Hello KNIME Community,
I am trying to follow a simple LSTM tutorial from Kaggle which uses an IBM stock dataset. For it I have created a LSTM network in KNIME which did work before I added lagged values. Now after having added lagged values it is giving a content accuracy of 0. I’ve tried messing around with the different optimization parameters and activation functions but to no avail.
Keras model accuracy constant 0
Hi @Repletion,
Looks like you’re doing a numeric prediction on a double value there? Just guessing based on the accuracy and the use case.
If that’s true then seeing 0% for accuracy makes sense as it is calculated based on when your prediction exactly matches the correct value.
For numeric prediction models it’s more useful to look at the second tab in the learning monitor, the Loss tab. That will show you how your loss function (hopefully) decays over time. MSE being a popular one for tasks like this. That loss function is set in the learner node right below were you choose the target data.
Configuring Loss Function:
Viewing Loss Function in Monitor:
Hope this helps! Let me know if that was the problem.
1 Like
Unfortunately that is not the problem, yes I am running with double values. But I have been using MSE/MAPE/MAE all the time, cycle between them with no result. Also the loss is not falling like a logistic growth curve. I have attached some screenshots below.
Note: I have two screenshots of the loss curve, the first showing the total curve, the next being zoomed in to filter out the extreme outliers.
Fully zoomed in loss curve:
Hmm, Thanks for sharing those screenshots.
I see that your loss function is behaving oddly.
The first thing that comes to mind is to normalize your input features in some way. It can be difficult to train a network when the inputs are on such massively different scales(you get some weird gradients). Your volume feature in particular being many orders of magnitude larger than the rest. Even just dividing that value by 100,000 might help your model’s training process.
If that doesn’t help exploring different network architectures and activation functions would be the next things try.
2 Likes
Taking into consideration that the volume might be affecting the loss and accuracy of the model in a negative way I have opted to completely remove it from the model for testing purposes, the result being… the same as before 
The loss function still doesnt go towards 0 and the accuracy remains constant 0.
As an extra thing I have tried normalizing the data aswel however without any sucess. I suspect the problem being in using lagged values to predict a current value, but how else would you do forecasting with an LSTM network in KNIME?
If any kind soul could perhaps look at the workflow, maybe there is a KNIME newbie mistake in there?
Hmm, first thing I may try from here is using more than just 1 past lagged value from each feature in your model. Maybe 10, 25, 100, try a few values.
That’ll produce a fairly large input space but it can be more effective than solely relying on the LSTM unit to produce the useful “memories”.
If you don’t see much progress there can you share more information about your network architecture? Layers, number of neurons, activation functions.
Maybe we can troubleshoot a bit more from that direction.
I think it’s just a matter of getting the right network architecture and Keras configurations figured out. Everything is running through the trainer so the “KNIME” part of the KNIME workflow seems to be going well!
1 Like
From a quick look at your attached workflow, looks like you are missing a step (create column collection) that should occur prior to training the network.
To see how to do that, take a look at this workflow:
You may also want to add additional lags to your data prior to training.
Hope this helps.
2 Likes