DL4J Neural Network vs. RPROP algorithms

Hello everyone!

I have a problem with the efficiency of the DL4J algorithm for neural networks. More specifically, I want to train a neural network so that it correctly predicts the values from some dataset (so I have a regression problem). I have used as parameters the following values: 

-Optimization algorithm: Stochastic Gradient Descent

-Updater: Nesterovs

-0.0001 as L2 regularization

-learning rate: 0.01

-23 epochs on a dataset containing 307 input samples

-loss function: Sum of errors.

 

Now my question is why this configuration (and all the others I tried) is orders of magnitude slower and only about equally accurate as the standard RPROP algorithm in KNIME? I need to use DL4J since I want to perform computations on a GPU which doesn´t work with the standard algorithms like RPROP.

 

Also one more question: How is it possible to use an already trained neural network to compute outputs for previously unseen input values? Say, I trained my network and want to predict the corresponding output for some new input, which node will facilitate this?

 

Thanks in advance,

Leon

 

 

HI Leon,

the speed of training mostly depends on the used network architecture. What kind of network did you use for the DL4J integration and how did you configure the RPROP Learner node? Also, if you want to can attach your workflow and I could have a look at it.

Regarding your last question: Do you want to load a pretrained network from the Internet? For that I would point you to our Keras Integration. If you mean a DL4J network, just use one of the DL4J Predictor nodes (in your case the regression predictor).

Cheers

David

Hello David,

 

thanks for your answer. I used a 2 hidden layer network with 10 neurons each. See attached my workflow. It looks like a mess because I wanted to normalise and renormalise all columns and still see the real and predicted target values next to each other, please excuse this ;-). 

My second question was regarding the same network I attached. I want to first train it and then load a new file with unseen data and predict the target values for the data set contained in the file. I still do not exactly understand how to do this to be honest. How do I use the predictor node after applying the file reader?

 

Best regards,

Leon

I forgot to say that I configured the RPROP algorithm in the same way as the DL4J network (with regard to iterations, layers and # of nodes).

Hi Leon,

I had a short look at your workflow and could identify several things:

1. You configured the DL4J Learner to do 4000 iterations. For DL4J this means something else than for the RPROP Learner. In DL4J this is the number of updates per batch which will be done and in the RPROP Learner the number of epochs. If your batch size covers the whole dataset then iterations (RPROP) is the same as epochs (DL4J)

2. The general performance strongly depends on the used optimization Algorithm. You configured the DL4J Learner to do Conjugate Gradient Descent which is very slow in comparison to Stochatic Gradient Descent. If you choose the latter, The DL4J Learner executes very fast. Explaining the difference between these two algorithms would extend the scope of this post but you can easily look it up if you are interested.

3. You can create predictions exactly like you did it in the attached workflow with the DL4J Regression Predictor node. Just make sure that the column names match and that the data is normalized (use the same normalization you used for learning) before you pass it to the Predictor node.

4. There is a view on the DL4J Learner node which shows the training error. This is very useful to monitor the training process. I saw that it is NaN so the network probably didn't learn anything. Consider reducing the learning rate to fix this.

How big is your dataset in total? If it is fairly small (like in the example) then it would probably easier to use the RPROB Learner and not the DL4J Integration. For DL4J there are many hyper parameters which can be hard to configure if you are new to the topic.

Cheers

David

Hello David,

 

thanks for your very helpful answer. I think that given your last lines it will probably be for the best to use the RPROP algorithm although it is not easily executable on a GPU. My dataset contains about 20.000 samples and about 15 input nodes, so it is certainly not very large. Transfering this to a GPU might not be worth the effort.

 

Cheers,

Leon

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.