I am exploring the use of Knime framework for machine learning development. I would like to learn from the community what the best practices are when using extensions like Keras.
Specifically, I would like to know when there is a Python related error in running the node, how would you best advice someone to go about debugging in the Knime framework?
For example, I tried to the MNIST example in the Keras extension and I got some error ins the DL Keras Network Learner:
ERROR DL Keras Network Learner 0:89 Node input/target data size does not match the expected size of network input/target ‘conv2d_1_input:0’. Neuron count is 784, batch size is 200. Thus, expected input/target size is 156800. However, node input/target data size is 2000. Please check the column selection for this input/target and validate the node’s input/target data.
ERROR DL Keras Network Learner 0:89 Execute failed: An error occured during training of the Keras deep learning network. See log for details.
So I suppose this will be the first of many debugging I will encounter down the road, so I would like to educate myself before embarking on something entirely new.
I am a bit confused by your title.
Do you want to develop new KNIME nodes that build on our Keras integration or do you want to use the nodes available in the Keras integration?
Concerning your example, the best practice I can recommend is to go to the KNIME log (View->Open KNIME log), search for the full stack trace (it should be somewhere close to the bottom of the log) and see if that already helps you to figure out the problem.
If it doesn’t help you, I’d recommend to open a thread on the forum that contains
An explanation of what you are trying to achieve
Ideally a sample workflow
The stack trace of the error extracted from the KNIME log.
A very common problem that many new users of the Keras integration have is that some python packages are missing, so here are the libraries you will need:
Keras (obviously)
Tensorflow or CNTK (as backend for Keras)
h5py (used by Keras to save and load models)
pandas (used by us to represent data in python)
The one that is easiest to miss is the h5py in my experience.
As for the specific error you mentioned, it isn’t Python related. It sounds more like somehow the wrong columns ended up being selected under the input data tab of the Keras learner node. Please make sure that only the image column is selected there. Also check that - under the training target tab - the label columns (columns 0 to 9) are the ones that are selected.
I tried to open the DL Keras Network Learner, and I got this message box. I am using the example work-flow as you can see in the tab. I hope it’s just a newbie mistake.
Has anybody else tried to run the MNIST example using the Keras extension?
Thanks for the advice. I will open a new thread regarding the sample workflow used in the Keras extension. I installed keras integration using the conda environment, so I have reasonable confidence it is installed properly.
This particular error is caused because the DL Python Network Creator has not run yet. It should disappear once you run the creator and by this load your network.
This seems to be an issue on our side. I’ll have a look at the example workflow.
What version of KNIME are you using?
If you still want to try out the workflow in the meantime, you can try to replace the Keras learner node with a fresh one from the node repository (simply drag the new learner node onto the old one). Select the training and target columns as described in my previous post and select “Categorical crossentropy” as loss function under the “Target Data” tab.
I just ran the example workflow on my local machine using KNIME 3.5.2 which seemed to work properly. Could you provide me with your knime.log to help tracking down the issue? Thanks!
did you install the KNIME Image Processing - Deep Learning extension? This extension offers the converters necessary to feed images into the deep learning node. You can find it on the Community Contributions (Trusted) update-site.
