knime-sdk keras node problem

syinkim · July 5, 2019, 2:09am

I was testing keras-network-learner node. (03_Train_MNIST_classifier)
But suddenly keras node executeion failed since a few day ago.
The installer version KNIME (3.7.2) will run keras-node normally and will run problem when run as KNIME-SDK (nightly).
The error stack is as follows.
Anyone please help me now what’s problems.

ERROR KNIME-Worker-2 DLKerasLearnerNodeModel W0705 14:24:04.790926 20308 deprecation.py:323] From C:\Users\kisy777\Anaconda3\envs\keras\lib\site-packages\tensorflow\python\ops\nn_impl.py:180: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where

ERROR KNIME-Worker-16 Node Execute failed: An error occured during training of the Keras deep learning network. See log for details.
java.lang.RuntimeException: An error occured during training of the Keras deep learning network. See log for details.
at org.knime.dl.keras.base.nodes.learner.DLKerasLearnerNodeModel.handleGeneralException(DLKerasLearnerNodeModel.java:720)
at org.knime.dl.keras.base.nodes.learner.DLKerasLearnerNodeModel.executeInternal(DLKerasLearnerNodeModel.java:696)
at org.knime.dl.keras.base.nodes.learner.DLKerasLearnerNodeModel.execute(DLKerasLearnerNodeModel.java:303)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:567)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1192)
at org.knime.core.node.Node.execute(Node.java:979)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:559)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
Caused by: org.knime.python2.kernel.PythonIOException: W0705 10:19:01.867142 17844 deprecation.py:323] From C:\Users\kisy777\Anaconda3\envs\keras\lib\site-packages\tensorflow\python\ops\nn_impl.py:180: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where

at org.knime.dl.python.core.DLPythonAbstractCommands$DLPythonNetworkTrainingTaskHandler.handleSuccessMessage(DLPythonAbstractCommands.java:831)
at org.knime.dl.python.core.DLPythonAbstractCommands$DLPythonNetworkTrainingTaskHandler.handleSuccessMessage(DLPythonAbstractCommands.java:1)
at org.knime.python2.kernel.messaging.AbstractTaskHandler.handle(AbstractTaskHandler.java:86)
at org.knime.python2.kernel.messaging.DefaultTaskFactory$DefaultTask.runInternal(DefaultTaskFactory.java:256)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

bwilhelm · July 5, 2019, 6:54am

Hi @syinkim,

I cannot identify the problem in the error log. I do not think that:

nn_impl.py:180: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where

is the real problem because this looks like a warning to me. You are not using TensorFlow 2.0, right?

Do you use the same Python environment in both of you setups? Can you post a list of installed packages (conda list -n <env name>)?

What exactly do you mean with “KNIME-SDK”? Do you use the target definition from knime/knime-sdk-setup or from knime-ip/knip-sdk-setup?

Best,
Benjamin

syinkim · July 5, 2019, 7:20am

@bwilhelm Thank you for your attention.

You are not using TensorFlow 2.0, right?

I use to Tensorflow is 1.14.

Do you use the same Python environment in both of you setups? Can you post a list of installed packages ( conda list -n <env name> )?

My conda list here : conda_list.txt (17.6 KB)

What exactly do you mean with “KNIME-SDK”

I’m using the target definition KNIME-AP-complete.target from knime/knime-sdk-setup.

bwilhelm · July 5, 2019, 7:50am

I cannot reproduce your problem yet.

The KNIME-AP-complet target definiton does not contain the Image Processing community contribution which is needed for the example workflow. Did you add it? If so, did you also add the “KNIME Image Processing - Deep Learning Extension”?

Can you maybe post a larger portion of your log? Maybe the real cause of the problem is hidden earlier in the log.

syinkim · July 5, 2019, 8:24am

@bwilhelm

The KNIME-AP-complet target definiton does not contain the Image Processing community contribution which is needed for the example workflow. Did you add it? If so, did you also add the “KNIME Image Processing - Deep Learning Extension”?

I added target definition url ‘KNIME Community Extensions (Trusted) - http://update.knime.com/community-contributions/trusted/4.0’ for Image Processing.
KNINE-AP-complete%20(ImageProcessing)

Can you maybe post a larger portion of your log? Maybe the real cause of the problem is hidden earlier in the log.

All logs is here. Keras mnist error log.txt (6.9 KB)
Only ‘Keras Network Learner node’ is a running log.

bwilhelm · July 5, 2019, 9:35am

@syinkim,

Perfect. I now managed to reproduce the issue. It seems to be related to TensorFlow 1.14 and changes that we made to the message passing + error handling of executed Python code.
I will dig deeper into the issue and report back.

For now you can create a conda environment with exactly the Python library versions that we officially support (and test) using the new “Python Deep Learning” preference page:
https://docs.knime.com/2019-06/deep_learning_installation_guide/index.html#_configure_python_for_knime_deep_learning

Best,
Benjamin

l.thomas · March 27, 2020, 12:24pm

Hello guys,
I also come across this issue. Downgrading to tf 1.12 indeed fix it.
I had to downgrade from python 3.7.x to 3.6.9 to make this possible for some reason.

Is there a github issue or so that one can follow?