I ran the Time Series Prediction workflow on my machine just to make sure that it is not somehow malfunctioning in general, and it worked well. Can you consistently reproduce the error (e.g., even when trying to rerun the Keras Network Learner node a couple of times, or resetting the entire workflow and then rerunning, etc.)?
Also, would you mind uploading your entire KNIME log? I am hoping to find some more details buried elsewhere, possibly not in the proximity of the entries that report the error.
Correct - I can consistently reproduce the error every time I run it, either from the start (as-is), or just the Learner node (when it encounters the error).
Log attached. Please do let me know if this is what you were expecting re: entire log - a somewhat new to KNIME keras_error_log.txt (122.0 KB)
Yes, that is what I meant. Sadly it does not provide any more information than the previous one.
No, that is alright (although confusing). I also got this message when I opened the workflow, it should not be related to the error.
Altogether, I am a little stumped right now. Best we can do, is to gradually try some more complex combinations and see if/when things break. I uploaded another workflow here. Could you again try and see if it works for you?
Sorry that this is not a really satisfying procedure right now, I hope it leads to a solution eventually.
Sorry for the late reply. I will get back to you as soon as possible with a further test workflow.
If you already want to do some investigation on your own, the next step would probably be to let the entire training of the LSTM network run directly in Python, e.g., in a single Python Script (1⇒1) node (or even outside of KNIME) to see if it is a problem with the underlying Python installation (Keras, TensorFlow, …) or with the KNIME Keras integration.
If you choose to do it in the Python node, please make sure to select the same Python 3 environment under File > Preferences > KNIME > Python as is selected on File > Preferences > KNIME > Python Deep Learning.
Hi Marcel, I connected the Python script node to the last node of the workflow, and got the same error.
Separately, I was also able to replicate the same error across 2 different Macs running different versions of MacOS (Mojave, Catalina), but not when I run the same workflow on a PC.
Please let me know what we can do to further diagnose/resolve.
I uploaded another workflow here. It contains a single Python scripting node that performs the entire model building and training that was previously done by the Keras layer nodes (Input, LSTM, Dense) and the Keras Network Learner node in the other workflows. This effectively takes the KNIME Keras integration out of the equation.
Could you try and execute this and see if it logs any more descriptive errors, given that the workflow also fails?
Additionally, if the workflow fails, could you:
First, copy and paste the Python code from the Python Source node’s dialog into a text editor and save it as a Python file (e.g., “train_lstm.py”)
Then, open a terminal and cd into the directory where train_lstm.py is located
In the same terminal, activate the deep learning conda environment by running conda activate py3_knime_dl
Again in the terminal, run the Python file via python train_lstm.py
And see what happens?
This, in turn, takes KNIME completely out of the equation.
I am also pretty sure that this is a Mac-specific problem. The question is just whether the problem only occurs in combination with KNIME or also when using Python directly. That is what I am trying to find out with those additional steps. Unfortunately, I have not found the time yet to replicate the problem on one of our Macs.
Do you know if Keras (and other required packages for this workflow) are installed when KNIME creates the default Python environment? Or should this workflow point to the default Deep Learning environment created by KNIME?
Here’s what my Python environment screen looks like:
When activating the py3_knime_dl environment via command line, am getting this error:
xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun
Thank you Marcel. I was able to activate the py3_knime_dl environment. Here’s the error I now get from running in Python:
Using TensorFlow backend.
Epoch 1/50
2020-02-21 11:56:09.360065: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2020-02-21 11:56:09.360368: I tensorflow/core/common_runtime/process_util.cc:69] Creating new thread pool with default inter op setting: 4. Tune using inter_op_parallelism_threads for best performance.
OMP: Error #15: Initializing libiomp5.dylib, but found libiomp5.dylib already initialized.
OMP: Hint: This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.
