Integration of Python and Keras (in Anaconda) in to Knime for MAC

I am having a lot or trouble trying to install and integrate Python and Keras with Knime for a MAC Book Pro.

I can run Keras and Python fine inside my Anaconda environment and I have made the environment in Anaconda = py35_knime and can see python and all extension in the py35_knime environment in Anaconda?

I have looked at:
https://www.knime.com/blog/setting-up-the-knime-python-extension-revisited-for-python-30-and-20
https://www.knime.com/forum/knime-labs-general/knime-keras-integration-issue

But still have a lot of trouble with the integration to make it work in Knime

In the Preferences keep getting: "Could not find Python executable at the given location … "

Is there someone who can spell this out for me step by step … what am I missing? Thank you!!!

conda list -n py35_knime

packages in environment at /Users/xxxxx/anaconda3/envs/py35_knime:

ca-certificates 2017.08.26 ha1e5d58_0
certifi 2017.11.5 py35hd00889a_0
intel-openmp 2018.0.0 h8158457_8
jedi 0.11.0 py35_2
libcxx 4.0.1 h579ed51_0
libcxxabi 4.0.1 hebd6815_0
libedit 3.1 hb4e282d_0
libffi 3.2.1 h475c297_4
libgfortran 3.0.1 h93005f0_2
mkl 2018.0.1 hfbd8650_4
ncurses 6.0 hd04f020_2
numpy 1.13.3 py35hfd7066c_0
openssl 1.0.2n hdbc3d79_0
pandas 0.22.0 py35h0a44026_0
parso 0.1.1 py35hbda7c10_0
pip 9.0.1 py35h33ce766_4
python 3.5.4 he720263_23
python-dateutil 2.6.1 py35h10515e0_1
pytz 2017.3 py35heeb7564_0
readline 7.0 hc1231fa_4
setuptools 36.5.0 py35h52cde6a_0
six 1.11.0 py35h39a4c60_1
sqlite 3.20.1 h7e4c145_2
tk 8.6.7 h35a86e2_3
wheel 0.30.0 py35h5c0b906_1
xz 5.2.3 h0278029_2
zlib 1.2.11 hf3cbc9b_2

Hi @tw349,

It seems that there is a problem with your Python configuration. Can you paste the .sh script you created and pointed the python preference page to? This will help us debugging your problem!

Best,

Christian

OK, maybe that is the part I am not clear on … the exact script, where to place it etc. etc. … sorry can you put a simple step by step explanation of these instructions … thank you.

My shell script for the Python environment (is below) I named this script py35.sh, tried to make it executable (“chmod gou+x py35.sh”), and put it in my home directory?? (maybe I put in in the wrong place - I put it in the /Users/xxxxx/anaconda3/py35.sh directory??)

#! /bin/bash

export PATH="/Users/xxxxx/anaconda3/bin:$PATH"

source activate py35_knime
python “$@” 1>&1 2>&2

Instructions at: https://www.knime.com/blog/setting-up-the-knime-python-extension-revisited-for-python-30-and-20 say that:

In order to use this new Python environment from inside of KNIME, you need to create a script (shell script on linux and the Mac, bat file on Windows) to launch it.

If you are using linux or the Mac, here’s an example shell script for the Python environment defined above:

#! /bin/bash

start by making sure that the anaconda directory is on the PATH

so that the source activate command works.

This isn’t necessary if you already know that

the anaconda bin dir is on the PATH

export PATH=“PATH_WHERE_YOU_INSTALLED_ANACONDA/bin:$PATH”

source activate py35_knime
python “$@” 1>&1 2>&2

You will need to edit that to replace PATH_WHERE_YOU_INSTALLED_ANACONDA with wherever you installed Anaconda. I named this script py35.sh, made it executable (“chmod gou+x py35.sh”), and put it in my home directory.

My Anaconda3 direectory:

anaconda3 xxxx$ ls

Anaconda-Navigator.app
node-webkit
Applications
org.freedesktop.dbus-session.plist
Launcher.app
phrasebooks
bin
pkgs
conda-meta
plugins
doc
py35.sh
envs
python.app
etc
qml
fonts
resources
include
sbin
lib
share
libexec
ssl
man
translations
mkspecs
var

Hi @tw349,

Did you also set the path to the .sh script in the KNIME preferences? You can set the path under File -> Preferences -> KNIME -> Python.

Best,

Christian

Yes and seems to find path to executable Python 3.5.4 now and have set default for:

/Users/xxxxx/anaconda3/envs/py35_knime/bin/python3.5

Do I set path in Knime preferences to .sh script or the above?

Where did you put the .sh file exactly and is the below script correct (with the quotation marks etc)?

#! /bin/bash

export PATH="/Users/xxxxx/anaconda3/bin:$PATH"

source activate py35_knime
python “$@” 1>&1 2>&2

Still can’t get Cats/Dogs example to work?

EXAMPLES Server: 04_Analytics/14_Deep_Learning/02_Keras/04_Cats_and_Dogs/02_Train_simple_CNN*

see log below:

ERROR NativeLibBundleActivator /Applications/KNIME 3.5.2.app/Contents/Eclipse/plugins/org.knime.knip.tess4j.bin.macosx.amd64_1.3.1.v201607080717/lib/macosx/amd64/libtesseract.3.dylib: dlopen(/Applications/KNIME 3.5.2.app/Contents/Eclipse/plugins/org.knime.knip.tess4j.bin.macosx.amd64_1.3.1.v201607080717/lib/macosx/amd64/libtesseract.3.dylib, 1): Library not loaded: /usr/local/lib/liblept.4.dylib
Referenced from: /Applications/KNIME 3.5.2.app/Contents/Eclipse/plugins/org.knime.knip.tess4j.bin.macosx.amd64_1.3.1.v201607080717/lib/macosx/amd64/libtesseract.3.dylib
Reason: image not found
ERROR NativeLibBundleActivator Could not load org.knime.knip.tess4j
WARN Create File Name 1:25:9 Selected directory ‘/private/var/folders/5p/1df2pd75557f_9n47t60wgbw0000gn/T/knime_tc_vsvi8udnnbfm’ cannot be accessed!
WARN Create File Name 1:26:9 Selected directory ‘/private/var/folders/5p/1df2pd75557f_9n47t60wgbw0000gn/T/knime_tc_rcm73hr5ptkj’ cannot be accessed!
WARN Create File Name 1:27:9 Selected directory ‘/private/var/folders/5p/1df2pd75557f_9n47t60wgbw0000gn/T/knime_tc_jtbzbspwok76’ cannot be accessed!
WARN Create File Name 1:28:9 Selected directory ‘/private/var/folders/5p/1df2pd75557f_9n47t60wgbw0000gn/T/knime_tc_clrpmukq2nox’ cannot be accessed!
WARN Qed Calculator 1:29 Could not find templates folder templates/ in plugin org.knime.knip.knimepython
WARN Create File Name 1:30:9 Selected directory ‘/private/var/folders/5p/1df2pd75557f_9n47t60wgbw0000gn/T/knime_tc_lrdcgez4xcet’ cannot be accessed!
WARN Create File Name 1:31:9 Selected directory ‘/private/var/folders/5p/1df2pd75557f_9n47t60wgbw0000gn/T/knime_tc_h7hzngsoezim’ cannot be accessed!
ERROR ServerRecommendationSettings Can’t load the requested node recommendations: Unexpected character (’<’ (code 60)): expected a valid value (number, String, array, object, ‘true’, ‘false’ or ‘null’)
at [Source: sun.nio.ch.ChannelInputStream@31d2060e; line: 1, column: 2]
ERROR NodeRecommendationManager Can’t load the requested node recommendations: Unexpected character (’<’ (code 60)): expected a valid value (number, String, array, object, ‘true’, ‘false’ or ‘null’)
at [Source: sun.nio.ch.ChannelInputStream@113ba3c3; line: 1, column: 2]
WARN Table Reader 0:84 Input file ‘/private/var/folders/5p/1df2pd75557f_9n47t60wgbw0000gn/T/knime_temp_space_xxxx_10346/02_Train_simple_CNN10363/02_Train_simple_CNN/…/preprocessed_150x150.tbl’ does not exist
ERROR DL Python Network Creator 0:222 Execute failed: Traceback (most recent call last):
File “/Applications/KNIME 3.5.2.app/Contents/Eclipse/plugins/org.knime.python2_3.5.0.v201712011355/py/PythonKernel.py”, line 491, in execute
exec(source_code, self._exec_env, self._exec_env)
File “”, line 7, in
ImportError: No module named ‘keras’

Hi @tw349,

you have to point the entry in the KNIME preference page on the .sh file and also make sure to install the keras extension into your python environment (see http://knime.com/deeplearning).

See ImportError: No module named ‘keras’.

I hope this helps,

Christian

OK, I think I got it to work … Keras and Tensorflow work for me for: knime://EXAMPLES/04_Analytics/14_Deep_Learning/02_Keras/01_Classify_images_using_InceptionV3

But now I have a new problem that have been trying many things but does not seem to work for the below?!?!

List Files Node Not Working For: /01_Pre-process_image_data to classify cats and dogs

knime://EXAMPLES/04_Analytics/14_Deep_Learning/02_Keras/04_Cats_and_Dogs
/01_Preprocess_image_data

After I download the cats/dogs data from Kaggle I can see the files of images in the folder with cats.jpg first then dogs.jpg files on the bottom of the file list (see screen shot on bottom). But when I try to upload the files (or zip folder) the first node (List Files) mixes up the order of the files … see first image below Then when I use the next node (Rule Engine) the first 12500 rows get classified as cats (no matter if they are a cat or dog image? Either need to fix the List Files node or change the java script in the Rule Engine node?? Any suggestions is greatly appreciated … thank you.

Can you try to add the Sorter node after the List Files? Does that help? If not, then we’ll upload a small workflow solving your problem.

Thank you … I think the sorter did work … see the below image:

BUT THE IMAGE CALCULATOR IN THE LOAD AND PREPROCESS IMAGES METANODE LOSES THE PHOTOS?!? See images below …

JUST FOR YOU INFORMATION THE IMAGE READER (TABLE) NODE JUST BEFORE THE IMAGE CALCULATOR NODE IS FINE … SEE BELOW.

The images are not normalized and therefore displayed in grey, but the information is there. You can use the “Image Normalizer” to normalize them. However, for learning it’s important to use them as they are.

thank you for getting back to me so soon!!!.. after when I save the processed data and try the workflow below the accuracy comes out at 50% … for some reason it is not seeing the images. Maybe I don’t have the tensorflow set up right for my MacBook Pro using High Sierra … don’t have GPU just CPU … see image and log below.
MacBookPro

knime://LOCAL/14_Deep_Learning/02_Keras/04_Cats_and_Dogs/02_Train_simple_CNN

WARN RowID 2:224 No row key column selected generate a new one
WARN RowID 2:224 No row key column selected generate a new one
WARN RowID 2:224 No row key column selected generate a new one
WARN PythonKernel Using TensorFlow backend.
WARN PythonKernel 2018-03-23 11:22:11.440348: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
WARN RowID 2:224 No row key column selected generate a new one
WARN PythonKernel Using TensorFlow backend.
WARN PythonKernel 2018-03-23 11:22:50.937370: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
WARN PythonKernel /Users/tracy/anaconda3/envs/py35_knime/lib/python3.5/site-packages/keras/models.py:252: UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually.
WARN PythonKernel warnings.warn('No training configuration found in save file: ’
WARN PythonKernel Using TensorFlow backend.
WARN PythonKernel 2018-03-23 11:23:43.319876: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA

WARN RowID 3:232 No row key column selected generate a new one
WARN RowID 3:232 No row key column selected generate a new one
WARN PythonKernel Using TensorFlow backend.
WARN PythonKernel 2018-03-24 20:29:20.682779: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
WARN PythonKernel /Users/tracy/anaconda3/envs/py35_knime/lib/python3.5/site-packages/keras/models.py:252: UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually.
WARN PythonKernel warnings.warn('No training configuration found in save file: ’

In the 04_Cats_and_Dogs Train _Simple_CNN model the DLPythonNetworkExecutor all the numbers come out above 0.5 and get predicted as dogs?? (see image below)… why? Why is this not working?

I’m not sure if I understand what you mean by “seeing”. Keep in mind, there is no tensorflow / keras support for GPU on macOS.

The “Simple CNN” isn’t expected to perform very well. However, there should be some “cats” in there as well :wink: How many epochs and how many batches did you actually train your network?

steps = 10
epochs = 5

what does the below error message mean? Does the compile section of code need updating ???

Using TensorFlow backend.
2018-03-26 09:59:00.481670: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
/Users/tracy/anaconda3/envs/py35_knime/lib/python3.5/site-packages/keras/models.py:252: UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually.
warnings.warn('No training configuration found in save file: ’

ERROR PythonKernel Exception ignored in: <bound method BaseSession.del of <tensorflow.python.client.session.Session object at 0x115736470>>
ERROR PythonKernel TypeError: ‘NoneType’ object is not callable

ok, maybe i figured it out … do I need to hit the Execute Script button in the Python Network Learner Node (i.e. manual execution) … says 5 Epochs with 2000 steps each … will take over 2 hours ETA?!? mac laptop can’t handle this?!?

Any comments or suggestions is greatly appreciated! Thank you.