Use pipenv to install packages like Keras?

Is it possible to use pipenv to install python packages like Keras for use with KNIME?
I rather like the idea of pipenv and wonder if there are any pointers how to use it along with KNIME?
Thanks

Hi knick,

Yes, this should be possible (in fact, I just tried it myself and it worked :slight_smile:).

Here’s what I did (Linux):

pipenv --three # or --two
pipenv install keras tensorflow pandas
pipenv shell

pipenv shell will output the command to activate its virtual environment. The command should look like this:

Launching subshell in virtual environment…
 . /home/<user>/.local/share/virtualenvs/<project-name>-E4Euwy4K/bin/activate

You can now take the example script from our Python setup guide for conda and replace the line source activate py35_knime (Linux/Mac) or the respective part of the the line @CALL activate py35_knime || ECHO Activating py35_knime failed (Windows) by the command that was output by pipenv shell. You can also remove the line where Anaconda is added to the PATH variable. Then save the script. If you’re on Linux/Mac, you’ll also have to make it executable.

In my case, the script for use with pipenv looks like this:

#! /bin/bash
. /home/marcel/.local/share/virtualenvs/marcel-E4Euwy4K/bin/activate
python "$@" 1>&1 2>&2

All that’s left to do now, is to point to that file via File > Preferences > KNIME > Python > Path to Python (2|3) executable. The preference page should give you immediate feedback on whether the environment was detected.

Please let me know if you experience any problems :slight_smile:.

Marcel

4 Likes

Hi Marcel, thanks for the quick answer! Unfortunately I get stuck at Locking [packages] dependencies…
Might have something to do with my connection, on the road. Will try it again when I get home

Ok, installation finished and I did $ pipenv shell and put the resulting path in a .sh file which I made executable and entered in knime. But then Knime shows an error: python cant open file, and then the path to my knime plugins directory and the file PythonKernelTester.py and [Errno 2] No such file or directory
Any ideas?

Hi knick,

Can you double-check whether the file PythonKernelTester.py at the given path actually exists? Maybe something went wrong during the installation of the Python extension.
Can you also take a look at your knime.log file and see if there is any additional information? Thanks!

Additionally, you could try to run the command you got from pipenv shell outside of KNIME, e.g., by simply pasting it to a command line. Maybe there’s something wrong.

Marcel

The file exists and the command from pipenv shell seems to work. Looking at the last lines in the logfile, it may contain related stuff. But It is 8mb!

Thanks for double-checking. Can you maybe upload these last lines of the file? Or delete the log file and reproduce the python cant open file error again. Then only the relevant lines should be in the (now much smaller) log file.

This should be the relavant part

2018-10-27 13:25:55,201 : DEBUG : KNIME-ConfigurationArea-Checker : ConfigurationAreaChecker :  :  : Configuration area check completed in 0.0s
2018-10-27 13:27:24,467 : DEBUG : Thread-13 : PythonKernelTester :  :  : Error occurred during testing Python2 installation
2018-10-27 13:27:24,468 : DEBUG : Thread-13 : PythonKernelTester :  :  : Executed command: /home/user/st/ml/knime/keras-test/activate-env.sh /home/user/st/ml/knime/knime_3.6.1.linux.gtk.x86_64/knime_3.6.1/plugins/org.knime.python2_3.6.1.v201808311614/py/PythonKernelTester.py 2.7.0
PYTHONPATH=:
PATH=/home/user/.local/bin:/home/user/.nvm/versions/node/v10.1.0/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/user/built/bin
Error during execution: python: can't open file '“/home/user/st/ml/knime/knime_3.6.1.linux.gtk.x86_64/knime_3.6.1/plugins/org.knime.python2_3.6.1.v201808311614/py/PythonKernelTester.py': [Errno 2] No such file or directory

Raw test output: 


2018-10-27 13:27:24,526 : DEBUG : Thread-14 : PythonKernelTester :  :  : Error occurred during testing Python3 installation
2018-10-27 13:27:24,526 : DEBUG : Thread-14 : PythonKernelTester :  :  : Executed command: /home/user/st/ml/knime/keras-test/activate-env.sh /home/user/st/ml/knime/knime_3.6.1.linux.gtk.x86_64/knime_3.6.1/plugins/org.knime.python2_3.6.1.v201808311614/py/PythonKernelTester.py 3.0.0
PYTHONPATH=:
PATH=/home/user/.local/bin:/home/user/.nvm/versions/node/v10.1.0/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/user/built/bin
Error during execution: python: can't open file '“/home/user/st/ml/knime/knime_3.6.1.linux.gtk.x86_64/knime_3.6.1/plugins/org.knime.python2_3.6.1.v201808311614/py/PythonKernelTester.py': [Errno 2] No such file or directory

Raw test output:

Does the log say anything interesting?

Sorry for the delay.
Not really. There’s a double quotation mark at the beginning of the path that Python complains about ('“/home/user/st/ml...'). Maybe that’s just an error in the print output but it isn’t displayed when I try to execute some nonesense like this: python lala.py (output: python: can't open file 'lala': [Errno 2] No such file or directory). Can you try this yourself and see if there’s a double quotation mark?

Could you also share your activation script (activate-env.sh)? I just want to make sure everything’s alright there.

Also, did you already try to execute the entire command /home/user/st/ml/knime/keras-test/activate-env.sh /home/user/st/ml/knime/knime_3.6.1.linux.gtk.x86_64/knime_3.6.1/plugins/org.knime.python2_3.6.1.v201808311614/py/PythonKernelTester.py 3.0.0 from a shell?

1 Like

Solved! The problem was I copied the below. For some reason " had turned into “.

2 Likes

Oh, I didn’t expect that. I fixed my post. Thanks for working it out!

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.