KNIME on nvidia 3090 30xx series

KNIME Extensions
1

Hey There,

i have just tried to run KNIME on the new nvidia 3090.
I am not sure if i got this correctly, but it seems like i need cuda 11, cudnn 8 and a later version of TF.
The installation script for KNIME’S python environment is using an old version of TF which does not support the latest driver to use the new gpu architecture.
Any experience or ideas on that?
Cheers
Niko

2

Hi Niko,
We haven’t tested it yet, but you might be able to install TF 2.4 in the environment and use our Tensorflow 2 integration with that. However, our Keras integration still uses TF 1 and you won’t be able to use the new GPU there, I believe.
Kind regards,
Alexander

3

Hi Alex,

thanks for your reply!
Unfortunately i am using keras in my workflows.
Is there a plan to provide later keras versions in KNIME soon?

I found a nice post about nvidia providing TF 1.15. But only for linux if i see that correctly.

Would that one work with KNIME and current Keras (2.1.6 in installation script for KNIME’s python environment) verision?
If so - how do I install that with anaconda on windows 10 :S ? Any experience with that??

Best,
Niko

4

Hi Niko,

This post looks promising. I did not know about the Nvidia TensorFlow builds but they look like a good option to use our Keras integration with the newest GPUs. I am confident that this would work but we have not tested it.

You can take a look at the section about the “Manual Python Installation” at the KNIME Deep Learning Integration Installation Guide. After doing the “Manual” environment creation from the KNIME Python Integration Guide. You can install tensorflow in the environment using pip as described in the article. Then, you can install keras==2.2.4 and h5py using also using pip. This should leave you with a Python environment that can be used by KNIME (if there are no incompatibilities between Keras 2.2.4 and TF 1.15).

How did you get to Keras 2.1.6? I think 2.2.4 is the version we currently recommend (and test).

Let me know if you have success with this or if you need more advice.

2 Likes
5

Hi @bwilhelm,

so I finally moved to TF 2 which works with cuda 11 on nvidia 3090.
Unfortunately i made the magic happen on one machine. But i am not able to reproduce it on another machine.

My current .yml file looks like this (NOT working since KNIME complains about missing pyarrow, or wrong numpy version …i just could not make it work again).

name: KNIMEenv_TF2 # Name of the created environment
channels: # Repositories to search for packages

  • defaults
  • anaconda
  • conda-forge
    dependencies: # List of packages that should be installed
  • python=3.6 # Python
  • pandas=0.23 # Table data structures
  • jedi=0.13 # Python script autocompletion
  • python-dateutil=2.7 # Date and Time utilities
  • numpy=1.15 # N-dimensional arrays
  • cairo=1.14 # SVG support
  • pillow=5.3 # Image inputs/outputs
  • matplotlib=3.0 # Plotting
  • pyarrow=0.11
  • scipy=1.1
  • scikit-learn
  • gitpython
  • pip
  • pip:
    • xgboost==1.2.0
    • tensorflow-gpu==2.4.1

I also tried the “New environment” button in the Preferences->KNIME->Python Deep Learning-> Tensorflow2. Which did not create a working environment.

Could you help me with that anaconda environment - how do i have to set it up to use KNIME with tensorflow 2 and cuda 11?
Best
Niko

6

… adding one more question to that:
is it possible to use KNIME with the latest pandas version and TF2 ?
using pandas=0.23 leads to trouble reading pickle files created with a later version of pandas…

→ can you provide a .yml file for using TF2 and an up to date version of pandas ?

7

Hi @niko314159,

Using pandas > 0.23 sadly won’t work. This is a known issue with the deep-learning integration that we did not find time to fix yet. We just re-prioritized this, so hopefully, you will be able to use newer pandas versions soon.

I managed to use the following environment in KNIME. I just tested it with some example workflows for TF2 and it seemed to work. Since it’s using TensorFlow 2.4.1 it should work with a 3090.

name: py3_knime_tf2_3090
channels:
- defaults
- anaconda
- conda-forge
dependencies:
- python=3.7          # Python
- pandas=0.23         # Table data structures
- jedi=0.13           # Python script autocompletion
- python-dateutil=2.7 # Date and Time utilities
- numpy=1.16.6        # N-dimensional arrays
- cairo=1.14          # SVG support
- pillow=5.3          # Image inputs/outputs
- matplotlib=3.0      # Plotting
- pyarrow=2.0         # Arrow serialization
- cudatoolkit=11.0    # Can be removed if installed globally
- cudnn=8             # Can be removed if installed globally
- pip
- pip:
  - xgboost==1.2.0
  - tensorflow==2.4.1
1 Like
8

Hi @bwilhelm ,

thanks for your reply!
So i have restarted the machine, prayed for good karma, used your script for installation and i still get “library pyarrow is not properly installed. Details: DLL load failed: The specific module could not be found” .

If i use “flatbuffers collumn serialization” the environment passes the test. But i end up with
“Execute failed: Builder object has no atrribute CreateByteArray”
when i try to run it.

Currentlly i am using KNIME 4.2.0. Do i need to upgrade to newer KNIME version?

Best
Niko

9

Hi @niko314159,

There are some known issues with loading DLLs in our Python integration in KNIME versions prior to 4.3.2 (which is the most recent version). Could you try upgrading to that version and see if the error persists?

Marcel

10

Hi @MarcelW,

thx for the reply!
Upgrading to KNIME 4.3 solves the pyarrow issue.
So i moved on to the next error message…
DL python network creator throws the error:

"execute failed: No deep learning network type associated with Python type ‘class…’

I am using code for TF 2 from the homepage (TensorFlow 2 quickstart for beginners  |  TensorFlow Core) :
import tensorflow as tf
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation=‘relu’),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10)
])

output_network = model

“print(‘tf version’, tf.version)” in the DL python network creator shows
tf version 2.4.1

Any ideas on that?
Greetings
Niko

1 Like
11

Hi @niko314159,

Can you confirm that the KNIME Deep Learning - TensorFlow 2 Integration is actually installed in your current KNIME instance? I tried to reproduce the error message on my machine and I only managed to do so after having uninstalled the integration.

Marcel

12

Hey @MarcelW,
i was not aware that “KNIME Deep Learning Tensorflow 2” plugin needs to be installed for other nodes (not from that package) to work with TF2.
I have installed the plugin and now everything works fine.
Thanks :slight_smile: !!
Niko

P.S.: what is the timeline on new pandas version ?

2 Likes
13

Hi @niko314159,

The idea is that the TensorFlow 2 integration (and other ones such as the Keras integration) does not only extend the set of available nodes but also extends the functionality of the DL Python scripting nodes. (That is, as soon as a Python-based deep learning integration is installed, the node “understands” the models associated with the integration.) So the scripting nodes can be thought of as “universal” nodes that work with all Python-based deep learning integrations.
But this is actually not the first time that this approach has confused users. So maybe introducing dedicated scripting nodes per deep learning integration instead would make more sense.

I cannot promise anything specific, unfortunately, but we tentatively scheduled the upgrade for one of the next releases. So hopefully this should not take too long.

Marcel

14

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.

15

It’s been a while, but I quickly wanted to let you know that beginning with KNIME Analytics Platform 4.5.0, which was released two days ago, the deep learning nodes now also support the more recent versions of pandas and Python in general.

3 Likes