I am trying to run a neural network with CuDNN Layer nodes and I am having some problems with NVIDIA driver and conda environment.
I am using a Tesla V100 GPU on GCP and those are the problems that I am facing:
The NVIDIA driver for CUDA Toolkit 10.0(Ubuntu 18.04) is open source and after I installed it and used the command nvidia-smi is not showing the GPU (I am having one single GPU in the system), but after selecting a higher proprietary version of the driver the GPU appears in the nvidia-smi output command.
It’s OK to use NVIDIA driver CUDA Toolkit 10.1 or 10.2?
When I’m creating a conda environment from KNIME UI for GPU Anaconda is installing cuda-toolkit-9.2-0 and other version of cuDNN package in the Anaconda folder. Also, I tried to install the cuda-toolkit-10.0 and the cuDNN package version v7.6.0 CUDA 10.0 by using conda commands from the terminal before creating any environemnt from KNIME UI and the same behavior.
Another question: KNIME can run on TPU hardware units?
I managed to configure the NVIDIA driver CUDA Toolkit 10.0 after reinstalling it several times and seems to be stable.
I created a conda environment from the KNIME UI and the GPU usage is around 1-2%. I am having one single Tesla V100 GPU in the system and I am having 2 hidden Keras CuDNN LSTM Layer node with max 100 neurons each of them. I am using a Loop for trying different parameters. It’s OK to have the GPU usage at 1-2%? Or it’s a bad configuration for CUDA or Keras/Tensorflow?
I attached the network topology.
a V100 is quite powerful so it is possible that your network only causes such a low GPU usage.
As far as I know the CuDNN layers will only run in a GPU setup (at least when I played with them that was kind of a bummer, when I wanted to show my results on a non-GPU machine…) but to be sure, can you check if there is a process showing up in the nvidia-smi when you are training?
If it does than you should be good to go
Regarding the TPUs I have to admit that I don’t know if anyone has tried that yet and I will have to investigate a bit to figure out if it is possible.
