AutoML - All Models Post Process Component Failing

rdlc1 · August 9, 2023, 6:08am

Hello,

I use knime in both mac (version 4.7) and pc (version 5.1). I loaded the mac workflow with automl in it to take advantage of XGBoost in PC. My problem is that using the same dataset and ML workflow, the PC workflow hangs at the Model to Cell for all the ML models. I tried deleting old AutoML and dragging a new / fresh AutoML component from Knime Hub but still same error. I have been trying to find the specific cause of the bug the whole day but no success. Thanks for any input!

Not sure if its related but Error log is below: !SESSION 2023-08-08 23:17:50.417 -----------------------------------------------
eclipse.buildId=unknown
java.version=17.0.5
java.vendor=Eclipse Adoptium
BootLoader constants: OS=win32, ARCH=x86_64, WS=win32, NL=en_US
Command-line arguments: -os win32 -ws win32 -arch x86_64

!ENTRY org.eclipse.e4.ui.workbench 4 0 2023-08-08 23:17:54.859
!MESSAGE Unable to load class ‘org.knime.product.rcp.addons.ToolbarFixAddon’ from bundle ‘295’
!STACK 0
java.lang.ClassNotFoundException: org.knime.product.rcp.addons.ToolbarFixAddon cannot be found by org.knime.product_5.1.0.v202307121409
at org.eclipse.osgi.internal.loader.BundleLoader.generateException(BundleLoader.java:541)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass0(BundleLoader.java:536)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:416)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:168)
at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
at org.eclipse.osgi.internal.framework.EquinoxBundle.loadClass(EquinoxBundle.java:622)
at org.eclipse.e4.ui.internal.workbench.ReflectionContributionFactory.createFromBundle(ReflectionContributionFactory.java:89)
at org.eclipse.e4.ui.internal.workbench.ReflectionContributionFactory.doCreate(ReflectionContributionFactory.java:60)
at org.eclipse.e4.ui.internal.workbench.ReflectionContributionFactory.create(ReflectionContributionFactory.java:37)
at org.eclipse.e4.ui.internal.workbench.swt.E4Application.createE4Workbench(E4Application.java:276)
at org.eclipse.ui.internal.Workbench.lambda$3(Workbench.java:572)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:338)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:550)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:171)
at org.knime.product.rcp.KNIMEApplication.start(KNIMEApplication.java:193)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:136)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:402)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:659)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:596)
at org.eclipse.equinox.launcher.Main.run(Main.java:1467)

Artem · August 9, 2023, 7:14am

Hello @rdlc1

I think you are not the only one who faced this issue. It seems that the problem is with 5.1. Similar issue was posted here: Solutions to "Just KNIME It!" Challenge 19 - Season 2 - #25 by paolotamag

I hope it will be fixed in 5.1.1 soon.

rdlc1 · August 9, 2023, 1:52pm

@Artem - Thank you for the quick reply. Hope it does get fixed soon.

rdlc1 · August 9, 2023, 3:48pm

@artem. Two more issues that I have been dealing with using autoML for ML work (Knime 4.7 on Mac M1 pro OS Monterey):

Missing XGBoost nodes . I noticed it when running AutoML and also when running a separate XGBoost workflow. I already installed and uninstalled the XGBoost extension and the XGBoost nodes still do not appear in the node repository and the workflow shows the nodes as missing. Maybe the extensions are installed but the node repository and Knime is unable to see XGBoost nodes?

Screen Shot 2023-08-09 at 9.19.28 AM1868×1682 146 KB

Looking at the forum, I have seen this has been an ongoing issue but am having a hard time setting up Deep Learning for Keras and Tensorflow because of old Python build requirements that I cannot find : . Hope it gets resolved soon.

Thank you,
Richard

Artem · August 10, 2023, 5:16am

It looks like a bug to me. But there a couple of things you can try:

Try to drag (copy) the component from the hub once you have the XGBoost extension installed. Maybe you saved the workflow in this state, where the node is missing;
Try to find the this node in Node repository and try to replace the missing nodes. However it also means that you need to configure them yourself as well.

Just in case I will tag @paolotamag since it seems that he is investigating the issues with the AutoML component. Maybe he will have some ideas.

paolotamag · August 10, 2023, 8:48am

Hi all,

in general i recommend dragging and dropping the automl component again after installing extensions and restarting KNIME.

Regarding the KNIME XGBoost Integration…
We are investigating issues in even installing the extension at this moment in Mac systems. We will post here when we know more. A workaround for autoML is executing the AutoML component even with the missing nodes indies but with XGBoost deactivated in the component dialogue. Maybe you can try the same?

Regarding Deep Learning enviroments, you should be able to install the enviroment automatically if you go in “KNIME Preferences > KNIME > Python Deep Learning” then you can create a new enviroment for either Keras or “TensorFlow 2” automatically by selecting “Conda”.

You need to install first conda though via miniconda (Miniconda — conda documentation) or similar. If you defined a custom path for the miniconda installation provide such path in “KNIME Preferences > KNIME > Conda”

paolotamag · August 10, 2023, 11:17am

Regarding KNIME XGBoost Integration – KNIME Community Hub

I talked to our devs and it seems that XGBoost nodes are missing on M1 Mac in KNIME 4.7.x
XGBoost on Mac on 5.1.0 should work but there AutoML is failing.

To use XGBoost in the AutoML component on M1 Mac please wait for 5.1.1 (likely later this month).

Sorry for the inconvenience.

rdlc1 · August 10, 2023, 12:17pm

No worries. I really appreciate the work you guys do answering questions from non-coders like me. Thank you very much!

I will watch out for 5.1.1.

rdlc1 · August 10, 2023, 12:24pm

I have had success using autoML on Mac M1 and Knime 4.7 running the ML workflows except for XGBoost and DL (w/ Keras).

On deep learning, I will try the miniconda install. Thank you!

rdlc1 · August 10, 2023, 12:40pm

Still no luck. I followed the instructions and got the error below:

I also tried to make my own yml file following this very thorough articles: KNIME and Python — Setting up and managing Conda environments | by Markus Lauber | Low Code for Data Science | Medium
KNIME and Python — Setting up Deep Learning Environments for Keras and TensorFlow | by Markus Lauber | Low Code for Data Science | Medium
Also getting the similar error messages after “create new environment” step. I wa sable to download Python 3.6 from the Python website but I do not know how to direct the yml to fetch it locally. Plus the other packages needed are also hard to find. Versions loaded after making conda environment based on knime_py_keras_macos.yml file from the article are newest versions. When I force it to try to load older versions using “=” , env creation fails.

I hope what I wrote makes sense. lol.

Thanks,
Richard

paolotamag · August 10, 2023, 1:13pm

Sorry to hear.

Did you check the blog post:

https://www.knime.com/blog/how-to-setup-the-python-extension

scroll down to the section 4. Set Up your Python Deep Learning Environment

or check the documentation:

https://docs.knime.com/latest/deep_learning_installation_guide

P.S. the knime website is currently under maintenance if any page does not load now please try again later

marzukim · August 11, 2023, 7:26am

it just for references,
here are the details of my knime 4.7.5 installation.
i’m using the following conda environment setting to use the DL approach:

a. conda version: 23.5.0
b. keras DL
-python 3.5.4
-keras 2.2.2
-tensorflow 1.10.0
-numpy 1.14.2
-pandas 0.20.3
-h5py 2.8.0

c. tensorflow 2 DL
-python 3.10.11

rgds

mlauber71 · August 11, 2023, 8:59am

I also noticed the problem of installing the XGBoost extension on Mac with this workflow (Binary Classification - use Python XGBoost package and other nodes to build model and deploy that thru KNIME Python nodes – KNIME Community Hub) in June and there is a ticket already in the knime system AP-20587.

There have been similar problems in the past AP-19894, AP-19918. I hope an underlying cause can be found and resolved.

rdlc1 · August 11, 2023, 10:38am

This is very helpful. Thank you. I will definitely try to get Tensorflow 2 working since it works on a newer Python version.

Can I get your opinion or help (if you think it will work) on a workaround on the errors that pop up by either of the two published choices for setting up Keras deep learning on knime, which are: (1) “setup new environment” on Knime DL pop up - which leads to error I get on the screenshot above and (2) making a yml file that has the packages because the sites that the yml tries to get the packages only has the newest versions of the packages. Here, the error I get is that it wants me to remove the old packages or it downloads newer versions of the packages for python, tensorflow, etc.

I was able to find Python 3.6 on the official Python website. I was able to download it but my problem is how to direct Knime to see this older version of Python that I have installed. For example, if I write a new yml environment file, is there a way or command line to direct the yml file to fetch python from a local install (instead of online sites like anaconda, etc.). Or do you guys know of an online site from where I can direct my yml file to fetch Python 3.6 and the older packages of tensorflow, h5py, etc. that running Keras on Knime 4.7 is dependent on?

Thanks, Richard

mlauber71 · August 11, 2023, 10:57am

@rdlc1 I think the issue is, that the current combination of packages for the KNIME nodes and Deep Learning on Apple Silicon is not there. The packages currently required are older and for M1/m2 only newer versions are available.

I was not able to find a working combination of slightly newer packages to get all functions to work with Apple Silicon. I think KNIME would have to rework the nodes to fit to the newer versions of Keras and Tensorflow.

My impression is that is also due to quite dynamic developments in these packages and changes in the way some basic structures are used and communicated and where the nodes might have to be adapted.

Only solution I would see is to just use a Python node and do all the coding by hand - which sort of defeats the idea of low code unfortunately.

Also the installation descriptions might need some updates since there seem to be several cross-references that are not always on point. If the automatic installation is working it is OK - otherwise you have to search for yourself (hence the article).

As mentioned: You can see all the YAML files and settings by extracting them from the KNIME installation - you could try and follow the versions and packages but I do not think this is currently possible for Apple Silicon (or I was too stupid doing it).

rdlc1 · August 11, 2023, 11:30am

@mlauber71 Thank you for the thorough reply. Looks like I will have to wait for the updates for using DL on Apple silicon. Using mostly ML for now so…getting XGBoost to work on Apple M1would suffice.

Also, I have been using the yml workaround on an Apple M1. So if I use the yml file on my windows 10 (64bit) laptop, then theoretically the yml will be able to fetch python 3.6 and the other older package versions?

Thanks,
Richard

marzukim · August 11, 2023, 1:08pm

the easiest and preferred way to handle the ‘pop-up’ dependencies of knime 4.7.x, especially in python DL, is to use conda installation.
i’m using a linux machine and am not quite familiar and expert with other operating systems. although i can install conda env within the knime dialog page, i prefer to do it manually.

the concept of conda installation is the same on all operating systems: create an environment, activate it, and install the required packages. once conda is installed, use the preferences to select the required environment.

for TensorFlow 2 DL, a python 3.10+ environment will be able to fulfill the dependencies. however, for Keras DL, I set the environment to Python 3.5.

rgds

mlauber71 · August 11, 2023, 3:36pm

It should work on windows. I also have two meta nodes with Conda Environment Propagation for Keras and TensorFlow for Windows. I have not tested them for some time though.

rdlc1 · August 24, 2023, 4:30pm

Thanks for the suggestions and appreciate all the help!

What finally worked for me was running v4.7 on a windows laptop. Knime was able to pull up the correct python environments for keras and tensorflow.

rdlc1 · September 11, 2023, 12:32am

Hello @mlauber71 - would you happen to have the similar metanode for env propagation for GPU execution of Keras. My CPU Keras execution works just fine (testing done using the Image_Classification_MNIST_Solution workflow). When I switch to GPU Keras execution, the analysis hangs during 1st second of GPU usage.
Keras GPU python env modules installed by Knime is below:

cmd: C:\Users..\miniforge3\Scripts\conda-env-script.py create --file C:\Users..\Desktop\knime_4.7.7\plugins\org.knime.python2.envconfigs_4.7.0.v202211181342\envconfigs\windows\py36_knime_dl_gpu.yml --name py3_knime_kerasv2_gpu --json

conda version: 23.3.1

+anaconda/win-64::hdf5-1.8.18-vc14h7a021fe_0
+anaconda/win-64::icu-58.2-vc14hc45fdbb_0
+anaconda/win-64::jpeg-9b-vc14h4d7706e_1
+anaconda/win-64::pixman-0.34.0-vc14h00fde18_1
+anaconda/win-64::qt-5.9.7-vc14h73c81de_0
+anaconda/win-64::yaml-0.1.7-vc14h4cb57cf_1
+anaconda/win-64::zlib-1.2.11-vc14h1cdd9ab_1
+conda-forge/win-64::arrow-cpp-5.0.0-py36he9238d2_8_cpu
+conda-forge/win-64::aws-c-cal-0.5.11-he19cf47_0
+conda-forge/win-64::aws-c-event-stream-0.2.7-h70e1b0c_13
+conda-forge/win-64::aws-c-io-0.10.5-h2fe331c_0
+conda-forge/win-64::aws-checksums-0.1.11-h1e232aa_7
+conda-forge/win-64::aws-sdk-cpp-1.8.186-hb0612c5_3
+conda-forge/win-64::grpc-cpp-1.40.0-h2431d41_1
+conda-forge/win-64::jpype1-0.6.3-py36h79cbd7a_1001
+conda-forge/win-64::libprotobuf-3.18.0-h7755175_1
+conda-forge/win-64::protobuf-3.18.0-py36he2d232f_0
+conda-forge/win-64::pyarrow-5.0.0-py36h6720e24_8_cpu
+conda-forge/win-64::python_abi-3.6-2_cp36m
+conda-forge/win-64::re2-2021.09.01-h0e60522_0
+defaults/noarch::absl-py-0.15.0-pyhd3eb1b0_0
+defaults/noarch::attrs-21.4.0-pyhd3eb1b0_0
+defaults/noarch::backcall-0.2.0-pyhd3eb1b0_0
+defaults/noarch::colorama-0.4.4-pyhd3eb1b0_0
+defaults/noarch::cycler-0.11.0-pyhd3eb1b0_0
+defaults/noarch::dataclasses-0.8-pyh4f3eec9_6
+defaults/noarch::decorator-5.1.1-pyhd3eb1b0_0
+defaults/noarch::gast-0.5.3-pyhd3eb1b0_0
+defaults/noarch::importlib_metadata-4.8.1-hd3eb1b0_0
+defaults/noarch::ipython_genutils-0.2.0-pyhd3eb1b0_1
+defaults/noarch::jsonschema-3.2.0-pyhd3eb1b0_2
+defaults/noarch::keras-applications-1.0.8-py_1
+defaults/noarch::keras-preprocessing-1.1.2-pyhd3eb1b0_0
+defaults/noarch::parquet-cpp-1.5.1-h34088ae_4
+defaults/noarch::parso-0.8.3-pyhd3eb1b0_0
+defaults/noarch::pickleshare-0.7.5-pyhd3eb1b0_1003
+defaults/noarch::prompt_toolkit-2.0.10-py_0
+defaults/noarch::py4j-0.10.9.2-pyhd3eb1b0_0
+defaults/noarch::pygments-2.11.2-pyhd3eb1b0_0
+defaults/noarch::pyparsing-3.0.4-pyhd3eb1b0_0
+defaults/noarch::pytz-2021.3-pyhd3eb1b0_0
+defaults/noarch::six-1.16.0-pyhd3eb1b0_1
+defaults/noarch::typing_extensions-4.1.1-pyh06a4308_0
+defaults/noarch::wcwidth-0.2.5-pyhd3eb1b0_0
+defaults/noarch::werkzeug-2.0.3-pyhd3eb1b0_0
+defaults/noarch::wheel-0.37.1-pyhd3eb1b0_0
+defaults/noarch::zipp-3.6.0-pyhd3eb1b0_0
+defaults/win-64::_tflow_select-2.1.0-gpu
+defaults/win-64::abseil-cpp-20210324.2-hd77b12b_0
+defaults/win-64::astor-0.8.1-py36haa95532_0
+defaults/win-64::aws-c-common-0.6.2-h2bbff1b_0
+defaults/win-64::blas-1.0-mkl
+defaults/win-64::bzip2-1.0.8-he774522_0
+defaults/win-64::c-ares-1.19.1-h2bbff1b_0
+defaults/win-64::ca-certificates-2023.08.22-haa95532_0
+defaults/win-64::cairo-1.14.12-hf171d8a_3
+defaults/win-64::certifi-2021.5.30-py36haa95532_0
+defaults/win-64::coverage-5.5-py36h2bbff1b_2
+defaults/win-64::cudatoolkit-9.0-1
+defaults/win-64::cudnn-7.6.5-cuda9.0_0
+defaults/win-64::cython-0.29.24-py36hd77b12b_0
+defaults/win-64::freetype-2.10.4-hd328e21_0
+defaults/win-64::gflags-2.2.2-ha925a31_0
+defaults/win-64::glog-0.5.0-hd77b12b_0
+defaults/win-64::grpcio-1.36.1-py36hc60d5dd_1
+defaults/win-64::h5py-2.8.0-py36hf7173ca_0
+defaults/win-64::icc_rt-2022.1.0-h6049295_2
+defaults/win-64::importlib-metadata-4.8.1-py36haa95532_0
+defaults/win-64::intel-openmp-2023.1.0-h59b6b97_46319
+defaults/win-64::ipython-7.1.1-py36h39e3cac_0
+defaults/win-64::jedi-0.13.3-py36_0
+defaults/win-64::jupyter_core-4.8.1-py36haa95532_0
+defaults/win-64::keras-2.2.4-0
+defaults/win-64::keras-base-2.2.4-py36_0
+defaults/win-64::kiwisolver-1.3.1-py36hd77b12b_0
+defaults/win-64::lerc-3.0-hd77b12b_0
+defaults/win-64::libbrotlicommon-1.0.9-h2bbff1b_7
+defaults/win-64::libbrotlidec-1.0.9-h2bbff1b_7
+defaults/win-64::libbrotlienc-1.0.9-h2bbff1b_7
+defaults/win-64::libcurl-7.82.0-h86230a5_0
+defaults/win-64::libdeflate-1.17-h2bbff1b_0
+defaults/win-64::libpng-1.6.37-h2a8f88b_0
+defaults/win-64::libssh2-1.10.0-hcd4344a_2
+defaults/win-64::libthrift-0.15.0-he1d8c1a_0
+defaults/win-64::libtiff-4.5.0-h8a3f274_0
+defaults/win-64::libutf8proc-2.6.1-h2bbff1b_1
+defaults/win-64::lz4-c-1.9.3-h2bbff1b_1
+defaults/win-64::markdown-3.3.4-py36haa95532_0
+defaults/win-64::matplotlib-3.0.3-py36hc8f65d3_0
+defaults/win-64::mkl-2020.2-256
+defaults/win-64::mkl-service-2.3.0-py36h196d8e1_0
+defaults/win-64::mkl_fft-1.2.0-py36h45dec08_0
+defaults/win-64::mkl_random-1.1.1-py36h47e9c7a_0
+defaults/win-64::nbformat-4.4.0-py36_0
+defaults/win-64::numpy-1.16.1-py36h19fb1c0_1
+defaults/win-64::numpy-base-1.16.1-py36hc3f5095_1
+defaults/win-64::olefile-0.46-py36_0
+defaults/win-64::openssl-1.1.1v-h2bbff1b_0
+defaults/win-64::pandas-0.23.4-py36h830ac7b_0
+defaults/win-64::pillow-5.3.0-py36hdc69c19_0
+defaults/win-64::pip-21.2.2-py36haa95532_0
+defaults/win-64::pyqt-5.9.2-py36h6538335_2
+defaults/win-64::pyrsistent-0.17.3-py36he774522_0
+defaults/win-64::python-3.6.13-h3758d61_0
+defaults/win-64::python-dateutil-2.7.5-py36_0
+defaults/win-64::pywin32-228-py36hbaba5e8_1
+defaults/win-64::pyyaml-5.3.1-py36he774522_0
+defaults/win-64::scipy-1.1.0-py36h29ff71c_2
+defaults/win-64::setuptools-58.0.4-py36haa95532_0
+defaults/win-64::sip-4.19.8-py36h6538335_0
+defaults/win-64::snappy-1.1.9-h6c2663c_0
+defaults/win-64::sqlite-3.41.2-h2bbff1b_0
+defaults/win-64::tensorboard-1.12.2-py36h33f27b4_0
+defaults/win-64::tensorflow-1.12.0-gpu_py36ha5f9131_0
+defaults/win-64::tensorflow-base-1.12.0-gpu_py36h6e53903_0
+defaults/win-64::tensorflow-gpu-1.12.0-h0d30ee6_0
+defaults/win-64::termcolor-1.1.0-py36haa95532_1
+defaults/win-64::tk-8.6.12-h2bbff1b_0
+defaults/win-64::tornado-6.1-py36h2bbff1b_0
+defaults/win-64::traitlets-4.3.3-py36haa95532_0
+defaults/win-64::vc-14.2-h21ff451_1
+defaults/win-64::vs2015_runtime-14.27.29016-h5e58377_2
+defaults/win-64::wincertstore-0.2-py36h7fe50ca_0
+defaults/win-64::xz-5.4.2-h8cc25b3_0
+defaults/win-64::zstd-1.5.0-h19a0ad4_1

update specs: [‘numpy=1.16.1’, ‘h5py=2.8’, ‘ipython=7.1’, “pyarrow[version=‘>=5.0.0’]”, ‘matplotlib=3.0’, ‘pillow=5.3’, ‘pandas=0.23’, ‘python=3.6’, ‘jedi=0.13’, ‘tensorflow-gpu=1.12.0’, ‘keras=2.2.4’, ‘pip’, ‘jpype1=0.6.3’, ‘cairo=1.14’, ‘python-dateutil=2.7’, ‘py4j’, ‘scipy=1.1’, ‘nbformat=4.4’]

and error I am seeing in the log is:
2023-09-10 18:22:05,558 : ERROR : KNIME-Worker-46-Keras Network Learner 3:96 : : DLKerasLearnerNodeModel : Keras Network Learner : 3:96 : Blas GEMM launch failed : a.shape=(200, 100), b.shape=(100, 10), m=200, n=10, k=100
[[{{node output_1/MatMul}} = MatMul[T=DT_FLOAT, _class=[“loc:@training/Adadelta/gradients/output_1/MatMul_grad/MatMul”], transpose_a=false, transpose_b=false, _device=“/job:localhost/replica:0/task:0/device:GPU:0”](dense_1/Relu, output_1/kernel/read)]]
[[{{node loss/mul/_97}} = _Recvclient_terminated=false, recv_device=“/job:localhost/replica:0/task:0/device:CPU:0”, send_device=“/job:localhost/replica:0/task:0/device:GPU:0”, send_device_incarnation=1, tensor_name=“edge_605_loss/mul”, tensor_type=DT_FLOAT, _device=“/job:localhost/replica:0/task:0/device:CPU:0”]]
2023-09-10 18:22:05,558 : ERROR : KNIME-Worker-46-Keras Network Learner 3:96 : : Node : Keras Network Learner : 3:96 : Execute failed: An error occured during training of the Keras deep learning network. See log for details.
java.lang.RuntimeException: An error occured during training of the Keras deep learning network. See log for details.
at org.knime.dl.keras.base.nodes.learner.DLKerasLearnerNodeModel.handleGeneralException(DLKerasLearnerNodeModel.java:751)
at org.knime.dl.keras.base.nodes.learner.DLKerasLearnerNodeModel.executeInternal(DLKerasLearnerNodeModel.java:721)
at org.knime.dl.keras.base.nodes.learner.DLKerasLearnerNodeModel.execute(DLKerasLearnerNodeModel.java:320)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:549)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1267)
at org.knime.core.node.Node.execute(Node.java:1041)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:595)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:98)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:201)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:117)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:367)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:221)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
Caused by: org.knime.python2.kernel.PythonIOException: Blas GEMM launch failed : a.shape=(200, 100), b.shape=(100, 10), m=200, n=10, k=100
[[{{node output_1/MatMul}} = MatMul[T=DT_FLOAT, _class=[“loc:@training/Adadelta/gradients/output_1/MatMul_grad/MatMul”], transpose_a=false, transpose_b=false, _device=“/job:localhost/replica:0/task:0/device:GPU:0”](dense_1/Relu, output_1/kernel/read)]]
[[{{node loss/mul/_97}} = _Recvclient_terminated=false, recv_device=“/job:localhost/replica:0/task:0/device:CPU:0”, send_device=“/job:localhost/replica:0/task:0/device:GPU:0”, send_device_incarnation=1, tensor_name=“edge_605_loss/mul”, tensor_type=DT_FLOAT, _device=“/job:localhost/replica:0/task:0/device:CPU:0”]]
at org.knime.python2.kernel.messaging.AbstractTaskHandler.handleFailureMessage(AbstractTaskHandler.java:146)
at org.knime.python2.kernel.messaging.AbstractTaskHandler.handle(AbstractTaskHandler.java:92)
at org.knime.dl.python.core.DLPythonAbstractCommands$DLTrainingTask.runInternal(DLPythonAbstractCommands.java:931)
at org.knime.core.util.ThreadUtils$CallableWithContextImpl.callWithContext(ThreadUtils.java:383)
at org.knime.core.util.ThreadUtils$CallableWithContext.call(ThreadUtils.java:269)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)