DL4J is broken after upgrade to knime 3.5 cannot find libnd4jcuda.so

DL4J worked before on the same machine. After I upgraded to KNIME 3.5 it stopped working.
The error message says a library is missing. However, I can see that file.

[uXXX@elc7papp02 libnd4jcuda.so]$ ls -l
total 195242
-rw-r–r-- 1 uXXX users 166239828 Dec 31 1969 libnd4jcuda.so
[uXXX@elc7papp02 libnd4jcuda.so]$ pwd
/home/uXXX/.javacpp/cache/libnd4jcuda.so

java.lang.ClassNotFoundException: An error occurred while automatically activating bundle org.knime.ext.dl4j.libs (934).
at org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:116)
at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:529)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:325)
at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:345)
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(BundleLoader.java:423)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:372)
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:364)
at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:161)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.eclipse.osgi.internal.framework.EquinoxBundle.loadClass(EquinoxBundle.java:564)
at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.createExecutableExtension(RegistryStrategyOSGI.java:174)
at org.eclipse.core.internal.registry.ExtensionRegistry.createExecutableExtension(ExtensionRegistry.java:905)
at org.eclipse.core.internal.registry.ConfigurationElement.createExecutableExtension(ConfigurationElement.java:243)
at org.eclipse.core.internal.registry.ConfigurationElementHandle.createExecutableExtension(ConfigurationElementHandle.java:55)
at org.eclipse.ui.internal.WorkbenchPlugin$1.run(WorkbenchPlugin.java:291)
at org.eclipse.swt.custom.BusyIndicator.showWhile(BusyIndicator.java:70)
at org.eclipse.ui.internal.WorkbenchPlugin.createExtension(WorkbenchPlugin.java:286)
at org.eclipse.ui.internal.dialogs.WorkbenchPreferenceNode.createPage(WorkbenchPreferenceNode.java:48)
at org.eclipse.jface.preference.PreferenceDialog.createPage(PreferenceDialog.java:1280)
at org.eclipse.ui.internal.dialogs.FilteredPreferenceDialog.createPage(FilteredPreferenceDialog.java:355)
at org.eclipse.jface.preference.PreferenceDialog.showPage(PreferenceDialog.java:1167)
at org.eclipse.ui.internal.dialogs.FilteredPreferenceDialog.showPage(FilteredPreferenceDialog.java:608)
at org.eclipse.jface.preference.PreferenceDialog$5.lambda$0(PreferenceDialog.java:656)
at org.eclipse.swt.custom.BusyIndicator.showWhile(BusyIndicator.java:70)
at org.eclipse.jface.preference.PreferenceDialog$5.selectionChanged(PreferenceDialog.java:653)
at org.eclipse.jface.viewers.StructuredViewer$3.run(StructuredViewer.java:878)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
at org.eclipse.ui.internal.JFaceUtil$1.run(JFaceUtil.java:50)
at org.eclipse.jface.util.SafeRunnable.run(SafeRunnable.java:173)
at org.eclipse.jface.viewers.StructuredViewer.firePostSelectionChanged(StructuredViewer.java:875)
at org.eclipse.jface.viewers.StructuredViewer.handlePostSelect(StructuredViewer.java:1244)
at org.eclipse.jface.viewers.StructuredViewer$5.widgetSelected(StructuredViewer.java:1270)
at org.eclipse.jface.util.OpenStrategy.firePostSelectionEvent(OpenStrategy.java:265)
at org.eclipse.jface.util.OpenStrategy.access$5(OpenStrategy.java:259)
at org.eclipse.jface.util.OpenStrategy$1.lambda$1(OpenStrategy.java:431)
at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:35)
at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:182)
at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4528)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:4146)
at org.eclipse.jface.window.Window.runEventLoop(Window.java:818)
at org.eclipse.jface.window.Window.open(Window.java:794)
at org.eclipse.ui.internal.dialogs.WorkbenchPreferenceDialog.open(WorkbenchPreferenceDialog.java:211)
at org.eclipse.ui.internal.OpenPreferencesAction.run(OpenPreferencesAction.java:63)
at org.eclipse.jface.action.Action.runWithEvent(Action.java:473)
at org.eclipse.jface.action.ActionContributionItem.handleWidgetSelection(ActionContributionItem.java:565)
at org.eclipse.jface.action.ActionContributionItem.lambda$4(ActionContributionItem.java:397)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:5219)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1340)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4553)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:4143)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$4.run(PartRenderingEngine.java:1121)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:336)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1022)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:150)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:687)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:336)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:604)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:148)
at org.knime.product.rcp.KNIMEApplication.start(KNIMEApplication.java:141)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:388)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:243)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:673)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:610)
at org.eclipse.equinox.launcher.Main.run(Main.java:1519)
at org.eclipse.equinox.launcher.Main.main(Main.java:1492)
Caused by: org.osgi.framework.BundleException: Exception in org.knime.ext.dl4j.libs.DL4JPluginActivator.start() of bundle org.knime.ext.dl4j.libs.
at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:795)
at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:724)
at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:932)
at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:309)
at org.eclipse.osgi.container.Module.doStart(Module.java:581)
at org.eclipse.osgi.container.Module.start(Module.java:449)
at org.eclipse.osgi.framework.util.SecureAction.start(SecureAction.java:470)
at org.eclipse.osgi.internal.hooks.EclipseLazyStarter.postFindLocalClass(EclipseLazyStarter.java:107)
… 72 more
Caused by: java.lang.ExceptionInInitializerError
at org.nd4j.jita.concurrency.CudaAffinityManager.getNumberOfDevices(CudaAffinityManager.java:185)
at org.nd4j.jita.constant.ConstantProtector.purgeProtector(ConstantProtector.java:38)
at org.nd4j.jita.constant.ConstantProtector.(ConstantProtector.java:31)
at org.nd4j.jita.constant.ConstantProtector.(ConstantProtector.java:20)
at org.nd4j.jita.constant.ProtectedCudaConstantHandler.(ProtectedCudaConstantHandler.java:48)
at org.nd4j.jita.constant.CudaConstantHandler.(CudaConstantHandler.java:17)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:6150)
at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:6087)
at org.nd4j.linalg.factory.Nd4j.(Nd4j.java:201)
at org.knime.ext.dl4j.libs.DL4JPluginActivator.start(DL4JPluginActivator.java:173)
at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:774)
at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:767)
… 79 more
Caused by: java.lang.RuntimeException: ND4J is probably missing dependencies. For more information, please refer to: http://nd4j.org/getstarted.html
at org.nd4j.nativeblas.NativeOpsHolder.(NativeOpsHolder.java:51)
at org.nd4j.nativeblas.NativeOpsHolder.(NativeOpsHolder.java:19)
… 95 more
Caused by: java.lang.UnsatisfiedLinkError: no jnind4jcuda in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:976)
at org.bytedeco.javacpp.Loader.load(Loader.java:777)
at org.bytedeco.javacpp.Loader.load(Loader.java:684)
at org.nd4j.nativeblas.Nd4jCuda.(Nd4jCuda.java:10)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.bytedeco.javacpp.Loader.load(Loader.java:739)
at org.bytedeco.javacpp.Loader.load(Loader.java:684)
at org.nd4j.nativeblas.Nd4jCuda$NativeOps.(Nd4jCuda.java:63)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.nd4j.nativeblas.NativeOpsHolder.(NativeOpsHolder.java:29)
… 96 more
Caused by: java.lang.UnsatisfiedLinkError: /home/uXXX/.javacpp/cache/libjnind4jcuda.so/libjnind4jcuda.so: libnd4jcuda.so: cannot open shared object file: No such file or directory
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:956)
… 107 more

Hi yupl,

which version of CUDA do you have installed? Also, do the Nodes work when executing on CPU? Could you maybe attach the full KNIME log file?

Thanks
David

Hi David,

Mu CUDA version is 8.0. I am not even able to open a deeplearning example. The log file is attached.

Best

ERROR RepositoryManager MetaNode alexnet’ from plugin ‘org.knime.ext.dl4j.base’ could not be created: Could not initialize class org.knime.ext.dl4j.base.DLModelPortObject
ERROR RepositoryManager MetaNode deepmlp’ from plugin ‘org.knime.ext.dl4j.base’ could not be created: Could not initialize class org.knime.ext.dl4j.base.DLModelPortObject
ERROR RepositoryManager MetaNode lenet’ from plugin ‘org.knime.ext.dl4j.base’ could not be created: Could not initialize class org.knime.ext.dl4j.base.DLModelPortObject
ERROR RepositoryManager MetaNode simplemlp’ from plugin ‘org.knime.ext.dl4j.base’ could not be created: Could not initialize class org.knime.ext.dl4j.base.DLModelPortObject
ERROR RepositoryManager MetaNode deepbelief’ from plugin ‘org.knime.ext.dl4j.base’ could not be created: Could not initialize class org.knime.ext.dl4j.base.DLModelPortObject

knime.zip (830.1 KB)

Hi yupl,

Sorry for the troubles.

Do you have GPU enabled in the DL4J preferences?
I’m not sure what causes the problem, however you could try some things:

  • You could add the line -Djava.library.path="" to the end of your knime.ini file which is located in your KNIME installation folder
  • Delete the cache folder ( /home/uXXX/.javacpp/cache*) and try again

Cheers
David

Hi David,

I did what you said and still got the same error.

My knime.ini

-startup
plugins/org.eclipse.equinox.launcher_1.3.200.v20160318-1642.jar
–launcher.library
plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.400.v20160518-1444
-vm
plugins/org.knime.binary.jre.linux.x86_64_1.8.0.152-01/jre/bin
-vmargs
-server
-Dsun.java2d.d3d=false
-Dosgi.classloader.lock=classname
-XX:+UnlockDiagnosticVMOptions
-XX:+UnsyncloadClass
-Dsun.net.client.defaultReadTimeout=0
-XX:CompileCommand=exclude,javax/swing/text/GlyphView,getBreakSpot
-Xmx10137m
-Dorg.eclipse.swt.internal.gtk.disablePrinting
-Dorg.bytedeco.javacpp.maxbytes=8G
-Dorg.bytedeco.javacpp.maxphysicalbytes=8G
-Djava.library.path=""

Have you tried reinstalling KNIME? Furthermore, what OS are you using?

Hi David,

I tried reinstalling and got the same result. My OS is CentOS Linux release 7.4.1708 (Core).

Best

Hi yupl,
sorry for the late answer. Unfortunately, I have not yet found a solution for your problem as I can’t reproduce it on my machine. I’ll let you know if there is any progress on this.

Cheers
David

Hi David,

Do you have an old version that I can use to test? I think before upgrade it worked fine. Thanks!

Best

Hi yupl,
You can download an older version of KNIME from here.

Cheers
David