Memory issue in running Gaussian processes

sjanbakhsh · February 12, 2016, 10:12pm

Hi,
I'm using Knime 3.1.0, I tried to execute a Guassian process but it just goes to 50% and then it freeze. I also try to change the Guassian confguration e.g filtertype/noise but it didn't work.
I increased JavaHeapsize in Knime/knime.ini file (made it to-Xmx5120m)
But still no differences.
I execute the same process with the linear regression and it works fine, but i want to try it with Gaussian too.
Can you please advice how I can make it work?
Sanaz

sjanbakhsh · February 13, 2016, 2:29pm

For more information, Please find the command line options below.

-L 1.0 -N 0 -K "weka.classifiers.functions.supportVector.PolyKernel -C 250007 -E 1.0"

I also tried to increase -C (chache size) too but still not luck and it freezed at 50% progress.

when i execute the process, Kinme uses more than 70% of the CPU.

I hope someone can help me here.

Sanaz

wiswedel · February 15, 2016, 3:11pm

That appears to be a weka "issue". I guess you have to reach out to the people behind weka as this is not KNIME code (we only do the integration).

sjanbakhsh · February 16, 2016, 12:47am

Hi wiswede

Thanks for the reply.

Since we are in the stage of the evaluatung repid miner and Knime, So i tested in Rapidminer too and I could completed the process successfuly with Rapidminer; However it fails with Knime. That's why i suspected the issue is more related to the integration with knime not weka. I will try to see if i can find any solutions related to weka.

Sanaz

wiswedel · February 16, 2016, 12:25pm

Interesting. Do you know if RM also runs this through Weka or is it a native RM module/node/operator/function?

sjanbakhsh · February 16, 2016, 9:50pm

Hi Wiswedel,

I tried different WEKA functions in Knime e.g linear regression function, .. but it hangs when i use Gaussian Prcoess.

So maybe we can confirm that there is no issue with WEKA and just in this function in Knime.

Any thoughts?

Sanaz

wiswedel · February 17, 2016, 11:52am

Memory and runtime consumption is heavily algorithm (and implementation) dependent. So just because Weka-Linear Regression works (which, btw, is a relatively trivial algorithm unless you have many independent variables) doesn't tell anything about the runtime of other algorithms.

I guess the question still stands: Does RM have their own native implementation or are they using Weka? If it's also weka and it works in there then it might be an issue with our integration.

Another possibility for diagnosis is to extract a 'jstack' from the running process - this would allow us to track where it gets stuck. Here is an explanation how to get to the jstack output: https://tech.knime.org/forum/knime-users/auto-save-bugs#comment-33754

Happy to review!

sjanbakhsh · February 17, 2016, 3:58pm

Hi wiswedel,

Yes RM has native impelemtation as well as Weka. I tested Guassian in both and it works fine.

Thanks,

Sanaz

wiswedel · February 17, 2016, 4:04pm

Thanks for the clarification. Are you saying you also ran this natively in Weka and it works there (on the same data)?

If so, we are really interested in a jstack output and, possibly, a workflow to reproduce it.

Thanks,
Bernd

sjanbakhsh · February 17, 2016, 4:47pm

Hi wiswedel,

For the first time after 2 weeks try, i could run Guassian process in Knime ( just one time) but after again no luck,

Please find a part of log below. which shows the java heap size issue.

ENTRY org.eclipse.ui 4 0 2016-02-17 08:33:05.391
!MESSAGE Unhandled event loop exception
!STACK 0
org.eclipse.swt.SWTException: Failed to execute runnable (java.lang.OutOfMemoryError: Java heap space)
at org.eclipse.swt.SWT.error(SWT.java:4491)
at org.eclipse.swt.SWT.error(SWT.java:4406)
at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:138)
at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4155)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3772)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$4.run(PartRenderingEngine.java:1127)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:337)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1018)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:156)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:654)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:337)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:598)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
at org.knime.product.rcp.KNIMEApplication.start(KNIMEApplication.java:132)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:380)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:669)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:608)
at org.eclipse.equinox.launcher.Main.run(Main.java:1515)
Caused by: java.lang.OutOfMemoryError: Java heap space

!ENTRY org.eclipse.ui 4 0 2016-02-17 08:33:06.795
!MESSAGE Unhandled event loop exception
!STACK 0
org.eclipse.swt.SWTException: Failed to execute runnable (java.lang.OutOfMemoryError: Java heap space)
at org.eclipse.swt.SWT.error(SWT.java:4491)
at org.eclipse.swt.SWT.error(SWT.java:4406)
at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:138)
at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4155)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3772)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$4.run(PartRenderingEngine.java:1127)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:337)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1018)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:156)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:654)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:337)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:598)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
at org.knime.product.rcp.KNIMEApplication.start(KNIMEApplication.java:132)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:380)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:669)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:608)
at org.eclipse.equinox.launcher.Main.run(Main.java:1515)
Caused by: java.lang.OutOfMemoryError: Java heap space

!ENTRY org.eclipse.ui 4 0 2016-02-17 08:33:08.290
!MESSAGE Unhandled event loop exception
!STACK 0
org.eclipse.swt.SWTException: Failed to execute runnable (java.lang.OutOfMemoryError: Java heap space)
at org.eclipse.swt.SWT.error(SWT.java:4491)
at org.eclipse.swt.SWT.error(SWT.java:4406)
at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:138)
at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4155)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3772)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$4.run(PartRenderingEngine.java:1127)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:337)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1018)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:156)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:654)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:337)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:598)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
at org.knime.product.rcp.KNIMEApplication.start(KNIMEApplication.java:132)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:380)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:669)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:608)
at org.eclipse.equinox.launcher.Main.run(Main.java:1515)
Caused by: java.lang.OutOfMemoryError: Java heap space

!ENTRY org.eclipse.ui 4 0 2016-02-17 08:33:09.451
!MESSAGE Unhandled event loop exception
!STACK 0
org.eclipse.swt.SWTException: Failed to execute runnable (java.lang.OutOfMemoryError: Java heap space)
at org.eclipse.swt.SWT.error(SWT.java:4491)
at org.eclipse.swt.SWT.error(SWT.java:4406)
at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:138)
at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4155)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3772)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$4.run(PartRenderingEngine.java:1127)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:337)
at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1018)
at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:156)
at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:654)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:337)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:598)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
at org.knime.product.rcp.KNIMEApplication.start(KNIMEApplication.java:132)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:380)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:669)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:608)
at org.eclipse.equinox.launcher.Main.run(Main.java:1515)
Caused by: java.lang.OutOfMemoryError: Java heap space

sjanbakhsh · February 17, 2016, 5:29pm

Thanks Wiswedel for the quick response.

Please find the attached jstack log. I will send you the project shortly too.

Sanaz

output.txt

wiswedel · February 25, 2016, 9:27am

Just a summary. We concluded that it's a weka problem[1] and that it's not completely clear how it compares to the other implementation that was available to Sanaz.

[1]

at weka.core.SerializedObject.<init>(SerializedObject.java:69)
at weka.classifiers.functions.supportVector.Kernel.makeCopy(Kernel.java:279)
at weka.classifiers.functions.GaussianProcesses.buildClassifier(GaussianProcesses.java:391)
at org.knime.ext.weka37.utils.WekaClassifierCreationTask.run(WekaClassifierCreationTask.java:124)

sjanbakhsh · March 9, 2016, 3:38pm

Hi Wiswedel,

I just wanted to update you that I couldn't fix my issue with Gussian, Instead I am using Bagging (REP tree) and the result is much more accurate.

Thanks,

Sanaz

PacoMasip · March 19, 2019, 8:37pm

Hi,

I wander if there has been any update on the issue mentioned above. I am using the weka gaussian process and I am seeing same issues very slow sometimes hangs up and so on.

Thanks
Paco