4.0 update and Heap Space

hafidz · June 28, 2019, 4:31pm

I just updated to version 4.0. The heap size at the bottom bar seems to max out at 512M.

KNIME seems to be able to read correct .ini file for the memory size but does not affect them. Kindly comment.

Mark_Ortmann · June 28, 2019, 7:34pm

could you please mouse over the heap status bar? It should say something like
Heap size: <current size> of total 512M (in your case) max: 2048M

The last number depends on your knime.ini settings. In my case I have an entry
-Xmx2048m.

Once you execute a workflow that requires more than 512M RAM the JVM will automatically ask for more and increase the total value to at most the value provided by -Xmx. Just give it a try by using, e.g., a Data Generator node and increase the number of rows created to something big.

If you want KNIME to use more than 512M right from the start simply add the command
-Xms to your ini. This will increase the initial heap space used by the JVM and change the total value to .

Btw. since we update with 4.0 the garbage collection algorithm resizing your heap-space is not as costly as it used to be anymore. However, please feel free to adapt Xms and Xmx according to your needs. If you are dealing with larger data increasing the maximum heap space can potentially save you a lot of execution time :).

I hope this answers your question!

Best,
Mark

hafidz · June 28, 2019, 10:48pm

Untitled

C:\Program Files\KNIME\knime.ini

-startup
plugins/org.eclipse.equinox.launcher_1.4.0.v20161219-1356.jar
–launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.551.v20171108-1834
–launcher.defaultAction
openFile
-vm
plugins/org.knime.binary.jre.win32.x86_64_1.8.0.202-b08/jre/bin
-vmargs
-server
-Dsun.java2d.d3d=false
-Dosgi.classloader.lock=classname
-XX:+UnlockDiagnosticVMOptions
-XX:+UnsyncloadClass
-XX:+UseG1GC
-Dsun.net.client.defaultReadTimeout=0
-XX:CompileCommand=exclude,javax/swing/text/GlyphView,getBreakSpot
-Xmx12000m
-Dorg.eclipse.swt.browser.IEVersion=10001
-Dsun.awt.noerasebackground=true
-Dequinox.statechange.timeout=30000

CONFIG

*** Date: Saturday, 29 June 2019 at 06:46:21 Singapore Standard Time

*** Platform Details:

*** System properties:
WEKA_HOME=C:\Users\hafid\wekafiles
applicationXMI=org.eclipse.ui.workbench/LegacyIDE.e4xmi
awt.toolkit=sun.awt.windows.WToolkit
ds.delayed.keepInstances=true
eclipse.application=org.knime.product.KNIME_APPLICATION
eclipse.commands=-os
win32
-ws
win32
-arch
x86_64
-showsplash
-launcher
C:\Program Files\KNIME\knime.exe
-name
Knime
–launcher.library
C:\Program Files\KNIME\plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.551.v20171108-1834\eclipse_1630.dll
-startup
C:\Program Files\KNIME\plugins/org.eclipse.equinox.launcher_1.4.0.v20161219-1356.jar
–launcher.overrideVmargs
-launcher
C:\Program Files\KNIME\knime.exe
-launcher
C:\Program Files\KNIME\knime.exe
-launcher
C:\Program Files\KNIME\knime.exe
-launcher
C:\Program Files\KNIME\knime.exe
-data
file:/E:/knime-workspace/
-launcher
C:\Program Files\KNIME\knime.exe
-vm
C:\Program Files\KNIME\plugins/org.knime.binary.jre.win32.x86_64_1.8.0.202-b08/jre/bin\server\jvm.dll
eclipse.home.location=file:/C:/Program Files/KNIME/
eclipse.launcher=C:\Program Files\KNIME\knime.exe
eclipse.launcher.name=Knime
eclipse.p2.data.area=@config.dir/…/p2/
eclipse.p2.profile=KNIMEProfile
eclipse.product=org.knime.product.KNIME_PRODUCT
eclipse.startTime=1561738543439
eclipse.stateSaveDelayInterval=30000
eclipse.vm=C:\Program Files\KNIME\plugins/org.knime.binary.jre.win32.x86_64_1.8.0.202-b08/jre/bin\server\jvm.dll
eclipse.vmargs=-server
-Dsun.java2d.d3d=false
-Dosgi.classloader.lock=classname
-XX:+UnlockDiagnosticVMOptions
-XX:+UnsyncloadClass
-XX:+UseG1GC
-Dsun.net.client.defaultReadTimeout=0
-XX:CompileCommand=exclude,javax/swing/text/GlyphView,getBreakSpot
-Xmx12000m
-Dorg.eclipse.swt.browser.IEVersion=10001
-Dsun.awt.noerasebackground=true
-Dequinox.statechange.timeout=30000
-Djava.class.path=C:\Program Files\KNIME\plugins/org.eclipse.equinox.launcher_1.4.0.v20161219-1356.jar
equinox.init.uuid=true
equinox.statechange.timeout=30000
equinox.use.ds=true

quaeler · June 28, 2019, 10:59pm

Strange (and strange that you have the

-launcher
C:\Program Files\KNIME\knime.exe

couplet listed so many times…) What happens if you change the Xmx to something smaller, like -Xmx2560m ? How much physical memory do you have on hand? Win10 or Win7 or Win__?

hafidz · June 28, 2019, 11:52pm

I’m not sure, but further down in the configurations are as follows:

/instance/org.knime.product//us_info-http://update.knime.com/analytics-platform/3.5-added=true
/instance/org.knime.product//us_info-http://update.knime.com/analytics-platform/3.5-removed=true
/instance/org.knime.product//us_info-http://update.knime.com/analytics-platform/3.6-added=true
/instance/org.knime.product//us_info-http://update.knime.com/analytics-platform/3.6-removed=true
/instance/org.knime.product//us_info-http://update.knime.com/analytics-platform/3.7-added=true
/instance/org.knime.product//us_info-http://update.knime.com/analytics-platform/3.7-removed=true
/instance/org.knime.product//us_info-http://update.knime.com/analytics-platform/4.0-added=true

hafidz · June 28, 2019, 11:53pm

Its 16GB DDR3 memory, Win 10, and 1TB SSD

quaeler · June 28, 2019, 11:56pm

and nothing in the heap-meter mouse-over changed if you lowered the -Xmx value?

hafidz · June 29, 2019, 12:00am

Ya, nothing moved. Like its hard coded to not to be changed!

quaeler · June 29, 2019, 12:01am

For ha-ha’s-sake, what happens if you add an -Xms greater than 512m to your .ini?

hafidz · June 29, 2019, 1:16am

I did that before. In the config, KNIME reads the info, but does not acted on it.

quaeler · June 29, 2019, 3:23am

It’s the damnedest thing… could you try downloading a fresh 4.0.0 in its entirety from knime.org - perhaps there’s something about the multiple-updates that has done something untoward.

hafidz · June 29, 2019, 10:01am

Still the same. I will wait for 4.0.0.1 instead

*** Date: Saturday, 29 June 2019 at 17:59:35 Singapore Standard Time

*** Platform Details:

*** System properties:
applicationXMI=org.eclipse.ui.workbench/LegacyIDE.e4xmi
awt.toolkit=sun.awt.windows.WToolkit
ds.delayed.keepInstances=true
eclipse.application=org.knime.product.KNIME_APPLICATION
eclipse.commands=-os
win32
-ws
win32
-arch
x86_64
-showsplash
-launcher
C:\Program Files\KNIME\knime.exe
-name
Knime
–launcher.library
C:\Program Files\KNIME\plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.551.v20171108-1834\eclipse_1630.dll
-startup
C:\Program Files\KNIME\plugins/org.eclipse.equinox.launcher_1.4.0.v20161219-1356.jar
–launcher.overrideVmargs
-launcher
C:\Program Files\KNIME\knime.exe
-data
file:/E:/knime-workspace/
-launcher
C:\Program Files\KNIME\knime.exe
-vm
C:\Program Files\KNIME\plugins/org.knime.binary.jre.win32.x86_64_1.8.0.202-b08/jre/bin\server\jvm.dll
eclipse.home.location=file:/C:/Program Files/KNIME/
eclipse.launcher=C:\Program Files\KNIME\knime.exe
eclipse.launcher.name=Knime
eclipse.p2.data.area=@config.dir/…/p2/
eclipse.p2.profile=KNIMEProfile
eclipse.product=org.knime.product.KNIME_PRODUCT
eclipse.startTime=1561802125153
eclipse.stateSaveDelayInterval=30000
eclipse.vm=C:\Program Files\KNIME\plugins/org.knime.binary.jre.win32.x86_64_1.8.0.202-b08/jre/bin\server\jvm.dll
eclipse.vmargs=-server
-Dsun.java2d.d3d=false
-Dosgi.classloader.lock=classname
-XX:+UnlockDiagnosticVMOptions
-XX:+UnsyncloadClass
-XX:+UseG1GC
-Dsun.net.client.defaultReadTimeout=0
-XX:CompileCommand=exclude,javax/swing/text/GlyphView,getBreakSpot
-Xmx12000m
-Dorg.eclipse.swt.browser.IEVersion=10001
-Dsun.awt.noerasebackground=true
-Dequinox.statechange.timeout=30000
-Djava.class.path=C:\Program Files\KNIME\plugins/org.eclipse.equinox.launcher_1.4.0.v20161219-1356.jar
equinox.init.uuid=true
equinox.statechange.timeout=30000
equinox.use.ds=true
file.encoding=Cp1252
file.encoding.pkg=sun.io
file.separator=
gosh.args=–nointeractive
java.awt.graphicsenv=sun.awt.Win32GraphicsEnvironment
java.awt.printerjob=sun.awt.windows.WPrinterJob
java.class.path=C:\Program Files\KNIME\plugins/org.eclipse.equinox.launcher_1.4.0.v20161219-1356.jar
java.class.version=52.0
java.endorsed.dirs=C:\Program Files\KNIME\plugins\org.knime.binary.jre.win32.x86_64_1.8.0.202-b08\jre\lib\endorsed
java.ext.dirs=C:\Program Files\KNIME\plugins\org.knime.binary.jre.win32.x86_64_1.8.0.202-b08\jre\lib\ext;C:\Windows\Sun\Java\lib\ext
java.home=C:\Program Files\KNIME\plugins\org.knime.binary.jre.win32.x86_64_1.8.0.202-b08\jre
java.io.tmpdir=E:\KNIME
java.library.path=C:\Program Files\KNIME;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:/Program Files/KNIME//plugins/org.knime.binary.jre.win32.x86_64_1.8.0.202-b08/jre/bin/server;C:/Program Files/KNIME//plugins/org.knime.binary.jre.win32.x86_64_1.8.0.202-b08/jre/bin;C:/Program Files/KNIME//plugins/org.knime.binary.jre.win32.x86_64_1.8.0.202-b08/jre/lib/amd64;C:\Program Files (x86)\Google\Chrome\Application;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files\IBM\ILOG\CPLEX_Studio128\opl\bin\x64_win64;C:\Program Files\IBM\ILOG\CPLEX_Studio128\opl\oplide;C:\Program Files\IBM\ILOG\CPLEX_Studio128\cplex\bin\x64_win64;C:\Program Files\IBM\ILOG\CPLEX_Studio128\cpoptimizer\bin\x64_win64;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files\Microsoft VS Code\bin;C:\Program Files\PuTTY;C:\Program Files\gs\gs9.16\lib;C:\Program Files\gs\gs9.16\bin;C:\Program Files (x86)\Gow\bin;C:\Users\hafid\Anaconda3\envs\python3.7;C:\Users\hafid\Anaconda3\envs\python3.7\python.exe;C:\Users\hafid\Anaconda3\Scripts;C:\Users\hafid\Anaconda3\Scripts\conda.exe;“C:\Users\hafid\Anaconda3\envs\python3.7\python.exe;C:\Users\hafid\Anaconda3\envs\python3.7\python.exe\Lib;C:\Users\hafid\Anaconda3\envs\python3.7\python.exe\DLLs;C:\Users\hafid\Anaconda3\envs\python3.7\python.exe\Lib\lib-tk;”;C:\Program Files\Java\jdk1.8.0_201\bin;C:\Program Files\Java\jre1.8.0_201\bin;C:\Program Files (x86)\Wolfram Research\WolframScript;C:\Program Files\Git\cmd;C:\Program Files\MIT\Kerberos\bin;C:\Users\hafid\AppData\Local\Microsoft\WindowsApps;C:\Users\hafid\AppData\Local\GitHubDesktop\bin;C:\Program Files\JetBrains\PyCharm Community Edition 2019.1\bin;;C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2019.1\bin;;C:\Progra~1\Java\jdk1.8.0_201\bin;E:\spark-2.4.2-bin-hadoop2.7\bin;;C:\Program Files\Docker Toolbox;C:\Program Files\KNIME;;.
java.runtime.name=OpenJDK Runtime Environment
java.runtime.version=1.8.0_202-b08
java.specification.name=Java Platform API Specification
java.specification.vendor=Oracle Corporation
java.specification.version=1.8
java.vendor=AdoptOpenJdk
java.vendor.url=https://adoptopenjdk.net/
java.vendor.url.bug=https://github.com/AdoptOpenJDK/openjdk-build/issues
java.version=1.8.0_202
java.vm.info=mixed mode
java.vm.name=OpenJDK 64-Bit Server VM
java.vm.specification.name=Java Virtual Machine Specification
java.vm.specification.vendor=Oracle Corporation
java.vm.specification.version=1.8
java.vm.vendor=Oracle Corporation
java.vm.version=25.202-b08
line.separator=

quaeler · June 29, 2019, 2:55pm

Hmm… this configuration shows your modified -Xmx in it though; sorry i wasn’t clearer - i meant “what happens if you download a fresh 4.0.0 from knime.org, unzip it, and run it fresh (keep it vanilla, don’t touch anything)” does the mouse-over on the heap bar then show the default (2048) max setting?

hafidz · June 29, 2019, 3:09pm

Actually, I did just that. It max out at 512M. Not responding to any type of input from the .ini file.

quaeler · June 29, 2019, 3:15pm

I’m stumped :- (

acommons · June 29, 2019, 11:15pm

So maybe it’s not reading that .ini file. Three suggestions; [1] change something else in the .ini file that should have a visible impact and see what happens, [2] rename the knime.ini you are looking at so that Knime won’t find it and try again (I don’t know if this will have side effects), [3] search to see if you have more than one knime.ini on your drive.

quaeler · June 29, 2019, 11:18pm

(The dump from KNIME that he lists above - the one that starts

*** Date: Saturday, 29 June 2019 at 17:59:35 Singapore Standard Time

*** Platform Details:

*** System properties:

lists his custom -Xmx value… so it’s being read, oddly.)

acommons · June 30, 2019, 12:23am

@hafidz & @quaeler Showing my ignorance here

Where are you getting that dump from in Knime? None of my log files look anything like it.

quaeler · June 30, 2019, 12:37am

No worries - KAP has all sorts of little twists and turns within it. GIF attached!

acommons · June 30, 2019, 12:43am

@quaeler Nice!