I set up knime.ini in order to increase the java memory knime should use. I set:
-Xmx16G-XX
MaxPermSize=2048m
but when executing heavy streams, that perform joins and transformation over millions records tables, and setting some nodes to 'memory execution', PC reaches a maximum 7.5GB use of memory and then has to use HDD.
knime 3.1.1 64b
System: i7 recent generation, 24GB RAM, windows 10 64b
Can anyone help?? I need knime to use all the memory allocated, as it would dramatically improve performance (I expect so), which is quite poor now.
First of all, but actually not really important: I think you don't have to set MaxPermSize anymore since Java 8. Second, I think your problem might be related to https://tech.knime.org/faq#q18. You can try to increase the cellsInMemoryValue to 1000k and see if your memory consumption goes up.
Add the following line after the -vmargs line in the knime.ini file to keep at most 1000 cells in memory (you can choose a different value, even 0 in which case it always swaps to disk):
-Dorg.knime.container.cellsinmemory=1000
The parameter is optional, thats why it's not there.
Christian, I found the way to change that line as:
-Dorg.knime.container.cellsinmemory=1000000
but if I go then to a node and hover on the option 'keep only small tables in memory' it kees saying 'tables with less than 0 cells are kept in main memory, otherwise...'
There was a problem in 3.1 that the tooltip in the dialog wasn't updated correctly. This will be fixed with the official release of 3.2.2 (or 3.3 ... which is due next week and which will come before we release 3.2.2.). There is another thread similar to this here and as a result of this we fixed it in the code.
Hope this helps/clarifies. (Note, you can already double-check values by looking at the log file .. it's only a bug in how we compose the label that is shown when you hover over the options.)
thanks Christian, that's what i did but it doesn't seem to be working.
I add the parameter,
-Dorg.knime.container.cellsinmemory=1000
restart knime, add a new stream-> new node -> hover over 'keep only small tables in memory' and the messagge is "tables with less than 0 cells are kept in main memory".
It should be "tables with less than 1000000 cells are kept in main memory", shouldn't it? Is there another way to test it? am I doing something wrong?
I've had a look at the thread, but yet can't find a message starting with "Setting max cell count to be held in memory to " nor "Unable to parse property org.knime.container.cellsinmemory, using default" in knime.log.
To make sure I changed again the parameter, closed knime, checked last rows in logs... nothing.
in knime.ini parameter: -Dorg.knime.container.cellsinmemory=1000000
tooltip: "tables with less than 100000 cells are kept in main memory"
So, 1M in .ini but 100k in tooltip (???)
Anyway, the memory consumption should increase after icresasing the cellsinmemory parameter but doesn't, it keeps increasing until 7,5GB, although the stream is far to be finished.
no suggestions for the issue that knime only uses up to 7.5GB with very large and wide sets instead of reaching either the kime.ini -Xmx16G-XX or the system limit?
it's a problem for me not using all the memory allocated when working with large and wide datasets.. no one has seen this same issue?
My machine has 16GB Ram. By default KNIME could use 8GB, but it looks like that it never use more than 3GB. I changed the value for cells in memory to 1000k, but the same behavior like before.