I set up knime.ini in order to increase the java memory knime should use. I set:
but when executing heavy streams, that perform joins and transformation over millions records tables, and setting some nodes to 'memory execution', PC reaches a maximum 7.5GB use of memory and then has to use HDD.
knime 3.1.1 64b
System: i7 recent generation, 24GB RAM, windows 10 64b
Can anyone help?? I need knime to use all the memory allocated, as it would dramatically improve performance (I expect so), which is quite poor now.
Thanks a lot in advance!
First of all, but actually not really important: I think you don't have to set MaxPermSize anymore since Java 8. Second, I think your problem might be related to https://tech.knime.org/faq#q18. You can try to increase the cellsInMemoryValue to 1000k and see if your memory consumption goes up.
Hope this helps,
thanks a lot for the answer Christian, but I there is no cellsInMemoryValue in my knime.in.
Should I add the parameter? could you please provide me with the exact syntax?
yes, just add the parameter according to https://tech.knime.org/faq#q18
Add the following line after the -vmargs line in the knime.ini file to keep at most 1000 cells in memory (you can choose a different value, even 0 in which case it always swaps to disk):
The parameter is optional, thats why it's not there.
Let me know if it helped.
Christian, I found the way to change that line as:
but if I go then to a node and hover on the option 'keep only small tables in memory' it kees saying 'tables with less than 0 cells are kept in main memory, otherwise...'
Same with 1000k instead of 1000000.
Can you please help with this? thanks a lot!
There was a problem in 3.1 that the tooltip in the dialog wasn't updated correctly. This will be fixed with the official release of 3.2.2 (or 3.3 ... which is due next week and which will come before we release 3.2.2.). There is another thread similar to this here and as a result of this we fixed it in the code.
Hope this helps/clarifies. (Note, you can already double-check values by looking at the log file .. it's only a bug in how we compose the label that is shown when you hover over the options.)
thanks Christian, that's what i did but it doesn't seem to be working.
I add the parameter,
restart knime, add a new stream-> new node -> hover over 'keep only small tables in memory' and the messagge is "tables with less than 0 cells are kept in main memory".
It should be "tables with less than 1000000 cells are kept in main memory", shouldn't it? Is there another way to test it? am I doing something wrong?
I've had a look at the thread, but yet can't find a message starting with "Setting max cell count to be held in memory to " nor "Unable to parse property org.knime.container.cellsinmemory, using default" in knime.log.
To make sure I changed again the parameter, closed knime, checked last rows in logs... nothing.
There must be something I am doing wrong...
The tooltip fix is now officially online so please double check with 3.3.x or 3.2.2, whichever you prefer.
Hi Bernd, happy new year!
After updating to 3.3.1,
in knime.ini parameter: -Dorg.knime.container.cellsinmemory=1000000
tooltip: "tables with less than 100000 cells are kept in main memory"
So, 1M in .ini but 100k in tooltip (???)
Anyway, the memory consumption should increase after icresasing the cellsinmemory parameter but doesn't, it keeps increasing until 7,5GB, although the stream is far to be finished.
Any suggestion? thanks a lot in advance!
no suggestions for the issue that knime only uses up to 7.5GB with very large and wide sets instead of reaching either the kime.ini -Xmx16G-XX or the system limit?
it's a problem for me not using all the memory allocated when working with large and wide datasets.. no one has seen this same issue?
thanks a lot!
I have the same issue.
My machine has 16GB Ram. By default KNIME could use 8GB, but it looks like that it never use more than 3GB. I changed the value for cells in memory to 1000k, but the same behavior like before.
Is there anything else which I can configure?
Thanks for help!
Does one of you found a solution for the mentionned issue of RAM allocation ?
Thanks in advance,