GC Overhead limit

I have used Knime for over 2 years however with the new project I have close to 25 million records.
I am running into GC Overhead issue.

I have a I-7 machine with 16 gigs of ram and Knime is the only software running on the computer

Please help me improve and reduce errors/

Hi @rkagrawal,

Please have a look at this blog post: https://www.knime.com/blog/optimizing-knime-workflows-for-performance

The two most important factors in your case are probably the -Xmx and -Dorg.knime.container.cellsinmemory parameters in your knime.ini. What values are you using here? Note that by default, cellsinmemory is not listed in the knime.ini. In that case, it uses the default of 100,000.
With your machine, A much higher number would be appropriate. You can first try around 10,000,000, and maybe increase even further if things run smoothly.
For -Xmx, you can use 12G, which will still leave enough room for some other processes outside KNIME :slight_smile:

Hope that helps!

Cheers,
Roland

1 Like