FILE READER freezes - Part (2)


I noticed you have the line -Dorg.knime.container.cellsinmemory=10000000 in your knime.ini. This is much larger than the default of 1,000 and entails that KNIME AP holds tables with up to 10 million cells in memory no-matter-what until memory becomes critical. Before this critical memory condition is met, the garbage collector will take up more and more of your CPU cycles, which could explain the slow-down. Note that, for instance, a String cell can easily require 100 byte or more memory. Therefore, holding multiple tables with 10 million cells in memory can quickly fill up your 53 GB of main memory. I suggest to take this line out of your knime.ini and leave the caching of tables in memory to KNIME AP. The -Dorg.knime.container.cellsinmemory option is mostly a remnant from earlier days, when KNIME AP did not have an elaborate table caching mechanism.

Also, I was wondering:

  1. Which version of KNIME are you using? KNIME 4.0 is more aggressive in using heap space than KNIME 4.1, so I suggest an upgrade if you are still on 4.0.
  2. If you set your console / log level to DEBUG in your KNIME preferences, do you observe memory alerts in your console or knime.log? If so, what exactly is being reported?
  3. If you configure the memory policy of nodes in your loop to “Write tables to disc”, does the heap still fill up and the workflow still slow down?



1 Like