I noticed you have the line -Dorg.knime.container.cellsinmemory=10000000 in your knime.ini. This is much larger than the default of 1,000 and entails that KNIME AP holds tables with up to 10 million cells in memory no-matter-what until memory becomes critical. Before this critical memory condition is met, the garbage collector will take up more and more of your CPU cycles, which could explain the slow-down. Note that, for instance, a String cell can easily require 100 byte or more memory. Therefore, holding multiple tables with 10 million cells in memory can quickly fill up your 53 GB of main memory. I suggest to take this line out of your knime.ini and leave the caching of tables in memory to KNIME AP. The -Dorg.knime.container.cellsinmemory option is mostly a remnant from earlier days, when KNIME AP did not have an elaborate table caching mechanism.
Also, I was wondering:
- Which version of KNIME are you using? KNIME 4.0 is more aggressive in using heap space than KNIME 4.1, so I suggest an upgrade if you are still on 4.0.
- If you set your console / log level to DEBUG in your KNIME preferences, do you observe memory alerts in your console or knime.log? If so, what exactly is being reported?
- If you configure the memory policy of nodes in your loop to “Write tables to disc”, does the heap still fill up and the workflow still slow down?
Regards,
Marc