Knime 3.6 Crash when dealing with massive data

Hello,
It is the Knime 3.6 with win 7 (4G rom) platform for my work condition. However, when I was dealing massive data, around 6M record, especially for the sort node, it always crashed…
is there any suggestion that can avoid the situation

Thanks in advanced,
Kevin

you could try and increase the assigned memory in the knime.ini configuration file but with only 4 GB there are limits to that (https://www.knime.com/blog/optimizing-knime-workflows-for-performance).

On other option is to set the “Memory Policy” of the (sorter) node to “Write tables to disc”. Obviously this increases the load on the hard drive but it might save on RAM.

image

Also you could try and close other workflows, restart KNIME an try to use the garbage collector.

image

image

Then you could try to split your task into smaller subgroups or try to split it into several workflows so that not all data from one big workflow has to be present any time.

Also some programs find it easier to sort by numeric variables (Long Integers) instead of strings. So if you might convert or replace a string variable with a numeric one this could help. Obviously if you create some dictionary to have numeric IDs instead of strings this could add to performance issues.

Often in the end KNIME needs more power.

3 Likes

Hi mlauber71, thanks for your information. I will try.
regards,
Kevin