If this is not related to problems we are seeing with Loops in the latest KNIME release 3.7 you could check out the various hints concerning KNIME performance.
It could make sense to try and split the workflow up into several parts and see if that makes any difference. You could also try and monitor the Java Heap Space and see when that reaches a critical point.
KNIME performance
Will the large dataset be imported and the problems will occur later or do you experience problems also when loading data? If you have problems with import it might help to use the R library Readr instead of KNIME file reader.
I know of no formal restriction for data in KNIME but it depends on how much power your system might bring. These points might be able to help you
check out the links below with hints from KNIME and other information about performance
with large datasets from CSV it mig…
you could try and increase the assigned memory in the knime.ini configuration file but with only 4 GB there are limits to that (https://www.knime.com/blog/optimizing-knime-workflows-for-performance ).
On other option is to set the “Memory Policy” of the (sorter) node to “Write tables to disc”. Obviously this increases the load on the hard drive but it might save on RAM.
[image]
Also you could try and close other workflows, restart KNIME an try to use the garbage collector.
[image]
[image]
T…
You might want to check your virus scanner. On certain systems with quite aggressive virus scanners I experienced KNIME to be slow. Since it can contain several 1,000 single files an ‘agressive’ virus scanner might scan all of them every time they are accessed/opened. When we changed the settings KNIME started faster. Of course there is a trade off between security and speed. But modern virus scanners might be able to have a flexible configuration.
But of course the number of AddOns and the ove…
That is certainly odd. To the best of my knowledge, there haven’t been any recent major changes to KNIME that are known to have a degrading effect on performance. From my experience, the most dominating performance factor in KNIME is the materialization of intermediate data tables on disk.
Have you noticed the performance of your hard disk degrading independently of KNIME?
While we have some performance improvements for KNIME in the pipeline, here are some small adjustments you can try out:
Y…
1 Like