Out of heap errors reading excel files with 16GB RAM

[srimalj]

Hi

I’m running Knime on a 16GB RAM Intel I7 machine.

I have a sizeable workflow.
knime stops and comes up with out of heap space errors. I understand that knime keeps objects in memory which is what allows me to run each node at a time. This is with 8GB set off in the ini file for knime.

I’ve managed to work around the issue by using flow variables to force knime to read the excel files one at a time.

Is there a better way to do this like a generic option which stops knime from trying to read several excel files at the same time and running out of heap space.

Also is there a global option or a parameter I can set to make knime not keep objects in memory when I just want to run the the workflow in production and don’t need to debug each step but prefer knime to use less memory?

Any adivice or experience would be much appreciated.

Thanks

[mlauber71]

You can tell each node to store its results on the disc that could save RAM

Knime 3.6 Crash when dealing with massive data

you could try and increase the assigned memory in the knime.ini configuration file but with only 4 GB there are limits to that (https://www.knime.com/blog/optimizing-knime-workflows-for-performance). On other option is to set the “Memory Policy” of the (sorter) node to “Write tables to disc”. Obviously this increases the load on the hard drive but it might save on RAM. [image] Also you could try and close other workflows, restart KNIME an try to use the garbage collector. [image] [image] T…

Managing the flow of data with flow variables is a good idea.

The you could check other tips to increase performance.

Optimizing KNIME workflows for performance | KNIME

The another idea could be to increase the RAM allocated to KNIME to 10 or 12 GB and not have other big programs running.

Then you might give us a better idea how your workflow looks like and what it does so we might see how to improve things.

[ipazin]

Hi there @srimalj,

What KNIME version are you using?

@srimalj

I’ve managed to work around the issue by using flow variables to force knime to read the excel files one at a time.

Does this impacts your time execution a lot? When you say flow variables you mean flow variables connections (the red ones), right?

Br,
Ivan

[srimalj]

KNIME Analytics Platform 4.0.2

Release date: October 1, 2019

[srimalj]

Well its way faster than when knime tries to read in all the excel files at the same time, and runs out of heap

[ipazin]

Hi there @srimalj,

@srimalj

Well its way faster than when knime tries to read in all the excel files at the same time, and runs out of heap

Then it seems you found a solution

Additionally to learn a bit more on memory and KNIME check out these two topics:

Heap space exhaustion (PC)

KNIME V4.0.1 Windows 10, 64 bit 32G RAM -xMx16384m I’m suffering from JAVA Heap exhaustion which appears to occur after a period of time when performing a series of operations. I can’t pin it down exactly as it sometimes occurs when performing Joins, Splits, Filters, and other table operations. The only common theme appears to be when the workflow has several branches which may execute at the same time, though I have also had a heap exhaustion when these branches are concatenated and there w…

Memory/Heap space issues in version 4+

KNIME V4.0.1 Windows 10, 64 bit 16G RAM -xMx10240m Hello folks, Not sure if specific to me - or to the particular dataset that I’m processing - but I’m experiencing severe heap/memory related issues in some nodes - like column appender, column list loop and even round double. Feels like memory leak - memory is not released and after some time/couple of iterations it results in out of memory exception. Had to revert back to 3.7 and it works without issues now. This is just FYI - would appre…

Br,
Ivan