Memory upgrade for KNIME

[j_ochoada]

Hi everyone,

Thanks in advance for taking a look at this post. I seem to be hitting memory issues in every type of workflow I do. Clustering large numbers of molecules, getting JSON info from API’s running recursive loops on large datasets. I’ve hit the ceiling and the most memory currently I have is 30G. The machine can be upgraded to 256GB. will this help can the client use all of it? Anyone else running KNIME (Client) with larger memory allocations?

Thanks,
Jason

[Snowy]

Have you allocated your machine memory by modifying your knime.ini file? By default I think KNIME only uses half of your available memory. I have 32GB and have let KNIME use up to 26 gigs by modifying the ini file.

And generally, yes, upgrading the amount of memory will help KNIME workflows run quicker, though some of it depends on the workflow and the nodes used.

One way you could test relatively easily is set up a virtual machine with upgraded specs to see if your KNIME workflow runs better with more memory/cores. A lot of times AWS or Azure will provide free credits, so you may be able to check for free.

[j_ochoada]

Hi @Snowy Thanks for the quick reply. Yes I have my ini file maxed and enough so that I actually crash the computer. Re the AWS or Azure, you are referring to the server yes? I’d eventually like to to get the server set up but right now I’m just trying to get my own work done

[Snowy]

You can get a virtual machine, install KNIME desktop on it, then run your workflow from there. Although KNIME server has an hourly rate as well, which could make the cost fairly effective.

[j_ochoada]

@Snowy Great to hear. I’ll take a look into it.

Thanks!
Jason

[mlauber71]

You might want to have a look at a collection of discussions about KNIME performance and how to tweak it. More memory will definitely help but there seems to be a threshold in relation to the overall memory of the system (KNIME might not start or freeze if there is not enough memory left for other processes and the system).

KNIME performance

Optimizing KNIME workflows for performance | KNIME

Process 900+ CSV files

Processing hundreds of millions of records

A few remarks since it is difficult to judge from the screenshots: if you have to process that many files and lines it could make sense to do it in chunks and have some method in place to log the current status and be able to restart at a certain point. Eg. write results to disk after each 50 out of 900 processed files or something. Because if you do not have a very powerful infrastructure, compartmentalization might be a thing for you also this saving of steps in-between might function as som…

Large data tables missing?

Will the large dataset be imported and the problems will occur later or do you experience problems also when loading data? If you have problems with import it might help to use the R library Readr instead of KNIME file reader. I know of no formal restriction for data in KNIME but it depends on how much power your system might bring. These points might be able to help you check out the links below with hints from KNIME and other information about performance with large datasets from CSV it mig…

Knime 3.6 Crash when dealing with massive data

you could try and increase the assigned memory in the knime.ini configuration file but with only 4 GB there are limits to that (https://www.knime.com/blog/optimizing-knime-workflows-for-performance). On other option is to set the “Memory Policy” of the (sorter) node to “Write tables to disc”. Obviously this increases the load on the hard drive but it might save on RAM. [image] Also you could try and close other workflows, restart KNIME an try to use the garbage collector. [image] [image] T…

KNIME Takes Long to Load?

You might want to check your virus scanner. On certain systems with quite aggressive virus scanners I experienced KNIME to be slow. Since it can contain several 1,000 single files an ‘agressive’ virus scanner might scan all of them every time they are accessed/opened. When we changed the settings KNIME started faster. Of course there is a trade off between security and speed. But modern virus scanners might be able to have a flexible configuration. But of course the number of AddOns and the ove…

Knime is slowing down

That is certainly odd. To the best of my knowledge, there haven’t been any recent major changes to KNIME that are known to have a degrading effect on performance. From my experience, the most dominating performance factor in KNIME is the materialization of intermediate data tables on disk. Have you noticed the performance of your hard disk degrading independently of KNIME? While we have some performance improvements for KNIME in the pipeline, here are some small adjustments you can try out: Y…

[j_ochoada]

@mlauber71 Thanks for the comprehensive view of the different topics. Other than the R loader, I had come across them but it’s always good to refresh. I have no issues in loading the data it’s the later processing and I am especially having issues using a recursive loop. It makes sense as everything is held and iterated on perhaps up to 50-100K times. I’ve chunked where I could but I recently thought of an idea. I was thinking instead of using a recursive loop, trying to find a way to run a regular loop or perhaps a recursive number of loops and write the file after each iteration. If I have the file reader with the same file listed 100K times then I’m using the IO over the memory and if it dies I have a partially processed file I can pick up from.

I’m going to see if I can’t implement that.
Thanks again!

[ipazin]

Hi there @j_ochoada,

probably should try different logic if possible (without recursive loop) and additionally optimize workflow execution. If you share your workflow design and logic (more details and possibly example) maybe community will have some ideas that help you speed it up and avoid memory problems

Br,
Ivan