Optimizing KNIME Workflow on a Smart PC: Tips for Heavy Data Processing

[mlauber71]

@harry_1 welcome to the KNIME forum.

@harry_1

tasks like reading multiple large Excel sheets with the Excel Reader node sometimes lead to freezes or “Java Heap Space” errors.

For Excel in the past sometimes employing Python /OpenPyxl could help:

Increased Excel file size and Excel Writer sheet append issue with KNIME 4.6.0

@alexkrah you might have found another solution but at least on a Mac, it is possible to write a file of your size into a large existing excel file with the help of the bundled Python and OpenPyxl although I would not necessarily recommend it … [image]

You can also try and use Apache Arrow which is more ‘native’ to KNIME and might speed up the process.

Excel - Import very large File with Python and Apache Arrow – mlauber71

You should leave some memory for other processes besides KNIME in general. How much RAM do you have and how large are your files.

Some further ideas about KNIME and performance are in this article:

Mastering KNIME: Unlocking Peak Performance with Expert Tips and Smart Settings

And about the handling of very large files:

KNIME Snippets (1): Collect and Restore — or how to handle many large files...