Memory usage - Excel Cell Updater \Read Sheet Names Nodes

[asdsad]

Hello everyone,
I want to describe some issue I encountered,
I tried to write to an exist Excel file, using the node Excel Cell Updater,
the size of the file is 74 MB, pretty big file with more than 20 sheets,
Excel Reader Node can read it on something like a 10 sec using 1.5g of heap size,
but when I try to write\update some cells in the file(11k cells) within a sheet with 17k rows and 80 columns, KNIME crushes with ‘java heap size error’ , so I increased the heap size and noticed that to write to this file I need to use more than 25g

*Similar issue when using Read Sheet Names Node

I would like to hear solution to the issue or other options that will decrease heap size usage

Thanks,
Rami

[mlauber71]

@asdsad not sure about the specific new excel cell updated node but these two entries might get you ideas how to increase performance. This might also help with the Excel nodes.

Knime processing power blocking at every single step after finished flow

@Gonzo welcome to the KNIME forum. You could try and give KNIME 10-12 GB but not more and not run any other jobs. Is this a Windows machine and does it have an SSD hard drive. Then you could explore this collection about KNIME and performance You might try: Garbage collection columnar table backend splitting the workflow into several parts and calling them separately while resetting them at every start (maybe a last resort)

Knime processing power blocking at every single step after finished flow

@Gonzo I have no experience with Windows 11 as of now. These things come to my mind when you mention a large ungroup (or any other complex operation for that matter): the Cache node. Sometimes I have experienced that before complex operations after a long data manipulation stream KNIME ‘likes’ a Cache node in order to bring all the changes back to one place then you could see if forcing the node to write something to disk can help, also if you try different underlying encodings in the knime.i…

The other option you could explore would be to use python and openpyxl to do more complex manipulations on Excel files.

Data extraction from .xls file/s

@Matt9301 there is a way to do this with the use of Python pandas and openpyxl [image] You would have to provide an existing excel file and a sheet name and then sheets with the same structure. [image]

Append data to an existing Excel file with the help of Python Pandas and...