I have a workflow which is not to complex and in total it takes 20-30 seconds to execute. When I then add nodes which should write to CSV and Excel files a major issue occurs. Generally I don’t want to write to CSV but I had to use it as an option in my case.
I have a table of approximately 260.000 rows and 25 columns. There was never a problem during all node executions until I start to write to a new Excel file. This table size took me 50 minutes to write ending up in an Excel file of 26MB. I then added CSV writing node to see if it went quicker, and it did. It was done in about 10-15 seconds and finished with a size of 57MB.
This slow speed of writing to Excel feel very wierd. I tried to increase the xmg in the Knime.ini file to 12111MB (I have 16GB RAM) and also the number of cells in memory to 10.000.000 but the speed did not improve at all. When I opened up taskmanager in Win10 to see if it was using 12GB of memory it was only using 1.5GB. What is going on here and is there some way to speed this extremely slow speed up?
Currently I feel like I could have done the entire operations with Excel formular or VBA within that writing time that it took me in this case. If I can’t speed it up I don’t see the benefit of continuing to use Knime.
Knime.ini file content: