@mlauber71 Thanks for your answer!
The main problem with this approach is the way I want to process the data. I would like to keep the data in memory, and work with it in multiple different components afterwards.
I can only guess how much slower would it be to write it to a ZIP and read in every time there is a python script to python script connection.
In every file there are ~7 million float values (there are some way smallers aswell), and the breakpoint seems to be at the 18. file
For the context 7 million floats equals a 2 hours long record of ECG data.
@botichello maybe you give it a try. How large would the ZIP file be?
Can you save the data as a ‘standard’ Pandas Data frame? Then you might be able to benefit from the new Columnar storage integration mentioned that would speed up the transfer of data between KNIME and Python.
Then you could try and optimise your settings within the Python code. You could delete unnecessary items or you could try and run Garbage Collection from within the Python environment.
One last thing you could try is if the large individual job would run to put a conda environment propagation into the loop and reset the Python environment in every step in order to freiem memory. You would then have to have a loop in KNIME and restart the Python environment in every step.