I use a Python Script (2 => 2) (Labs) node to create 6 csv files, which are then used as an input for a certain function, ran from the command line. This creates 6 new csv files which are read and put together in two KNIME tables.
After looping through 4 files, everything just stops (as seen in the Task Manager). Meaning that Python stops doing anything, as well as the .exe that is ran from the command line. The node is stuck at 30%, without warning or error, no matter how long I wait. But after cancelling the Python Script node, the acitivity continues (seen in Task Manager) and I see the 2 remaining output csv's appearing in the folder.
If I only loop through 3 files, the node is executed untill the end.
If I use smaller files, I can go up to 5 files that are processed, after which everything stops again.
If I run in Spyder, everything works perfectly.
From all this, I can only conclude it has to do with the KNIME node and maybe its memory usage. I saw one question with the exact same problem, but without a solution.
I considered splitting it in two Python Script nodes. However, in the end I will use the workflow to process larger input files thus creating larger csv files in the node. My worry is it will get stuck even if I process 1 csv file per node.
Thanks in advance