Python scripts in Loops

@ActionAndi I think if you wrap a loop around a Python node the node will always restart a python session which might take some time - you might only shorten that to some extent by creating a small conda environment or use the internal one (How to Set Up Your Python Extensions | KNIME) and also if data is being transferred make sure to use the new columnar backend (Data Transfer between KNIME & Python Just Got Faster | KNIME).

Parallelisation might help but only to a certain extent maybe and I have not tried that with python (Parallel Chunk Start – KNIME Community Hub).

Other than that you might want to think about doing the loop within Python then the setup would only occur once. You would then export the PNG files directly from within Python:

Also: with such a large number of items it might make sense to store the ones you already have processed in a table so you can restart the process if something should happen.

2 Likes