Looping of File Reader is slow

KNIME Analytics Platform
1

Hi,

I used a List Files node and then loop over few thousand .csv files (small file sizes). Looping is very slow (~1 per sek), if the Loop End node is connected to the rest of the workflow. If I delete the connection between Loop End and the following node, it immediately goes 20-30-times faster. It seems that KNIME checks the configuration state of the downstream nodes each iteration. I noticed that especially String Manipulator nodes behind Loop End slows down looping. 

I am running KNIME on a MAC and sometimes notice bad performace, which I suppose is due to the OS. However, this issue looks like a KNIME (String Manipulator) thing...

Cheers,

Martin

2

Hello Martin,

interesting... - but we have trouble reproducing this effect. It could be related to the type of loop nodes that you are using, could you post/send the workflow or at least an image of the relevant parts?

Thanks! Michael

3

Hello Michael,

I could reproduce this effect on another MAC, on Windows and also loafin files from local HHD instead of from server. Here is the workflow and 100 .txt files in a folder, zipped. I hope this helps to trouble shoot this...

Best,

Martin

 

 

4

Hi Martin, 

Thanks for the repot, we've ientified and fixed the bug.  The fix will be available in KNIME 2.8 which is due out very soon.

 

Regards,

 

Aaron Hart

5

Hello Martin,

in the meantime you can also avoid this problem by adding a "Domain Calculator" node to your workflow, right behind the LoopEnd node.

Cheers, Michael