Recursive loop very slow after 10 iterations

Recursive loop is a very powerful and wise however its speed slows down significantly after 10 iterations. Hope it improves with the time.

Hi @DeMytrych,

do you have an example where is performance gets bad over time?
Maybe then we could give you suggestions how to keep it going faster :slight_smile:
I think it highly depends on how the data is forwarded to the loop :thinking:

The data quantity is not too big. However nodes like JOIN and Rule based row filter gives all the problem.
Join actually even stop working somewhere around iteration 30+. It gives some error don’t remember exactly but it says Java… So i have created another workflow that don’t remove rows but rather do some data modification via nodes like lag column, Rule Engine, math formula etc… Also i have chosen an option to pick up last iteration only. Actually excel can do it in meter of seconds… with something like countif formula :slight_smile:

Hello @DeMytrych,

loops do know to become slower as iterations go further. As @AnotherFraudUser it would be best to share example workflow so someone can check it or suggest optimization steps.

One suggestion is to use new and improved joiner node available with KNIME 4.2.0. - Joiner (Labs) :wink:

Br,
Ivan

2 Likes

Thank you. 4.2 is not yet integrated for our organization but definitely if will try it whenever its possible. Sorry but i’ve delted to wf so cannot share it. The newer ones looks like this:

Hi @DeMytrych,

I see. Well then I can suggest this blog post about optimizing KNIME workflows:
https://www.knime.com/blog/optimizing-knime-workflows-for-performance

Hope you’ll find something applicable for your workflow :wink:

Br,
Ivan

@ipazin and @AnotherFraudUser hi. Thank you for your help and useful discussion!!!

1 Like

Hello @DeMytrych,

check this topic: Recursive loop performance issue

Probably same issue you had. RowIDs growing longer and longer with each iteration due to Joiner node.

Br,
Ivan

1 Like

Hello @ipazin . Thank you very much for your help. i’ve solved it in my own way however what you wrote make sense! Hopefully i will remember it next time when i need to use recursive loop.

Thank you again!!!

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.