Recursive loop performance issue

Hi guys,
yesterday i’ve uploaded a workflow that use recursive loop in order to achieve the result wanted by another knimer in the forum.

i suddendly realized how bad is the performance using this kind of loop.
disclaimer: i’ve allocated only 4gb to knime (yeah i know, really low…), but we’re talking about 30 rows by 4 columns to be looped… i mean, nothing enormus.

first iteration go very well, but then the ram usage start to drastically increase like a bad resource allocator here.

i’ve tried to switch all the node to cache the data on disk, seems slightly better, but still unusable.

any guess?

Lucarecursive loop performance issue.knwf (36.1 KB)

This isn’t going to help your situation, but I can confirm that I see the same behavior in that it’s a very expensive loop to run. If you are able to increase your ram allocation that is good as well as maybe using the nodes to force garbage collection after each iteration may help. Another thought would be to read in the starting data do the manipulation overwrite the first data table and use a counting loop to run the number of iterations if you know how many times it needs to be processed.

Hope this helps,
Jason

1 Like

Hello @Luca_Italy,

have checked it and problem is in Cross Joiner node as it concatenates RowIDs. Doing so in a Recursive Loop leads to RowID values being longer and longer with each iteration which obviously slows down execution. Solution is to add RowID node to “reset” RowIDs anywhere between Cross Joiner and Loop End :wink:

Br,
Ivan

4 Likes

wow, now we are talking about performance ;D

thank you very much

2 Likes

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.