Hi,
I have to do ~200 times of row filtering within a loop that takes most of the time of the loop. Now the total filtering does take about 10 seconds what is 0,05 s per row filtering. That does you seem much but it add and is to much for a liver operation.
I already have done this:
parallelize the loop, so I have now 8 parallel loops. (the number of CPU cores)
put a “Cache” node before the loops. This improves performance by factor of 6-12!
So you run loop around 200 times with Row Filter within and some other operations or? Anyways I would try to create loopless solution. Is that possible?
I do interpolation (via missing value node) and the loops are for separating the different signals in the same data table. If I interpolate without loops I get interpolation errors at the “border” of the signal.
You mean an index for one or more columns in the database context? I did not know you can do it in KNIME. You mean the rank node as already told by you on Add index column with respect to group - #2 by morpheus . Does it really have speed advantages for filtering?
I did some improvements due to your hints.
I have 500.000 rows in 220 groups and do for each group the linear interpolation. I already tried to put the sorter out of the loop but it had no performance improvement.
Grouping with the rank node has no improvement if I consider the time for the rank node and the cache node has now also no advantages anymore.
This lasts 10-12 seconds.