I have a workflow in KNIME that looks for certain attributes of adjacent rows and removes rows that are too similar to the row before them. Hence going in the table might be 5,800 rows and going out the table might be 5,740 rows. I repeat this process until the count of rows at the start is the same as the count at the end - hence no rows were removed.
Is there a way to set up a loop that will handle this? Right now I keep adding more nodes to my workflow until I reach the condition where rows_in == rows_out, it would be advantageous to have a loop handle this for me. I looked at āTable Row to Variableā but that doesnāt quite do it (unless I do some additional math to add the table length to a given row before I start the loop).
This is of course not the same as doing X number of loops where X is the number of rows; I generally do far fewer than that many loops. While it wouldnāt hurt to keep doing more loops after the condition is met, it would be a waste of time to do 5,800 loops when I only need to do 6.
My current workflow - which seems to only remove rows once regardless of how many iterations it goes through:
Math Formula (to count the initial rows)
Generic Loop Start
Several math processes to identify which rows to remove, and remove them
Math Formula (to count the rows after the above processes)
Math Formula (to subtract the new count from the initial count)
Table Row to Variable (takes the row count difference and makes it into a flow variable for the loop close)
Variable Condition Loop End
The last Math Formula node is connected to the input of Variable Condition Loop End; while the output of Table Row to Variable is connected to the same Variable Condition Loop End as a flow variable. I then set the Variable Condition Loop End to watch available flow variable ārow_changeā until it equals 0 and end the loop when that happens. The result unfortunately is that the table shrinks once and then never again; the difference never changes from the difference between the original table and the result of the first iteration. It seems as if the loop is reloading the original table instead of loading the table that was created by the iteration.
I just added another Math Formula node to the start of the loop that counts the rows in the table; it confirms that the table is being reloaded from its original state instead of receiving the result of the steps in the loop. How do I correct this?
Thank you for the suggestion, but I canāt come up with a way to apply it to what I am trying to do. In my case I donāt know how many times I will need to run the loop, I need it to run until the difference in row count between successive runs is 0. Iāve tried this with a recursive loop but it runs infinitely; I tried using a variable condition loop (where I set a variable to pass ātrueā if the row count difference described above is met) but that doesnāt work correctly either.
Iām not sure what Iām doing wrong here. I could have written this in Perl in about 10 minutes but Iāve been running in circles on this in KNIME for days now.
With the recursive loop, try using a Reference Row Filter to see how many rows from the start of the loop (i.e. the output table of the Loop Start node) remain immediately before the loop end. Use Extract table dimensions and then Java Edit variable to set s String variable as ātrueā is the number of rows is 0, and āfalseā otherwise. then use this flow variable to control the āEnd loopā setting in the recursive loop end
I finally got it to work with a chain of āRow Filterā -> āRule Engineā -> āTable Column to Variableā; then passing that variable on to āRecursive Loop Endā. It was not the most intuitive thing to put together but it works well now. I ran in to a large number of ways that did not work before then
There is Extract Table Dimension node which you can use to extract number of rows for two output tables you want to compare. This node outputs table dimensions in both table output and as flow variables. As KNIME can not have two flow variables with same name in one node (and you need to bring those two flow variables together to compare two table lengths) you need to rename at least one of them (you can use Rule Engine Variable node for example) and then follow it with Merge Variables node. After that again use already mentioned Rule Engine Variable node to compare two flow variables and output ātrueā in new flow variable if values are equal to stop Recursive Loop End.
Give it a try and if itās a bit too much I can make you workflow example.
