When i want to iterate over individual rows of a table, I mostly use the “Chunk Loop Start” with the “Rows per Chunk” set to 1. However, there are plenty of situations where it would be nice to have the current Row ID available in a flow variable within the loop. This would be similar to the column name being available as a flow variable when using a “Column List Loop Start”. As a workaround I sometimes use a “Group Loop Start” on the RowIDs but in that case I first have to use the “RowID” node to copy the row IDs into a normal column, and afterwards remove this column again; it is also a extra unnecessary work for the system trying to look for groups that are not there.
So my feature request is as follows: a “Single Row Loop Start” where the row ID of the current row is available as a flow variable within the loop.
The Table Row to Variable node is exactly what I am trying to avoid, is kind of ugly… Also, I noticed that there are two kind of Chunk Loops in my workflows: one type is where I can increase the chunk size and the loop still works, and the other type is where the chunk can only be 1, or else the loop will fail. I think it would be elegant to have different types of Loop Start nodes for these different constructs.
Example. Suppose I have a table with, for example, the body weights of the ostriches that get born at Berlin Zoo, one column per weighing session for each week after hatching (columns “week 1”, “week 2”, etc). I want to fit these weights with the polynomial regression learner. To use the polynomial regression learner, I need to have all the weights for one hatchling in a column, not in a row… one option would be to transpose the whole table and loop through the different ostriches with a Column List Loop start. Problem: transposing is slow for large tables. A Column List loop is also VERY slow for large wide tables. If I ever want to use that same workflow with a chicken farm where the workflow has to deal with millions of chickens, the workflow will take hours or days to complete.
Much more efficient to loop through the individual rows of the table, transpose that one row, and use that with the polynomial regression learner. It could look like this:
With a Single Row Loop Start the lowest two nodes would not be necessary. The Column List Loop Start already works like this, why not have a rowwise loop that works in the same way?
Checked it. Unfortunately seems a bit special use case so not in plans for now. But let’s keep topic open and maybe someone else has same/similar request and then it can be re-evaluated.
I just would like to add that the behaviour of this suggested “Single row loop start” would be identical to the “Group loop start” if this could be executed on the RowIDs as groups. Unfortunately, it is not possible to select the Row IDs as groups at the moment. Maybe this feature could be easily added, but in that case, it should not waste time looking for groups.
As a side note, the feature would be identical to the Python Pandas iterrows() method, where both the index and the row are made available to the inside of the loop.
