I have a situation in which there is a file reader within a flow variable loop, where the number of columns read from each file will vary by one or two from one file to the next. I have no way of controlling or predicting when this will happen, or what the additional columns will be called. The problem is that the loop end node wants all iterations to end with the same number and type of columns. Could the loop end node(s?) be modified such that they allow options for "union of columns" or "intersection of columns", as is seen for the concatenate nodes?
Failing that, does anyone have an workaround for this situation? (Or is there a node I have missed, equivalent to the Add Empty Rows node, for adding empty columns? If not, perhaps one might be useful?)
Thanks
Steve
Hmm.
Thats an interesting problem, and I agree, its something that would be useful if the Loop End node could cope with differing column names and/or differing number of columns.
The only suggestion I can think of is to try using a Transpose node, and then a Loop End (Column Append) node and then Transpose back again.
Would that work?
Simon.
Another option is to use the Extract Column Header node, this will give all the columns with default names "Column x".
Now concetenate this dataset with the "Create Table Structure" node. In this specify a maximum number of columns you think you will need. You should now end up with the same number of columns reaching the Loop End each time.
Now connect this concetenate node to the Loop End node.
Simon.
Thanks, Simon. I used something similar to your second solution - I just added an arbitrary number of columns to the most that I had seen and used a column rename node to make sure that the column names matched before the concatenation.
For anyone else trying this, it seems to be important to put the output of the "Create Table Structure" node branch into the top in port of the concatenate node, otherwise the loop end node still gets confused because it treats the columns in the order they appear in the table, not by trying to match them by name. Perhaps this is also a bug in need of repair?
Steve
I am now also having this issue with several workflows I am working on.
Ideally need the Loop End node to tolerate columns with variable names between loops. Not supporting this is adding significant complexity to my workflows.
I assume this wouldnt be too difficult a feature request ? Column names appearing in one loop but not another loop can simply have the column cells filled with missing values.
Simon.