Hi again @takbb , first of all, thank you for such a detailed post.
As a component, that column rename element is a nice addition. I haven’t checked the component, but based on what you’ve written (and please correct me if I’m wrong), for this particular issue of mine, it seems that the component needs to be fed by #Iter.* strings right? I have no doubt that it’ll work when all column names contain that substring. As I mentioned in the post, the issue with the Loop End (Column Append) Node is that the first iteration (Iteration 0) doesn’t have that substring. Here’s a dummy outcome from a real result of that node to re-emphasize my point:
Since the info we have at hand is how many columns we have per group of iterations, it’s possible to use this info to rename the first x columns such that they include #Iter 0 substrings, before feeding the table into your component. What I can do is to manually rename the first x columns using the Column Rename Node. I can see that this is doable. Do you have any pointers as to how can I do it in a less tedious manner using the Column Rename (Regex)? (I’m guessing @ArjenEX might have figured out something by now, since this node was mentioned earlier.) I would appreciate any workaround that automates things as much as possible.
Let me first point out that as far as example workflows with Twitter API in the hub is concerned, it can be considered that iterating query variations in loops is uncommon. Most of the workflows up there shows that people are querying high volumes of Tweets and analyzing them one query at a time. While that’s something I do too, my task also involves many variations from that one query.
The typical loop end node is not able to hold the result from a previous query, as the incoming results overwrite the table. The typical way to overcome this hurdle is by writing the results on an output file, as shown in —> this workflow <— on the hub (not mine), or suggested —> on this post. <—
It’s just my personal choice to find my own way around it, which is by using the Loop End Column Append Node. I like to keep everything inside the workflow, and I don’t like creating unnecessary files that I won’t use. Saves me space, and avoids cluttering in the destination folder. Again, it’s just my personal choice 