I have a very large dataset that I need to filter with “tiered” filters. I have been successfully doing it is cascading workflows (the end of flow_1 starts flow_2 with a remote call) but it is really onerous to manage and update. The same result would be possible "if’ I could get a nested loop to work.
As soon as I connect nodes into the first loop I get the error “Unable to merge flow object stacks: Conflicting FlowObjects: <Loop Context (Head 3:57, Tail unassigned)> - iteration 0 vs. <Loop Context (Head 3:41, Tail 3:42)> - iteration 1 (loops/scopes not properly nested?)”
I understand that it is something to do with KNIME not being able to identify where the “start-end” of the looks are but I have not been able to find a way to resolve the problem, which I am sure is not unique to this flow.
First of all, I think it is a good idea for you to take a look at this blog post and know loops in KNIME better.
Then if you could not solve the issue by yourself, you can provide an example input and your desired output here and we can help you find the right solution.
Hi @armingrudd … I found the blog a great read and there some other articles that I will definitely return to read.
To return to this problem…
Ideally, when I get this sub-section of a very long process that takes about 2 hours complete (a new input file is created every 12 hours), this sub-flow will be built back into the flow that creates all the input for this one.
In the above flow (my original post).
Nodes ‘Filter List 0’ and ‘Filter List 1’ are output from nodes in the earlier KNIME flow. Each is a single column with values to be used in a filter node.
The CSV Reader ‘Source Data’ is the main output from the same preceding flow (contains approx 18 million rows). When this flow is reversed into the ‘master’ flow it will be output from a previous node, which is actually a column filter.
The rest of the flow is pretty straight forward, filtering the rows on category and sub-category that for now get written to output files but later will be passed directly out to the next section of the ‘master’ flow
What I need, is to (a) get is get this nested loop to work, or (b) if it cannot do what I need, find an alternative solution. I hope you have some insight and ideas.
Node 41 (the START node for the main loop) has to have a variable connector to Node 57 ( the START node of the nested loop)
Node 59 (the END node of the nested loop) has to have a variable connector to Node 42 (the END node of the main loop)
This makes sense. The main loop has to ‘notify’ the START node of the nested loop and in-turn the nested loop END has to ‘notify’ the END node of the main loop.
