Metanode execution order

Is it possible to make one metanode wait for a previous metanode to finish…
I can’t get it to work with the workflow variable connection (of which I found an example somewhere on the Knime sites).
That works for individual nodes, but not for metanodes.
Kr,
Piet

Hi @Piet_Van_de_Weghe,

two options come to my mind: you can either explicitly pull a flow variable connection through output of one and to the input of another metanode or you can wrap your metanodes - wrapped metanodes have the flow variable connectors on the top similar to regular nodes.

Cheers,
Misha

1 Like

Hi Misha,

Thank you for your input.

Now I’m actually watching this video on Wrapped Metanodes :
https://www.youtube.com/watch?v=v2eefyiwasI&t=281s
and at about 06:20, the lady shows the ‘Job Manager Selection’.
Now I don’t get that tab, I only get the first 4 tabs.

But then I noticed I was looking for it in the wrong spot. I was calling the Configuration Window INSIDE the Wrapped Metanode : She calls the ‘Configuration Window’ from OUTSIDE the wrapped metanode. However, when I do a Right Mouse Click on my Wrapped Metanode, when I’m outside the node, the option ‘Configure’ is grayed out/disabled…

Any ideas ?

Thx,
Kr,
Piet

1 Like

Hi Piet,

you are right, one can configure streaming of wrapped metanode execution in the ‘Job Manager Selection’ tab of the wrapped metanode configuration. This window is indeed opened from outside by either Right click -> Configure... or F6 or Double click.

Could you share a small example? I was not able to construct a use-case, when the configuration window of a wrapped metanode is disabled. Thanks in advance.

Cheers,
Misha

Hi Misha,
Maybe a silly question, but how can I best share my example with you ?
I mean, a screenshot of my Wrapped Metanode, and the RMC - menu with the Configure option disabled is hardly going to learn you anything, is it :wink: ?
(F6 and Double Click don’t react either, which is kinda logic in this case)

FYI, what I was actually trying to accomplish, was the following : Imagine a flow containing two DB extractions from some external DB, that can run in parallel, and those extractions are then written to two target tables in ‘my’ DB. Next, I want to continue with some more selections on those tables, that can again be done in parallel. All of those staging actions of course have to wait for the two ‘Extractions’ to finish. So obviously, I made an Extraction Metanode and a Staging Metanode. But no matter how I played around with the out and in ports of those Regular Metanodes and drawing connections between those ports, that just did NOT force the ‘Staging’ Metanode to wait for the ‘Extraction’ Metanode.

But now I’ve altered them into Wrapped Metanode. And I see that this automatically generates WrappedNode Input and Output Nodes inside the Wrapped Metanode.
And it seems that those Nodes immediately provide an Out-Port on the Wrapped Metanode. I’ve linked the two Wrapped Metanodes that way, and I clearly see that now,
the ‘Staging’ Wrapped Metanode is properly queuing after the ‘Extraction’.

And this even works without doing any linking INSIDE the ‘Extraction’ node.
What I mean by that is the following : My Extracion metanode finishes with two parallel, independent DB Writers. Those are NOT linked to anything else (so not to the Wrapped Node Output, and not to some manually defined Out port), and also the WrappedNode Output is not linked to ANYTHING.

And still, I get the ‘wait for the preceding metanode’ - effect I was searching for.

image

So that’s good.

The thing is, that I was actually investigating to get this effect via the ‘Job Manager Selection’ (No idea if that would have helped, as I’m not sure what’s precisely meant by ‘Streaming Excution’ in this context).
Hence my original question. And now I’ve stumbled on a solution by chance.

But still, any enlightment/tips/information that you can give about my lengthy explanation, is surely welcome …

Kind Regards,
Piet

1 Like

Hi Piet,

but how can I best share my example with you ?

You can export a workflow: Right click on the workflow in the KNIME explorer on the left -> Export KNIME Workflow.... This way you can get a .knwf file that you can share (for example, by attaching it to your post here on the forum). Of course, we do not advice to share any sensitive data on the forum, so the example would have to have personal data and DB connections stripped off.

And now I’ve stumbled on a solution by chance.

So you have already resolved it in the meantime? That’s good news. The only additional hint is that I would connect the Extractor node to the flow variable port on top-left of the Stager node. This way you do not need to have an explicit input port on the Stager input (unless you actually need it). But this would still achieve the goal of staging the two steps to run one after another.

No idea if that would have helped, as I’m not sure what’s precisely meant by ‘Streaming Excution’ in this context

The idea of the streaming in workflow execution means that a following node will start execution already when only a chunk of data from the preceding node is ready. There is a nice blog post on the topic here.

Cheers,
Misha

1 Like

Thx for the info !
Kr,
Piet

1 Like

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.