Call Workflow (Table Based) - execute failure (screenshot inside)

ipazin · February 27, 2019, 12:34pm

Hi there!

Usually there is a maximum but I haven’t encounter it. Doubt this is a problem.

The called workflow must contain at least one of the Container Input (Table) , Container Input (Variable) , Container Input (Credential) or Container Output (Table) nodes. From your print screen the called workflow is also calling another workflow. Does that workflow has at least one from above mentioned nodes? Also how does the first calling workflow looks like?

You can use Wrapped Metanodes as well. I personally like Call Local Workflow nodes more as things are a bit more organized from my point of view but they are usually used when being repeatedly used like in loop. This really depends on your use case and how you like to build your workflows

Here is link on example from KNIME public Server. Take a look.

https://nodepit.com/server/public-server.knime.com:80/06_Control_Structures/07_Workflow_Orchestration

Br,
Ivan

arp · February 27, 2019, 12:50pm

Here is the log output from trying to execute the call workflow node. It doesn’t tell me much apart from that the execution failed:

2019-02-27 13:41:28,995 : DEBUG : main : ExecuteAction :  :  : Creating execution job for 1 node(s)...
2019-02-27 13:41:28,995 : DEBUG : main : NodeContainer :  :  : Call Workflow (Table Based) 0:379 has new state: CONFIGURED_MARKEDFOREXEC
2019-02-27 13:41:28,996 : DEBUG : main : NodeContainer :  :  : Call Workflow (Table Based) 0:379 has new state: CONFIGURED_QUEUED
2019-02-27 13:41:28,996 : DEBUG : main : NodeContainer :  :  : dropzoneImagesNike 0 has new state: EXECUTING
2019-02-27 13:41:28,996 : DEBUG : KNIME-WFM-Parent-Notifier : NodeContainer :  :  : ROOT  has new state: EXECUTING
2019-02-27 13:41:28,997 : DEBUG : KNIME-Worker-5 : WorkflowManager : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 doBeforePreExecution
2019-02-27 13:41:28,997 : DEBUG : KNIME-Worker-5 : NodeContainer : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 has new state: PREEXECUTE
2019-02-27 13:41:28,997 : DEBUG : KNIME-Worker-5 : WorkflowManager : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 doBeforeExecution
2019-02-27 13:41:28,997 : DEBUG : KNIME-Worker-5 : NodeContainer : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 has new state: EXECUTING
2019-02-27 13:41:28,997 : DEBUG : KNIME-Worker-5 : WorkflowDataRepository : Call Workflow (Table Based) : 0:379 : Adding handler df94c2d1-4d50-437f-9c0f-6ee49000dd14 (Call Workflow (Table Based) 0:379: <no directory>) - 1 in total
2019-02-27 13:41:28,997 : DEBUG : KNIME-Worker-5 : LocalNodeExecutionJob : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 Start execute
2019-02-27 13:41:28,998 : DEBUG : KNIME-Worker-5 : Node : Call Workflow (Table Based) : 0:379 : reset
2019-02-27 13:41:29,000 : ERROR : KNIME-Worker-5 : Node : Call Workflow (Table Based) : 0:379 : Execute failed: Failure, workflow was not executed, current state is IDLE.
ROOT : EXECUTING (start)
ROOT (end)

2019-02-27 13:41:29,000 : DEBUG : KNIME-Worker-5 : Node : Call Workflow (Table Based) : 0:379 : Execute failed: Failure, workflow was not executed, current state is IDLE.
ROOT : EXECUTING (start)
ROOT (end)

java.lang.Exception: Failure, workflow was not executed, current state is IDLE.
ROOT : EXECUTING (start)
ROOT (end)

	at org.knime.productivity.callworkflow.table.CallWorkflowTableNodeModel.executeInternal(CallWorkflowTableNodeModel.java:132)
	at org.knime.productivity.callworkflow.table.CallWorkflowTableNodeModel.access$0(CallWorkflowTableNodeModel.java:112)
	at org.knime.productivity.callworkflow.table.CallWorkflowTableNodeModel$1.call(CallWorkflowTableNodeModel.java:99)
	at org.knime.productivity.callworkflow.table.CallWorkflowTableNodeModel$1.call(CallWorkflowTableNodeModel.java:1)
	at org.knime.core.util.ThreadPool.runInvisible(ThreadPool.java:615)
	at org.knime.productivity.callworkflow.table.CallWorkflowTableNodeModel.execute(CallWorkflowTableNodeModel.java:96)
	at org.knime.core.node.NodeModel.executeModel(NodeModel.java:567)
	at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1186)
	at org.knime.core.node.Node.execute(Node.java:973)
	at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:559)
	at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
	at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
	at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
	at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
	at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
	at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
2019-02-27 13:41:29,001 : DEBUG : KNIME-Worker-5 : WorkflowManager : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 doBeforePostExecution
2019-02-27 13:41:29,001 : DEBUG : KNIME-Worker-5 : NodeContainer : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 has new state: POSTEXECUTE
2019-02-27 13:41:29,001 : DEBUG : KNIME-Worker-5 : WorkflowManager : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 doAfterExecute - failure
2019-02-27 13:41:29,001 : DEBUG : KNIME-Worker-5 : Node : Call Workflow (Table Based) : 0:379 : reset
2019-02-27 13:41:29,001 : DEBUG : KNIME-Worker-5 : Node : Call Workflow (Table Based) : 0:379 : clean output ports.
2019-02-27 13:41:29,001 : DEBUG : KNIME-Worker-5 : WorkflowDataRepository : Call Workflow (Table Based) : 0:379 : Removing handler df94c2d1-4d50-437f-9c0f-6ee49000dd14 (Call Workflow (Table Based) 0:379: <no directory>) - 0 remaining
2019-02-27 13:41:29,001 : DEBUG : KNIME-Worker-5 : NodeContainer : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 has new state: IDLE
2019-02-27 13:41:29,002 : DEBUG : KNIME-Worker-5 : Node : Call Workflow (Table Based) : 0:379 : Configure succeeded. (Call Workflow (Table Based))
2019-02-27 13:41:29,002 : DEBUG : KNIME-Worker-5 : NodeContainer : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 has new state: CONFIGURED
2019-02-27 13:41:29,002 : DEBUG : KNIME-Worker-5 : NodeContainer : Call Workflow (Table Based) : 0:379 : dropzoneImagesNike 0 has new state: CONFIGURED
2019-02-27 13:41:29,002 : DEBUG : KNIME-WFM-Parent-Notifier : NodeContainer :  :  : ROOT  has new state: IDLE

The other workflow that’s being called is just a simple as the one displayed here. Some tables from a mariadb, some joining nothing and exactly one output node. Also it executes perfectly fine on its own and more importantly: it works when called from somewhere else inside the workspace.

I got permission from my superior to post all details about these workflows here, since we really like to solve this with KNIME and using multiple local table-based workflows.

ipazin · February 27, 2019, 1:41pm

Hi there!

I have created an example for you. While doing so I got the same error as you but it was something while configuring nodes so I just ran calling workflow and it disappeared.

Check out example, play with it a bit, inspect configuration and try to fix your workflow then. If you still encounter problems maybe you can share workflow so I can take a look. Instead of database connections and data from it you can put some dummy data.

Call_Workflow.knar (36.7 KB)

Br,
Ivan

arp · February 27, 2019, 2:32pm

Thanks for your example. Apart from the fact that your example uses an input-table and ours doesn’t it looks exactly the same. I have even saved/moved/recreated many of the nodes in question in order to make sure that there are no lingering cache/config/labeling problems.

Nothing so far seems to help. The error is exactly the same as always and even recreating the workflow from scratch didn’t help.

arp · February 27, 2019, 3:02pm

I rebuilt the node from scratch. Now it works. No idea what the problem was or how to reproduce it. If it happens again, I’ll file a bug report. Until then,
arp

ipazin · February 27, 2019, 3:02pm

Great!
Glad it works now
Br,
Ivan

Tobias_Urhaug · March 1, 2019, 8:27am

Hi arp,

I’m happy it worked out in the end. I’m not sure what the error might have been and the log doesn’t help me much either. If you figure out how to reproduce it please let us know so we can fix it and implement a proper error message.

All the best,
Tobias U.

arp · March 6, 2019, 4:19pm

Just wanted to let you know that this problem just occured for a colleague of mine who was working under very similar circumstances.

He tried to use Call local workflow (table-based) on a perfectly fine workflow but KNIME will not allow him to call it. Same error as above.

We tried everything and are running out of options. We are unable to recreate the problem on purpose or fix it. At the moment we are just trying to make it go away so that we can continue working on this project. As a last ressort we will recreate the workflow from scratch and see if that helps like it has for me before.

ipazin · March 7, 2019, 12:51pm

Hi there!

Hm… Seems there is something. Is there a possibility for you to share this workflows? You can use some dummy data instead of data from databases. That would be really helpful for troubleshooting as we are unable to reproduce it

Br,
Ivan

mlauber71 · March 7, 2019, 9:02pm

Maybe you would want to check out the structure I have proposed before using a table of the workflows to be called, then a loop to call them all and JSON information to ‘communicate’ between the main and subworkflows (even if there is nothing to communicate)

the sub workflow needs a start and finish with (in this case) JSON informations coming in and leaving the sub-workflow (I think it could be empty but it could be use it to track the time it took the sub-wf to finish so I can see if it worked at all).

Then it is very important that the sub-workflow are saved in a reset state otherwise they will just be run without changing anything. Mandatory reset I think is only possible in batch mode and with the KNIME server.

Instead of the Input and Output JSON Containers also Table input and output should be possible. Although I have not much experience with that. Instead of using them you could just save the data you want to transfer in a temporary table and reload it in the next step.

arp · April 10, 2019, 3:51pm

Quick follow-up:

found a somewhat reliable way to reproduce the “Execute failed: Failure, workflow was not executed, current state is IDLE.” problem.

It has to do with nesting Call Local Workflow nodes. After a certain depth, the knime client just gives up or has problems with ordering of tasks or waiting for them? We are not sure what goes wrong there but it all falls apart if you have workflows which call other workflows two tiers in.

If we nest Call Local Workflow nodes depth 2+ you can no longer run the highest tier workflow!

Our intermediate workaround for gui development is to reset all nodes and then run them from the bottom upwards, one by one, and manually.
Example: if A calls B and B calls C and C calls D, running A results in the above error code and everything goes to helll. If you reset all 4 workflows and run them manually in reverse order (D, C, B) then A runs perfectly. This is tedious and takes ages. Any hope for a fix here?

I hope this info can give you some hints on where to find the problem.

ipazin · April 11, 2019, 6:23am

Hi @arp!

Tnx for info. I will check this one and get back to you.

Br,
Ivan

ipazin · April 29, 2019, 10:03am

HI there @arp!

I have just created a simple workflow up to 4 called workflows (A calls B, B calls C, C calls D and D calls E) and it works just fine. So unfortunately I can not reproduce it. If you can share workflow with error that would be a huge help in investigating this further on and subsequently fixing it. Just use some dummy data if possible.

Br,
Ivan

arp · May 7, 2019, 3:05pm

I’ll try to compile you a dummy workspace but no promises as to when I get this done sorry.

ipazin · May 8, 2019, 10:18am

Hi there!

Sure. Tnx!

Br,
Ivan

arp · September 3, 2019, 1:03pm

Just a follow up which might be interesting info for you:

We have isolated the cause for the problem and it seems that it is some sort of internal ordering problem for KNIME. If call local workflows (table based) happen in a diamon shape then the execution fizzles and the error occurs.
We developed our own method to execute the workflows in a hierarchical order to circumvent this problem. Maybe you can take a look at your inner workings and see if you can resolve the issue.

Hope that piece of info helps. Cheers,

Tobias_Urhaug · September 10, 2019, 8:43am

Hi @arp

I’m no sure I understand what you mean by diamond shape. Is it the call order, i.e workflow A calls workflows B and C and then workflow B and C calls a workflow D? Could you share an example workflow or screenshot illustrating the problem?

thanks,
Tobias U.

arp · September 10, 2019, 10:40am

i.e workflow A calls workflows B and C and then workflow B and C calls a workflow D?

Exactly that.

kotlapullareddy · May 19, 2023, 11:27am

@arp
I wondered if you found any solutions for this issue.
recently, I encountered same problem while using call workflow node.