Call Workflow (Table Based) - execute failure (screenshot inside)

Hi,

we are trying to build several smaller workflows and then put things together with call workflow (table based). So far this worked out well, but when I try to call this workflow, it just crashes with an error:

ERROR Call Workflow (Table Based) 0:379      Execute failed: Failure, workflow was not executed, current state is IDLE.
ROOT : EXECUTING (start)
ROOT (end)

Here is a screenshot of the workflow we are trying to call. As you can see, it is perfectly simple and executes without any problems on its own.

  • Is there a maximum to how many encapsulated call workflow nodes one can use?
  • Why is there no error message? It just crashes.
  • Should we use wrapped meta nodes instead? From what I understand, call local workflow is the way to go.

From what I understand the “Call Workflow” node at the beginning would call a workflow so this might result in a cascading call. If this is not the intention the you might want to change that.

To be honest I have not fully figured the call workflow business out but I have constructed an example that calls local workflows from a list and is able to iterate that. The data sent back is just a structured information in JSON format but might as well be an Output (Table)

Another remark: there should be a log file entry in the central knime.log maybe you could look again if you see something that might give us a hint.

Then there is a Python script in the subworkflow. Initially I do not see why a sub workflow might not be able to handle Python scripts. But do the other sub workflows that work also have Python scripts.

Hi there!

Usually there is a maximum but I haven’t encounter it. Doubt this is a problem.

The called workflow must contain at least one of the Container Input (Table) , Container Input (Variable) , Container Input (Credential) or Container Output (Table) nodes. From your print screen the called workflow is also calling another workflow. Does that workflow has at least one from above mentioned nodes? Also how does the first calling workflow looks like?

You can use Wrapped Metanodes as well. I personally like Call Local Workflow nodes more as things are a bit more organized from my point of view but they are usually used when being repeatedly used like in loop. This really depends on your use case and how you like to build your workflows :wink:

Here is link on example from KNIME public Server. Take a look.

https://nodepit.com/server/public-server.knime.com:80/06_Control_Structures/07_Workflow_Orchestration

Br,
Ivan

Here is the log output from trying to execute the call workflow node. It doesn’t tell me much apart from that the execution failed:

2019-02-27 13:41:28,995 : DEBUG : main : ExecuteAction :  :  : Creating execution job for 1 node(s)...
2019-02-27 13:41:28,995 : DEBUG : main : NodeContainer :  :  : Call Workflow (Table Based) 0:379 has new state: CONFIGURED_MARKEDFOREXEC
2019-02-27 13:41:28,996 : DEBUG : main : NodeContainer :  :  : Call Workflow (Table Based) 0:379 has new state: CONFIGURED_QUEUED
2019-02-27 13:41:28,996 : DEBUG : main : NodeContainer :  :  : dropzoneImagesNike 0 has new state: EXECUTING
2019-02-27 13:41:28,996 : DEBUG : KNIME-WFM-Parent-Notifier : NodeContainer :  :  : ROOT  has new state: EXECUTING
2019-02-27 13:41:28,997 : DEBUG : KNIME-Worker-5 : WorkflowManager : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 doBeforePreExecution
2019-02-27 13:41:28,997 : DEBUG : KNIME-Worker-5 : NodeContainer : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 has new state: PREEXECUTE
2019-02-27 13:41:28,997 : DEBUG : KNIME-Worker-5 : WorkflowManager : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 doBeforeExecution
2019-02-27 13:41:28,997 : DEBUG : KNIME-Worker-5 : NodeContainer : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 has new state: EXECUTING
2019-02-27 13:41:28,997 : DEBUG : KNIME-Worker-5 : WorkflowDataRepository : Call Workflow (Table Based) : 0:379 : Adding handler df94c2d1-4d50-437f-9c0f-6ee49000dd14 (Call Workflow (Table Based) 0:379: <no directory>) - 1 in total
2019-02-27 13:41:28,997 : DEBUG : KNIME-Worker-5 : LocalNodeExecutionJob : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 Start execute
2019-02-27 13:41:28,998 : DEBUG : KNIME-Worker-5 : Node : Call Workflow (Table Based) : 0:379 : reset
2019-02-27 13:41:29,000 : ERROR : KNIME-Worker-5 : Node : Call Workflow (Table Based) : 0:379 : Execute failed: Failure, workflow was not executed, current state is IDLE.
ROOT : EXECUTING (start)
ROOT (end)

2019-02-27 13:41:29,000 : DEBUG : KNIME-Worker-5 : Node : Call Workflow (Table Based) : 0:379 : Execute failed: Failure, workflow was not executed, current state is IDLE.
ROOT : EXECUTING (start)
ROOT (end)

java.lang.Exception: Failure, workflow was not executed, current state is IDLE.
ROOT : EXECUTING (start)
ROOT (end)

	at org.knime.productivity.callworkflow.table.CallWorkflowTableNodeModel.executeInternal(CallWorkflowTableNodeModel.java:132)
	at org.knime.productivity.callworkflow.table.CallWorkflowTableNodeModel.access$0(CallWorkflowTableNodeModel.java:112)
	at org.knime.productivity.callworkflow.table.CallWorkflowTableNodeModel$1.call(CallWorkflowTableNodeModel.java:99)
	at org.knime.productivity.callworkflow.table.CallWorkflowTableNodeModel$1.call(CallWorkflowTableNodeModel.java:1)
	at org.knime.core.util.ThreadPool.runInvisible(ThreadPool.java:615)
	at org.knime.productivity.callworkflow.table.CallWorkflowTableNodeModel.execute(CallWorkflowTableNodeModel.java:96)
	at org.knime.core.node.NodeModel.executeModel(NodeModel.java:567)
	at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1186)
	at org.knime.core.node.Node.execute(Node.java:973)
	at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:559)
	at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
	at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
	at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
	at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
	at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
	at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
2019-02-27 13:41:29,001 : DEBUG : KNIME-Worker-5 : WorkflowManager : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 doBeforePostExecution
2019-02-27 13:41:29,001 : DEBUG : KNIME-Worker-5 : NodeContainer : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 has new state: POSTEXECUTE
2019-02-27 13:41:29,001 : DEBUG : KNIME-Worker-5 : WorkflowManager : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 doAfterExecute - failure
2019-02-27 13:41:29,001 : DEBUG : KNIME-Worker-5 : Node : Call Workflow (Table Based) : 0:379 : reset
2019-02-27 13:41:29,001 : DEBUG : KNIME-Worker-5 : Node : Call Workflow (Table Based) : 0:379 : clean output ports.
2019-02-27 13:41:29,001 : DEBUG : KNIME-Worker-5 : WorkflowDataRepository : Call Workflow (Table Based) : 0:379 : Removing handler df94c2d1-4d50-437f-9c0f-6ee49000dd14 (Call Workflow (Table Based) 0:379: <no directory>) - 0 remaining
2019-02-27 13:41:29,001 : DEBUG : KNIME-Worker-5 : NodeContainer : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 has new state: IDLE
2019-02-27 13:41:29,002 : DEBUG : KNIME-Worker-5 : Node : Call Workflow (Table Based) : 0:379 : Configure succeeded. (Call Workflow (Table Based))
2019-02-27 13:41:29,002 : DEBUG : KNIME-Worker-5 : NodeContainer : Call Workflow (Table Based) : 0:379 : Call Workflow (Table Based) 0:379 has new state: CONFIGURED
2019-02-27 13:41:29,002 : DEBUG : KNIME-Worker-5 : NodeContainer : Call Workflow (Table Based) : 0:379 : dropzoneImagesNike 0 has new state: CONFIGURED
2019-02-27 13:41:29,002 : DEBUG : KNIME-WFM-Parent-Notifier : NodeContainer :  :  : ROOT  has new state: IDLE

The other workflow that’s being called is just a simple as the one displayed here. Some tables from a mariadb, some joining nothing and exactly one output node. Also it executes perfectly fine on its own and more importantly: it works when called from somewhere else inside the workspace.

I got permission from my superior to post all details about these workflows here, since we really like to solve this with KNIME and using multiple local table-based workflows.

Hi there!

I have created an example for you. While doing so I got the same error as you but it was something while configuring nodes so I just ran calling workflow and it disappeared.

Check out example, play with it a bit, inspect configuration and try to fix your workflow then. If you still encounter problems maybe you can share workflow so I can take a look. Instead of database connections and data from it you can put some dummy data.

Call_Workflow.knar (36.7 KB)

Br,
Ivan

Thanks for your example. Apart from the fact that your example uses an input-table and ours doesn’t it looks exactly the same. I have even saved/moved/recreated many of the nodes in question in order to make sure that there are no lingering cache/config/labeling problems.

Nothing so far seems to help. The error is exactly the same as always and even recreating the workflow from scratch didn’t help.

I rebuilt the node from scratch. Now it works. No idea what the problem was or how to reproduce it. If it happens again, I’ll file a bug report. Until then,
arp

2 Likes

Great!
Glad it works now :slight_smile:
Br,
Ivan

1 Like

Hi arp,

I’m happy it worked out in the end. I’m not sure what the error might have been and the log doesn’t help me much either. If you figure out how to reproduce it please let us know so we can fix it and implement a proper error message.

All the best,
Tobias U.

Just wanted to let you know that this problem just occured for a colleague of mine who was working under very similar circumstances.

He tried to use Call local workflow (table-based) on a perfectly fine workflow but KNIME will not allow him to call it. Same error as above.

We tried everything and are running out of options. We are unable to recreate the problem on purpose or fix it. At the moment we are just trying to make it go away so that we can continue working on this project. As a last ressort we will recreate the workflow from scratch and see if that helps like it has for me before.

Hi there!

Hm… :thinking: Seems there is something. Is there a possibility for you to share this workflows? You can use some dummy data instead of data from databases. That would be really helpful for troubleshooting as we are unable to reproduce it :slightly_frowning_face:

Br,
Ivan

Maybe you would want to check out the structure I have proposed before using a table of the workflows to be called, then a loop to call them all and JSON information to ‘communicate’ between the main and subworkflows (even if there is nothing to communicate)

the sub workflow needs a start and finish with (in this case) JSON informations coming in and leaving the sub-workflow (I think it could be empty but it could be use it to track the time it took the sub-wf to finish so I can see if it worked at all).

Then it is very important that the sub-workflow are saved in a reset state otherwise they will just be run without changing anything. Mandatory reset I think is only possible in batch mode and with the KNIME server.

Instead of the Input and Output JSON Containers also Table input and output should be possible. Although I have not much experience with that. Instead of using them you could just save the data you want to transfer in a temporary table and reload it in the next step.

Quick follow-up:

found a somewhat reliable way to reproduce the “Execute failed: Failure, workflow was not executed, current state is IDLE.” problem.

It has to do with nesting Call Local Workflow nodes. After a certain depth, the knime client just gives up or has problems with ordering of tasks or waiting for them? We are not sure what goes wrong there but it all falls apart if you have workflows which call other workflows two tiers in.

If we nest Call Local Workflow nodes depth 2+ you can no longer run the highest tier workflow!

Our intermediate workaround for gui development is to reset all nodes and then run them from the bottom upwards, one by one, and manually.
Example: if A calls B and B calls C and C calls D, running A results in the above error code and everything goes to helll. If you reset all 4 workflows and run them manually in reverse order (D, C, B) then A runs perfectly. This is tedious and takes ages. Any hope for a fix here?

I hope this info can give you some hints on where to find the problem.

Hi @arp!

Tnx for info. I will check this one and get back to you.

Br,
Ivan

HI there @arp!

I have just created a simple workflow up to 4 called workflows (A calls B, B calls C, C calls D and D calls E) and it works just fine. So unfortunately I can not reproduce it. If you can share workflow with error that would be a huge help in investigating this further on and subsequently fixing it. Just use some dummy data if possible.

Br,
Ivan

I’ll try to compile you a dummy workspace but no promises as to when I get this done sorry.

Hi there!

Sure. Tnx!

Br,
Ivan

Just a follow up which might be interesting info for you:

We have isolated the cause for the problem and it seems that it is some sort of internal ordering problem for KNIME. If call local workflows (table based) happen in a diamon shape then the execution fizzles and the error occurs.
We developed our own method to execute the workflows in a hierarchical order to circumvent this problem. Maybe you can take a look at your inner workings and see if you can resolve the issue.

Hope that piece of info helps. Cheers,

Hi @arp

I’m no sure I understand what you mean by diamond shape. Is it the call order, i.e workflow A calls workflows B and C and then workflow B and C calls a workflow D? Could you share an example workflow or screenshot illustrating the problem?

thanks,
Tobias U.

1 Like

i.e workflow A calls workflows B and C and then workflow B and C calls a workflow D?

Exactly that.