Exported workflow missing vital files in batch mode

I export my workflow group and then try to test it by running some of the workflows in batch mode. However, I’m unable to run the exported workflows due to errors related to missing files, for example:

Status: DataLoadError: Unable to load port content for node "JSON Reader": No such data file: [...]/JSON Path (#44)/port_1/data.xml
Status: DataLoadError: Unable to load port content for node "String Manipulation (Variable)": Can't read spec file: [...]/String Manipulation _Variable_ (#37)/port_1/spec/spec.zip

Comparing to the workspace, it seems the exported KNAR file is missing many files. Running directly in the workspace and in the GUI everything works fine. I also imported the exported KNAR file to another workspace and the GUI is able to run the workflow even though I don’t see those files that appear in the another workspace. Any idea why batch mode fails?

Hi!

you are calling .knar file in batch? Not sure it works like that. How your batch call looks like?

Br,
Ivan

I unzip the KNAR file and use -workflowDir option.

I think I found what this is related to. I use my custom configuration file so I pass it to KNIME with -preferences option. If I run it without, the workflows runs without a problem. Now the next question is why I can’t use my preference file.

1 Like

Oh. I see. So you point it to .epf file which you export from Knime?
If you are not using custom configuration file which one is used?
Ivan

If I don’t pass an .epf file, I think KNIME uses the config that GUI uses. I, however, assume that if I export the config from the GUI, then it should be the same as in the GUI, but the batch run still fails. It seems to fail even when I pass empty config file.

Interesting :slight_smile:
You are transferring your workflow to a different machine/location? Are workflows saved?
Ivan

Yes, I want to transfer my workflows and run in batch mode. I have saved my workflows in the workspace.

I have ran into issues between GUI and batch mode before as well and it’s quite worrisome that batch mode cannot always replicate results in GUI.

Windows to Linux transfer?
But without -preferences option it runs successfully, right?
The above error actually indicates workflow can not be open I think…
Ivan

I’m running both GUI and batch in the same Linux. Knime can open the workflow but it fails when files are not found.

Now I noticed the problem doesn’t seem to depend on the -preferences option. I removed one useless file for test and exported the work again, but this time the batch run failed without -preferences option. I ran again with -preferences and suddenly it worked. I repeated a few times by changing some minor thing, then exported and tried to run in batch mode. In some cases the batch run fails with some combinations of the parameters or all the time. I cannot pinpoint when this happens exactly. Seems to be quite random.

Here is the call:

$KNIME_HOME/knime -nosplash -nosave -reset -failonloaderror -consoleLog --launcher.suppressErrors -application org.knime.product.KNIME_BATCH_APPLICATION -workflowDir="[...]"

And the relevant log entries:

ERROR	 main BatchExecutor	 Status: DataLoadError: [...] 0 loaded with error during data load
  Status: DataLoadError: [...] 0
    Status: DataLoadError: Read configuration 0:1
      Status: DataLoadError: JSON Reader 0:1:43
        Status: DataLoadError: Unable to load port content for node "JSON Reader": No such data file: [...]/Read configu (#1)/JSON Reader (#43)/port_1/data.xml
        Status: DataLoadError: State has changed from EXECUTED to CONFIGURED
      Status: DataLoadError: JSON Path 0:1:44
        Status: DataLoadError: Unable to load port content for node "JSON Path": No such data file: [...]/Read configu (#1)/JSON Path (#44)/port_1/data.xml
      Status: DataLoadError: Table Row to Variable 0:1:45
        Status: DataLoadError: Unable to load port content for node "Table Row to Variable": Can't read spec file [...]/Read configu (#1)/Table Row to Variable (#45)/port_1/spec/spec.zip
      Status: DataLoadError: String Manipulation (Variable) 0:1:37
        Status: DataLoadError: Unable to load port content for node "String Manipulation (Variable)": Can't read spec file [...]/Read configu (#1)/String Manipulation _Variable_ (#37)/port_1/spec/spec.zip
      Status: DataLoadError: State has changed from EXECUTED to CONFIGURED
ERROR	 main BatchExecutor	 IO error while loading the workflow: Error(s) during workflow loading. Check log file for details.
INFO 	 main BatchExecutor	 ========= Workflow did not execute sucessfully ============

It keeps getting more mysterious. I tried to run my workflow with the command below, but it failed with the above error:

$KNIME_HOME/knime -nosplash -nosave -reset -failonloaderror -consoleLog --launcher.suppressErrors -application org.knime.product.KNIME_BATCH_APPLICATION -workflowDir="workgroup/workflow/"

Now, I ran it again only by removing the last slash in the workflowDir param:

$KNIME_HOME/knime -nosplash -nosave -reset -failonloaderror -consoleLog --launcher.suppressErrors -application org.knime.product.KNIME_BATCH_APPLICATION -workflowDir="workgroup/workflow"

This time it run succesfully. Strange, isn’t it?

I tried with Knime 3.5.3 and 3.6.0 and the problem appears in both.

Hi!

:smiley:
Does it always runs successfully without Last slash in workflowDir param?

Br,
Ivan

No, it depends. I also tried to run in different environment and while it might run in one, it might fail in another.

Not sure I can help here more. Seems you encountered multiple problems.
Try going back from beginning to resolve your original problem.
Good luck!
Ivan

Thank you, Ivan. Seems like I’d need help from Knime.

Dear @jvj0,

Have you tried executing the workflow with -workflowFile=SomeFile.knwf instead of extracting and using -workflowDir=...?

Also, could you please provide a minimal workflow that reproduces the issue?

Best,
Stefan