I apologize for the lengthly post but have been working on it, reading through many Knime community posts, working through Hub examples, try & erroring over quite some time, in order to provide a comprehensive picture.
I am trying to replace the Bash Node from NGS, which got removed entirely with Knime 4.7 (Corresponding posts here and here) , by any of the external tools nodes like External Tool (Labs) – KNIME Community Hub
However, the input and output files as well as other issues do make even the simplest commands virtually impossible to execute.
Most simplest approach: Just External Tool + Manually created Shell Script
First off, when using “Generate unique file name on execute” the following error is thrown:
ERROR External Tool (Labs) 4:3 Execute failed: Can't access 'file:/var/folders/jg/zcb9yqn90w54f3szwrsnqrl80000gn/T/bash_20230104_23758/output/port0.csv'. (/var/folders/jg/zcb9yqn90w54f3szwrsnqrl80000gn/T/bash_20230104_23758/output/port0.csv (No such file or directory))
This applies for both External Tools, the regular and labs node respectively.
More complex approach: Dynamically create SH file and try to set permissions
The Set Files/Folders Permissions – KNIME Community Hub Node fails to set permissions.
ERROR Set Files/Folders Permissions 4:6 Configure failed (IllegalStateException): The file system Current workflow data area does not support setting POSIX permissions
Worth to note that Knime is executed with my OS user (not root, www-data eclipse or whatever) which apparently has the necessary permissions (confirmed recursively).
In- and Output
About the high difficulty to use the external tools. That relates the confusing aspect of the input and output files. Why bother ingesting a table and offering options to loop over the entire table or individual lines when in- and egress files must get served as well. Maybe something is not well explained about that causing the confusion?
The External Tools (Labs) node breaks with the following error is files are defined manually:
ERROR External Tool (Labs) 4:10 Execute failed: Index 0 out of bounds for length 0
File Path and Flow-Variables
The some time ago introduces Path variables are not supported and the flow variables to not allow to populate the locations to any file let it be Input, Output or the Executable. Actually (almost) nothing is manageable via Flow Variables.
Many community examples lack documentation and are not really working (anymore). The two created by Knime however do (partially) work but are quite complex.
Actually only the first and form that only the branch with the data generator.
Unaddressed Quotation Issue?
Please see External tool node - a problem with double quotes
Manually choosing files
Usually, when manually choosing a file, a file name could get provided which would result, upon execution, that the file would get created. However, as seen below, one must manually create in- and output files first.
In- and Output Files getting deleted
Whilst it’s good to prevent unexpected data garbage to accumulate, I noticed that the manually created and empty csv files got deleted without my notice or seeing any configuration in the nodes.
- The NGS Tools got decommissioned without any advanced warning time and “against” the usual process of marking them as deprecated. Mentioned in this and following posts it revealed an insufficient processes which a couple of users got frustrated about (latest here). Is there any process improvement considered for the future?
- How to work with the external tool node in general or details as per aforementioned issues?
- How to handle the POSIX Permission issue?
- Would it be possible to provide a really simple example workflow to showcase how to i.e. run a simple shell command like “printenv”?
Here is the workflow displaying the many issues mentioned above: