Execute bash command fails – what's my mistake?

[mwiegand]

Hi all,

I am trying to execute a bash / terminal command via Knime nodes via one of the following nodes but getting an error which likely is caused by a mistake of mine. Hope anyone has an idea …

The command is:

awk '(NR == 1) || \(FNR > 1\)' table*.csv > bigfile.csv

The following two nodes fail with:

ERROR CmdwInput            0:45       Execute failed: STDERR message:
awk: syntax error at source line 1
 context is
 >>> ' <<< 
 missing )
awk: bailing out at source line 1

CmdwInput – KNIME Hub

Bash

External Toll fails with

ERROR External Tool (Labs) 0:58       Execute failed: Failed with exit code 2

External Tool (Labs)

Thanks in advance
Mike

[ipazin]

Hi there @mw,

have you figured it out? Does this command works from outside KNIME?

Br,
Ivan

[mwiegand]

Good afternoon Ivan,

not yet, have tried several variations except a bash script but that would increase complexity a lot. Any idea how to debug. All googling I did, did not provide any more approaches nor insights.

Best
Mike

[ipazin]

Hi @mw,

I would start by making sure command works outside of KNIME first and then trying to call it from within KNIME.

Br,
Ivan

[mwiegand]

Hi @ipazin,

that’s the funny thing. I copy the command I generate in Knime and can successfully execute it in the terminal. Which leads me to assume the command execution or interpretation by Knime is failing. But the error output is not very specific which I was reaching out to the forum is anyone has an idea.

Best
Mike

[ipazin]

Hi @mw,

I see. Wasn’t sure of that. Well I would try with different quotes as one error mentions them but that is all I can think of.

Hope someone will have more (and better) ideas how to apply command in KNIME.

Br,
Ivan

[mwiegand]

Yeah, have seen and tired that too. Guess I can only work that out myself. Wish me luck. Will update this post upon success.

[mlauber71]

Could you elaborate more on what you want to do and what system you are on.

Is it possible to have a minimal demonstration that one could reproduce?

I once had a problem with a python wrapper. It was a progress bar with exotic characters that the KNIME system would not want to handle while it ran without problems in the world outside KNIME. Took me some time to figure that out.

Python Script and H2O Data Frames - Error under Windows

@MarcelW thank you for your effort and the suggestions. I think they put me on the right track with actually studying the error message and thinking about waht is going on. It was the progress bar in H2O that would use characters that the KNIME/Windows console would not like and would refuse to fix using you suggestions. In the end I just introduced: h2o.no_progress() So no more progress bars inside the wrapper (which would not be of any real help) and now it does work. Still might be a poi…

[mwiegand]

Hi @mlauber71,

thanks for catching it up. When using parallel start / end nodes and writing the result of each branch into a single file, chances are that branches while saving run into conflict. This breaks the file structure.

The idea is to create separate file i.e. by amending the batch / chunk / iteration of each branch. As a post process those files shall be concatenated into one. Using the (CSV) file readers takes an abnormally large amount of time compared to using awk which takes just seconds.

awk '(NR == 1) || \(FNR > 1\)' table*.csv > bigfile.csv

I created a test workflow to showcase the issue about parallel writing to eliminate possible data issues.

Best
Mike

[mwiegand]

Hi there,

I’ve four shell / terminal / bash examples which, if manually executed via command line, even by running via bash script, work like a charm. However in Knime I get various errors like:

ERROR CmdwInput            0:1278     Execute failed: STDERR message:
cat: *.csv: No such file or directory
cat: >>: No such file or directory
cat: finalout.csv: No such file or directory

ERROR External Tool        0:10       Execution failed (with exception): Cannot run program "/Users/mw/Documents/Knime-Workspace/Knime" (in directory "/Users/mw/Documents/Knime-Workspace/Knime Forum Support/Concat files/data/_sample-data"): error=13, Permission denied
ERROR External Tool        0:10       Execute failed: Cannot run program "/Users/mw/Documents/Knime-Workspace/Knime" (in directory "/Users/mw/Documents/Knime-Workspace/Knime Forum Support/Concat files/data/_sample-data"): error=13, Permission denied
WARN  External Tool        0:10       Existing input file will be overridden!

ERROR CmdwInput            0:1278     Execute failed: STDERR message:
find: >>: unknown primary or operator

The workflow with all examples I missed uploading the last time. Sorry for that …

Execute bash shell terminal fails – mw

Hope anyone has an idea.

System: macOS Catalina 10.15.7
Knime: Latest version
Installed extensions

KNIME Amazon Cloud Connectors	4.2.0.v202006171540	org.knime.features.cloud.aws.feature.group	KNIME AG, Zurich, Switzerland
KNIME Analytics Platform	4.2.3.v202011031551	org.knime.desktop.product	null
KNIME BigQuery	4.2.0.v202007101231	org.knime.features.database.extensions.bigquery.feature.group	KNIME AG, Zurich, Switzerland
KNIME Connectors for Common Databases	4.2.0.v202006221532	org.knime.features.dbdrivers.feature.group	KNIME AG, Zurich, Switzerland
KNIME Database	4.2.1.v202008251536	org.knime.features.database.feature.group	KNIME AG, Zurich, Switzerland
KNIME Ensemble Learning Wrappers	4.2.0.v202004061016	org.knime.features.ensembles.feature.group	KNIME AG, Zurich, Switzerland
KNIME Excel Support	4.2.1.v202008251158	org.knime.features.ext.poi.feature.group	KNIME AG, Zurich, Switzerland
KNIME Expressions	4.2.2.v202009241054	org.knime.features.expressions.feature.group	KNIME AG, Zurich, Switzerland
KNIME Extension providing the Chromium Browser	83.0.4103.116	org.knime.features.ext.chromium.feature.group	KNIME AG, Zurich, Switzerland
KNIME External Tool Support	4.2.0.v202005261930	org.knime.features.ext.exttool.feature.group	KNIME AG, Zurich, Switzerland
KNIME External Tool Support (Labs)	4.2.0.v201912131002	org.knime.features.exttool.feature.group	KNIME AG, Zurich, Switzerland
KNIME File Handling Testing Framework	4.2.3.v202011021201	org.knime.features.filehandling.core.testing.feature.group	KNIME AG, Zurich, Switzerland
KNIME Google Cloud Storage Connection	4.2.2.v202009241054	org.knime.features.google.cloud.storage.feature.group	KNIME AG, Zurich, Switzerland
KNIME Google Connectors	4.2.2.v202009241054	org.knime.features.google.api.feature.group	KNIME AG, Zurich, Switzerland
KNIME Image Processing	1.8.3.202010230604	org.knime.knip.feature.feature.group	University of Konstanz / KNIME
KNIME JavaScript Views	4.2.3.v202011021201	org.knime.features.js.views.feature.group	KNIME AG, Zurich, Switzerland
KNIME Javasnippet	4.2.0.v202006150933	org.knime.features.javasnippet.feature.group	KNIME AG, Zurich, Switzerland
KNIME JSON-Processing	4.2.1.v202008251158	org.knime.features.json.feature.group	KNIME AG, Zurich, Switzerland
KNIME Math Expression (JEP)	4.2.2.v202009241054	org.knime.features.ext.jep.feature.group	KNIME AG, Zurich, Switzerland
KNIME NGS tools	0.2.200.qualifier	org.pasteur.pf2.ngs.feature.feature.group	Plate-forme 2 - Transcriptome et Epigenome, Institut Pasteur, Paris.
KNIME Personal Productivity Tools	4.2.0.v202001210919	org.knime.features.personalproductivity.feature.group	KNIME AG, Zurich, Switzerland
KNIME Plotly	4.2.0.v202006071334	org.knime.features.js.plotly.feature.group	KNIME AG, Zurich, Switzerland
KNIME Public Server Access	4.2.3.v202011021202	com.knime.features.enterprise.client.exampleserver.feature.group	KNIME AG, Zurich, Switzerland
KNIME Quick Forms	4.2.3.v202011021201	org.knime.features.js.quickforms.feature.group	KNIME AG, Zurich, Switzerland
KNIME REST Client Extension	4.2.0.v202006301416	org.knime.features.rest.feature.group	KNIME AG, Zurich, Switzerland
KNIME ServerSpace	4.11.3.v202011021202	com.knime.features.explorer.serverspace.feature.group	KNIME AG, Zurich, Switzerland
KNIME Statistics Nodes	4.2.0.v202008251159	org.knime.features.stats.feature.group	KNIME AG, Zurich, Switzerland
KNIME SVG Support	4.2.2.v202009241055	org.knime.features.ext.svg.feature.group	KNIME AG, Zurich, Switzerland
KNIME Testing Framework	4.2.0.v202006171549	org.knime.features.testing.core.feature.group	KNIME AG, Zurich, Switzerland
KNIME Testing Framework - JavaScript support	4.2.1.v202008251343	org.knime.features.testing.js.feature.group	null
KNIME Testing Framework UI	4.2.1.v202008251158	org.knime.features.testing.application.feature.group	KNIME AG, Zurich, Switzerland
KNIME Timeseries nodes	4.2.0.v202003271454	org.knime.features.timeseries.feature.group	KNIME AG, Zurich, Switzerland
KNIME Virtual Nodes	4.2.0.v202004271350	org.knime.features.virtual.feature.group	KNIME AG, Zurich, Switzerland
KNIME XML-Processing	4.2.1.v202008251159	org.knime.features.xml.feature.group	KNIME AG, Zurich, Switzerland
Lhasa public plugin	1.1.500.v202007031132	org.lhasalimited.knime.public.feature.feature.group	Lhasa Limited (http://www.lhasalimited.org)
Palladian for KNIME	2.3.0.202009251618	ws.palladian.nodes.feature.feature.group	palladian.ws; Philipp Katz, Klemens Muthmann, David Urbansky
Vernalis KNIME Nodes	1.27.2.v202010191232	com.vernalis.knime.feature.feature.group	Vernalis Research Ltd, Cambridge, UK

Cheers
Mike