Execute bash command fails – what's my mistake?

Hi all,

I am trying to execute a bash / terminal command via Knime nodes via one of the following nodes but getting an error which likely is caused by a mistake of mine. Hope anyone has an idea …

The command is:

awk '(NR == 1) || \(FNR > 1\)' table*.csv > bigfile.csv

The following two nodes fail with:

ERROR CmdwInput            0:45       Execute failed: STDERR message:
awk: syntax error at source line 1
 context is
 >>> ' <<< 
 missing )
awk: bailing out at source line 1

External Toll fails with

ERROR External Tool (Labs) 0:58       Execute failed: Failed with exit code 2

Thanks in advance
Mike

1 Like

Hi there @mw,

have you figured it out? Does this command works from outside KNIME?

Br,
Ivan

Good afternoon Ivan,

not yet, have tried several variations except a bash script but that would increase complexity a lot. Any idea how to debug. All googling I did, did not provide any more approaches nor insights.

Best
Mike

Hi @mw,

I would start by making sure command works outside of KNIME first and then trying to call it from within KNIME.

Br,
Ivan

Hi @ipazin,

that’s the funny thing. I copy the command I generate in Knime and can successfully execute it in the terminal. Which leads me to assume the command execution or interpretation by Knime is failing. But the error output is not very specific which I was reaching out to the forum is anyone has an idea.

Best
Mike

Hi @mw,

I see. Wasn’t sure of that. Well I would try with different quotes as one error mentions them but that is all I can think of.

Hope someone will have more (and better) ideas how to apply command in KNIME.

Br,
Ivan

Yeah, have seen and tired that too. Guess I can only work that out myself. Wish me luck. Will update this post upon success.

1 Like

Could you elaborate more on what you want to do and what system you are on.

Is it possible to have a minimal demonstration that one could reproduce?

I once had a problem with a python wrapper. It was a progress bar with exotic characters that the KNIME system would not want to handle while it ran without problems in the world outside KNIME. Took me some time to figure that out.

1 Like

Hi @mlauber71,

thanks for catching it up. When using parallel start / end nodes and writing the result of each branch into a single file, chances are that branches while saving run into conflict. This breaks the file structure.

The idea is to create separate file i.e. by amending the batch / chunk / iteration of each branch. As a post process those files shall be concatenated into one. Using the (CSV) file readers takes an abnormally large amount of time compared to using awk which takes just seconds.

awk '(NR == 1) || \(FNR > 1\)' table*.csv > bigfile.csv

I created a test workflow to showcase the issue about parallel writing to eliminate possible data issues.

Best
Mike