Bash Node: Command not found

[mwiegand]

Hi there,

the Bash – KNIME Hub form the KNIME NGS tools – KNIME Hub works well on commands like curl.

However, other commands, which can be executed w/o issues directly in the terminal, like the following, cause an error to be thrown: “Execute failed: Cannot execute command”

nmap -v -F google.com

Does Knime has its own file which references executables? I checked the preferences but could not find anything, though.

Best
Mike

[mwiegand]

PS: I suspect the path variables being at “fault” due to

Exit code 127 error status in Bash script

Which I once encountered in

Execute bash command fails – what's my mistake?

Hi all, I am trying to execute a bash / terminal command via Knime nodes via one of the following nodes but getting an error which likely is caused by a mistake of mine. Hope anyone has an idea … The command is: awk '(NR == 1) || \(FNR > 1\)' table*.csv > bigfile.csv The following two nodes fail with: ERROR CmdwInput 0:45 Execute failed: STDERR message: awk: syntax error at source line 1 context is >>> ' <<< missing ) awk: bailing out at source line 1 External Toll…

But also others

External Tool (Labs): command not found. External commands terminated with exit code: 127

Hi everybody, I’m trying to use the External Tool (Labs) node but I have problems because it seems it doesn’t know where executable files are. In practice I’m launching the node with the command: bash -c “my_script.sh %inFile% %outFile%” Upon execution the command is translated to something like: bash -c my_script.sh /tmp/bash_20190326_38800/input/port0.csv /tmp/bash_20190326_38800/output/port0.csv my_script.sh is a bash script inside one of my bash paths and with executable permissions. I…

Cheers
Mike

[mwiegand]

Update: It is indeed as I thought that the PATH environment variable does not match.

Knime: PATH=/usr/bin:/bin:/usr/sbin:/sbin
OS: PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/Apple/usr/bin:/Applications/Wireshark.app/Contents/MacOS

Anyone knows how to adjust this as I could not find anything in the Knime preferences?

Many thanks in advance
Mike

[mwiegand]

Hi @baj,

I possibly posted this in the wrong forum section as I thought this is more a general question but it seems I was wrong. In search for a solution to instruct Knime respectively your bash Node to use the actual PATH pariable, I wonder if you could shed some light onto this?

Googling around to adjust the Eclipse Path variable I could not come to a solution (please see my prev. comment). Do you happen to have an idea?

Many thanks in advance
Mike

[baj]

Hi @mwiegand,

with respect to the path issue, can you print the environment using “set” or “env”?

One quick way to solve it, is to use the full path…

Other things to consider are where you set your path in the login script or .profile… I am not sure which one JAVA is using to start the command.

It has been a few moons since I programmed this and haven’t had time to return to KNIME… What a shame!!!

Glad to see that people are still using the nodes.

Hope this helps.

Best,

Bernd

[baj]

Did you figure out the awk issue?

to me it sounds like you need either other quotes, or need to escape them with with 1-3 ""…

B

[mwiegand]

Hi Baj,

I haven’t continue working on the awk issue but with the recently gained knowledge I might do that. Though, that project I worked on back then was closed (I switched employers). For the sake of interest – and to satisfy my German urge to do things right – I might come back to this in the future.

About the environment I printed the variables in a previous comment. When chekcing the env vars in terminal they differ from those available in Knime. Therefore, i suspect that Knime is using another configuration file.

However, I could not find anything. Even by declaring the PATH variable in the knime.ini it did not changed anything. The HOME and SHEEL Variables are identical, though. Therefore, I thought it’s using the same bash which I confirmed via

which bash true false

Setting the absolute directory for the command not found by first identifying the executable “/usr/local/bin” via (in terminal):

which nmap true false

Did not resolve the command not found issue.

Creating a .bash_profile and declaring the PATH variable, applying via source .bash_profile, didn’t resolve it either. I assume Eclipse / Knime is spun up in a separate user / config or so not utilizing the system nor my actual user configuration (to the full extend).

The answer from Stackoverflow, declaring a key in the Info.plist file of Knime did not work too.

I also declared the PATH variable in /etc/profile to make it accessible for all users to no avail. Knime was restarted each single time.

Best
Mike

[mwiegand]

PS: I believe it is less related to your bash node but more to Knime in general. using a Java Snippet to retrieve the PATh variable I too get the same result.

out_path = System.getenv("PATH");

@ipazin you are always a great and knowledgeable source. Do you maybe have an idea?

Cheers
Mike

[baj]

Where do you set your PATH variable in the shell?
(.bashrc, .profile)?
Bernd

[mwiegand]

Hi Bernd,

I tried both as per this explanation but neither did work: About bash_profile and bashrc on macOS – Scripting OS X

Is this actually reproducible on other OS or unique to OSX? Shall I start a VM for further testing?

Cheers
Mike

[baj]

did you check that you are actually using bash? the newer OS is recommending something else.

just saw it… you are using zsh… I don’t use that one, so not entirely sure how to things there…

B

[mwiegand]

I suppose zsh is used per the SHELL variable shown in Knime

Even the TMPDIR variable matches which I’d suspect getting randomly regenerated with each session. There is a higher level of alignment but for some reason if ignores or overwrites the PATH variable.

[baj]

Zsh Profile File (.profile, .zprofile, .zshrc) – LinuxTect.

[mwiegand]

I changed from zsh to bash but Knime did not pick that up. The shell profiles, whichever I configured, are not picked up either.

Despite being odd it further indicates that Knime is “minding it’s own business” not fully caring about OS preferences. WDYT?

[baj]

did you log-out after changing and potentially reboot…?

[rhm21k]

To confirm mwiegand’s assessment of the problem:

• it appears that the $PATH declaration made by a user in either .bash_profile or .zshrc (the profile for zsh) has no effect on the PATH established by the Bash node.
• for this reason, any executable residing outside of that path variable definition is not found.
• for some reason, variables (which use the “$” character) are not recognized, so one is prevented from executing either a temporary addition to the path or a ‘permanent’ change to the profile

At this point, I’ve all but given up. My last effort will be to determine whether the conda environment that prevails when I try to use the Bash node might have some means of affecting the ability to temporarily (for the duration of the Bash node session) modify the $PATH variable.

This same issue appears in my attempts with Cmdwinput node and External Tool node. All of them put you in $PATH jail.
= RM

[mwiegand]

Just for completeness, I rebooted but it did not have any effect. I tried setting the PATH variable in Knime but it failed with Execute failed: Cannot execute command as well.

I think the only two options left at present are to:

1. Create a shell script, which could get done via Knime, and make it executable OR
2. Using Java to make bash calls like in Shell Command in knime

While attempting the two options I encountered an issue with approach #1. I can’t make the shell script executable. Setting POSIX permission fails.

With approach #2 I noticed a possible cause of the command not found. Using the Bash Node to get all env variables it gave back a PATH variable. However, using the Java Snipped the PATH variable was not found. Could it be that the PATH variable is not properly setup / defined and Knime respectively Eclipse falls back to a default?

I put all bits and pieces in this workflow together:

NOTE: There is a mistake with the Java Snippet which I noticed only after upload. Will update the workflow once resolved.

Bash Command not Found – mwiegand

There are a few additional interesting findings as well. Using the whereis YOUR-CMD command to locate the executable the commands which do NOT work still could be found. The where YOUR-CMD (without “is”) command also works in the terminal but not in Knime.

where and whereis CMD in Terminal

where and whereis CMD in Knime

Since not everyone uses OS X, even though it’s linux, in case anyone wants to debug something, we could arrange a remote debugging session.

Best
Mike

[ipazin]

Hello @mwiegand,

missed this one somehow. Unfortunately I can’t help here…

Br,
Ivan

[mwiegand]

Thanks anyways @ipazin!

The other thread showcasing an approach to execute shell commands via a Java Snippet Node is unfortunately dead I noticed.

Does maybe anyone here notice the flaw in the code I attempted to refurbish but which does not give any feedback but just an empty response?

The example is also in the workflow I shared in my previous comment.

Best
Mike

[mwiegand]

Just for the record. Thanks to @carstenhaubold there is a Python based approach of executing bash commands.

External Tool "issues" and high difficulty to use

Totally agree, we should make the External Tool node more usable. Still, to give you a possible workaround right now, I just rebuilt the NGS Bash node’s functionality using the Python Script node in KNIME 4.7, and using the bundled Python environment (which it’ll use by default if you freshly install the Python Integration in KNIME). So you don’t even need to deal with setting up a Python installation. Place a Python Script node in your workflow, click the three dots to remove the input table …

Unfortunately, though, it has the same issue that some commands are not found. That pretty much makes it abundantly clear that the PATH variable difference between Knime / Eclipse and vs. user under which it runs is at fault.

Bug Report

Knime PATH does not match to system user PATH Variable

Hi, for quite some time, interrupted by the sudden decommissioning of NGS (in particular the Bash Node), I attempted to resolve an issue in details explored in: https://forum.knime.com/t/bash-node-command-not-found/46256 In a nutshell Executing external commands only partially works (i.e. ping, nc) whilst other commands are not found Using Python or Java to execute shell commands resulted in the same issue Checking the Path variable indicates that Knime / Eclipse is not using the users Path…