Problem on External tool node !!!

KNIME Analytics Platform
1

I tried to use External tool node to exchange information between Windows and the Linux. I created a Windows bat file which contains command for copying files (pscp) and run remote command (plink). For the bat file I tested, it functioned as expected when running on DOS. But it does not when running through the External tool node. What I see is that the command in the second or the third line of the bat file continue showing its process in Windows. For example, pscp process will hang there forever and as a result, External tool node also keeps running. When I kill the Windows process, the node also finishes with errors.

Any idea what is going on?

Thanks!

2

Hi qliu,

your problems are exactly the reason why we didn't include this external tool node in our release (up to now anyway). This node takes the control out of KNIME's hands. If something hangs in the external process it could be harmful to the entire KNIME application. Also, there are so many different requirements from external tools how to format/deliver/create/prepare the data that it is impossible to fullfill all these requests.

But having said that, let's try to find out why your process hangs.
Is the external script you call producing a lot of output to std out or std err?
The ExtToolNode is picking up the output, but I believe not in the best way, and huge output could get stuck in the buffer and therefore block the external process. If this is the case, you could try working around it by redirecting the output into a file.

Regards,
- Peter.

3

The external tool is a Windows batch file with multiple commands in it. The first command is to copy a local Windows file to the remote Linux using pscp (a tool in Putty package). This part is successful. Then the second command (also pscp) which should copy a Linux file back to the Windows does not finish somehow. The file it is trying to copy is a small one (about 50k). Only part of the contents were copied over, and there is no reason it couldn't finish copying the whole file by simply checking the contents of the file. As your information, the two commands are like,

pscp.exe -pw xxx c\temp\file1.hits user@Linux/home/remote_folder
pscp.exe -pw xxx user@Linux/home/remote_folder/file2.hits c\temp

Thanks!

4

Hi qlin,

to be honest I'm running out of ideas here.
It sounds really like a pscp problem, because you say, only a part of the file is being copied. But then, if the job completes without problem outside KNIME, it can't be something in the file that causes pscp to hang...

I have tried it here, with the same commands in a .bat file - and it worked fine. It's really mystical.
And your script does run through - without any interaction (no password prompt, no connection confirmation) - when you start it from the DOS prompt, right?

Do you have a log file by any chance I could look at? Maybe that gives me some clue...
- Peter.

5

For those who have interest on this topics, I have found a temporary solution by adding "1>&2" at the end of each line in the batch file. I believe it worked by suppressing the output from the command, which might have confused the process control by the node.

Thanks to Peter for his wonderful support on this issue. I am looking forward to the next bug fixed ExtToolNode.

6

We've just released the external tool node in an add-on plug-in (that includes the fix for the mentioned output/blockage problem):
http://knime.org/download_extensions.html#extTool

And, while we are at it, we are throwing in an XLS Writer plug-in too:
http://knime.org/download_extensions.html#xlsWriter

Have fun with it!
- Peter.

7