Python in KNIME doesn't recognise process finished

I am using KNIME 4.3.3 Python Script node on Windows 10.

As part of a larger function, I need python to call a Windows program, that program does something independently, then python does some more things afterwards. I am doing this process iteratively over a data frame. I need python to wait for the Windows process to finish before it continues.

In a jupyter notebook this can be achieved using one of the functions below:

os.system(run_command)
os.popen(run_command).read()
subprocess.popen(run_command).wait()

The process works well and will iterate over the data frame.

When I attempt to run this function through knime (either directly in a python script node or via a jupyter notebook in a python script node) the process successfully calls the first windows command, the command runs and completes but then hangs and does not continue with the python code. I presume this is because the python script thinks the process is still running.

I can’t seem to find a way to tell it that the process has completed, nor why it has different behaviour to when it is run in the same Anaconda environment from a notebook.

Does anyone have any suggestions?

1 Like

Hi @tom123,

When you say

do you mean that the Windows command’s process in fact ended properly (e.g. is not visible as active process in the Task Manager anymore) but Python somehow did not notice that? Or that it is actually still alive and hanging (visible in the Task Manager)?

Does iteratively running some simpler command work? E.g. does the following snippet

import subprocess
from pandas import DataFrame

for i in range(10):
	os.system(f"python -c \"print('os.system: Hello world #{i}!')\"")
	print(os.popen(f"python -c \"print('os.popen: Hello world #{i}!')\"").read())
	subprocess.Popen(["python", "-c", f"print('subprocess.Popen: Hello world #{i}!')"]).wait()

output_table = DataFrame()

produce output that looks like this:

os.system: Hello world #0!
os.popen: Hello world #0!
subprocess.Popen: Hello world #0!
os.system: Hello world #1!
os.popen: Hello world #1!
subprocess.Popen: Hello world #1!
os.system: Hello world #2!
os.popen: Hello world #2!
...

?

Also, could you provide more information on the Windows program that you are trying to run? Is that program known to be in some way sensitive to the circumstances under which its process was created and run (environment variables, etc.)?

Marcel

1 Like

Thanks @MarcelW , your reply made me think about how python is calling the Windows command.

In the Jupyter notebook,

os.system(cmd /k 'command')

works well. i.e. Python runs the command, waits for it to finish and carry on. Even though this would equate to start cmd and remain. I assume this was a typo when I was writing the function and happened to work in Jupyter… The windows program must send a terminate signal that is recognised by python, I don’t know why this behaviour changes when the cmd prompt is started from python in KNIME. Any thoughts (from an acedemic point of view)?

Changing the code to:

os.system(cmd /c 'command')

i.e. start and terminate cmd gives me the desired results in KNIME. So python runs the windows program, it terminates the windows command, then carries on with the rest of its code.

Thanks for your help!

3 Likes

Glad I could help!

But no idea why the behavior of os.system("cmd ...") differs between Jupyter and KNIME when using the /k option. However, I quickly tested both versions, i.e.:

for i in range(10):
    os.system("cmd /c 'command'")

and

for i in range(10):
    os.system("cmd /k 'command'")

using Python’s interactive mode directly from the Windows Command Prompt, and found that this replicates the behavior you observed when using KNIME (i.e. using /c works as intended, using /k only executes the first call). So there seems to be something special about how Jupyter executes this code…

Marcel

1 Like