Parallel Chunk Nodes: Could not connect to the Python process

[mwiegand]

Hi,

I have setup and use my conda environment frequently. Since I started to use Python in combination with the Parallel Chunk nodes I constantly face the issue that some chunks fail to connect to Python.

2024-10-01 13:02:07,292	ERROR Python Script        3:852      Execute failed: Could not connect to the Python process.
2024-10-01 13:02:08,199	WARN  Python Script        3:867:857  Python standard output: 
2024-10-01 13:02:08,199	WARN  Python Script        3:867:857  Python standard error: Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.

2024-10-01 13:02:08,200	ERROR Python Script        3:867:857  Execute failed: Could not connect to the Python process.
2024-10-01 13:02:08,777	WARN  Python Script        3:867:852  Python standard output: 
2024-10-01 13:02:08,777	WARN  Python Script        3:867:852  Python standard error: Python was not found; run without arguments to install from the Microsoft Store, or disable this shortcut from Settings > Manage App Execution Aliases.

2024-10-01 13:02:08,778	ERROR Python Script        3:867:852  Execute failed: Could not connect to the Python process.

I configured the knime.ini to adjust the python time out a per @mlauber71 suggestion but to no avail.

Python nodes failing on parallel loop

You could try to increase the timeout setting in the KNIME.ini. This is mentioned in the log file. -Dknime.python.connecttimeout=30000 2019-10-08 13:10:51,154 : ERROR : KNIME-Worker-1060 : Node : Python Source : 0:284:380:334:394:310 : Execute failed: The connection attempt timed out. Please consider increasing the socket timeout using the VM option ‘-Dknime.python.connecttimeout=’. Also make sure that the communication between KNIME and Python is not blocked by a firewall and that your ho…

While further debugging the issue I noticed that there many Python processes still running in the background despite all nodes being reset. IT seems after starting knime, several Python processes get launched, stopped and started again automatically.

After node reset

After restart w/o execution

Executing w/o the parallel nodes does work but leave a lot of resources unleveraged on the table.

Best
Mike

[k10shetty1]

Hello @mwiegand,

Thanks for the detailed explanation.

I tried replicating the issue on my end by using multiple loops with a custom Conda environment, but I was not able to reproduce the problem. Everything worked fine for me with parallel execution.

Since the issue could be environment specific, could you please share a sample workflow along with some dummy data so I can try to replicate it ?

Can you also share your Python version, Conda setup, and whether there are any specific packages that might cause this issue?

Best,
Keerthan