knime external python process crashed for unknown error

Hello to you all.
I am facing problems with Python nodes, which sometimes, not always, fails under the error “knime external python process crashed for unknown error”.
The thing is that the same process works fine most of the time, but sometimes arises this error and I cannot find a way to solve it. It is annoying because the python node is located inside a 50-iterations-loop and most of the times makes it fail.
Any clue on what’s going on?
The Knime versión is 4.1.3 and the Python environment is Python 3 (conda versión 4.7.12).

Thanks in advance.

Could you provide us with more details like what python packages were used and what kind of data was processed. And a log file of what happened might also help.

And which python nodes did you use? Maybe a screenshot of the workflow.

If possible a workflow reproducing the error with dummy data.

Hello, thanks for your response.

I am using Auto-Arima learner and Auto-Arima predictor, as they come from the knime node repository. The input data is a double-formated numeric series without nulls. Most of the time they work fine, but occasionally one of them does not. When they fail, if I restart the workflow with the same data they run fine…
I tested that behaviour many times: one of the AUTO-ARIMA components fails and, changing nothing, it runs fine when executing again the node.
Attached are some images:

  • Workflow with error message![Arima predictor failure workflow
  • incoming data to Auto-Arima Learner component
  • Node where it fails within Arima predictor component

Hope you could help me with this, because I cannot execute the Project since the Auto-ARIMA is located inside a loop and always fail at any iteration…

Thank you in advance!




I Forgot to upload the LOG message:

2020-06-07 16:51:05,795 : ERROR : KNIME-Worker-23-Python Predictor 0:605:606:573:526:521:520:564:0:342 : : Node : Python Predictor : 0:605:606:573:526:521:520:564:0:342 : Execute failed: The external Python process crashed for unknown reasons while KNIME set up the Python environment. See log for details.
org.knime.python2.kernel.PythonIOException: The external Python process crashed for unknown reasons while KNIME set up the Python environment. See log for details.

  • at org.knime.python2.kernel.PythonKernel.(PythonKernel.java:279)*
  • at org.knime.python2.nodes.predictor2.PythonPredictorNodeModel2.execute(PythonPredictorNodeModel2.java:79)*
  • at org.knime.core.node.NodeModel.executeModel(NodeModel.java:571)*
  • at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1236)*
  • at org.knime.core.node.Node.execute(Node.java:1016)*
  • at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:557)*
  • at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)*
  • at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:218)*
  • at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:124)*
  • at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:334)*
  • at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:210)*
  • at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)*
  • at java.util.concurrent.FutureTask.run(FutureTask.java:266)*
  • at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)*
  • at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)*
    Caused by: java.util.concurrent.ExecutionException: java.net.SocketTimeoutException: Accept timed out
  • at java.util.concurrent.FutureTask.report(FutureTask.java:122)*
  • at java.util.concurrent.FutureTask.get(FutureTask.java:192)*
  • at org.knime.python2.kernel.PythonKernel.(PythonKernel.java:273)*
  • … 14 more*
    Caused by: java.net.SocketTimeoutException: Accept timed out
  • at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method)*
  • at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:135)*
  • at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)*
  • at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)*
  • at java.net.ServerSocket.implAccept(ServerSocket.java:545)*
  • at java.net.ServerSocket.accept(ServerSocket.java:513)*
  • at java.util.concurrent.FutureTask.run(FutureTask.java:266)*
  • at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)*
  • at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)*
  • at java.lang.Thread.run(Thread.java:748)*

@mlauber71 Today I had a similar error “knime external python process creashed for unknown error”. Not in the Auto Arima learner, but in an own made component using a Python Source node.

As with @rdoce it happened once, but when I changed the logging to debug, it didn’t happen anymore.

I use Knime 4.1.3 on Windows7 with python3 (used standard setup via Anaconda for a py3_knime environment). I added two packages elasticsearch and pandasticsearch to this environment.

My workflow looks like this:

One of the ElasticSearch Reader components failed on this “unknown error”. Each of the them look like this:
afbeelding
The only packages used in the python code are pandas, elasticsearch and pandasticsearch.

What I found in the log (not with debug info, because all new attempts after switching it on went okay) is in this part of the logfile.
knime.log (2.7 KB)

Not sure if it’s important but I noticed in this log “org.knime.python2.kernel.PythonIOException” while I’m using python3.

@mlauber71 this morning I had the same error again AND the debug info was still on.

This is how the workflow looks like now:

In the crashed compont it looks like this:

The part of the knime.log where the execution starts is given here: knime.log (190.0 KB)

On line 812 (and further) of this log it shows the following (the log shows it partially in Dutch, I translated it here):

The process does not have access to the file because it’s used by another C:\Users\JanDuo\Appdata\Local\Temp\conda-21583.tmp process.
Failed to activate the Conda environment. Please make sure Conda is properly installed and added to your PATH enrivonment variable.
Cannot find file C:\Users\JanDuo\Appdata\Local\Temp\conda-21583.tmp.
The process does not have access to the file because it’s used by another process.

As if 2 of the 3 parallel flows containing the same ES Table Reader component try to use the exact same temporary names for the conda environments.

Hi @JanDuo,

What version of Conda are you using? You can find that out by, for example, executing
conda --version
in the Anaconda Prompt. The version should also be displayed on KNIME’s Python preference page (File > Preferences > KNIME > Python). Does upgrading (or perhaps even downgrading) Conda to a different version resolve the problem? I remember versions around 4.6.14 having problems when activating environments in parallel.

Marcel

1 Like

Hi @MarcelW
I’m using 4.6.14 (still).
I know it’s an older version. A couple of months ago I tried to do an update to 2019.[something], but the result was that nothing worked anymore (installations of Jupyter Notebook and RStudio kept on failing),
So I was glad I kept the older installation executable and reinstalled the one which does work for me.

Hi @MarcelW
I started checking the version immediately without reading the rest of your post … (oops). As mentioned I use this version 4.6.14 which popped up from your memory.
Maybe I should try to upgrade Anaconda again later, but right now is not a good time for me since I need to have something working right now (my whole R set up depends on it too; the reinstall last time took quite some time, especially on all those R packages I’ve installed).
I hope to find time for this upgrade later this week.

Hi @MarcelW
Performed the upgrade of Anaconda last week (conda version is now 4.8.3) and I did not have this issue (in the parallel processing of Python tasks) anymore since.
Thanks!

3 Likes

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.