Workflow in Executor fails when started in Webportal or as "new Job on Server" with Python (not found)

[sven-abx]

Hello,

If I execute the Workflow with the Analytics Platform (Executor) on the Server everything works fine. But when I start the Workflow at the Webportal or as a new Job on the Server with my local AP the following happens:

Short: The Python Directory is not found.

Node (Keras Network Learner):

There are messages for workflow “08_Sentiment_Analysis_with_Deep_Learning_KNIME_nodes 2020-11-18 21.17.07”
Keras Network Learner 3:16 - ERROR: Execute failed: An error occurred while trying to launch Python: Could not start Python kernel. Error during Python installation test: Could not start Python executable at the given location (no_conda_environment_selected/bin/python): Cannot run program “no_conda_environment_selected/bin/python”: error=2, Datei oder Verzeichnis nicht gefunden
DL Network Executor (deprecated) 3:15 - WARNING: Network input ‘input_1_0:0’ is not yet configured.

epf-File:

#Thu Nov 19 01:09:33 CET 2020
!/=
/configuration/org.eclipse.core.net/org.eclipse.core.net.hasMigrated=true
/configuration/org.eclipse.ui.ide/MAX_RECENT_WORKSPACES=10
/configuration/org.eclipse.ui.ide/RECENT_WORKSPACES=/home/knime/knime-workspace
/configuration/org.eclipse.ui.ide/RECENT_WORKSPACES_PROTOCOL=3
/configuration/org.eclipse.ui.ide/SHOW_RECENT_WORKSPACES=false
/configuration/org.eclipse.ui.ide/SHOW_WORKSPACE_SELECTION_DIALOG=true
/instance/de.mpicbg.knime.scripting.python/python.exec=/home/knime/anaconda3/bin/python
/instance/org.eclipse.ant.ui/useAnnotationsPrefPage=true
/instance/org.eclipse.ant.ui/useQuickDiffPrefPage=true
/instance/org.eclipse.core.net/org.eclipse.core.net.hasMigrated=true
/instance/org.eclipse.core.resources/version=1
/instance/org.eclipse.datatools.connectivity.sqm.core.ui/org.eclipse.datatools.connectivity.sqm.core.ui.column.decoration=[{datatype} {nullable} {primaryKey} {foreignKey}]
/instance/org.eclipse.e4.ui.css.swt.theme/themeid=org.knime.product.theme.knime
/instance/org.eclipse.ui.browser/internalWebBrowserHistory=file:/tmp/intro8351848283408531324.html||file:/tmp/intro4573988660112740087.html||file:/tmp/intro5319186241369164354.html||file:/tmp/intro8399573192369477438.html||file:/tmp/intro1394992779683066946.html||file:/tmp/intro9171895247179337483.html||file:/tmp/intro7824891912785552340.html||file:/tmp/intro3382308186863939610.html||file:/tmp/intro4114670003466010241.html||file:/tmp/intro1460452153031790154.html||file:/tmp/intro3941646937364837140.html||file:/tmp/intro7754595036654242435.html||file:/tmp/intro8565427831637422203.html||file:/tmp/intro6255495381847521298.html||file:/tmp/intro5276595102179920742.html||file:/tmp/intro4779133863314997099.html||file:/tmp/intro6417170387666390803.html||file:/tmp/intro188121484914847688.html||file:/tmp/intro8216711701108190133.html|*|
/instance/org.eclipse.ui.editors/overviewRuler_migration=migrated_3.1
/instance/org.eclipse.ui.workbench//org.eclipse.ui.commands/state/org.eclipse.ui.navigator.resources.nested.changeProjectPresentation/org.eclipse.ui.commands.radioState=false
/instance/org.knime.core/knime.core.row-container-factory=org.knime.core.data.container.BufferedRowContainerFactory
/instance/org.knime.core/knime.core.table-store-format=org.knime.core.data.container.DefaultTableStoreFormat
/instance/org.knime.dl.python/condaDirectoryPath=/home/knime/anaconda3
/instance/org.knime.dl.python/kerasCondaEnvironmentDirectoryPath=/home/knime/anaconda3/envs/knime_dl_keras
/instance/org.knime.dl.python/librarySelection=keras
/instance/org.knime.dl.python/manualConfig=python3
/instance/org.knime.dl.python/pythonConfigSelection=dl
/instance/org.knime.dl.python/pythonEnvironmentType=conda
/instance/org.knime.dl.python/serializerId=org.knime.serialization.flatbuffers.column
/instance/org.knime.dl.python/tf2CondaEnvironmentDirectoryPath=/home/knime/anaconda3/envs/knime_dl_tf2
/instance/org.knime.dl.python/tf2ManualConfig=python3
/instance/org.knime.python2/condaDirectoryPath=/home/knime/anaconda3
/instance/org.knime.python2/python2CondaEnvironmentDirectoryPath=/home/knime/anaconda3/envs/knime_py2
/instance/org.knime.python2/python2Path=/home/knime/anaconda3/bin/python
/instance/org.knime.python2/python3CondaEnvironmentDirectoryPath=/home/knime/anaconda3/envs/knime_py3
/instance/org.knime.python2/python3Path=/home/knime/anaconda3/bin/python
/instance/org.knime.python2/pythonEnvironmentType=conda
/instance/org.knime.python2/serializerId=org.knime.serialization.flatbuffers.column
/instance/org.knime.workbench.core/knime.askedToSendStatistics=true
/instance/org.knime.workbench.core/knime.tempDir=/home/knime/knime-tmp
/instance/org.knime.workbench.core/knime.workspace.version=20190627
/instance/org.knime.workbench.core/logging.logfile.location=true
/instance/org.knime.workbench.core/logging.logfile.logGlobalInWfDir=true
/instance/org.knime.workbench.core/logging.loglevel.logfile=ERROR
/instance/org.knime.workbench.explorer.view/mountpointNode/EXAMPLES/active=true
/instance/org.knime.workbench.explorer.view/mountpointNode/EXAMPLES/defaultMountID=EXAMPLES
/instance/org.knime.workbench.explorer.view/mountpointNode/EXAMPLES/factoryID=com.knime.explorer.server.examples
/instance/org.knime.workbench.explorer.view/mountpointNode/EXAMPLES/mountID=EXAMPLES
/instance/org.knime.workbench.explorer.view/mountpointNode/EXAMPLES/mountpointNumber=1
/instance/org.knime.workbench.explorer.view/mountpointNode/EXAMPLES/useRest=false
/instance/org.knime.workbench.explorer.view/mountpointNode/LOCAL/active=true
/instance/org.knime.workbench.explorer.view/mountpointNode/LOCAL/defaultMountID=LOCAL
/instance/org.knime.workbench.explorer.view/mountpointNode/LOCAL/factoryID=org.knime.workbench.explorer.workspace
/instance/org.knime.workbench.explorer.view/mountpointNode/LOCAL/mountID=LOCAL
/instance/org.knime.workbench.explorer.view/mountpointNode/LOCAL/mountpointNumber=2
/instance/org.knime.workbench.explorer.view/mountpointNode/LOCAL/useRest=false
/instance/org.knime.workbench.explorer.view/mountpointNode/My-KNIME-Hub/active=true
/instance/org.knime.workbench.explorer.view/mountpointNode/My-KNIME-Hub/defaultMountID=My-KNIME-Hub
/instance/org.knime.workbench.explorer.view/mountpointNode/My-KNIME-Hub/factoryID=com.knime.explorer.server.workflow_hub
/instance/org.knime.workbench.explorer.view/mountpointNode/My-KNIME-Hub/mountID=My-KNIME-Hub
/instance/org.knime.workbench.explorer.view/mountpointNode/My-KNIME-Hub/mountpointNumber=0
/instance/org.knime.workbench.explorer.view/mountpointNode/My-KNIME-Hub/useRest=false
/instance/org.knime.workbench.workflowcoach/community_node_triple_provider=true
@de.mpicbg.knime.scripting.python=4.0.0.v202006300844
@org.eclipse.ant.ui=3.7.700.v20200304-2203
@org.eclipse.core.net=1.3.800.v20200218-0937
@org.eclipse.core.resources=3.13.700.v20200209-1624
@org.eclipse.datatools.connectivity.sqm.core.ui=1.4.101.201811012051
@org.eclipse.e4.ui.css.swt.theme=0.12.600.v20200124-0005
@org.eclipse.ui.browser=3.6.700.v20200217-0820
@org.eclipse.ui.editors=3.13.100.v20200130-1507
@org.eclipse.ui.workbench=3.118.0.v20200222-0719
@org.knime.core=4.2.2.v202009250802
@org.knime.dl.python=4.2.1.v202008251157
@org.knime.python2=4.2.2.v202009241055
@org.knime.workbench.core=4.2.1.v202008251912
@org.knime.workbench.explorer.view=8.6.1.v202009241342
@org.knime.workbench.workflowcoach=4.2.1.v202008251912
file_export_version=3.0

knime.log:

2020-11-19 11:54:23,322 : ERROR : KNIME-Worker-10-Keras Network Learner 0:16 : 32461b51-8773-48d9-8383-7fada006ffc0 : Node : Keras Network Learner : 0:16 : Execute failed: An error occurred while trying to launch Python: Could not start Python kernel. Error during Python installation test: Could not start Python executable at the given location (no_conda_environment_selected/bin/python): Cannot run program “no_conda_environment_selected/bin/python”: error=2, Datei oder Verzeichnis nicht gefunden
java.io.IOException: An error occurred while trying to launch Python: Could not start Python kernel. Error during Python installation test: Could not start Python executable at the given location (no_conda_environment_selected/bin/python): Cannot run program “no_conda_environment_selected/bin/python”: error=2, Datei oder Verzeichnis nicht gefunden
at org.knime.dl.keras.base.portobjects.DLKerasUnmaterializedPortObjectContent.materialize(DLKerasUnmaterializedPortObjectContent.java:118)
at org.knime.dl.keras.base.portobjects.DLKerasUnmaterializedNetworkPortObject.getNetwork(DLKerasUnmaterializedNetworkPortObject.java:123)
at org.knime.dl.keras.base.portobjects.DLKerasUnmaterializedNetworkPortObject.getNetwork(DLKerasUnmaterializedNetworkPortObject.java:1)
at org.knime.dl.keras.base.nodes.learner.DLKerasLearnerNodeModel.executeInternal(DLKerasLearnerNodeModel.java:624)
at org.knime.dl.keras.base.nodes.learner.DLKerasLearnerNodeModel.execute(DLKerasLearnerNodeModel.java:303)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:576)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1236)
at org.knime.core.node.Node.execute(Node.java:1016)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:558)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:201)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:117)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:334)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:210)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
Caused by: org.knime.dl.core.DLInvalidEnvironmentException: An error occurred while trying to launch Python: Could not start Python kernel. Error during Python installation test: Could not start Python executable at the given location (no_conda_environment_selected/bin/python): Cannot run program “no_conda_environment_selected/bin/python”: error=2, Datei oder Verzeichnis nicht gefunden
at org.knime.dl.python.core.DLPythonAbstractContext.createKernel(DLPythonAbstractContext.java:253)
at org.knime.dl.python.core.DLPythonAbstractContext.getKernel(DLPythonAbstractContext.java:122)
at org.knime.dl.python.core.DLPythonAbstractContext.executeInKernel(DLPythonAbstractContext.java:168)
at org.knime.dl.python.core.DLPythonAbstractCommands.getContext(DLPythonAbstractCommands.java:217)
at org.knime.dl.keras.core.layers.DLKerasNetworkMaterializer.materialize(DLKerasNetworkMaterializer.java:181)
at org.knime.dl.keras.base.portobjects.DLKerasUnmaterializedPortObjectContent.materialize(DLKerasUnmaterializedPortObjectContent.java:107)
… 17 more
Caused by: org.knime.python2.kernel.PythonInstallationTestException: Could not start Python kernel. Error during Python installation test: Could not start Python executable at the given location (no_conda_environment_selected/bin/python): Cannot run program “no_conda_environment_selected/bin/python”: error=2, Datei oder Verzeichnis nicht gefunden
at org.knime.python2.kernel.PythonKernel.testInstallation(PythonKernel.java:378)
at org.knime.python2.kernel.PythonKernel.(PythonKernel.java:261)
at org.knime.python2.kernel.PythonKernelQueue$KeyedPooledPythonKernelFactory.createKernel(PythonKernelQueue.java:342)
at org.knime.python2.kernel.PythonKernelQueue$KeyedPooledPythonKernelFactory.populateHolder(PythonKernelQueue.java:335)
at org.knime.python2.kernel.PythonKernelQueue$KeyedPooledPythonKernelFactory.makeObject(PythonKernelQueue.java:318)
at org.knime.python2.kernel.PythonKernelQueue$KeyedPooledPythonKernelFactory.makeObject(PythonKernelQueue.java:1)
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1051)
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:347)
at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:270)
at org.knime.python2.kernel.PythonKernelQueue.dequeueHolder(PythonKernelQueue.java:235)
at org.knime.python2.kernel.PythonKernelQueue.getNextKernelInternal(PythonKernelQueue.java:219)
at org.knime.python2.kernel.PythonKernelQueue.getNextKernel(PythonKernelQueue.java:144)
at org.knime.dl.python.core.DLPythonAbstractContext.createKernel(DLPythonAbstractContext.java:247)
… 22 more

I tried several things like:

Could not start python kernel error

A colleague installed Python on KNIME (WebPortal v 4.6.0) as per the instructions here (on Ubuntu 16.04): https://www.knime.com/blog/setting-up-the-knime-python-extension-revisited-for-python-30-and-20 When we try an execute a workflow in the browser we get an error, then checking in the logs I see the following (which is the same error but just a little more detailed): 17-Jul-2018 15:32:04.861 INFO [http-nio-8080-exec-38] com.knime.enterprise.server.jobs.WorkflowJobManagerImpl.execute Execut…

Keras To TensorFlow Network Converter node issue

Hi Alexander, I ran the KNIME from terminal and I got this: Exception in thread “Timer-2” java.lang.SecurityException: Could not lock User prefs. Lock file access denied. at java.util.prefs.FileSystemPreferences.checkLockFile0ErrorCode(FileSystemPreferences.java:949) at java.util.prefs.FileSystemPreferences.lockFile(FileSystemPreferences.java:937) at java.util.prefs.FileSystemPreferences.sync(FileSystemPreferences.java:741) at java.util.prefs.FileSystemPreferences.flush(FileSystemPreferen…

I hope someone can help me out.

BR,
Sven

[MarcelW]

Hi Sven,

Thanks for the detailed error report!

That is a bug in how the Python Deep Learning Preferences are set in the Executor when it is used to run jobs. We are currently working on fixing that – apologies!
In the meantime, you could specify the paths to the correct Python binaries manually instead of via Conda. You would need to change the following Preferences entries as shown:

org.knime.dl.python/pythonEnvironmentType=manual
org.knime.dl.python/manualConfig=/home/knime/anaconda3/envs/knime_dl_keras/bin/python
org.knime.dl.python/tf2ManualConfig=/home/knime/anaconda3/envs/knime_dl_tf2/bin/python

Marcel

[sven-abx]

Hi Marcel,

thanks for your reply and it is working now.
Searched a few hours yesterday and today.

Sven