I’m trying to use the Python Script (DB) node on a Win7(64bit) computer. I have the Python extensions installed in Knime and configured to Anaconda python 3.5. From the node description I know i need JayDeBeApi and JPype. I have JayDeBeApi installed but I’m having a hard time finding win64bit version of JPype. Can anyone point me to a 64bit version (or an alternative to JaydeBeApi for Python Script (DB). thanks
Hi @rscottrowland2,
I followed the link in the node description of the Python Script (DB) node (https://www.lfd.uci.edu/~gohlke/pythonlibs/#jpype). This shows a number of different versions of JPype. This one seems to be for Windows 64bit: JPype1‑0.6.3‑cp37‑cp37m‑win_amd64.whl
is this what you were looking for?
Cheers,
Roland
I saw that file. I’m assuming the amd64 refers to AMD 64bit chips and not Intel chips which is what I have. (I actually tried it anyway with pip and pip complains about not supported which is consistent with my belief its for AMD chips).
Hi @rscottrowland2,
amd64 is commonly used in file names to refer to the 64-bit version of the x86 instruction set, which covers both Intel and AMD CPUs. There are some slight differences between AMD and Intel, but when it comes to installers those shouldn’t matter (else I would expect them to provide a version for both).
What is the error you are seeing when you tried to install?
Cheers,
Roland
Hi,
i want to use a Python Script (DB) node.
I have installed Python 3 without Ananconda.
I also have installed jpype and JayDeBeApi.
When i try to run this workflow
i get this error:
ERROR PythonKernelManager Failed to clean up Python. See log for details.
ERROR Python Script (DB) 2:13 Execute failed: An exception occured while running the Python kernel. See log for details.
Can anyone help me, please?
BR,
ptrwlr
Hi MarcelW,
when i open the Python Script (DB) Node the following is written to the knime.log file:
2019-05-17 08:54:26,906 : DEBUG : main : NodeContainerEditPart : : : Python Script (DB) 2:86 (IDLE)
2019-05-17 08:54:27,613 : DEBUG : Thread-91 : PythonKernel : Python Script (DB) : 2:86 : Using serialization library: Flatbuffers Column Serialization.
2019-05-17 08:54:29,269 : DEBUG : Thread-91 : PythonKernel : Python Script (DB) : 2:86 : Python PID: 11604
Hi ptrwlr,
That doesn’t include the information I’m looking for, unfortunately. The root of the error is possibly buried in the knime.log file quite deeply, so it would be easiest for you to upload the entire file here. Please see here (third and fourth paragraph) to find out how to locate the knime.log file.
Also, please let me know if you have concerns uploading the entire file (for privacy/business reasons), then I’ll gladly point you to the right spots in the file but this might get more complicated.
Marcel
Hi MarcelW,
does this include the information you are looking for?
2019-05-21 07:52:41,789 : DEBUG : Thread-15 : PythonKernel : Python Script (DB) : 2:129 : Using serialization library: Flatbuffers Column Serialization.
2019-05-21 07:52:43,577 : DEBUG : Thread-15 : PythonKernel : Python Script (DB) : 2:129 : Python PID: 1096
2019-05-21 07:52:57,876 : DEBUG : main : NodeContainer : Python Script (DB) : 2:129 : Setting dirty flag on Python Script (DB) 2:129
2019-05-21 07:52:57,882 : WARN : main : Node : Python Script (DB) : 2:129 : Database does not support insert or update
2019-05-21 07:52:58,391 : DEBUG : Thread-18 : ThreadUtils : : : Unnecessary call to runnableWithContext, because no context is available
org.knime.core.util.ThreadUtils$UnnecessaryCallException
at org.knime.core.util.ThreadUtils.runnableWithContext(ThreadUtils.java:381)
at org.knime.core.util.ThreadUtils.runnableWithContext(ThreadUtils.java:357)
at org.knime.core.util.ThreadUtils$ExecutorServiceWithContext.submit(ThreadUtils.java:94)
at org.knime.python2.kernel.messaging.DefaultTaskFactory$DefaultTask.run(DefaultTaskFactory.java:184)
at org.knime.python2.kernel.messaging.DefaultTaskFactory$DefaultTask.get(DefaultTaskFactory.java:202)
at org.knime.python2.kernel.PythonKernel.close(PythonKernel.java:1443)
at org.knime.python2.kernel.PythonKernelManager.close(PythonKernelManager.java:375)
at org.knime.python2.config.PythonSourceCodePanel$6.run(PythonSourceCodePanel.java:334)
at java.lang.Thread.run(Thread.java:748)
2019-05-21 07:52:58,392 : ERROR : Thread-18 : PythonKernelManager : : : Failed to clean up Python. See log for details.
2019-05-21 07:52:58,392 : DEBUG : Thread-18 : PythonKernelManager : : : Failed to clean up Python. See log for details.
org.knime.python2.kernel.PythonKernelCleanupException: Failed to clean up Python. See log for details.
at org.knime.python2.kernel.PythonKernel.close(PythonKernel.java:1452)
at org.knime.python2.kernel.PythonKernelManager.close(PythonKernelManager.java:375)
at org.knime.python2.config.PythonSourceCodePanel$6.run(PythonSourceCodePanel.java:334)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: Failed to receive message from Python or forward received message. Cause: Connection reset
at org.knime.python2.kernel.messaging.AbstractMessageLoop.throwExceptionInLoop(AbstractMessageLoop.java:77)
at org.knime.python2.kernel.messaging.DefaultMessageReceiverLoop.loop(DefaultMessageReceiverLoop.java:98)
at org.knime.python2.kernel.messaging.AbstractMessageLoop.doLoop(AbstractMessageLoop.java:176)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
… 1 more
Suppressed: java.lang.Exception: Failed to distribute message from Python. Cause: Message receiver loop terminated.
at org.knime.python2.kernel.messaging.AbstractMessageLoop.throwExceptionInLoop(AbstractMessageLoop.java:77)
at org.knime.python2.kernel.messaging.MessageDistributorLoop.loop(MessageDistributorLoop.java:93)
… 6 more
Caused by: java.io.IOException: Message receiver loop terminated.
at org.knime.python2.kernel.messaging.DefaultMessageReceiverLoop.receive(DefaultMessageReceiverLoop.java:82)
at org.knime.python2.kernel.messaging.MessageDistributorLoop.loop(MessageDistributorLoop.java:82)
… 6 more
[CIRCULAR REFERENCE:java.lang.Exception: Failed to distribute message from Python. Cause: Message receiver loop terminated.]
[CIRCULAR REFERENCE:java.lang.Exception: Failed to distribute message from Python. Cause: Message receiver loop terminated.]
Suppressed: java.lang.Exception: Failed to send message to Python. Cause: Connection reset by peer: socket write error
at org.knime.python2.kernel.messaging.AbstractMessageLoop.throwExceptionInLoop(AbstractMessageLoop.java:77)
at org.knime.python2.kernel.messaging.DefaultMessageSenderLoop.loop(DefaultMessageSenderLoop.java:88)
… 6 more
Caused by: java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:134)
at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
at org.knime.python2.kernel.messaging.DefaultMessageSender.send(DefaultMessageSender.java:80)
at org.knime.python2.kernel.messaging.DefaultMessageSenderLoop.loop(DefaultMessageSenderLoop.java:86)
… 6 more
[CIRCULAR REFERENCE:java.lang.Exception: Failed to distribute message from Python. Cause: Message receiver loop terminated.]
[CIRCULAR REFERENCE:java.lang.Exception: Failed to send message to Python. Cause: Connection reset by peer: socket write error]
[CIRCULAR REFERENCE:java.lang.Exception: Failed to distribute message from Python. Cause: Message receiver loop terminated.]
[CIRCULAR REFERENCE:java.lang.Exception: Failed to send message to Python. Cause: Connection reset by peer: socket write error]
[CIRCULAR REFERENCE:java.lang.Exception: Failed to distribute message from Python. Cause: Message receiver loop terminated.]
[CIRCULAR REFERENCE:java.lang.Exception: Failed to send message to Python. Cause: Connection reset by peer: socket write error]
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.knime.python2.kernel.messaging.PythonMessagingUtils.readInt(PythonMessagingUtils.java:91)
at org.knime.python2.kernel.messaging.DefaultMessageReceiver.receive(DefaultMessageReceiver.java:80)
at org.knime.python2.kernel.messaging.DefaultMessageReceiverLoop.loop(DefaultMessageReceiverLoop.java:92)
… 6 more
Hi ptrwlr,
The warning
2019-05-21 07:52:57,882 : WARN : main : Node : Python Script (DB) : 2:129 : Database does not support insert or update
could be interesting in general. But given that you only want to run the (unmodified) example workflow, this shouldn’t be a problem. Other than that, I don’t see anything helpful, unfortunately. I hoped to find some Python error tracebacks that might help to lead us in the right direction. Could you please try and search the log for the term Traceback and post all of the matches here?
Also: Did you try to run any other Python example workflow (maybe one where no databases are involved)? This would help to narrow down the problem to either databases or Python in general.
Thanks for your assistance and sorry for the inconvenience!
Marcel
Hi MarcelW,
a Python script without DB connection works fine. I talked to our technical support for the Hadoop Cluster. We do not quite understand how the Python code (DB) is executed via Hive when we use the node. I have a hive connector then used a Python Script (DB) node. Could you give me some information? Possibly the problem is not at the node itself.
Thank you very much.
BR,
ptrwlr
Hi,
the Python Script (DB) node is using the JayDeBeApi to create a connection to a database. JayDeBeApi allows us to use the same JDBC driver and connection settings in Python that we use in KNIME to establish the connection. So the Python code is not executed in the DB but on the machine where KNIME is running on. The benefit of the node is that you can work in Python directly with a ResultSet from the database instead of loading the data from the database into KNIME and then transferring it to Python.
If you want to execute Python code in your Hadoop cluster you can use the new PySpark nodes.
Bye
Tobias
3 Likes
Hi Tobias,
i spoke with our technical support for Hadoop. They said the node should actually work. Unfortunately, the error is still there. Do you have any ideas ahead?
BR,
ptrwlr
Hi,
the error message that I see in the log file is Connection reset by peer: socket write error
This indicates a problem with the communication. Is your cluster a secured cluster? If that is the case are you using the Apache Hive driver or a vendor specific driver e.g. Cloudera or Hortonworks?
We also have a dedicated Python Script (Hive) node that uses the same way to establish a connection. However the major difference is that it uses the same mechanism as the Hive Loader to upload new created or updated Python data frames.
Bye
Tobias