ERROR PythonKernel MemoryError

allen_n · November 6, 2018, 1:06am

This is a duplicate, but my original issue is back. Sometimes the script node executes okay the first time after I start KNIME, but it always fails the next time I run it. It’s a simple script, and executes fine in the Configure window:

indat = input_table.copy()
import numpy as np
import pandas as pd
import pandas_profiling  as pro

def get_var_category(series):
    unique_count = series.nunique(dropna=False)
    total_count = len(series)
    if pd.api.types.is_numeric_dtype(series):
        return 'Numerical'
    elif pd.api.types.is_datetime64_dtype(series):
        return 'Date'
    elif unique_count==total_count:
        return 'Text (Unique)'
    else:
        return 'Text'

def print_categories(df):
    for column_name in df.columns:
        print(column_name, ": ", get_var_category(df[column_name]))

def get_mode(series):
    y = series.mode()
if isinstance(y, pd.DataFrame):
    y = y.iloc[0, 0]
    return y

dat = pd.DataFrame(indat.dtypes)
dat.reset_index(inplace = True)

dat = pd.concat([dat, pd.Series([get_var_category(indat[column_name]) for column_name in  indat.columns])], axis = 1)
dat = pd.concat([dat, pd.Series([pd.isna(indat[column_name]).sum() for column_name in indat.columns])], axis = 1)
dat = pd.concat([dat, pd.Series([len(indat[column_name]) - indat[column_name].count() for column_name in indat.columns])], axis = 1)

dat.columns = ['col_name', 'dtype', 'data_type', 'NA', 'missing']
output_table = dat

The trouble seems to start with “OSError: [WinError 10038]”, and the memory error happens soon after:

2018-11-06 11:20:04,220 : DEBUG : Thread-14 : PythonKernel : : : OSError: [WinError 10038] An operation was attempted on something that is not a socket
2018-11-06 11:20:04,220 : DEBUG : Thread-15 : PythonKernel : : : Traceback (most recent call last):
2018-11-06 11:20:04,220 : DEBUG : Thread-15 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\PythonKernelLauncher.py”, line 88, in
2018-11-06 11:20:04,222 : DEBUG : Thread-15 : PythonKernel : : : raise ex
2018-11-06 11:20:04,223 : DEBUG : Thread-15 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\PythonKernelLauncher.py”, line 84, in
2018-11-06 11:20:04,225 : DEBUG : Thread-15 : PythonKernel : : : kernel.start()
2018-11-06 11:20:04,225 : DEBUG : Thread-15 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\PythonKernel.py”, line 93, in start
2018-11-06 11:20:04,227 : DEBUG : Thread-15 : PythonKernel : : : self._monitor.wait_for_close()
2018-11-06 11:20:04,227 : DEBUG : Thread-15 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\PythonKernel.py”, line 172, in wait_for_close
2018-11-06 11:20:04,229 : DEBUG : Thread-15 : PythonKernel : : : self.check_exception()
2018-11-06 11:20:04,229 : DEBUG : Thread-15 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\PythonKernel.py”, line 168, in check_exception
2018-11-06 11:20:04,231 : DEBUG : Thread-15 : PythonKernel : : : raise self._exception
2018-11-06 11:20:04,231 : DEBUG : Thread-15 : PythonKernel : : : File “C:\Users\allen.nugent\AppData\Local\Programs\Python\Python37-32\lib\concurrent\futures\thread.py”, line 57, in run
2018-11-06 11:20:04,232 : DEBUG : Thread-15 : PythonKernel : : : result = self.fn(*self.args, **self.kwargs)
2018-11-06 11:20:04,232 : DEBUG : Thread-15 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\messaging\AbstractMessageLoop.py”, line 126, in _do_loop
2018-11-06 11:20:04,234 : DEBUG : Thread-15 : PythonKernel : : : raise ex
2018-11-06 11:20:04,234 : DEBUG : Thread-15 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\messaging\AbstractMessageLoop.py”, line 123, in _do_loop
2018-11-06 11:20:04,235 : DEBUG : Thread-15 : PythonKernel : : : self._loop()
2018-11-06 11:20:04,235 : DEBUG : Thread-15 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\messaging\MessageReceiverLoop.py”, line 70, in _loop
2018-11-06 11:20:04,236 : DEBUG : Thread-15 : PythonKernel : : : message = self._receiver.receive()
2018-11-06 11:20:04,236 : DEBUG : Thread-15 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\MessageReceiver.py”, line 62, in receive
2018-11-06 11:20:04,238 : DEBUG : Thread-15 : PythonKernel : : : header_size = self._read_size()
2018-11-06 11:20:04,238 : DEBUG : Thread-15 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\MessageReceiver.py”, line 77, in _read_size
2018-11-06 11:20:04,239 : DEBUG : Thread-15 : PythonKernel : : : data.extend(self._connection.recv(4 - len(data)))
2018-11-06 11:20:04,239 : DEBUG : Thread-15 : PythonKernel : : : OSError: [WinError 10038] An operation was attempted on something that is not a socket
2018-11-06 11:20:06,134 : DEBUG : Thread-19 : PythonMessaging : : : Python messaging system could not be shut down gracefully. Process will be killed.
2018-11-06 11:20:07,395 : DEBUG : main : ExecuteAction : : : Creating execution job for 1 node(s)…
2018-11-06 11:20:07,395 : DEBUG : main : NodeContainer : : : Python Script (1â‡’1) 0:15 has new state: CONFIGURED_MARKEDFOREXEC
2018-11-06 11:20:07,397 : DEBUG : main : NodeContainer : : : Python Script (1â‡’1) 0:15 has new state: CONFIGURED_QUEUED
2018-11-06 11:20:07,398 : DEBUG : main : NodeContainer : : : KNIME_project1a_2 0 has new state: EXECUTING
2018-11-06 11:20:07,399 : DEBUG : KNIME-Worker-5 : WorkflowManager : Python Script (1â‡’1) : 0:15 : Python Script (1â‡’1) 0:15 doBeforePreExecution
2018-11-06 11:20:07,399 : DEBUG : KNIME-Worker-5 : NodeContainer : Python Script (1â‡’1) : 0:15 : Python Script (1â‡’1) 0:15 has new state: PREEXECUTE
2018-11-06 11:20:07,399 : DEBUG : KNIME-Worker-5 : WorkflowManager : Python Script (1â‡’1) : 0:15 : Python Script (1â‡’1) 0:15 doBeforeExecution
2018-11-06 11:20:07,400 : DEBUG : KNIME-Worker-5 : NodeContainer : Python Script (1â‡’1) : 0:15 : Python Script (1â‡’1) 0:15 has new state: EXECUTING
2018-11-06 11:20:07,401 : DEBUG : KNIME-Worker-5 : WorkflowFileStoreHandlerRepository : Python Script (1â‡’1) : 0:15 : Adding handler ab5365be-b9b7-4e3a-98fd-f5d8ef911f8b (Python Script (1â‡’1) 0:15: ) - 1 in total
2018-11-06 11:20:07,401 : DEBUG : KNIME-WFM-Parent-Notifier : NodeContainer : : : ROOT has new state: EXECUTING
2018-11-06 11:20:07,401 : DEBUG : KNIME-Worker-5 : LocalNodeExecutionJob : Python Script (1â‡’1) : 0:15 : Python Script (1â‡’1) 0:15 Start execute
2018-11-06 11:20:07,403 : DEBUG : KNIME-Worker-5 : PythonKernel : Python Script (1â‡’1) : 0:15 : Using serialization library: Flatbuffers Column Serialization.
2018-11-06 11:20:09,497 : DEBUG : KNIME-Worker-5 : PythonKernel : Python Script (1â‡’1) : 0:15 : Python PID: 23328
2018-11-06 11:20:09,516 : DEBUG : KNIME-Worker-5 : Buffer : Python Script (1â‡’1) : 0:15 : Opening input stream on file “C:\Users\allen.nugent\AppData\Local\Temp\knime_container_20181106_7140224257793559723.tmp”, 0 open streams
2018-11-06 11:20:37,025 : INFO : Thread-20 : PythonKernel : : : Traceback (most recent call last):
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\PythonKernelLauncher.py”, line 84, in
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : kernel.start()
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\PythonKernel.py”, line 93, in start
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : self._monitor.wait_for_close()
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\PythonKernel.py”, line 172, in wait_for_close
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : self.check_exception()
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\PythonKernel.py”, line 168, in check_exception
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : raise self._exception
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : File “C:\Users\allen.nugent\AppData\Local\Programs\Python\Python37-32\lib\concurrent\futures\thread.py”, line 57, in run
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : result = self.fn(*self.args, **self.kwargs)
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\PythonKernel.py”, line 235, in _monitored_fn
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : raise ex
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\PythonKernel.py”, line 232, in _monitored_fn
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : return fn(*args, **kwargs)
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\Task.py”, line 185, in run
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : raise ex
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\Task.py”, line 180, in run
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : self._runnable()
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\Task.py”, line 145, in _run_internal
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : received, self._message_handlers, self._message_id_supplier, self._set_result, self._workspace)
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\AbstractTaskHandler.py”, line 76, in handle
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : if self._handle_custom_message(message, message_id_supplier, message_to_send, result_consumer, workspace):
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\RequestHandlers.py”, line 91, in _handle_custom_message
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : response = self._respond(message, response_message_id_supplier(), workspace)
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\RequestHandlers.py”, line 139, in _respond
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : data_bytes = payload_decoder.get_next_bytes()
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\Message.py”, line 124, in get_next_bytes
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : return self._read_bytes(size)
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\Message.py”, line 139, in _read_bytes
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : return self._buffer[start_pointer:self._pointer]
2018-11-06 11:20:37,026 : INFO : Thread-20 : PythonKernel : : : MemoryError
2018-11-06 11:20:37,028 : DEBUG : Thread-21 : PythonKernel : : : Python error with stacktrace:

2018-11-06 11:20:37,028 : DEBUG : Thread-21 : PythonKernel : : : Traceback (most recent call last):
2018-11-06 11:20:37,028 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\PythonKernelLauncher.py”, line 88, in
2018-11-06 11:20:37,029 : DEBUG : Thread-21 : PythonKernel : : : raise ex
2018-11-06 11:20:37,029 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Users\allen.nugent\AppData\Local\Programs\Python\Python37-32\lib\concurrent\futures\thread.py”, line 57, in run
2018-11-06 11:20:37,030 : DEBUG : Thread-21 : PythonKernel : : : result = self.fn(*self.args, **self.kwargs)
2018-11-06 11:20:37,030 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\messaging\AbstractMessageLoop.py”, line 126, in _do_loop
2018-11-06 11:20:37,032 : DEBUG : Thread-21 : PythonKernel : : : raise ex
2018-11-06 11:20:37,032 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\messaging\AbstractMessageLoop.py”, line 123, in _do_loop
2018-11-06 11:20:37,033 : DEBUG : Thread-21 : PythonKernel : : : self._loop()
2018-11-06 11:20:37,033 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\messaging\MessageDistributorLoop.py”, line 97, in _loop
2018-11-06 11:20:37,041 : DEBUG : Thread-21 : PythonKernel : : : raise ex
2018-11-06 11:20:37,041 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\messaging\MessageDistributorLoop.py”, line 92, in _loop
2018-11-06 11:20:37,043 : DEBUG : Thread-21 : PythonKernel : : : message = self._receiver.receive()
2018-11-06 11:20:37,043 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\messaging\MessageReceiverLoop.py”, line 62, in receive
2018-11-06 11:20:37,044 : DEBUG : Thread-21 : PythonKernel : : : message = self._receive_queue.get()
2018-11-06 11:20:37,044 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\PythonKernel.py”, line 209, in get
2018-11-06 11:20:37,046 : DEBUG : Thread-21 : PythonKernel : : : self._monitor.check_exception()
2018-11-06 11:20:37,046 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\PythonKernel.py”, line 168, in check_exception
2018-11-06 11:20:37,047 : DEBUG : Thread-21 : PythonKernel : : : raise self._exception
2018-11-06 11:20:37,047 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\PythonKernelLauncher.py”, line 84, in
2018-11-06 11:20:37,049 : DEBUG : Thread-21 : PythonKernel : : : kernel.start()
2018-11-06 11:20:37,049 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\PythonKernel.py”, line 93, in start
2018-11-06 11:20:37,050 : DEBUG : Thread-21 : PythonKernel : : : self._monitor.wait_for_close()
2018-11-06 11:20:37,050 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\PythonKernel.py”, line 172, in wait_for_close
2018-11-06 11:20:37,052 : DEBUG : Thread-21 : PythonKernel : : : self.check_exception()
2018-11-06 11:20:37,052 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\PythonKernel.py”, line 168, in check_exception
2018-11-06 11:20:37,053 : DEBUG : Thread-21 : PythonKernel : : : raise self._exception
2018-11-06 11:20:37,053 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Users\allen.nugent\AppData\Local\Programs\Python\Python37-32\lib\concurrent\futures\thread.py”, line 57, in run
2018-11-06 11:20:37,054 : DEBUG : Thread-21 : PythonKernel : : : result = self.fn(*self.args, **self.kwargs)
2018-11-06 11:20:37,054 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\PythonKernel.py”, line 235, in _monitored_fn
2018-11-06 11:20:37,056 : DEBUG : Thread-21 : PythonKernel : : : raise ex
2018-11-06 11:20:37,056 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\python3\PythonKernel.py”, line 232, in _monitored_fn
2018-11-06 11:20:37,058 : DEBUG : Thread-21 : PythonKernel : : : return fn(*args, **kwargs)
2018-11-06 11:20:37,058 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\Task.py”, line 185, in run
2018-11-06 11:20:37,059 : DEBUG : Thread-21 : PythonKernel : : : raise ex
2018-11-06 11:20:37,059 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\Task.py”, line 180, in run
2018-11-06 11:20:37,061 : DEBUG : Thread-21 : PythonKernel : : : self._runnable()
2018-11-06 11:20:37,061 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\Task.py”, line 145, in _run_internal
2018-11-06 11:20:37,062 : DEBUG : Thread-21 : PythonKernel : : : received, self._message_handlers, self._message_id_supplier, self._set_result, self._workspace)
2018-11-06 11:20:37,062 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\AbstractTaskHandler.py”, line 76, in handle
2018-11-06 11:20:37,064 : DEBUG : Thread-21 : PythonKernel : : : if self._handle_custom_message(message, message_id_supplier, message_to_send, result_consumer, workspace):
2018-11-06 11:20:37,064 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\RequestHandlers.py”, line 91, in _handle_custom_message
2018-11-06 11:20:37,065 : DEBUG : Thread-21 : PythonKernel : : : response = self._respond(message, response_message_id_supplier(), workspace)
2018-11-06 11:20:37,065 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\RequestHandlers.py”, line 139, in _respond
2018-11-06 11:20:37,067 : DEBUG : Thread-21 : PythonKernel : : : data_bytes = payload_decoder.get_next_bytes()
2018-11-06 11:20:37,067 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\Message.py”, line 124, in get_next_bytes
2018-11-06 11:20:37,068 : DEBUG : Thread-21 : PythonKernel : : : return self._read_bytes(size)
2018-11-06 11:20:37,068 : DEBUG : Thread-21 : PythonKernel : : : File “C:\Program Files\KNIME\plugins\org.knime.python2_3.6.1.v201808311614\py\messaging\Message.py”, line 139, in _read_bytes
2018-11-06 11:20:37,069 : DEBUG : Thread-21 : PythonKernel : : : return self._buffer[start_pointer:self._pointer]
2018-11-06 11:20:37,069 : ERROR : Thread-21 : PythonKernel : : : MemoryError
2018-11-06 11:20:39,028 : DEBUG : KNIME-Worker-5 : Buffer : Python Script (1â‡’1) : 0:15 : Closing input stream on “C:\Users\allen.nugent\AppData\Local\Temp\knime_container_20181106_7140224257793559723.tmp”, 0 remaining
2018-11-06 11:20:39,029 : DEBUG : KNIME-Worker-5 : Node : Python Script (1â‡’1) : 0:15 : reset
2018-11-06 11:20:39,029 : ERROR : KNIME-Worker-5 : Node : Python Script (1â‡’1) : 0:15 : Execute failed: An exception occured while running the Python kernel. See log for details.
2018-11-06 11:20:39,031 : DEBUG : KNIME-Worker-5 : Node : Python Script (1â‡’1) : 0:15 : Execute failed: An exception occured while running the Python kernel. See log for details.
org.knime.python2.kernel.PythonIOException: An exception occured while running the Python kernel. See log for details.
at org.knime.python2.kernel.PythonKernel.getMostSpecificPythonKernelException(PythonKernel.java:1526)
at org.knime.python2.kernel.PythonKernel.putDataTable(PythonKernel.java:766)
at org.knime.python2.kernel.PythonKernel.putDataTable(PythonKernel.java:785)
at org.knime.python2.nodes.script.PythonScriptNodeModel.execute(PythonScriptNodeModel.java:88)
at org.knime.core.node.NodeModel.execute(NodeModel.java:733)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:567)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1177)
at org.knime.core.node.Node.execute(Node.java:964)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:561)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
Caused by: java.util.concurrent.ExecutionException: java.lang.Exception: Failed to receive message from Python or forward received message.
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.knime.python2.kernel.messaging.DefaultTaskFactory$DefaultTask.get(DefaultTaskFactory.java:196)
at org.knime.python2.kernel.PythonKernel.putDataTable(PythonKernel.java:764)
… 16 more
Caused by: java.lang.Exception: Failed to receive message from Python or forward received message.
at org.knime.python2.kernel.messaging.AbstractMessageLoop.throwExceptionInLoop(AbstractMessageLoop.java:76)
at org.knime.python2.kernel.messaging.DefaultMessageReceiverLoop.loop(DefaultMessageReceiverLoop.java:98)
at org.knime.python2.kernel.messaging.AbstractMessageLoop.doLoop(AbstractMessageLoop.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.lang.Exception: Failed to distribute message from Python. Cause: Message receiver loop terminated.
at org.knime.python2.kernel.messaging.AbstractMessageLoop.throwExceptionInLoop(AbstractMessageLoop.java:76)
at org.knime.python2.kernel.messaging.MessageDistributorLoop.loop(MessageDistributorLoop.java:93)
… 4 more
Caused by: java.io.IOException: Message receiver loop terminated.
at org.knime.python2.kernel.messaging.DefaultMessageReceiverLoop.receive(DefaultMessageReceiverLoop.java:82)
at org.knime.python2.kernel.messaging.MessageDistributorLoop.loop(MessageDistributorLoop.java:82)
… 4 more
Caused by: java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.knime.python2.kernel.messaging.PythonMessagingUtils.readInt(PythonMessagingUtils.java:91)
at org.knime.python2.kernel.messaging.DefaultMessageReceiver.receive(DefaultMessageReceiver.java:80)
at org.knime.python2.kernel.messaging.DefaultMessageReceiverLoop.loop(DefaultMessageReceiverLoop.java:92)
… 4 more
2018-11-06 11:20:39,034 : DEBUG : KNIME-Worker-5 : WorkflowManager : Python Script (1â‡’1) : 0:15 : Python Script (1â‡’1) 0:15 doBeforePostExecution
2018-11-06 11:20:39,034 : DEBUG : KNIME-Worker-5 : NodeContainer : Python Script (1â‡’1) : 0:15 : Python Script (1â‡’1) 0:15 has new state: POSTEXECUTE
2018-11-06 11:20:39,035 : DEBUG : KNIME-Worker-5 : WorkflowManager : Python Script (1â‡’1) : 0:15 : Python Script (1â‡’1) 0:15 doAfterExecute - failure
2018-11-06 11:20:39,035 : DEBUG : KNIME-Worker-5 : Node : Python Script (1â‡’1) : 0:15 : reset
2018-11-06 11:20:39,035 : DEBUG : KNIME-Worker-5 : Node : Python Script (1â‡’1) : 0:15 : clean output ports.
2018-11-06 11:20:39,035 : DEBUG : KNIME-Worker-5 : WorkflowFileStoreHandlerRepository : Python Script (1â‡’1) : 0:15 : Removing handler ab5365be-b9b7-4e3a-98fd-f5d8ef911f8b (Python Script (1â‡’1) 0:15: ) - 0 remaining
2018-11-06 11:20:39,035 : DEBUG : KNIME-Worker-5 : NodeContainer : Python Script (1â‡’1) : 0:15 : Python Script (1â‡’1) 0:15 has new state: IDLE
2018-11-06 11:20:39,036 : DEBUG : KNIME-Worker-5 : Node : Python Script (1â‡’1) : 0:15 : Configure succeeded. (Python Script (1â‡’1))
2018-11-06 11:20:39,036 : DEBUG : KNIME-Worker-5 : NodeContainer : Python Script (1â‡’1) : 0:15 : Python Script (1â‡’1) 0:15 has new state: CONFIGURED
2018-11-06 11:20:39,036 : DEBUG : KNIME-Worker-5 : NodeContainer : Python Script (1â‡’1) : 0:15 : KNIME_project1a_2 0 has new state: CONFIGURED
2018-11-06 11:20:39,037 : DEBUG : KNIME-WFM-Parent-Notifier : NodeContainer : : : ROOT has new state: IDLE
2018-11-06 11:20:39,070 : DEBUG : KNIME-Node-Usage-Writer : NodeTimer$GlobalNodeStats : : : Successfully wrote node usage stats to file: C:\Users\allen.nugent\knime-workspace.metadata\knime\nodeusage_3.0.json

allen_n · November 6, 2018, 1:19am

Actually, don’t even worry about my code: I get the same issue if I create a new “Python Script (1=>1)” node and execute it.

MarcelW · November 9, 2018, 11:53am

Hi Allen,

How large is the input table of that Python node in terms of number of rows, number of columns and data types (e.g., images)? Is it again from a ~200 MB CSV file?

Apart from that, here are some general things which would make sense to check (some of these points are probably unlikely since you sometimes are able to run the script once):

Did you assign a large portion of the machine’s main memory to KNIME (e.g., via the -Xmx option in the knime.ini file)? If yes, then it might be possible that there’s not enough memory left for the Python process (which does not run inside KNIME) after the first run of the node. Decreasing the value could solve the problem.
Can you check how many Python processes are running before and after you are successfully running the script after restarting KNIME? Maybe the Python process is kept open for some reason and still occupies memory. You could check that using some system monitor/task manager.
Can you observe that the (failing) Python process or the KNIME process utilize an “unreasonably” large amount of memory or that your overall memory usage is exceptionally high at that moment?
Do you happen to accidentally run a 32bit version of Python? (Unlikely since it runs once.)
You can check that via

import sys
sys.maxsize

which should return something larger than 2^32 if you’re Python installation is 64bit.
Is your operating system configured to impose memory limits on a per-process level? This is probably unlikely since you’d have run into problems earlier, e.g., when using native KNIME nodes.

Marcel

allen_n · November 19, 2018, 5:25am

Thanks, Marcel

Sorry, I’m just getting back to this now.

sys.maxsize returns 2147483647 when executed from the node’s script window. The Python 3 version in Preferences looks like 32-bit, so I’m going to install the 64-bit version and change the Preferences and see what happens.

system · June 2, 2023, 9:30pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.