Python Script Node: error occurred during serialization

KNIME Analytics Platform
1

I’m currently getting this error from a Python Script Node where the input is very large.

ERROR Python Script        0:45:0:1   Execute failed: An error occurred during serialization. See log for errors.

Here’s the log:

2022-04-27 14:53:36,717 : ERROR : KNIME-Worker-38-Python Script 0:45:0:1 :  : Node : Python Script : 0:45:0:1 : Execute failed: An error occurred during serialization. See log for errors.
org.knime.python2.extensions.serializationlibrary.SerializationException: An error occurred during serialization. See log for errors.
	at org.knime.python2.serde.flatbuffers.Flatbuffers.tableToBytes(Flatbuffers.java:149)
	at org.knime.python2.kernel.Python2KernelBackend.putDataTable(Python2KernelBackend.java:841)
	at org.knime.python2.kernel.Python2KernelBackend.putDataTable(Python2KernelBackend.java:872)
	at org.knime.python2.kernel.PythonKernel.putDataTable(PythonKernel.java:301)
	at org.knime.python2.ports.DataTableInputPort.execute(DataTableInputPort.java:116)
	at org.knime.python2.nodes.script2.PythonScriptNodeModel2.execute(PythonScriptNodeModel2.java:127)
	at org.knime.core.node.NodeModel.executeModel(NodeModel.java:549)
	at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1267)
	at org.knime.core.node.Node.execute(Node.java:1041)
	at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:559)
	at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
	at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:201)
	at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:117)
	at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:367)
	at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:221)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
	at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
Caused by: org.knime.python2.kernel.PythonIOException: FlatBuffers: cannot grow buffer beyond 2 gigabytes.
	at org.knime.python2.util.PythonUtils$Misc.executeCancelableUnwrapExecutionException(PythonUtils.java:312)
	at org.knime.python2.util.PythonUtils$Misc.executeCancelable(PythonUtils.java:283)
	at org.knime.python2.serde.flatbuffers.Flatbuffers.tableToBytes(Flatbuffers.java:146)
	... 18 more
Caused by: java.lang.AssertionError: FlatBuffers: cannot grow buffer beyond 2 gigabytes.
	at com.google.flatbuffers.FlatBufferBuilder.growByteBuffer(FlatBufferBuilder.java:198)
	at com.google.flatbuffers.FlatBufferBuilder.prep(FlatBufferBuilder.java:244)
	at com.google.flatbuffers.FlatBufferBuilder.startVector(FlatBufferBuilder.java:414)
	at org.knime.python2.serde.flatbuffers.flatc.IntColumn.createValuesVector(IntColumn.java:122)
	at org.knime.python2.serde.flatbuffers.inserters.IntInserter.createColumn(IntInserter.java:105)
	at org.knime.python2.serde.flatbuffers.Flatbuffers.tableToBytesInternal(Flatbuffers.java:275)
	at org.knime.python2.serde.flatbuffers.Flatbuffers.lambda$0(Flatbuffers.java:146)
	at org.knime.core.util.ThreadUtils$CallableWithContextImpl.callWithContext(ThreadUtils.java:383)
	at org.knime.core.util.ThreadUtils$CallableWithContext.call(ThreadUtils.java:269)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)

Is there anything I can do?

2

@azsb1g19 you could try and save the file from within the node as Parquet without exporting it to KNIME and read it back:

You could try and use the new Columnar storage backend with Python and try to use that.

Then you could try and split your data into several ORC files and bring them back together later in KNIME. ORC and Parquet both support combining several physical files into one file that KNIME can use:

3

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.