Python Source: Extreme Slow Data Transfer to Knime

[mlauber71]

@ActionAndi I think in order to take full advantage of the new columnar storage you might have to switch your Python code to the new nodes, which are still in labs status:

Data passing to Python node still slow

@Experimenter two things about that. With the version 4.5 KNIME launched a new Python file handling support that should greatly increase the speed. You could try a sample here: You will have to active the Apache/Parquet based columnar storage to take full advantage: https://docs.knime.com/latest/analytics_platform_workbench_guide/index.html#table-backend Since the new integration is yet in Labs status you could also try and use Parquet to bring data in and out of KNIME and Python. Maybe no…

What I sometimes do in the meantime is save the data from within the Python node as Parquet and then read that file back into KNIME (yes not the most sophisticated solution, but it should work) - cf. old and new Python nodes.