Handling Many Input Channels in Python Script Node

[mlauber71]

@ekadagami question is if the files all have the same structure. If yes you could put them in a folder as parquet files and address them as one file in Python.

use the new (KNIME 4.6+) Python Script node and bundled Python version to...

In general parquet or SQLite could be formats to communicate between KNIME and Python. SQLite could store several tables and the tables can be accessed within Python / KNIME nodes. Question is if the code to handle them would be based on the structure of the table.

knime assign failed, request status: data overflow incoming data too big

@RVC2023 one way to get data from and to R is to use Parquet files. You can use them in one go or you can use chunks which is especially useful if you have large data. [image] The large file can be split into smaller parts. If they have the same structure you could also write them in a loop and later import them back as one file: [image]

KNIME Snippets (1): Collect and Restore — or how to handle many large files...