@Erich_Gstrein these things come to my mind:
- use a cache node right in front of the operation and maybe force KNIME to write the result to disk.
- try to revert to the traditional internal storage and maybe tweak the settings by going for gzip instead of snappy. Arrow was still in Labs before KNIME 4.5 and over the time (although it is a great format) the formats (namely Parquet) were not that stable (hopefully with the new columnar storage out of labs status that might change)
- if possible you could think about doing work in chunks or splitting it into several Workflows (not very elegant or popular, I know)
- check out this post
Collection of further things to explore:
And if it is about transferring data from and to Python (besides what I just said about Parquet :-)) I have use the generic format and Reader and Writer to transfer data between KNIME, R and Python without using the data connection (old version, new version). Also SQLite is an option.