Ways to speed this up? 10M + rows

[mlauber71]

@Gavin_Attard you could try and put a Cache node in front of the writer node.

@Gavin_Attard

Faster if i use parquet, but can’t seem to read the data back into the nodes.

What does this mean? What kind of data is in there that you cannot re-load it?

Have you tried to write the parquet data maybe in chunks?

knime assign failed, request status: data overflow incoming data too big

@RVC2023 one way to get data from and to R is to use Parquet files. You can use them in one go or you can use chunks which is especially useful if you have large data. [image] The large file can be split into smaller parts. If they have the same structure you could also write them in a loop and later import them back as one file: [image]

Have you tried suggestions from the article like using streaming? Maybe put a cache node in fron of a streaming part.