How to filter big CSV files?

Hello.

I have some CSV files, which have about 15gb each. I would like to filter them and save only 2 or 3 columns as a new file. So I use File Reader -> Column Filter -> CSV Writer.

 

But File Reader throws an error:  ERROR File Reader          0:4        Execute failed: Java heap space

 

I've edited the knime.ini file and set -Xmx8013m

Also, I've set a File Reader module to save everything to a HDD. 

And it still doesn't work.

 

Do you have any ideas on how to filter these CSV files? 

 

 

Hi Tomek,

 

yes this is the perfect usecase for KNIME Streaminig.

Just wrap the complete workflow into a wrapped meta node, enable the Simple Streaming and it will be a lot faster.

Here is a video, demoing you the functionality: https://www.youtube.com/watch?v=OKyjTCWqemQ

Best, Iris