i have been trying to load a 4gb csv file. it has a table of 400,000 columns over 5000 rows. 99% of the information is either 1 or 0 (8 bits) while a couple are actually strings.
i am running knime with -Xmx3000m .
the file reader node will fault at 5% with or without the “-Xmx” with "execute failed: java heap space: which suggest that there isnt enough memory to load the file, but i did load knime with enough memory according to what people said in the forum. plus i dont have more than 3gb ram.
what can i do to fix this ?
Hi,
I don’t have a solution (yet) - 400K columns is just awfully big…
But to better locate the problem, I have a question:
From your post I assume that you are able to configure the File Reader, is that right? Are you able to open the dialog, adjust all settings - you see a preview table that somehow makes sense - and OK the dialog?
Only when you execute the file reader this out of mem error occures, right?
Thanks,
yes you are right.
I can configure the file, and if i push the “quick” option i can preview the file and OK the dialog.
i havent tried the processing version though.
i might have found a GUI bug somewhere when you load up the list of features somewhere, but that is beside the point.
I have seen the same type of behavior with the database reader node. If the table is too large N*M (not necessarily just M) then you can even OK the dialog.
i managed to reduce my data to 5000x40000 (10 times smaller), the file itself is 400mb .
but still there it reports a heap problem. although this time it manages to load it up to 77%.