cant load 4gb csv file, available memory 3gb on vista

hi,

i have been trying to load a 4gb csv file. it has a table of 400,000 columns over 5000 rows. 99% of the information is either 1 or 0 (8 bits) while a couple are actually strings.
i am running knime with -Xmx3000m .
the file reader node will fault at 5% with or without the “-Xmx” with "execute failed: java heap space: which suggest that there isnt enough memory to load the file, but i did load knime with enough memory according to what people said in the forum. plus i dont have more than 3gb ram.
what can i do to fix this ?

thanks!

come on people or dear devs, someone must know how to deal with this :slight_smile:

Hi,
I don’t have a solution (yet) - 400K columns is just awfully big…
But to better locate the problem, I have a question:
From your post I assume that you are able to configure the File Reader, is that right? Are you able to open the dialog, adjust all settings - you see a preview table that somehow makes sense - and OK the dialog?
Only when you execute the file reader this out of mem error occures, right?
Thanks,

  • Peter.

yes you are right.
I can configure the file, and if i push the “quick” option i can preview the file and OK the dialog.
i havent tried the processing version though.
i might have found a GUI bug somewhere when you load up the list of features somewhere, but that is beside the point.

update: the gui bug was in another application, so please ignore what i said about it.

It is due to the huge amount of columns. It is something we need to investigate further. I’m sorry, I have no workaround or fix.

I have seen the same type of behavior with the database reader node. If the table is too large N*M (not necessarily just M) then you can even OK the dialog.

Might this be related?

i managed to reduce my data to 5000x40000 (10 times smaller), the file itself is 400mb .
but still there it reports a heap problem. although this time it manages to load it up to 77%.

G:\knime\plugins\org.eclipse.equinox.launcher_1.0.1.R33x_v20080118.jar
-clean
-Xmx3000m
-vm

i hope this info will help you somehow.

i figured that if 77% loads then i’ll just chuck 1/4 of the file and now that i have 5000x30000 it loads just fine.

buts its a fraction of what i had.