Memory problem within Distance Matrix Calculation node

KNIME Analytics Platform
1

Dear list members,

First of all, I'm a newbee of the KNIME package, but I already have a problem which I can't solve by reading and google-ing... :-(

I run into a memory problem when I try to cluster (fingerprint) my 180.000 ligands. The error I get is

ERROR     Distance Matrix Calculate     Execute failed: GC overhead limit exceeded

I already used "Write tables to disk" within the Memory Policy menu. No changes.

Any hints how to change the limits are highly appreciated!! Or even a hint that it is not possible to use such a large number of ligands...

I'm using KNIME 2.3.3, OS openSUSE 11.4 x86_64, Java is java-1_6_0-openjdk-1.6.0.0_b20.1.9.7-1.2.1.x86_64

Regards,

Michael

2

Running into a similar issue on my end.  I have created fingerprints of 200K molecules and I am running a distance matrix calculation on these as well.  In my case I chose to 'write tables to disk' as the memory policy, problem is KNIME wrote to 120GB of disk space.  I understand that each fingerprint is taking up 512 bytes, and I am calculating 200K rows by 300+ columns but I dont think that would result in 120 GB file.  

One question I have, is it possible to change the location of where KNIME writes its temp files as I could mount a larger drive and brute force this through?

Thanks for any help,

3

Well, the distance matrix for 200,000 molecules has 200,000*200,000/2 entries. Given that each entry is a double value with 8 Bytes this gives 149GB. The compression brings it down to 120GB. You can change the temp location in the KNIME preferences (File->Preferences->KNIME).

4

Thank you for the response Thor, yup you are right about the size I did not multiply by 8 bytes per double.  The ability to change the write location of the temp files is key, thank you so much I was getting a bit frustrated!