Concatenate problem

KNIME Analytics Platform
1

Hi,

I have three tables each with about 20,000,000 rows having mostly the same row_keys. Now, when I use Concatenate (Optional in) I get a Heap-space error or the JVM crashes...

These are JVM options I am using (I believe they are rather genourous)

-Dknime.expert.mode=true 
-XX:PermSize=128m
-XX:MaxPermSize=1G
-Xmx16G
-Dknime.tmpdir=/pasteur/solexa2/solexa_travail/PF2/tmp
-Dknime.logfile.maxsize=10000000
-Dorg.knime.container.cellsinmemory=1000000

 

Do you have any suggestion on how to get this working? In the future I am expecting to have more like 100M rows per table...

(My solution would be to make the checking for uniq keys optional...)

 

Thanks,

 

Bernd

2

In KNIME v2.3 we have a "Fail on duplicate" option in the Concatenate node, which does not get all row IDs into memory. In KNIME v2.2.2 (I guess that's the version that you have problems with) you only have the choice of skipping duplicate rows or appending a suffix to each subsequent occurrence -- both options need all IDs in a memory-expensive hash.