Database writter using an insane amount of memory

KNIME Analytics Platform
1

i'm monitoring the knime memory usage (using task explorer on windows, bah).

And i notice a huge memory usage when the Database Writer is running.

it try to insert (in a loop) many 500Mb CSV file and this memory usage go from ~1GB to ~2GB when this node is active.

It hang and i have the following error :

ERROR     Loop End     Execute failed: Found duplicate row ID "Row0" (at unknown position)
ERROR     Database Writer     Execute failed: GC overhead limit exceeded

 

What's happening and is there a workaround please ?

 

my knime.ini :

-startup
plugins/org.eclipse.equinox.launcher_1.1.0.v20100507.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.1.R36x_v20100810
-vmargs
-Xmx2048m
-XX:MaxPermSize=1024m
-server
-Dsun.java2d.d3d=false
-Dosgi.classloader.lock=classname
-XX:+UnlockDiagnosticVMOptions
-XX:+UnsyncloadClass
-Dknime.enable.fastload=true
-Dfile.encoding=UTF-8
-XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode
-XX:SurvivorRatio=16

 

EDIT : I solved the loop-end error by using "Uniquify row IDs", and i changed my worflow, i'll see if it works now :)

2

1) i solved the row id problem

2) i changed the workflow so the database writter isn't in the loop anymore.

 

it's inserting now 43 millions lines * 9 columns.

using 446MB of ram right now ... it's seems to work.

(it's going to take a lot of time...)

3

And now a very good idea and a big thank you \o/

Thank you for doing this :

WARN      DatabaseWriterConnection     Error in row #25112117: Row430526#15, Disk is full writing '.\knime_hitv2js\hit_2011_04.MYD' (Errcode: 28). Waiting for someone to free space... (Expect up to 60 secs delay for server to continue after freeing disk space)
 

instead of canceling the execution

 

it saved me some hours !! (i misplaced the mysql db directory in :(( )

4