heap space problem

KNIME Development
#1

I am writing a new module that uses the column rearranger.
Specifically, from my execute() method in the nodeModel I call createColumnRearranger
that calls a CellFactory method, which in turn provides a getCells method.
this getCells method is then called from RearrangeColumnsTable. I modified one of the database node for this if I remember correctly.
Now I am always running into heap space problems. Changing (reducing) the heap space still brings up the problem (originally I was thinking this has something to do with my underlying data).

I don’t see where there are allocation problems and trying to profile the application using Profiling tools proves to be very difficult and painful.

Do you have any ideas where I could start looking? Or, could you explain to me a bit more how the column rearranger works?

Thanks for any hints and suggestions.

Best,

Bernd

#2

I know that these heap space problem can come from anywhere but the latest debugging result is the following:
Exception in thread “KNIME Progress Timer” java.lang.OutOfMemoryError: Java heap space
But I guess this doesn’t mean that org.knime.core.node.DefaultNodeProgressMonitor$1.run is the one with the memory leak…

#3

One more thing: I now suspend the worker that I wrote and still get the heap error…
B

#4

The OOME in the Progress Time might only be a side effect. I don’t quite understand how the column rearranger, the worker that you created and the database nodes get together. Can you clarify?

Apart from that, two suggestions:

  • You could set the "memory policy" of the node causing the trouble to "Write to disc" to ensure that the ColumnRearranger doesn't keep anything in memory.
  • (More on the hard-core end:) You could get used to one of the memory profilers that are publicly available and analyse the heap of your VM to find the bottle neck. I found jhat/jmap to be very useful.

Hope it helps,
Bernd

#5

Thanks a lot Bernd! the jhat/jmap pointed me to the mysql connections. And it seems that closing the Statement and ResultSet using st.close() and rs.close() is a good idea…

This is for a node that connects to a specific MySQL instance and executes requests one per line…

Thx.

Bernd