Java Heap Space Error

Node Development
1

Hi,

I'm currently creating a few KNIME nodes with the GenericKnimeNodes (https://github.com/genericworkflownodes/GenericKnimeNodes) functionality. So far, all was working well, but now one node fails with a "Java Heap Space Error" during execution.

The node's functionality is pretty simple: It reads a .sam/.bam file (the output of another node which sorts the reads in the file first), processes it and writes a .csv file. If I execute the program which I want to turn into a node on its own with a small test dataset (15 MB sam file), it doesn't use more than 20MB RAM and the output file also is just about 1.5 MB or so.

I don't use any KNIME tables to pass data around but only file input/output ports (if that is what you call them, I'm not entirely sure about the terminology). I guess I could try to increase the Heap Space, but I can't imagine that's actually the issue here since the external payload that gets executed runs in C++ and as I mentioned, does not use much memory. Changing the memory policy also does not help (of course, I don't use any KNIME tables that might be stored in memory). When I manually execute the program (the C++ payload) on the data of the previous step that KNIME stores in /tmp everything also executes perfectly. It just does not work in Node form.

Is there anything I have to consider or might have forgotten?

Thanks for the help.

2

Leaving my resolution of the problem for posterity:

The payload wrote some debugging information to stdout and this can result in many lines being written. I didn't think of it as a potential problem source but apparently it overwhelms KNIME (which stores the output in memory, it seems) and leads to the Heap Space error.

So try not to write anything but constant length output to stdout I suppose.

3

The KNIME framework doesn't handle the output of external programs at all. It's your node that may read in the output of the external program and store it in memory. If the node is auto-generated by GKN then it may also be a general problem in GKN. You should ask the developers in this case.

4

Using Windows 10, with 16 GB and AMD Phenom II 2.80 GHz

Dynamic heap shows only up to 504M, sometimes reaching the limit over the redline

I have changed the node Memory policy to save to disc.

Even after setting the following in Knime.ini, Simple Reg Tree Learner is giving the Java Heap Error.

-startup
plugins/org.eclipse.equinox.launcher_1.3.100.v20150511-1540.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.300.v20150602-1417
--launcher.defaultAction
openFile
-vmargs
--- -Xmx8064m
-Xmx8G
-XX:MaxPermSize=256m