TIL: heap size gotcha above 32 GB

[beginner]

So today I just learned that due to how the JVM work, if you go to 32gb or higher heap space, then the way object pointer work changes to pure 64-bit pointers which need more memory.

This means if you go from say 30gb to 32gb heap you now have LESS available memory. And in fact if you go to 32GB you need to go up to 48GB heap to offset this pointer cost. So any setting between 32 and 48 GB of heap space is basically useless.

This also lead me to this comment about using off-heap memory I wonder if this is planned or even possible to store the result tables in memory mapped files off-heap which would also reduce stress and issues with GC?

[marc-bux]

Hi @beginner,

Thanks for bringing this up. I was not aware of the steep increase in “pointer cost” when going beyond 32 GB heap space. Always something new to learn in these forums

Yes, storing least-recently-used KNIME tables off-heap is something we are actively discussing. I agree that there is likely a lot to be gained from.

I’ll keep this post updated if there are any updates on this from our end.

Regards,

Marc