This topic somewhat rings a bell.
I had some nasty issues with leaks in the “GET Request” node a while ago (they are fixed by now, but back then simply running the GC would not have helped, as there were references which just did not get cleaned up).
I’ve also observed with other nodes that sometimes the memory consumption with long-running workflows steadily grows without being released, but I didn’t have time or energy to investigate this further.
This old thread here suggests that there might an issue with the Java Snippet node – are you probably using this node? Did you check what kind of objects consume the heap (it’s in one of VisualVM’s tabs).
Just my “you’re not alone”, without being able to present a solution. Sorry 
– Philipp