Thanks for the information, that is the missing piece in the jigsaw for me.
I’ve been suffering a lot with KNIME 4.0, in particular problems with heap allocations and excessive amounts of garbage collection as the heap fills up resulting in KNIME slowing to a crawl.
This tends to happen when the tenured regions in G1GC exceed the -XX:InitiatingHeapOccupancyPercent (IHOP) which by default is 45% of the total heap size. Once the tenured region reaches this amount then garbage collection becomes continuous and KNIME effectively freezes.
Data ends up in the tenured regions either because it has passed through Eden and Survivor, or quite fequently because it is a humongous allocation which bypasses Eden and Survivor and goes straight to tenured.
As the tenured region fills up to 45% the garbage collector is running continuously and, because Eden/Survivor doesn’t get to fill the remaining 55%, the heap doesn’t hit 90% occupancy and the k most recently cached tables are not released.
I’ve managed to ameliorate some of this isse by setting IHOP to 75%, which results in Eden/Survivor space startvation, however, I am still not hitting the 90% threshold for releasing the cached tables.
If most of the data is cycling through Eden/Survivor and not getting to Tenured then I am guessing that the heap does hit 90% occupancy and flushes the tables.
You may want to consider flushing the tables against the size of the tenured space. As tenured space will start driving garbage collection as it hits IHOP you may want to trigger a cache release when tenured hits 90% of the IHOP target, rather than the total heap utilisation hits 90%. As cached data is long lasting I would expect the cached data would be in tenured heap space anyway.
I could have misunderstood everything, but the new G1GC garbage collection and caching strategy is not working.