Occasionaly I get an error message along the lines of "no chance to free up extra memory, may cause an endless loop".
Is there a way of terminating the workflow run when this occurs rather than having to forcefully exit KNIME?
Thanks for reporting the problem. It's an issue we will address in 2.8.
Joining data is a memory expensive operation and the current implementation uses a memory observer that triggers some cleanup when the memory consumption is above a certain threshold. That trigger only affects the data kept by the Joiner itself but not any other tables in the workflow -- which is a problem if the rest of the nodes already consume a lot of memory. In 2.8 we'll have a workflow global trigger that cleans up all tables in all nodes to release memory, which then leaves more resources to the Joiner. (One bug targeted for 2.8 is on the exact same behavior you describe, including a workflow to reproduce the problem ... and that has been fixed).
Current workaround is either to give more heap to KNIME (modify the knime.ini and increase the -Xmx1024M line to -Xmx4G?) or to make sure the rest of the nodes in the workflow are memory friendly, e.g. make sure they write their output to disc.
Hope that helps!
Nice to hear that it has been looked at already. I've already got a 5G heap space but hadn't set any of the memory management in the other nodes in the workflow. I was being particularly greedy with the work I was trying to carry out but the error has cropped up on other occasions.
Thank you for the response
Sorry to resurrect an old thread, but I wanted to check if this is still a known bug.
I'm seeing the error in KNIME 2.10, specifically in joiner nodes.
We changed the memory handling of the framework in 2.8 (July last year). In case the system runs short on memory it swaps all tables to disc. You should see messages like this in your log file before it hits the memory limits:
DEBUG Service Thread MemoryObjectTracker Low memory encountered. Used memory: 2 GB; maximum memory: 2 GB.
DEBUG KNIME-Memory-Cleaner MemoryObjectTracker Trying to release 3 memory objects
DEBUG KNIME-Memory-Cleaner Buffer Wrote 5400 rows in order to free memory
DEBUG KNIME-Memory-Cleaner Buffer Buffer file (/tmp/knime_Workflow Manage60515/knime_container_20140731_7573434359227819524.bin.gz) is 250.33MB in size
DEBUG KNIME-Memory-Cleaner MemoryObjectTracker Removing org.knime.core.data.container.Buffer$BufferMemoryReleasable (1 remaining)
DEBUG KNIME-Memory-Cleaner Buffer Wrote 4 rows in order to free memory
DEBUG KNIME-Memory-Cleaner Buffer Buffer file (/tmp/knime_Workflow Manage60515/knime_container_20140731_2371751231334993760.bin.gz) is 0MB in size
DEBUG KNIME-Memory-Cleaner MemoryObjectTracker Removing org.knime.core.data.container.Buffer$BufferMemoryReleasable (2 remaining)
Regarding the Joiner: There were no memory-related changes in the code. It's still an resource-intensive operation in particular if you have large matching chunks (say a cross join). If you can send me some data I'm happy to take a look.