I noticed one of my workflow generated ~50G temp files in the knime server/workflow_repository/jobs/xxx/flowContextTmp folder while executing in knime server, I listed several largest files below:
This WF is processing ~5000*~20 level of data each time, so it’s not a big data. from the temp file name looks like they’re from duplicated filtering somewhere. I’m checking the whole workflow, haven’t found where a node forms huge dumps.
Appreciate any ideas on:
experience on what process may generate such huge temp files.
is there a way to automatically clear this “jobs” folder in knime?
I’ve checked the “discard workflow job after execution”, but looks like the folders were not deleted right after the execution (may stay for days). I manually deleted the folders previously so cannot confirm now, will keep monitoring and update.
thanks Thor
One of my complex workflow is to read SQLite dataset which is 20G in size, the executor (running on knime server under linux ) gives exception since it needs 1Tera Temp folder space, meanwhile I have only 53G available for that folder.
The following exception triggered when executing “DB Reader” node.
ERROR KNIME-BackgroundTableWriter-1 Buffer Writing of table to file at nod e DB Reader 3:283:11 at workflow CIC-IoT-Analysis_v2.0_a 3 encountered error: IO Exception: The partition of the temp file “/tmp/knime_CIC_IoT_Analysi_12494/knim e_container_20240801_1482821592515136867.bin.snappy” is too low on disc space (5 3472MB available but at least 1048576MB are required). You can tweak the limit by changing the “org.knime.container.minspace.temp” java property.
