our customer executes workflows in parallel , which results the server filling up the temp storage, which size is about 500 GBytes.
First he could execute these workflows one after another which would lead to smaller demand of resources. But the main question is why does knime server not free the resources after executing, or is it so only that it does not free them when reaching the limit.
After restarting the knime-executor service the temp storage was freed and knime executed test workflows successfully.
A solution would probably be the scheduling of workflows in a task-list. The task-list would be scheduled only , not the individual workflows and only if the previous workflow hat stopped and freed his resources the next one would start. How could this be done?
In general the used resources (utilized heap space as well as temporary files) of a workflow job should be released/freed during the swapping process where the Job is transferred from the executor’s memory back to the KNIME Server job repository.
In case you experience com.knime.enterprise.server.jobs.WorkflowJobManagerImpl.swapOutJob Failed in the KNIME Server logs you should raise the timeout parameter for this to a high amount, either via the Webportal Administration page or via editing the knime-server.config file: com.knime.server.job.default_swap_timeout=60m
It could also help to set the scheduled job to be discarded immediately after succeeding which would release the used resources instantly after the job has finished. There is also an option to start a scheduled job only if the previous job (of the same workflow) already finished.
If you monitor your data processing including timestamps externally, e.g. in a database, you can use a small workflow to check the source against new data and only starts the processing if newer data exists.
Regarding your idea of processing a task list I know of some customers already using such a processing while I haven’t found public resources on the workflow design. Maybe this forum topic might help you although it is not covering exactly your use case: workflow orchestration process
This workflow on the KNIME Hub is pretty old but implements the same logic you want to use: Fault tolerant Workflow orchestration – KNIME Community Hub
This Errormessage occured multiple times:
15-Aug-2023 23:50:05.543 WARNING [KNIME-Job-Lifecycle-Handler_1] com.knime.enterprise.server.jobs.WorkflowJobManagerImpl.swapOutJob Failed to swap job ‘theWorkflow’ to disc, keeping it in memory: Waiting time has elapsed before state change occurred
I will apply the mentioned timeout-setting and observe , if the problem still exits.
Further action will only be needed , if the problem still persists.