I have some workflows running in the batch mode every 5,10,30 minutes.
Sometimes, they get stuck and when I open it from Knime, it says workflow is already in use. I try to delete the .knimeLock file from the workflow directory but it says the file is already in use. I have the solution for that is that I delete the process which uses that file by using an application called process explorer but that is not optimal at all.
Can someone please guide me about this?
My Knime version is 4.3.1
Hi @MajidAbbasi , could it be that your workflow takes a long time to run, and so long that it’s still running when the next scheduled execution of the workflow kicks in?
And just to clarify what you mean by “every 5,10,30 minutes”, do you mean at the 5 minute mark, 10 minute mark and 30 minute mark, as in *:05, *:10, *:30? or do you literally mean it runs at every 5, 10 and 30 minutes, which would mean a combination of:
1 Every 5 minutes: *:00, *:05, *:10, *:15, …, *:30, …, *:55
2 Every 10 minutes: *:00, *:10, *:20, …, *:50
3 Every 30 minutes: *:00, *:30
As you can see, if this is the case, Every 30 minutes is a subset of both Every 10 minutes and Every 5 minutes, which basically means both job will try to run the same workflow at the same time, and Every 10 minutes is a subset of Every 5 minutes, which again will be in conflict. So hopefully this is NOT what you meant.
Even if you meant my former option (runs at the 5 minute mark, 10 minute mark and 30 minute mark, as in *:05, *:10, *:30), does the workflow take more than 5 minutes to run?
Not only it’s not optimal, hopefully this does not corrupt your workflow - FYI, you can also kill the process via the Task Manager which is native to Windows. Or you can also do this via the command line (command “tasklist” will give you the list of tasks that are running, along with their PID, and then use “taskkill” to kill the process). The Process Explorer simply gives you a UI. But you should not have to kill processes to run a workflow.
First of all, you must design your workflows the way that they have the least chance of failure and by failure I mean workflow execution failure. I know that issues and errors may occur but you need to handle any possible errors with Error Handling nodes so your workflow finishes successfully even if some error occurs in the middle of an execution. This way you ensure that the workflow would finish even if something wrong happens. You can include some nodes or components to output the workflow status in a file or in your DB. With error handling nodes you have the error message at the end of the workflow and you can write it somewhere and check in case.
Now about the “get stuck” issue. Of course, you can not have a workflow be initiated more than once at a time when you are NOT using KNIME Server. In KNIME Server you have the option of executing several jobs of the same workflow at the same time.
But if you still want to handle this issue in KNIME AP, you have to make copies of your workflows and then schedule the copies instead of the original ones. This way you always have access to the original workflow even if its copy is still running. Also, if there is a chance of an overlap between executions of a workflow and you are OK with that, you can handle it by scheduling several copies of the same workflow.
I hope I could help you here. If you have more questions I would be more than happy to answer.