I have a workflow which receive an input and provide the same input as output and a wait node in between. Now I created a job out of this workflow and executed with an input, for e.g., a=10. While this job was executing, I call the same job again with input a=20.
URL used in the POST Request node to create the job: https://knime.server.com/knime/rest/v4/repository/Test/simple_restapi:jobs
URL used in the POST Request node to call the job: https://knime.server.com/knime/rest/v4/jobs/<jobid>?&reset=true
Request body:
{
"json-input-1": {
"a": "10"
}
}
Both POST Request nodes finished the execution at the same time and provided the same result; a=10.
Is this the expected behaviour?
I expected the the second job call will wait until the first one to finish and execute the job again with the second input. Is it possible to achieve this?
As I understand it, you called a workflow to start a job, the wait node keeps it executing, and you then called the job to reset it with a different input.
However, a job will not stop executing and re-run with different inputs if it is still running.
You may wish to examine Job Pools [1] to see if they are a fit for your use-case.
Thank you for the reply
My expectation is, when a job is executing while called again, the latest request will wait, the job finishes it’s current execution, and execute again with the new request. In general I don’t want this workflow executing in parallel (i.e. one job at a time for this workflow). Since I cannot control the trigger to execute this workflow, I want KNIME server to handle this situation. And I don’t think I can achieve this with job pooling.
Usecase: The workflow read a file, process data, and write output into the same file. To make the operations deterministic, there shouldn’t be no parallel job executions. Parallel jobs create inconsistent state (non-deteministic state) of the file. I have to make sure that current job read the file only after the write operation of the previous job.
Is it possible with KNIME server, without using any external components such as Redis?
Another solution would be locking the file.
Is there a file lock mechanism in KNIME, and job waits for the file to be unloacked if it is already in lock state?
How often du you need to run this sequence? If the number of times is somewhat manageable, I would build a KNIME workflow that calls the job via a sequence of REST nodes, hitting the /rest/v4/jobs/{uuid} in async mode. That way, you can ensure that each invocation of the job only starts after the previous call has returned.
The call to the workflow comes from a external entity which we don’t have access. The will happen every 7 seconds and 24x7 but the workflow will take ~ 28 seconds to finish one execution. So it will leads to parallel executions. So the sequential calls to the job will not work. If knime could queues the calls to the same job and execute sequentially that would be a great solution.
Got it. From a more practical perspective, if you have jobs coming in every 7 seconds, and they take 28 sec to execute, and then you want to queue them - wouldn’t this queue grow to a massive size very quickly?
There will be 4 jobs running parallel. At the end of first 28 sec there will be 4 (or 5) jobs running parallel and first job will be finished. From that point onwards for each incoming job, there will be one job finished and discarded. . The parent workflow takes 28 secs, but the job I want to call only 1 or 2 secs. So if 4 parallel workflows are running and all of them call same the job, even if they are queued, there will not be more than 5 jobs in parallel.
