Using the Knime exporer, I see that several jobs on my server are in the running state (triple green arrows). But none of them are actually doing anything. If I understand things correctly, this is due to one workflow that is still running, which basically has all the others standing in line, waiting for the executor to finish the current job.
My server setup is basically used as a batch executor: I have a dozen or so scheduled jobs, some of which run every hour. Unfortunately, one of the bigger jobs is either hanging, or waiting for some external query to finish. The server on the other hand shows 99% idle.
So I would like to have the Knime server run more things in parallel, for the same user. The manual only mentions that you can configure it to have one executor per user, instead of one global one.
But how do I configure it for multiple global executors?
And secondly: how do I figure out which workflow is taking so long?
The server doesn't have a limit for the number of parallel jobs (other than hardware resources).
If you enable per-workflow log files in the preferences you can easily inspect the message of workflows which are supposed to execute but don't show any progress. Just cancel the job, save it as a workflow and then download it to your local workspace.
After some investigating and mails with Knime support, here are a few take-aways that might be good to know for others as well:
- (as stated by thor) workflows indeed run in parallel, but ...
- database connections that use the same connectionstring (driver+server+user) are shared. This effectively blocks one workflow whilst another is doing any work on the connection.
- there is a knime.ini setting that allows concurrent processes within the same workflow to use multiple connections with the same user. Add the following line to knime.ini to enable:
- db connection pooling (the way most app servers handle this issue) is not supported yet.
(No clue of what 'yet' means, perhaps someone from Knime can tell us more)
So the trick, if you want to make sure one workflow does not block another while it's doing something on the db, is to have different db users for each workflow.