I have a question regarding resource management on KNIME Hub.
We are currently running ML-related workflows on a KNIME Hub with 8 CPU cores available. When a workflow is executed, it quickly consumes all 8 cores.
Because of this, subsequent jobs are queued and cannot start until the current job finishes, which makes it difficult to run multiple workflows concurrently.
My question is:
Is there a way to limit the number of CPU cores that a single job can use, either through node-level settings or Hub-level configuration?
Any guidance on how to control or restrict core usage per job would be greatly appreciated.
On KNIME Business Hub Standard and higher tiers, you can define multiple, dedicated execution contexts and allocate specific portions of the total available execution cores to each context. For example, you may choose to reserve four cores for machine learning–related workflows and assign the remaining four cores to other workloads.
Within an execution context, each executor will utilize the maximum number of cores assigned to that context to run workflows in parallel. As a result, a single workflow execution can consume all allocated cores, while multiple concurrent workflow executions will share the available threads using a round-robin scheduling approach.
If you need to restrict parallel execution within an individual workflow, this can be achieved by using flow variable connections to introduce explicit dependencies. A node that receives a flow variable connection will wait until the source node has completed execution before it is triggered, thereby enforcing sequential execution where required.