Hi everyone,
I have a question about KNIME Server queue system. For many requests via REST at the same time, the response time of the workflow increases considerably because KNIME Server puts requests in a queue. I think it should be a structure that can run more than one request at the same time. If my hypothesis is correct, what should I do at this point? Installing RabbitMQ can be a solution? If yes, how can i install RabbitMQ to server? Is there another solution for this issue, can someone explain?
Any help would be really appreciated,
Best,
Hi @KKERROXXX ,
more than often in these scenarios not the message queue, but the executor becomes a bottleneck. Too many concurrent executions might utilize all of the assigned memory and threads. For these scenarios, adding additional executors might help, such that concurrent requests for executions can be distributed over several executors. For such distributed setups RabbitMQ should be used. So yes, you might consider switching to RabbitMQ, but the reasoning behind this is not to have a more potent message broker, but the ability to have several executors at the same time.
The solution described above works well for scenarios where several, computational expensive workflows need to be executed concurrently. In another scenario, where you have rather lightweight workflows which are requested on highest frequency and where low latency is crucial, we will soon have a new offering called KNIME Edge. It was introduced during this years spring summit. If this sounds interesting, you can have a sneak peak here: https://youtu.be/HxLtTznvDTQ?t=443
1 Like
Thanks for the answer @Marten_Pfannenschmidt !,
How can I install the RabbitMQ?
One last thing, I wonder that Is RabbitMQ required a specific KNIME Server version? I am using 4.10.3 KNIME Server version.
Best,
Preface: Distributed setups require a Server Large license.
Installation instructions and required versions you can find in the docs. Here is a link to the 4.10 docs: KNIME Server Administration Guide