Please tell me how to set up distributed execution of Knime tasks on Apache Spark. I would like to implement automatic task management and distribution of tasks across the nodes of the home LAN. I would like to do without the knime server and other paid servers. As I understand it, the required can be implemented via Apache Spark Standalone. But how? My node is two ordinary computers. I deployed Knime and Spark on both Windows computers, I configured master and slave. The Spark web interface perfectly sees the worker on the slave. I’m running a workflow with Spark nodes. They work. But there is no distribution of tasks to the slave. What do I need to do to make this happen?
Hi @R72, have you looked into adding Livy to a standalone spark cluster?
Also, you can read more about how to set up Livy with Spark here, and about how to connect KNIME to Livy to submit Spark jobs here.
1 Like
Thank you for your reply. I have Knime running under Windows, all the instructions for your link and a lot of links to the Internet are made for Linux. The second point concerning Livy, including, at least what I see from the links, all this requires working with products under a paid license. My project is still non-commercial and more for self-development and self-education, so it is still economically unjustified for me to use paid licenses.
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.