JAR & LIBRARY KNIME ON SPARK

[And]

Hi, I have a question, so far i have used the big data envoirment node with spark,hive,ecc in my local jvm, and it works.

Now I have to use knime connected to spark on Hadoop, I have read that I have to load the libraries on the cluster that I intend to use for the java on spark nodes, but using the Spark remote server jvm, I have to load also the jar package to run knime?
If yes, where do I get the jar package or anyway the libraries used by knime?

thanks.

[dora_gcs]

Hello @And ,

To be honest, I don’t fully understand the described problem, but I found these guides that can help you. Have you already seen them?

https://docs.knime.com/latest/bigdata_extensions_admin_guide/index.html#_overview

https://docs.knime.com/latest/bigdata_extensions_user_guide/index.html#_overview

Best regards,
Dora

[sascha.wolke]

Hi @And,

KNIME uploads the required jar packages to your cluster to run the Spark nodes. You don’t have to deal with any jar files in that case.

If you like to use some special jar in the Spark Dataframe Snippet, then this jar has to be present on both sides, the KNIME AP and the cluster.

More details can be found in documentation, see the links posted by Dora above.

Cheers
Sascha