Create Spark Context(Livy) in Azure

[filum]

’m trying to configure KNIME in Azure to run jobs on my HDInsight Cluster using the Create Spark Context(Livy) -Node wutj a WebHDFS-Connection

From the KNIME Livy node I get the message:

ERROR Create Spark Context (Livy) 0:4 Execute failed: File /tmp/.knime-spark-staging-b09b868f-0a6f-4742-ad37-b7a6752e238e/ef9e9ea2-3e50-4bf2-b53f-9b3376b2dbb5 not found. (RemoteException)

I’m using KNIME 3.7 on a Windows 2019 Server and Spark 2.3 (HDI 3.6) on Linux, all in Azure.

Can anybody help me?

Cheers
filum

[filum]

Hello,

can anybody tell me what the status of the Livy-Node is? It would be an elegant way to connect KNIME with an existing Spark cluster. The currently available ‘Spark Job Server for KNIME® Spark Executor’ on Azure requires the old Spark version 1.6, current version of Spark is 2.4.

filum

[mareike.hoeger]

Hi @filum
the Livy node needs to create a staging folder in the cluster so KNIME can exchange files. We actually recommend to use the Azure Blob Store connector with the node if used on HDInsight. Could you try this? Note that you have to set a staging area in this case.

best regards Mareike

[filum]

Hi Mareike,

it works. Thx for your support.

br filum