Create Spark Context (Livy) - Knime ending session direct after creating it

bigdata
#1

Hi,

We running Knime 4.0.1 and trying to connect to our Spark cluster that runs Spark 2.3 and Livy 0.5.

We have created a small workflow that is a HttpFS Connection node connected to a Create Spark Context (Livy) node. The user in the HttpFS Connection node is Livy and the connection is ok when we test it. When we execute the Create Spark Context (Livy) node it fails after a while and we get the error: “ERROR Create Spark Context (Livy) 3:6 Execute failed: Staging area access from inside Spark failed: nas:/user/livy/.knime-spark-staging-b3c20e38-b7f9-4716-9ca1-90a51dcef127/310fd832-f89c-4b40-b5b9-422b4ef2033c”

We inspected what is happening on the Spark cluster side and it shows that the session was correct started and directly ended by Knime. No faults reported.

When we run this code in a Python Script node it creates a session that stays stable and goes to idle after a while until we deletes it (the input table has the url and port to livy):

import json, pprint, requests, textwrap

host = input_table[‘column1’][0]
data = {‘kind’: ‘spark’}
headers = {‘Content-Type’: ‘application/json’,‘X-Requested-By’:‘livy’}
r = requests.post(host + ‘/sessions’, data=json.dumps(data), headers=headers)
print(r.text)
d = json.loads(r.text)

Do you have any suggestions on what can be wrong?
We tried different staging areas like nas:/user/livy/ without any success and we can’t find any problem with access rights for the livy-user in the cluster.

0 Likes

#2

Hi @Max74,
could you give me some further Infos to your setup? Are you running your cluster on a NAS or on HDFS? Are you able to access the provided staging area from within a spark job (e.g. save a file in it)?

best regards Mareike

0 Likes

#3

Hi,
thank you for the response.
It´s a hybrid solution from Huawei with a HDFS ontop of a NAS.
From pure Python we can run Spark job via Livy without any problem.

Best regards
Max

0 Likes

#4

Hey Max,
could you please open the KNIME.log (You can do that via View>Open KNIME Log)
This file should contain the full Stacktrace to the error, it could be helpful if you could provide this.
The Error indicates, that the spark job is not able to read or write the test files we uploaded into the file system.
Could you please check the Livy Web UI and see who is the owner of the application that is created by the LivyContext node?

best regards,
Mareike

0 Likes