EMR Spark Connection Error (Livy)

KNIME Extensions
,
1

Hello,

I am trying to connect Knime Desktop to my EMR Spark cluster, but I am constantly getting the following error when I am trying to create the spark context.

Execute failed: Failed to create Livy session. Please consult the Livy server log. (Exception)

Here is my Spark cluster info,
Release label: emr-5.9.0
Applications: Hue 4.0.1, Spark 2.2.0, Zeppelin 0.7.2, Livy 0.4.0

The livy server logs are also attached to this topic.
livy_logs.log (7.8 KB)

Please advice.

2

Hi @sguleria,
welcome to the community.
Could you also take a look into the log files of the failing application?
In this case: application_1557152545038_0039
Do you use the S3 Connector for the Remote File System in the Livy node?

best regards Mareike

3

Yes, I am using S3 Connector for the FileSystem. I was able to connect to the EMR cluster yesterday upon starting a new EMR cluster, but once the connection terminated I was not able to connect again.

I will try to get the logs of failing application.

4

@sguleria
could you solve the issue?

5

Not exactly but we have a temporary solution which is working. It seems like the Knime Spark Livy node is able to connect to a fresh instance of EMR but once the connection is terminated the node no longer connects to the same EMR cluster.

6

Hey @sguleria,
that should not be the case. It think it would be worth to do some more debugging on this. What do you mean by the connection is terminated do you reset the node? Do you restart the Livy Server?
Where you able to look into an application log, for a failing attempt?

7

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.