ERROR Create Spark Context (Livy) 0:65 Execute failed: Connection refused (Connection refused) (ConnectException)

[erbert_merlo]

Hi! I’m trying to connect the create spark contest (Livy) node to the Amazon S3 connection node but the execution fails ( ERROR Create Spark Context (Livy) 0:65 Execute failed: Connection refused (Connection refused) (ConnectException) ).
The Amazon S3 connection node is well configured and the connection test works. The create spark contest (Livy) node is configured with the staging area in a buket configured with access: buket and non-public objects and region: Frankfurt (but I also tried with Ohio).

please don’t slap me if I made some stupid mistake but I am new to knime and S3

[mareike.hoeger]

Hi @erbert_merlo,
first of all: Welcome to the KNIME community! Rest assured we do not slap anybody here.

Could you give me some more details about your setup? Are you running an AWS EMR Cluster?
Could you take a look into the KNIME log (You can do that with “View” > “Open KNIME log”) and provide me the stacktrace of the error?

best regards Mareike

[erbert_merlo]

Thanks for your answer Mareike!

2019-10-11 16:44:43,922 : ERROR : KNIME-Worker-9 : : Node : Create Spark Context (Livy) : 0:65 : Execute failed: Connection refused (Connection refused) (ConnectException)

java.util.concurrent.ExecutionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8998 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)

at org.apache.livy.client.common.AbstractHandle.get(AbstractHandle.java:198)

at org.apache.livy.client.common.AbstractHandle.get(AbstractHandle.java:221)

at org.knime.bigdata.spark.core.livy.context.LivySparkContext.waitForFuture(LivySparkContext.java:374)

at org.knime.bigdata.spark.core.livy.context.LivySparkContext.createRemoteSparkContext(LivySparkContext.java:355)

at org.knime.bigdata.spark.core.livy.context.LivySparkContext.open(LivySparkContext.java:265)

at org.knime.bigdata.spark.core.context.SparkContext.ensureOpened(SparkContext.java:144)

at org.knime.bigdata.spark.core.livy.node.create.LivySparkContextCreatorNodeModel.executeInternal(LivySparkContextCreatorNodeModel.java:125)

at org.knime.bigdata.spark.core.node.SparkNodeModel.execute(SparkNodeModel.java:240)

at org.knime.core.node.NodeModel.executeModel(NodeModel.java:567)

at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1192)

at org.knime.core.node.Node.execute(Node.java:979)

at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:559)

at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)

at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)

at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)

at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)

at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)

at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)

Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8998 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)

at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159)

at org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:325)

at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)

at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)

at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)

at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)

at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)

at org.apache.livy.client.http.LivyConnection.executeRequest(LivyConnection.java:276)

at org.apache.livy.client.http.LivyConnection.sendRequest(LivyConnection.java:269)

at org.apache.livy.client.http.LivyConnection.sendJSONRequest(LivyConnection.java:238)

at org.apache.livy.client.http.LivyConnection.post(LivyConnection.java:213)

at org.apache.livy.client.http.CreateSessionHandleImpl.performStart(CreateSessionHandleImpl.java:144)

at org.apache.livy.client.common.AbstractHandle$5.run(AbstractHandle.java:267)

at org.apache.livy.client.common.AbstractHandle$4.run(AbstractHandle.java:231)

at org.apache.livy.client.common.AbstractHandle$2.call(AbstractHandle.java:156)

at org.apache.livy.client.common.AbstractHandle$2.call(AbstractHandle.java:153)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.net.ConnectException: Connection refused (Connection refused)

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)

at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)

… 23 more

[mareike.hoeger]

Hey @erbert_merlo,
looks like you have still localhost configured in the node. You have to set the hostname of the server that runs your Livy service in the configuration (That is most likely not localhost).
If you do not have such a server and want to try the BigData KNIME nodes on your local machine, I recommend you to use the “Create Local Big Data Environment” node.

best regards
Mareike