Can I connect Knime with the node Create Spark Context to my local Spark Apache Cluster version 1.6?
I tried it but Knime returns the next error log:
2017-06-13 12:38:15,505 : DEBUG : main : ExecuteAction : : : Creating execution job for 1 node(s)...
2017-06-13 12:38:15,524 : DEBUG : main : NodeContainer : : : Create Spark Context 0:21 has new state: CONFIGURED_MARKEDFOREXEC
2017-06-13 12:38:15,547 : DEBUG : main : NodeContainer : : : Create Spark Context 0:21 has new state: CONFIGURED_QUEUED
2017-06-13 12:38:15,569 : DEBUG : main : NodeContainer : : : iberdorla 0 has new state: EXECUTING
2017-06-13 12:38:15,588 : DEBUG : KNIME-Worker-21 : WorkflowManager : Create Spark Context : 0:21 : Create Spark Context 0:21 doBeforePreExecution
2017-06-13 12:38:15,603 : DEBUG : KNIME-Worker-21 : NodeContainer : Create Spark Context : 0:21 : Create Spark Context 0:21 has new state: PREEXECUTE
2017-06-13 12:38:15,619 : DEBUG : KNIME-Worker-21 : WorkflowManager : Create Spark Context : 0:21 : Create Spark Context 0:21 doBeforeExecution
2017-06-13 12:38:15,633 : DEBUG : KNIME-Worker-21 : NodeContainer : Create Spark Context : 0:21 : Create Spark Context 0:21 has new state: EXECUTING
2017-06-13 12:38:15,645 : DEBUG : KNIME-Worker-21 : WorkflowFileStoreHandlerRepository : Create Spark Context : 0:21 : Adding handler 2e45c240-21c2-4bb0-9365-fe9205c94296 (Create Spark Context 0:21: <no directory>) - 3 in total
2017-06-13 12:38:15,658 : DEBUG : KNIME-Worker-21 : LocalNodeExecutionJob : Create Spark Context : 0:21 : Create Spark Context 0:21 Start execute
2017-06-13 12:38:15,658 : DEBUG : KNIME-WFM-Parent-Notifier : NodeContainer : : : ROOT has new state: EXECUTING
2017-06-13 12:38:15,672 : INFO : KNIME-Worker-21 : JobserverSparkContext : Create Spark Context : 0:21 : Spark context jobserver://localhost:8080/enrique changed status from CONFIGURED to CONFIGURED
2017-06-13 12:38:15,709 : DEBUG : KNIME-Worker-21 : JobserverSparkContext : Create Spark Context : 0:21 : Checking if remote context exists. Name: enrique
2017-06-13 12:38:15,779 : ERROR : KNIME-Worker-21 : GetContextsRequest : Create Spark Context : 0:21 : HTTP Status code: 200 | Response Body: <!DOCTYPE html><html> <head> <meta http-equiv="Content-type" content="text/html; charset=utf-8"/><link rel="stylesheet" href="/static/bootstrap.min.css" type="text/css"/><link rel="stylesheet" href="/static/vis.min.css" type="text/css"/><link rel="stylesheet" href="/static/webui.css" type="text/css"/><link rel="stylesheet" href="/static/timeline-view.css" type="text/css"/><script src="/static/sorttable.js"></script><script src="/static/jquery-1.11.1.min.js"></script><script src="/static/vis.min.js"></script><script src="/static/bootstrap-tooltip.js"></script><script src="/static/initialize-tooltips.js"></script><script src="/static/table.js"></script><script src="/static/additional-metrics.js"></script><script src="/static/timeline-view.js"></script> <title>Spark Master at spark://192.168.41.129:7077</title> </head> <body> <div class="container-fluid"> <div class="row-fluid"> <div class="span12"> <h3 style="vertical-align: middle; display: inline-block;"> <a style="text-decoration: none" href="/"> <img src="/static/spark-logo-77x50px-hd.png"/> <span class="version" style="margin-right: 15px;">1.6.0</span> </a> Spark Master at spark://192.168.41.129:7077 </h3> </div> </div> <div class="row-fluid"> <div class="span12"> <ul class="unstyled"> <li><strong>URL:</strong> spark://192.168.41.129:7077</li> <li> <strong>REST URL:</strong> spark://192.168.41.129:6066 <span class="rest-uri"> (cluster mode)</span> </li> <li><strong>Alive Workers:</strong> 1</li> <li><strong>Cores in use:</strong> 1 Total, 0 Used</li> <li><strong>Memory in use:</strong> 6.8 GB Total, 0.0 B Used</li> <li><strong>Applications:</strong> 0 Running, 0 Completed </li> <li><strong>Drivers:</strong> 0 Running, 0 Completed </li> <li><strong>Status:</strong> ALIVE</li> </ul> </div> </div><div class="row-fluid"> <div class="span12"> <h4> Workers </h4> <table class="table table-bordered table-condensed table-striped sortable"> <thead><th width="" class="">Worker Id</th><th width="" class="">Address</th><th width="" class="">State</th><th width="" class="">Cores</th><th width="" class="">Memory</th></thead> <tbody> <tr> <td> <a href="http://192.168.41.129:8081">worker-20170613112553-192.168.41.129-34592</a> </td> <td>192.168.41.129:34592</td> <td>ALIVE</td> <td>1 (0 Used)</td> <td sorttable_customkey="6970.0"> 6.8 GB (0.0 B Used) </td> </tr> </tbody> </table> </div> </div><div class="row-fluid"> <div class="span12"> <h4> Running Applications </h4> <table class="table table-bordered table-condensed table-striped sortable"> <thead><th width="" class="">Application ID</th><th width="" class="">Name</th><th width="" class="">Cores</th><th width="" class="">Memory per Node</th><th width="" class="">Submitted Time</th><th width="" class="">User</th><th width="" class="">State</th><th width="" class="">Duration</th></thead> <tbody> </tbody> </table> </div> </div><div> </div><div class="row-fluid"> <div class="span12"> <h4> Completed Applications </h4> <table class="table table-bordered table-condensed table-striped sortable"> <thead><th width="" class="">Application ID</th><th width="" class="">Name</th><th width="" class="">Cores</th><th width="" class="">Memory per Node</th><th width="" class="">Submitted Time</th><th width="" class="">User</th><th width="" class="">State</th><th width="" class="">Duration</th></thead> <tbody> </tbody> </table> </div> </div><div> </div> </div> </body> </html>
2017-06-13 12:38:15,965 : INFO : KNIME-Worker-21 : JobserverSparkContext : Create Spark Context : 0:21 : Spark context jobserver://localhost:8080/enrique changed status from CONFIGURED to CONFIGURED
2017-06-13 12:38:16,147 : DEBUG : KNIME-Worker-21 : Create Spark Context : Create Spark Context : 0:21 : reset
2017-06-13 12:38:16,330 : DEBUG : KNIME-Worker-21 : SparkNodeModel : Create Spark Context : 0:21 : In reset() of SparkNodeModel. Calling deleteRDDs.
2017-06-13 12:38:16,505 : ERROR : KNIME-Worker-21 : Create Spark Context : Create Spark Context : 0:21 : Execute failed: Spark Jobserver gave unexpected response (for details see View > Open KNIME log). Possible reason: Incompatible jobserver version, malconfigured Spark jobserver
2017-06-13 12:38:16,692 : DEBUG : KNIME-Worker-21 : Create Spark Context : Create Spark Context : 0:21 : Execute failed: Spark Jobserver gave unexpected response (for details see View > Open KNIME log). Possible reason: Incompatible jobserver version, malconfigured Spark jobserver
com.knime.bigdata.spark.core.exception.KNIMESparkException: Spark Jobserver gave unexpected response (for details see View > Open KNIME log). Possible reason: Incompatible jobserver version, malconfigured Spark jobserver
at com.knime.bigdata.spark.core.context.jobserver.request.AbstractJobserverRequest.createUnexpectedResponseException(AbstractJobserverRequest.java:156)
at com.knime.bigdata.spark.core.context.jobserver.request.AbstractJobserverRequest.handleGeneralFailures(AbstractJobserverRequest.java:124)
at com.knime.bigdata.spark.core.context.jobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:63)
at com.knime.bigdata.spark.core.context.jobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:1)
at com.knime.bigdata.spark.core.context.jobserver.request.AbstractJobserverRequest.send(AbstractJobserverRequest.java:73)
at com.knime.bigdata.spark.core.context.jobserver.JobserverSparkContext.remoteSparkContextExists(JobserverSparkContext.java:411)
at com.knime.bigdata.spark.core.context.jobserver.JobserverSparkContext.access$3(JobserverSparkContext.java:409)
at com.knime.bigdata.spark.core.context.jobserver.JobserverSparkContext$1.run(JobserverSparkContext.java:241)
at com.knime.bigdata.spark.core.context.jobserver.JobserverSparkContext.runWithResetOnFailure(JobserverSparkContext.java:342)
at com.knime.bigdata.spark.core.context.jobserver.JobserverSparkContext.open(JobserverSparkContext.java:231)
at com.knime.bigdata.spark.core.context.SparkContext.ensureOpened(SparkContext.java:58)
at com.knime.bigdata.spark.node.util.context.create.SparkContextCreatorNodeModel.executeInternal(SparkContextCreatorNodeModel.java:155)
at com.knime.bigdata.spark.core.node.SparkNodeModel.execute(SparkNodeModel.java:235)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:566)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1128)
at org.knime.core.node.Node.execute(Node.java:915)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:561)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
2017-06-13 12:38:16,715 : DEBUG : KNIME-Worker-21 : WorkflowManager : Create Spark Context : 0:21 : Create Spark Context 0:21 doBeforePostExecution
2017-06-13 12:38:16,737 : DEBUG : KNIME-Worker-21 : NodeContainer : Create Spark Context : 0:21 : Create Spark Context 0:21 has new state: POSTEXECUTE
2017-06-13 12:38:16,755 : DEBUG : KNIME-Worker-21 : WorkflowManager : Create Spark Context : 0:21 : Create Spark Context 0:21 doAfterExecute - failure
2017-06-13 12:38:16,773 : DEBUG : KNIME-Worker-21 : Create Spark Context : Create Spark Context : 0:21 : reset
2017-06-13 12:38:16,791 : DEBUG : KNIME-Worker-21 : SparkNodeModel : Create Spark Context : 0:21 : In reset() of SparkNodeModel. Calling deleteRDDs.
2017-06-13 12:38:16,813 : DEBUG : KNIME-Worker-21 : Create Spark Context : Create Spark Context : 0:21 : clean output ports.
2017-06-13 12:38:16,833 : DEBUG : KNIME-Worker-21 : WorkflowFileStoreHandlerRepository : Create Spark Context : 0:21 : Removing handler 2e45c240-21c2-4bb0-9365-fe9205c94296 (Create Spark Context 0:21: <no directory>) - 2 remaining
2017-06-13 12:38:16,853 : DEBUG : KNIME-Worker-21 : NodeContainer : Create Spark Context : 0:21 : Create Spark Context 0:21 has new state: IDLE
2017-06-13 12:38:16,875 : DEBUG : KNIME-Worker-21 : SparkContextCreatorNodeModel : Create Spark Context : 0:21 : Reconfiguring old context with same ID.
2017-06-13 12:38:16,892 : WARN : KNIME-Worker-21 : Create Spark Context : Create Spark Context : 0:21 : Context is the same as in the Spark preferences (see File > Preferences > KNIME > Spark)
2017-06-13 12:38:16,909 : DEBUG : KNIME-Worker-21 : Create Spark Context : Create Spark Context : 0:21 : Configure succeeded. (Create Spark Context)
2017-06-13 12:38:16,929 : DEBUG : KNIME-Worker-21 : NodeContainer : Create Spark Context : 0:21 : Create Spark Context 0:21 has new state: CONFIGURED
2017-06-13 12:38:16,947 : DEBUG : KNIME-Worker-21 : NodeContainer : Create Spark Context : 0:21 : iberdorla 0 has new state: IDLE
2017-06-13 12:38:16,961 : DEBUG : KNIME-WFM-Parent-Notifier : NodeContainer : : : ROOT has new state: IDLE
I think that the problem is that I don´t have a Job server because i don´'t know how to install it.
Some idea?