Connect Knime to a local Spark Cluster

Can I connect Knime with the node Create Spark Context to my local Spark Apache Cluster version 1.6?

I tried it but Knime returns the next error log:

2017-06-13 12:38:15,505 : DEBUG : main : ExecuteAction :  :  : Creating execution job for 1 node(s)...
2017-06-13 12:38:15,524 : DEBUG : main : NodeContainer :  :  : Create Spark Context 0:21 has new state: CONFIGURED_MARKEDFOREXEC
2017-06-13 12:38:15,547 : DEBUG : main : NodeContainer :  :  : Create Spark Context 0:21 has new state: CONFIGURED_QUEUED
2017-06-13 12:38:15,569 : DEBUG : main : NodeContainer :  :  : iberdorla 0 has new state: EXECUTING
2017-06-13 12:38:15,588 : DEBUG : KNIME-Worker-21 : WorkflowManager : Create Spark Context : 0:21 : Create Spark Context 0:21 doBeforePreExecution
2017-06-13 12:38:15,603 : DEBUG : KNIME-Worker-21 : NodeContainer : Create Spark Context : 0:21 : Create Spark Context 0:21 has new state: PREEXECUTE
2017-06-13 12:38:15,619 : DEBUG : KNIME-Worker-21 : WorkflowManager : Create Spark Context : 0:21 : Create Spark Context 0:21 doBeforeExecution
2017-06-13 12:38:15,633 : DEBUG : KNIME-Worker-21 : NodeContainer : Create Spark Context : 0:21 : Create Spark Context 0:21 has new state: EXECUTING
2017-06-13 12:38:15,645 : DEBUG : KNIME-Worker-21 : WorkflowFileStoreHandlerRepository : Create Spark Context : 0:21 : Adding handler 2e45c240-21c2-4bb0-9365-fe9205c94296 (Create Spark Context 0:21: <no directory>) - 3 in total
2017-06-13 12:38:15,658 : DEBUG : KNIME-Worker-21 : LocalNodeExecutionJob : Create Spark Context : 0:21 : Create Spark Context 0:21 Start execute
2017-06-13 12:38:15,658 : DEBUG : KNIME-WFM-Parent-Notifier : NodeContainer :  :  : ROOT  has new state: EXECUTING
2017-06-13 12:38:15,672 : INFO  : KNIME-Worker-21 : JobserverSparkContext : Create Spark Context : 0:21 : Spark context jobserver://localhost:8080/enrique changed status from CONFIGURED to CONFIGURED
2017-06-13 12:38:15,709 : DEBUG : KNIME-Worker-21 : JobserverSparkContext : Create Spark Context : 0:21 : Checking if remote context exists. Name: enrique
2017-06-13 12:38:15,779 : ERROR : KNIME-Worker-21 : GetContextsRequest : Create Spark Context : 0:21 : HTTP Status code: 200 | Response Body: <!DOCTYPE html><html>      <head>        <meta http-equiv="Content-type" content="text/html; charset=utf-8"/><link rel="stylesheet" href="/static/bootstrap.min.css" type="text/css"/><link rel="stylesheet" href="/static/vis.min.css" type="text/css"/><link rel="stylesheet" href="/static/webui.css" type="text/css"/><link rel="stylesheet" href="/static/timeline-view.css" type="text/css"/><script src="/static/sorttable.js"></script><script src="/static/jquery-1.11.1.min.js"></script><script src="/static/vis.min.js"></script><script src="/static/bootstrap-tooltip.js"></script><script src="/static/initialize-tooltips.js"></script><script src="/static/table.js"></script><script src="/static/additional-metrics.js"></script><script src="/static/timeline-view.js"></script>        <title>Spark Master at spark://192.168.41.129:7077</title>      </head>      <body>        <div class="container-fluid">          <div class="row-fluid">            <div class="span12">              <h3 style="vertical-align: middle; display: inline-block;">                <a style="text-decoration: none" href="/">                  <img src="/static/spark-logo-77x50px-hd.png"/>                  <span class="version" style="margin-right: 15px;">1.6.0</span>                </a>                Spark Master at spark://192.168.41.129:7077              </h3>            </div>          </div>          <div class="row-fluid">          <div class="span12">            <ul class="unstyled">              <li><strong>URL:</strong> spark://192.168.41.129:7077</li>              <li>                    <strong>REST URL:</strong> spark://192.168.41.129:6066                    <span class="rest-uri"> (cluster mode)</span>                  </li>              <li><strong>Alive Workers:</strong> 1</li>              <li><strong>Cores in use:</strong> 1 Total,                0 Used</li>              <li><strong>Memory in use:</strong>                6.8 GB Total,                0.0 B Used</li>              <li><strong>Applications:</strong>                0 Running,                0 Completed </li>              <li><strong>Drivers:</strong>                0 Running,                0 Completed </li>              <li><strong>Status:</strong> ALIVE</li>            </ul>          </div>        </div><div class="row-fluid">          <div class="span12">            <h4> Workers </h4>            <table class="table table-bordered table-condensed table-striped sortable">      <thead><th width="" class="">Worker Id</th><th width="" class="">Address</th><th width="" class="">State</th><th width="" class="">Cores</th><th width="" class="">Memory</th></thead>      <tbody>        <tr>      <td>        <a href="http://192.168.41.129:8081">worker-20170613112553-192.168.41.129-34592</a>      </td>      <td>192.168.41.129:34592</td>      <td>ALIVE</td>      <td>1 (0 Used)</td>      <td sorttable_customkey="6970.0">        6.8 GB        (0.0 B Used)      </td>    </tr>      </tbody>    </table>          </div>        </div><div class="row-fluid">          <div class="span12">            <h4> Running Applications </h4>            <table class="table table-bordered table-condensed table-striped sortable">      <thead><th width="" class="">Application ID</th><th width="" class="">Name</th><th width="" class="">Cores</th><th width="" class="">Memory per Node</th><th width="" class="">Submitted Time</th><th width="" class="">User</th><th width="" class="">State</th><th width="" class="">Duration</th></thead>      <tbody>              </tbody>    </table>          </div>        </div><div>                  </div><div class="row-fluid">          <div class="span12">            <h4> Completed Applications </h4>            <table class="table table-bordered table-condensed table-striped sortable">      <thead><th width="" class="">Application ID</th><th width="" class="">Name</th><th width="" class="">Cores</th><th width="" class="">Memory per Node</th><th width="" class="">Submitted Time</th><th width="" class="">User</th><th width="" class="">State</th><th width="" class="">Duration</th></thead>      <tbody>              </tbody>    </table>          </div>        </div><div>                  </div>        </div>      </body>    </html>
2017-06-13 12:38:15,965 : INFO  : KNIME-Worker-21 : JobserverSparkContext : Create Spark Context : 0:21 : Spark context jobserver://localhost:8080/enrique changed status from CONFIGURED to CONFIGURED
2017-06-13 12:38:16,147 : DEBUG : KNIME-Worker-21 : Create Spark Context : Create Spark Context : 0:21 : reset
2017-06-13 12:38:16,330 : DEBUG : KNIME-Worker-21 : SparkNodeModel : Create Spark Context : 0:21 : In reset() of SparkNodeModel. Calling deleteRDDs.
2017-06-13 12:38:16,505 : ERROR : KNIME-Worker-21 : Create Spark Context : Create Spark Context : 0:21 : Execute failed: Spark Jobserver gave unexpected response (for details see View > Open KNIME log). Possible reason: Incompatible jobserver version, malconfigured Spark jobserver
2017-06-13 12:38:16,692 : DEBUG : KNIME-Worker-21 : Create Spark Context : Create Spark Context : 0:21 : Execute failed: Spark Jobserver gave unexpected response (for details see View > Open KNIME log). Possible reason: Incompatible jobserver version, malconfigured Spark jobserver
com.knime.bigdata.spark.core.exception.KNIMESparkException: Spark Jobserver gave unexpected response (for details see View > Open KNIME log). Possible reason: Incompatible jobserver version, malconfigured Spark jobserver
    at com.knime.bigdata.spark.core.context.jobserver.request.AbstractJobserverRequest.createUnexpectedResponseException(AbstractJobserverRequest.java:156)
    at com.knime.bigdata.spark.core.context.jobserver.request.AbstractJobserverRequest.handleGeneralFailures(AbstractJobserverRequest.java:124)
    at com.knime.bigdata.spark.core.context.jobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:63)
    at com.knime.bigdata.spark.core.context.jobserver.request.GetContextsRequest.sendInternal(GetContextsRequest.java:1)
    at com.knime.bigdata.spark.core.context.jobserver.request.AbstractJobserverRequest.send(AbstractJobserverRequest.java:73)
    at com.knime.bigdata.spark.core.context.jobserver.JobserverSparkContext.remoteSparkContextExists(JobserverSparkContext.java:411)
    at com.knime.bigdata.spark.core.context.jobserver.JobserverSparkContext.access$3(JobserverSparkContext.java:409)
    at com.knime.bigdata.spark.core.context.jobserver.JobserverSparkContext$1.run(JobserverSparkContext.java:241)
    at com.knime.bigdata.spark.core.context.jobserver.JobserverSparkContext.runWithResetOnFailure(JobserverSparkContext.java:342)
    at com.knime.bigdata.spark.core.context.jobserver.JobserverSparkContext.open(JobserverSparkContext.java:231)
    at com.knime.bigdata.spark.core.context.SparkContext.ensureOpened(SparkContext.java:58)
    at com.knime.bigdata.spark.node.util.context.create.SparkContextCreatorNodeModel.executeInternal(SparkContextCreatorNodeModel.java:155)
    at com.knime.bigdata.spark.core.node.SparkNodeModel.execute(SparkNodeModel.java:235)
    at org.knime.core.node.NodeModel.executeModel(NodeModel.java:566)
    at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1128)
    at org.knime.core.node.Node.execute(Node.java:915)
    at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:561)
    at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
    at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
    at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
    at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
    at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
    at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
2017-06-13 12:38:16,715 : DEBUG : KNIME-Worker-21 : WorkflowManager : Create Spark Context : 0:21 : Create Spark Context 0:21 doBeforePostExecution
2017-06-13 12:38:16,737 : DEBUG : KNIME-Worker-21 : NodeContainer : Create Spark Context : 0:21 : Create Spark Context 0:21 has new state: POSTEXECUTE
2017-06-13 12:38:16,755 : DEBUG : KNIME-Worker-21 : WorkflowManager : Create Spark Context : 0:21 : Create Spark Context 0:21 doAfterExecute - failure
2017-06-13 12:38:16,773 : DEBUG : KNIME-Worker-21 : Create Spark Context : Create Spark Context : 0:21 : reset
2017-06-13 12:38:16,791 : DEBUG : KNIME-Worker-21 : SparkNodeModel : Create Spark Context : 0:21 : In reset() of SparkNodeModel. Calling deleteRDDs.
2017-06-13 12:38:16,813 : DEBUG : KNIME-Worker-21 : Create Spark Context : Create Spark Context : 0:21 : clean output ports.
2017-06-13 12:38:16,833 : DEBUG : KNIME-Worker-21 : WorkflowFileStoreHandlerRepository : Create Spark Context : 0:21 : Removing handler 2e45c240-21c2-4bb0-9365-fe9205c94296 (Create Spark Context 0:21: <no directory>) - 2 remaining
2017-06-13 12:38:16,853 : DEBUG : KNIME-Worker-21 : NodeContainer : Create Spark Context : 0:21 : Create Spark Context 0:21 has new state: IDLE
2017-06-13 12:38:16,875 : DEBUG : KNIME-Worker-21 : SparkContextCreatorNodeModel : Create Spark Context : 0:21 : Reconfiguring old context with same ID.
2017-06-13 12:38:16,892 : WARN  : KNIME-Worker-21 : Create Spark Context : Create Spark Context : 0:21 : Context is the same as in the Spark preferences (see File > Preferences > KNIME > Spark)
2017-06-13 12:38:16,909 : DEBUG : KNIME-Worker-21 : Create Spark Context : Create Spark Context : 0:21 : Configure succeeded. (Create Spark Context)
2017-06-13 12:38:16,929 : DEBUG : KNIME-Worker-21 : NodeContainer : Create Spark Context : 0:21 : Create Spark Context 0:21 has new state: CONFIGURED
2017-06-13 12:38:16,947 : DEBUG : KNIME-Worker-21 : NodeContainer : Create Spark Context : 0:21 : iberdorla 0 has new state: IDLE
2017-06-13 12:38:16,961 : DEBUG : KNIME-WFM-Parent-Notifier : NodeContainer :  :  : ROOT  has new state: IDLE

 

I think that the problem is that I don´t have a Job server because i don´'t know how to install it.

Some idea?

 

This link will give you the Spark Executor : https://www.knime.org/knime-spark-executor 

Choose the one that matches your knime Analytics platform in the "Installation Steps" section.

Step 2: also has the Installaiton guide and how to enable that on your cluster.

If you have done the above steps it should connect automatically.