Spark-Python code Integration

Community Extensions
1

Hi,

Is there a way I can use the "Create Spark Context" node to create a new spark context that can be levereged in Python code ?  I tried to do so, but the python0script node didn't recognize the contect created .

 

if this is not the correct way, is there a workaround to write custom Spark-Python code ?

 

 

2

Hi,

no this is not possible right now as we do not support Python code for Spark jobs. Currently you can only use the Spark Java Snippet node to write your own Spark job using Java. With the next Spark Executor release we will also provide a public API which will allow anybody to write Spark nodes and Spark jobs themselve.

Bye

Tobias

3

Thanks Tobias for your answer.

So, there is no work around to even execute a spark application that is written outside KNIME ? or execute Spark-Scala code ? or to invoke a shell command from KNIME ?

What is expected date of the next spark executer release ? 

 

Thanks

Ahmed

4

Hello Ahmed,

no right now you can not execute any custome spark application from within KNIME. However this will change with the next release which is planned for May.

Bye

Tobias

5

Hi Tobias,

Just wanted to follow up on the date of next release of KNIME Big Data Extensions that will support Spark 1.6 and will fix the above mentioned issues . I've checked the website but it seems it is not released yet.

6

Hello aelnabawi,

The KNIME Big Data Extension that was released today supports Spark v1.6.

Best,
Ferry
 

7

Hi Tobias,

Would like to understand, that the latest Spark Extension support PySpark code to execute Spark jobs.

Thanks in advance.

8

Please refrain from double posting questions (the other question is at https://tech.knime.org/forum/knime-general/can-we-run-pyspark-code-thru-knime). It won't result in faster answers instead you are causing more work for the people answering posts in the forum.

9

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.