Spark context was created by version 2.1.0.v201709091016 of the KNIME Extension for Apache Spark, but you are running 2.2.0.v201712021420.

Hi,

 

I have just updated my Knime to 4.5.0

I tried to run my previous workflow using spark, however an error appears

Spark context was created by version 2.1.0.v201709091016 of the KNIME Extension for Apache Spark, but you are running 2.2.0.v201712021420. Please destroy and reopen this Spark context or use a different one.
 

how should I deal with this?
Thank you in advance.

Hi,

with your KNIME you also updated the Spark extension (now called "KNIME Extensions for Apache Spark"). Whenever you update the Spark extensions you cannot work with your existing Spark context anymore, so either you destroy it (Destroy Spark Context node) or you create a new one with a different name (Create Spark Context node).

A Spark context is supposed to be a temporary session. It is best practice to only keep it open while you are working with it. When you don't (e.g. when you are not at work), you save your current results to persistent storage (HDFS) and destroy the context.

 

Best,

Björn
 

 

 

 

 

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.