Error while using Create Databricks Environment Node

[ShinagdeS]

Hi Team,
I am using Create Databricks Environment Node in my workflow.
But I am getting below error

WARN  Create Databricks Environment 0:27       Job jar uninstall failed, library in unknown state: UNINSTALL_ON_RESTART
ERROR Create Databricks Environment 0:27       Execute failed: Invalid job jar on cluster detected. Please restart or terminate the cluster and run this node again.

I tried to restart my cluster and execute the node again but still, I got the same error.

How can I fix this issue?

Thank you
Srinivas

[sascha.wolke]

Hi @ShinagdeS,

what Spark and Databricks Cluster Version are you using?

Can you make sure there is no knime library attached to the cluster after you restart the cluster?

Cheers
Sascha

[ShinagdeS]

Hi @sascha.wolke
I am using Amazon AWS cluster.
How can I make sure that knime library is not getting attached ?

Thank you.
Srinivas

[sascha.wolke]

Hi @ShinagdeS,

you have to select a Databricks Runtime Version that uses a Spark Version supported by KNIME. You can select the Spark Version in the Create Databricks Environment configuration dialog. The latest Version supported by KNIME is 3.0. Does this help?

If not, navigate to your cluster (as you already have in your screenshot), click on the Libraries tab and remove all knime-spark-jobs-jar entries.

Cheers
Sascha

[marvin.kickuth]

Dear @ShinagdeS

did you manage to resolve the issue? If you need any further help, please feel free to ask

Kind regards
Marvin