Cannot create Spark node

rastasanu · November 29, 2023, 3:05pm

Hello everyone,
I am trying to use Spark connected to an HDFS but I cannot create the node “Create Spark Context (Livy)”. When I drag and drop the node on the workflow I am getting the following error:

It seems to be a problem only with this specific node, as I can create other nodes from the Apache Spark extension.
I have already tried to change networks and restart my KNIME Analytics Platform.
Here is also the log just in case:

2023-11-29 15:44:36,625 : DEBUG : main :  : CreateNodeCommand :  :  : Node cannot be created.
java.lang.NullPointerException: Cannot invoke "org.knime.bigdata.spark.core.version.SparkVersion.getLabel()" because "org.knime.bigdata.spark.core.livy.LivySparkContextProvider.HIGHEST_SUPPORTED_AND_AVAILABLE_SPARK_VERSION" is null
	at org.knime.bigdata.spark.core.livy.node.create.LivySparkContextCreatorNodeSettings.<init>(LivySparkContextCreatorNodeSettings.java:139)
	at org.knime.bigdata.spark.core.livy.node.create.AbstractLivySparkContextCreatorNodeModel.<init>(AbstractLivySparkContextCreatorNodeModel.java:51)
	at org.knime.bigdata.spark.core.livy.node.create.LivySparkContextCreatorNodeModel2.<init>(LivySparkContextCreatorNodeModel2.java:54)
	at org.knime.bigdata.spark.core.livy.node.create.LivySparkContextCreatorNodeFactory2.createNodeModel(LivySparkContextCreatorNodeFactory2.java:45)
	at org.knime.bigdata.spark.core.livy.node.create.LivySparkContextCreatorNodeFactory2.createNodeModel(LivySparkContextCreatorNodeFactory2.java:1)
	at org.knime.core.node.NodeFactory.callCreateNodeModel(NodeFactory.java:520)
	at org.knime.core.node.Node.<init>(Node.java:307)
	at org.knime.core.node.workflow.WorkflowManager.addNodeAndApplyContext(WorkflowManager.java:844)
	at org.knime.core.ui.wrapper.WorkflowManagerWrapper.createAndAddNode(WorkflowManagerWrapper.java:140)
	at org.knime.workbench.editor2.commands.CreateNodeCommand.lambda$0(CreateNodeCommand.java:183)
	at org.knime.workbench.ui.async.AsyncUtil.wfmAsyncSwitch(AsyncUtil.java:130)
	at org.knime.workbench.editor2.commands.CreateNodeCommand.execute(CreateNodeCommand.java:182)
	at org.eclipse.gef.commands.CommandStack.execute(CommandStack.java:228)
	at org.knime.workbench.editor2.WorkflowEditorDropTargetListener.handleDropAsync(WorkflowEditorDropTargetListener.java:531)
	at org.knime.workbench.editor2.WorkflowEditorDropTargetListener.lambda$5(WorkflowEditorDropTargetListener.java:494)
	at org.eclipse.swt.widgets.RunnableLock.run(RunnableLock.java:40)
	at org.eclipse.swt.widgets.Synchronizer.runAsyncMessages(Synchronizer.java:132)
	at org.eclipse.swt.widgets.Display.runAsyncMessages(Display.java:4043)
	at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3648)
	at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$5.run(PartRenderingEngine.java:1155)
	at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:338)
	at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1046)
	at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:155)
	at org.eclipse.ui.internal.Workbench.lambda$3(Workbench.java:644)
	at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:338)
	at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:551)
	at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:156)
	at org.knime.product.rcp.KNIMEApplication.start(KNIMEApplication.java:191)
	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:136)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:402)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:659)
	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:596)
	at org.eclipse.equinox.launcher.Main.run(Main.java:1467)

Thanks in advance!

Kind regards,

Cristian

sascha.wolke · November 29, 2023, 4:10pm

Hi @rastasanu,

I am not sure why there are now Spark Plugins available. What KNIME version are you using, did you update KNIME recently?

If you installed the Spark Extension recently, make sure to update KNIME. You can verify that the Spark Job Plugins are installed this way:

Help → About KNIME Analytics Platform
Installation Details (left bottom in the Dialog)
Navigate to the Plug-Ins Tab and enter Spark in the Search Box

Cheers,
Sascha

rastasanu · November 29, 2023, 4:33pm

Hello @sascha.wolke ,
Thanks for your reply!
I am using KNIME 4.7.3 and I checked the plug-ins and it seems I have them all

including the extension:

Cheers,
Cristian

sascha.wolke · November 29, 2023, 4:50pm

Hi @rastasanu,

Can you restart KNIME, add the node to a workflow, and check if there are errors in the console logged since the start?

Maybe you can try to update to the latest 4.7.7 release via File → Update KNIME.

Cheers,
Sascha

rastasanu · December 5, 2023, 9:09am

Hi @sascha.wolke ,

Thanks again for the reply!

I tried restarting and adding the node but I still cannot add it as I am getting the same error as before.

There is however an error in the console since the start of the KNIME Analytics Platform:

******************************************************************************
***         Welcome to KNIME Analytics Platform v4.7.3.v202305100921       ***
***               Copyright by KNIME AG, Zurich, Switzerland               ***
******************************************************************************
Log file is located at: C:\Users\crastasanu\knime-workspace\.metadata\knime\knime.log
ERROR SparkProviderRegistry            Problems during initialization of Spark provider with id 'org.knime.bigdata.spark.core.databricks.DatabricksNodeFactoryProvider'. Exception: Plug-in org.knime.bigdata.databricks was unable to load class org.knime.bigdata.spark.core.databricks.DatabricksNodeFactoryProvider.
ERROR SparkProviderRegistry            Extension org.knime.bigdata.databricks ignored.
ERROR RepositoryManager               Node 'org.knime.bigdata.dbfs.node.connector.DBFSConnectionInformationNodeFactory' from plugin 'org.knime.bigdata.databricks' could not be created. The corresponding plugin bundle could not be activated!
ERROR RepositoryManager               Node 'org.knime.bigdata.dbfs.filehandling.node.DbfsConnectorNodeFactory' from plugin 'org.knime.bigdata.databricks' could not be created. The corresponding plugin bundle could not be activated!
ERROR SparkProviderRegistry            Problems during initialization of Spark provider with id 'org.knime.bigdata.spark2_4.base.Spark_2_4_JobRunFactoryProvider'. Exception: Plug-in "org.knime.bigdata.spark2_4" was unable to instantiate class "org.knime.bigdata.spark2_4.base.Spark_2_4_JobRunFactoryProvider".
ERROR SparkProviderRegistry            Extension org.knime.bigdata.spark2_4 ignored.
ERROR SparkProviderRegistry            Problems during initialization of Spark provider with id 'org.knime.bigdata.spark3_0.base.Spark_3_0_JobRunFactoryProvider'. Exception: Plug-in "org.knime.bigdata.spark3_0" was unable to instantiate class "org.knime.bigdata.spark3_0.base.Spark_3_0_JobRunFactoryProvider".
ERROR SparkProviderRegistry            Extension org.knime.bigdata.spark3_0 ignored.
ERROR SparkProviderRegistry            Problems during initialization of Spark provider with id 'org.knime.bigdata.spark3_1.base.Spark_3_1_JobRunFactoryProvider'. Exception: Plug-in "org.knime.bigdata.spark3_1" was unable to instantiate class "org.knime.bigdata.spark3_1.base.Spark_3_1_JobRunFactoryProvider".
ERROR SparkProviderRegistry            Extension org.knime.bigdata.spark3_1 ignored.
ERROR SparkProviderRegistry            Problems during initialization of Spark provider with id 'org.knime.bigdata.spark3_2.base.Spark_3_2_JobRunFactoryProvider'. Exception: Plug-in "org.knime.bigdata.spark3_2" was unable to instantiate class "org.knime.bigdata.spark3_2.base.Spark_3_2_JobRunFactoryProvider".
ERROR SparkProviderRegistry            Extension org.knime.bigdata.spark3_2 ignored.
ERROR SparkProviderRegistry            Problems during initialization of Spark provider with id 'org.knime.bigdata.spark3_3.base.Spark_3_3_JobRunFactoryProvider'. Exception: Plug-in "org.knime.bigdata.spark3_3" was unable to instantiate class "org.knime.bigdata.spark3_3.base.Spark_3_3_JobRunFactoryProvider".
ERROR SparkProviderRegistry            Extension org.knime.bigdata.spark3_3 ignored.
ERROR SparkProviderRegistry            Problems during initialization of Spark provider with id 'org.knime.bigdata.spark3_4.base.Spark_3_4_JobRunFactoryProvider'. Exception: Plug-in "org.knime.bigdata.spark3_4" was unable to instantiate class "org.knime.bigdata.spark3_4.base.Spark_3_4_JobRunFactoryProvider".
ERROR SparkProviderRegistry            Extension org.knime.bigdata.spark3_4 ignored.

Perhaps it can give more insights about why it’s failing.

About the update of the platform since I am in a business environment and we are more people using the same version and working on the KNIME server so this would mean that we would all need to update it passing through the admin, so I would do it only as a last resort and if we can tell that this is most likely the cause of the error.

Thanks in advance!

Cheers,
Cris

rastasanu · December 5, 2023, 9:37am

Hi again @sascha.wolke , I’ve got some additional info that might be helpful.

I tried to upload a workflow containing just the HDFS connector on the KNIME Server, then I created a job out of it and I added the Spark Context node on the job and this worked. What doesn’t work is opening the node with the same error I am getting when I am trying to add the node on my local environment.

Then I checked the version of the executor and indeed it is a slightly different version. On the server is 4.7.4 while locally we have 4.7.3

What I am thinking is that perhaps if we update the local version from 4.7.3 to 4.7.4 then it could work?

rastasanu · December 6, 2023, 9:10am

Hi @sascha.wolke ,
Some updates. I got the KNIME 4.7.4 in the .zip archive for test purposes and it works to create the node, however I am running against another error for which I think it is best to create another forum thread. FYI the error is Execute failed: Found: (LivyHttpException).

Thanks for your support!

Cheers,
Cris

system · December 13, 2023, 9:10am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.