Hello,
when trying to use Apache workflow executor together with local big data environment, the component does not execute and throws in the following error :
“Could not find required Spark job for Spark version: 3.5. Possible reason: The KNIME Extension for Apache Spark that provides the jobs for your Spark version is not installed.”
This is likely to the fact that the workflow executor only supports spark 3.4, whereas the local big data env node was updated (in 5.3) to use spark 3.5.
“BD-1271: (Big Data Extensions): Update Create Local Big Data Environment to Apache Spark 3.5”
I have tested the issue in KNIME 5.2.6 and the component executes without issues.
Since there is no spark version config available in “Create Local Big Data Environment” node, is using older versions of KNIME the only available workaround?
are there plans to update the workflow executor extension to align the versions?
Thank you
Adam