Amazon EMR Spark 3.0

KNIME Extensions
,
1

Hi,

I´d like to use the “Create Spark Context (Livy)” node to connect to an existing Amazon EMR Cluster with Spark 3 installed.
The node only allows 2.x Spark versions and throws errors when trying to connect/execute because of the version mismatch 2.4->3.0

Is there a workaround to connect to such a cluster, just for demo / test purposes?

Thanks

2

Hello,
we do not support Spark 3.0 as of today and there is no workaround. However when creating the EMR cluster you can select another version e.g. 6.0.0 or 5.x which are compatible with the big data extension. Support for Spark 3.0 is on the roadmap for next year.
Bye
Tobias

3

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.