Connecting Spark 3.2 with livy

Hello Guys,

Does anyone know how to connect spark 3.2 with livy? I am trying to use spark 3.2 clusters from knime but I saw some threads that livy supports until spark 3.0 and which means knime bigdata extension supports until 3.0 only.

Any good idea?

Thanks

@KyuHo welcome to the KNIME forum. I think at the moment the latest Spark version supported is 3.0.

Hi @KyuHo,

welcome to the KNIME community! Right now, KNIME supports only Spark up to 3.0 and there is no workaround to use Spark 3.2. We are already working on Spark 3.2 support and hopefully ship this with the KNIME summer release.

Cheers
Sascha

2 Likes