I have KNIME 4.0.1 and KNIME Extension for Apache Spark 4.0.0.v201908141011.
While KNIME manages to start a Spark session via Livy on EMR, it does not seem to be able to register and keep hold of that connection. I use EMR version 5.26.0.
The error that is thrown is:
ERROR Create Spark Context (Livy) 0:9 Execute failed: Unrecognized field "name" (class org.apache.livy.client.common.HttpMessages$SessionInfo), not marked as ignorable (8 known properties: "state", "owner", "kind", "proxyUser", "id", "appId", "appInfo", "log"])
at [Source: org.apache.http.client.entity.LazyDecompressingInputStream@60b93581; line: 1, column: 20] (through reference chain: org.apache.livy.client.common.HttpMessages$SessionInfo["name"]) (UnrecognizedPropertyException)
If I open the Livy UI on EMR I can see the sessions open.
you are connecting to an EMR cluster that has Livy 0.6. Unfortunately, KNIME is currently only compatible with Livy 0.5 and lower, because Livy has changed its response message format with 0.6 and we need make adjustments in our Livy client library to adapt to those. Right now you need to choose an EMR version that has Livy 0.5. We will work on fixing that with the next bugfix release, sorry for the inconveniences.
I have exactly the same issue connecting to AWS EMR. Unfortunately, I cannot choose the version of Livy when configuring the EMR cluster. By default is 0.6.
I would like to know if the fix in Knime is already available. if not, when are you planning to release it?
