we are currently struggeling with connecting to a Kerberos enabled Livy endpoint using KNIME 4.0.2.
What we basically have on the server side is the following:
- a KDC containing all Principals (HDFS, Livy and Test Principal for KNIME)
- a Kerberos enabled HDFS setup (Namenode, Datanode…)
- a Livy endpoint which is also Kerberos enabled
The connection to HDFS can be established using the Test Principal’s Keytab.
When trying to submit jobs to Livy the only error message being shown is
ERROR Create Spark Context (Livy) 0:1 Execute failed: Authentication required:Error 401
HTTP ERROR: 401
Problem accessing /sessions/. Reason:
Powered by Jetty://
When using curl (with --negotiate) after having called kinit using the exact same Keytab as used for KNIME it is possible to create a session in Livy.
We’ve been basically trying to follow all guides available (which are probably outdated…):
Did anybody ever accomplish doing something like this and is able to help?
Has this even been tested before?
Fun fact for everybody trying to use Livy 0.6.0. Stop trying, use Livy 0.5.0 as 0.6.0 is not supported by KNIME at all.