Hive Connector & Kerberos (Cloudera) - Unable to connect

KNIME Extensions Big Data
#1

Hi,

I try to connect to a Cloudera Cluster with Kerberos enabled.
Followed the steps in the BlogPost https://www.knime.org/blog/speaking-kerberos-with-knime-big-data-extensions
Still getting timeout-messages, when trying to connect the Hive-Server.

Any suggestions how to fix this?

Thanks !

2018-03-23 22:02:47,289 : DEBUG : main : ExecuteAction : : : Creating execution job for 1 node(s)…
2018-03-23 22:02:47,290 : DEBUG : main : NodeContainer : : : Hive Connector 0:5 has new state: CONFIGURED_MARKEDFOREXEC
2018-03-23 22:02:47,290 : DEBUG : main : NodeContainer : : : Hive Connector 0:5 has new state: CONFIGURED_QUEUED
2018-03-23 22:02:47,290 : DEBUG : main : NodeContainer : : : Datencheck 0 has new state: EXECUTING
2018-03-23 22:02:47,290 : DEBUG : KNIME-WFM-Parent-Notifier : NodeContainer : : : ROOT has new state: EXECUTING
2018-03-23 22:02:47,290 : DEBUG : KNIME-Worker-18 : WorkflowManager : Hive Connector : 0:5 : Hive Connector 0:5 doBeforePreExecution
2018-03-23 22:02:47,290 : DEBUG : KNIME-Worker-18 : NodeContainer : Hive Connector : 0:5 : Hive Connector 0:5 has new state: PREEXECUTE
2018-03-23 22:02:47,290 : DEBUG : KNIME-Worker-18 : WorkflowManager : Hive Connector : 0:5 : Hive Connector 0:5 doBeforeExecution
2018-03-23 22:02:47,291 : DEBUG : KNIME-Worker-18 : NodeContainer : Hive Connector : 0:5 : Hive Connector 0:5 has new state: EXECUTING
2018-03-23 22:02:47,291 : DEBUG : KNIME-Worker-18 : WorkflowFileStoreHandlerRepository : Hive Connector : 0:5 : Adding handler c3233870-a35d-4868-94c5-2a661b5d375a (Hive Connector 0:5: ) - 1 in total
2018-03-23 22:02:47,291 : DEBUG : KNIME-Worker-18 : LocalNodeExecutionJob : Hive Connector : 0:5 : Hive Connector 0:5 Start execute
2018-03-23 22:02:47,291 : DEBUG : KNIME-Worker-18 : HiveConnectorNodeModel : Hive Connector : 0:5 : Using jdbc url: jdbc:hive2://mndemucsva01500.mn-man.biz:10000/trend;AuthMech=1;KrbRealm=MN-MAN.BIZ;KrbHostFQDN=mndemucsva01500.mn-man.biz;KrbServiceName=hive;
2018-03-23 22:02:47,294 : DEBUG : KNIME-Worker-18 : NodeContext : Hive Connector : 0:5 : Workflow user found: d0519
2018-03-23 22:02:47,294 : DEBUG : KNIME-Worker-18 : DatabaseDriverLoader : Hive Connector : 0:5 : Database driver retrieved from user defined drivers: com.cloudera.hive.jdbc41.HS2DriverDriver info: Driver class name: com.cloudera.hive.jdbc41.HS2Driver major version: 2 minor version: 5 jdbc compliant: false
2018-03-23 22:02:47,294 : DEBUG : KNIME-Worker-18 : PriorityDriverFactory : Hive Connector : 0:5 : DB driver: com.cloudera.hive.jdbc41.HS2Driver found in driver factory: org.knime.core.node.port.database.RegisteredDriversConnectionFactory. Driver info: Driver class name: org.knime.core.node.port.database.DatabaseWrappedDriver major version: 2 minor version: 5 jdbc compliant: false
2018-03-23 22:02:47,294 : DEBUG : pool-5-thread-12 : RegisteredDriversConnectionFactory : Hive Connector : 0:5 : Opening database connection to “jdbc:hive2://mndemucsva01500.mn-man.biz:10000/trend;AuthMech=1;KrbRealm=MN-MAN.BIZ;KrbHostFQDN=mndemucsva01500.mn-man.biz;KrbServiceName=hive;”…
2018-03-23 22:02:47,300 : DEBUG : pool-5-thread-12 : UserGroupUtil : Hive Connector : 0:5 : Retrieving user for Kerberos authentication
2018-03-23 22:02:47,301 : DEBUG : pool-5-thread-12 : UserGroupUtil : Hive Connector : 0:5 : Determining Kerberos TGT user based on keytab settings (keytab file: C:\Users\d0519\Documents\d0519.keytab / keytab user: d0519).
2018-03-23 22:02:47,301 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : Java config name: D:\SW\KNIME\plugins\org.knime.binary.jre.win32.x86_64_1.8.0.152-01\jre\lib\security\krb5.conf
2018-03-23 22:02:47,302 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : Loaded from Java config
2018-03-23 22:02:47,302 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : >>> KdcAccessibility: reset
2018-03-23 22:02:47,302 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : Looking for keys for: d0519@MN-MAN.BIZ
2018-03-23 22:02:47,302 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : Added key: 23version: 1
2018-03-23 22:02:47,302 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : Looking for keys for: d0519@MN-MAN.BIZ
2018-03-23 22:02:47,302 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : Added key: 23version: 1
2018-03-23 22:02:47,302 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : Using builtin default etypes for default_tkt_enctypes
2018-03-23 22:02:47,302 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : default etypes for default_tkt_enctypes:
2018-03-23 22:02:47,302 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : 18
2018-03-23 22:02:47,302 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : 17
2018-03-23 22:02:47,302 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : 16
2018-03-23 22:02:47,303 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : 23
2018-03-23 22:02:47,303 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : .
2018-03-23 22:02:47,303 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : >>> KrbAsReq creating message
2018-03-23 22:02:47,303 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : >>> KrbKdcReq send: kdc=mndemucsva01500.mn-man.biz UDP:88, timeout=30000, number of retries =3, #bytes=128
2018-03-23 22:02:47,304 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : >>> KDCCommunication: kdc=mndemucsva01500.mn-man.biz UDP:88, timeout=30000,Attempt =1, #bytes=128
2018-03-23 22:03:17,303 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : SocketTimeOutException with attempt: 1
2018-03-23 22:03:17,304 : DEBUG : pool-5-thread-12 : KerberosLogger : Hive Connector : 0:5 : >>> KDCCommunication: kdc=mndemucsva01500.mn-man.biz UDP:88, timeout=30000,Attempt =2, #bytes=128
2018-03-23 22:03:18,294 : DEBUG : KNIME-Worker-18 : Node : Hive Connector : 0:5 : reset
2018-03-23 22:03:18,294 : ERROR : KNIME-Worker-18 : Node : Hive Connector : 0:5 : Execute failed: Could not create connection to database: Connection to database ‘jdbc:hive2://mndemucsva01500.mn-man.biz:10000/trend;AuthMech=1;KrbRealm=MN-MAN.BIZ;KrbHostFQDN=mndemucsva01500.mn-man.biz;KrbServiceName=hive;’ timed out
2018-03-23 22:03:18,294 : DEBUG : KNIME-Worker-18 : Node : Hive Connector : 0:5 : Execute failed: Could not create connection to database: Connection to database ‘jdbc:hive2://mndemucsva01500.mn-man.biz:10000/trend;AuthMech=1;KrbRealm=MN-MAN.BIZ;KrbHostFQDN=mndemucsva01500.mn-man.biz;KrbServiceName=hive;’ timed out
java.sql.SQLException: Could not create connection to database: Connection to database ‘jdbc:hive2://mndemucsva01500.mn-man.biz:10000/trend;AuthMech=1;KrbRealm=MN-MAN.BIZ;KrbHostFQDN=mndemucsva01500.mn-man.biz;KrbServiceName=hive;’ timed out
at org.knime.bigdata.hive.node.connector.HiveConnectorNodeModel.execute(HiveConnectorNodeModel.java:136)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:567)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1172)
at org.knime.core.node.Node.execute(Node.java:959)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:561)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:179)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:110)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:328)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:204)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
Caused by: java.io.IOException: Connection to database ‘jdbc:hive2://mndemucsva01500.mn-man.biz:10000/trend;AuthMech=1;KrbRealm=MN-MAN.BIZ;KrbHostFQDN=mndemucsva01500.mn-man.biz;KrbServiceName=hive;’ timed out
at org.knime.core.node.port.database.connection.CachedConnectionFactory.getConnection(CachedConnectionFactory.java:245)
at org.knime.core.node.port.database.DatabaseConnectionSettings.createConnection(DatabaseConnectionSettings.java:450)
at org.knime.bigdata.hive.node.connector.HiveConnectorNodeModel.execute(HiveConnectorNodeModel.java:128)
… 13 more
2018-03-23 22:03:18,295 : DEBUG : KNIME-Worker-18 : WorkflowManager : Hive Connector : 0:5 : Hive Connector 0:5 doBeforePostExecution
2018-03-23 22:03:18,295 : DEBUG : KNIME-Worker-18 : NodeContainer : Hive Connector : 0:5 : Hive Connector 0:5 has new state: POSTEXECUTE
2018-03-23 22:03:18,295 : DEBUG : KNIME-Worker-18 : WorkflowManager : Hive Connector : 0:5 : Hive Connector 0:5 doAfterExecute - failure
2018-03-23 22:03:18,295 : DEBUG : KNIME-Worker-18 : Node : Hive Connector : 0:5 : reset
2018-03-23 22:03:18,295 : DEBUG : KNIME-Worker-18 : Node : Hive Connector : 0:5 : clean output ports.
2018-03-23 22:03:18,295 : DEBUG : KNIME-Worker-18 : WorkflowFileStoreHandlerRepository : Hive Connector : 0:5 : Removing handler c3233870-a35d-4868-94c5-2a661b5d375a (Hive Connector 0:5: ) - 0 remaining
2018-03-23 22:03:18,295 : DEBUG : KNIME-Worker-18 : NodeContainer : Hive Connector : 0:5 : Hive Connector 0:5 has new state: IDLE
2018-03-23 22:03:18,296 : DEBUG : KNIME-Worker-18 : HiveDriverDetector : Hive Connector : 0:5 : Using Cloudera Hive driver: com.cloudera.hive.jdbc41.HS2Driver
2018-03-23 22:03:18,296 : DEBUG : KNIME-Worker-18 : HiveConnectorNodeModel : Hive Connector : 0:5 : Using jdbc url: jdbc:hive2://mndemucsva01500.mn-man.biz:10000/trend;AuthMech=1;KrbRealm=MN-MAN.BIZ;KrbHostFQDN=mndemucsva01500.mn-man.biz;KrbServiceName=hive;
2018-03-23 22:03:18,300 : DEBUG : KNIME-Worker-18 : Node : Hive Connector : 0:5 : Configure succeeded. (Hive Connector)
2018-03-23 22:03:18,300 : DEBUG : KNIME-Worker-18 : NodeContainer : Hive Connector : 0:5 : Hive Connector 0:5 has new state: CONFIGURED
2018-03-23 22:03:18,301 : DEBUG : KNIME-Worker-18 : NodeContainer : Hive Connector : 0:5 : Datencheck 0 has new state: CONFIGURED
2018-03-23 22:03:18,301 : DEBUG : KNIME-WFM-Parent-Notifier : NodeContainer : : : ROOT has new state: IDLE

#2

Hi,

the log indicates that you are trying to authenticate against a KDC (Kerberos authentication server) at mndemucsva01500.mn-man.biz (udp, port 88). However there is no KDC running at that address, or there is a firewall that prevents you from connecting to the KDC.

Note: The KDC server is not the Hive server. The Hive server is specified in the Hive Connector configuration dialog. The KDC server is specified in the krb5.conf.

Hence this needs to be fixed in your krb5.conf file which is at:

D:\SW\KNIME\plugins\org.knime.binary.jre.win32.x86_64_1.8.0.152-01\jre\lib\security\krb5.conf

If you are unsure about the correct address of your KDC, then you could download a /etc/krb5.conf from one of your Hadoop cluster nodes, or you could ask one of the Hadoop admins to provide you with the correct krb5.conf.

Best,
Björn

#3

Hi Björn,

Thanks for the hint.
Using the correct krb5.conf file (copied from existing node), fixed it.
Now I can connect KNIME via Hive and Impala to the kerberized cluster.

Best regards,
Dieter