I am attempting to connect to HIVE using the KNIME Big Data Connector from a Windows PC. The Windows PC and the Hadoop cluster on in different realms. I'm using MIT Kerberos on the PC to manage tickets.
I am having an odd problem connecting KNIME to Hive using the Big Data Connectors. The error message I get in KNIME is:
ERROR Hive Connector 0:1 Exception creating Kerberos based jdbc connection: Illegal principal name myusername@MYDOMAIN.NET
ERROR Hive Connector 0:1 Execute failed: Could not create connection to database: No rules applied to myusername@MYDOMAIN.NET
The error comes back immediately (i.e. not a timeout problem), and there is nothing in the log file.
Things I know:
1. myusername@MYDOMAIN.NET is correct. I've got a valid Kerberos ticket.
2. I can use applications on this PC to conenct using Kerberos (i.e. TOAD for Hadoop works perfectly)
3. I can use my account on another computer to connect KNIME to HIVE with the Big Data conenctor. We installed the same things on one of my team mate's PCs and it worked for both of us.
From what I can tell, the message usually indicates an authentication problem with the Kerberos principal, but I am kind of stuck. I'd appreciate any help in debugging this that the community can offer.
this seems to be a problem with cross realms. You might want to have a look into this post. It might be also releated to the dfs.namenode.kerberos.principal.pattern property of the hdfs-site.xml file which is by default *. You can overwrite the default pattern in the KNIME preferences. To do so open KNIME and go to File->Preferences. In the Preferences dialog go to KNIME->KNIME Big Data Extensions->Hadoop and enter the path to your hdfs-site.xml file. You might also want to check your krb5.conf file if it contains the same default realm as the one of your colleague.
Attached you can find a KNIME workflow with a Java Edit Variable node that enables Kerberos debugging and forwarding of the log messages to the KNIME log as warning messages. To use it simply import the workflow into your KNIME via File->Import Workflow and execute the Java Edit Variable node before executing the Hive Connector node.
That is an extreemly handy bit of Java code. I was able to debug the problem with what you provided. I'll post the solution here for others who may have similar issues:
First - wtih Windows connecting to Hadoop, you may encounter a cross-domain authentication scenario. The typical approach is to install the MIT Kerberos client so you can authenticate seperately from the Windows client auth. In my case, this worked perfectly.
Important tip: In the debug code generated using the module attached to this thread, I got these lines (among others). You can see that the krb5.ini file that Java uses by default is in the Windows root directory, which was NOT where I had put the file. (The default for MIT is C:\ProgramData\MIT\Kerberos\}.
WARN Java Edit Variable 0:1 Kerberos debug log redirected to KNIME log with warning level
WARN Hive Connector 0:2 Your database timeout (15 s) is set to a rather low value for Hive. If you experience timeouts increase the value in the preference page.
WARN Hive Connector 0:2 Java config name: null
WARN Hive Connector 0:2 Native config name: C:\WINDOWS\krb5.ini
You have a couple of options to fix this:
1. Copy the krb5.ini file to the Windows root so Java will find it by default (and maybe redirect the MIT client).
2. Add an environment variable to help Java find the right one:
Varialbe name = _JAVA_OPTIONS
Variable value = -Djava.security.krb5.conf=C:\ProgramData\MIT\Kerberos5\krb5.ini
Either way, all works as execpected thanks to some debugging help!
I'm gald it helped you to solve the problem. Thanks a lot for sharing your insights.
Just for documentation:
This tutorial describes how to set the global _JAVA_OPTIONS environment variable in Windows. However this is a global variable and will effect all Java programs.
To set this variable or any other Java option (e.g. memory size) specifically for KNIME you can append the variable to the end of the knime.ini file which is located in your KNIME installation directory (on Windows this is usually C:\Program Files\knime). For the Kerberos configuration file you would add the following line: