connection hdfs -> webhdfs failed

And · May 25, 2023, 3:17pm

Hello,
Often when I connect hdfs with these parameters :

I can connect, but often the connection fails, returning this error :

IS RANDOM…

Can anyone help me understand?

I use knime 4.6.0

sascha.wolke · May 26, 2023, 9:10am

Hi @And,

I guess your principal contains invalid characters or characters that do not work with rules to map them. See auth_to_local in the Hadoop config or MIT Kerberos.

Are you using random or frequently changing principals?

Cheers,
Sascha

And · May 31, 2023, 7:19am

Hi Sasha,
We always log in from a keytab using the same principal (example my_username@COMPANY.COM).

In the hadoop.security.auth_to_local configuration (specified in the core-site.xml file), we have set up the standard rule to remove the domain/realm from the principal name. This means that the user in Hadoop should simply be my_username.
The core-site.xml file is copied from the Hadoop cluster and works fine with various command-line tools such as hdfs dfs, yarn, and spark-submit.

However, in Knime, we have encountered an intermittent issue. Sometimes it works, and sometimes it doesn’t. When it doesn’t work, we receive the following error:

2023-05-25 16:30:00,718: ERROR: KNIME-Worker-6-HDFS Connector 3:1453: Node: HDFS Connector: 3:1453: Execution failed: Invalid value for the webhdfs parameter ‘user.name’: Invalid value: ‘my_username@COMPANY.COM’ does not belong to the domain ^[A-Za-z_][A-Za-z0-9.-][$]?$
org.apache.hadoop.ipc.RemoteException(java.lang.IllegalArgumentException): Invalid value for the webhdfs parameter ‘user.name’: Invalid value: ‘my_username@COMPANY.COM’ does not belong to the domain ^[A-Za-z][A-Za-z0-9._-][$]?$

Based on my limited knowledge, it appears that Knime doesn’t always apply the auth_to_local rule and instead attempts to use the full principal name (which is not a valid user.name) instead of the shortened version.

thanks,
And

sascha.wolke · May 31, 2023, 8:20am

Hi @And,

The hadoop.security.auth_to_local configuration should happen on the cluster side.

Are you using the command-line tools on your local machine or a node of the cluster? Your first screenshot looks like you are using WebHDFS, does the command-line tools use WebHDFS too?

Maybe your cluster logs contain more details about what goes wrong.

Cheers,
Sascha

And · June 5, 2023, 12:24pm

Hello Sasha,

We are using command-line tools on a cluster machine (running on Linux/RHEL).
We don’t have these tools installed on our local Knime machines, which are Windows-based. We don’t have any problems with webhdfs using the native client.

For example, the following command works :
…
hdfs dfs -ls webhdfs:///folder
…
On the cluster/server side, we don’t see any problems or errors when Knime fails.
Below, you can find a full stack trace for our connection attempt. From my limited understanding of the Hadoop client API, the error seems to be completely client-side:

2023-05-25 16:50:42,962 : ERROR : KNIME-Worker-9-HDFS Connector 3:1453 : : Node : HDFS Connector : 3:1453 : Execute failed: Invalid value: “my_username@COMPANY.COM” does not belong to the domain ^[A-Za-z_][A-Za-z0-9.-]*[$]?$
java.io.IOException: Invalid value: “my_username@COMPANY.COM” does not belong to the domain ^[A-Za-z][A-Za-z0-9.-]*[$]?$
at org.knime.filehandling.core.defaultnodesettings.ExceptionUtil.wrapAsIOException(ExceptionUtil.java:204)
at org.knime.bigdata.hadoop.filehandling.fs.ExceptionMapper.mapException(ExceptionMapper.java:103)
at org.knime.bigdata.hadoop.filehandling.fs.ExceptionMapper.mapException(ExceptionMapper.java:81)
at org.knime.bigdata.hadoop.filehandling.fs.HdfsFileSystemProvider.exists(HdfsFileSystemProvider.java:237)
at org.knime.bigdata.hadoop.filehandling.fs.HdfsFileSystemProvider.exists(HdfsFileSystemProvider.java:1)
at org.knime.filehandling.core.connections.base.BaseFileSystemProvider.existsCached(BaseFileSystemProvider.java:656)
at org.knime.filehandling.core.connections.base.BaseFileSystemProvider.readAttributes(BaseFileSystemProvider.java:733)
at java.base/java.nio.file.Files.readAttributes(Unknown Source)
at java.base/java.nio.file.Files.getLastModifiedTime(Unknown Source)
at org.knime.bigdata.hadoop.filehandling.node.HdfsConnectorNodeModel.testConnection(HdfsConnectorNodeModel.java:111)
at org.knime.bigdata.hadoop.filehandling.node.HdfsConnectorNodeModel.execute(HdfsConnectorNodeModel.java:103)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:549)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1267)
at org.knime.core.node.Node.execute(Node.java:1041)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:595)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:201)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:117)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:367)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:221)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
Caused by: java.lang.IllegalArgumentException: Invalid value: “my_username@COMPANY.COM” does not belong to the domain ^[A-Za-z][A-Za-z0-9._-]*[$]?$
at org.apache.hadoop.hdfs.web.resources.StringParam$Domain.parse(StringParam.java:54)
at org.apache.hadoop.hdfs.web.resources.StringParam.(StringParam.java:25)
at org.apache.hadoop.hdfs.web.resources.UserParam.(UserParam.java:68)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:448)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:459)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:727)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:620)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:472)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:502)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:498)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1352)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:246)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:437)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:459)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:727)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:620)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:472)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:502)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:498)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:884)
at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:894)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1425)
at org.knime.bigdata.hadoop.filehandling.fs.HdfsFileSystemProvider.exists(HdfsFileSystemProvider.java:225)

Thanks for support
And

sascha.wolke · June 5, 2023, 12:53pm

Hi @And,

Not sure how your user account is named, but does it start with numbers?

This might be related: [HDFS-4983] Numeric usernames do not work with WebHDFS FS - ASF JIRA

Cheers,
Sascha

And · June 5, 2023, 1:41pm

Hi Sasha

No start with sa_xxxxxx

thanks for support

And

sascha.wolke · June 5, 2023, 4:02pm

Hi @And,

There is a Hadoop option, called dfs.webhdfs.user.provider.user.pattern, that can be used to set a less restrictive pattern that matches your username (see the linked ticket above).

Cheers,
Sascha

And · June 7, 2023, 2:49pm

Hi Sasha,
We added this part to the hdfs-site.xml file

<property>
<name>dfs.webhdfs.user.provider.user.pattern</name>
<value>^[A-Za-z_][A-Za-z0-9@._-]*[$]?$</value>

but it still doesn’t work here’s what knime console returns me:

What can it be now that blocks this?
it’s random, probably if i restart knime it works.

thanks
And

system · June 14, 2023, 2:49pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.

sascha.wolke · June 29, 2023, 3:21pm

Hi @And,

It’s quite difficult to help you with those specific problems, without having any knowledge about your setup and/or any log files. Please open a support request instead.

Cheers,
Sascha