Connection to Azure Gen2 data lake with SAS key

Dear Knimers,

I am trying to connect to Azure using the Microsoft Authentication node with the "Shared access signiture (SAS) authentication (Azure Storage only)" Option.

I get the below error msg, when I click the browse button on the Azure Data lake Storage Gen2 Connector before executing this node.

If i run the node,i get the below error:

ERROR Azure Data Lake Storage Gen2 Connector 4:2 Execute failed: Server encountered an internal error. Please try again after some time.

If I consume the SAS URL through my web browser, it works.

I am using the latest 4.7 release. Thanks in advance for your help!

@AlexanderFillbrunn is it possible that you can help with this, please? thanks in advance

Hi Bashar,
Internal Server Error is an error occurring in the backend, so in Azure Gen2 and I am not sure if KNIME can affect this in any way. Do you see more information about the error in the KNIME log via View → Open KNIME Log?
Kind regards,
Alexander

Thanks,Alexander for your reply, attached is the relevant part of the log.
Gen2.txt (29.0 KB)
Kind regards,
Bashar

Hi Bashar,
Thanks for the log files. Unfortunately they do not contain additional information. The error occurs when the node checks if you are successfully authenticated, but the server only returns the error code. Generally, error codes starting with 4 mean that the client sent the request in the wrong fashion, but error codes starting with 5 indicate a problem on the server side. Maybe you can inquire with your cloud team if they can help.
Kind regards,
Alexander

Hi Alexander,
Thanks again for your help on this, I have checked with my cloud team and they are wondering since I am able to access the ADLS from my laptop with the same SAS URL ( consuming this with chrome and IE) That I am using in Knime, while Knime is sending its requests to the cloud, will it use it will use any other IP than my local IP?? and is there way to check what IP knime is using to communicate with the Cloud?

the reason behind this question is that the datalake is currently open for my internet IP only.

thanks again.
Kind regards,
Bashar

Hi @AlexanderFillbrunn ,
we communicated with Microsoft on this issue and we have the following diagnosis :

##########################
While using a user-delegated SAS token to grant access to a single container, this wouldn’t allow your application to perform operations outside of that specific container e.g. any management-level operation should fail with a 403-status code, this is expected behavior Create a user delegation SAS - Azure Storage | Microsoft Learn

KNIME Request:

From the request above, we can see KNIME trying to perform a list operation on the account List Containers (REST API) - Azure Storage | Microsoft Learn . With the SAS token granted to KNIME on the container level, this request would fail as it does not have enough permission to perform the list operation. From our replication on KNIME, we could see the application returning the error below:

#####################

we are having Knime as part of our enterprise architicture, Is there a way to access the resources in the lake with the SAS token?

Much appreciated.

Kind regards,
Bashar

Hi @Bashar,
Does the error just occur in the dialog when you want to browse? I think browsing might call the list operation, as it needs you to select a container. What if you enter a file path manually without browsing? Does that work?
Kind regards,
Alexander

Hi Alexander,
The error happens when when browsing and when putting a folder path.
When granted the rights to list containers, both options work. But this violates our security policy as the SAS is very specific for what I can consume.

I am available to go through this over a zoom call to demonstrate.

Kind Regards,
Bashar

Hi Bashar,
I just tried it with my colleagues and it worked fine. They will contact you regarding the issue and set up a call. If you can, please update us here as well so the community knows the solution :slight_smile:
Kind regards,
Alex