we are having a problem configuring the Azure Blob Storage Connector. Unfortunately, we are not able to get it to work; we always get the error message “Authentication failed, or the account doesn’t have enough permissions to list containers” during execution of the node (in the KNIME console). When we then try to access the node via e.g. a .csv reader, we get an “Access denied: /: Access denied” message.
yes, we tried the Azure Data Lake Storage Gen2 Connector node. If we use a SAS token that is valid for the whole storage account the blob storage we want to connect to is part of, that node works to access files. The problem we have with that, though, is that the SAS token authorizes for ALL blobs in the account, which from a security perspective is not practicable
we have seen Sasha Wolke’s response concerning the hierarchical namespace, but could not figure out how that might help us
yes, the blob storage roles associated with the SAS token for it have read, write and list permissions; we have literally given that role ALL permissions there are to be “checked” in Azure
we are using KNIME 4.4
we registered KNIME with Office 365 so that it is, e.g., able to access files in SharePoint. Since the Gen2 node works, it does not seem as if we need to register KNIME somewhere in Azure in order to be able to access files there (right?)
we tried to configure the blob storage node both with a “/” in the working directory folder field and with a “/TestFolder/” entry (after creating such a folder directly in the blob storage using the Azure Storage Explorer)
Maybe someone out there has an idea what we might be doing wrong?
If you need screenshots or more detailed descriptions, please ask!
Thanks to everyone who takes their time to read this,
Jan Schuppius
Deutsche Afrika-Linien / John T. Essberger
Hamburg, Germany
Hello Julian
we have not heard back from anyone and are still having issues.
Currently, we are building up a data lake + data pipeline architecture for our company with Azure + KNIME, and using the Blob Storage node is a central part of many of our intended processes.
We would really appreciate input.
If such a thing is an option, we would also pay for an hour of an expert’s time to set this straight for us.
Best regards,
Jan Schuppius
We have made some more experimenting, and think we have an idea what the problem is. Please find attached a screenshot with a data scientist’s approach to finding a solution
What’s happening, is that when the Azure Blob Storage Connector node executes, it performs a test to verify that the connection is successful and that the credentials are correct. Right now, this operation is to list all containers in the storage account. If this test fails, you will see the particular warning you are getting ("“Authentication failed, or the account doesn’t have enough permissions to list containers”"). However, the connector node still successfully executes and becomes green because there are some corner cases like yours where actually some things might still work.
In your case, what should work is file browsing inside the container, that your SAS token has permissions for. Unfortunately, due to a bug on our side, it seems that you cannot browse in the container root, but you have to start in a subfolder of the container (*). Hence with your credentials, the browsing behavior is as follows:
Browsing starts at…
Result
/ (i.e. storage account root)
“Access Denied” (to be expected. You don’t have permissions to list containers)
/test1 (i.e. container root)
Should work, but “Access Denied” due to bug
/test1/myfolder
Works
You can define in which location the file browser window opens by:
Setting a working directory in the Blob Storage Connector.
or by typing/pasting a path such as /test1/myfolder into the File/Folder textfield of the CSV Reader. If you leave the field empty, browsing will start in the working directory (defined by the connector).
Furthermore, what should also work, is reading file(s) directly by typing/pasting the path in CSV Reader, for example /test1/file.csv.
We will try to fix the bug with the next release (4.5.0 in Dec’21, or next bugfix release on the 4.4.x release line).
Hope this helps,
Björn
(*) Reason: When browsing a container root, we are currently calling this REST API method to fetch metadata for the container. Unfortunately this REST call is only allowed when you have read permissions for the whole storage account (which your SAS token has not).
Hello Björn,
the nodes behave exactly as you describe above, and I was able to get access to a file in a blob container using the method of creating a folder first in which to put it.
Thank you very much for your precise input!
Best regards from Hamburg,
Jan Schuppius
