Knime and Amazon S3 Connection with AWS Session token

Hi, 

we have temporary access to S3 specifiied in $HOME/.aws/credentials by 

aws_access_key_id = .....

aws_secret_access_key = ......

aws_session_token = ...................................

Accessing a bucket via aws s3 ls bucket works

 

I defined a Amazon S3 connection within Knime and specified to use the "default credentials provider chan" 

but this fails with Access Denied (403)

 

How can I configure Knime to use the aws_session_token ?

 

Thank you 

 

We want to add some information to this....
The try with "default credentials provider chain"  was already described

Whenever we trying the Access Key Id and Secret Key we are also failing because it seems, that Knime take the key id as part of the S3 bucket address...

s3://<AccessKeyId>@eu-central-1 fails
 

Any ideas on that ?

I am also trying to connect to an S3 resource using my credentials that are protected by multi-factor authentication. It seems important that the Amazon S3 Connection node support MFA.

Hi,

If you select the Default Credential Provider Chain local ENV variables or the credentials file should work. Looking into the aws-sdk-java, session tokens should be supported too, but i have not tested it so far. Do you use IAM roles?

This is only the representation inside of KNIME. What Node do you use access S3 after the S3 Connection node?

This should work with the Default Credentials Provider Chain. Can you test this with ENV variables (AWS_ACCESS_KEY_ID, AWS_SECRET_KEY and AWS_SESSION_TOKEN)?

1 Like

This sounds great! I will try again on Monday.

Reviving this very old thread :slight_smile:
I am trying to build an automated process to convert some files into Parquet files and then load them straight in an S3 bucket.
In order to do this, I have a few nodes that get the temporary access tokens:
aws_access_key_id, aws_secret_access_key and aws_session_token.

Unfortunately, I cannot pass these values automatically to the Amazon Authentication Node via variables, as there are no equivalent Variables defined in the node. I have tried assigning the values to the Credentials, username and session token fields, but no luck.

image

Is there any way around this as dumping the values in a file then manually copy pasting isn’t very … automated?
Thank you.

Hi @MMarius,

Welcome to the KNIME community!

Did you select the Access Key ID and Secret Key option in the dialog and activated the use Session Token checkbox? If this does not help, maybe create a new thread with the error message.

Cheers,
Sascha

Hello Sascha

Yes I have

image

As stated above, the problem is that I can’t pass the data I receive from the Authentication nodes to the Amazon Authentication node.
At the end of the chain in the first picture, it dumps the current logon information into a CSV file from where I have to manually copy paste the values in the Amazon Authentication node because I can’t find a Access Key ID or a Secret Key mapping in the Flow Variables tab.
Thank you.

Hi @MMarius,
You can convert flow variables with two strings for username and password to a Credentials flow variable with the Variable to Credentials node (Variable to Credentials – KNIME Hub). Does that help?
Kind regards,
Alexander

That part is done, but what do I map it to?
Again, I have 3 values that I receive and need to set:
Access Key ID
Secret Key
Session Token

image

So, the Metanode (details in original post, first pic) retrieves the 3 values.
As you can see, now I dump them in a CSV.
The Table Row to Variable Node takes care on the conversion to Flow Variables, BUT, and this is the key, WHAT do I map it to?
If I map to Credentials and Username, it doesn’t work and I don’t have Access Key ID or Secret Key mappings in the Node.
Thank you.

Hi,
It does not work with the Table Row to Variable node, because credentials are protected in a special fashion. You will need the Variable to Credentials node to convert Access Key ID and Secret Key to a special credentials flow variable (Access Key ID = username, Secret Key = password). Then you can select the option “Credentials” in the dialog (currently grayed out for you because there is no credentials flow var). The session token you can map as-is in the Flow Variables tab.
Kind regards,
Alexander

2 Likes

That worked.
Awesome!
Thank you very much.

2 Likes

Hello,
we just release version 4.7 of the KNIME Analytics Platform which now supports AWS session tokens.
For an overview of all the new features have a look at the What’s new page. For all the details go to the changelog.
Bye
Tobias

4 Likes

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.