Hey there,
It’s my first post in the Knime Forum, I will try to explain my issue.
Node : new S3 connector
Context : we need to sync a folder with S3 bucket’s vendor. It gives me:
The access key + access secret key id (I have used to connect through the Amazon authentication node)
The bucket and the folder name
I guess I have permission only to explore folders/files belonging to the Bucket that the vendor has created for me.
Issue : when I try to configure my working directory using the Amazon s3 connector > Browse… it’s returning a 403 error.
See below my error message:point_down:
WARN NioFile Could not list files in ‘/’: Access Denied (Service: S3, Status Code: 403, Request ID: 4Q5T3K7…, Extended Request ID: GJkZzqbT6HZF9ZWmEQCY…)
Hi @Diego_Rodriguez
welcome to the KNIME forum!
I guess I have permission only to explore folders/files belonging to the Bucket that the vendor has created for me.
Well that explains it
If all you want to do is browse inside that one bucket where you have access, then you need to manually set /bucketname as the working directory.
Also please note that strictly speaking you can just leave the working directory at the root (“/”), you don’t need to set it in order to read/write files in that bucket. The working directory is only a convenience feature that allows you to work with relative paths more easily. More information can be found in our documentation:https://docs.knime.com/latest/analytics_platform_file_handling_guide/index.html#working-directory
4 Likes
Thanks for the warming welcome @bjoern.lohrmann
I have tried leaving the root ("/") path in the working directory string as you suggested (‘Normalize Paths’ flagged
But when I try to put my personal path (bucket and folder created for me) in the CSV writer-node connected to the S3-node (see below):
and then inside the CSV node I fill the file path with my personal bucket and folder (the one that the vendor configure for me) is still giving a 403 error (see below 2):
Instead, I was able to upload a file from my local machine to the S3 vendor bucket using AWS CLI command-line tool
But I prefer doing with Knime because I can integrate multiple data wrangler operations in one workflow
Hey @bjoern.lohrmann or @knime team any news on this issue?
It will be very helpful to be able to use the S3 connector
Hi @Diego_Rodriguez ,
welcome to the forum
Can you double check that you do not have any typos in the bucket name? In your screenshot, the bucket name contains dots and this is not recommended, but should work.
If this does not help, can you enable debug logging in KNIME and post the log lines here (go to the preferences KNIME > KNIME GUI):
Hola @sascha.wolke
Thanks again for your help and support.
But unfortunately, there is no typo in my bucket name
Below you can find
Logs when I have reset the S3 connector node putting /bucketname in the ‘Workning directory’ field:
DEBUG NodeContainerEditPart CSV Writer 3:5 (CONFIGURED)https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 0 of 50; total allocated: 0 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]s3.amazonaws.com/52.217.46.198:443 https://s3.amazonaws.com:443 ] can be kept alive for 60.0 secondshttps://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ] can be kept alive for 60.0 secondshttps://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]
when I try to write a test .csv file in my bucket through the ‘CSV Writer’ node:
DEBUG PoolingHttpClientConnectionManager Closing connections idle longer than 60000 MILLISECONDShttps://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]s3.amazonaws.com/52.217.169.0:443 https://s3.amazonaws.com:443 ] can be kept alive for 60.0 secondshttps://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 0 of 50; total allocated: 0 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 0 of 50; total allocated: 0 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]s3.amazonaws.com/52.217.169.0:443 https://s3.amazonaws.com:443 ] can be kept alive for 60.0 secondshttps://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 0 of 50; total allocated: 0 of 50]https://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]s3.amazonaws.com/52.217.169.0:443 https://s3.amazonaws.com:443 ] can be kept alive for 60.0 secondshttps://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ] can be kept alive for 60.0 secondshttps://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ] can be kept alive for 60.0 secondshttps://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 2 of 50; total allocated: 2 of 50]s3.amazonaws.com/52.217.169.0:443 https://s3.amazonaws.com:443 ] can be kept alive for 60.0 secondshttps://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 2 of 50; total allocated: 2 of 50]https://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]s3.amazonaws.com/52.217.79.150:443 https://s3.amazonaws.com:443 ] can be kept alive for 60.0 secondshttps://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ] can be kept alive for 60.0 secondshttps://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ] can be kept alive for 60.0 secondshttps://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ][total kept alive: 0; route allocated: 1 of 50; total allocated: 1 of 50]https://s3.amazonaws.com:443 ] can be kept alive for 60.0 secondshttps://s3.amazonaws.com:443 ][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50]
Hope I am sharing the correct info with you and it could be useful.
Thanks again!
Hi @Diego_Rodriguez , could it be that you are initiating the S3 connection too early to the point that it’s idle long enough for the connection to close?
Can you try linking your Node 20 to your Node 1 with a flow variable link? (Notice the direction, from Node 20 to Node 1, meaning flow variable output of Node 20 (right side) to flow variable input of Node 1 (left side)), that way the workflow will ensure that Node 1 is executed only after Node 20 is done.
1 Like
system
October 13, 2021, 1:45pm
8
This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.
When you start browsing at the root (“/”), you need permission to get the list of buckets which apparently you don’t have.
