I’m trying to build a generic workflow which fetches all the files from S3 and then does computation on top of those and then uploads the changes files back to S3. In between I’m using python scripts for manipulation.
The main problem is that the python nodes don’t take in the path of files as flow variables even after creating a connection. As a workaround I’m using String input but that definitely doesn’t solve the purpose as my solution is not generic.
Also, to write the files back to S3 I’m using python script that uses boto3 but for that as well I need paths as my flow variable which are not available to me even after connecting.
Any help would be really appreciated. Kindly refer to the screenshot for your ready reference.