I have a local big data environment running a spark job, I want to use the spark to parquet node which is connected to S3. My S3 connection is setup and working via the Default credential provider chain, but I need some way of supplying the default credential provider chain to the local spark environment.
Is there a way to do this with the local big data environment, e.g. access to the core-site.xml. And what is the options to set to pickup the credentials chain?