I am trying to connect to a remote Spark cluster (via Livy) and provide it access to an S3 bucket that is in the same VPC. The direct connection between our EMR cluster and the S3 does not require credentials, as it is behind a protected VPN.
When adding an Amazon S3 Connection it asks to use my local AWS credentials, which are protected by multi-factor authentication.
Regardless, I don’t want to transfer any of the S3 data to my laptop, but rather would have Spark access S3 directly in the remote cluster.
How can I create a Spark Context (Livy) that uses the remote S3 file system directly, without using my local laptop/credentials?