I’m running spark nodes on KNIME 4.0.0 using EMR 5.23 (Hive 2.3.4, Spark 2.4.0, Livy 0.5.0). In order to connect I use S3 Connection node, when creating Spark Context it gives an error:
ERROR Create Spark Context (Livy) 0:2637 Execute failed: Remote file system upload test failed: java.lang.UnsupportedOperationException: openOutputStream must not be used for S3 connections.
Create Spark Context node gets to 70-80% and stops on openOutputStream error.
Perhaps on older KNIME version 3.7.2 it doesn’t give any error and works well.
- Staging area is set on Spark Context for S3 ( tried to configure for local - same result)
- Access keys are set up correctly as S3 Connection passes
How can I solve this problem?