Azure and local Spark

Hi,
I am trying to use the “Create Local Big Data Environment” with the “Azure Blob Store Connection” to process some Avro File. I’ve added in the Custom Spark Settings the spark.jars: /path/hadoop-azure-2.7.0.jar, /path/azure-storage-4.0.0.jar but I still get error Execute failed: No FileSystem for scheme: wasb (IOException) when I try to run it, any suggestion?

image

1 Like

Hi @madape,
the Local Big Data Environment is not set up to connect to Azure Blob Storage, so I am not sure you can get it working, but let us try.
Adding the jars, would also be my first suggestion to you.
You would also have to add some spark configs:

" spark.hadoop.fs.azure", "org.apache.hadoop.fs.azure.NativeAzureFileSystem"
" spark.hadoop.fs.AbstractFileSystem.wasb.impl", "org.apache.hadoop.fs.azure.Wasb"
and
" spark.hadoop.fs.azure.account.key.YOUR_ACCOUNT.blob.core.windows.net", "YOUR_KEY"

I hope that will do the trick.
best Mareike

1 Like

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.