Start Spark Context (Livy) not working

Hi all.

I was trying to initiate a spark context in my cluster using the Livy node but with no success. To this Livy node I’m giving as input a HDFS connection, which is working, but when I run the Livy node I get the following error:

ERROR Create Spark Context (Livy) 0:5        Execute failed: org.apache.hadoop.security.AccessControlException: Permission denied: user=livy, access=EXECUTE, inode="/user/hdfs":hdfs:hdfs:drwx------
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:399)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:315)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:242)
at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkDefaultEnforcer(RangerHdfsAuthorizer.java:589)
at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer$RangerAccessControlEnforcer.checkPermission(RangerHdfsAuthorizer.java:350)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:193)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1857)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1841)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1800)
at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.resolvePathForStartFile(FSDirWriteFileOp.java:315)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2407)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2351)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:774)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:462)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)

sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:278)
org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1211)
org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1190)
org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1128)
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:537)
org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:534)
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:548)
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:475)
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1118)
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1098)
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:987)
org.apache.hadoop.fs.FileSystem.create(FileSystem.java:975)
org.knime.bigdata.spark.core.livy.jobapi.StagingArea.createUploadStream(StagingArea.java:122)
org.knime.bigdata.spark.core.livy.jobapi.StagingAreaUtil.lambda$0(StagingAreaUtil.java:39)
org.knime.bigdata.spark.core.livy.jobapi.LivyJobSerializationUtils.serializeObjectsToStream(LivyJobSerializationUtils.java:147)
org.knime.bigdata.spark.core.livy.jobapi.StagingAreaUtil.toSerializedMap(StagingAreaUtil.java:37)
org.knime.bigdata.spark2_3.base.LivySparkJob.call(LivySparkJob.java:106)
org.knime.bigdata.spark2_3.base.LivySparkJob.call(LivySparkJob.java:1)
org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:40)
org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:27)
org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:57)
org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:42)
org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:27)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745) (RuntimeException)

Edit: Sorry about this error being all weird but I can’t seem to format it properly.

It seems as though it is trying to use a user called livy that does not have access. I then tried to do a curl to the same IP using:

curl -X POST --data ‘{“kind”: “pyspark”, “proxyUser”:“livy”}’ -H “Content-Type: application/json” http://ip-address/sessions -H “Sec-Fetch-Mode: cors” -H “Accept-Encoding: gzip, deflate, br” -H “Content-Type: application/json;charset=UTF-8” -H "Accept: application/json, text/plain, / " -H “Connection: keep-alive” -H “X-Requested-By: livy”

curl -X POST -d “{"code":"1 + 1"}” -H “Content-Type: application/json” http://ip-address/sessions/10/statements -H “Connection: keep-alive” -H “X-Requested-By: livy”
{“id”:1,“code”:“1 + 1”,“state”:“running”,“output”:null,“progress”:0.0}
curl http://ip-address:8999/sessions/10/statements/0 {“id”:0,“code”:“1 + 1”,“state”:“available”,“output”:{“status”:“ok”,“execution_count”:0,“data”:{“text/plain”:“2”}},“progress”:1.0}.

This apparently ran with success so I’m not sure what the problem is in KNIME…

Hope someone can help.

I believe this is a duplicate of the issue you posted in the other thread. I’m closing this one to keeps things tidy.

1 Like