same problem CONTEXT LIVY

KNIME Extensions
,
1

HI,
I have a problem creating spark contexts due to an error on the Livy connection node.
Specifically, this error returns:

image

If I dig into the driver log I find this additional information:

23/10/19 15:44:16 DEBUG rpc.Rpc: [Rpc] Received RPC message: type=CALL id=4 payload=org.apache.livy.rsc.BaseProtocol$EndSession
23/10/19 15:44:16 DEBUG driver.RSCDriver: Shutting down due to EndSession request.

23/10/19 15:44:16 INFO repl.PythonInterpreter: Shutting down process

I found this topic already open which seems very similar to me.
Has it been resolved, if so, how?

Unable to create Spark Context (Livy) - KNIME Extensions - KNIME Community Forum

@andreis
Hi, please, have you solved it by any chance?

Thanks !!
And

2

HI
Is there any news regarding the request above?
Thank you !!!

And

3

Hi @And,

The mentioned error from the logs is very generic and sounds like there was a problem and the session was stopped.

Are you running Spark on YARN? The logs from the containers might contain some more logs. Was there anything in the Livy logs before the mentioned lines?

Cheers,
Sascha

1 Like
4

Hi,
Yes, we use Yarn.

This is the log Livy:
…
2023-09-29 15:31:31,055 INFO org.apache.livy.rsc.RSCClient: Received result for 29f7c00b-8888-4189-af71-9e6f8e1a4ab4
2023-09-29 15:31:31,599 ERROR org.apache.livy.server.SessionServlet$: internal error
org.eclipse.jetty.io.EofException: Early EOF
…

this is the expanded driver log:
…
23/10/19 15:44:16 DEBUG security.SaslRpcClient: unwrapping token of length:1550
23/10/19 15:44:16 DEBUG ipc.Client: IPC Client (951822991) connection to xxxxxx.xxxxxxxxxx from sa_xxxxx got value #20
23/10/19 15:44:16 DEBUG ipc.ProtobufRpcEngine: Call: getBlockLocations took 2ms
23/10/19 15:44:16 DEBUG hdfs.DataStreamer: DFSClient seqno: 6 reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 849738 flag: 0 flag: 0
23/10/19 15:44:16 DEBUG hdfs.DFSClient: newInfo = LocatedBlocks{; fileLength=1045; underConstruction=false; blocks=[LocatedBlock{BP-1194774438-10.210.45.20-1442928748731:blk_1201885799_128258504; getBlockSize()=1045; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.211.45.54:9866,DS-d7c3c363-f2f7-4005-9d66-d28f8bd9d97a,DISK], DatanodeInfoWithStorage[10.211.45.55:9866,DS-6b8fbd6c-08ed-4fb4-9b85-f04ae4575824,DISK]]}]; lastLocatedBlock=LocatedBlock{BP-1194774438-10.210.45.20-1442928748731:blk_1201885799_128258504; getBlockSize()=1045; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[10.211.45.54:9866,DS-d7c3c363-f2f7-4005-9d66-d28f8bd9d97a,DISK], DatanodeInfoWithStorage[10.211.45.55:9866,DS-6b8fbd6c-08ed-4fb4-9b85-f04ae4575824,DISK]]}; isLastBlockComplete=true; ecPolicy=null}
23/10/19 15:44:16 DEBUG hdfs.DFSClient: Connecting to datanode xx.xxxx.xxxx.xxxxxx
23/10/19 15:44:16 INFO spark.SparkContext: Added JAR hdfs://nameservice/user/sa_xxxxxx/.livy-sessions/6e85d114-32c0-437b-ad4f-b24a352d800c/livy-kryo-version-detector.jar at hdfs://nameservice/user/sa_xxxxxx/.livy-sessions/6e85d114-32c0-437b-ad4f-b24a352d800c/livy-kryo-version-detector.jar with timestamp 1697723056155
23/10/19 15:44:16 DEBUG driver.RSCDriver: Send job(a245e91e-7707-4fe1-b37d-4eef1aa7c9ad) result to Client.
23/10/19 15:44:16 DEBUG rpc.Rpc: [Rpc] Registered outstanding rpc 6 (org.apache.livy.rsc.BaseProtocol$JobResult).
23/10/19 15:44:16 DEBUG rpc.KryoMessageCodec: Encoded message of type org.apache.livy.rsc.rpc.Rpc$MessageHeader (22 bytes)
23/10/19 15:44:16 DEBUG rpc.KryoMessageCodec: Encoded message of type org.apache.livy.rsc.BaseProtocol$JobResult (102 bytes)
23/10/19 15:44:16 DEBUG rpc.KryoMessageCodec: Decoded message of type org.apache.livy.rsc.rpc.Rpc$MessageHeader (22 bytes)
23/10/19 15:44:16 DEBUG rpc.KryoMessageCodec: Decoded message of type org.apache.livy.rsc.rpc.Rpc$NullMessage (22 bytes)
23/10/19 15:44:16 DEBUG rpc.Rpc: [Rpc] Received RPC message: type=REPLY id=6 payload=org.apache.livy.rsc.rpc.Rpc$NullMessage
23/10/19 15:44:16 DEBUG hdfs.DFSOutputStream: enqueue full packet seqno: 7 offsetInBlock: 195584 lastPacketInBlock: false lastByteOffsetInBlock: 260096, src=/user/spark/spark2ApplicationHistory/application_1696743370355_4859_1.inprogress, bytesCurBlock=260096, blockSize=xxxxxx, appendChunk=false, blk_1201885795_128258500
23/10/19 15:44:16 DEBUG hdfs.DataStreamer: Queued packet seqno: 7 offsetInBlock: 195584 lastPacketInBlock: false lastByteOffsetInBlock: 260096, blk_1201885795_128258500
23/10/19 15:44:16 DEBUG hdfs.DFSClient: computePacketChunkSize: src=/user/spark/spark2ApplicationHistory/application_1696743370355_4859_1.inprogress, chunkSize=516, chunksPerPacket=126, packetSize=65016
23/10/19 15:44:16 DEBUG hdfs.DataStreamer: stage=DATA_STREAMING, blk_1201885795_128258500
23/10/19 15:44:16 DEBUG hdfs.DFSClient: WriteChunk allocating new packet seqno=8, src=/user/spark/spark2ApplicationHistory/application_1696743370355_4859_1.inprogress, packetSize=65016, chunksPerPacket=126, bytesCurBlock=260096, DFSOutputStream:blk_1201885795_128258500
23/10/19 15:44:16 DEBUG hdfs.DataStreamer: blk_1201885795_128258500 sending packet seqno: 7 offsetInBlock: 195584 lastPacketInBlock: false lastByteOffsetInBlock: 260096
23/10/19 15:44:16 DEBUG hdfs.DFSOutputStream: enqueue full packet seqno: 8 offsetInBlock: 260096 lastPacketInBlock: false lastByteOffsetInBlock: xxxxxxx1, src=/user/spark/spark2ApplicationHistory/application_1696743370355_4859_1.inprogress, bytesCurBlock=xxxxxxx1, blockSize=xxxxxx, appendChunk=false, blk_1201885795_128258500
23/10/19 15:44:16 DEBUG hdfs.DataStreamer: Queued packet seqno: 8 offsetInBlock: 260096 lastPacketInBlock: false lastByteOffsetInBlock: xxxxxxx1, blk_1201885795_128258500
23/10/19 15:44:16 DEBUG rpc.KryoMessageCodec: Decoded message of type org.apache.livy.rsc.rpc.Rpc$MessageHeader (22 bytes)
23/10/19 15:44:16 DEBUG hdfs.DFSClient: computePacketChunkSize: src=/user/spark/spark2ApplicationHistory/application_1696743370355_4859_1.inprogress, chunkSize=516, chunksPerPacket=126, packetSize=65016
23/10/19 15:44:16 DEBUG hdfs.DFSClient: WriteChunk allocating new packet seqno=9, src=/user/spark/spark2ApplicationHistory/application_1696743370355_4859_1.inprogress, packetSize=65016, chunksPerPacket=126, bytesCurBlock=xxxxxxx1, DFSOutputStream:blk_1201885795_128258500
23/10/19 15:44:16 DEBUG hdfs.DataStreamer: stage=DATA_STREAMING, blk_1201885795_128258500
23/10/19 15:44:16 DEBUG hdfs.DataStreamer: blk_1201885795_128258500 sending packet seqno: 8 offsetInBlock: 260096 lastPacketInBlock: false lastByteOffsetInBlock: xxxxxxx1
23/10/19 15:44:16 DEBUG hdfs.DataStreamer: DFSClient seqno: 7 reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 457168 flag: 0 flag: 0
23/10/19 15:44:16 DEBUG hdfs.DataStreamer: DFSClient seqno: 8 reply: SUCCESS reply: SUCCESS downstreamAckTimeNanos: 307682 flag: 0 flag: 0
23/10/19 15:44:16 DEBUG rpc.KryoMessageCodec: Decoded message of type org.apache.livy.rsc.BaseProtocol$xxxxxxx2 (182 bytes)
23/10/19 15:44:16 DEBUG rpc.Rpc: [Rpc] Received RPC message: type=CALL id=2 payload=org.apache.livy.rsc.BaseProtocol$xxxxxxx2
23/10/19 15:44:16 INFO driver.RSCDriver: Received bypass job request 7af2f966-37fb-49b7-accf-82d1b938e82e
23/10/19 15:44:16 DEBUG rpc.KryoMessageCodec: Encoded message of type org.apache.livy.rsc.rpc.Rpc$MessageHeader (22 bytes)
23/10/19 15:44:16 DEBUG rpc.KryoMessageCodec: Encoded message of type org.apache.livy.rsc.rpc.Rpc$NullMessage (22 bytes)
23/10/19 15:44:16 DEBUG rpc.KryoMessageCodec: Decoded message of type org.apache.livy.rsc.rpc.Rpc$MessageHeader (22 bytes)
23/10/19 15:44:16 DEBUG rpc.KryoMessageCodec: Decoded message of type org.apache.livy.rsc.BaseProtocol$GetBypassJobStatus (110 bytes)
23/10/19 15:44:16 DEBUG rpc.Rpc: [Rpc] Received RPC message: type=CALL id=3 payload=org.apache.livy.rsc.BaseProtocol$GetBypassJobStatus
23/10/19 15:44:16 DEBUG rpc.KryoMessageCodec: Encoded message of type org.apache.livy.rsc.rpc.Rpc$MessageHeader (22 bytes)
23/10/19 15:44:16 DEBUG rpc.KryoMessageCodec: Encoded message of type org.apache.livy.rsc.BypassJobStatus (70 bytes)
23/10/19 15:44:16 DEBUG rpc.KryoMessageCodec: Decoded message of type org.apache.livy.rsc.rpc.Rpc$MessageHeader (22 bytes)
23/10/19 15:44:16 DEBUG rpc.KryoMessageCodec: Decoded message of type org.apache.livy.rsc.BaseProtocol$EndSession (70 bytes)
23/10/19 15:44:16 DEBUG rpc.Rpc: [Rpc] Received RPC message: type=CALL id=4 payload=org.apache.livy.rsc.BaseProtocol$EndSession
23/10/19 15:44:16 DEBUG driver.RSCDriver: Shutting down due to EndSession request.
23/10/19 15:44:16 INFO repl.PythonInterpreter: Shutting down process
23/10/19 15:44:16 DEBUG yarn.ApplicationMaster: Number of pending allocations is 0. Slept for 3001/3000.
23/10/19 15:44:16 DEBUG yarn.ApplicationMaster: Sending progress
…

But how was the old ticket mentioned resolved?

Thanks
And

5

Hi @And,

There are more logs in the other contains running on YARN as part of the Livy Job. The executors might contain errors or details in the logs, and always a good thing to check.

Cheers,
Sascha

6

Hi sasha,
This is the log executor:

…
…
Container: container_e169_1696743370355_4859_01_000002 on xxxxxx11.sist.group_8041_1697723063195
LogAggregatiiiiiiiinType: AGGREGATED

LogType:stderr
LogLastModifiedTime:Thu Oct 19 15:44:21 +0200 2023
LogLength:17368
LogContents:
23/1111/19 15:44:1111 INFO executor.CoarseGrainedExecutorBackend: Started daemon with process name: 67888@xxxxxx1111
23/1111/19 15:44:1111 INFO util.SignalUtils: Registered signal handler for TERM
23/1111/19 15:44:1111 INFO util.SignalUtils: Registered signal handler for HUP
23/1111/19 15:44:1111 INFO util.SignalUtils: Registered signal handler for INT
23/1111/19 15:44:1111 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformatiiiiiiiin$UgiMetrics.loginSuccess with annotatiiiiiiiin @org.apache.hadoop.metrics2.annotatiiiiiiiin.Metric(always=false, sampleName=Ops, about=, type=DEFAULT, valueName=Time, value=[Rate of successful kerberos logins and latency (milliseconds)])
23/1111/19 15:44:1111 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformatiiiiiiiin$UgiMetrics.loginFailure with annotatiiiiiiiin @org.apache.hadoop.metrics2.annotatiiiiiiiin.Metric(always=false, sampleName=Ops, about=, type=DEFAULT, valueName=Time, value=[Rate of failed kerberos logins and latency (milliseconds)])
23/1111/19 15:44:1111 DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformatiiiiiiiin$UgiMetrics.getGroups with annotatiiiiiiiin @org.apache.hadoop.metrics2.annotatiiiiiiiin.Metric(always=false, sampleName=Ops, about=, type=DEFAULT, valueName=Time, value=[GetGroups])
23/1111/19 15:44:1111 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformatiiiiiiiin$UgiMetrics.renewalFailuresTotal with annotatiiiiiiiin @org.apache.hadoop.metrics2.annotatiiiiiiiin.Metric(always=false, sampleName=Ops, about=, type=DEFAULT, valueName=Time, value=[Renewal failures since startup])
23/1111/19 15:44:1111 DEBUG lib.MutableMetricsFactory: field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformatiiiiiiiin$UgiMetrics.renewalFailures with annotatiiiiiiiin @org.apache.hadoop.metrics2.annotatiiiiiiiin.Metric(always=false, sampleName=Ops, about=, type=DEFAULT, valueName=Time, value=[Renewal failures since last successful login])
23/1111/19 15:44:1111 DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics
23/1111/19 15:44:1111 DEBUG security.SecurityUtil: Setting hadoop.security.token.service.use_ip to true
23/1111/19 15:44:1111 DEBUG util.Shell: setsid exited with exit code 0
23/1111/19 15:44:1111 DEBUG security.Groups: Creating new Groups object
23/1111/19 15:44:1111 DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000; warningDeltaMs=5000
23/1111/19 15:44:1111 DEBUG deploy.SparkHadoopUtil: creating UGI for user: xxxxxx11112
23/1111/19 15:44:1111 DEBUG security.UserGroupInformatiiiiiiiin: hadoop login
23/1111/19 15:44:1111 DEBUG security.UserGroupInformatiiiiiiiin: hadoop login commit
23/1111/19 15:44:1111 DEBUG security.UserGroupInformatiiiiiiiin: using local user:UnixPrincipal: xxxxxx11112
23/1111/19 15:44:1111 DEBUG security.UserGroupInformatiiiiiiiin: Using user: “UnixPrincipal: xxxxxx11112” with name xxxxxx11112
23/1111/19 15:44:1111 DEBUG security.UserGroupInformatiiiiiiiin: User entry: “xxxxxx11112”
23/1111/19 15:44:1111 DEBUG security.UserGroupInformatiiiiiiiin: Reading credentials from locatiiiiiiiin set in HADOOP_TOKEN_FILE_LOCATION: /hadoop/disk1/yarn/nm/usercache/xxxxxx11112/appcache/applicatiiiiiiiin_1696743370355_4859/container_e169_1696743370355_4859_01_000002/container_tokens
23/1111/19 15:44:1111 DEBUG security.UserGroupInformatiiiiiiiin: Loaded 4 tokens
23/1111/19 15:44:1111 DEBUG security.UserGroupInformatiiiiiiiin: UGI loginUser:xxxxxx11112 (auth:SIMPLE)
23/1111/19 15:44:1111 DEBUG security.UserGroupInformatiiiiiiiin: PrivilegedActiiiiiiiin as:xxxxxx11112 (auth:SIMPLE) from:org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:65)
23/1111/19 15:44:1111 INFO spark.SecurityManager: Changing view acls to: xxxxxx11112
23/1111/19 15:44:1111 INFO spark.SecurityManager: Changing modify acls to: xxxxxx11112
*23/1111/19 15:44:1111 INFO spark.SecurityManager: Changing view acls groups to: *
*23/1111/19 15:44:1111 INFO spark.SecurityManager: Changing modify acls groups to: *
23/1111/19 15:44:1111 INFO spark.SecurityManager: SecurityManager: authenticatiiiiiiiin disabled; ui acls disabled; users with view permissiiiiiiiins: Set(xxxxxx11112); groups with view permissiiiiiiiins: Set(); users with modify permissiiiiiiiins: Set(xxxxxx11112); groups with modify permissiiiiiiiins: Set()
23/1111/19 15:44:1111 DEBUG logging.InternalLoggerFactory: Using SLF4J as the default logging framework
23/1111/19 15:44:1111 DEBUG internal.InternalThreadLocalMap: -Diiiiiiii.netty.threadLocalMap.stringBuilder.initialSize: 111124
23/1111/19 15:44:1111 DEBUG internal.InternalThreadLocalMap: -Diiiiiiii.netty.threadLocalMap.stringBuilder.maxSize: 4096
23/1111/19 15:44:1111 DEBUG channel.MultithreadEventLoopGroup: -Diiiiiiii.netty.eventLoopThreads: 128
23/1111/19 15:44:1111 DEBUG niiiiiiii.NiiiiiiiiEventLoop: -Diiiiiiii.netty.noKeySetOptimizatiiiiiiiin: false
23/1111/19 15:44:1111 DEBUG niiiiiiii.NiiiiiiiiEventLoop: -Diiiiiiii.netty.selectorAutoRebuildThreshold: 512
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent0: -Diiiiiiii.netty.noUnsafe: false
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent0: Java versiiiiiiiin: 8
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent0: sun.misc.Unsafe.theUnsafe: available
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent0: sun.misc.Unsafe.copyMemory: available
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent0: java.niiiiiiii.Buffer.address: available
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent0: direct buffer constructor: available
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent0: java.niiiiiiii.Bits.unaligned: available, true
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent0: jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable priiiiiiiir to Java9
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent0: java.niiiiiiii.DirectByteBuffer.(long, int): available
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent: sun.misc.Unsafe: available
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent: -Diiiiiiii.netty.tmpdir: /hadoop/disk1/yarn/nm/usercache/xxxxxx11112/appcache/applicatiiiiiiiin_1696743370355_4859/container_e169_1696743370355_4859_01_000002/tmp (java.iiiiiiii.tmpdir)
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent: -Diiiiiiii.netty.bitMode: 64 (sun.arch.data.model)
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent: -Diiiiiiii.netty.maxDirectMemory: 111129177344 bytes
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent: -Diiiiiiii.netty.uninitializedArrayAllocatiiiiiiiinThreshold: -1
23/1111/19 15:44:1111 DEBUG internal.CleanerJava6: java.niiiiiiii.ByteBuffer.cleaner(): available
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent: -Diiiiiiii.netty.noPreferDirect: false
23/1111/19 15:44:1111 DEBUG internal.PlatformDependent: org.jctools-core.MpscChunkedArrayQueue: available
23/1111/19 15:44:1111 DEBUG util.ResourceLeakDetector: -Diiiiiiii.netty.leakDetectiiiiiiiin.level: simple
23/1111/19 15:44:1111 DEBUG util.ResourceLeakDetector: -Diiiiiiii.netty.leakDetectiiiiiiiin.targetRecords: 4
23/1111/19 15:44:1111 DEBUG buffer.PooledByteBufAllocator: -Diiiiiiii.netty.allocator.numHeapArenas: 1111
23/1111/19 15:44:1111 DEBUG buffer.PooledByteBufAllocator: -Diiiiiiii.netty.allocator.numDirectArenas: 1111
23/1111/19 15:44:1111 DEBUG buffer.PooledByteBufAllocator: -Diiiiiiii.netty.allocator.pageSize: 8192
23/1111/19 15:44:1111 DEBUG buffer.PooledByteBufAllocator: -Diiiiiiii.netty.allocator.maxOrder: 11
23/1111/19 15:44:1111 DEBUG buffer.PooledByteBufAllocator: -Diiiiiiii.netty.allocator.chunkSize: 16777216
23/1111/19 15:44:1111 DEBUG buffer.PooledByteBufAllocator: -Diiiiiiii.netty.allocator.smallCacheSize: 256
23/1111/19 15:44:1111 DEBUG buffer.PooledByteBufAllocator: -Diiiiiiii.netty.allocator.normalCacheSize: 64
23/1111/19 15:44:1111 DEBUG buffer.PooledByteBufAllocator: -Diiiiiiii.netty.allocator.maxCachedBufferCapacity: 32768
23/1111/19 15:44:1111 DEBUG buffer.PooledByteBufAllocator: -Diiiiiiii.netty.allocator.cacheTrimInterval: 8192
23/1111/19 15:44:1111 DEBUG buffer.PooledByteBufAllocator: -Diiiiiiii.netty.allocator.cacheTrimIntervalMillis: 0
23/1111/19 15:44:1111 DEBUG buffer.PooledByteBufAllocator: -Diiiiiiii.netty.allocator.useCacheForAllThreads: true
23/1111/19 15:44:1111 DEBUG buffer.PooledByteBufAllocator: -Diiiiiiii.netty.allocator.maxCachedByteBuffersPerChunk: 111123
23/1111/19 15:44:1111 DEBUG client.TransportClientFactory: Creating new connectiiiiiiiin to xxxxxx14.sist.group/1111.311.11.54:36189
23/1111/19 15:44:1111 DEBUG channel.DefaultChannelId: -Diiiiiiii.netty.processId: 67888 (auto-detected)
23/1111/19 15:44:1111 DEBUG util.NetUtil: -Djava.net.preferIPv4Stack: false
23/1111/19 15:44:1111 DEBUG util.NetUtil: -Djava.net.preferIPv6Addresses: false
23/1111/19 15:44:1111 DEBUG util.NetUtilInitializatiiiiiiiins: Loopback interface: lo (lo, 127.0.0.1)
23/1111/19 15:44:1111 DEBUG util.NetUtil: /proc/sys/net/core/somaxconn: 4096
23/1111/19 15:44:1111 DEBUG channel.DefaultChannelId: -Diiiiiiii.netty.machineId: 34:80:0d:ff:fe:dd:f3:64 (auto-detected)
23/1111/19 15:44:1111 DEBUG buffer.ByteBufUtil: -Diiiiiiii.netty.allocator.type: pooled
23/1111/19 15:44:1111 DEBUG buffer.ByteBufUtil: -Diiiiiiii.netty.threadLocalDirectBufferSize: 0
23/1111/19 15:44:1111 DEBUG buffer.ByteBufUtil: -Diiiiiiii.netty.maxThreadLocalCharBufferSize: 16384
23/1111/19 15:44:11 DEBUG buffer.AbstractByteBuf: -Diiiiiiii.netty.buffer.checkAccessible: true
23/1111/19 15:44:11 DEBUG buffer.AbstractByteBuf: -Diiiiiiii.netty.buffer.checkBounds: true
23/1111/19 15:44:11 DEBUG util.ResourceLeakDetectorFactory: Loaded default ResourceLeakDetector: iiiiiiii.netty.util.ResourceLeakDetector@3c5c112a
23/1111/19 15:44:11 DEBUG client.TransportClientFactory: Connectiiiiiiiin to xxxxxx14.sist.group/1111.311.11.54:36189 successful, running bootstraps…
23/1111/19 15:44:11 INFO client.TransportClientFactory: Successfully created connectiiiiiiiin to xxxxxx14.sist.group/1111.311.11.54:36189 after 71 ms (0 ms spent in bootstraps)
23/1111/19 15:44:11 DEBUG util.Recycler: -Diiiiiiii.netty.recycler.maxCapacityPerThread: 4096
23/1111/19 15:44:11 DEBUG util.Recycler: -Diiiiiiii.netty.recycler.maxSharedCapacityFactor: 2
23/1111/19 15:44:11 DEBUG util.Recycler: -Diiiiiiii.netty.recycler.linkCapacity: 16
23/1111/19 15:44:11 DEBUG util.Recycler: -Diiiiiiii.netty.recycler.ratiiiiiiii: 8
23/1111/19 15:44:11 DEBUG util.Recycler: -Diiiiiiii.netty.recycler.delayedQueue.ratiiiiiiii: 8
23/1111/19 15:44:11 INFO deploy.SparkHadoopUtil: Updating delegatiiiiiiiin tokens for current user.
23/1111/19 15:44:11 DEBUG deploy.SparkHadoopUtil: Adding/updating delegatiiiiiiiin tokens List(Kind: HBASE_AUTH_TOKEN, Service: e69237d6-4dfc-46f7-b546-b97b91dc1696, Ident: ((username=xxxxxx11112, keyId=1698, issueDate=1697723036201, expiratiiiiiiiinDate=1698327836201, sequenceNumber=973)); (username=xxxxxx11112, keyId=1698, issueDate=1697723036201, expiratiiiiiiiinDate=1698327836201, sequenceNumber=973), Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { applicatiiiiiiiin_id { id: 4859 cluster_timestamp: 1696743370355 } attemptId: 1 } keyId: 989769412); appAttemptId { applicatiiiiiiiin_id { id: 4859 cluster_timestamp: 1696743370355 } attemptId: 1 } keyId: 989769412, Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:nameservice, Ident: (token for xxxxxx11112: HDFS_DELEGATION_TOKEN owner=xxxxxx11112, renewer=yarn, realUser=livy/sgsvrashads13lx.sist.group@SISTEMI.GROUP, issueDate=1697723027929, maxDate=1698327827929, sequenceNumber=1969023, masterKeyId=3304); token for xxxxxx11112: HDFS_DELEGATION_TOKEN owner=xxxxxx11112, renewer=yarn, realUser=livy/sgsvrashads13lx.sist.group@SISTEMI.GROUP, issueDate=1697723027929, maxDate=1698327827929, sequenceNumber=1969023, masterKeyId=3304; Renewer: yarn; Issued: 1111/19/23 3:43 PM; Max Date: 1111/26/23 3:43 PM, Kind: kms-dt, Service: kms://https@sgsvrashads13lx.sist.group;sgsvrashads14lx.sist.group:9494/kms, Ident: (kms-dt owner=xxxxxx11112, renewer=yarn, realUser=livy, issueDate=1697723028812, maxDate=1698327828812, sequenceNumber=139699, masterKeyId=681); kms-dt owner=xxxxxx11112, renewer=yarn, realUser=livy, issueDate=1697723028812, maxDate=1698327828812, sequenceNumber=139699, masterKeyId=681; Renewer: yarn; Issued: 1111/19/23 3:43 PM; Max Date: 1111/26/23 3:43 PM)
23/1111/19 15:44:11 INFO spark.SecurityManager: Changing view acls to: xxxxxx11112
23/1111/19 15:44:11 INFO spark.SecurityManager: Changing modify acls to: xxxxxx11112
*23/1111/19 15:44:11 INFO spark.SecurityManager: Changing view acls groups to: *
*23/1111/19 15:44:11 INFO spark.SecurityManager: Changing modify acls groups to: *
23/1111/19 15:44:11 INFO spark.SecurityManager: SecurityManager: authenticatiiiiiiiin disabled; ui acls disabled; users with view permissiiiiiiiins: Set(xxxxxx11112); groups with view permissiiiiiiiins: Set(); users with modify permissiiiiiiiins: Set(xxxxxx11112); groups with modify permissiiiiiiiins: Set()
23/1111/19 15:44:11 DEBUG spark.SparkEnv: Using serializer: class org.apache.spark.serializer.KryoSerializer
23/1111/19 15:44:11 DEBUG client.TransportClientFactory: Creating new connectiiiiiiiin to xxxxxx14.sist.group/1111.311.11.54:36189
23/1111/19 15:44:11 DEBUG client.TransportClientFactory: Connectiiiiiiiin to xxxxxx14.sist.group/1111.311.11.54:36189 successful, running bootstraps…
23/1111/19 15:44:11 INFO client.TransportClientFactory: Successfully created connectiiiiiiiin to xxxxxx14.sist.group/1111.311.11.54:36189 after 1 ms (0 ms spent in bootstraps)
23/1111/19 15:44:11 INFO storage.DiskBlockManager: Created local directory at /hadoop/disk3/yarn/nm/usercache/xxxxxx11112/appcache/applicatiiiiiiiin_1696743370355_4859/blockmgr-2ed3ab3f-2011-4f82-8877-c028253d5221
23/1111/19 15:44:11 INFO storage.DiskBlockManager: Created local directory at /hadoop/disk8/yarn/nm/usercache/xxxxxx11112/appcache/applicatiiiiiiiin_1696743370355_4859/blockmgr-d3014a2a-0bd1-49a7-8dc9-ede3316d1383
23/1111/19 15:44:11 INFO storage.DiskBlockManager: Created local directory at /hadoop/disk5/yarn/nm/usercache/xxxxxx11112/appcache/applicatiiiiiiiin_1696743370355_4859/blockmgr-a1208d47-fc32-4693-8fd9-150fb5276543
23/1111/19 15:44:11 INFO storage.DiskBlockManager: Created local directory at /hadoop/disk7/yarn/nm/usercache/xxxxxx11112/appcache/applicatiiiiiiiin_1696743370355_4859/blockmgr-054dbf6f-d468-4af0-8344-14011cddbfec
23/1111/19 15:44:11 INFO storage.DiskBlockManager: Created local directory at /hadoop/disk4/yarn/nm/usercache/xxxxxx11112/appcache/applicatiiiiiiiin_1696743370355_4859/blockmgr-ac18b28a-c41c-4006-92e9-f9f1318f2c07
23/1111/19 15:44:11 INFO storage.DiskBlockManager: Created local directory at /hadoop/disk6/yarn/nm/usercache/xxxxxx11112/appcache/applicatiiiiiiiin_1696743370355_4859/blockmgr-b2c4603d-5533-46e5-abbc-6cec5873338b
23/1111/19 15:44:11 INFO storage.DiskBlockManager: Created local directory at /hadoop/disk1/yarn/nm/usercache/xxxxxx11112/appcache/applicatiiiiiiiin_1696743370355_4859/blockmgr-dde6e503-911a-4d97-9de6-a3688447c4d7
23/1111/19 15:44:11 INFO storage.DiskBlockManager: Created local directory at /hadoop/disk2/yarn/nm/usercache/xxxxxx11112/appcache/applicatiiiiiiiin_1696743370355_4859/blockmgr-f38fc670-0364-474a-83c1-e1cef2edee9c
23/1111/19 15:44:11 DEBUG storage.DiskBlockManager: Adding shutdown hook
23/1111/19 15:44:11 DEBUG util.ShutdownHookManager: Adding shutdown hook
23/1111/19 15:44:11 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
23/1111/19 15:44:11 INFO executor.CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGraineXXXXXXXX@xxxxxx14.sist.group:36189
23/1111/19 15:44:11 INFO executor.CoarseGrainedExecutorBackend: Successfully registered with driver
23/1111/19 15:44:11 INFO executor.Executor: Starting executor ID 1 on host xxxxxx11.sist.group
23/1111/19 15:44:11 DEBUG server.TransportServer: Shuffle server started on port: xxxxxx11111
23/1111/19 15:44:11 INFO util.Utils: Successfully started service ‘org.apache.spark.network.netty.NettyBlockTransferService’ on port xxxxxx11111.
23/1111/19 15:44:11 INFO netty.NettyBlockTransferService: Server created on xxxxxx11.sist.group:xxxxxx11111
23/1111/19 15:44:11 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicatiiiiiiiinPolicy for block replicatiiiiiiiin policy
23/1111/19 15:44:11 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(1, xxxxxx11.sist.group, xxxxxx11111, None)
23/1111/19 15:44:11 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(1, xxxxxx11.sist.group, xxxxxx11111, None)
23/1111/19 15:44:11 INFO storage.BlockManager: external shuffle service port = 7337
23/1111/19 15:44:11 INFO storage.BlockManager: Registering executor with local external shuffle service.
23/1111/19 15:44:11 DEBUG client.TransportClientFactory: Creating new connectiiiiiiiin to xxxxxx11.sist.group/xxxxxx11114
23/1111/19 15:44:11 DEBUG client.TransportClientFactory: Connectiiiiiiiin to xxxxxx11.sist.group/xxxxxx11114 successful, running bootstraps…
23/1111/19 15:44:11 INFO client.TransportClientFactory: Successfully created connectiiiiiiiin to xxxxxx11.sist.group/xxxxxx11114 after 1 ms (0 ms spent in bootstraps)
23/1111/19 15:44:11 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(1, xxxxxx11.sist.group, xxxxxx11111, None)
23/1111/19 15:44:11 INFO executor.Executor: Using REPL class URI: spark://xxxxxx14.sist.group:36189/classes
23/1111/19 15:44:21 INFO executor.CoarseGrainedExecutorBackend: Driver commanded a shutdown
23/1111/19 15:44:21 INFO memory.MemoryStore: MemoryStore cleared
23/1111/19 15:44:21 INFO storage.BlockManager: BlockManager stopped
23/1111/19 15:44:21 INFO util.ShutdownHookManager: Shutdown hook called
23/1111/19 15:44:21 DEBUG util.ShutdownHookManager: Completed shutdown in 0.003 seconds; Timeouts: 0
23/1111/19 15:44:21 DEBUG util.ShutdownHookManager: ShutdownHookManger completed shutdown.

End of LogType:stderr

…
…

I managed to upload all the logs. can you give me a hand?

I need to understand what’s happening

thanks !!!

7

Let me know for the other ticket as well

8

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.