Good morning,
I’m starting Knime flow jobs with scheduling, both connecting and running the flow have always worked well for me.
My system group installed @channel encryption in the application yesterday, since then the streams go wrong with the following error:
…
was aborted by the software in your host machine (SocketException)
2023-09-26 00:09:18,657 : ERROR : KNIME-Worker-6-Create Spark Context (Livy) 3:2114:1460 : : Node : Create Spark Context (Livy) : 3:2114:1460 : Execute failed: An established connection was aborted by the software in your host machine (SocketException)
java.util.concurrent.ExecutionException: java.net.SocketException: An established connection was aborted by the software in your host machine
at java.base/java.util.concurrent.FutureTask.report(Unknown Source)
at java.base/java.util.concurrent.FutureTask.get(Unknown Source)
at org.knime.bigdata.spark.core.livy.context.LivySparkContext.waitForFuture(LivySparkContext.java:495)
at org.knime.bigdata.spark.core.livy.context.LivySparkContext.uploadJobJar(LivySparkContext.java:467)
at org.knime.bigdata.spark.core.livy.context.LivySparkContext.open(LivySparkContext.java:329)
at org.knime.bigdata.spark.core.context.SparkContext.ensureOpened(SparkContext.java:145)
at org.knime.bigdata.spark.core.livy.node.create.LivySparkContextCreatorNodeModel2.executeInternal(LivySparkContextCreatorNodeModel2.java:85)
at org.knime.bigdata.spark.core.node.SparkNodeModel.execute(SparkNodeModel.java:240)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:549)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1267)
at org.knime.core.node.Node.execute(Node.java:1041)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:595)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:201)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:117)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:367)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:221)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
Caused by: java.net.SocketException: An established connection was aborted by the software in your host machine
at java.base/sun.nio.ch.NioSocketImpl.implWrite(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.write(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl$2.write(Unknown Source)
at java.base/java.net.Socket$SocketOutputStream.write(Unknown Source)
at java.base/sun.security.ssl.SSLSocketOutputRecord.deliver(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source)
at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
at org.apache.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:136)
at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:167)
at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113)
at org.apache.http.entity.mime.content.FileBody.writeTo(FileBody.java:121)
at org.apache.http.entity.mime.AbstractMultipartForm.doWriteTo(AbstractMultipartForm.java:134)
at org.apache.http.entity.mime.AbstractMultipartForm.writeTo(AbstractMultipartForm.java:157)
at org.apache.http.entity.mime.MultipartFormEntity.writeTo(MultipartFormEntity.java:113)
at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
…
Good morning,
I’m launching Knime flow jobs with scheduling, both the connection and flow execution have always worked well for me.
My systems group ran on the @channel encryption application, from that moment the flows go wrong with the following error:
Are you running the scheduled jobs using the KNIME Hub or Server?
The error message seems very generic and sounds like something with the connection goes wrong. Maybe can you check the Livy logs, verify that Livy is reachable and/or possibly restart Livy?
Hi Sasha, Thanks for the support,
I tried to restart Livy but without success.
The server is ours. I’ll give you some details, after the @channel encryption by systems groups I modified the cacerts file under the path
…\Program Files\KNIME\plugins\org.knime.binary.jre.win32.x86_64_17.0.3.20220429\jre\lib\security\
this is the debug’s log knime:
2023-09-29 11:03:17,172 : WARN : KNIME-Database-Connection-Closer-1 : : DBConnectionManager : : : An error occurred while closing the database connection [URL=“jdbc:hive2://hadop-hivesvil.intranet.servizi:10000/”]: Error while cleaning up the server resources
java.sql.SQLException: Error while cleaning up the server resources
at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:641)
at org.knime.database.connection.wrappers.AbstractConnectionWrapper.close(AbstractConnectionWrapper.java:170)
at org.knime.database.connection.impl.DBConnectionManager.lambda$1(DBConnectionManager.java:198)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.apache.thrift.transport.TTransportException
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:376)
at org.apache.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:453)
at org.apache.thrift.transport.TSaslTransport.read(TSaslTransport.java:435)
at org.apache.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:37)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
at org.apache.hadoop.hive.thrift.TFilterTransport.readAll(TFilterTransport.java:62)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hive.service.cli.thrift.TCLIService$Client.recv_CloseSession(TCLIService.java:179)
at org.apache.hive.service.cli.thrift.TCLIService$Client.CloseSession(TCLIService.java:166)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1309)
at jdk.proxy19/jdk.proxy19.$Proxy48.CloseSession(Unknown Source)
at org.apache.hive.jdbc.HiveConnection.close(HiveConnection.java:639)
… 5 more
2023-09-29 11:03:19,154 : WARN : KNIME-Worker-24-Hive Connector 3:2114:1459 : : DefaultDBMetadata : Hive Connector : 3:2114:1459 : Couldn’t fetch SQL keywords from database.
java.sql.SQLException: Method not supported
at org.apache.hive.jdbc.HiveDatabaseMetaData.getSQLKeywords(HiveDatabaseMetaData.java:547)
at org.knime.database.connection.wrappers.AbstractDatabaseMetaDataWrapper.getSQLKeywords(AbstractDatabaseMetaDataWrapper.java:222)
at org.knime.database.connection.wrappers.AbstractDatabaseMetaDataWrapper.getSQLKeywords(AbstractDatabaseMetaDataWrapper.java:222)
at org.knime.database.connection.wrappers.AbstractDatabaseMetaDataWrapper.getSQLKeywords(AbstractDatabaseMetaDataWrapper.java:222)
at org.knime.database.connection.wrappers.AbstractDatabaseMetaDataWrapper.getSQLKeywords(AbstractDatabaseMetaDataWrapper.java:222)
at org.knime.database.session.impl.DefaultDBMetadata.fetchSQLKeywords(DefaultDBMetadata.java:87)
at org.knime.database.session.impl.DefaultDBMetadata.(DefaultDBMetadata.java:144)
at org.knime.database.session.impl.DefaultDBSession.(DefaultDBSession.java:229)
at org.knime.database.session.impl.DBSessionFactory.create(DBSessionFactory.java:81)
at org.knime.database.session.DBSessionCache.getOrCreate(DBSessionCache.java:151)
at org.knime.database.node.connector.AbstractDBConnectorNodeModel.registerSession(AbstractDBConnectorNodeModel.java:389)
at org.knime.database.node.connector.AbstractDBConnectorNodeModel.execute(AbstractDBConnectorNodeModel.java:219)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:549)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1267)
at org.knime.core.node.Node.execute(Node.java:1041)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:595)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:201)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:117)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:367)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:221)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
2023-09-29 11:03:51,660 : ERROR : KNIME-Worker-25-Create Spark Context (Livy) 3:2114:1460 : : Node : Create Spark Context (Livy) : 3:2114:1460 : Execute failed: An established connection was aborted by the software in your host machine (SocketException)
java.util.concurrent.ExecutionException: java.net.SocketException: An established connection was aborted by the software in your host machine
at java.base/java.util.concurrent.FutureTask.report(Unknown Source)
at java.base/java.util.concurrent.FutureTask.get(Unknown Source)
at org.knime.bigdata.spark.core.livy.context.LivySparkContext.waitForFuture(LivySparkContext.java:495)
at org.knime.bigdata.spark.core.livy.context.LivySparkContext.uploadJobJar(LivySparkContext.java:467)
at org.knime.bigdata.spark.core.livy.context.LivySparkContext.open(LivySparkContext.java:329)
at org.knime.bigdata.spark.core.context.SparkContext.ensureOpened(SparkContext.java:145)
at org.knime.bigdata.spark.core.livy.node.create.LivySparkContextCreatorNodeModel2.executeInternal(LivySparkContextCreatorNodeModel2.java:85)
at org.knime.bigdata.spark.core.node.SparkNodeModel.execute(SparkNodeModel.java:240)
at org.knime.core.node.NodeModel.executeModel(NodeModel.java:549)
at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1267)
at org.knime.core.node.Node.execute(Node.java:1041)
at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:595)
at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:95)
at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:201)
at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:117)
at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:367)
at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:221)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)
Caused by: java.net.SocketException: An established connection was aborted by the software in your host machine
at java.base/sun.nio.ch.NioSocketImpl.implWrite(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl.write(Unknown Source)
at java.base/sun.nio.ch.NioSocketImpl$2.write(Unknown Source)
at java.base/java.net.Socket$SocketOutputStream.write(Unknown Source)
at java.base/sun.security.ssl.SSLSocketOutputRecord.deliver(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl$AppOutputStream.write(Unknown Source)
at org.apache.http.impl.io.SessionOutputBufferImpl.streamWrite(SessionOutputBufferImpl.java:124)
at org.apache.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:136)
at org.apache.http.impl.io.SessionOutputBufferImpl.write(SessionOutputBufferImpl.java:167)
at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:113)
at org.apache.http.entity.mime.content.FileBody.writeTo(FileBody.java:121)
at org.apache.http.entity.mime.AbstractMultipartForm.doWriteTo(AbstractMultipartForm.java:134)
at org.apache.http.entity.mime.AbstractMultipartForm.writeTo(AbstractMultipartForm.java:157)
at org.apache.http.entity.mime.MultipartFormEntity.writeTo(MultipartFormEntity.java:113)
at org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:156)
at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:238)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.apache.livy.client.http.LivyConnection.executeRequest(LivyConnection.java:292)
at org.apache.livy.client.http.LivyConnection.access$000(LivyConnection.java:68)
at org.apache.livy.client.http.LivyConnection$3.run(LivyConnection.java:277)
at java.base/java.security.AccessController.doPrivileged(Unknown Source)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.livy.client.http.LivyConnection.sendRequest(LivyConnection.java:274)
at org.apache.livy.client.http.LivyConnection.post(LivyConnection.java:228)
at org.apache.livy.client.http.HttpClient$3.call(HttpClient.java:256)
at org.apache.livy.client.http.HttpClient$3.call(HttpClient.java:253)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
--------------------------------------------------------------------------------- This is the log that comes from the hadoop world ----------------------------------------------------------------------------------
23/09/29 14:41:56 WARN util.Utils: spark.executor.instances less than spark.dynamicAllocation.minExecutors is invalid, ignoring its setting, please update your configs.
23/09/29 14:41:56 INFO util.Utils: Using initial executors = 1, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
23/09/29 14:41:56 INFO conf.Configuration: resource-types.xml not found
23/09/29 14:41:56 INFO resource.ResourceUtils: Unable to find ‘resource-types.xml’.
23/09/29 14:41:56 INFO yarn.YarnAllocator: Will request 1 executor container(s), each with 1 core(s) and 1408 MB memory (including 384 MB of overhead)
23/09/29 14:41:56 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@xxxxxxxxxxx)
23/09/29 14:41:56 INFO yarn.YarnAllocator: Submitted 1 unlocalized container requests.
23/09/29 14:41:56 INFO yarn.ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
23/09/29 14:41:57 INFO yarn.YarnAllocator: Launching container container_e64_1695541571491_24487_01_000002 on host sxxxxx for executor with ID 1
23/09/29 14:41:57 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
23/09/29 14:42:02 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.209.45.41:50788) with ID 1
23/09/29 14:42:02 INFO dynalloc.ExecutorMonitor: New executor 1 has registered (new total is 1)
23/09/29 14:42:02 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
23/09/29 14:42:02 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
23/09/29 14:42:02 INFO driver.SparkEntries: Spark context finished initialization in 8963ms
23/09/29 14:42:02 INFO conf.HiveConf: Found configuration file file:/hadoop/disk18/yarn/nm/usercache/sa_mkcho-prod/filecache/86/spark_conf.zip/hadoop_conf/hive-site.xml
23/09/29 14:42:02 INFO storage.BlockManagerMasterEndpoint: Registering block manager xxxxxxx with 366.3 MB RAM, BlockManagerId(1, xxxxxxx, 34912, None)
23/09/29 14:42:02 INFO driver.SparkEntries: Created Spark session (with Hive support).
23/09/29 14:42:08 INFO driver.RSCDriver: Received job request c761fd39-d5ae-4e02-a23b-633cd3e1e397
23/09/29 14:42:09 INFO driver.SparkEntries: Created HiveContext.
23/09/29 14:42:09 WARN spark.SparkContext: Using an existing SparkContext; some configuration may not take effect.
23/09/29 14:42:09 INFO spark.SparkContext: Added file /tmp/tmpvpnhuY/livy/livy-kryo-version-detector.jar at spark://xxxxxxxxxx:37422/files/livy-kryo-version-detector.jar with timestamp 1695991329238
23/09/29 14:42:09 INFO util.Utils: Copying /tmp/tmpvpnhuY/livy/livy-kryo-version-detector.jar to /hadoop/disk20/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-9cae3dde-38f9-470b-8865-97263fef9035/userFiles-f4b1b76b-4de7-4d69-af70-ed2a500809e3/livy-kryo-version-detector.jar
23/09/29 14:42:09 INFO spark.SparkContext: Added JAR hdfs://nameservice/user/sa_mkcho-prod/.livy-sessions/76f982c7-7f84-47e8-8d03-6f96fd15b7b7/livy-kryo-version-detector.jar at hdfs://nameservice/user/sa_mkcho-prod/.livy-sessions/76f982c7-7f84-47e8-8d03-6f96fd15b7b7/livy-kryo-version-detector.jar with timestamp 1695991329280
23/09/29 14:42:09 INFO driver.RSCDriver: Received bypass job request efca6596-290a-449c-9270-489e27eaebf4
23/09/29 14:42:09 INFO repl.PythonInterpreter: Shutting down process
23/09/29 14:42:14 INFO server.AbstractConnector: Stopped Spark@13be0720{HTTP/1.1, (http/1.1)}{0.0.0.0:0}
23/09/29 14:42:14 INFO ui.SparkUI: Stopped Spark web UI at hxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
23/09/29 14:42:14 INFO yarn.YarnAllocator: Driver requested a total number of 0 executor(s).
23/09/29 14:42:14 INFO cluster.YarnClusterSchedulerBackend: Shutting down all executors
23/09/29 14:42:14 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
23/09/29 14:42:14 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
23/09/29 14:42:14 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
23/09/29 14:42:14 INFO memory.MemoryStore: MemoryStore cleared
23/09/29 14:42:14 INFO storage.BlockManager: BlockManager stopped
23/09/29 14:42:14 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
23/09/29 14:42:14 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
23/09/29 14:42:14 INFO spark.SparkContext: Successfully stopped SparkContext
23/09/29 14:42:14 INFO repl.PythonInterpreter: process has been shut down
23/09/29 14:42:14 INFO spark.SparkContext: SparkContext already stopped.
23/09/29 14:42:14 INFO spark.SparkContext: SparkContext already stopped.
23/09/29 14:42:14 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
23/09/29 14:42:14 WARN channel.AbstractChannelHandlerContext: An exception ‘java.lang.IllegalArgumentException: not existed channel:[id: 0x66a6c7c9, L:0.0.0.0/0.0.0.0:10000 ! R:/10.209.45.15:36492]’ [enable DEBUG level for full stacktrace] was thrown by a user handler’s exceptionCaught() method while handling the following exception:
java.lang.IllegalArgumentException: not existed channel:[id: 0x66a6c7c9, L:0.0.0.0/0.0.0.0:10000 ! R:/10.209.45.15:36492]
at org.apache.livy.rsc.rpc.RpcDispatcher.getRpc(RpcDispatcher.java:66)
at org.apache.livy.rsc.rpc.RpcDispatcher.channelInactive(RpcDispatcher.java:84)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
at org.apache.livy.rsc.rpc.Rpc$5.channelInactive(Rpc.java:245)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:389)
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:354)
at io.netty.handler.codec.ByteToMessageCodec.channelInactive(ByteToMessageCodec.java:118)
at org.apache.livy.rsc.rpc.KryoMessageCodec.channelInactive(KryoMessageCodec.java:104)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:497)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.lang.Thread.run(Thread.java:748)
23/09/29 14:42:14 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED
23/09/29 14:42:14 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
23/09/29 14:42:14 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://nameservice/user/sa_mkcho-prod/.sparkStaging/application_1695541571491_24487
23/09/29 14:42:14 INFO util.ShutdownHookManager: Shutdown hook called
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk3/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-5fa3f7ab-bd3e-4168-8f24-c4ab3c68195f/pyspark-a7633ce9-bbd9-47cd-bdd9-2799a248ebc6
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk12/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-33573dd4-f9b6-4fd3-9800-16766d559c69
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk14/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-08dbc4f5-4fbe-418a-95ad-ca69edac73cf
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk13/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-27c9df23-4fc3-4bf7-8313-f1865adef3df
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk6/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-a5f83191-aba1-49c4-8d20-9d6a427fd4c2
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk1/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-f0f47a7f-34ea-443c-84e9-721e56ff8e50
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk8/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-6a9fb14e-1b94-4809-9d7f-e50e0fd13809
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk16/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-8f7d83f7-63da-45ff-a825-93ffeb41d01b
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk9/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-a3b9d559-c451-4f96-acdd-bfd83ccc04cc
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk18/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-d7bd600d-9f65-4a69-97b3-80882bf3df77
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk20/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-9cae3dde-38f9-470b-8865-97263fef9035
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk22/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-60bb8b44-4143-42a2-9ab6-6624d6825352
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk15/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-d3692d3a-4b70-4b34-9c70-e899ec989926
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk3/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-5fa3f7ab-bd3e-4168-8f24-c4ab3c68195f
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk17/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-9a0ffddd-d1ad-43d2-aeda-464532de7df0
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk7/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-be5931fe-7ab8-468b-990c-4bcd90bec48c
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk4/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-339f6bb2-2fe3-400e-86ff-c152adfd16f5
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk11/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-d7bbd770-39a0-48f3-ba55-06f61df7281a
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk19/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-e903c842-4e8e-438a-aed8-b24fc11ea7ad
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk2/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-eb0f7c9d-9ee3-4a7e-9d43-dab152bce2bc
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk21/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-0ed05247-33c5-42d1-b934-87e374863678
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk5/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-a120479a-a4c5-42cc-908d-1c25e13de876
23/09/29 14:42:14 INFO util.ShutdownHookManager: Deleting directory /hadoop/disk10/yarn/nm/usercache/sa_mkcho-prod/appcache/application_1695541571491_24487/spark-d3787951-53a8-4482-bd75-d80cb3a6f481
would also like to know if there are any custom spark settings on the Knime Livy node that I can try
Hi Sasha,
Knime is installed on a Windows server, I’m not using knime hub.
More or less a .bat starts the knime flow.
While Hadoop is installed on another server (Linux) that I connect to, and livy has been restarted there.
Yes, the problem occurs after installing new certificates.
I use other flows that also use certificates (e.g. Amazon s3) and they have no problems.
2023-09-29 15:31:28,345 INFO org.apache.livy.server.interactive.InteractiveSession: Interactive session 475 created appid: application_1695541571491_24539, owner: sa_XXXXXX, state: idle, kind: shared, info: {driverLogUrl=https://xxxxxxxxxxxxx/node/containerlogs/container_e64_1695541571491_24539_01_000001/sa_xxxxxxxxx, sparkUiUrl=[https://xxxxxxxx.xxxxxxx.group:xxxx/proxy/application_1695541571491_24539/}]
2023-09-29 15:31:31,055 INFO org.apache.livy.rsc.RSCClient: Received result for 29f7c00b-8888-4189-af71-9e6f8e1a4ab4
2023-09-29 15:31:31,599 ERROR org.apache.livy.server.SessionServlet$: internal error
org.eclipse.jetty.io.EofException: Early EOF
at org.eclipse.jetty.server.HttpInput$3.getError(HttpInput.java:1187)
at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:1175)
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:333)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at org.eclipse.jetty.util.ReadLineInputStream.read(ReadLineInputStream.java:134)
at org.eclipse.jetty.util.MultiPartInputStreamParser.parse(MultiPartInputStreamParser.java:747)
at org.eclipse.jetty.util.MultiPartInputStreamParser.getParts(MultiPartInputStreamParser.java:475)
at org.eclipse.jetty.server.MultiParts$MultiPartsUtilParser.getParts(MultiParts.java:112)
at org.eclipse.jetty.server.Request.getParts(Request.java:2437)
at org.eclipse.jetty.server.Request.getParts(Request.java:2421)
One question, is it possible to load libraries onto the knime livy node (therefore client side) to be used by livy by overwriting those present on the server?
One question, is it possible to load libraries onto the knime livy node (therefore client side) to be used by livy by overwriting those present on the server?
Not sure what that means. The Livy KNIME node contains only a REST client, LIvy itself runs on the Server side.
