problem with rabbit mq

joracab · September 30, 2024, 6:44pm

Hi i ahve a knime server over aws machine medium size. there are workflows running for over one year suddenly the ser ver is blocking and i have found the following errors

Blockquote2024-09-25 18:12:28,845 : WARN : AMQP Connection 127.0.0.1:5672 : : RabbitMQHandler : : : Connection to message queue lost, attempting to re-establish connections
2024-09-25 18:13:42,588 : ERROR : main : : ProfileManager : : : Could not download profiles from http://localhost:8080/knime/rest/v4/profiles/contents: Connect to localhost:8080 [localhost/127.0.0.1] failed: Connect timed out. Will use existing but potentially outdated profiles.
org.apache.http.conn.ConnectTimeoutException: Connect to localhost:8080 [localhost/127.0.0.1] failed: Connect timed out
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.knime.product.profiles.ProfileManager.downloadProfiles(ProfileManager.java:356)
at org.knime.product.profiles.ProfileManager.fetchProfileContents(ProfileManager.java:271)
at org.knime.product.profiles.ProfileManager.applyProfiles(ProfileManager.java:171)
at com.knime.enterprise.executor.KNIMEExecutorApplication.start(KNIMEExecutorApplication.java:80)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:136)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:402)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:659)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:596)
at org.eclipse.equinox.launcher.Main.run(Main.java:1467)
at org.eclipse.equinox.launcher.Main.main(Main.java:1440)

Blockquote2024-09-25 19:40:04,317 : ERROR : KNIME-Worker-127-Rule Engine (Dictionary) 21:1056:854:855:1010 : 1f48b34e-f9b6-4c09-9d06-d8465d3c38c4 : Node : String Manipulation : 21:1056:854:855:1029 : Configure failed (ClassFormatError): Truncated class file2024-09-25 21:18:05,429 : ERROR : pool-1-thread-39 : 43263684-d413-4282-8b1d-9c3ceda737a4 : DiscardJobTask : : : Could not stop job dispatcher for job 43263684-d413-4282-8b1d-9c3ceda737a4: null
java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:140)
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:136)
at com.rabbitmq.client.impl.ChannelN.basicCancel(ChannelN.java:1528)
at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicCancel(AutorecoveringChannel.java:650)
at com.knime.enterprise.executor.amqp.RabbitMQMessageConsumer.stop(RabbitMQMessageConsumer.java:265)
at com.knime.enterprise.executor.amqp.AbstractMessageDispatcher.stop(AbstractMessageDispatcher.java:211)
at com.knime.enterprise.executor.amqp.actions.DiscardJobTask.stopJobDispatcher(DiscardJobTask.java:113)
at com.knime.enterprise.executor.amqp.actions.DiscardJobTask.perform(DiscardJobTask.java:102)
at com.knime.enterprise.executor.amqp.RabbitMQMessageConsumer$Consumer.processMessage(RabbitMQMessageConsumer.java:133)
at com.knime.enterprise.executor.amqp.RabbitMQMessageConsumer$Consumer.handleDelivery(RabbitMQMessageConsumer.java:107)
at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149)
at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.rabbitmq.client.ShutdownSignalException: connection error
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:552)
at com.rabbitmq.client.impl.ChannelN.basicCancel(ChannelN.java:1522)
… 12 more
Caused by: com.rabbitmq.client.MissedHeartbeatException: Heartbeat missing with heartbeat = 60 seconds
at com.rabbitmq.client.impl.AMQConnection.handleSocketTimeout(AMQConnection.java:869)
at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:769)
at com.rabbitmq.client.impl.AMQConnection.access$400(AMQConnection.java:48)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:688)
… 1 more
2024-09-25 21:18:05,429 : ERROR : pool-1-thread-38 : badf8d1d-b915-4080-a52b-66bad6f3cae3 : DiscardJobTask : : : Could not stop job dispatcher for job badf8d1d-b915-4080-a52b-66bad6f3cae3: null
java.io.IOException
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:140)
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:136)
at com.rabbitmq.client.impl.ChannelN.basicCancel(ChannelN.java:1528)
at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicCancel(AutorecoveringChannel.java:650)
at com.knime.enterprise.executor.amqp.RabbitMQMessageConsumer.stop(RabbitMQMessageConsumer.java:265)
at com.knime.enterprise.executor.amqp.AbstractMessageDispatcher.stop(AbstractMessageDispatcher.java:211)
at com.knime.enterprise.executor.amqp.actions.DiscardJobTask.stopJobDispatcher(DiscardJobTask.java:113)
at com.knime.enterprise.executor.amqp.actions.DiscardJobTask.perform(DiscardJobTask.java:102)
at com.knime.enterprise.executor.amqp.RabbitMQMessageConsumer$Consumer.processMessage(RabbitMQMessageConsumer.java:133)
at com.knime.enterprise.executor.amqp.RabbitMQMessageConsumer$Consumer.handleDelivery(RabbitMQMessageConsumer.java:107)
at com.rabbitmq.client.impl.ConsumerDispatcher$5.run(ConsumerDispatcher.java:149)
at com.rabbitmq.client.impl.ConsumerWorkService$WorkPoolRunnable.run(ConsumerWorkService.java:111)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.rabbitmq.client.ShutdownSignalException: connection error
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66)
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:552)
at com.rabbitmq.client.impl.ChannelN.basicCancel(ChannelN.java:1522)
… 12 more
Caused by: com.rabbitmq.client.MissedHeartbeatException: Heartbeat missing with heartbeat = 60 seconds
at com.rabbitmq.client.impl.AMQConnection.handleSocketTimeout(AMQConnection.java:869)
at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:769)
at com.rabbitmq.client.impl.AMQConnection.access$400(AMQConnection.java:48)
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:688)

Cna any one helpme as i understand rammit mq is just for distributed enviroments. If not please provide adavice on how to handle this error because after this error appears only viable solution rigth now is to restart server

thor · October 1, 2024, 9:56am

The errors in the executor indicate that the KNIME Server process isn’t running any more. Neither the embedded message broker is available nor can the executor get customization profiles. Please check whether the process is running and look for errors in the server logs.

joracab · October 3, 2024, 9:40pm

There is another error we recieve from an specific job run the erros appears below :

	 DBConnectionManager :  :  : An error occurred while closing the database connection  IO Error: Connection timed out java.sql.SQLRecoverableException

is there a way to debug this type of error?

NDekay · October 4, 2024, 7:32pm

All you can tell from that is that the DB Connection Manager’s connection to whatever database it was connected to at the time had an I/O error - specifically, that the connection timed out. Maybe there’s an upper limit on how long a connection can be open before it gets shut down from the remote side?

We’d need more context to help debug that, really. Either executor logs (if that’s where it was taken from ), or maybe review of the workflow. And if it’s being closed from the remote side then you’d want to review the logs from the remote server to see what they say about why it was closed.