Knime Server job disabled automatically after 3 failed attempts to execute

We have a Knime Server scheduled job that runs on an hourly interval. The job was running correctly but suddenly disabled itself. After inspecting the logs the job had failed to execute 3 times and was then disabled.

Here is an extract from the log:

15-Nov-2018 13:47:24.787 INFO [KNIME-Job-Scheduler_1] com.knime.enterprise.server.jobs.ScheduledJobManagerImpl.schedule Rescheduling execution of ‘Workflow ‘/Marketplace/PO OIs Incoming’; target name = PO OIs Incoming; next execution at 2018-11-16T13:45+02:00[Africa/Harare], repeating every day; User: hilton.rossenrode; ID: a442ad80-9e8c-4f3d-91ca-f07729c97d59 (failed 1 times)’
15-Nov-2018 14:00:00.001 INFO [KNIME-Job-Scheduler_1] com.knime.enterprise.server.jobs.ScheduledJobManagerImpl.runWorkflow Timer expired for scheduled execution ‘Workflow ‘/StockCartCheck’; resetting; discarding after execution; target name = StockCartCheck; next execution at 2018-11-15T14:15+02:00[Africa/Johannesburg], repeating every 15 minutes; User: johann.vandermerwe; ID: bd20bb71-5ba3-4d2e-9708-67038bbeda3b’.
15-Nov-2018 14:00:00.001 INFO [KNIME-Job-Scheduler_1] com.knime.enterprise.server.jobs.ScheduledJobManagerImpl.runWorkflow Schedule ‘Workflow ‘/StockCartCheck’; resetting; discarding after execution; target name = StockCartCheck; next execution at 2018-11-15T14:15+02:00[Africa/Johannesburg], repeating every 15 minutes; User: johann.vandermerwe; ID: bd20bb71-5ba3-4d2e-9708-67038bbeda3b’ is currently disabled, not executing it.
15-Nov-2018 14:00:00.001 INFO [KNIME-Job-Scheduler_1] com.knime.enterprise.server.jobs.ScheduledJobManagerImpl.schedule Schedule ‘Workflow ‘/StockCartCheck’; resetting; discarding after execution; target name = StockCartCheck; next execution at 2018-11-15T14:15+02:00[Africa/Johannesburg], repeating every 15 minutes; User: johann.vandermerwe; ID: bd20bb71-5ba3-4d2e-9708-67038bbeda3b’ is currently disabled
15-Nov-2018 14:00:00.001 INFO [KNIME-Job-Scheduler_1] com.knime.enterprise.server.jobs.ScheduledJobManagerImpl.runWorkflow Timer expired for scheduled execution ‘Workflow ‘/Promotions/PO Pull’; resetting; discarding after execution; target name = PO Pull; next execution at 2018-11-15T15:00+02:00[Africa/Harare], repeating every hour; User: johann.vandermerwe; ID: e1701251-28c6-415f-870e-f4ef49af864b (failed 2 times)’.
15-Nov-2018 14:00:00.001 INFO [KNIME-Job-Scheduler_1] com.knime.enterprise.server.jobs.WorkflowJobManagerImpl.loadWorkflow Loading workflow ‘/Promotions/PO Pull’ for user ‘johann.vandermerwe’
15-Nov-2018 14:00:00.008 INFO [KNIME-Job-Scheduler_1] com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.startNewRMI Launching KNIME Executor…
15-Nov-2018 14:03:56.226 SEVERE [KNIME-Job-Scheduler_1] com.knime.enterprise.server.executor.rmi.RMIExecutorImpl.startSlave Version check for KNIME RMI slave failed: error during JRMP connection establishment; nested exception is:
java.net.SocketTimeoutException: Read timed out
java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
java.net.SocketTimeoutException: Read timed out
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:304)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:129)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179)
at com.sun.proxy.$Proxy142.getVersion(Unknown Source)
at com.knime.enterprise.server.executor.rmi.RMIExecutorImpl.checkVersion(RMIExecutorImpl.java:430)
at com.knime.enterprise.server.executor.rmi.RMIExecutorImpl.startSlave(RMIExecutorImpl.java:383)
at com.knime.enterprise.server.executor.rmi.RMIExecutorImpl.(RMIExecutorImpl.java:222)
at com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.startNewRMI(RMIExecutorManagerImpl.java:310)
at com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.getExecutorInstance(RMIExecutorManagerImpl.java:392)
at com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.getExecutorInstance(RMIExecutorManagerImpl.java:63)
at com.knime.enterprise.server.jobs.WorkflowJobManagerImpl.loadWorkflow(WorkflowJobManagerImpl.java:595)
at com.knime.enterprise.server.jobs.ScheduledJobManagerImpl.loadAndExecute(ScheduledJobManagerImpl.java:316)
at com.knime.enterprise.server.jobs.ScheduledJobManagerImpl.runWorkflow(ScheduledJobManagerImpl.java:368)
at com.knime.enterprise.server.jobs.ScheduledJobManagerImpl$ScheduledJobRunnable.run(ScheduledJobManagerImpl.java:101)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at java.io.DataInputStream.readByte(DataInputStream.java:265)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:246)
… 22 more

15-Nov-2018 14:03:56.227 WARNING [KNIME-Job-Scheduler_1] com.knime.enterprise.server.jobs.ScheduledJobManagerImpl.runWorkflow Scheduled execution of ‘/Promotions/PO Pull/e1701251-28c6-415f-870e-f4ef49af864b’ failed: Version check for KNIME RMI slave failed: error during JRMP connection establishment; nested exception is:
java.net.SocketTimeoutException: Read timed out
com.knime.enterprise.utility.ExecutorException: Version check for KNIME RMI slave failed: error during JRMP connection establishment; nested exception is:
java.net.SocketTimeoutException: Read timed out
at com.knime.enterprise.server.executor.rmi.RMIExecutorImpl.startSlave(RMIExecutorImpl.java:387)
at com.knime.enterprise.server.executor.rmi.RMIExecutorImpl.(RMIExecutorImpl.java:222)
at com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.startNewRMI(RMIExecutorManagerImpl.java:310)
at com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.getExecutorInstance(RMIExecutorManagerImpl.java:392)
at com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.getExecutorInstance(RMIExecutorManagerImpl.java:63)
at com.knime.enterprise.server.jobs.WorkflowJobManagerImpl.loadWorkflow(WorkflowJobManagerImpl.java:595)
at com.knime.enterprise.server.jobs.ScheduledJobManagerImpl.loadAndExecute(ScheduledJobManagerImpl.java:316)
at com.knime.enterprise.server.jobs.ScheduledJobManagerImpl.runWorkflow(ScheduledJobManagerImpl.java:368)
at com.knime.enterprise.server.jobs.ScheduledJobManagerImpl$ScheduledJobRunnable.run(ScheduledJobManagerImpl.java:101)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
java.net.SocketTimeoutException: Read timed out
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:304)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:129)
at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227)
at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179)
at com.sun.proxy.$Proxy142.getVersion(Unknown Source)
at com.knime.enterprise.server.executor.rmi.RMIExecutorImpl.checkVersion(RMIExecutorImpl.java:430)
at com.knime.enterprise.server.executor.rmi.RMIExecutorImpl.startSlave(RMIExecutorImpl.java:383)
… 15 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at java.io.DataInputStream.readByte(DataInputStream.java:265)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:246)
… 22 more

15-Nov-2018 14:03:56.227 INFO [KNIME-Job-Scheduler_1] com.knime.enterprise.server.jobs.ScheduledJobManagerImpl.schedule Schedule ‘Workflow ‘/Promotions/PO Pull’; resetting; discarding after execution; target name = PO Pull; next execution at 2018-11-15T15:00+02:00[Africa/Harare], repeating every hour; User: johann.vandermerwe; ID: e1701251-28c6-415f-870e-f4ef49af864b (failed 3 times)’ is currently disabled

Hi Richardd,

The KNIME Server is designed to disable a scheduled job after three failed attempts at execution. That is to prevent all server resources being used attempting to launch jobs that aren’t working. In this case it looks like an issue with the executor setup. Most likely the KNIME Server version does not match the executor slave version. See the Server-Executor compatibility chart here: https://www.knime.com/knimecom-product-downloads

Make sure that the executor (server-side) extension matches the KNIME Server version.

Best,

Jon

Hi Jon

Thanks for the response. Our Knime Server is 4.6. How do I determine the executor slave version?

Regards

Are you running on Windows or Linux?

Under Linux cd into the executor directory:
/opt/knime-executor/features

and run
ls | grep slave
which for me returns
com.knime.features.enterprise.slave_4.7.0.v2018…

That indicates compatibility with the KNIME Server 4.7

Under Windows, you will need to manually inspect the directory for the feature.

Best,

Jon

Hi Jon

Knime Server - 4.6.0
com.knime.features.enterprise.slave_4.6.0.v201712041209

So this doesn’t seem to be a server / slave compatibility problem.

Regards

Hi @richardd

we would recommend to update to the most recent release. Actually we always recommend to only use the >=.1 releases in production.

Would updating the server to our latest stable built be possible for you?

Best wishes, Iris