An error occurred while trying to automatically restart the executor

Good afternoon.
To clear the memory, we planned to restart executor every night.
Unfortunately, an error occurs at the time of restart:

05-Feb-2020 13:31:26.813 SEVERE [KNIME-RMI-Retirer_0] com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.rmiReachedEndOfLife Failed to start new RMI instance - using the old instance instead com.knime.enterprise.utility.ExecutorException: Could not start new RMI instance as all available ports are currently in use at com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.startNewRMI(RMIExecutorManagerImpl.java:310) at com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.rmiReachedEndOfLife(RMIExecutorManagerImpl.java:206) at com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.lambda$startNewRMI$0(RMIExecutorManagerImpl.java:325) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

Hi,

this could be related to a problem concerning a Java update:

Which version of KNIME Server do you run? Check for your version if there’s already an update (we’re plannin on also releasing one today).

Cheers,
Moritz

AdoptOpenJDK:
image

Oracle JDK:
image

Knime server version:
4.10.0.3fa2e72f6

Now I will update to the last December version and write you the status of the problem

I checked, the December version is installed. The problem with the error and the lack of a reboot persists.

Hi,

I see that you run on a Windows machine. IIRC there has been a Windows update that blocked the ports used by the executor. Could you check that the necessary ports are available? They start at port 50100 and then the port will be increased by 1 if the port isn’t available.

Cheers,
Moritz

Hi!

I checked the ports and in the range 50101-51800 there are no ports used. In the log I do not see that knime is trying to use another port only 50100.

Another interesting fact is that after an error occurs (with a reserved port), knime no longer tries to restart according to a schedule, even there are no error messages.

netstat:

task manager for pid 9892:
image

The same problem on version 4.10.1. I changed the port to 60100 (just to give it a try).

Hi,

the error might be misleading. Windows might block the ports using their firewall and hence no program is listening to that port. It could be that 50100 is unblocked while all others are blocked.

Cheers,
Moritz

Hi!

We checked the settings and logs of the firewall, it is disabled and there are no messages about blocking ports in the logs. From experience, there may still be options, what can block ports?

Respectfully,
Vladimir

Good day!

Disabled firewall and antivirus, unfortunately the situation and the error did not change. I tried with the changed port and completely without specifying the port. Maybe you have any tips or suggestions for diagnosing the problem?

Hi @arcdevil,

Can you please try to use a different port range? I.e., set com.knime.server.executor.start_port=60100 or something else outside the 50000 range.

Cheers,
Roland

1 Like

Hi Roland!
Changed the port to 40100, unfortunately the situation is the same.

10-Feb-2020 13:03:07.340 INFO [KNIME-RMI-Retirer_0] com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.rmiReachedEndOfLife RMI Executor 0 (user: (none), port: 40100, uptime: 00:10:00, status: AcceptingJobs, #jobs: 0) needs to be retired; starting new instance and shutting it down
10-Feb-2020 13:03:07.340 SEVERE [KNIME-RMI-Retirer_0] com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.rmiReachedEndOfLife Failed to start new RMI instance - using the old instance instead
com.knime.enterprise.utility.ExecutorException: Could not start new RMI instance as all available ports are currently in use
at com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.startNewRMI(RMIExecutorManagerImpl.java:310)
at com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.rmiReachedEndOfLife(RMIExecutorManagerImpl.java:206)
at com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.lambda$startNewRMI$0(RMIExecutorManagerImpl.java:325)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

10-Feb-2020 13:03:07.340 INFO [KNIME-RMI-Retirer_0] com.knime.enterprise.server.executor.rmi.RMIExecutorManagerImpl.printExecutorListToLog KNIME Executors associated with user: (none):
0: RMI Executor 0 (user: (none), port: 40100, uptime: 00:10:00, status: OutDatedButAcceptingJobs, #jobs: 0)

Attached logs from the moment the system was launched:
logs.zip (22.8 KB)

Good afternoon.
Please tell me. Do you have any ideas where to look for a problem?

Hi Vladimir,

Is it possible that an executor is still running in the background and prevents the start of the new one? Could you please try to shutdown the server and stop/kill all remaining KNIME executor processes? An other possible way to make sure that there is no remaining process would be a restart of the Windows on the server.

What did you do before the issue occurs, did you perform an update of the KNIME server or a change in the configuration? I would like to have a look on your configuration file /config/knime-server.config. Please send me a PM if you don’t want to attach it here.

Thanks in Advance,

Best,
Michael

Good afternoon.

Completely stopped the server, checked that there were no executor, and started the server again. The server automatically starts the executor. At that moment, when the executor must restart, the old one is still active.

Vladimir

Is it possible that you set the option com.knime.server.executor.max_instances=1 in the knime-server.config, so that the start of the second instance is blocked?

The old one will be shut down after the successful start of the new one.

Best,
Michael

3 Likes

Hi, Michael!

You are absolutely right, that was the problem.
Thank you very much!

Vladimir.

2 Likes

Hi Vladimir,

happy to hear that it works now and the issue is finally solved.

Michael

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.