Executor don't start after installing Server 4.11.1

KNIME Server
1

First, updating to the newest version of the knime server did not worked. Therefore, I tried to reinstall the knime server with Apache Tomcat now.
But still there are problems running workflows from the web portal. I tried to copy the parts of the log files that could be necessary. I tried both to start the executor and to install it as service. The service looks to be connected still to tomee but I exchanged the executor by the downloaded version completly (%NSSM% set knime-executor DependOnService TomEE).
As I don’t know the log files very well I forgot something import possibly.
Thank you very much for your help.

04-Sep-2020 15:04:58.582 SEVERE [http-nio-8080-exec-9] com.knime.enterprise.webportal.WebportalUI.displayCustomError Load error
java.util.concurrent.TimeoutException: Job wasn’t loaded within 1 minute. This usually happens if no executor is available. Please check back with your server administrator.

04-Sep-2020 15:09:38.713 INFO [KNIME-JobStatusUpdater-Background-Thread_1] com.knime.enterprise.server.executor.msgq.RabbitMQJobStatusUpdaterImpl.logJobStates Current job states:
Jobs known by server:
/Examples/WebPortal/DataVisualization_AirlineDataset (DataVisualization_AirlineDataset 2020-09-02 15.11.39; 078714e6-1149-4b65-8bb3-b3dda405e6ad) (swapped)
Jobs known by executors: none

04-Sep-2020 15:23:12.798 INFO [pool-12-thread-8] com.knime.enterprise.server.tokens.TokenServerImpl$1.onRemoval 4 core tokens from executor 502c9f35-e67e-4efc-b486-438777c0ac43@s-c05-gn-appl01 just expired. Now there are 0 available core tokens.
04-Sep-2020 15:23:12.798 INFO [pool-12-thread-8] com.knime.enterprise.server.tokens.TokenServerImpl.acquireTokens Executor 1540ea5a-bf34-4789-a6af-a7e3791b1446@s-c05-gn-appl01 acquired 4 core tokens. Now there are 0 available core tokens.
04-Sep-2020 15:23:27.628 INFO [http-nio-8080-exec-4] com.knime.enterprise.server.jobs.WorkflowJobManagerImpl.loadWorkflow Loading workflow ‘/Reports/Statistic_jahrweise_v5’ for user ‘baderi’
04-Sep-2020 15:23:27.659 INFO [http-nio-8080-exec-4] com.knime.enterprise.server.executor.msgq.RabbitMQExecutorImpl.loadWorkflow Loading workflow ‘/Reports/Statistic_jahrweise_v5 (Statistic_jahrweise_v5 2020-09-04 15.23.27; b8e0a174-310c-411a-9a1c-3ca13f4cdcc9)’ via executor group ‘knime-jobs’
04-Sep-2020 15:24:27.677 INFO [http-nio-8080-exec-4] com.knime.enterprise.server.executor.msgq.RabbitMQExecutorImpl.unload Unloading job ‘/Reports/Statistic_jahrweise_v5 (Statistic_jahrweise_v5 2020-09-04 15.23.27; b8e0a174-310c-411a-9a1c-3ca13f4cdcc9)’ via executor group ‘knime-jobs’, executor ‘’
04-Sep-2020 15:24:27.677 WARNING [http-nio-8080-exec-4] com.knime.enterprise.webportal.presenters.ExecutionHandler. Failed to load workflow, full stack trace below
java.util.concurrent.TimeoutException: Job wasn’t loaded within 1 minute. This usually happens if no executor is available. Please check back with your server administrator.
at com.knime.enterprise.server.jobs.WorkflowJobManagerImpl.loadWorkflow(WorkflowJobManagerImpl.java:789)
at com.knime.enterprise.server.application.jobs.PermissionAwareWorkflowJobManagerImpl.loadWorkflow(PermissionAwareWorkflowJobManagerImpl.java:108)
at com.knime.enterprise.webportal.presenters.ExecutionHandler.(ExecutionHandler.java:137)
at com.knime.enterprise.webportal.presenters.ServerItemHandler.getExecutionHandler(ServerItemHandler.java:495)
at com.knime.enterprise.webportal.components.wkfpanels.WorkflowJobInputFirstPagePanel.checkLegacyQuickforms(WorkflowJobInputFirstPagePanel.java:161)
at com.knime.enterprise.webportal.components.wkfpanels.WorkflowJobInputFirstPagePanel.(WorkflowJobInputFirstPagePanel.java:108)
at com.knime.enterprise.webportal.presenters.ServerItemHandler.showWorkflowItem(ServerItemHandler.java:341)
at com.knime.enterprise.webportal.presenters.ServerItemHandler.treeSelectionChanged(ServerItemHandler.java:880)
at com.knime.enterprise.webportal.presenters.ServerItemHandler.selectInTree(ServerItemHandler.java:903)
at com.knime.enterprise.webportal.presenters.ViewHandler.getView(ViewHandler.java:190)
at com.knime.enterprise.webportal.presenters.WebportalNavigator.navigateTo(WebportalNavigator.java:85)
at com.knime.enterprise.webportal.components.WorkflowTree.navigateToItem(WorkflowTree.java:159)
at com.knime.enterprise.webportal.components.WorkflowTree.access$000(WorkflowTree.java:63)
at com.knime.enterprise.webportal.components.WorkflowTree$1.itemClick(WorkflowTree.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.vaadin.event.ListenerMethod.receiveEvent(ListenerMethod.java:510)
at com.vaadin.event.EventRouter.fireEvent(EventRouter.java:200)
at com.vaadin.event.EventRouter.fireEvent(EventRouter.java:163)
at com.vaadin.server.AbstractClientConnector.fireEvent(AbstractClientConnector.java:1015)
at com.vaadin.ui.Tree.changeVariables(Tree.java:511)
at com.vaadin.server.communication.ServerRpcHandler.changeVariables(ServerRpcHandler.java:616)
at com.vaadin.server.communication.ServerRpcHandler.handleInvocation(ServerRpcHandler.java:463)
at com.vaadin.server.communication.ServerRpcHandler.handleInvocations(ServerRpcHandler.java:406)
at com.vaadin.server.communication.ServerRpcHandler.handleRpc(ServerRpcHandler.java:273)
at com.vaadin.server.communication.UidlRequestHandler.synchronizedHandleRequest(UidlRequestHandler.java:90)
at com.vaadin.server.SynchronizedRequestHandler.handleRequest(SynchronizedRequestHandler.java:41)
at com.vaadin.server.VaadinService.handleRequest(VaadinService.java:1422)
at com.vaadin.server.VaadinServlet.service(VaadinServlet.java:380)
at com.knime.enterprise.webportal.WebportalServlet.service(WebportalServlet.java:139)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:666)
at com.knime.enterprise.tomcat.authenticator.KnimeServerAuthenticator.invoke(KnimeServerAuthenticator.java:442)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:373)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)

2

Some more details:
Server 4.11.1, Executor 4.2.1, Windows Server 2016 Standard (Version 10.0.14393 Build 14393). The Webportal, that works well, says it is connected with a distributed executor (in older installations executor only and I haven’t installed a distributed version willinglly). The webportal shows a load error when I try to start a workflow - so for some reason there is no connection to the executor. The installation was done fresh with an unpacked executor downloaded from the server download section.
When trying to run a workflow there is no log generated in workflow/.metadata.

3

Additionaly; I found a Java exception in the executors directory that is to large to post completly:

A fatal error has been detected by the Java Runtime Environment:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000058fd6014, pid=1196, tid=0x0000000000001ab4

JRE version: OpenJDK Runtime Environment (8.0_252-b09) (build 1.8.0_252-b09)

Java VM: OpenJDK 64-Bit Server VM (25.252-b09 mixed mode windows-amd64 compressed oops)

Problematic frame:

V [jvm.dll+0x126014]

Current thread (0x0000000024b0b800): JavaThread “pool-2-thread-7” [_thread_in_vm, id=6836, stack(0x0000000029550000,0x0000000029650000)]

4

Hi @DiHa,

Welcome to the KNIME community. Did you install the knime executor after the server installation or within in the installer? It looks as if the knime.ini of the executor has to be modified as described here: https://docs.knime.com/2020-07/server_update_guide/index.html#switch-to-qpid

Please stop all running executors before the modifications and start it afterwards first with the startup skript (not as a service) using the same user that owns the server installation.

All the best,
Michael

5

Hi Michael,
I installed the executor within the server installation (by copying from the zip and then choosing the executors directory). The modifications in the knime.ini file were done, too. When restarting the executor it seems to abort with the error log cited above (exactly the executor start script).
All the best and thank you,
Dirk

6

Do you need some additional information? As I think I did all steps that are neccecary. But still the installation doesn’t work. I tried as several times, too. At the moment we cannot work with the server (or, more precise, cannot start workflows at the server).
All the best, Dirk

7

Knime Server Config:
com.knime.server.default_mount_id=knime-server
#com.knime.enterprise.executor.msgq=amqp://knime:knime@rabbitmq-host/knime-server
com.knime.enterprise.executor.embedded-broker=true
com.knime.enterprise.executor.embedded-broker.port=
com.knime.server.executor.knime_exe=C:/knime_server/knime_executor

und knime.ini:
-startup
plugins/org.eclipse.equinox.launcher_1.5.700.v20200207-2156.jar
–launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.1100.v20190907-0426
–launcher.defaultAction
openFile
-vm
plugins/org.knime.binary.jre.win32.x86_64_1.8.0.252-b09/jre/bin
-profileLocation
http://127.0.0.1:8080/knime/rest/v4/profiles/contents
-profileList
executor
-vmargs
-server
-Dsun.java2d.d3d=false
-Dosgi.classloader.lock=classname
-XX:+UnlockDiagnosticVMOptions
-XX:+UnsyncloadClass
-XX:+UseG1GC
-Dsun.net.client.defaultReadTimeout=0
-XX:CompileCommand=exclude,javax/swing/text/GlyphView,getBreakSpot
-Xmx2048m
-Dorg.eclipse.swt.browser.IEVersion=11001
-Dsun.awt.noerasebackground=true
-Dequinox.statechange.timeout=30000
-Dknime.server.executor.memory=128G
-Dcom.knime.enterprise.executor.msgq=amqp://knime:20knime16@localhost/

I think that is configured correct. I installed the server as well as the exectuor several times but this doesn’t help. I tried to use rabbit as message queue, too, but it doesn’t work (am I right that it is not possible to use rabbit with the small knime server? I am not interested in distributed executors but I thought there was an systematic problem in my qpid installation).
Thank you, Dirk

8

Hi Dirk,

already had a call with your colleague about this topic. We fixed the installation in general but are facing some network/permission issues now. We will investigate further to find the root cause.

All the best,
Michael

9

We also dealing with the same problem.

10

Hi @nael,

Could you please elaborate what exactly is failing? If it is a fresh installation of 4.11.1 Tomcat variant the executor is not started via the server process per default, as it uses the Apache Qpid for the communication. So it has to be started manually or by installing a service for it.

More details are available here: https://docs.knime.com/2020-07/server_installation_guide/index.html#_startingstopping_knime_server
and here: https://docs.knime.com/2020-07/server_installation_guide/index.html#_running_executors_as_services

All the best,
Michael

1 Like
11

Michael, thank you! We don’t knew that the executor doesn’t started per default.

Best regards

2 Likes