Unable to start executor after upgrading from Server 4.10 and Exec 4.1 to Server 4.11 and Exec 4.2.4

KNIME Server
1

Overview

This may be similar to this post however I haven’t been able to derive an exact solution from that response.

I’m working on updating a test server as above. Tomee starts fine, but when I run ./knime -nosplash -consolelog -application com.knime.enterprise.slave.KNIME_REMOTE_APPLICATION I get the output below. In particular it looks like there’s a problem related to

Could not download profiles from http://127.0.0.1:8080/knime/rest/v4/profiles/contents

However my knime.ini looks okay. I’ve also tried adding the public ip address of my server after

com.knime.server.canonical-address=

in knime-server.config

but I’m still getting the same error. Note that our server is not equipped with a server certificate.

Much appreciate any suggestions.

Console Output

CompilerOracle: exclude javax/swing/text/GlyphView.getBreakSpot
log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAddCookies).
log4j:WARN Please initialize the log4j system properly.
Feb 08, 2021 4:34:36 AM org.apache.cxf.bus.osgi.CXFExtensionBundleListener addExtensions
INFO: Adding the extensions from bundle org.apache.cxf.cxf-rt-transports-http (521) [org.apache.cxf.transport.http.HTTPTransportFactory, org.apache.cxf.transport.http.HTTPWSDLExtensionLoader, org.apache.cxf.transport.http.policy.HTTPClientAssertionBuilder, org.apache.cxf.transport.http.policy.HTTPServerAssertionBuilder, org.apache.cxf.transport.http.policy.NoOpPolicyInterceptorProvider]
Feb 08, 2021 4:34:36 AM org.apache.cxf.bus.osgi.CXFExtensionBundleListener addExtensions
INFO: Adding the extensions from bundle org.apache.cxf.cxf-rt-transports-http-hc (522) [org.apache.cxf.transport.http.HTTPConduitFactory, org.apache.cxf.transport.ConduitInitiator]
ERROR	 main ProfileManager	 Could not download profiles from http://127.0.0.1:8080/knime/rest/v4/profiles/contents: <!doctype html><html lang="en"><head><title>HTTP Status 404 ��� Not Found</title><style type="text/css">h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 ��� Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> &#47;knime&#47;rest&#47;v4&#47;profiles&#47;contents</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat</h3></body></html>. No profiles will be applied.
java.io.IOException: <!doctype html><html lang="en"><head><title>HTTP Status 404 ��� Not Found</title><style type="text/css">h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 404 ��� Not Found</h1><hr class="line" /><p><b>Type</b> Status Report</p><p><b>Message</b> &#47;knime&#47;rest&#47;v4&#47;profiles&#47;contents</p><p><b>Description</b> The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.</p><hr class="line" /><h3>Apache Tomcat</h3></body></html>
	at org.knime.product.profiles.ProfileManager.downloadProfiles(ProfileManager.java:396)
	at org.knime.product.profiles.ProfileManager.fetchProfileContents(ProfileManager.java:268)
	at org.knime.product.profiles.ProfileManager.applyProfiles(ProfileManager.java:168)
	at com.knime.enterprise.executor.KNIMEExecutorApplication.start(KNIMEExecutorApplication.java:44)
	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:137)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:107)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:657)
	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:594)
	at org.eclipse.equinox.launcher.Main.run(Main.java:1447)
	at org.eclipse.equinox.launcher.Main.main(Main.java:1420)
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
WARN 	 main RabbitMQHandler	 Couldn't connect to message queue, retry to connect in 10000ms
java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:607)
	at com.rabbitmq.client.impl.SocketFrameHandlerFactory.create(SocketFrameHandlerFactory.java:60)
	at com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:62)
	at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)
	at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1104)
	at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1063)
	at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:885)
	at com.knime.enterprise.executor.msgq.RabbitMQHandler.createConnection(RabbitMQHandler.java:172)
	at com.knime.enterprise.executor.msgq.RabbitMQHandler.<init>(RabbitMQHandler.java:138)
	at com.knime.enterprise.executor.KNIMEExecutorApplication.start(KNIMEExecutorApplication.java:59)
	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:137)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:107)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:657)
	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:594)
	at org.eclipse.equinox.launcher.Main.run(Main.java:1447)
	at org.eclipse.equinox.launcher.Main.main(Main.java:1420)

knime.ini

-startup
plugins/org.eclipse.equinox.launcher_1.5.700.v20200207-2156.jar
--launcher.library
plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.1100.v20190907-0426
-vm
plugins/org.knime.binary.jre.linux.x86_64_1.8.0.252-b09/jre/bin

-profileLocation
http://127.0.0.1:8080/knime/rest/v4/profiles/contents
-profileList
executor

-vmargs

-Dcom.knime.enterprise.executor.msgq=amqp://knime:20knime16@localhost/

-server
-Dsun.java2d.d3d=false
-Dosgi.classloader.lock=classname
-XX:+UnlockDiagnosticVMOptions
-XX:+UnsyncloadClass
-XX:+UseG1GC
-Dsun.net.client.defaultReadTimeout=0
-XX:CompileCommand=exclude,javax/swing/text/GlyphView,getBreakSpot
-Xmx2048m
-Dorg.eclipse.swt.internal.gtk.disablePrinting

-Djava.io.tmpdir=/srv/knime_server_tmp/

knime-server.config

#
# Example configuration file. Copy this file to <server repository>/config/knime-server.config and adjust the values
# to your needs. Defaults will be used if no values are specified.
#
com.knime.server.admin_email=
com.knime.server.arctorus_report_formats=
com.knime.server.canonical-address=
com.knime.server.config.watch=true
com.knime.server.csp-report-only=
com.knime.server.default_mount_id=knime-server
# com.knime.enterprise.executor.msgq=amqp://<user>:<password>@<rabbitmq-host>/<vhost>
com.knime.enterprise.executor.embedded-broker=true
# com.knime.server.executor.knime_exe=/opt/knime/knime-latest/knime
com.knime.server.executor.max_instances=
com.knime.server.executor.max_lifetime=
com.knime.server.executor.prestart=
com.knime.server.executor.reject_future_workflows=true
com.knime.server.executor.skip_teamspace_mount=
com.knime.server.executor.start_port=
com.knime.server.executor.sudo_cmd=
com.knime.server.executor.update_metanodelinks_on_load=
com.knime.server.job.discard_after_timeout=
com.knime.server.job.max_execution_time=
com.knime.server.job.max_lifetime=
com.knime.server.job.max_time_in_memory=
com.knime.server.job.status_update_interval=
com.knime.server.job.swap_check_interval=
com.knime.server.login.allowed_groups=
com.knime.server.login.consumer.allowed_accounts=
com.knime.server.login.consumer.allowed_groups=
com.knime.server.login.jwt-lifetime=
com.knime.server.login.user.allowed_accounts=
com.knime.server.login.user.allowed_groups=
com.knime.server.repository.update_recommendations_at=01:00
com.knime.server.server_admin_groups=admin
com.knime.server.server_admin_users=knimeadmin
com.knime.server.webportal.csp=
com.knime.server.webportal.disable_report_preview=
com.knime.server.webportal.disable_warning_messages=true
com.knime.server.webportal.hide_version=
com.knime.server.webportal.ie_compatibility=
com.knime.server.webportal.restrict_x_frame_options=
com.knime.server.webportal.sketcher_page=
com.knime.server.webportal.sketcher_size=
com.knime.server.webportal.title_label=
com.knime.server.user_directories.directory_location=/Users
com.knime.server.user_directories.parent_directory_owner=knimeadmin
com.knime.server.user_directories.owner_permissions=
com.knime.server.user_directories.inherit_permissions=
com.knime.server.user_directories.groups=
com.knime.server.user_directories.users=
com.knime.server.user_directories.world_permissions=
2

Hi Andrew,

If you updated from 4.10.* or below, you might need to rearrange your profile folders like described in here

Therefore each profile must have a folder in /config/client-profiles
The profileList line is a list of the profile folders the executor should be using. So just create a folder called “executor” in the repository we mentioned before, and more the default.epf – or whatever profile you are using – to the executor folder (I always like to rename it to executor too, just to keep it understandable).

If you have more profiles, you can create different folders and also list them in the profileList argument.

Best wishes
Ana

3

Thanks Ana.

I do already have an executor.epf profile at /srv/knime_server/config/client-profiles/executor/executor.epf

In case they’re relevant, the contents of that file are as listed below - these are just copied and pasted from our version 4.10 preferences.epf file.

For what it’s worth something else that I’ve noticed is that if I comment out the line after -profileLocation I no longer get the profileManager error, but I do still get the subsequent errors related to the RabbitMQ queue.

Any additional suggestions would be much appreciated.

/instance/org.knime.python/pythonPath=/usr/bin/python
/instance/org.knime.python2/defaultPythonOption=python3
/instance/org.knime.python2/python3Path=/home/knime/python/python36.sh
/instance/org.knime.workbench.core/database_timeout=7200

/instance/org.knime.workbench.core/database_drivers=/srv/knime_server/RedshiftJDBC42-1.2.16.1027.jar
4

Hi Andrew,

Do you have rabbitmq running in this machine? Sometimes there could be some other service already using the same port.

Even without the executor, can you access the webportal?

Best wishes
Ana

5

Hi Andrew,

Did you find the source of the problem? I asked about rabbitmq running the same machine, as this can cause problems when running the KNIME Server using the embedded message queue.

Please let me know if you need further assistance.

Best wishes
Ana

6

Hi @andrew.dun,

In my server.config there is no # in front of “com.knime.enterprise.executor.msgq=amqp://” - so it looks like to me, that your server don’t know where to find the executor (in your case rabbitmq).

BR,
Sven

7

Hi @sven-abx

You are right in the case someone is using rabbitmq. But when using the embedded broker (as you can see he is, as embedded broker is set to true), that is not necessary.

The problem of having Rabbitmq running and the KNIME server running (when using the embedded broker) on the same machine is that qpid uses the same port as Rabbitmq. You can either stop the rabbitmq service, or you can change the embedded broker port.

Best wishes
Ana

8

Thanks @ana_ved and @sven-abx

Much appreciate your replies, and please accept my apologies for the delayed response.

Certainly, we were aiming to use the embedded broker. @ana_ved to answer your question, we were finding that we were unable to access the web server, even though we could connect with the Apache Tomee page at the web server root address.

What’s happened since my last message is that we’ve overwritten our test machine (which we’d been attempting to upgrade), with the server 4.12.1 AMI from the AWS marketplace. As would be expected, this is working fine, but still doesn’t give us a clear path for updating our production machine. What we’re looking at doing is launching the 4.12.1 AMI in our production environment, and then reassigning the IP address of the existing EC2 machine to the new one, as well as transferring our license (initially, without removing the old machine). The plan had been to do this using the AWS Elastic Network Interface functionality, and then use the same approach as the basis for all future upgrades (of course we’d need to do some configuration on any new machine, to copy over existing workflows, install nodes, set up our python environments etc).

However, we’re not actually sure at this point whether the IP reassignment will be possible, or whether we’ll need to restore our old test instance and continue with the upgrade process. If we find that we can’t go ahead with that approach, then likely we’ll be going back to continuing to troubleshoot the above problem, probably sometime in the next few days.

Apologies for the lengthy response, and for the lack of clarity on where we’d need to go next with the issue on this thread. But I hope this at least helps explain the current status of the issue. Again, thank you for your responses to date.

9

Hi @andrew.dun

I have PM’d you via support to ask for logs and further steps. Please verify your email when possible.

Best wishes
Ana

1 Like