Accessing Hub triggers high CPU load

[mwiegand]

Description:

Steps to reproduce:

1. Open Task Manager and filter Knime
2. Open Knime (Classic or Modern UI is irrelevant)
3. Let CPU load settle after app start
4. Log into Hub
5. Observer CPU load increasing significantly even without a workflow being opened

Actual results:

Expected results:

Attachments:
Screen Recording for download showing that 5.2.6, close to the end of thee video, does not exhibit that issue:

Screenshots

OS:
Windows 11

[wiswedel]

Hi hi.

This is tricky to explain. Do you mind extracting a jstack while you see the cpu load? There are so many factors, could indeed be a bug but could also be Windows Defender or something else. (In general, you see high cpu use or KNIME is “stuck”, a jstack output is useful for developers.)

– Bernd

[mwiegand]

Sure, here you go.

BEFORE - 0 to 2 % Load
80657 Knime 5-3 High CPU Load BEFORE accessign Hub - threaddump-1719836794855.tdump.txt (122.1 KB)

AFTER - 8 - 12 % Load (on 4 Cores)
Only logged in, nothing else. I also downloaded the full zip starting with an “unused” app folder
80657 Knime 5-3 High CPU Load WHILE accessign Hub - threaddump-1719836862095.tdump.txt (111.5 KB)

Update
In an effort to find possible explanations I used some extensive system monitoring trying to figure out what is happening. The tools used were:

Fan Control: Using properly adjusted fan control as an indicator of higher CPU utilization

Microsoft Sysinternals Porcmon and Process-Explorer as more advances means to assess processes compared to the task manager i.e. also allowing to monitor network traffic

Microsoft WinDbg to analyze the threads of a process of dumpfiles.

HWiNFO® 64 Pro to check hardware details like power consumption and double verify i.e. CPU utilization

What I found was:

BASELINE: System IDLE - Knime and all other main apps closed

• Temps and wattage as expected
  
  

DEFAULT: Knime Started but not connected to Hub

• Temps + 5 to 8 C°, Wattage ok, no significant fan utilization
  
  

EXCEPTION: Knime connected to Hub, nothing opened

• CPU utilization up to 22 %
• Wattage increase of +15 W (Task Manager reports high power usage)
• Temps increase by 20 °C to max. of 95°C causing fan speed to ramp up (see Fan Control)
• Only a few packages send in the background (see process monitor)
• HWInfo and Task Manager both report similar core usage (polling rate differs)
• Process Explorer sees constant CPU utilization (not split by cores or threads) of circa 7%

There doesn’t seems to be much IOPS not Network throughout. Hence, the issue seems to be local. Here is the process summary from the Process Monitor.

That leads me to analyzing the threads of the knime.exe PID process.

BASELINE: HUB Disconnected

EXCEPTION: HUB Connected

Upon connecting to the HUB there are two threads with the start address “ucrtbase.dll!recalloc+0x10”

I am currently trying to work myself through WinDbg but Knime crashed likely because I unintentionally interrupted the process.

Update
Trying to use WinDbg, either by attaching it to a running process or using “File > Launch Executable”, fails. Attaching it to a process causes Knime to crash and and launching an executable fails.

WinDbg Output:

************* Preparing the environment for Debugger Extensions Gallery repositories **************
   ExtensionRepository : Implicit
   UseExperimentalFeatureForNugetShare : true
   AllowNugetExeUpdate : true
   NonInteractiveNuget : true
   AllowNugetMSCredentialProviderInstall : true
   AllowParallelInitializationOfLocalRepositories : true

   EnableRedirectToV8JsProvider : false

   -- Configuring repositories
      ----> Repository : LocalInstalled, Enabled: true
      ----> Repository : UserExtensions, Enabled: true

>>>>>>>>>>>>> Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.000 seconds

************* Waiting for Debugger Extensions Gallery to Initialize **************

>>>>>>>>>>>>> Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.032 seconds
   ----> Repository : UserExtensions, Enabled: true, Packages count: 0
   ----> Repository : LocalInstalled, Enabled: true, Packages count: 41

Microsoft (R) Windows Debugger Version 10.0.27553.1004 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

CommandLine: C:\Program Files\knime-full_5.3.0\knime.exe

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*
Symbol search path is: srv*
Executable search path is: 

+------------------------------------------------------------------------+
| This target supports Hardware-enforced Stack Protection. A HW based    |
| "Shadow Stack" may be available to assist in debugging and analysis.   |
| See aka.ms/userhsp for more info.                                      |
|                                                                        |
| dps @ssp                                                               |
|                                                                        |
+------------------------------------------------------------------------+

ModLoad: 00007ff6`ba0e0000 00007ff6`ba164000   image00007ff6`ba0e0000
ModLoad: 00007ffc`53570000 00007ffc`53787000   ntdll.dll
ModLoad: 00007ffc`519a0000 00007ffc`51a64000   C:\WINDOWS\System32\KERNEL32.DLL
ModLoad: 00007ffc`50a20000 00007ffc`50dcc000   C:\WINDOWS\System32\KERNELBASE.dll
ModLoad: 00007ffc`514b0000 00007ffc`5165e000   C:\WINDOWS\System32\USER32.dll
ModLoad: 00007ffc`509c0000 00007ffc`509e6000   C:\WINDOWS\System32\win32u.dll
ModLoad: 00007ffc`51ad0000 00007ffc`51af9000   C:\WINDOWS\System32\GDI32.dll
ModLoad: 00007ffc`508a0000 00007ffc`509b9000   C:\WINDOWS\System32\gdi32full.dll
ModLoad: 00007ffc`50dd0000 00007ffc`50e6a000   C:\WINDOWS\System32\msvcp_win.dll
ModLoad: 00007ffc`51190000 00007ffc`512a1000   C:\WINDOWS\System32\ucrtbase.dll
ModLoad: 00007ffc`40a50000 00007ffc`40ce3000   C:\WINDOWS\WinSxS\amd64_microsoft.windows.common-controls_6595b64144ccf1df_6.0.22621.3672_none_2713b9d173822955\COMCTL32.dll
ModLoad: 00007ffc`53000000 00007ffc`530a7000   C:\WINDOWS\System32\msvcrt.dll
(15ed0.645c): Break instruction exception - code 80000003 (first chance)
ntdll!LdrpDoDebuggerBreak+0x30:
00007ffc`5364bed4 cc              int     3

[thor_landstrom]

Hello @mwiegand,

Doing a quick update and there is a ticket open for this issue.

Ticket AP-22906

Thanks!
TL

[mwiegand]

Hi @thor_landstrom,

after familiarizing / playing around with VisualVM I was able to furth pin down the cause. What I did:

1. Started VisualVM

2. Started Knime

3. Immediately initiated monitoring of the process

4. Waited a minute to give KNime time to start

5. Connected to the Hub w/o opening a workflow (just the start screen) and let VisualVM collect some stats (Time stamp when opening the hub is 15:42 after which you see the flapping heap space usage too
   

   

6. Navigated in VisualVM to Sampler > Thread CPU Time to find, after some time has lapsed, the threads with elevated CPU usage
   

   

7. Got the call tree in the CPU Samples tab
   

   

As follows the snap snapshot and threaddump. I can also share with you the heapdump (>300 MB). Send me a DM to share the link with you.

Thread-114
java.lang.Thread.run()
com.knime.enterprise.client.rest.JobFetcherImpl.fetchJobs()
com.knime.enterprise.client.rest.RestServerContent$$Lambda$3465.0x00000008029c5738.getAsBoolean()
com.knime.enterprise.client.rest.RestServerContent.lambda$25()
com.knime.enterprise.client.rest.HubContent.isExecutionEnabled()
com.knime.enterprise.client.rest.HubContent.hasExecutionContext()

Thread-115
java.lang.Thread.run()
com.knime.enterprise.client.rest.ScheduleFetcherImpl.run()
com.knime.enterprise.client.rest.ScheduleFetcherImpl.fetchSchedules()
com.knime.enterprise.client.rest.RestServerContent$$Lambda$3472.0x00000008029c6cb8.getAsBoolean()
com.knime.enterprise.client.rest.RestServerContent.lambda$32()
com.knime.enterprise.client.rest.HubContent.isExecutionEnabled()
com.knime.enterprise.client.rest.HubContent.hasExecutionContext()

Best
Mike

[forum]

Internal ticket ID: AP-22906
Fix version(s): 5.1.4, 5.2.6, 5.3.1
Other related topic(s): -