Overoccupation of cpu in Python Script

KNIME Server
1

Hello, I have a question about the CPU of Python script.

Currently, when running Python script node on KNIME Server, other WorkFlows are not working because parallel processing is not performed due to excessive CPU usage.

Therefore, we are looking for ways to limit CPU usage in code within Python Scripts in WorkFlow that is overoccupied.

Can you help me with a way to solve the above issue or code that limits CPU usage? It’s very urgent.

Thank you.

2

Hi @JaeHwanChoi,

pure Python code does not run multi-threaded. Could it be that you use a library which does heavy multi-processing? Is there a way that you configure that library to not use all CPU?

Best regards
Steffen

2 Likes
3

It is unfortunately not possible to limit RAM or CPU of a specific node.

1 Like
4

Hi @JaeHwanChoi

As stated by @steffen_KNIME, this is an issue related to Python more than to KNIME, eventually “process and system” management in general. Any Python code is run by a python interpreter/environment which is called by KNIME to delegate the job. Once a Python code is running, these Python issues will need to be handled based on Python solutions.

A possible solution is to import and use in your Python code the psutil Python library:

As stated in its documentation, psutil (process and system utilities) :

  • is a cross-platform library for retrieving information on running processes and system utilization (CPU, memory, disks, network, sensors) in Python.
  • is useful mainly for system monitoring, profiling and limiting process resources and management of running processes.
  • implements many functionalities offered by classic UNIX command line tools such as ps, top, iotop, lsof, netstat, ifconfig, free and others.

psutil currently supports Linux, Windows & macOS.

The following stackoverflow thread provides a nice example on how to use it:

Limit total CPU usage in python multiprocessing

Hope these hints and links on Python process and system utilities help to better balance the work load on your server.

Best,
Ael

3 Likes
5

Thank you for your answer. @steffen_KNIME , @aworker

Are you saying that you can limit the CPU usage within Python Scripts that learn ML/DL with the psutil python library you suggested as a solution?

The psutil python library seems to be used simply for monitoring purposes to check cpu and memory.

What I want to do is to limit the use of Python Script executor to less than 4 cores, assuming that KNIME executor (=cpu) is 8 cores.

Is that possible?

Thank you.

1 Like
6

Yes, that is exactly what @aworker wrote:

is useful mainly for […] limiting process resources […].

Could you try the example he mentioned and tell us whether that worked?
I am also curious what the outcome is.

Best regards
Steffen

1 Like