Using Python "multiprocessing" capabilities within Python Script Nodes

KNIME Extensions
1

Hi,

I know this is similar to those two topics:

But the question was never fully answered. So I wanted to run a really simple script using “multiprocessing”. The script looks like this:

from multiprocessing import Pool

def square(n):
    return n * n

with Pool() as pool:
    squares = pool.map(square, range(1, 11))
    print(squares)

But that won’t work, it throws an exception like this:

Can’t pickle <function square at 0x7f877b5c4400>: attribute lookup square on main failed
Traceback (most recent call last):
File “<string>”, line 8, in <module>
File “…/python3.6/multiprocessing/pool.py”, line 266, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File “…/python3.6/multiprocessing/pool.py”, line 644, in get
raise self._value
File “…/python3.6/multiprocessing/pool.py”, line 424, in _handle_tasks
put(task)
File “…/python3.6/multiprocessing/connection.py”, line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File “…/python3.6/multiprocessing/reduction.py”, line 51, in dumps
cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can’t pickle <function square at 0x7f877b5c4400>: attribute lookup square on main failed

I think it has something to do with the function “square” not being defined on the “main” level of this script. And that has something to do with how Java is executing the Python code within the Python Script Node I guess. I still wonder if there is way around it. Any ideas?

2

Hi @karbon_kreature,

That is correct. KNIME uses the exec command to execute the user-provided script.

Afaik, square is not required to be defined inside the main module but rather on the top-level of any module.
So you could create a file mymodule.py that contains square, place it somewhere where Python can find it, and use the following script in the node:

from multiprocessing import Pool
from mymodule import square

with Pool() as pool:
    squares = pool.map(square, range(1, 11))
    print(squares)

This seems to work on my machine.

Marcel

3

Hi @MarcelW that is so cool! I suspected some hack like that could help me out. Great stuff. I will try that with my real function (not “square” of course) and see how that goes. Thank you very much!

4

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.