Knime 4.7 keeps crashing with 'new python script'

[Dino]

Updated Knime to 4.7 (Mac m1). Knime keeps crashing when I use the new python script node.

The same script works when I use the legacy python node.
Also tested on Knime 4.7 Windows x64 with parallels. It also crashes there the same way.

I included the crash log.

error log 4.7 mac1 python script.txt (281.3 KB)

[steffen_KNIME]

Hi @Dino,

thanks for sharing. Could you provide the used Python script or some minimal version which causes the same error?
Does it also happen if the node has some small script or no script at all?

Best regards
Steffen

[Dino]

Hi Steffen,

It’s a bit difficult to provide the script (it’s not in knime self). The script sometimes runs, when I execute the the nodes one by one. Machine is Mac M1 Max with 64gb ram.

Best Regards Dino

[Dino]

Did some more testing. Changed my script from async http requests to single requests. Got a error for a memory leak but not a crash.

For now I am back to the Python legacy node, no issues at all.

Dio

[DiaAzul]

@Dino

Are you using asyncio library for the asynchronous requests? Without seeing your script it is not possible to comment definitively, however, you could be deleting objects and leaving the handle on the event loop. This prevents the objects being garbage collected with the result that the heap fills up resulting in a programme failure.

I wrote about my experience with this issue on Medium. Apologies, the link might be behind a paywall.

A Bug of Such Cunning That I Admire It

Then again, it could be something completely different

DiaAzul
LinkedIn | Medium | GitHub

[Dino]

I read your article, interesting! The strange thing is that I have been using the script with async requests for a long time and it has always worked with python legacy node (on windows and Mac m1 Windows arm and 4.7 native m1 nightly with de legacy node).

The 4.7 python node (new style) has a single change to the script to get the correct output table naming and flow variables (knio.flow_vars en knio.Table…). No other changes. It makes Knime crash without a Knime error.

As a test I modified my python script to send ‘normal’ requests and see if the issue is with async requests. Knime did not crash when testing, single node at a time. But when I run the entire script (part where I make the requests, Knime is not crashing but i now get a Knime memory error (memory leak). The request is nothing special. It makes a couple of requests to get some tables (couple 100 rows).

Maybe it’s something to do with the new way of the way the python node is build or the combination with the apple m1 cpu.

Is there a way to increase the memory available for knime on macOS arm version?

Dino