Python Script stops working

Hello everybody,

I'm working with Knime to handle a list of files and want to work with them with a python script. For one file everything works fine. But when I load all 45 files the node stops working after two minutes. There is no warning or something like that. It just stops at the percentage where it is at this moment. I only can see that it's doing nothing by looking at the CPU-chronic.

Does anybody has a sloution for this problem or do I have to accept that knime does not like python, because it is written in Java?

Thanks for help, Julia

Hi Julia,

Could you please give me a bit more details? Which operating system do you use? Which KNIME-version? Which version of the Python-plugin? Which Python version?

The input table does contain the filenames?

Would it run in a loop (one file per iteration)?

Does the script with multiple input files run by using OpenInPython?

Antje

Hi Antje,

sure I can give you the details of my systems: I use Ubuntu 14.04, my Knime version is 2.11.3 and the Knime Python Scripting extension is version 2.0.2.201505280911. And I have Python 2.7.6.

The input table contains the Locations and the URL of my files. It also stops by using a loop.

I'm not familiar with OpenInPython. Could you shortly explain the Input and Output to me. Because I looked at it and I'm not sure if it works for me. I want to manipulate the output in further steps. But do I get with this node an output table?

Thanks for your help.

Julia

Hi Julia,

I don't get the difference between your test with a single file and the test running your script within a loop with a single file per iteration. It shoul d behave similar, no?

Would it be possible to provide me some data and the workflow (the essential part)? I would like to whether I can reproduce the problem

The OpenInPython node can only help troubleshooting to find a problem in your script. It should open python (command line) containing an object 'kIn' which allows you to run the same code like in the snippet. Then you might see error messages and not just an execution failure like in KNIME...

Ciao,

Antje

Hello Antje,

I thougth about providing you some of my workflow, but you would need further scripts aside to my own and I cannot give these away. I will try to write one script with similar problems.

I don't know why it stops when I run it in a loop. The first time when I check if my script is working I always only use one File. And then when it works I try to apply it to all files.

There is also the problem that in the first run it  manages like 10 loops before it stops. The next time when I try to run the exact same workflow it is only able to run like 5 loops before stopping. Every time I try to run again the workflow it is able to manage less iterations. Even closing Knime and open it again does not solve this problem. I think I have also tried to restart the computer.

I don't think that it has something to do with my scripts. When I run them for example in Spyder they work perfectly fine.

I want to make clear, that Knime does not give me an execution failure when the script stops working. As I said it just stops like it has hung up. I can wait a big amount of time and nothing happens. I can only see in my CPU-chronic that Knime does not work anymore.

I hope I can provide a workflow which makes my problem clear.

Have a nice day,

Julia

Hi Julia,

that is a weird behaviour.

I'm curious if you have something for me to test at some point. Does it work, if you use the standard script within a loop? (I mean the one, which is default for the node and just passes the table through)

For the moment I have no further idea... I'm sorry

Ciao, Antje

Hi Julia,

you might want to try the KNIME Python Integration nodes (a KNIME Labs product). It's a different set of Python nodes for KNIME. In my experience the most common cause of friction between KNIME and Python is when KNIME tries to convert the data structure created by Python to a KNIME data structure.

The KNIME Python Scripting extension (Community Nodes) creates a CSV file which KNIME then attempts to read, whereas the KNIME Python Integration nodes use the Google Protobuf format, which is much more robust.

The only difference in usage is that you will need to write your ouput data into a Pandas dataframe in your script, instead of into a Dictionary.

Also remember to switch the Debug mode of your KNIME GUI to 'DEBUG' (File -> Preferences -> KNIME -> KNIME GUI -> Console View Log Level)

Hope this helps

Gideon

Hi

any news here ?

 

My Python-Scripts also Stuck at 30% without an error msg.

Thanks 4  u're help.

KnimeLearner14