Portable Python Components

Hello KNIME Community,

TLDR: Is there a way to embed python module(s) in a component and reference them from the Python Script node?

I recently read this article. It was a great introduction but I don’t understand how the component is portable if the Python Script node is referring to a module that lives inside the workflow and not inside the component itself.

It seems that when you execute the Python Script node the current working directory is always the workflow directory and not the directory of the current component. Any modules you import into the Python Script node would then have to be placed in the workflow directory and would not exist if someone were to pull the component from the knime hub etc.

You could put the script in the component directory and use extract context properties to get the absolute path to the workflow directory and then add the component directory to that and add this to sys.path. This isn’t very robust though because the way knime names nodes you aren’t always going to be sure what the node ID for the current component is. If you choose to match the name regardless of the node ID it wouldn’t hold up to you having multiple components in the same workflow. Even if there was a way to make this work it would not be overcome by the issue below.

I have also tried sharing a component to my local workspace then adding the module inside of this component folder and editing the component to import the module. This all works fine when editing and testing the component but when I drag the shared component into a new workflow the module file does not come with it. It is the same when sharing a component. If the component has files in it knime does not recognize they will not be shared to your local workspace / knime hub.

I am hoping there is something obvious I am missing. I was assuming that components should operate like workflows where they are essentially containers that can be moved around and deployed with their dependencies all contained inside of them but that does not seem to be the case right now.

Any help or pointing me in the right direction would be much appreciated!! Thank you for your time!
Kaegan

1 Like

Hi Kaegan,

So I am not an expert on this topic and actually there are a lot of changes currently being made to make shipping python scripts. Have you tried resolving the issue with a conda environment propagation node? Also, are you already running version 4.6?

Best,
Karen

1 Like

Hey Karen,

Thanks for replying! I am still using knime 4.3.4 because that is what our server is using at the moment.

The problem I am describing doesn’t have to do with the conda environment propagation node, it is more about importing local modules in a python component and how that will work when the component is posted on knime hub or shared in general. The conda environment propagation node works great for including external modules but I don’t believe it would solve the problem described above unless I wanted to go share my code publicly as a package.

Thank you for your thoughts!
Kaegan

If you got some REST endpoint you could publish your packages there and then use request node with API key (some kind of security layer) to pull the modules from there.

Hello @Kaegan,

Yes, you are right, the python script nodes use the workflow directory as default. One way to export python modules with Components is to make the component write the python module (.py) to the workflow directory whenever the Component is executed, this will make the module available in the Workflow directory and then the Python Script Node in the Component can then use this module for running the script of the user.

Please look at the Component that I have created which allows the user to select the .py file and write it to the workflow directory, you can use this component and embed it in your Python based Components.

I believe this will help you because I used this hack while working with KNIME 4.3. Let me know if you need further assistance.

5 Likes

Hey @Mpattadkal,

Thank you for responding!! I have taken a look at this potential solution but I’m not sure it solves my problem. It looks like this component would only work for myself locally if I always have a path to the .py file I want to use. If I were to try to distribute my component via the KNIME hub no one would have access to the module that I pointed to locally. This could maybe work for the KNIME server by storing the module there but if I am making components for other people to use at my company who aren’t logged into the KNIME server they also wouldn’t have access to the python module. It would be ideal if the .py file could be saved inside of the component and then referenced or copied somehow from there but as I mentioned in my earlier post:

Thank you again for your reply! Please let me know if I am mistaken in my understanding of how this component works.

@Kaegan well, well. I came up with this ‘solution’ though it is not a very nice one. The component would write .PY file(s) with functions dynamically to a local folder of the workflow.

The code itself would be written into a string variablen and then be exported to a local file so all logic can be stored within the component … in a way … There also is a init file to mark the module. Adapting from this workflow with individual .PY files.

The created module with init file and .PY file will then be imported via the sys-path idea.
Not sure if this is a solution you would want to roll out. But it does work in general.

3 Likes