Node development using Python

Hello,

We are looking at developing a collection of nodes that use our Python API to make our tools
accessible via KNIME. The Python scripting nodes allow us to access our API very easily, and when
combined with Components/Metanodes and the Workflow Abstraction tools we though we had everything
we needed to put a KNIME Extension package together.

However, digging deeper has raised some issues, and, while the developer documentation has plenty of great material on node development in Java, we haven’t found answers to our questions around development using Python. So, our main questions are:

  • Eventually, we would like our nodes to appear in the Node Repository, and I assume the way to do this is via an Extension package. Is it actually possible to create such an Extension without creating the nodes in Java?

  • If so, how do people manage development of non-Java nodes? For example, how does one keep a configured Python Scripting Node or a Component in a version control system? Workflows can be exported and imported, but this doesn’t seem to be the case for individual nodes.

Any help on these matters would be greatly appreciated, even if it’s just pointers to documentation we’ve missed. Thanks!

Francis
2 Likes

I’ve never written a python based node myself but wonder if maybe this git repo will be helpful to you:

This will require some java though.

3 Likes

I’ll give it a look. Thanks!

Hi Francis,

Yes, creating an extension should be the way to go here.

KNIME’s Node Repository plug-in (org.knime.workbench.repository) offers an extension point (org.knime.workbench.repository.metanode) which lets you add your own metanodes (not components, as far as I know) to the repository. If only using metanodes works for you, then this would be a solution that requires no coding in Java. KNIME’s time-series plug-in, for example, makes use of this extension point. If you type “Seasonality Correction” into the search bar of the Node Repository in the KNIME UI, you should see the corresponding metanode listed. And in the time-series plug-in itself, it is located here and registered here.

If you wish for a more native integration of your functionality into KNIME (or metanodes alone simply do not cut it), then there will probably be no way around writing the skeletons (including the dialogs) of your nodes in Java and letting them reach out to Python to perform their actual tasks. There is currently no way of creating nodes directly in Python.
@swebb already posted an example on how this can be done. Additionally, there are also a couple of nodes provided by KNIME such as the MDF Reader node (source) and also some nodes of the KNIME Deep Learning - Keras Integration (in particular the learner (source) and executor (source) nodes) that do the same. They might be a bit too complicated to serve as a good reference, though.

As for the metanode-based approach outlined above, all you need to do to “export” the metanode from KNIME into your extension is copying over the corresponding directory from KNIME’s workspace directory. This is also explained in the description of the extension point mentioned above. There are still some details that you need to consider. For example, the KNIME version that you used to create the metanodes must not be newer than the KNIME versions for which you want to provide your extension.

All of that being said, the recent releases of the KNIME Analytics Platform tried to make sharing reusable functionality easier by bundling them into Components and sharing these via the KNIME Hub. While these components will not show up in the Node Repository (but in the KNIME Explorer), maybe they still offer a good trade-off with respect to the development effort that you need to put into them compared to the extension-based approaches.

Marcel

4 Likes

Thanks very much, Marcel, this is very helpful.