add python library to existing environment via the KNIME way

KNIME Development
#1

Hi everybody,

we would like to include in our own nodes the way how KNIME is handling conda environments. I got the feedback from our developer that we can create a new environment with the desired libraries via the KNIME way. What he did not find is how KNIME is adding a new library to an existing environment. Does that functionality not exist or could you lead us the right way?

Thanks a lot
Lars

#2

Hi @laval

We currently do offer a way to add packages to conda environments from the Python preferences.
In our documentation you can read how to add packages from the command line: KNIME Python Integration Guide

You should also take note of the Conda Environment Propagation node, it helps you recreate conda environments on target machines, so that you have to create a specific conda env only once.

best,
Gabriel

#3

Hi Gabriel,

sorry my explanation was a bit misleading. We want to solve that via KNIME nodes automatically.
The issue with the node you mention is that it creates a new environment - we did not find a way to simply add a package via KNIME node to an environment we already created to include this procedure in our self developed KNIME nodes. We want to avoid to add new environments each time we need a new package.

Best,
Lars

#4

Hi @laval –

To help create reusable nodes written either partially or entirely in Python, a Component can provide a convenient way of encapsulating your nodes and functionality with a Conda Environment Propagation node. This ensures portability of your Component – if used on a system where all the Python dependencies are already satisfied, the Conda Environment Propagation node does nothing, but if used on a system where dependencies are absent, it recreates an env with the required dependencies.

The goals of portability and reusability are achievable when all dependencies are specified as part of any node or Component. The cost of creating yet another conda env on disk, especially when only the minimal required dependencies are used to define the conda env, is generally expected to be small enough that it will not be a concern. Unless you have an exceptionally large conda env?

There is currently no node for the purpose of adding a single package to an existing conda env (and no node to first investigate if that package is already installed, and no node to remove a package, etc.) but I can offer the following options to achieve this:

  • use the conda API in a Python Script node
  • use the External Tool node to invoke conda commands directly

I hope what I wrote provides some encouragement to further consider using the Conda Environment Propagation node to ensure all dependencies are satisfied for a node/Component. If that is still not a good fit for your situation, I would be very interested to hear more about it but I will also encourage looking at the conda API before looking at the option of the External Tool node.

Hope this helps,

Davin

#5

Hi Davin,

thanks a lot for your comment! I will forward that to our developers and maybe they can make use of your information already or give additional information.

Best,
Lars