Stuck with errors in the python extension development tutorial

settonull · September 20, 2024, 9:43pm

Hi, using Knime 5.3.2, I’m trying to integrate some scripts we have to allow our users to use knime, but getting stuck starting up.

I’m using the tutorial from Create a New Python based KNIME Extension

I was able to successfully launch and run the original but seeing odd errors as I progress which makes me think I have something basic wrong.

When uncommenting the parameters, (tutorial step 10). it warns on start up - “Loading model settings failed: Parameter missing for key “some_param”.” Interestingly, it says this for whatever is the first parameter being loaded, but not the others. Once I click ok it proceeds to run, and I can see the parameters available in the config for my node.

Next going through step 11, it gives an import error when trying to run “to_pandas()” on my input table - “ImportError: DLL load failed while importing _ctypes: The specified module could not be found.”

This is being run from conda environment created clean with the command from earlier in the tutorial… It causes the node execute to fail obviously, which is a hard stop for me.

Anything obvious I might be missing? Thanks!

MartinDDDD · September 21, 2024, 8:00am

Hey there and welcome to the forum!

I moved this topic into the Node Development section to make sure the right people see it.

With regards to the issues you are encountering - can you share the .py file in the status where it is failing at step 10 (trying to solve that first to ensure you have something that is running before moving on to step 11)

I take after making the changes in the .py file and then saving it you also:

restarted KNIME
after re-opening it, you deleted the node and re-dragged it into the workflow area ?

settonull · September 21, 2024, 12:34pm

Thanks!

It gives the warning at whatever is the first paramater defined in the class -

class TemplateNode:
    # This node has a description

    # ### Tutorial step 10: Uncomment all of the following parameters and the 'is_numeric' method to create your first dialogue ###
    # (Restart KAP and drag&drop the node again from the node repository to let changes take effect)
    some_param = knext.IntParameter("Some Int Parameter", "The answer to everything", 42, min_value=0)

    another_param = knext.StringParameter("Some String parameter", "The classic placeholder", "foobar")

    double_param = knext.DoubleParameter("Double Parameter", "Just for test purposes", 3.0)

It gives the warning on startup, so before it even fully loads, but yes I deleted the node and re-added it. It always just gives one warning on whatever the first parameter is, if I make another_param first it warns about that one instead…

Other than having to click “ok” through the warning at startup it seems to work though? still shows them in the config -

Thanks for any help!

MartinDDDD · September 21, 2024, 1:10pm

Alright thanks a lot for the additional information!

Good news it that everything is working.

Let me try to explain what I think is happening here - I have seen this whilst I was developing an extension as well.

What you are seeing is not “really” an error - it is more a warning.

Let’s say you completed the previous step before 10. and then dragged the node into a workflow to test it out.

Then you went ahead and made changes to the .py file (uncommenting for step 10.), then saved the .py file, saved your workflow (with the node in its “state” as at step “9”), then you restart KNIME.

When KNIME re-opens and you open up the workflow that has the node with its “state” as at step 9, KNIME loads the node config for this node that still matches your code from step 9 (i.e. there were no parameters yet) and because your extension/code has changed it throws that warning.

I think it should go away by either:

after opening the config dialog and seeing that everything works, saving the workflow - as long as you don’t make any further changes to your code between your next saving / closing and re-opening, the error should go away
deleting the node from last time and dragging it back in (now with the state as at step 10)

Now on to step 11:

I take that you followed the instructions to comment out (green squares) and uncomment (red squares)

Can you share the detailed error again and maybe also the code again?

settonull · September 21, 2024, 1:40pm

Ah yes, if I save the workflow and reopen without changing anything it skips the warning. Does that mean the first time anyone new opened the workflow with my module it would give his warning?

And yes, just changed to comments around -

import logging
import knime.extension as knext
LOGGER = logging.getLogger(__name__)


@knext.node(name="My Template Node", node_type=knext.NodeType.LEARNER, icon_path="icon.png", category="/")
@knext.input_table(name="Input Data", description="We read data from here")
@knext.input_table(name="Tutorial: Input Data 2", description="We also read data from here") ### Tutorial step 11: Uncomment to create a new input port
@knext.output_table(name="Output Data", description="Whatever the node has produced")
class TemplateNode:
    #
    # This node has a description
    #

    # ### Tutorial step 10: Uncomment all of the following parameters and the 'is_numeric' method to create your first dialogue ###
    # (Restart KAP and drag&drop the node again from the node repository to let changes take effect)
    some_param = knext.IntParameter("Some Int Parameter", "The answer to everything", 42, min_value=0)

    another_param = knext.StringParameter("Some String parameter", "The classic placeholder", "foobar")

    double_param = knext.DoubleParameter("Double Parameter", "Just for test purposes", 3.0)

    boolean_param = knext.BoolParameter("Boolean Parameter", "also just for testing", True)

    column_param = knext.ColumnParameter()

    def is_numeric(column):  # Filter columns visible in the column_param for numeric ones
        return (
            column.ktype == knext.double()
            or column.ktype == knext.int32()
            or column.ktype == knext.int64()
        )

    column_param = knext.ColumnParameter(label="label", description="description", port_index=0, column_filter=is_numeric)

    #def configure(self, configure_context, input_schema_1):
    def configure(self, configure_context, input_schema_1, input_schema_2):  ### Tutorial step 11: Uncomment to configure the new port (and comment out the previous configure header)
        # return input_schema_1
        ### Tutorial step 12: Uncomment the following to adjust to the changes we do in this step in the execute method (and comment out the previous return statement)
        return input_schema_1.append(knext.Column(knext.double(), "column2"))
        ### Tutorial step 13: Uncomment to set a warning for the configuration, which will be shown in the workflow
        # configure_context.set_warning("This is a warning during configuration")

 
    # def execute(self, exec_context, input_1):
    def execute(self, exec_context, input_1, input_2):  ### Tutorial step 11: Uncomment to accept the new port (and comment out the previous execute header)
        # return input_1
        ### Tutorial step 12: Uncomment the following lines to work with the new port (and comment out the previous return statement)
        input_1_pandas = input_1.to_pandas() # Transform the input table to some processable format (pandas or pyarrow)
        input_2_pandas = input_2.to_pandas()
        input_1_pandas['column2'] = input_1_pandas['column1'] + input_2_pandas['column1']
        return knext.Table.from_pandas(input_1_pandas)
        ### Tutorial step 13: Uncomment the following line to use the parameters from the configuration dialogue (and comment out the previous return statement)
        # input_1_pandas['column2'] = input_1_pandas['column2'] + self.double_param
        # LOGGER.warning(self.double_param) # Tutorial step 14: Logging some warning to the console
        # exec_context.set_warning("This is a warning") # Tutorial step 14: Set a warning to be shown in the workflow
        # return knext.Table.from_pandas(input_1_pandas) ### Tutorial step 13: Uncomment

And here is the whole error from the log -

2024-09-21 09:35:30,535 : WARN  : KNIME-Worker-0-My Template Node 3:3 :  : CloseablePythonNodeProxy : My Template Node : 3:3 : Traceback (most recent call last):
  File "C:\Users\chris\AppData\Local\Programs\KNIME\plugins\org.knime.python3.nodes_5.3.2.v202409031144\src\main\python\_node_backend_launcher.py", line 999, in execute
    outputs = self._node.execute(exec_context, *inputs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\chris\AppData\Local\Programs\KNIME\plugins\org.knime.python3.nodes_5.3.2.v202409031144\src\main\python\knime\extension\nodes.py", line 1202, in wrapper
    results = func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\chris\grid3\knime\tutorial_extension\my_extension.py", line 50, in execute
    input_1_pandas = input_1.to_pandas() # Transform the input table to some processable format (pandas or pyarrow)
                     ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\chris\AppData\Local\Programs\KNIME\plugins\org.knime.python3.arrow_5.3.2.v202409040940\src\main\python\knime\_arrow\_table.py", line 249, in to_pandas
    import knime._arrow._pandas as kap
  File "C:\Users\chris\AppData\Local\Programs\KNIME\plugins\org.knime.python3.arrow_5.3.2.v202409040940\src\main\python\knime\_arrow\_pandas.py", line 50, in <module>
    import pandas as pd
  File "C:\Users\chris\anaconda3\envs\knime_env\Lib\site-packages\pandas\__init__.py", line 22, in <module>
    from pandas.compat import is_numpy_dev as _is_numpy_dev  # pyright: ignore # noqa:F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\chris\anaconda3\envs\knime_env\Lib\site-packages\pandas\compat\__init__.py", line 25, in <module>
    from pandas.compat.numpy import (
  File "C:\Users\chris\anaconda3\envs\knime_env\Lib\site-packages\pandas\compat\numpy\__init__.py", line 4, in <module>
    from pandas.util.version import Version
  File "C:\Users\chris\anaconda3\envs\knime_env\Lib\site-packages\pandas\util\__init__.py", line 8, in <module>
    from pandas.core.util.hashing import (  # noqa:F401
  File "C:\Users\chris\anaconda3\envs\knime_env\Lib\site-packages\pandas\core\util\hashing.py", line 24, in <module>
    from pandas.core.dtypes.common import (
  File "C:\Users\chris\anaconda3\envs\knime_env\Lib\site-packages\pandas\core\dtypes\common.py", line 26, in <module>
    from pandas.core.dtypes.base import _registry as registry
  File "C:\Users\chris\anaconda3\envs\knime_env\Lib\site-packages\pandas\core\dtypes\base.py", line 24, in <module>
    from pandas.errors import AbstractMethodError
  File "C:\Users\chris\anaconda3\envs\knime_env\Lib\site-packages\pandas\errors\__init__.py", line 6, in <module>
    import ctypes
  File "C:\Users\chris\anaconda3\envs\knime_env\Lib\ctypes\__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ImportError: DLL load failed while importing _ctypes: The specified module could not be found.

2024-09-21 09:35:30,713 : ERROR : KNIME-Worker-0-My Template Node 3:3 :  : Node : My Template Node : 3:3 : Execute failed: DLL load failed while importing _ctypes: The specified module could not be found.
org.knime.python3.nodes.PythonNodeRuntimeException: DLL load failed while importing _ctypes: The specified module could not be found.
	at org.knime.python3.nodes.CloseablePythonNodeProxy$FailureState.throwIfFailure(CloseablePythonNodeProxy.java:798)
	at org.knime.python3.nodes.CloseablePythonNodeProxy.execute(CloseablePythonNodeProxy.java:562)
	at org.knime.python3.nodes.DelegatingNodeModel.lambda$4(DelegatingNodeModel.java:180)
	at org.knime.python3.nodes.DelegatingNodeModel.runWithProxy(DelegatingNodeModel.java:237)
	at org.knime.python3.nodes.DelegatingNodeModel.execute(DelegatingNodeModel.java:178)
	at org.knime.core.node.NodeModel.executeModel(NodeModel.java:588)
	at org.knime.core.node.Node.invokeFullyNodeModelExecute(Node.java:1286)
	at org.knime.core.node.Node.execute(Node.java:1049)
	at org.knime.core.node.workflow.NativeNodeContainer.performExecuteNode(NativeNodeContainer.java:594)
	at org.knime.core.node.exec.LocalNodeExecutionJob.mainExecute(LocalNodeExecutionJob.java:98)
	at org.knime.core.node.workflow.NodeExecutionJob.internalRun(NodeExecutionJob.java:198)
	at org.knime.core.node.workflow.NodeExecutionJob.run(NodeExecutionJob.java:117)
	at org.knime.core.util.ThreadUtils$RunnableWithContextImpl.runWithContext(ThreadUtils.java:367)
	at org.knime.core.util.ThreadUtils$RunnableWithContext.run(ThreadUtils.java:221)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at org.knime.core.util.ThreadPool$MyFuture.run(ThreadPool.java:123)
	at org.knime.core.util.ThreadPool$Worker.run(ThreadPool.java:246)

C:\Users\chris\anaconda3\envs\knime_env was created from the tutorial’s command -

conda create -n knime_env python=3.11 knime-python-base=5.3 knime-extension=5.3 -c knime -c conda-forge

MartinDDDD · September 23, 2024, 2:44pm

Nope - this should only be happening in development I think. That said I have not yet tested what happens if someone uses a node in say your initial version and updates to a new version where there were changes (e.g. fields added to config dialogue)

Regarding the error that you encounter from what appears to be step 12:

Not sure what is happening there - going through the logs there’s some import from a module _ctypes failing… have not had the time to try and replicate, but will try and do so later this week.

Maybe someone from the KNIME team can help shed some light.

settonull · September 23, 2024, 3:29pm

It occurred to me over the weekend that it must certainly be my environment. Switching to the conda environment, simply importing pandas cause the same error. On a lark I just ran a conda update, and that fixed it. Not sure why the tutorial environment didn’t work, guessing something from conda-forge wasn’t playing nice? Seems like I’m all good now though, thanks for your help.

MartinDDDD · September 23, 2024, 5:01pm

Well I always love if problems solve themselves :-). Happy developing!