Inconsistent error message of python source node

Hi, KNIMEer!

Set code in python source node:

print("Hi There!")
print("Fatal error!", file=sys.stderr)
  1. As default log level(warn) in KNIME, we see nothing in KNIME console, but we can see these two lines display at View: Standard output and View: Error output separately. Should we see the second line at KNIME console?

  2. If we set log level to info at *File > Preferences > KNIME > KNIME GUI > Console View Log Level*. We can see these two lines display at KNIME console. Both of them start with INFO Python Source .....

  3. If we add raise(ValueError("Oops")) to the code below. We see nothing at View: Standard output and View: Error output. We only see message at KNIME console.

It is not a big problem, but it is inconsistent.

Have a nice day! :smiley:

Btw, if I want log message to KNIME console, but don’t want change Console View Log Level to Info(keep warn).

What can I do?

It seems there are warning exceptions in Python.

raise(UserWarning("wow"))

But this would fail the node.

ERROR Python Source        0:4        Execute failed: wow
Traceback (most recent call last):
File "<string>", line 38, in <module>
UserWarning: wow

Any suggestion?

Thanks

Hi @HaveF,

Thanks for the feedback! :slight_smile: I hope the following helps illustrating our rationale behind the handling of the various outputs of a Python program.

The problem with stderr is that it does not perfectly match any of KNIME’s logging categories (DEBUG, INFO, WARN, ERROR). Here is what the POSIX standard has to say about the standard streams:

At program start-up, three streams shall be predefined and not be opened explicitly: standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output).

That is, stderr is not exclusively used to report errors but also to output less critical technical information about a program. (So it would partly match KNIME’s ERROR logging category but partly also its DEBUG category.)
Prior to KNIME Analytics Platform 4.1, we in fact logged stderr to ERROR. This became a problem in practice because there are some popular Python libraries that we also use in KNIME (most notably TensorFlow and Keras in our deep learning integrations) that – in line with POSIX – use stderr to output rather trivial diagnostics. E.g. Keras would output via stderr that it uses TensorFlow as its computational backend. TensorFlow in turn would report that it successfully opened an external library, etc. This resulted in each execution of a deep learning KNIME node first producing at least two big fat red ERROR lines in the log before continuing and successfully completing its task, which we considered being really confusing to users.
That is why we decided to treat stderr like stdout in terms of logging and redirected it to the INFO log category. In contrast to logging, we have the luxury of just adding as many Views to Nodes as appropriate. That is why the Python nodes still have separate Views for stdout and stderr.

So you can see, KNIME’s current behavior is really the (intermediate?) result of a longer process and not set in stone at all. We appreciate any feedback to further improve the behavior (given the technical constraints described above) and make it feel less inconsistent to users! :slight_smile:

That is because Errors raised in Python are not necessarily output via stderr all of the time. They are when e.g. running Python from the terminal and an Error is not handled by the Python program (via try/except), but they do not have to. Think of Errors as a type of program-internal signalling mechanism (and stdout, stderr as external ones) that can produce an external signal (e.g. an output to stderr).
In the case of KNIME, we handle the raised Errors and translate (=external signal) them to the perfect counterpart in KNIME: node execution failure (i.e. the white cross on red background displayed over a KNIME node when its execution failed). The execution failure of a Python node (or any KNIME node) is logged to KNIME’s log, which is why you can see a message there but not in the node’s Views.

As above, this is probably a very technical answer, so we welcome any suggestions to better reconcile the technical details with what users actually expect :slight_smile:.


Warnings in Python are not raised like Errors but via the warnings module:

import warnings
warnings.warn("wow")

This will produce a WARN log entry in KNIME as intended:

WARN  Python Script    0:12    <string>:2: UserWarning: wow

Marcel

3 Likes

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.