How do you use KNIME Testing Framework on KNIME Server? No readmes out there yet

Hi there,
I was very excited to find the KNIME Testing Framework. Unfortunately, according to Google, there are no how-to’s to explain how to use this, especially not on KNIME Server.
There is one example workflow here, but it doesn’t work on its own – there are a bunch of problems with the workflow itself and it does not run as-is, and seems extremely complex without explaining why.

Could someone from KNIME please write a how-to for this framework and share it? I’m looking for the basics, including:

  1. How does a workflow know whether it’s run as a test? The “Table Difference Checker” must only be run when it’s a test, with a specified input. And it must not be run when real data is in, or else the workflow will fail. Knowing that, how do we create a workflow that can use test data when it’s supposed to and use real data when it’s sent in without failing the tests?
  2. How do we run a test on KNIME Server? (There’s no menu item for “Run as workflow test” on the server – that menu exists only on the desktop version).

And, FYI, in case anyone at KNIME wants to improve the example workflow, here’s the reasons that workflow doesn’t work as-is:

  1. It errors on all “Table Reader” nodes, starting with the very first one – it can’t read any local files (are they corrupted?). The error is “Unable to read spec from file” and will neither let me reset the node nor execute it.
  2. As a side problem, this workflow depends on many optional plugins – which makes it hard to load. I can strip those sections out or install it, but that takes a lot of time – this is clearly not a basic workflow nor one that can be understood if you’re not familiar with these plugins.
  3. The workflow seems to have a lot of extra nodes – not just the necessary ones to explain the idea.
  4. The workflow has no documentation.
  5. To do testing & deployment, the workflow needs to be run in two different modes: (a) called as a workflow with real data, and (b) for testing. It should only run the “Table Difference Checker” when it’s run as a test. How can the workflow figure out which mode it was run in and do the table comparison only in the right case?
1 Like

Hey @YetAnotherKnimeUser,

Thank you for the feedback. I’ll check for any internal documentation we have on running testflows on KNIME Server. In the interim, here are some details on usage in KNIME Analytics Platform:

Getting Started with KNIME Testing Framework extension

The KNIME Testing Framework UI Extension enables the development of test cases for workflows and components.


Why create test workflows (or “testflows”)?

  • The purpose of a test workflow is to detect issues with workflows and components as early as possible.
  • Issues could mean…
    • A workflow or component fails to execute for unexpected reasons
    • A workflow or component produces wrong output
    • A workflow or component doesn’t show the appropriate expected error messages
  • Reasons for these issues could include…
    • Updates to KNIME Analytics Platform
    • Changes to node inputs/outputs during an update
    • Changes in external services (e.g. REST API integrations)
    • Changes to a nested component

Testflow Configuration Node

The Testflow Configuration node is the primary utility for configuring testflows. It allows us to define which components and/or nodes in the workflow are expected to fail or succeed when the workflow executes.

As you can see below, there are a number of configuration options available.

image

The “Node settings” tab in the configuration allows us to specify which nodes are expected to succeed or fail as well as allowing us to specify required warning messages. Use the pattern _!_.+_!_ to replace parts of the error/warning messages that are variable across workflow runs. You can read more in the node description of the Testflow Configuration node.

image


Executing testflows

  1. Right click on a workflow from the KNIME Explorer and select Run as workflow test
  2. Click OK
  3. The testflow will then begin executing and the results will be displayed in the JUnit view.
  4. By browsing these results it’s possible to audit any failing nodes or unexpected behavior as well as the failure trace.


Helpful nodes for testing the output of a node or component

There are many nodes which are useful for building testflows, but here are a few data validation nodes which frequently come in handy:

  • Table Difference Checker
    • Determines difference between two tables
    • Flexibility for ignoring certain factors
  • Table Validation
    • Validates the schema of a table
    • Flexibility for handling extra columns
  • File Difference Checker
  • Image Difference Checker
  • MODEL Difference Checker
  • PMML Difference Checker

image

1 Like

Here’s an example workflow for you: Example Testflow – KNIME Hub

1 Like

Hi @YetAnotherKnimeUser

did you already see the blog post we wrote last year about this as well?

Best, Iris

2 Likes