I was very excited to find the KNIME Testing Framework. Unfortunately, according to Google, there are no how-to’s to explain how to use this, especially not on KNIME Server.
There is one example workflow here, but it doesn’t work on its own – there are a bunch of problems with the workflow itself and it does not run as-is, and seems extremely complex without explaining why.
Could someone from KNIME please write a how-to for this framework and share it? I’m looking for the basics, including:
- How does a workflow know whether it’s run as a test? The “Table Difference Checker” must only be run when it’s a test, with a specified input. And it must not be run when real data is in, or else the workflow will fail. Knowing that, how do we create a workflow that can use test data when it’s supposed to and use real data when it’s sent in without failing the tests?
- How do we run a test on KNIME Server? (There’s no menu item for “Run as workflow test” on the server – that menu exists only on the desktop version).
And, FYI, in case anyone at KNIME wants to improve the example workflow, here’s the reasons that workflow doesn’t work as-is:
- It errors on all “Table Reader” nodes, starting with the very first one – it can’t read any local files (are they corrupted?). The error is “Unable to read spec from file” and will neither let me reset the node nor execute it.
- As a side problem, this workflow depends on many optional plugins – which makes it hard to load. I can strip those sections out or install it, but that takes a lot of time – this is clearly not a basic workflow nor one that can be understood if you’re not familiar with these plugins.
- The workflow seems to have a lot of extra nodes – not just the necessary ones to explain the idea.
- The workflow has no documentation.
- To do testing & deployment, the workflow needs to be run in two different modes: (a) called as a workflow with real data, and (b) for testing. It should only run the “Table Difference Checker” when it’s run as a test. How can the workflow figure out which mode it was run in and do the table comparison only in the right case?
Thank you for the feedback. I’ll check for any internal documentation we have on running testflows on KNIME Server. In the interim, here are some details on usage in KNIME Analytics Platform:
Getting Started with KNIME Testing Framework extension
The KNIME Testing Framework UI Extension enables the development of test cases for workflows and components.
Why create test workflows (or “testflows”)?
- The purpose of a test workflow is to detect issues with workflows and components as early as possible.
- Issues could mean…
- A workflow or component fails to execute for unexpected reasons
- A workflow or component produces wrong output
- A workflow or component doesn’t show the appropriate expected error messages
- Reasons for these issues could include…
- Updates to KNIME Analytics Platform
- Changes to node inputs/outputs during an update
- Changes in external services (e.g. REST API integrations)
- Changes to a nested component
Testflow Configuration Node
The Testflow Configuration node is the primary utility for configuring testflows. It allows us to define which components and/or nodes in the workflow are expected to fail or succeed when the workflow executes.
As you can see below, there are a number of configuration options available.
The “Node settings” tab in the configuration allows us to specify which nodes are expected to succeed or fail as well as allowing us to specify required warning messages. Use the pattern
_!_.+_!_ to replace parts of the error/warning messages that are variable across workflow runs. You can read more in the node description of the Testflow Configuration node.
- Right click on a workflow from the KNIME Explorer and select Run as workflow test
- Click OK
- The testflow will then begin executing and the results will be displayed in the JUnit view.
- By browsing these results it’s possible to audit any failing nodes or unexpected behavior as well as the failure trace.
Helpful nodes for testing the output of a node or component
There are many nodes which are useful for building testflows, but here are a few data validation nodes which frequently come in handy:
- Table Difference Checker
- Determines difference between two tables
- Flexibility for ignoring certain factors
- Table Validation
- Validates the schema of a table
- Flexibility for handling extra columns
- File Difference Checker
- Image Difference Checker
- MODEL Difference Checker
- PMML Difference Checker
Here’s an example workflow for you: Example Testflow – KNIME Hub
did you already see the blog post we wrote last year about this as well?
Hi @Iris & @sjporter thank you so much for your replies! I’ve been slow to repost because I’ve been reading through all of that great information and making sure I wasn’t missing anything.
@sjporter – thank you for the great writeup and the example flow. That’s really helpful! I can see how this works now.
@Iris – thanks for the link. Thanks for that blog post.
I think documentation I cannot find is how this works with KNIME Server. On KNIME desktop, if you right click a workflow, or even a folder, you can “Run as workflow test.” However, that does not show on my server (I did install the test feature UI on the server but that doesn’t seem to add it).
In the blog Iris posted, the very end of the article ends saying “In a future post we’ll show how you can use this foundation and the KNIME Server to starting doing continuous delivery and deployment of production workflows”. Was there ever a follow-up post? Or was that functionality built? In particular, is there any way to run these tests in an automated way using KNIME Server?
Here’s a related forum post: Run knime as workflow test in a batch mode
Executing testflows in batch is done via command line. I don’t believe there’s an easy way to run the testflows (with expected output) on KNIME Server today.
Hi @sjporter ,
Thanks. In that case, if KNIME Server cannot be to automate these tests, I highly recommend correcting the following two articles about the test framework (both of them say we can use KNIME Server with the test framework):
Automated Workflow Testing and Validation | KNIME says “With KNIME Server execution of these tests can be scheduled, allowing the process to be fully automated.”
Enter the era of automated workflow testing and validation | KNIME says “you can use this foundation and the KNIME Server to starting doing continuous delivery and deployment of production workflows”
After more internal discussion, I believe the KNIME Testing Framework extension is what you’re looking for. The Testing Framework is a completely separate application - after you install the extension, look in the
plugins/org.knime.testing_*/doc directory of your KNIME Analytics Platform installation to find the documentation!
Thank you @sjporter ! I have some reading to do (I had no idea that any plugins stored local pdf documentation! Great suggestion to look there, thank you).