Hi there,
I was very excited to find the KNIME Testing Framework. Unfortunately, according to Google, there are no how-to’s to explain how to use this, especially not on KNIME Server.
There is one example workflow here, but it doesn’t work on its own – there are a bunch of problems with the workflow itself and it does not run as-is, and seems extremely complex without explaining why.
Could someone from KNIME please write a how-to for this framework and share it? I’m looking for the basics, including:
How does a workflow know whether it’s run as a test? The “Table Difference Checker” must only be run when it’s a test, with a specified input. And it must not be run when real data is in, or else the workflow will fail. Knowing that, how do we create a workflow that can use test data when it’s supposed to and use real data when it’s sent in without failing the tests?
How do we run a test on KNIME Server? (There’s no menu item for “Run as workflow test” on the server – that menu exists only on the desktop version).
And, FYI, in case anyone at KNIME wants to improve the example workflow, here’s the reasons that workflow doesn’t work as-is:
It errors on all “Table Reader” nodes, starting with the very first one – it can’t read any local files (are they corrupted?). The error is “Unable to read spec from file” and will neither let me reset the node nor execute it.
As a side problem, this workflow depends on many optional plugins – which makes it hard to load. I can strip those sections out or install it, but that takes a lot of time – this is clearly not a basic workflow nor one that can be understood if you’re not familiar with these plugins.
The workflow seems to have a lot of extra nodes – not just the necessary ones to explain the idea.
The workflow has no documentation.
To do testing & deployment, the workflow needs to be run in two different modes: (a) called as a workflow with real data, and (b) for testing. It should only run the “Table Difference Checker” when it’s run as a test. How can the workflow figure out which mode it was run in and do the table comparison only in the right case?
Thank you for the feedback. I’ll check for any internal documentation we have on running testflows on KNIME Server. In the interim, here are some details on usage in KNIME Analytics Platform:
Getting Started with KNIME Testing Framework extension
The purpose of a test workflow is to detect issues with workflows and components as early as possible.
Issues could mean…
A workflow or component fails to execute for unexpected reasons
A workflow or component produces wrong output
A workflow or component doesn’t show the appropriate expected error messages
Reasons for these issues could include…
Updates to KNIME Analytics Platform
Changes to node inputs/outputs during an update
Changes in external services (e.g. REST API integrations)
Changes to a nested component
Testflow Configuration Node
The Testflow Configuration node is the primary utility for configuring testflows. It allows us to define which components and/or nodes in the workflow are expected to fail or succeed when the workflow executes.
As you can see below, there are a number of configuration options available.
The “Node settings” tab in the configuration allows us to specify which nodes are expected to succeed or fail as well as allowing us to specify required warning messages. Use the pattern _!_.+_!_ to replace parts of the error/warning messages that are variable across workflow runs. You can read more in the node description of the Testflow Configuration node.
Executing testflows
Right click on a workflow from the KNIME Explorer and select Run as workflow test
Click OK
The testflow will then begin executing and the results will be displayed in the JUnit view.
By browsing these results it’s possible to audit any failing nodes or unexpected behavior as well as the failure trace.
Hi @Iris & @sjporter thank you so much for your replies! I’ve been slow to repost because I’ve been reading through all of that great information and making sure I wasn’t missing anything.
@sjporter – thank you for the great writeup and the example flow. That’s really helpful! I can see how this works now. @Iris – thanks for the link. Thanks for that blog post.
I think documentation I cannot find is how this works with KNIME Server. On KNIME desktop, if you right click a workflow, or even a folder, you can “Run as workflow test.” However, that does not show on my server (I did install the test feature UI on the server but that doesn’t seem to add it).
In the blog Iris posted, the very end of the article ends saying “In a future post we’ll show how you can use this foundation and the KNIME Server to starting doing continuous delivery and deployment of production workflows”. Was there ever a follow-up post? Or was that functionality built? In particular, is there any way to run these tests in an automated way using KNIME Server?
Executing testflows in batch is done via command line. I don’t believe there’s an easy way to run the testflows (with expected output) on KNIME Server today.
Hi @sjporter ,
Thanks. In that case, if KNIME Server cannot be to automate these tests, I highly recommend correcting the following two articles about the test framework (both of them say we can use KNIME Server with the test framework):
After more internal discussion, I believe the KNIME Testing Framework extension is what you’re looking for. The Testing Framework is a completely separate application - after you install the extension, look in the plugins/org.knime.testing_*/doc directory of your KNIME Analytics Platform installation to find the documentation!
Thank you @sjporter ! I have some reading to do (I had no idea that any plugins stored local pdf documentation! Great suggestion to look there, thank you).