How do other people go about debugging loops with KNIME? It is something that has puzzled me for quite a while.
Sure you can manually pause execution and then “step loop execution” but what do you do when you have a loop of around 10000 iterations, and you know that something is occurring at about iteration 7000, so you want to stop the loop at iteration 6999 and then step it onwards so you can see what is happening?
You can add row filters to narrow down your dataset, but how do you do it without changing anything about the data that is being processed. How do you get it to stop at iteration 6999 and then step through the processing so that you can manually inspect what is happening?
This is where I sometimes find myself, and perhaps I am missing something with the Breakpoint node, but once it has “broken” the flow, I have yet to see how to “continue” the flow as you would with a traditional breakpoint in a typical programming IDE.
Let’s take an example of a smaller loop using the example breakpoint workflow on the hub, which demonstrates that a breakpoint can be inserted to stop at iteration 15 on a 1000 iteration loop.
Yes, it breaks at iteration 15, and I can check the data/flow variables in the loop up to that breakpoint but how do I debug “what happens next”? If I am missing something, then sorry for wasting your time, because the rest of this will turn into an academic exercise 
But if, for the sake of argument, there is no way to restart the flow from that breakpoint, then stay with me because perhaps this will be useful 
I’ve been thinking for a while that it would be good to stop a workflow programmatically, when a given condition occurs (e.g. the 6999th iteration), but then be able to continue the loop (e.g. step it onwards from that point, or just simply do a few checks and then allow it to continue on its way up to iteration 10000).
In the absence of finding a way to make this happen with the breakpoint node, I decided there was nothing for it but to have a think about how this could be done using the standard nodes. You may have already guessed that this came about through a very specific need that had arisen in a workflow I was debugging for work that contained such a loop.
Here is the result of my attempts at making this happen, in the form of a component I have just put on the hub:
This component doesn’t modify any data, and contains no configuration. All it does is halt the workflow. The idea is that you put it on a conditional branch using something like a Java If node, to handle the conditions for it to “Halt”.
(At some point I might add config to give it a sister with similar “powers” to the breakpoint node, but for now it does the job intended)
Let’s look again at the workflow from “Examples.06_Control Structores/04_Loops/17_Usage_of_Breakpoints_in_Loops” as an example:
Here it shows the use of a standard Breakpoint node, to stop the loop at iteration 15, which it does very successfully. However, as the annotation on the Loop End node says, the end is never reached because once stopped, the breakpoint remains in a “failed” state. What I want is to be able to somehow “un-fail” it after some manual debugging and inspection, and allow the loop to complete.
So here is the same workflow but with a conditional call to my component:
(NB The “deprecated” nodes remain simply because this is an old example, but I wanted to write an exact equivalent)
Here, a Java If contains a condition that directs the flow to the lower branch on iteration 15, and also on iteration 150.
if ($${IcurrentIteration}$$ == 15 || $${IcurrentIteration}$$ == 150 )
{return 1;}
else
{return 0;}
The image shows the workflow stopping at iteration 15
but the cool thing is that I can then click on the Loop End node, and step the loop on… something that I am seemingly unable to do with the Breakpoint node.
and if I step it twice (once to re-commence execution of the current iteration and the second to go onto the next iteration), you can see the loop continues and now arrives at iteration 16…
I can also resume loop execution, and it continues until the next breakpoint condition is reached at iteration 150
Like I said at the beginning, if there was already a way to do this, I never found it, and this turned out to be an exercise in how such a component could be written. In this case, it uses Python to achieve the “break” in execution. It does this by generating an error condition which halts the flow.
I wanted to do it using a java snippet instead but I didn’t found a way as the equivalent code in a java snippet causes KNIME to report the error, but doesn’t actually halt the workflow. [EDIT - I realise now I could possibly have done this by throwing an Abort(), in java snippet, so I will port this to java snippet at some point. Then it will work even where python isn’t configured]
Having stopped the flow though, how does it allow it to clear the error that made it stop? Well, that was achieved by having python write a “flag” file to the local workflow temporary area. This allows the python to alternate between generating an error and not. So if executed after it has generated an error, it clears the error and continues, but otherwise it generates an error. Other nodes such as “Fail in Execution” couldn’t be incorporated because they have one job “fail” and they are too good at it! 
Thanks for reading. I hope you found it useful.
Oh, here is the demo…
Debugging Loops with Halt and Await Execution Component.knwf (31.0 KB)
