I sometimes have the problem that when I interrupt the execution of a loop, and then restart it, the results of the previous execution get piled on top of the new one. For example, if the loop processes a table with 500 rows, and the interrupted execution reached 200 rows until the interruption, the restarted execution will sometimes have 700 rows at the loop end output port.
In other words, Knime sometimes seems to fail to erase the results of previous interrupted loop executions. This happens even when I actively reset the loop nodes. Perhaps it is OS-dependent? This last happened on a Ubuntu Linux 18.04 machine, Knime 4.3.4.
Unfortunately, I have not been able to reproduce this phenomenon in a simple example workflow. Perhaps it only happens when working with large tables. I will post an update when it happens again or when I succeed in making an example workflow.
Hi @Aswin , that is a weird behaviour, I’ve never encountered this, and you say that you can reproduce this many times (not on simple example workflow, but on your project)?
Not that it should make a difference, but could you tell us what kind of loop you are doing?
the last time it happened it was in a loop beginning with Group Loop Start and ending with a Loop End (2 ports). But I don’t think it is exclusive to group loops.
And it doesn’t happen “many times”, but it does happen with some regularity, yes… Sometimes I cut loops or loop-containing components/metanodes and paste them back to the same place after an interruption, just to make sure I have a fresh copy of these nodes where this phenomenon does not happen.
This is a “real-world” Interval Loop for optimizing the “alpha” value (-log(alpha) actually) of a Lasso regression from the Python sklearn package. I interrupted the loop execution and then restarted it. I am pretty sure that, before restarting, I manually reset the Interval Loop Start node and the Component node inside the loop that contains the Python node and some other calculations. The configuration of the Interval Loop Start node is:
Result:
As you can see, somehow the first 4 rows are 2 duplicates.
When I reset the loop and re-execute it, all is fine again. No changes to the workflow.
In this case the computer has Ubuntu 20.04, 64GB, 24GB assigned to Knime, Knime 4.3.4
Hi @Aswin , sorry to hear that this happened again.
Are you able to put a demo workflow together and share it, and also explain the steps to reproduce this? It will be much easier for us to then fully understand how this is happening, and also see if with these steps, it happens on different systems.
As I said, it only happens occasionally, and it seems more likely to happen in data-heavy workflows. I also thought that parallel executing workflow branches increase the likelihood of this occurring, but this was not the case for the “real-world” occurence that I describe above. In this attempt to make an demo workflow, I once succeeded in achieving the opposite: rows disappearing instead of duplication:
The workflow on the left and the right are identical. As you can see, all the traffic lights are green and one would assume the two workflows to produce identical results, and most of the time, they do. But sometimes, after interrupting and restarting, they produce different results.
In this case, both empty table creators produce a table with a million rows and no columns. All Componets are identical and contain a Group Loop doing some dummy operations:
They all produce table with 3000030 rows and two columns:
…except the second component of the workflow on the right, which only produced with 600006 rows:
The workflow + data is huge, so I will attach the workflow in the reset state. However, i have not been able to reproduce the bug today; it somehow is a “Heisenbug”
So, trying to reproduce the bug with the attached workflow may be futile. I will post again when have a better idea of the circumstances that increase the likelihood of this bug manifesting itself.
