Issue with recursive loop

[Julie_sha]

Dear all,
I’m a real beginner on knime and I’m currently facing to issue with recurcive loop. I’m working on it since several days without find a solution. That’s why I really need your help please.

Node 498 are data updated that I would like to reuse them for the next iteration of the loop “chunk loop”.
Workflow stops at the beginning of the metanode 445. The first node I have is a cross joiner : the message “can’t merge FlowVariable stacks! (likely a loop problem.)”.

I read some topic on the subject but I didn’t to fix my issue.

Thank you in advance for your help.

[Julie_sha]

Please fin here the workflow and the input file.
Julie_warehouse_planning.knwf (101.2 KB)
Revue des DLP.xlsx (1.0 MB)

[takbb]

Hi @Julie_sha ,

Welcome to the KNIME community. As a real beginner in KNIME, I don’t envy you trying to cut your teeth not just on loops, but recursive loops. They are possibly the most complex loop structure that KNIME has, and there are many experienced KNIMERS who stay away from them.

So well done for getting this far, and also thank you for posting a sample workflow, and some data. That always helps to give better assistance.

Incidentally, with the spreadsheet supplied separately, on adding this to the workflow, it causes the spreadsheet config to change, which stops it even getting to the cross-joiner.

I have re-uploaded you flow (with no other changes as far as I know), but with the spreadsheet accessed from the workflow’s data folder. This will make it easier for others to look and assist, without having to fight to get it working initially.

forum - Julie_warehouse_planning with embedded data.knwf (992.2 KB)

So that all said, I think we will need more information as it’s difficult to work out what it is you are wanting the workflow to do. I can see you are trying to process the two sheets from Excel file and join them together in some way, but what is the overall purpose, and maybe if you can give a summary of the steps you are trying to achieve, we can help in pointing you at a solution.

As you say, the flow doesn’t work and the reason for that is that the two loops are partially coinciding whereas KNIME is going to need one loop to be fully inside the other, or separate from it.

In terms of the current workflow, here are the specific problems that I can see without further investigation. They are mistakes easily made when first starting out, so don’t treat this as a criticism, but I am trying to indicate why what you have cannot currently work.

(1) The two Loop Start nodes are appearing to merge together here. One loop needs to be inside the other, or both be separate. When you consider that you have two “cycles”, how are these cycles supposed to be related to each other when one is not inside the other? This is where we need to know what you are wanting it to do.

(2) The Loop End node. As far as KNIME is concerned, this just ends a loop but it doesn’t know which one, and even if we can say from the name (i.e one loop is recursive but the other isn’t) without one loop being wholly contained inside the other, who many iterations of one loop are you expecting to occur, for each iteration of the other?

(3) The branch exits one of the loops and bypasses the first Loop End. What are you expecting it to do at this point, as data here is involved in an iteration, so it cannot just do its own thing.

(4) What data are you actually wanting to use for the next iteration? This is where the data that has branched out from the inner loop has appeared, so what is this data?

The cross-joiner “flow variable” stack message has come about because of point (1) above, in that two loops appear to be merging which isn’t allowed.

So if you can give an overall summary of what task you are trying to achieve (i.e. we have the input data, but what is the form of the intended output?) somebody will be able to help. Just a small example of partial output should be enough.

[Daniel_Weikert]

Can you build your workflow in a way that your loops are nested?
I personally always want to make sure that my loops are set up that way so that the inner loops are executed before returning to the outer loop
So in your example something like connecting the flow variable output port of the recursive loop to your chunk loop to because you also first close the chunk loop before closing the recursive loop start.
Beside this listen to @takbb
He always has great ideas and solutions
br

[Julie_sha]

Many thanks for your reply and comments.

I will try to explain what I want to do :

Workflow objectives :

• To be able to affect new date to each object’s reference according the “Nouvelle Date de DLP possible” (data already calculated in the excel file). That means this date is the first possible date.
• Respect the planning capacity

In the workflow, I process object’s reference for which the associated date (“Nouvelle Date de DLP possible” name of the field in excel file input) exceeds the capacity.

• Excel reader : planning capacity (Node called “Date ouverte”)**
  Node 443 = formatting planning capacity. Expected results : date and associated capacity. Example : date = 2022-11-07 à capa = 25 (that means we plan 25 objects for this date).
  Node 443 = determines all possible combinations.

• Excel reader : objects (reference) to plan (Node called Export)

The workflow must respect the date constraint : we need to affect a new date to the objects which the “Nouvelle Date de DLP possible” is the closer first. That’s why I put a “chunk loop start” after the GroupBy Node 189.

Count(Actual_DLP) = number of objects to replace in the planning

In this case, there is 57 objects (row0) to replace in the planning before treat the next line (row1).

Node 445 : determines the needs :
For 2022-11-24, 57 objects to replace. That means I will use 25 positions on 2022-11-24, 25 positions on 2022-11-25, and 7 positions on 2022-11-24 :

Node 446 :
First output : Objects references are affected to new date.

Second output : I get back the capacity used during this first iteration because I need to update node 443 (there is no more capacity for date 24-11-2022 and 25-11-2022 and for 28-11-2022 free capacity is decreased (of 7).

→ The next iteration must treat row1 (node 189) in the same way as steps previously described but we need to take in account the new values for the capacity.

Hope it’s more clear. Many thanks for your advices
Julie

[takbb]

Hi @Julie_sha

Thank you for your explanation of the workflow which is helpful, but I have not been able to spend much time on this. I also don’t fully understand the data so at the moment I’m making guesses at a possible re-structure of the workflow without really knowing it in detail, so I may be off the mark here.

Assuming you still haven’t got it working, take a look at the attached and see if it gets you anywhere closer. I used one of @Daniel_Weikert’s suggestions, at (1) to nest the main loops (node 499 and node 495), and also you will see that the Loop End at Node 496 now passes the lower data out through the loop at (2) so that the branch doesn’t “exit the loop early”. Whether this is right or not I am not yet sure, but it does at least allow it to process!

It at least begins to iterate now, but it then fails inside Node 446

The failure there is because the Counting Loop uses CX_Capa as the parameter, but at one point CX_Capa is assigned a value of -5 which, being negative, is not valid for a loop counter. I wasn’t sure of the purpose of this loop, as it simply iterates with nothing in between.

forum - Julie_warehouse_planning with embedded data 2.knwf (998.3 KB)

Anyway, maybe this moves you in the right direction (but maybe not!). Please feel free to come back with more questions. I think, as the workflow is quite complex, this is something that may require a few questions/responses here and maybe I will also begin to understand it more, and possibly be of more help.

[Julie_sha]

Hello @takbb, @Daniel_Weikert
Many thanks for your help on this topic. I found the solution in a different way and that works! I mean the results expected are the results that the worflow provides. Not totally optimized…
After your post, I decided to think the workflow differently.
Please find in attached the workflow, maybe it could help someone else.

Have a good evening & thank you again.
Julie
DLP loop.knwf (408.7 KB)