Hierarchy from transactions

Hi! @takbb @evert.homan_scilifelab.se

You both helped me with the hierarchy transaction building workflow, and I tested this today first time with real live data.

Result of running the workflow was 1.4 million iterations and a runtime on 20-30 hours (estimate) :smiley: I don’t see any way of making this to run faster, but would either of you have any ideas? :smiley:

I have attached the model and the example data.

hierarchy builder.knwf (146.7 KB)
hierarchy test.xlsx (42.5 KB)

Hi @Data_consumer ,

Yes that is rather a lot of iterations for a loop! I was trying to remind myself how this all worked, and then I noticed something I think I overlooked in the original.

Add two Duplicate row Filters at the position shown here:

The pair of Joiner nodes had caused duplications to occur which I hadn’t spotted. By removing those duplications the cross joiner comes down to 109000 rows instead of 1.4 million. That should at least reduce the time down to 10% of the original estimate, which is a start :-).

On my pc it is taking around 40 seconds to perform 1000 iterations, so if it maintains that it is 109 x 40 seconds = 70 ish minutes.

If I can think of anything else I’ll let you know.

2 Likes

Yes, a lot better!!! :pray: :smiley:

Thanks Takbb! this should be feasible, as I don’t need to build this hierarchy too often :slight_smile:

1 Like

:+1: @Data_consumer,

It looks like the config of the Loop End needs modifying too as it needs to be set to !Generate New RowIDs" or it can fail (you may already have discovered this after an hour of processing!)

1 Like

Thanks @takbb! Yes it froze after running for a while. Now working perfect, about an hour of running time! :slight_smile: Thanks for the help!

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.