Efficient way to split data .. too many rule-based row splitters!

[TigerCole]

I have a set of data that I need to split many, many times … My data set is approx. 372 million rows of transaction data for products and sub-products that need to be split into individual sets.

At the moment I have had to create a “splitter” tree with about 35 rule-based row splitters and I will need another 10 or 20 more. Just keeping track of what is going where and trying to balance the data flow through the tree is a nightmare. Ideally, I would like a single rule-based splitter that splits everything once, or maybe twice (product then sub-product).

Is there a more efficient way to do this disaggregation of the data.

tC./.

[ipazin]

Hi there,

Splitter nodes only have two outputs so can’t do it in less splits I think.

What you can try is to use Rule Engine node or Rule Engine (Dictionary) node to associate each set with a unique identifier (if you don’t have one already). Then using Group Loop Start node gives you option to write that data somewhere, send it to report, perform same calculations for each set or something else depending on your need.

Hope this helps.

Br,
Ivan