java leap - split database for workflow execution

KNIME Analytics Platform
1

Hi all

i have a workflow containing a few pivot table.

i have collecing customer ID and all the transaction => the same customer ID has differnent line

i have doing several Pivot table to combine the file.

when i run my workflow up to 10 millions of line, it is fine but for my entire base 50 millions line the system doenst follow and i have java heap space.

i have optimize my computer ressource, my workflow add garbage node but it still can not fix my problem

is there a way to split my file and execute it then combine it at the end?

my file has customer ID (String) that i can split and combine at the end when all pivot table of datas linked to the customer ID are done

many thanks in advance for your help

Benoit

2

You can try a Group Loop on the whole process as long as you can find a way to group them into adequate sized batches.

Not sure what you are working on, but perhaps grouping it by business location / region, some kind of customer categorization, or transaction #?

It won’t be fast though…

Alternatively you could write to a temp H2 database in memory and then do complex joins. That is likely how I would approach it.

2 Likes
3

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.