Parallel chunk loop not working with pivot node

[Gavin_Attard]

Hi All

I am using the pivot node in a parallel chunk loop and it doesn’t seem t like it returning error
Execute failed: Cell count in row “Row0_#1” is not equal to length of column names array: 36 vs. 40

understand the node is in Lab - so one for devs to pick up i guess

ciao
Gavin

[aworker]

Hi @Gavin_Attard

Unfortunately the -Parallel Chunk End- node does not allow to concatenate tables with different number of columns :

as it is the case with other -* Loop End- nodes, for instance:

This is most probably the problem.

Therefore the solution is to make sure that all the tables arriving to the input of the -Parallel Chunk End- node have the same number of columns.

For instance, the following workflow shows how to solve this after transposing chunked tables with different number of columns:

The example is based on transposing but pivoting could be solved in the same way.

The dummy worflow example can be found in the hub here:

20230521 Pikairos Parallel Chunk Loop not Working if transposing or pivoting...

Hope it helps.

Best
Ael

[Gavin_Attard]

Thanks @aworker

Ya, i suspected as much, and moved the pivoting node out of the chunk loop.
I am now hitting the Pivot node being really slow on large data sets…

Anyone got a way to speed this up, or knows why it takes so long?