Parallel chunk loop not working with pivot node

Hi All

I am using the pivot node in a parallel chunk loop and it doesn’t seem t like it returning error
Execute failed: Cell count in row “Row0_#1” is not equal to length of column names array: 36 vs. 40

understand the node is in Lab - so one for devs to pick up i guess :smiley:

ciao
Gavin

Hi @Gavin_Attard

Unfortunately the -Parallel Chunk End- node does not allow to concatenate tables with different number of columns :

as it is the case with other -* Loop End- nodes, for instance:

This is most probably the problem.

Therefore the solution is to make sure that all the tables arriving to the input of the -Parallel Chunk End- node have the same number of columns.

For instance, the following workflow shows how to solve this after transposing chunked tables with different number of columns:

The example is based on transposing but pivoting could be solved in the same way.

The dummy worflow example can be found in the hub here:

Hope it helps.

Best
Ael

3 Likes

Thanks @aworker

Ya, i suspected as much, and moved the pivoting node out of the chunk loop.
I am now hitting the Pivot node being really slow on large data sets…

Anyone got a way to speed this up, or knows why it takes so long?

Hi @Gavin_Attard

Thanks for the feedback. It sounds then that my answer replies to your question about why the parallel chunk loop was failing.

If you think this question is solved, please feel free to check :white_check_mark: my post with the workflow solution as solved so that other people can more easily find it when searching for a solution to this problem.

Best
Ael

1 Like