transpose node is getting very slow

#1

I have a table of 8000rows and 600 columns that I transpose iteratively in a loop before I do other data manipulation. I noticed in the first few hundreds of iteration, the run of the transpose is quite fast (one or 2 seconds). Then it gets slower and slower. I have enougt memory Xm 8000mb.
The only paprameter I see in the node is the table chunk that I keep at 10.
Is there a way to make the execution of this node faster?
Thanks,

0 Likes

#2

HI there @zizoo,

you are using the latest version? How many iterations you got? In each iteration you are transposing 8.000 rows and 600 columns or?

Maybe it would be best if you share this example workflow and someone could take a look.

Br,
Ivan

0 Likes

#3

Hi @ipazin,
I have 8000 iterations for a manipulation on a table to 8000 rows and 600columns.
I have the latest version.
How I can choose the optimum value of table chunk in transpose node. The highest is the better?

0 Likes

#4

Hi there @zizoo,

if it is the same table every iteration why not first transpose it and then loop over it 8000 times? What loop are you using?

Br,
Ivan

0 Likes

#5

Hi @ipazin
I am using column list loop. So in each iteration, I use one column of the big table and at the same iteration, I work on that column as a row after I do the transpose on the full table.

0 Likes

#6

Hi there @zizoo,

I think I get it now :slight_smile:

Can you share example for someone to check it out?

Br,
Ivan

1 Like