Transpose Node is slow

I have a workflow where I need to transpose data. When the data is around a few hundred rows, the transpose works fast. Now, I am looking at a much larger data set (1754785 rows), and the transpose is where the workflow is slowing down by almost 2 hrs.

My machine is a Windows Server 2016 with 8 CPU and 512 GB of memory out of which 400 GB is allocated to the KNIME analytics platform in the knime.ini file. There is nothing else running on this server except KNIME. When running this workflow, only 15% CPU is used and only 200 GB of memory is used. With other workflows, I have seen that CPU and Memory are utilized at a much higher levels.

Why is this node not taking more resources? Why is this node taking so long to execute? What can I do to improve performance of this node?

I’ll recommend you to look at discussion here

Anyway, what you try to accomplish? If it really requires to transpose that huge amount?

I am trying to convert a data set that we are bringing in from AWS DynamoDB and converting it into a specific table format before we can load it into our database.

I know and understand that we can always change the approach…but my question is why is the node not using system resources…am I missing anything?

I will review that link…i think that is something we have not tried…i will test it and keep u posted.

This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.