I have a workflow where I need to transpose data. When the data is around a few hundred rows, the transpose works fast. Now, I am looking at a much larger data set (1754785 rows), and the transpose is where the workflow is slowing down by almost 2 hrs.
My machine is a Windows Server 2016 with 8 CPU and 512 GB of memory out of which 400 GB is allocated to the KNIME analytics platform in the knime.ini file. There is nothing else running on this server except KNIME. When running this workflow, only 15% CPU is used and only 200 GB of memory is used. With other workflows, I have seen that CPU and Memory are utilized at a much higher levels.
Why is this node not taking more resources? Why is this node taking so long to execute? What can I do to improve performance of this node?
I am trying to convert a data set that we are bringing in from AWS DynamoDB and converting it into a specific table format before we can load it into our database.
I know and understand that we can always change the approach…but my question is why is the node not using system resources…am I missing anything?