@mlauber71 Thanks for your answer and various exploration direction.
But saving in a file and reading it back is not working. Too many values for KNIME.
Indeed, I think that is a limitation of KNIME with the number of columns.
With a subset it work well and I needed to do a transposition in order to reduce the number of columns.
My dataset have 48 rows and 3’000’000 columns. That was the issue.
I tried to export the second dataframe column as a “list” but then KNIME is not managing it anymore.
My solution (sure I can find something better):
- Create the DataFrame and transpose it
- GroupBy to create a list (failed because too many values)
- Use only Python Nodes to do all the manipulation until the minimum number of parameters are reached.
So I found the limitation with KNIME to manipulate a huge amount of data.
I’ll have to use KNIME Python Node in order to keep going with that project. And use KNIME as a orchestrator and visualisation tool to show the process.
Thanks all for the support!