Hello
How one can process, read a huge EXCEL file (size about 3.5 GB). Also then if we succeed I need to perfom transpose on the table that was generated. It will take a very long long time.
Malik
Lots of data = long time. Nothing you can do.
Maybe increase RAM for knime in knime.ini and make sure knime workspace is on an ssd.
Mabye try export excel to csv and load the csv (if it isn’t actually already a csv).
Source of the file? if it’s a database extract best to get the data out of database in desired format.
2 Likes
For a lot of operations, the new Streaming execution will help. If you need to do any pre-processing in KNIME before the pivot, check if any of the nodes you use are streamable. The Excel Sheet Reader is. (You can find out by clicking the little dropdown arrow next to the search box in the node repository, and selecting ‘Show additional information’ - nodes which are streamable will then say ‘[Streamable]’ after them).
The transpose node is not streamable, and I dont think ever can be unfortunately. You could try wrapping the transpose node in a Chunk Loop Start (and set the chunk size to e.g. 5000 rows) -> Transpose -> Loop End (Column Append) sequence, but I dont think that is going to do anything that the node itself doesnt do. Unfortunately, as @beginner says, you have a lot of data and a complicated operation, it will take time.
Steve
3 Likes
This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.