I am trying to find a solution to reading in huge csv files (60mil+ rows) fast. Is there a node or custom coding option? As of now, I have tried to following:
- Base Knime Nodes (CSV/File Reader)
- Creating Custom Knime Node (and keeping all data in memory - not writing to disk)
- R Snippet (Data.table to knime.out)
- Increasing the ram in the Knime ini file
The best results I have had is the creation of a Knime node. This is usually faster than the other methods, though I do have some slowdown converting from a List<String> to Knime data container (basically looping through rows & cell values to populate the data table).
Are there other solutions to reading in large csv files?