Hi. This is a simple question and I’m relatively new to KNIME.
I have a large CSV file (10GB). I would like to save it in native format to speed up working with it in future/other workflows. I understand you can use the table writer node.
However, I do not have a “file” to select to write this table to in this node. Do I need to create an empty table with the appropriate rows and columns and write it to this? Can KNIME not be used to just create this file itself?
What is the simplest way to convert this data to KNIME NATIVE format and save to disk? I find it faster to work this way than load CSV each time.
Thanks
Hey there and welcome to the forum!
As you mentioned you can use the Table Writer node
In the configuration dialogue just select e.g. local file system and then hit the Browse Button to select a folder and the define the file name and add „.table“… e.g. myfile.table
When you execute the node it will create that file at the location.
You can then use a Table Reader node to read it by selecting the same file and location
In my experience .table format can be more performant, but with that file size there for sure will be quite some latency to get it loaded.
3 Likes
@will74 another option if you are dealing with very large files is to use the parquet format which can split data into several files but treating them as one.
https://medium.com/p/c57795b65d7e#bdcd
1 Like