Writing large csv files + memory issues

[mlauber71]

Could you try and write the data to CSV in loops? Only so much at a time and then append it to an existing CSV file. We had a similar discussion recently:

Export huge MS sql table to csv without fetching all records together

have you thought about doing it in chunks? And then appending the result to a CSV file. The example attached uses ROWIDs from SQLite. You could see if you either have a ROWIDs present or you could create one yourself. You can choose the size of the chunk/batch yourself (I use integers you might have to see if with very large numbers you might need big integers or something. Even a workaround with using strings might be possible). Only other idea would be to write a CSV file directly on a Big Da…