@tttpham maybe you can check out how to handle large CSV or parquet files in chunks. They will later then be one big file again (or at least work as one).
Check the parts “Collect results step by step” / “Several Parquet files into one”:
Some file systems have limitations how large a single file can be. Question is what kind of system your data will be stored on.