Large files split to excel / gsheet requirements

[MichaelB]

newish to using knime for some basic data splitting coming from powerquery.

I can’t seem to find / understand how to split a file that exceeds gsheets data limits.
Currently have a profiles.txt that has 3.5m rows of data. I’d like to find a way to break that out into 4 files at 1m each?

Ultimate goal is that this data may only need to be refreshed once a quarter, and I plan on using it with Google DataStudio/Looker to be a reference set. Currently Google Looker with corporate restrictions only allows me to use gsheets so I need to break up the data source into 4 files to meet the requirements.

TLDR;
How do you break a file (.txt) into multiple CSV’s and keep the output under 1M rows for each file?

[ArjenEX]

Hi @MichaelB

Welcome to the KNIME Community!

I can recommend having a look around the forum, this topic is frequently returning. Some examples that should help you out:

4 files, 12k records, divided by 500 rows each files = 24 files

Hi there as i mentioned in the title, i tried to split 4 files with 12k rows in total (file 1= 3000 rows; file 2= 4000 rows; file 3 = 2000 rows; file 4= 3000 rows) in multiple files with only 500 record each. I merged all in with a csv reader and then i split with row filter limiting by 500 rows, then i put 24 excel reader. But i would ask u if there s a faster way, maybe with a loop but i can t find it. Setting 24 row filters it s a bit long, anyone know a simplier and faster way? THANKS SO…

Row splitting tables into several tables at once

Hi guys, I made a simple solution with loops and files path to make it. For the example, I used the Chunck Loop start, because it can split it in fixed numbers of rows or parts/groups, make it easier to manipulate as you wish. Inside the loop session, I just build a string for the path with a counter, manipulating the information by the Interactive end loop session. and before the end loop node, I insert a write csv node to export the data for a file. file_split.knwf (363.8 KB) [image] [i…

Write multiple CSV file to a loop

Hi all, My goal is to write multiple csv files via a loop, with the titles of the rows as the filenames of the csv. So title of csv file 1 = B, csv 2 = B etc. (inspired by Write multiple files. ) I’ve tried to adjust multiple solutions from this forum but couldn’t get it to work as desired. Does anyone has experience with this and can show me what i need to adjust? Attached is my attempt, where i only get 1 file and not the desired 8 with the correct names. csv_loop_test.knwf.knwf (17.8 KB) T…

How to split data into multiple csv files?

Hi, I’m new to KNIME and I want to split data into multiple csv files. I hope someone can help. I have a dataset of 20,000 rows (could be 30,000 next week). I want to split this dataset into multiple csv files of up to 2500 rows. The dataset contains information about multiple customers. It is important that a customer’s data stays together in one csv file. A csv file will contain multiple customers. Thanks in advance!

Common ground is a chunk loop for which its size is set to your desired output maximum per file.