newish to using knime for some basic data splitting coming from powerquery.
I can’t seem to find / understand how to split a file that exceeds gsheets data limits.
Currently have a profiles.txt that has 3.5m rows of data. I’d like to find a way to break that out into 4 files at 1m each?
Ultimate goal is that this data may only need to be refreshed once a quarter, and I plan on using it with Google DataStudio/Looker to be a reference set. Currently Google Looker with corporate restrictions only allows me to use gsheets so I need to break up the data source into 4 files to meet the requirements.
TLDR;
How do you break a file (.txt) into multiple CSV’s and keep the output under 1M rows for each file?
Hi @MichaelB
Welcome to the KNIME Community!
I can recommend having a look around the forum, this topic is frequently returning. Some examples that should help you out:
Hi there
as i mentioned in the title, i tried to split 4 files with 12k rows in total (file 1= 3000 rows; file 2= 4000 rows; file 3 = 2000 rows; file 4= 3000 rows) in multiple files with only 500 record each.
I merged all in with a csv reader and then i split with row filter limiting by 500 rows, then i put 24 excel reader. But i would ask u if there s a faster way, maybe with a loop but i can t find it.
Setting 24 row filters it s a bit long, anyone know a simplier and faster way?
THANKS SO…
Hi guys, I made a simple solution with loops and files path to make it.
For the example, I used the Chunck Loop start, because it can split it in fixed numbers of rows or parts/groups, make it easier to manipulate as you wish.
Inside the loop session, I just build a string for the path with a counter, manipulating the information by the Interactive end loop session. and before the end loop node, I insert a write csv node to export the data for a file.
file_split.knwf (363.8 KB)
[image]
[i…
Hi all,
My goal is to write multiple csv files via a loop, with the titles of the rows as the filenames of the csv. So title of csv file 1 = B, csv 2 = B etc. (inspired by Write multiple files. )
I’ve tried to adjust multiple solutions from this forum but couldn’t get it to work as desired. Does anyone has experience with this and can show me what i need to adjust? Attached is my attempt, where i only get 1 file and not the desired 8 with the correct names.
csv_loop_test.knwf.knwf (17.8 KB)
T…
Hi,
I’m new to KNIME and I want to split data into multiple csv files. I hope someone can help.
I have a dataset of 20,000 rows (could be 30,000 next week). I want to split this dataset into multiple csv files of up to 2500 rows. The dataset contains information about multiple customers. It is important that a customer’s data stays together in one csv file. A csv file will contain multiple customers.
Thanks in advance!
Common ground is a chunk loop for which its size is set to your desired output maximum per file.
2 Likes
system
Closed
June 18, 2023, 4:00pm
3
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.