Automate concatenation of same CSV file type read from different folders

I have a weekly CSV filing reading and concatenation project I’d like to automate.

On Monday I access the prior seven days of sales data on AWS S3 using the appropriate KNIME nodes to download the folders/data to a target local location.

Say a CSV file type is called ‘Orders’. Each day has a separate file in a folder labeled with that day’s date (yyyymmdd = folder name).

What is the basic structure using loops and/or file type list reading to have KNIME CSV import each ‘Order’ CSV from each of the seven folders for concatenation?

Images for context:

image

Files listed are the same in each dated folder:

image

Thanks in advance!

You could combine two lists. One with the (last three) directories and another with the relevant Excel files and then loop over them. You could change the conditions to your needs.

The files and folders would have to follow the same pattern every time.

kn_example_loop_folder_import.knar (2.1 MB)

3 Likes

Thanks @mlauber71 I’ll try this out and let you know how it goes!

Hi there!

I have come up with a solution that might be handy. If I got everything right :slight_smile:

So I have created a week definition data. Each data is assigned a week number. It is created in external File and read into KNIME.

WeekDef

Based on your input week number Row Filter node only provides 7 rows i.e. 7 folder names for that week. With String Manipulation node you create folder path for each folder and then you loop over it and read it.

Data is attached with workflow and all paths are relative so you should be able to run it with no modifications. Hopefully :slight_smile:
2019_01_18_Automating_File_Reading.knwf (79.2 KB)

Have a nice day,
Ivan

2 Likes

@ipazin This didn’t quite work for the problem at hand. I like your approach of creating a week number table to use, but @mlauber71 provided a solution that can be automated a bit more.

Thank you!

1 Like

Thanks @mlauber71 this workflow solved the issue with slight modification.

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.