I have a weekly CSV filing reading and concatenation project I’d like to automate.
On Monday I access the prior seven days of sales data on AWS S3 using the appropriate KNIME nodes to download the folders/data to a target local location.
Say a CSV file type is called ‘Orders’. Each day has a separate file in a folder labeled with that day’s date (yyyymmdd = folder name).
What is the basic structure using loops and/or file type list reading to have KNIME CSV import each ‘Order’ CSV from each of the seven folders for concatenation?
Images for context:
Files listed are the same in each dated folder:
Thanks in advance!
You could combine two lists. One with the (last three) directories and another with the relevant Excel files and then loop over them. You could change the conditions to your needs.
The files and folders would have to follow the same pattern every time.
kn_example_loop_folder_import.knar (2.1 MB)
Thanks @mlauber71 I’ll try this out and let you know how it goes!
I have come up with a solution that might be handy. If I got everything right
So I have created a week definition data. Each data is assigned a week number. It is created in external File and read into KNIME.
Based on your input week number Row Filter node only provides 7 rows i.e. 7 folder names for that week. With String Manipulation node you create folder path for each folder and then you loop over it and read it.
Data is attached with workflow and all paths are relative so you should be able to run it with no modifications. Hopefully
2019_01_18_Automating_File_Reading.knwf (79.2 KB)
Have a nice day,
@ipazin This didn’t quite work for the problem at hand. I like your approach of creating a week number table to use, but @mlauber71 provided a solution that can be automated a bit more.
Thanks @mlauber71 this workflow solved the issue with slight modification.
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.