Delete Files with zero size before reading

Hello friends. :laughing:
I would like to share an idea for a flow I created, but I feel like I took too many detours and it could have been better.

The idea is simple:
Read several .csv files from a folder.

But there is a catch:

  1. Some CSV files need to be deleted from the folder before they are read. These are the files with a size of zero.
  2. The CSV files do not have headers; therefore, when a file has “zero” size, the CSV Reader node triggers an error because it cannot build the Data Type structure.
  3. After some files have been read, the flow must also work in cases where no files are found to be deleted.

What is the ultimate goal?
To simplify my flow.

But why?
Because I think I took too many detours. When the folder contains files with greater than zero size, the flow works; however, when it does contain files with size of “zero”, the flow fails due to the Row Filter nodes I used. In other words, no zero-size files can reach the CSV node because those files has no header and are empty.

One idea led to another, and, with each attempt, I kept changing the flow. It took me about 3.5 hours to reach the final model to not breaks.

But as I said, there were so many detours that I became intrigued and would like a more professional perspective.

I am attaching the final flow I completed, but I would like to know if there is a simpler approach.

csv.Help_01.knwf (958.4 KB)

How about this one?
The only thing different to your solution is that I move the valid files to a temporary folder.

2 Likes

@ActionAndi without knowing the file sizes we are talking about and potential storage locations, I wouldnt recommend moving files (which can incur egress costs) to a user.

@Felipereis50 the CSV reader should be ok ignoring the blank csv files if you uncheck this option:

but, i would recommend to e.g. add a column filter or table validator afterwards to ensure you are not getting new columns, and if you want to cover the case of ALL files being blank, then insert a table creater with the expected columns and concat it into the flow.

alternatively, deleting the blank ones and then loading (if at least 1 file is left) is the reasonable choice.