Thanks for all the links. They were very useful to learn new things but unfortunately, they didn’t match my use case (I didn’t try R or Java nodes as I don’t know those languages).
If the file I’m expecting has new columns, the “File Reader” fails but is not caught by the Try/Catch and that is extremely FRUSTRATING. Maybe this feature should be taken into account in the future.
My workaround came from this answer which suggested to read the csv without a delimiter (none), and then use the “Cell Splitter” to generate all the data. This had an extra advantage (depending on how you see it) to cast all the data as strings.
There was an issue with this approach:
I change the name of the file dynamically to point to today’s folder through a flow variable. When there is no file, I point it to a file with the same columns and just one row of dummy values. Then, some nodes do the cleaning (filtering, reordering, renaming, etc). The problem shows when the file is not there, eventhough I connected to the dummy file which has the same format. Some nodes showed this error and it didn’t let the flow continue:
“Column range filter: Input table doesn’t contain specified column name”
(this happened on datasets that had a column named “Date” which was recognized as a different column after reconnecting. Bug?)
After A LOT of trial and error, I managed to get past it by adding a “Domain Calculator” and a “Table Validator” node after getting the data.
Sorry for the long text, I hope this serves someone if they find a similar issue!