Hi, I have like 40 csv files coming from 5 different systems with invoice formation. Every system has a different layout with different data in the csv (some data is similar, but the column names differ). The file names are consistent per system and can be used for selections. At this moment I have multiple csv reader nodes (one per system) pointing to a folder where are files are located and I use the filter function in the csv reader to process data per system. In the end all data concatenated. I chose this way, so that I can simply update the folder location instead of selecting files. I would like to have workflow whereby I only need to update the file location once instead of each csv reader node. Subsequently a filter on filename would process the data per system. Finally all should be consolidated. I there a way to handle this?

Similar to @mlauber71 his suggestion, have you considered a CASE Switch approach? Since it should be based on the filename, you can assign a port index value with your business rules in mind and then distribute them accordingly to the correct CSV Reader. To illustrate: [image] It works like this: List File node to get the Path’s of the files, in my example 0.csv and 1.csv. Both contain one row of data. [image] Define the system with a Rule Engine, Column Expression, Java Snippet, etc. In this example case I want to derive the port for the CASE Switch directly from the filename so I use replace(getFileName(column("Path")),getFileExtension(column("Path")),"") Note: the PortIndex of the CASE Switch is of type string. [image] Start a Variable Loop with both the filepath and the “system type” as input colums. Used a flow variable CASE Switch whereby the PortIndex is variable controlled by the “type” column. [image] Set-up as many CSV Readers to your liking for each system whereby the filepath is flow variable controlled (Path). Apply the required transformations for each system (which I disregarded here). [image] With the CASE Switch end plus Loop end, the contents of both files will be concatenated into one. [image] Hope this helps!

Multiple different files in different folders load

KNIME Analytics Platform

mlauber71 January 25, 2023, 9:44pm 6

@robvp maybe you can provide us with an example of what would constitute a system and what part you want to change. You can read various paths and set different flow variables in order to have a dynamic setting. Of course, you can extract paths and file names from the results and filter them later.