How to read all files in a folder with a loop

I am trying to find an elegant way to loop over a series of files contained in a folder. I tried implementing community examples (https://www.knime.com/nodeguide/control-structures/loops/example-for-reading-a-list-of-files) and a few variants, but I keep getting an error that says: Errors loading flow variables into node : Coding issue: Cannot create URL of data file from \my_folder.

I created a flow variable in the List Files Node. My understanding is that this will create a list My_list, pass it onto the file reader (which I set to use My_list as the variable). The node throws an error telling me that the “DataUrl” parameter is controlled by a variable. Isn’t this exactly what I am trying to/supposed to do?

In the file location browser it says that the input location is a directory, not the list of files I want to loop over. How can i fix this?

47%20AM

03%20AM

Hi Emilio,

The Table Row To Variable Loop Start node will automatically create a variable out of the list passed on by the List Files node, so no need to create the My_list variable. Simply select the folder in the List Files node that you want to scan, add any filters and your loop will go through all of those outputs in the File Reader. Just choose for example the URL column in the File Reader node as the location of the files you want to read.

Let me know if this helps!

Cheers,
Medzi

4 Likes

Hi,

I am using Table to row variable loop start node and it works fine. But how if the folder contains many different file types and you only want to read .txt files?

Thanks,
Amiza

Hi Amiza,

Simply treat the list of files as any other record type and use a Row Filter to select only the .txt files for further processing before the Table Row To Variable Loop Start node.

Cheers,
mM

2 Likes

The list file node can select only specific extensions.

https://hub.knime.com/knime/extensions/org.knime.features.base/latest/org.knime.base.node.io.listfiles2.ListFilesNodeFactory

To get an idea about a lot of possible ways to import files this example workflow might be useful
https://hub.knime.com/tardispilot/spaces/Public/latest/Importing%20Multiple%20Files%20(ETL)

2 Likes

Hi @amizaamir

You don’t need a Row Filter. You can manage the files to select directly in the List Files node. If it’s not in the dropdow list, just type it in.
Knipsel
gr. Hans

4 Likes

Thanks @HansS @mlauber71 @mmedzihradszky for the reply. I am a beginner of KNIME and your answers are very helpfuI.

2 Likes

Can you help with the same case, I have listed the files but now I need knime to read a file according to data in cells of another excel sheet file

So when the cell in that sheet shows 9-Nov-2020 so the node goes and read file 9-Nov-2020 and so on

See here for @randadarwish case: Variables in Reading excel files