Workflow that reads the last file according to the date and reads all the resulting files with a loop

[IvaR]

Hi Knimers,

I’m working on a workflow, my idea is to read all the last files, I have they in folders according to month, my idea is with a “Rank” obtain the last files of the last month, already achieved it, what I cannot do is create a loop to read all the files I just filtered

As you can see the loop doesn’t work, what I’m doing wrong?

Really appreciate any help

[IvaR]

The first attempt was to do something like this to read all the files, but in the main categories sometimes one of the files is empty, that means that the streams have to be adjusted manually, which is not the idea, plus it looks horrible

[bruno29a]

Hi @IvaR , it’s hard to tell what the issue if we’re just looking at your screenshots. It would be more helpful if you could at least give us the error message, or some some of the nodes’ configurations.

For instance, I am assuming that you are attempting to open CSV files, and that your paths are correctly defined.

Once you have your filtered paths, you just need to pass them to the CSV reader.

I put something together that you can adapt/integrate to your workflow that look like this:

Basically I have 3 folders, 202106, 202107, 202108, and in each of the 3 folders, I have 2 csv files, file 1.csv and file 2.csv. All 6 files (2 files x 3 folders) are the same test files.

And we can see that the List Files/Folders node got all 6 files:

I do a Row Filter to keep only the paths with file 2. to simulate the fact that you want to keep the latest file (since you have this part covered, I’m just simulating a Row Filter here):

As far as I understand, you have reached this part.

After that, it’s just a matter of looping over each of these with the Table Row To Variable Loop Start, which in my case will generate a variable called Path which will contain the path to the current file in the current loop iteration.

Then just use that variable in the CSV Reader:

And that’s it.

Here’s the workflow:
Reading Multiple CSV Files.knwf (15.8 KB)

[IvaR]

Thank you so much for your answer!

Even with the little information that I gave you, you understood perfectly what I wanted to said

I removed the additional node and tried to replicate the same thing you did, however the CSV Reader is not allowing me to apply a variable

In my case, this is the result of the Row Filter node:

I would need to use the URI, but I don’t know why the CSV won’t let me apply the variable

Thanks in advance for your help, and for the workflow

[Daniel_Weikert]

The csv reader needs a path variable not an URI or string

[bruno29a]

Hi @IvaR , no problem.

For your issue with the CSV Reader, if you check my workflow, I’m passing a variable of type Path. In your case, it looks like you have URI instead.

You can convert the URL to Path using String to Path:

Choose what File system you are using (it looks like it’s Local File System from your screenshot)
Choose the column you want to convert (in your case the column name would be URI)
And then choose what the name of the column should be if it’s a new column.

You will then get a usable variable in the loop.

[IvaR]

Thanks @bruno29a and @Daniel_Weikert for your help,

I added the String to Path.
The information that I’m reading is in the Shared Folder of the company (kind of cloud), I put it by Mountpoint for that reason, and if I executed the node of the String to Path this one works, but the node of the CSV doesn’t execute, the same for when I change it to local file

MOUNTPOINT:

LOCAL FILES:

Both cases present same error

Thanks in advance for your help

[Daniel_Weikert]

Your path (sth. like C:\…) will only work with local file system selected
br

[bruno29a]

Hi @IvaR , the MountPoint in Knime has nothing to do with mounted/mapped drives/partitions in your system. The MountPoint refers to one of the items in your KNIME Explorer:

Any mounted/mapped drives/partitions on your system, as far as Knime knows, sees them just as a drive/partition on your local system, therefore would access it as a Local File System.

From your screenshot, I saw your URI as “URL: file:/...” which seems to be pointing to a file on the Local File System, which is why in my post, I mentioned “it looks like it’s Local File System from your screenshot” and which is why I chose Local File System in the instructions.

Can you expand that URI column from this table?

[IvaR]

hi @bruno29a, thanks a lot for your answer, and sure:

[bruno29a]

Thanks @IvaR , but it does not look like it’s the same column. The other one was of type URI, starting with URI: file:/

Nevertheless, as you can see, you are referencing a mounted drive mapped to the Z: drive, so this is considered as Local File System.