reading multiple files

KNIME Analytics Platform
1

Hello,

I am new to KNIME and I must say it is extremely useful.

I would appreciate any help for a problem I am facing. I have multiple SD files I would like to read in. Instead of using multiple "SDF Reader" nodes, I thought of using "List Files" and then "Iterate list of file" nodes. When I run the "List Files" node, I get a table with location and file names. However, for the "Iterate list of files" node, I get the warning message that there are no settings available. I have no idea what is wrong!

Thanks in advance,

Preeti

2

I dont believe the "Iterate list of files" metanode will work for SDFs.

Use the "List Files" node, then use a "Table Row to Variable Loop Start" node, then add a SDF Reader node. Right click on this and choose "Show Flow Variables", now connect a red line between the red blob on the "TableRow to Variable" node and the left hand red blob on the "SDF Reader" node.

In the config of the "SDF Reader" node, select any filename in the fileselection box, dont worry what it is, it will be replaced. Now goto the Flow Variables tab, expand the URL section, and next to "0", from the dropdown choose "Location".

Now add a "Loop End" node, and this should work for you.

Simon.

1 Like
3

Hi Preeti,

If you don't mind entering the sd files manually you can use the SD Reader with multiple files. I think it started in knime 2.4. After you entered the first filename a file selector box should appear where you can enter the next file and so on.

If you need to run the workflow on different lists of files Simon's solution is perfect.

Regards

Guido

4

Hello,

thanks a lot for the quick response. I tried Simon's solution and it worked like a charm.

Best wishes,

Preeti

5

i did, roughly, something like that. (this is actually my first "usefull" KNime projects, i discovered KNime yesterday).

 

here is a screenshot showing how to loop over a list of file :

http://fractals.s3.amazonaws.com/vrac/csvloop.jpg

I hope it's the right way to do it, isn't it ?

 

EDIT : nope, it's not.

I have only the last file in my database :(

6

Assuming you have set the filename in the CSV reader from the Flow Variables Control tab and NOT just entered one of the filenames in the CSV Reader, this ought to work.

In the flow control tab, select the variable from the dropdown next to URL.

In fact with this node,you can do it more easily by clicking on the little icon next to browse with the red blob and slider, choose the variable from here.

The List Files node should point to the directory of where the files are, and this directory should only contain the files you wish to import.

Simon.

7

Hello, I have used Simon's example suggestion above to send File List ---> Variable Loop Start and then used Flow connection (RED) to File Reader and then selected LOCATION from Flow Varable drop down.  Here is my issue:  In the console window I am getting:

"WARN      File Reader                        Errors loading flow variables into node : Loading model settings failed, caught "IllegalArgumentException": Cannot create URL of data file from 'C:\Users\Parker\Desktop\TOPS\SP Files\SP Files\Test-Load\141006 - 141010.csv' in filereader config"

C:\Users\Parker\Desktop\TOPS\SP Files\SP Files\Test-Load\141006 - 141010.csv is the first file in the list.

Any idea why it doesn't like this and is throwing an exception?

Jeff