Read, process and write multiple files

KNIME Analytics Platform
1

Hi,

I am very new to KNIME. As i've seen, several questions about reading multiple files had been asked, but I couldn't find any about processing and writing separate files from those read.

I have 33 csv files about 1.1 Gb each. I need to filter them and then write a filtered csv for every input file into the other directory. I don't understand if it's possible. There is a "reading multiple files" topic (https://tech.knime.org/forum/knime-users/reading-multiple-files) where sdf files are discussed. On the stage of reading files, for csv-tables there will be no difference from sdf, I suppose, but i can't see how separate files could be written (I need not to overwrite existing input files, but write new ones in new directory).
I guess, what I need is just "csv writer" node inside the loop, but what variable has to be used to define new file name and location? Any help/corrections are appreciated.

Thanks in advance,

 

Lana.

2

Hi Lana,

you can definitely use a CSV Writer node inside your loop and dynamically set the file name at each iteration with a flow variable linked to its filename parameter.

Since the filename is just a string, you can create a table holding the constant part of the URI (for example the directory), then use the String Manipulation node to add an extra part to it at each iteration, finally use the Table Row to Variable node to pass it to the CSV Writer node, linking it to the filename parameter in the Flow Variables tab.

Give it a try and post again here should you encounter an issue along the way.

Cheers,
Marco.

 

3

Hi Marco,

your advice was extremely useful.

Not to say I managed to create immediately a functioning workflow that I needed, but after several fails it finally worked as I wanted. The last version of algorithm looks exactly as you explained. All issues I faced were caused only by lack of practice, so it's not worth to mention them, I guess.

Thank you very much,

 

Lana.

4

Hi Lana,

Glad you made it work. This is how we all learned KNIME, by trying things out, failing and finally finding a solution.

Feel free to post here again in case you need additional help.

Cheers,
Marco.