upload of 10,000 documents

KNIME Extensions
1

Hello, I will appriciate your help with the issue of uploading multiple files, which are simple text files, into the Knime, and to also **convert them into csv files. **
My problem is the huge amount of written files- hoew to manage uploading all of then to one workflow? (more than 10,000 documents)?
Thank you!

2

Supposing the files are all in the same directory and have the same CSV format, you might want to

  • use the “List Files” Node,
  • then start a loop with “Table Row to Variable Loop Start”,
  • connect the flow variable output of it to a “File Reader” (after enabling Show Flow Variable Ports on File Reader node),
  • use variable “URL” for data file location (click on “V” button next to “Browse…”),
  • then collect the results in a “Loop End” node.

Make sure to configure the File Reader node to correctly read the CSV file (column separator, quote character).

2 Likes
3

@k-wizzz solution will work if you just want to concatenate the 10,000 files into a single .csv file. I read your post as needing to convert each individual .txt file into a .csv file. I have attached an example workflow for you.

Read_Save_files.knar (19.6 KB)

2 Likes