Moving files through a loop

KNIME Extensions
, ,
1

Hi friends, I have a lot of Excel files in a folder and I want to import them into SQL through knime! The problem I have is that when the number of files is large, the system hangs or the temp folder fills up and the workflow is canceled and …
I have 1000 Excel files and I want to design a workflow that takes files 1 to 100 from folder 1 and moves them to folder 2, and after performing an operation on them, it returns and takes files 101 to 200 and performs the operation on them again, and this is done until the last file
If anyone can help me with this, I would be happy.

2

@alex1368 you can list all the files. Maybe identify the sub folders and then loop thru the list of files in chunks.

1 Like
3

Hi,
why do you want to combine the data transfer to a SQL database and moving the excel files to a different folder? In the past I did similar things but in a sequential way:

  • Extract data from Excel to SQL using KNIME file by file (like @mlauber71 described)
  • Move files to a different folder in file browser afterwards.
1 Like
4

Most likely you can build a workflow like this:

  1. “List Files/Folders” node with "“Files in folder” option
  2. If necessary, create one per folder and concatenate the output with the “Concatenate” node
  3. Put the result in a “Chunk Loop Start” with chunk size = 1 (default)
  4. Do everything you want to do within this loop: Reading out the Excel, write it to the database, move the file (“Transfer File” or “Transfer File (Table)” nodes)
  5. “Loop End” node
    Test it with “Step loop execution” on the “Loop End”