I have a workflow that starts with a “List Files” node that is taking a folder with over 1600 .xml files to create a table. When I started the workflow it was not a problem given that I had a few xml files.
As time went on, the number of files grew, thus making the workflow a lot slower to complete.
I was hoping that someone could share a way to only process the “new xml files” that are added so that the workflow could be a lot faster. I have no idea if what I am asking is possible but I decided to ask anyways.
It’s almost certainly possible, but how to do it depends on how you will know that they are ‘new’? Are you looking simply at a file creation date-based approach, or is there some other way of looking up with files are already processed (e.g. from a database, a list of output files etc)?
Hi @stevens_albert and welcome back to the KNIME community forum,
Regarding @mlauber71’s second suggestion, You can export the output of the List Files node in a file and read it each time you run the workflow. Then use the Reference Row Filter node to exclude those files which are already listed in the exported file.