Read multiple files from filesystem in parallel and execute a workflow

KNIME Analytics Platform
1

Hi

I have a directory basically the knime anamoly detection example hosting 16 files.

I want KNIME to read all the 8 files in parallel and then call a workflow to execute certain tasks on it.

After the 8 files are fnished the next 8 should be picked up.The Server is having 8 CPU's and thats the reason i want to pickup 8 files at a time can KNIME do this ?

 

regards

Hrishy

 

 

2

Hi Hrishy,

in this case you need the paralel chunk loops. They are in the Virtual Loop Extension on Labs.

You would actually need to loops. The outer one is the parallel chunks and the inner one a basic table row to variable loop. The outer one will generate the processes and the inner one takes its chunk of data and read (or process its one by one)

Best, Iris

3

Hello Iris, could you point me towards an example?

4

Maybe you take a look at this example:

2 Likes
5

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.