How to do UNION with several csv files

Hi Knimers!

Playing with DCM data: (https://developers.google.com/doubleclick-advertisers/dtv2/reference/file-format)

This means that I have one CSV file per date and hour and this also means that if I desire to run some analysis of a display campaign in a specific month for instance, I should first make an UNION of those several files.

In my test, I'm using just files of the 25/01/2017 per hour and this means that I have 24 files and A LOT OF ROWS.

What is the easy way to make a UNION of all these files?

I found CONCATENATE node but it works just for 2 files per time and I would have to CONCATENATE 24 files just for one day and would be hard to do the same for several  months...

Best

Hi,

you need to loop over the files and compine them using a loop end.

Here is a nice example which demonstrates how this can be done.  https://www.knime.org/nodeguide/control-structures/loops/example-for-reading-a-list-of-files

Best, Iris

1 Like

HI Iris!

Wow! I was impressed! I didn't use the METANODE with the CREATE TEMP DIR and the UNZIP FILES. Why should I create a Temp file?
In my case I'm just using LIST FILES and the LOOPS

Thank you!

Actually, I have another issue.

I created a workflow without using a flow variable or determining the row prefix as in your example and worked.

I justed used this sequence of nodes LIST FILES>LOOP START>FILE READER>LOOP END,  but what I noticed is that using your workflow I have more rows than using my workflow and I still dont know why.

Can you give me a hand here?

Best 

Hi Fabio,

the meta node we only need to have an example which is usable out of the box for everyone.

About your question, did you set the flow variable in the file reader?

Best, Iris

 

Hi Iris

In my case no, I didn't set the flow variable.

Sorry but I still do get the difference.

Best