Hi all
I am trying to join 16 csv files with a loop with joiner node. However, afther 30% the joiner node gets extremely slow. Any ideas how to solve this, or suggestions for a different workflow to join the files. See the workflow below.
regards
Bas
I guess we need to understand what your data looks like, and what the Join operation is doing. Maybe you could post some sample data with a bit more detail?
(Also, is it possible that what you really need to be doing here is just a simple concatenate instead?)
Hi there @Bas1111,
from your screen shot you are feeding Joiner with same file. If you need to join (not concatenate!) 16 .csv files you would need to use 2 loops with some additional logic or simply cascade multiple CSV Reader and Joiner nodes.
Br,
Ivan
Hi Ipazin,
Thanks for the respons. I (think) I need to join, not concanate, becasue the data is not always in the same order and needs to be joined based on a unique identifier that is in all the datasets.
I would prefer to do it with a loop, the number of files will grow over time.
Could you point me in the right direction how to set up the workflow with two loops and additional logic as you suggested.
I can share some of the files if that helps (it is public data)
kind regards
Bas
Hi there @Bas1111,
actually now when I tried to create an example got a bit stuck 
Please share couple of files (3,4) to check do you really need to join them. Also maybe someone else gets an idea how to do that dynamically…
Br,
Ivan
Hi there @Bas1111,
What should be your joining criteria (unique identifier)? I have checked files and it seems there are no Item number in these 3 files so not sure what would you join. My guess is you need to concatenate…
Br,
Ivan
Hi Ivan
I will try with a concatenate loop then to see if that gives me the correct result.
Thanks for your advice.
regards
Bas
This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.