CSV Writer in Loop writes data file Over and Over Again

Hi everyone,
I have a loop that filters a file by Business Unit (variable in file) and saves it as a CSV. However, the loop continues to overwrite the csv file again and again until it finally moves onto the next Business Unit in the file. This makes the workflow take much longer than it should.
Here are my work flows:

And specifically it’s the last loop that I’m having trouble with… As you can see, it outputs the file again and again and again:

Could anyone provide any advice as to how to modify my loop so that this stops happening? I’m still new to loops and am struggling a little bit.

Thanks for your help!
-Snowy

Hi,

Did you edit the settings for output location in “CSV Writer” node to be read from the “filePath” variable created by “Create File Name” node earlier?
And if yes, Are you sure the file names created in loops are different from each other?

I think your problem is because of similar file names in all loops.

If this couldn’t help you to solve the issue, please export and share the workflow so that I can check it better.

Best,
Armin

1 Like

Hi Armin,
Thanks for your quick reply. Yes, I do have the CSV Writer set to create the file names based off the variable set in the previous node, as you can see here:


Also, the names are different from each other. Eventually they all save here:

I have attached my workflow for you to look at. Thanks again for your help!

Data Files.knwf (41.0 KB)

-Snowy

Would you please execute all nodes and then uncheck the “Reset workflow before export” when exporting the workflow to provide executed workflow containing the input data for nodes? (if it’s not an issue to share your data)

Will do. I have it executing now. It will take approx. 1 hour to complete the workflow, but will respond once it’s finished.

Meanwhile, please check your “Create File Name” node configuration. It seems you are generating the same file name in every loop so that the files get overwritten.

And it is not necessary to input all your data, just a sample of your data is enough.

This is how I have the Create File Name node set up:

It seems as though the loop goes down each row of the input file to write the output. currentIteration and RowID increase by one until it goes through all rows for a specific BU:

Maybe the Row Filter node is out of place?

If I’m getting your question correctly, you have a file containing records of data for several BU.
You want to have separate files for each BU.
Right?

That’s correct; I have a single file and I’m using the bottom workflow in the initial screenshot to resort the file, then create separate files for each BU.

Would you please just provide a sample data for 2 or 3 BUs in a Excel file?

Attached is a cut down version of the file the last workflow is using.
combined-sorted-small.xlsx (117.4 KB)

Also, to be clear, the workflow does do exactly what I want, it just takes a really long time because it rewrites the file over and over again before finally moving on.

Based on your sample data and 3rd part of your workflow:

I added a “GroupBy” node before the “Table Row to Variable Loop Start” node to make one loop for each BU not for each record of data (That’s why your workflow takes too much time to execute).
Then I filtered the data based on each BU in current iteration and created the name for the file and closed the loop.
Check the attached workflow and let me know if I’ve missed something.


Data-Files.knwf (106.7 KB)

4 Likes

Armin,
Thank you so much for your quick help. Your attached example helped me greatly!

-Snowy

1 Like

Are you sure the file is overwritten many times? I had a similar issue in other workflow but it was just the warning that was repeated multiple times (the warning is generated every time Knime verify if the block is executable not every time is executed)

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.