I’m trying to set up a scheduled workflow that gathers production data from multiple machines using the Rest Node.
However, some of these entries are changed retrospectively, so instead of just appending data for the current day, I would like to overwrite the last week or so in addition.
I’ve already created a workflow that takes the current date, shifts it back, loops over all the machines etc. so that I get data of the last 7 days, each day.
In addition, I have a huge table with all the data of the last 3 years that I now would like to expand on a daily base.
My idea was to import the large database using the csv reader, use something to append & overwrite, export it again as csv. and my desired output format and repeat this step each day.
However, I’m still trying to figure out how to overwrite parts of my dataset without losing relevant data.
Some entries from earlier runs might be filtered out in in the new dataset, so I’m afraid I cannot simply group by e.g. process ID and then use “last” for every entry.
Anyone with a solution?