Overwrite parts of data in scheduled Workflow

[peppapiglet]

Hi all,

I’m trying to set up a scheduled workflow that gathers production data from multiple machines using the Rest Node.

However, some of these entries are changed retrospectively, so instead of just appending data for the current day, I would like to overwrite the last week or so in addition.

I’ve already created a workflow that takes the current date, shifts it back, loops over all the machines etc. so that I get data of the last 7 days, each day.

In addition, I have a huge table with all the data of the last 3 years that I now would like to expand on a daily base.

My idea was to import the large database using the csv reader, use something to append & overwrite, export it again as csv. and my desired output format and repeat this step each day.

However, I’m still trying to figure out how to overwrite parts of my dataset without losing relevant data.
Some entries from earlier runs might be filtered out in in the new dataset, so I’m afraid I cannot simply group by e.g. process ID and then use “last” for every entry.

Anyone with a solution?

Cheers

[ipazin]

Hello @peppapiglet ,

and welcome to KNIME Community!

Considering both your data volume and needed operations I would suggest to use database instead of CSV file. Then whole process should be faster, simpler to design and easier to troubleshot&maintain (with appropriate error handling included in design). And perhaps a file-based DB is sufficient (SQLite, H2) so you don’t need to play around with PostgreSQL.

Here is link to the Database Guide:
https://docs.knime.com/latest/db_extension_guide/index.html

You can find workflow examples on KNIME Hub and search this forum on topics regarding database operations.

Br,
Ivan