I’m creating a dedupe mechanism where I’m pulling in recent data from a source, using “last 7 days” and joining to a ‘historical’ .TABLE file stored locally. I’m joining on a couple of fields, including date. I want to keep the most recent in the event values have been updated in the last week. In the joiner columns, I"m concatenating the left, right, and joined fields - filtering out the ‘historical’ last 7 days and keeping the most recent, then writing to a table writer, which becomes the next day’s "master input table’. The problem is that once the reader node is executed, the data is cached all the way through the workflow. So the question is, how do I configure the ‘master table’ to run every time i run the workflow? I appreciate that the data is cached for quick processing, but I need the file to read the new data each time without having to manually go into each reader and resetting them. I’m using .csv, Excel, and .TABLE files and have a handful that need to run. This is what the workflow looks like: [image] Thank you.

How to get any Reader node to automatically run when running downstream node

mlauber71 October 10, 2023, 6:21pm 5

@ebarr well, there is this node which is available from Selenium

Another approach could involve using a local SQL database like H2 and always pull the latest entry (per ID?) from the database:

1 Like