How to get any Reader node to automatically run when running downstream node

[ebarr]

I’m creating a dedupe mechanism where I’m pulling in recent data from a source, using “last 7 days” and joining to a ‘historical’ .TABLE file stored locally. I’m joining on a couple of fields, including date. I want to keep the most recent in the event values have been updated in the last week. In the joiner columns, I"m concatenating the left, right, and joined fields - filtering out the ‘historical’ last 7 days and keeping the most recent, then writing to a table writer, which becomes the next day’s "master input table’. The problem is that once the reader node is executed, the data is cached all the way through the workflow.

So the question is, how do I configure the ‘master table’ to run every time i run the workflow? I appreciate that the data is cached for quick processing, but I need the file to read the new data each time without having to manually go into each reader and resetting them. I’m using .csv, Excel, and .TABLE files and have a handful that need to run.

This is what the workflow looks like:

Thank you.

[ipazin]

Hello @ebarr,

don’t know how you start your workflow but connecting Google Authenticator node with Table Reader node using flow variable connection should do the trick.

Br,
Ivan

[ebarr]

For recent data I pull in through Authenticator, but to get history for past couple of years, I merge that flow into a .TABLE file to avoid sampling of data and only pull in recent data and blend with the history table. I’m the image I posted, the Authenticator is coming through the left join, the history table from the right. I’d like the .TABLE mode to reset every time I run workflow because I have upwards of 10 work streams that follow this pattern and I would prefer to not have to manually reset if possible.

[iCFO]

You can save the workflow in an unexecuted state and then “call” it from another workflow. This leaves it in a ready state. If a generic version can be customized for all 10 uses by passing the necessary info, then it can be incorporated into a larger series run from a main workflow via a loop, or incorporated into many different workflows.

[mlauber71]

@ebarr well, there is this node which is available from Selenium

Reset on Load — NodePit

Another approach could involve using a local SQL database like H2 and always pull the latest entry (per ID?) from the database:

DB Update won't progress - stuck on 'executing....' status

@summer_le I create an example how to UPDATE data within the database. In this case a H2 Db. The exact syntax might differ if you use it on another Db system. In this case in oder for the UPDATE … SET syntax to work a VIEW is being created that would do a LEFT JOIN between the old and the new data (of course you could just use that also). [image]

How to update database with scraped data

@RIchardC I put together a (hopefully) complete example using H2 and the Northwind database with customerid. Here we first create a database with a customerid as a primary key (randomly selected from the central DB). Then every 10 seconds another random batch gets drawn. Existing customerid will be updated. Then the workflow determines which customerid are new and would insert them. If a row is inserted initially a timestamp first_inserted will be stored. Then with every update another timestam…

[danielesser]

@mlauber71 Just a heads-up: Mentioned node is also available as part of the free NoOp Nodes. As we got a few requests for this very specific node, @qqilihq was so kind to fork it from the Selenium Nodes and make it available for free.

Reset on Load — NodePit

Best regards,
Daniel

[ebarr]

excellent! thank you everyone!