Row Filter on Database Level to only get incremental load of data

@kowisoft your workflow does not contain any data. SQLite might not be the best example because it has some challenges handling date and time variables. Maybe best to use H2 like in my example.

Maybe you take another look at that and try to understand since I think it provides the essential tools. I do not really get what you are doing with the update.

There is a Flow Variable as a date/time there derived from the database (could also be from local KNIME) and this Flow Variable gets used to filter the SQL Query. You will have to adapt the syntax to your specific database dialect.

Also you could try and just use a sub-query to get the latest date/time and then use that. If you want to process the data on your database, SQL might be the best way to go forward.

You can also use some insert functions. Question in a database will always be how to determine a unique key (if you have any) - of course you can base your insert solely on the date if you are sure about what you are doing. I have additional examples of update and insert procedures if that should be the case.

2 Likes