Incremental Data Load

[Kurobasa]

Hi guys

I’m new here and have a question.

I’m creating a workflow where I am executing SQL statements live on a database.
The results are sometimes quite huge (e.g. 85 million rows) in the initial query.
Because of performance reasons I don’t want to handle so much data every time I run it. I would like to implement something that only uses the newest data and adds it to the existing one.

Do you guys have any ideas?

regards and thanks in advance

[tobias.koetter]

Hello Kurobasa,
please have a look at this post.
Bye
Tobias

[pigreco]

Do you have a sorted field that you can use as boundary?

[Kurobasa]

Hi Tobias
Thank you for the quick response.
That post helps.
Kind regards
Ilija