@qdmt I like your spirit to crack this thing. What you might want to keep in mind: best to make some sort of plan what your workflow should do. In the previous approach there were examples about H2 and UPDATE functions. Also there is an example where only new lines or the ones that have changed are being inserted
In this thread we used SQLite and just appended data like in a Big Data environment and automatically assign IDs and TIMESTAMPS. All this approaches have their benefits. You might want to think about.
H2 and SQLite are both great standalone platforms. SQLite might be a little bit more ‘stable’ but also has fewer functions. H2 is more advanced but I do not have that much information about how it behaves with large sizes.
In theory you could also use a (local) big data environment where you would have the data as single parquet files and manage them (1, 2, 3).
H2 do an UPDATE on the server using a VIEW of the existing and the new data
MERGE an existing database and insert new rows with H2 - use the Northwind DB
Example of H2 database - handling of database structure and use of Primary Keys - insert only new lines by ID