top | item 43868445

(no title)

elmolino89 | 10 months ago

Ignoring for the moment issues of syncing a database where the source DB may be running inserts/updates:

if one dumps tables as separate CSV files/streams and using DuckDB converts them to individual parquet files the rsync should run faster since hopefully not every table is modified between each new syncing. There is an obvious overhead of the back and forth conversions but DuckDB can directly export a database to SQLite. I have not tested it myself, so it is just a brainstorming.

Last but not least: when compressing/decompressing text dumps use igzip or pigz if you want to speed things up. Also benchmark the compression levels.

discuss

order

No comments yet.