(no title)
lidavidm | 1 year ago
You can sort of see what benefits you might get from a post like this, though: https://willayd.com/leveraging-the-adbc-driver-in-analytics-...
While we're not using Arrow on the wire here, the ADBC driver uses Postgres's binary format (which is still row oriented) + COPY and can get significant speedups compared to other Postgres drivers.
The other thing might be to consider whether you can just dump to Parquet files or something like that and bypass the database entirely (maybe using Iceberg as well).
majoe|1 year ago
We will start a refactoring of the application in a few weeks to get rid of the performance problems. I will keep your advice in mind and do some thorough benchmarks in the meantime.