I do like this article a lot for showing how to do this pattern of slurping data and inserting it into a DB, in the context of Arrow Flight.
The concurrency rules of DuckDB is here [1]. Reads/writes need to happen in the same process, but multiple threads can do so.
This is putting a server in front of a DuckDB instance, so all read/writes are funneled there in that one process. DuckDB takes care of the concurrency within the process via MVCC.
You could do the same thing with an HTTP server or other system, but this shows it with the Flight RPC framework.
NOTE: I had an incorrect comment for about 2 minutes that I deleted and restructured here. Sorry if you saw that noise.
neomantra|1 year ago
The concurrency rules of DuckDB is here [1]. Reads/writes need to happen in the same process, but multiple threads can do so.
This is putting a server in front of a DuckDB instance, so all read/writes are funneled there in that one process. DuckDB takes care of the concurrency within the process via MVCC.
You could do the same thing with an HTTP server or other system, but this shows it with the Flight RPC framework.
NOTE: I had an incorrect comment for about 2 minutes that I deleted and restructured here. Sorry if you saw that noise.
[1] https://duckdb.org/docs/connect/concurrency
memhole|1 year ago
unknown|1 year ago
[deleted]