[stereosteve]

Similar question: how does it compare to using ClickHouse with http interface (which supports a number of output formats including Arrow and ArrowStream).

It does take a few more commands to start a ClickHouse server, create table and load data in... a friendly loader program could probably get pretty close to Roapi interface.

Both Roapi and Datafusion look very cool! Excited to learn about them.

[houqp]

I think these two systems explore different design spaces, the biggest difference I would say is Roapi can apply more read optimizations by exploiting the fact that it doesn't need to support frequent online updates from the client. Most of the datasets it serves will be static. For data-sources that supports streaming updates like delta tables, the update frequency will be much lower than what clickhouse supports.

[zX41ZdbW]

This is not a concern for ClickHouse.

You can load data into MergeTree table that support streaming data ingestion or into Memory table (that is read/write as well).

Also you can run queries on the data without any preprocessing (with `file`, `url` table functions or similar).

See also https://clickhouse.com/docs/en/operations/utilities/clickhou...