[aynyc]

I read that. But afaik, feather format is stable now. Hence my confusion. I use parquet at work a lot, where we store a lot of time series financial data. We like it. Creating the Parquet data is a pain since it's not append-able.

[yencabulator]

Generally Parquet files are combined in an LSM style, compacting smaller files into larger ones. Parquet isn't really meant for the "journal" of level-0 append-one-record style storage, it's meant for the levels that follow.

[aynyc]

So feather for journaling and parquet for long term processing?

[dionian]

Have you considered something like iceberg tables?

[aynyc]

Yes, but parquet hates small files.