[aynyc]

What's the difference between feather and parquet in terms of usage? I get the design philosophy, but how would you use them differently?

[tosh]

parquet is optimized for storage and compresses well (=> smaller files)

feather is optimized for fast reading

[aynyc]

Given the cost of storage is getting cheaper, wouldn't most firms want to use feather for analytic performance? But everyone uses parquet.

[twic]

And now there's Lance! https://lance.org/

[dionian]

https://stackoverflow.com/questions/48083405/what-are-the-di...

[aynyc]

I read that. But afaik, feather format is stable now. Hence my confusion. I use parquet at work a lot, where we store a lot of time series financial data. We like it. Creating the Parquet data is a pain since it's not append-able.

[willtemperley]

Feather (Arrow IPC) is zero copy and an order of magnitude simpler. Parquet has a lot of compatibility issues between readers and writers.

Arrow is also directly usable as the application memory model. It’s pretty common to read Parquet into Arrow for transport.

[aynyc]

When you say compatibility issues, you mean they are more problematic or less?

It’s pretty common to read Parquet into Arrow for transport.

I'm confused by this. Are you referring to Arrow Flight RPC? Or are you saying distributed analytic engine use arrow to transport parquet between queries?