[jeffinhat]

It's definitely a different use case but given they haven't had to tap into their follower replicas for scale, it must be pretty efficient and lightweight. I suspect not having ACLs helps. They also cite a minimum 2MB size, so not expecting exabtyes of little bytes.

I wonder if a major difference is listing a prefix in object storage vs performing recursive listings in a file system?

Even in S3, performing very large lists over a prefix is slow and small files will always be slow to work with, so regular compaction and catching file names is usually worthwhile.

[jleahy]

2MB median to be fair, so half of our files are under 2MB.