top | item 43681556

(no title)

smueller1234 | 10 months ago

Google's internal systems have been written against the Colossus semantics for many, many years and thus benefit from it's upsides (performance, cost efficiency, reliability, strong isolation for a multi tenant system, ability to scale byte and IO usage fairly independently, tremendously good abstraction against and automation of underlying physical maintenance, etc) while not really having too much of an issue with any of the conscious trade-offs (like no random writes).

On the other hand, if you've been building your applications against expectations of different semantics (like POSIX), retrofitting this into your existing application is really hard, and potentially awkward. This is (IMO) why there hasn't been an overtly Colossus based Google Cloud offering previously. (Though it's well publicized that both Persistent Disk and GCS use Colossus in their implementation.)

One of the reasons why it would be extremely hard to just set up or build CFS elsewhere or on a different abstraction level is that while it may look quite achievable to implement the high level architecture, there is vast complexity in the practical implementation side. The tremendous user isolation it affords for an MT system, the resilience it has against various types of failures and high throughput planned maintenance, the specialization it and its dependencies have to use specific hardware optimally.

(I work on Google storage part time, I am not a Colossus developer.)

discuss

order

No comments yet.