Google: 'At scale, everything breaks'

[DanielRibeiro]

It is always a pleasure to read from Urs Hölzle. I've been in touch with some minor spec of his work[1], but I'm always impressed by it.

[1] http://research.google.com/pubs/author79.html

[veb]

All the best (and fun) scaling issues is best solved by hacking. Is there really any proper decent methods out there that can actually take a company from 5 users a day to 5 billion without any problems at all? I'm aware that you could perhaps "foresee" it, and get a big data-center, and lots of hardware solutions -- but is this really the solution that's called an 'out of the box' solution?

Sounds like when Google gets a problem, they create something to fix that, which is pretty damn cool in my opinion.

[zaphar]

As a google employee I can say it really is "pretty damn cool".

[clarkbox]

"...most applications don't use [Google File System (GFS)] today. In fact, we're phasing out GFS in favour of the next-generation file system that is very similar, but it's not GFS anymore."

what could this be? home grown? an open source project?

[haberman]

Colossus: see a slide about halfway through this presentation: http://static.googleusercontent.com/external_content/untrust...

[delinquentme]

"We use tapes, still, in this age because they're actually a very cost-effective way as a last resort for Gmail. "

.. what kind of tape drives does Google use?

[dholowiski]

BFT... big freaking tapes. I wonder if it's really tape though. You can get 'tape' backup systems where the cartrige looks just like a tape, but it's actually a special hard drive. I wouldn't be surprised if that's it. Otherwise, I wonder how many miles (of tape) long my gmail inbox is?

[mv1]

I love this comment: "The reason why we put it in is not physical data loss, but once in a blue moon you will have a bug that destroys all copies of the online data and your only protection is to have something that is not connected to the same software system." I think that is often overlooked when designing HA storage systems.

[herf]

Am a little bit surprised Google hasn't incorporated SSD more.

[rxin]

For lots of applications, the workloads are either memory only or large sequential IOs. SSDs matter less in both situations.

[arkitaip]

Too expensive, too prone to failure http://www.codinghorror.com/blog/2011/05/the-hot-crazy-solid...

[clarkbox]

think commodity... have ssd's reached the commodity level yet? from a commodity stand point, i think it would be far more likely that google would use an array of SD cards per node. SSDs are really just an array of SD cards anyway, with a pile of hype and marketing piled on. you would still get most benefits of an SSD. power usage (esp at idle) might even be less than an ssd. google could develop their own wear leveling algorithms, and the rest of the stuff that an SSD controller provides for the internal flash. replacement costs could be less as well over time.

[Locke1689]

Where did you get the impression that Google hasn't incorporated SSDs?