top | item 28751033

(no title)

l9i | 4 years ago

I can assure you that Google has a procedure in place for that.

discuss

order

l9i|4 years ago

I unfortunately cannot edit the parent comment anymore but several people pointed out that I didn't back up my claim or provided any credentials so here they are:

Google has multiple independent procedures for coordination during disasters. A global DNS outage (mentioned in https://news.ycombinator.com/item?id=28751140) was considered and has been taken into account.

I do not attempt to hide my identity here, quite the opposite: my HN profile contains my real name. Until recently a part of my job was to ensure that Google is prepared for various disasterous scenarios and that Googlers can coordinate the response independently from Google's infrastructure. I authored one of the fallback communication procedures that would likely be exercised today if Google's network experienced a global outage. Of course Google has a whole team of fantastic human beings who are deeply involved in disaster preparedness (miss you!). I am pretty sure they are going to analyze what happened to Facebook today in light of Google's emergency plans.

While this topic is really fascinating, I am unfortunately not at liberty to disclose the details as they belong to my previous employer. But when I stumble upon factually incorrect comments on HN that I am in a position to correct, why not do that?

grayfaced|4 years ago

In future news: Waymo outage results in engineers unable to get to data center. Engineers don't even know where their servers are.

shemnon42|4 years ago

Give us the dirt on how google does it's disaster planning exercises please! Do you do these exercises all at once or slowly over the year?

ric2b|4 years ago

Yup, they make a new chat app if the previous one is down.

gadnuk|4 years ago

Google Talk, Google Voice, Google Buzz, Google+ Messenger, Hangouts, Spaces, Allo, Hangouts Chat, and Google Messages.

At some point, they must run out of names, right?

mr_toad|4 years ago

> Yup, they make a new chat app if the previous one is down.

Continuous Deployment.

knorker|4 years ago

For those who don't know who he is: l9i would know this. Just clarifying that this is not an Internet nobody guessing.

sam_lowry_|4 years ago

He is still an anonymous dude to me.

jaywalk|4 years ago

I don't know who either he or you are, so...

astrange|4 years ago

Why does it matter if he's guessing or not?

still_grokking|4 years ago

I've read here on HN that exactly this was the issue as they had one of the bigger outages (I think it was due to some auth service failure) and GMail didn't accept incoming mail.

l9i|4 years ago

A Gmail outage would be barely an inconvenience as Gmail plays a minor role in Google's disaster response.

Disclaimer: Ex-Googler who used to work on disaster reponse. Opinions are my own.