Amazon goes down; takes S3, Salesforce, Target, and others with them.

[cperciva]

And people told me I was crazy for hard-coding IP addresses for s3.amazonaws.com and sdb.amazonaws.com into the Tarsnap code... :-)

[akl]

You are crazy for doing that - one instance of downtime doesn't justify ignoring all the advantages DNS brings.

[justinsb]

Awesome ... was that for security or for redundancy? Did you have a solution for what would happen if the IP address changed?

[justinsb]

It is DNS. If you put the EC2/S3 address into /etc/hosts, the services work fine. Affecting lots of other big websites as well apparently (target, salesforce) because they all outsource to UltraDNS

[shaddi]

NANOG chatter confirming it is an issue with UltraDNS. Seems to be west coast related.

EDIT: Potentially a DOS attack. From NANOG: "We have some DNS providing type customers (not UltraDNS) receiving a few million packets/sec of UDP/53 DoS traffic, starting at about the same time as the UltraDNS problems. No clue if it's related, but it certainly sounds suspicious. :)"

[aristus]

I'm very surprised that they rely on a single vendor. But I guess DNS is one of those things you don't think about until it fails.

[justinsb]

I thought DNS was supposed to try backups servers automatically... any DNS experts able to explain what's going on? Some of the ultradns servers are returning (correct) values, others simply not responding.

[lsc]

yeah, uh, if you are smart, you have a secondary DNS provider. But that really requires you managing it yourself. the problem was that many people outsource, which usually means going with only one provider. (now, ultradns does have a good setup, they probably aren't a bad choice for a provider, but having only one is just plain stupid.)

[boredguy8]

Never let truth get in the way of a good headline.

[artagnon]

Haha.. true. What else can we expect from Techcrunch?

[jseifer]

It doesn't say in the TC article and I can't really tell from this thread. How long was Amazon actually down?

[ggrot]

I wonder how much money amazon loses per minute 2 days before christmas. Ouch.

[rlpb]

I don't think that Amazon itself will lose much, since they're well-known enough that most people who fail to reach them will retry later.

I think the small fry using S3 and EC2 will be the ones who are actually hit by this.

[cookiecaper]

I don't know, I don't think it'd be as much as 4 - 7 days before Christmas. Most people know that it's too late to order from Amazon or any other online-only store by the night of Dec. 23. It's probably more money than they'd lose normally, but maybe not that significant?

I have no data to back that up, it's pure speculation.

[alain94040]

At $20B annual, assuming Christmas is a big chunk, I'm guessing $2B (10%) in the last two weeks, 2 minutes would then be worth $200,000?

I think my 10% number is too low, so maybe $500K for two minutes? But this is not profit, just revenue.

[notmyname]

surely they mean "Amazon goes down and takes the Internet with it"

[slig]

Only if TC was hosted there.

[Sam_Odio]

It looks like a DNS issue, since while http://amazon.com isn't working for me http://72.21.207.65 is (kind of).

EDIT: "No A records were found for amazon.com" http://www.zoneedit.com/lookup.html?host=amazon.com&type...

[mattiss]

When Rackspace goes down and takes TechCrunch with them, the whole INTERNET goes down. When Amazon goes down, Salesforce, Target, and others...

[tinio]

It looks to be back up now.

[newhouseb]

Why wouldn't such a company run their own name servers? I understand it's "yet another thing to maintain," but I've set up bind before... didn't seem that bad.

[evgen]

This is one of those services that a dedicated provider can sometimes do better than internal IT. Ultradns, for example, has secure secondaries Colocated with large ISPs so you get some good protection against cache poisoning attacks. Everything they do you could do yourself, but it would cost you a lot more than what they charge. (full disclosure: I am a customer and until this evening I was quite happy with their service and reliability)

[rbranson]

It's relatively trivial to outsource. It's one of those services that's easy to measure, quantify, and manage (from an outsourced perspective). There's also a bit more to it then that. The Anycast routing can be quite difficult to setup and maintain. It's virtually useless outside of a very small set of protocols (DNS being one of them), so it wouldn't make sense for Amazon to bring that kind of talent in-house for something like DNS.

[tybris]

Internet can be so fragile.