top | item 11822022

Amazon search was down

153 points| gourou | 9 years ago |techcrunch.com | reply

159 comments

order
[+] granos|9 years ago|reply
Does something like this actually cost them a huge amount in sales or do people know enough about the internet today to just come back in a few hours and try again? I know that personally, if I'm buying from Amazon, I would just come back later and try again.
[+] splike|9 years ago|reply
I have a friend that worked in Amazon for a while. He told me that after outages are repaired, there is a spike in sales but that it is never enough to cover the loses. The revenue does just disappear.
[+] ripitrust|9 years ago|reply
They might still lose revenues from potential impulsive purchase

It just happened to me, I was trying to search a book and want to buy it, but since I can't search, I didn't. And now I don't want to buy it anymore

[+] hullo|9 years ago|reply
I think it comes down to how you define "a huge amount in sales". Let's optimistically say they lose 1% of sales over a 4 hour period although it could certainly be much higher. Is 1% of Amazon's sales a huge amount for them? Not in the short term. Is that absolute amount a huge amount for me, personally, or most of the other companies in the world? It sure is.
[+] mrdrozdov|9 years ago|reply
If there is no spike of sales, all hours of sales is worth the same, and assuming Amazon makes 100B per year in online sales, then it seems like a considerable loss, but still a drop in the bucket when compared to their total revenue:

  (1 - (365 * 24 - 4) / (365 * 24)) = 0.000456621...
  0.000456621... * 100,000,000,000 = $45,662,100.4566...
[+] Someone1234|9 years ago|reply
And a lot of people will do also.

But some won't. They'll either be buying on a whim (and the mood passes), or buy from a competitor. Those are lost sales that aren't coming back.

There are people around who never buy on a whim, but there are also a lot of people that buy stuff because the thought crosses their mind, forget about it, then remember when it turns up several days later. These aren't always things you "need" strictly speaking.

As to competition I'd name Jet.com, Target, and Walmart as being major ones. I buy a LOT of stuff from Target these days using the Red Card (effectively 5% pays for the tax, and free delivery on all purchases).

[+] trjordan|9 years ago|reply
Sort of. If the brand is good (e.g. Amazon), people will come back later to buy.

On the other hand, you can model some buying as periodic, not need-based. I go to the grocery store when I need food, but I go to J Crew once every six months. If I'm traveling this June, I'll delay my trip until July, so while the revenue isn't "lost", it's pushed into the future. My next trip is then in January, not December, and at this point, J Crew has lost one month of my sales forever.

There's a similar effect for many items at Amazon. I only spend so much time shopping, so if I can't do it now, there's some opportunity they miss out on in the future. The aggregate ends up looking like revenue that's lost forever.

[+] dsugarman|9 years ago|reply
they say they lose 1% of revenue from every extra 100ms of page load time, how much do you think they would lose from 4 hours of downtime?
[+] Aelinsaar|9 years ago|reply
Not only are they losing money, they're giving customers who otherwise never would, motivation to seek out competition they might not come back from.
[+] stanmancan|9 years ago|reply
I seem to recall reading an article or post previously about how they don't lose nearly as much as most people expect for this very reason. They would see a dip in sales when it was down, but it was more or less made up with a slight sales spike when it came back up. If I can find the article I'll be sure to share.
[+] ww520|9 years ago|reply
I was doing some late night search for a purchase in Amazon and saw the problem. Tried couple times and gave up. Now I lost the mood to continue. Yeah, it does cost loss of sales.
[+] jdimov10|9 years ago|reply
They should consider moving to AWS ;)
[+] 0xmohit|9 years ago|reply
In that event, AWS would respond by saying:

    The EC2 has responded, and informed me that they do not provide analysis of individual issues in AWS infrastructure.

    They referred to the AWS SLA that states that "AWS will use commercially reasonable efforts to make Amazon EC2 and Amazon EBS each available with a Monthly Uptime Percentage (defined below) of at least 99.95%".

    The AWS SLA is available here:
    https://aws.amazon.com/ec2/sla/
(The above is a part of the response I received when I attempted to ask AWS about sporadic reboots and outages on one of my instances.)
[+] jawilson2|9 years ago|reply
Nah, they should just host it themselves. Bandwidth is much cheaper than AWS.
[+] alecbaldwinlol|9 years ago|reply
Reminds me of Larry Ellison's old joke, "So when building our cloud, we decided to go with Oracle... we got a good price..."
[+] _nickwhite|9 years ago|reply
It just came back up it seems (9AM EST).

I imagine a room full of Amazon engineers all standing around freaking out and possibly yelling at interns.

[+] yomly|9 years ago|reply
I can only imagine the hell some team must be experiencing right now.

Godspeed, Amazon site reliability team...

[+] ionwake|9 years ago|reply
Do people get fired over things like this in the bigger companies?
[+] e40|9 years ago|reply
I have no idea where I read it, but I think it was about Bezos and a situation just like this. The quote was something like "why would I fire the person that caused the outage? They just learned a very valuable lesson and got great on the job training, why would I fire them now?"

When I read it I remember thinking how true it was. That person would be very unlikely to commit the same error, and might even work to prevent others from making that error.

[+] loliworkatamzn|9 years ago|reply
I once caused a 60 minute outage at Amazon and the next day my manager and my skip both begged me not to blame myself and freak out and quit.

Mistakes happen. Sometimes they are really expensive.

[+] umanwizard|9 years ago|reply
Not as bad as this, but my friend was an SDET contractor at amazon and bugs in his testing framework let a multimillion dollar bug to go to prod. Not only was he not fired, but he got a full-time offer at the end of his contract.
[+] witten|9 years ago|reply
Yes. One of my co-workers actually got fired from Amazon for causing downtime. And this was after following all procedures, documentation, etc.
[+] drhayes9|9 years ago|reply
I would hope not. I've broken things at big companies before and, mostly, everyone just wants it to work again. Mistakes happen sometimes.
[+] wrsh07|9 years ago|reply
At Google, the answer is a firm no. Someone else commented on the "we just spent a million dollars training them, why would we fire them now?" quote which is commonly thrown around.

People get fired for leaks and being intentionally malicious, not mistakes. Even expensive ones. [besides, a big outage is usually caused by multiple smaller issues if your systems are sufficiently reliable. Also, read the SRE book about "blameless postmortems"]

[+] xyzzy4|9 years ago|reply
I used to work in problem management at a big company. The answer is no unless it was deliberate.
[+] stevewilhelm|9 years ago|reply
Not for a single incident. If production outages become a frequent occurrence, the VP of engineering or Ops may not be around much longer.
[+] hmate9|9 years ago|reply
Someone probably will get fired.

Last quarter amazon earned $20.58bn from product sales.

So 4 hours of that is $38,000,000.

[+] vamur|9 years ago|reply
Could be due to HTTP/2 switch - e.g in the morning it was working in HTTPS and now it's HTTP only again. BTW, it's interesting how much bloat there is in their HTML code.
[+] evansekeful|9 years ago|reply
It's kind of interesting how being the purchasing agent for an Amazon seller account gives you visibility on Amazon's health. After getting my 9am sales report, I panicked and had our tech team check if our seller software was working properly. Conversely, I knew Bezos was laughing all the way to the bank when people ragged on Prime Day last year before the official numbers came out. We didn't even have a featured deal and could see the insane difference.
[+] BinaryIdiot|9 years ago|reply
I noticed this while I was up very late working but it only was an issue for some items. For instance when I searched for Item1 and Item2 it failed but Item3 just kept working. Not sure how they have search divided up but found that interesting.

Would love to see something come out about how it affected their users.

[+] artursapek|9 years ago|reply
Does Amazon do public post-mortems for things like this, the way they do for AWS outages?
[+] sabujp|9 years ago|reply
I expect to see a post mortem.
[+] misiti3780|9 years ago|reply
Is it back up now because I just searched from Croatia and it worked fine ?
[+] thomnottom|9 years ago|reply
I read that as "searched FOR Croatia" and thought, "Man, they really do sell everything these days..."
[+] lwhalen|9 years ago|reply
Would love to see the outage writeup and RCA for this, if/when it's published.
[+] ohitsdom|9 years ago|reply
Working for me right now. Hard to imagine search being down for 4 hours...
[+] known|9 years ago|reply
That hurts AMZN credibility