Tell HN: I accidentally ran up a $1000 Heroku bill

[erikpukinskis]

This is pretty typical. Every few years something slips my mind and I end up stuck with a "Erik, you're a moron tax".

I was experiencing a spike of traffic when my site was on national news a few weeks ago, and I turned my dynos and workers way up on Heroku to handle the traffic. The spike went down, but I forgot to turn off the dynos.

Flash forward a few weeks, and I am stuck with $1300 bill on a site that I'm not making any money on. I asked Heroku if they could help me out a little... I'm a good customer, and even though I'm a cash-strapped in-the-red bootstrapper, I gladly pay them hundreds of dollars a month to run my sites, bills payed on time and in full. I was hoping they'd maybe give me a little discount... not pay for my servers, but maybe not take a profit off of my stupid mistake.

They shot me down. No refunds available, even though these were virtual servers, and didn't contribute any real load to any of their machines. Although obviously they provisioned machines for me, so they had real costs.

Part of me wants to be upset, but obviously I was the one who screwed up. Still, it leaves a bad taste in my mouth. I guess a smaller host might offer more personal care, but I wouldn't get Heroku's world-class engineering baked into their platform, which I really, really appreciate.

What do other folks think? Am I just a dumbass who lost a chunk of paycheck, or is this poor customer care?

[patio11]

Heroku runs on top of EC2, and they get charged the same amount of money by Amazon whether your instances are idle or on fire. That doesn't matter, though, because businesses do not do cost-based pricing.

There comes a time in a young man's life when "haha, whoopsie, I'm a kid who is not responsible for his own decisions" is no longer sufficient for a do-over. I am not sure when that is, but if you're running a business, well, welcome to being an adult.

[staunch]

How horribly condescending and wrong. Any company that lets their customers go from huge fan to hater in a week has a major bug. The important question is why did their product turn a fan into a hater, and how can they prevent it?

Heroku should almost certainly have user defined cut off points. They should require every new user to set a maximum monthly bill to something they can afford (or "unlimited").

I did exactly that for a similar type of service I worked on. One of the biggest anxieties customers had was that they'd get a massive unexpected bill, just like this guy did.

Heroku's best move would be to cut his bill in half and institute a cut-off feature. Every service like it should follow suit.

[jacquesm]

There comes a time in a business owners life when he realizes that he may be able to take the money from this particular customer but that the PR damage done by that customer may be larger than being reasonable and understanding that mistakes happen. And if there is some egg on the face of the business owner for not alerting the user to their sudden suspicious consumption pattern then there are valuable lessons to be learned for all parties.

So, the adult business owner would likely split the bill with the customer to cover his costs, fore go his profits on this particular occasion, have a very happy customer (who might even write about that!) and would apply the lessons learned to their business.

[petercooper]

Heroku runs on top of EC2, and they get charged the same amount of money by Amazon whether your instances are idle or on fire.

Is the latter part of that true? (I'm not playing Devil's Advocate, honest!)

I'd assumed the lion share of Heroku's profits were coming from clever arbitrage of EC2 instances rather than markup alone.

For example, if they have 1000 users paying $108 per month for 4 dynos, they don't keep 4000 "dynos worth" of EC2 instances running constantly. Instead they can see the overall load and maintain that lower level of EC2 instances plus some "slack." When traffic increases among apps, the slack can be rapidly configured to pick it up. Heroku would then make not only a markup on EC2 but also profit from the difference between the maximum load and actual load. (Isn't this how cloud scaling systems work anyway, right? Heroku could/might be doing that over thousands of apps for collective benefit.)

Perhaps I'm over-thinking how I'd be trying to squeeze profits out of a Heroku-type system though ;-)

[davidedicillo]

I understand Heroku's decision, but I must say, when the opposite happened to me with SendGrid a couple of days ago (I didn't even asked for the refund, they did it on their own initiative), I was extremely positively surprised.

[davidw]

Refunding a portion of their profits on the 1K could be seen as money spent on advertising. It sounds like the guy realizes he made a mistake, so he'd probably be happy if he got something back, and think it perfectly fair not to seek to get everything back, so in the end it wouldn't be a great deal of cash, that, long term, they'll probably recoup in the course of a year anyway.

Since he's a good customer, they could even give it to him in the form of credit for their service, which gives him an incentive not to leave.

[tlrobinson]

I'm pretty sure that's not true, but I don't know exactly how Heroku works.

The way I understand it is if no one is making requests to your app they will shut down the dynos, and when you start getting requests they'll spin them back up, up to the number of dynos you're paying for. That's why if your app has been idle for a long time it takes a few seconds for the first request to complete.

This lowers their costs, and presumably the prices for their customers.

I could be wrong though, and perhaps they only turn off dynos for non-paying customers.

[alextgordon]

I'm going to go against the grain here and say it's partly Heroku's fault. If a customer is paying hundreds of dollars a week for a feature they're not using, that's the time to send a friendly automated email saying "Is everything okay?".

This is especially important with a service such as Heroku, where users will be changing these settings all the time and so there's a high probability of human error.

[plasticbuddha]

If it's not Heroku's fault in some partial way, then look at it from another perspective. It's an opportunity for Heroku's competitors in a market that is quickly becoming commoditized. At the very least it's a feature that I always look for when using pay on demand services, and will rank very highly on my list of important to have features.

Akamai comes to mind as an example of a company that has always had similar issues. If I ever were to use them in the future, I would have to factor into their monthly costs the partial FTE required to manage their usage and costs. More importantly, if there are other CDNs that offer better cost management features, I would almost certainly use them before ever using Akamai again, unless there was no other technical option.

[dotBen]

I can think of loads of reasons why you might be legitimately in that situation...

Perhaps you intentionally provisioned a large amount of headroom for a forthcoming expected/possible traffic spike.

In that example it would be annoying to get "are you sure you want to have this extra resource running?" emails.

(sure, you should be building the system to auto-provision but that would have required a better programmer and you just got the interns to build it, etc)

[StavrosK]

This reminds me of something my friend did. We had an assignment to try out EC2 with Hadoop and we needed 8 machines, and we were warned in class to turn them off or they'd keep charging.

Guess who didn't turn them off.

So he casually mentions it to me and tells me that he racked up a 500 pound bill, but his CC didn't have any money in it so it was fine. I asked "so did you turn them off?" He said "no, not really, but it's okay because they can't charge me".

So he racks up another 500 pounds, and only turns the instances off when he finds out that he didn't use the inactive CC but another, active one. Long story short, he's 500 pounds out (Amazon were kind enough to refund the first 500, which disappeared as soon as it hit his account, since he owed another 500).

[dotBen]

reminds me of a friend who was studying for his MSc Physics at Warwick... the students had to do a lab experiment with a radioactive isotope and were warned to only have the box containing the isotope open for a few minutes in order to reduce exposure to safe limits.

Guess who forgot to close this box and after 2 hrs of exposure now has to worry about cancer and/or infertility in later life. True story.

Look at it this way: at least your friend took CS and not Physics and thus is only £500 in the hole.

[spooneybarger]

I'm going to go with dumbass. Nothing personal there, but you turned up the level of service and continued to use it even though you didn't want/need it anymore. The fault is entirely on you. Think of it this way, they provisioned load to you which in turn even if you didn't use, couldnt be provisioned to others w/o potentially over provisioning.

[deno]

Making money on people's mistakes seems like a bad business model. Give him at least discount and you'll have an evangelist.

[jw84]

That's not how clouds work.

[fierarul]

What you are doing is distasteful because you made a mistake and now you are using bad publicity against Heroku to force them to lower your bill (or just out of spite).

Heroku might optimize for this usecase by detecting spikes somehow and notifying you that you still have machines running, but other than that, I don't see how any of this is their fault.

[100k]

I wish Heroku officially supported autoscaling. I've been using Google App Engine lately and the autoscaling and pay for what you _really_ use is very compelling.

[arfrank]

They do for their background workers, but not the dynos. http://github.com/pedro/delayed_job/tree/autoscaling

[collint]

Hmm, you might be out of luck, but it does bring up the question of why Heroku doesn't have load-triggered scaling, or some sort of dead-man's switch option.

[jmonegro]

There are third-party solutions for that, like this one: http://github.com/ddollar/heroku-autoscale

[chr15]

I am not surprised that they didn't give a refund; most businesses wouldn't. However, I think businesses these days should start integrating a one-time "whoopsie" allowance as part of their customer service (within reason, of course. It may or may not be feasible depending on your business model). You're probably going to have people that try to take advantage of this, but there are some people who make honest mistakes, and I would give them the benefit of the doubt. Just treat people well.

I believe if you take care of your customers, they'll take care of you. In general, the one thing that people will remember for life is how you make them feel.

The benefits of providing awesome customer service outweighs the cost IMO (I haven't done the math, but that's my gut feeling). I remember reading a story on reddit yesterday where a gamer experienced the almost exact same situation with Steam:

http://www.reddit.com/r/gaming/comments/dcnp7/one_of_many_re...

that provides a good example of how news of awesome customer service can be spread, and possibly grow sales (Zappos).

For example, I have several sites hosted with WebFaction.com and I'm unbelievably happy with their customer service. They respond within 10-30 minutes on any support tickets. I'll continue to use them in the future. (see how that was a good example of what awesome customer service can do?)

[jacquesm]

I'm actually surprised they didn't. Think of the goodwill a post reading "I messed up on Heroku, but they were cool about it" would have created.

After all, Heroku could have auto-detected idle instances easily and could have disabled them or alerted the user.

The whole 'dyno' concept without an automated scaling option is a bit strange. Why set an exact number of instances when web traffic is just about the most variable thing you could possibly want to measure. Even apache auto-scales the number of processes depending on how many workers it currently needs to serve your requests.

Auto-scaling and a web app hosting service go together like bread & butter.

[cglee]

Why would you say most businesses wouldn't? I'd think most business would do something to help a loyal customer stay loyal. I agree with you that it's absolutely good business practice to help customers recover from their own mistakes.

On many many occasions, businesses have literally given me money even when it was my stupid mistake. Some examples just over the past few years:

- Sprint credited me $100 when I racked up a $250 bill by not watching my usage

- Bank of America credited me $100 when I left my card in the ATM and someone withdrew $100 from my account

- Hyatt Place gave me a free room when I made a reservation on the wrong date (and never showed, or called)

- Southwest Airlines gives you full credit if you cancel your flight any time

- ATT gave me all my rollover minutes when I changed plans (by rule, you lose them - i had over 2700)

Those are just some examples that I can remember...I'm not sure it's a healthy mentality to expect this, but it certainly leaves a very positive impression. Leaving your customers in a state of feeling ripped off when you can obviously help is not the right business decision.

[steve19]

Apple apparently has a one-time whoopsie allowance. According to many reports on the 'net, they will one-time refund an app purchase.

[benatkin]

Expedia has this. I once booked a flight & hotel for the wrong month. I was told they could cancel the hotel free of charge, but for the flight they would be able to take care of it just this time and if it happened again it would be $100.

[sad]

It's tough to say. You really sound like a reasonable person and if you are the model customer you appear to be then I would probably side with you.

I like a very simple rule: Customer retention is far more important than customer acquisition. It may be a flawed rule, I'm not sure since I'm not a successful business person. But it sounds good to me. It seems vitally important to retain your good customers.

[ams6110]

Yes, the very fact the the OP is now considering (however causally) the idea of finding a smaller host with more personal care is something I'd be concerned about if I were his current host. Who knows, maybe he'll go out and find an awesome small host he never knew existed. Customer gone, and he'll probably start recommending the new host to all his friends and associates too.

If it were me, and a good customer made a mistake, I'd eat the cost to keep the goodwill.

[pclark]

Not sure why anyone but yourself is to blame, and I think it would have been nice for Heroku to give you a discount then, after this post they should go quiet and let you pay. I think this post is bad form.

[sr3d]

I'd go with "Dumbass" and add "whiner" to the list. Why is it poor customer care when all Heroku did was saying no to an unjust request? You can move your site to another hosting service, or host it yourself.

"I am stuck with $1300 bill on a site that I'm not making any money on." So that means if you make money on the site, then you'd pay? What's the difference here? You're saying that because you're poor now, you don't have enough integrity to say "oops I did something wrong" and take the responsibility?

I'm myself a boostrapper burning thru my savings and to tell you the truth, I do everything myself to cut on expenses. I'd love to use Heroku, but they are too expensive for me to run. So if you don't want to make another mistake, then start to accept responsibility and learn to take care of problems yourself.

[gfodor]

Happened to me on EC2 once. They cut me a break and gave me 20-30% discount. They can though, because the idle machines were not costing them any money.

[amock]

They weren't costing Amazon any money if they had excess capacity, but if those machines being in use prevented them from being sold to someone else then they did cost it.

[japherwocky]

this happened to me on ec2 this summer- dozen idle instances running for two months because of a bug in a script.

I didn't get any sort of break though. :( June was a lot of hotdogs and Budweiser till I could dig out of that hole.

[Emore]

This didn't happen to me, but that's only because I set up dozens of post-its and alarms reminding me to quit those tests I was running on EC2. That's perhaps the one web-service that has made me the most paranoid.

[audyyy]

If you could get away with turning up your dynos and workers in anticipation of a traffic spike and then not have to pay; it wouldn't be fun any more.

[jacquesm]

Why? It could be fun for everybody, just set a baseline and a peak level and use as much as required with a nice little graph of how close you came to your maximum allowance so you can plan your future hosting expenses.

[niico]

I happened something similar to me. I forgot I was running a recurring billing adwords campaign. I realized I had that campaign when the bill came with a lovely 3.700$ total. The worst part is I didnt got to sell anything.

[thibaut_barrere]

I don't find this poor customer care, really: I consider this normal.

You're talking about "a few weeks": did you check your expenses in the meantime ?

I would put some reminders (google calendar etc) or habits in place to regularly check your expenses.

[checoivan]

It's human error. Simple as that. Happens to absolutly everyone even the brightest individuals. I've seen in "the livesite" that this kind of spike problems are more common than you think, and the ideal solution is to automate an alarm that tells you both, when a spike is happening, and when X throughput passes over a threshold. This kind of automation is easy and spares lots of trouble.Could be as simple as scheduled task parsing some logs. Some more fancy option is always good though :)

[sliverstorm]

At this point, I'd probably just be thanking my lucky stars it was only $1.3k. You'll make it back.

[akshayubhat]

Something like this wont happen to Google App Engine, thankfully.

[dh]

I can see the problem from both sides but really Heroku had some more more to gain by just crediting it or most of it in what you would say about them. Clearly by the rules, you should have to pay, but that does not mean that a business should not do the right thing and work with a customer.

Imagine how big a raving fan you would have been if they had credited the entire thing. That is a tiny cost compared to acquiring a new customer and turning them into a raving fan.

[bdmac97]

I got burned by this on Heroku recently as well but to a much lesser extent $$$-wise. My app typically runs on 1 dyno because it's basically not supported anymore. At some point I must have been messing around in the interface and accidently bumped my dynos. Oops.

I personally think they should have a "WTF your load is like zero, you normally have X dynos, but you're using Y dynos for no real reason... dumbass" alert email... I certainly would have appreciated it!