Heroku is down

[carsongross]

I lose money when Heroku is down, and let me say this: Heroku is a f$(king miracle, and you'll pry our apps cold, dead fingers off of their stack.

They screwed up. They'll fix it. Any ops team and hosting solution I could hire for a comparable amount of money wouldn't even come close.

[edouard1234567]

Got to love their last two updates. Spot the difference :).

UPDATE: Process management error rates remain high. Engineers are continuing to investigate and work towards a resolution. FEB 25, 2012 – 20:27 UTC – 10 MINUTES AGO

UPDATE: Process management error rates remain high. Engineers are continuing to investigate and resolve the issue. FEB 25, 2012 – 20:11 UTC – 26 MINUTES AGO

[jared314]

Sounds normal. They have nothing to add, but they have to say something.

[matt2000]

Just out of interest sake what are people doing as a backup to heroku, and is failover automatic or do you have to make DNS changes?

[jvehent]

DNS changes does not provides high availability at all. Even with a ttl of 60 seconds, if a cache decides to overwrite your TTL, you're screwed.

Real high availability comes with a price: redundant datacenter with BGP failover.

[dholowiski]

Yet another reminder that well services like heroku are great, its up to you to make sure you have the appropriate redundancy in place to make sure your critical app doesn't go down. If you host anything critical on heroku, you _need_ a backup server elsewhere (not ec2!)

[lukev]

But doesn't that defeat the whole point of Heroku? If your app is running redundantly elsewhere, that means that you've already done all the work to set up the stuff that Heroku normally provides. And if you've already done that work, why use Heroku at all?

[DannyPage]

Having this problem too. The thing that I'm worried about is that my site just keeps trying to load without any error message or 404 page appearing. Is there a way to get something to show up to inform users of the downtime?

(Or ideally, a way to point towards another instance of the site quickly. I'm worried the DNS wouldn't propagate fast enough)

[wilfra]

Mine says:

Application Error An error occurred in the application and your page could not be served. Please try again in a few moments.

If you are the application owner, check your logs for details.

[overworkedasian]

isnt heroku based on EC2? they should be able offer some type of redundant package for customers willing to pay for it. i mean, lets be serious, people WILL pay for it, assuming it works.

[edouard1234567]

When heroku goes down. 1M+ apps go down.

[rhizome]

Now's the time to launch!

[edouard1234567]

Heroku should know better and not host their status page on their servers since it becomes unaccessible when most needed. This problem started last night and they detected it this morning, monitoring system FAIL.

[ohgodthecat]

It appears to be hosted on rackspace (right now at least). But it doesn't look automatic so perhaps the people were not available to update it til this morning.

[wilfra]

and not eating its own dog food...

http://blog.warsocial.com/post/18262715784/mmmm-dog-food

[mechanical_fish]

That comment is made from sour grapes.

If some bits of Heroku seem to be up, but not others, it could be because those bits are running older versions of the infrastructure. Or are being used as test beds for newer versions of the infrastructure. Or are deliberately running on a separate platform so that, when the main Heroku infrastructure starts having problems, other bits of Heroku's domain are still around to dispense advice on how to work around those problems.

(The most extreme example is a company's "status" subdomain, which ideally should be hosted on a completely different server, in a completely different datacenter, on a different continent located on a distant planet with different DNS regulations.)

[imbriaco]

The devcenter site is absolutely hosted on the Heroku platform.

I don't have first hand knowledge of this specific incident, but I do have a very deep understanding how the Heroku platform as a whole works. It is incredibly likely for many service disruptions to affect a only subset of applications, and not the platform as a whole. The system is, in fact, specifically designed to isolate failures of individual components to as small a failure domain as possible. That doesn't always work, but implying that there's something nefarious going on or that Heroku is not eating their own dogfood with the devcenter site is dead wrong.

[c00w]

It is. Heroku.com is down.

[wilfra]

Our app is back up.