Parse is down due to SSL certificate expiry

[seiji]

Do yourself a favor and run this nightly: http://prefetch.net/code/ssl-cert-check

A simple loop over your certs will solve so many problems:

  cd ssl/certs; for pem in *.pem; do ssl-cert-check -a -x 15 -e [email protected] -q -c $pem; done

[imagine a grumpy "silly companies with millions of dollars in funding, no actual processes, and too little systems knowledge" rant here. it's like a racecar driver who's team lets him run out of petrol every two laps. this is basic stuff, kiddies.]

[peterwwillis]

I recommend doing the test against the live SSL server. Accidents happen, and sometimes webservers get configured with a different SSL cert than you have in /etc/ssl/certs/. Heck, i've seen orgs with a whole slew of certs created in their issuer's domain management page, and none of them were ever applied to the server. Nobody would know the certs loaded in the webserver were bad until they get a page error.

[notaddicted]

Also, the nagios monitoring "check_http" will check the time to expiry of HTTPS certificates, and warn you when you have 30 days left. http://nagiosplugins.org/man/check_http

[gizmogwai]

I wish it was only an issue that occurs in such cases. Unfortunately, this kind of stupidity happen TWICE in a row in a company I worked for previously. The developpers told the manager litterally MONTHS ago, but it was just not considered as important enough. (Eh, if I click on "Approve" it works, so what the big deal?!)

[ses]

It's an important reminder that most startups are run by a handful of people who are normal, fallible human beings. They generally don't have access to the infrastructure and manpower required to be on the ball and maintain near 100% uptime of their services. I think the important information to be gleaned about a company is how they deal with problems like this (not necessarily the fact that they encountered the problem in the first place - if that is of concern, pick a more established organisation).

Also another issue, which probably doesn't apply to Parse is limited funding. One of my own venture's certificates has now expired and as a 'bootstrapped' web app, I simply can't justify spending more to renew the certificate. However I keep the service live as a demo to potential customers, if one takes a very keen interest I will happily renew it, or seek further funding.

[CoreDev]

This sounds true, but there is always something called calendar for important things which will expire.

I have a own calendar just for expire dates of software and certificats. My reminder is set to 2 months before the event.

[leftnode]

But certificates cost almost nothing these days, why not just spend the $20-$40 and get a valid one in there?

[neya]

This is why I think its always good to build your apps in the 'Good old way', instead of relying on a third party like Parse. It becomes a bottle-neck in situations like these, where in you are kind of stuck - Either, you will have to re-write your code from scratch, or wait for them to fix it, with a lot of uncertainty.

[leftnode]

Is there any value in companies like Parse selling their software as installable on your own servers (i.e., GitHub Enterprise).

I ask, rhetorically, because I built a somewhat competing service that isn't doing too well and I'm looking to make it so you can install it on your own servers.

[kordless]

The devil you know, right?

[kogir]

Hint: the web is one thing, but if you're deploying a library that talks to your own server, consider embedding your own root certificate. In addition to verifying the traffic isn't being proxied[1], you can also issue 10 or 20 year certs and be done.

[1] Users can add their own root certs and watch your traffic (typically with the intent of breaking your app), or worse, companies can deploy roots to all their devices and sniff their employees' traffic. Why be complicit?

[j_s]

If you have your own cert, wouldn't it be best to ignore expiration (and decide based on risk whether or not to implement revocation)? Unless you are saying to add the cert to the device/computer's list of trusted certs, which doesn't seem necessary to me.

All of Google's pinned certs are certainly evidence that this is a reasonable approach (I haven't looked at their expirations), though now it's kind of weird since it can't be MitM'd even to debug -- no one knows what is being shipped off to Google by Chrome anymore without reverse engineering binaries.

[sk5t]

Issuing inordinately long-lived certificates might not be the best security decision. To put it another way, "infinity" is not the appropriate lifetime for a 4096-bit cert.

[peterwwillis]

For a big org, SSL certs can sometimes be a bitch to manage. I recently started a project to track all the SSL-enabled devices on our medium-sized network, and it came out to something in the low four-digits. As you might assume, not all of them are CA-signed, and many of them expire without anyone ever realizing it.

I wrote a couple scripts to manage bulk-checking SSL certs on a network. One of them uses Curl's Mozilla root CA .pem file and follows the chain to verify a cert is really signed and not expired. https://github.com/psypete/public-bin/tree/public-bin/src/ne...

[stevencorona]

It costs $100 for a 5 year SSL certificate. Why do seemingly smart internet companies let this happen?

[michaelt]

Well, it's a 1 year wildcard certificate from DigiCert, which costs $595 according to [1] - while their 3 year certificate costs $1425.

That's quite a bit of cash when you're a small business - and your minimum viable product doesn't need a 5 year cert, you just have to remember to renew your cert if you're still in business after a year...

[1] http://www.digicert.com/wildcard-ssl-certificates.htm

[consultutah]

Because they are new, have never run into this problem before, and are bootstrapped to the hilt. At least you know they'll never let it happen again. ;-)

[whalesalad]

Where the hell are you buying your certs? I'd honestly love to know.

[xoail]

I started using Parse for my app and as I code I am starting to realize how painful it will be to move away from it in future. It would mean a complete rewrite of the app. Now, if they have problems like these (down due to certificate expiry), I am not sure what value I will get out from this service. Also, it is taking more time for me to learn their SDK than I thought, as I am beginning to write complex queries and relations. I encourage people to write data services on their own.

[jordantbro]

No mention of the issue on their Twitter account or self-proclaimed "Status" page:

http://status.parse.com/

[meritt]

It's on there now as of 7:39am PST

[TazeTSchnitzel]

There is now.

[csmajorfive]

We're working on it.

[kordless]

Been there. It sucks. FWIW, I recommend Zerigo monitoring for certs, and other various bits of web monitoring you might need.

Take a few minutes to update your Twitter feed.

[csmajorfive]

We're back up. We'll post more about the root cause soon.

[yuhong]

They just fixed it. FYI, in case you did not look at the certificate details, the old certificate expired 5AM today.

[zmitri]

Working now for me! 3h42m downtime :(

[yuhong]

And I am in the GMT-8 time zone and used Firefox.

[zmitri]

Put this up with hope that someone here can contact them. They aren't responsive to twitter or emails.

[neya]

Now, imagine the plight of all their customers who had built their apps based on their APIs. Hopefully, this is a very good lesson for all.

[hiddenstage]

For a company based around being convenient, they sure have been inconvenient as of late.

[freehunter]

In my line of work, I see this a lot (both expired and using a cert on a subdomain the cert isn't valid for). It gets really difficult trying to balance security with the needs of the business when employees are begging me to whitelist the site so it isn't blocked by the corporate proxy but I really don't want to whitelist invalid certificates. That sets a bad precedent.

There really should be another way that doesn't involve SSL certificates.

[hoffmanc]

This is a testament to the dangers of relying on third-party APIs to provide core functionality for your app.

[pallinder]

Yeah we are also seing issues. Fortunately we arent using them on that many places so should be fine anyway but my god this is a silly mistake to do.

[xoail]

On average, how much time does it take to renew the certificate and bring it back online?

[bwhalley]

This combined with their downtime episodes last week... Think twice before using.