GitHub's User Content certificate has expired

[koolba]

The cert for objects.githubusercontent.com has also expired:

    $ openssl s_client -connect objects.githubusercontent.com:443

    CONNECTED(00000005)
    depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
    verify return:1
    depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
    verify return:1
    depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io
    verify error:num=10:certificate has expired
    notAfter=Mar 21 23:59:59 2023 GMT
    verify return:1
    depth=0 C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io
    notAfter=Mar 21 23:59:59 2023 GMT
    verify return:1
    ---
    Certificate chain
     0 s:C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = *.github.io
       i:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
     1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
       i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA

What are the odds this happens the same day they rotate their SSH keys?

[brandur]

It's also the domain used for releases and other artifacts (after a redirect from github.com). There's going to be a lot of broken builds today:

    $ curl -i -L https://github.com/kyleconroy/sqlc/releases/download/v1.17.0/sqlc_1.17.0_linux_amd64.tar.gz
    HTTP/2 302
    server: GitHub.com
    date: Fri, 24 Mar 2023 20:51:56 GMT
    content-type: text/html; charset=utf-8
    location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/193160679/09048595-c7f4-45b5-858a-7f55baa2fd7d?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230324%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230324T205156Z&X-Amz-Expires=300&X-Amz-Signature=772d0aa8c5c19b0a5ef84d718d2faf0d81f24b224a4ef634d2410787e8f50bad&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=193160679&response-content-disposition=attachment%3B%20filename%3Dsqlc_1.17.0_linux_amd64.tar.gz&response-content-type=application%2Foctet-stream

    curl: (60) SSL certificate problem: certificate has expired
    More details here: https://curl.se/docs/sslcerts.html

    curl failed to verify the legitimacy of the server and therefore could not
    establish a secure connection to it. To learn more about this situation and
    how to fix it, please visit the web page mentioned above.

> What are the odds this happens the same day they rotate their SSH keys?

Definitely a bad for them. When it rains, it pours.

[AviationAtom]

Could be a good chance. I'd venture to guess they failed to update the known_hosts file for one of their systems that handles certificate management. Strictly me taking a stab at the answer though.

[belter]

Now they are owned by Microsoft, its a celebratory 10 year tradition. Cause a worldwide outage, by letting your certificate expire...

"Windows Azure Service Disruption from Expired Certificate" (2013) - https://azure.microsoft.com/en-us/blog/windows-azure-service...

[ksml]

They're serving the wrong cert on pkg-containers.githubusercontent.com (it's for *.githubassets.com) and their support site also expired 3/21... https://support.github.com/ What is going on over there?

[dz0ny]

Still some weird stuff around (* subject: CN=apistatus.chorus.co.nz).

    curl https://www.githubstatus.com/ -vvvv -I
    \*   Trying 52.215.192.131:443...
    \* Connected to www.githubstatus.com (52.215.192.131) port 443 (#0)
    \* ALPN: offers h2
    \* ALPN: offers http/1.1
    ...
    \* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
    \* ALPN: server accepted h2
    \* Server certificate:
    \*  subject: CN=apistatus.chorus.co.nz
    \*  start date: Mar  6 23:10:30 2023 GMT
    \*  expire date: Jun  4 23:10:29 2023 GMT
    \*  subjectAltName: host "www.githubstatus.com" matched cert's "www.githubstatus.com"
    \*  issuer: C=US; O=Let's Encrypt; CN=R3
    \*  SSL certificate verify ok.
    \* Using HTTP2, server supports multiplexing

[ollemasle]

Here is the report for this incident: https://www.githubstatus.com/incidents/x7njwb481j9b

[ccheney]

EDIT: this specific issue is resolved

Failing for us in GitHub Actions

For SEO purposes:

  npm ERR! code ERR_TLS_CERT_ALTNAME_INVALID
  npm ERR! errno ERR_TLS_CERT_ALTNAME_INVALID
  npm ERR! request to https://pkg- 
 npm.githubusercontent.com/npmregistryv2prod/blobs/\*\* failed, reason: 
  Hostname/IP does not match certificate's altnames: Host: pkg-npm.githubusercontent.com. is not in the cert's altnames: DNS:\*.githubassets.com, DNS:githubassets.com

[GOATS-]

This also applies to their avatars subdomain, causing them not to load anymore.

[sha-3]

Are you sure? avatars.githubusercontent.com works fine for me.

[radicalbyte]

I wonder if this has anything to do with the recent SNAFU from a Senior Security Engineer* there?

https://twitter.com/viibeeng/status/1639374358287118336

(*yeah we can all make mistakes, but it's 2023, if you've not build controls into your workflows by now you don't deserve to be a Senior anything)

[mattbillenstein]

I built a free monitoring service some years ago if anyone doesn't want to be the victim of this...

https://ismycertexpired.com/check?domain=objects.githubuserc...

[bvogelzang]

It looks as though it's back for me now. Status page is now showing the problem: https://www.githubstatus.com/

[deathanatos]

… well, the status page is back to green now, but AFAICT, the domains are still serving the expired cert.

  » TIMEZONE=UTC date; openssl s_client -connect support.github.com:443 2>&1 | grep 'cert.*has.*ex'
  Fri Mar 24 17:40:28 EDT 2023
  verify error:num=10:certificate has expired
      Verify return code: 10 (certificate has expired)

The previous incident seems pretty clearly to be this … so it seems like they think they fixed it…

[dz0ny]

Also serving wrong certificates for a lot of content domains.

https://news.ycombinator.com/item?id=35295191

[jmspring]

Sounds like whoever is in charge of certificates at GH must have come over from MSFT. Afterall, I think Microsoft has had 2-3 certificate expiry issues in the last several years.

[jiggawatts]

Azure had several global outages because of issues with certificates. One outage was caused by an incorrect date computation: the certificates last for one year, and this was computed with: "new DateTime(now.Year+1,now.Month,now.Day)".

If you do that on Feb 29th of a leap year, it'll throw an exception because the next year doesn't have a Feb 29th! Oops.

They "fixed" it and promptly had another related outage the very next day.

[gorjusborg]

And today of all days I have a moment to upgrade homebrew stuff.

[tonto]

got a "RequestError: certificate has expired" doing a release just now...as usual, not a good idea to release on a friday

[Kelamir]

Previously I had the same issue, but it works for me now, as well as for a friend in another EU country.

[gunshai]

For us dumb dumbs what does this mean?

[apetresc]

Seems to be resolved now. My `brew update` works again.

[artyom]

ChatGPT, rotate my certs

[slowmovintarget]

Why? Don't most cloud providers have auto-renewing certs now?

[pmontra]

It needs a plugin for that.

[jjice]

Well I'm kind of just waiting on PRs for the rest of the day today and it's a Friday, so I'll consider this a modern equivalent of https://xkcd.com/303/

[GOATS-]

It's back now!