GitHub Actions Incident 29.3

[temp_account_32]

Funnily enough, GitLab is also melting down at the moment, with pipelines not running and pull requests not functioning:

https://status.gitlab.com/

[bencevans]

Also causing issues due to a change that's made release source tars change [1] (changing hash), so build systems are rejecting [2].

1: https://gitlab.com/gitlab-org/gitlab/-/issues/402616 2: https://github.com/microsoft/vcpkg/issues/30481

[midasuni]

My on prem git lab has been fine with it’s pipelines, and given it’s impossible to have an on prem uptime higher than the cloud what you say can’t be true.

[kingds]

what's a pull request?

[rvz]

Once again, just two days ago [0], the whole of GitHub went down, after the RSA key leakage and the certificate key expiry on its user facing site.

It is also apparent that GitHub Actions has chronically been struggling to operate normally for at least once a month for years.

There is no question that GitHub has been more unreliable than if you were to use a self-hosted GitLab or Gittea instance yourself as I said before [1].

[0] https://news.ycombinator.com/item?id=35325850

[1] https://news.ycombinator.com/item?id=22867803

[carlmr]

Gittea is great but it doesn't replace GHA, only the rest of GitHub.

[chatmasta]

At least we don't need to scroll too far back in our comment history to copy/paste our arguments from the thread two days ago.

[jacobsenscott]

Nobody's stock goes down when actions go down. Nobody's stock goes up when actions are working. But everyone's stock goes up when you have mass layoffs. Working as designed.

[riffic]

not sure what the SLA is on Actions but outages are a regular occurrence with these kinds of systems and are incredibly expensive to move to the next 9 of availability.

it's certainly a risk you'll need to evaluate when planning your desired build process.

[nimbius]

how is it in just five years microsoft has managed to pedal this once vibrant and bustling community of developers and creatives into a roaring dumpster fire of sketcky GPL breaking copilot AI and endless seemingly random outages.

https://www.githubstatus.com/history

github has had 55 outages in 3 months. thats nearly an outage every two days.

the last six months of 2022 had 74 outages. In many shops thats tangibly worse than what their local greybeard Linux admin maintains.

arguments against spinning up my own gitlab/gitea/jenkins/whatever in podman under systemd are starting to ring pretty hollow lately.

[_gabe_]

> how is it in just five years microsoft has managed to pedal this once vibrant and bustling community of developers and creatives into a roaring dumpster fire of sketcky GPL breaking copilot AI and endless seemingly random outages.

I mean Github Actions was released 5 years ago[0]. I imagine the infrastructure for actions is more susceptible to outages than the fairly simple features Github offered previously. It makes sense that the number of outages would increase with the additional complexity in the infrastructure.

[0]: https://resources.github.com/devops/tools/automation/actions...

[xxpor]

comments like these are what end up incentivizing companies to hide outages and report all green all the time

[deltaci]

this is already the third time github actions is down this week at wednesday morning

[mdaniel]

maybe they have a "no deploy on Friday" rule :-D

[web3-is-a-scam]

Github is so crappy now, it feels like something is always wrong with it. Thanks Microsoft.

[sithlord]

Wonder if these were managed by a defunct team from India?

[racl101]

Oh I get it.

[touches nose]

[riffic]

what is the "29.3" in the title supposed to represent? is that supposed to indicate a date of March 29th? I do not see a reference to this on the incident page itself.

[nurgasemetey]

It is because titles will be duplicated on search if 29.3 is not added

[gxt]

At what point are organisations going to ask themselves wether this is intentional or not. Your velocity is disrupted by a likely competitor. I'd move out.

[aaomidi]

I don’t have an opinion on it is or not but layoffs do have a significant impact on that and it’s generally something that’s pretty impossible to study.

[jacooper]

Or simply just self host actions?

[usrme]

Azure is also having somewhat widespread issues at the moment, so I'd venture to guess that these two are also linked.