Days since last GitHub incident

[cedws]

I found out when Actions started failing again for the Nth time this month.

The internal conversation about moving away from Actions or possibly GitHub has been triggered. I didn't like Zig's post about leaving GitHub because it felt immature, but they weren't wrong. It's decaying.

[hinkley]

If you consider that an American maintainer was cheesed off enough to move an entire project off GitHub two days before Thanksgiving then the tone of the original post was completely in line with the energy involved.

Anger is a communication tool. It should absolutely be used when boundaries are being violated. Otherwise you’ll get walked all over.

[YetAnotherNick]

Idk, if being bad is the reason for leaving Github Actions, I think people would have left it ages ago. It stuck not because it is better than competitors but because it is included in the Github plans. It's decaying implies that it has somehow became worse, in fact it was one of the worst implementation to start with.

[IgorPartola]

The Zig post has since been updated and the objectionable parts have been removed. I think we can put that part to rest.

[DetroitThrow]

Combined with security concerns, this made us reconsider even our self-hosted GH Actions last month.

GH Packages is something we're extricating ourselves from after today too. One more outage in the next year and maybe we get the ammunition to move away from GH entirely.

It's still hard to believe that they couldn't even keep the lights on on this thing.

[stefan_]

GitHub has seem to come under the same management as VSCode, everything has to be made AI and that is the only priority. It's like the Google+ of old but stupider.

[toastal]

This is why I keep encouraging folks to a) have a mirror & b) make sure their tools automatically pick up the mirrors.

I recently got mirror support upstreamed into Nixpkgs for fetchdarcs & fetchpijul which actually work on my just-alpha-released pinning tool, Nixtamal <https://darcs.toastal.in.th/nixtamal/trunk/README.rst>, for just this sort of thing.

[barbazoo]

I envy you. Most of us struggle to get the resources to make our actual customer facing applications resilient, let alone our build pipeline.

[maccard]

That's great for the repository, but what about if you're using ghcr, actions, issues, or copilot?

[matheusmoreira]

I've been getting some weird cryptocurrency spam notifications on GitHub and they can't be cleared for some reason. Blue dot is gonna be there forever apparently. Some users made an issue out of it but nobody cared to fix it.

[cassidoo]

This should have been fixed here: https://github.blog/changelog/2025-12-04-notifications-trigg...

Are you still seeing it, would you mind checking? Our team will get on it if so.

[dennis-tra]

Had the same issue that the blue dot won’t disappear. I was able to clear the dot with:

gh api notifications -X PUT -F last_read_at=2025-10-06T00:00:00Z

Just change the date to today. I also got that line from a gh issue somewhere - maybe it was the same issue that you’re referring to.

[fastball]

Same happened to me. You can clear it via the CLI, hilariously.

```

gh api notifications\?all=true | jq -r 'map(select(.unread) | .id)[]' | xargs -L1 sh -c 'gh api -X PATCH notifications/threads/$0'

```

[ashton314]

Here’s how you can clear it:

https://github.com/orgs/community/discussions/174310#discuss...

I had the same issue too, and this was the only thing that fixed it for me.

[Aperocky]

Once GitHub fully migrates to Azure, it should be known as GitHub 11.

[ragall]

Github 3.11 for Workgroups.

[aranw]

GitHub Vista Cloud Edition

[Gazoche]

I think you mean "365 Code Copilot".

[OptionOfT]

GitHub Actions is a good example of systems thrown together that at face value have something to offer until they get put under stress.

Just now I found:

    * a job that's > 1 month old, still running
    * another job that started 2 hours ago that had 0 output
    * a job that was marked as pending, yet I could rerun it
    * auto-merges that don't happen
    * pull requests show (1), click it, no pull requests visible

Makes me wonder in how many places state is stored, because there is some serious disconnect between them.

[Nextgrid]

That's just post-Windows 8 Microsoft quality for you. Every product has been like that - looks "ok" on the outside (in reality it looks shit, but at least that's intentional), but the second you dig deeper and start using it you get all kinds of paper cuts like that.

[geophph]

Is GitHub deployed using GitHub Actions?

[Oakwhisper]

I was talking with some GH sales/marketing engineers last month and they said it deploys with actions, but they have a custom deploy queue

[samcheng]

Missed a chance to put this in meme format e.g. https://imgflip.com/memetemplate/439302803/Days-without-acci...

[llbbdd]

I've gotten accustomed lately to spending a lot of time in the Github Copilot / agent management page. In particular I've been having a lot of fun using agents to browse some of my decade-old throwaway projects; telling it to "setup playwright, write some tests, record screenshots/videos and commit them to the repo" works every time and it's a great way to browse memory lane without spending my own time getting some of these projects building and running again.

However this means I'm now using the Github website and services 1000x more than I was previously, and they're trending towards having coin-flip uptime stats.

If Github sold a $5000 box I could plug into a corner in my house and just use that entire experience locally I'd seriously consider it. I'm guessing maybe I could get partway there by spending twice that on a Mac Pro but I have no idea what the software stack would look like today.

Is there a fully local equivalent out-of-the-box experience that anyone can vouch for? I've used local agents primarily through VSCode, but AFAIK that's limited to running a single active agent over your repo, and obviously limited by the constraints of running on a single M1 laptop I currently use. I know at least some people are managing local fleets of agents in some manner, but I really like how immensely easy Github has made it.

[Aurornis]

None of the open weights models you can run locally will perform at the same level as the hosted frontier models. Some of them are becoming better, but the step-down in output quality is very noticeable for me.

> If Github sold a $5000 box I could plug into a corner in my house and just use that entire experience locally I'd seriously consider it. I'm guessing maybe I could get partway there by spending twice that on a Mac Pro but I have no idea what the software stack would look like today.

Right now, the only reasons to host LLMs locally are if you want to do it as a hobby or you are sensitive about data leaving your local network. If you only want a substitute for Copilot when GitHub is down, any of the hosted LLMs will work right away with no up front investment and lower overall cost. Most IDEs and text editors have built-in support for connecting to other hosted models or installing plugins for it.

> I know at least some people are managing local fleets of agents in some manner,

If your goal is to run fleets of agents in parallel, local LLM hosting is going to be a bottleneck. Familiarize yourself with some of the different tool options out their (Claude Code, Cline, even the new Mistral Vibe) and sign up for their cloud API. You can also check OpenRouter for some more options. The cloud hosted LLMs will absorb parallel requests without problem.

[colechristensen]

An NVIDIA DGX Spark is $4000, pair that with a relatively cheap second box to run GitLab in the corner and you would have pretty good local AI inference setup. (you'd probably have to write a nontrivial amount of software to get your setup where you want)

The local models are just right on the edge of being really useful, there's a tipping point to where accuracy is high enough so that getting things done is easy vs models getting continuously stuck. We're in the neighborhood.

Alternatively, just have local GitLab and use one of the many APIs, those are much more stable than github. Honestly just get yourself a Claude subscription.

[bastardoperator]

They do, it's called GHES.

https://docs.github.com/en/enterprise-server@3.19/admin/over...

"GitHub Enterprise Server is a self-hosted version of the GitHub platform"

[Lapalux]

Oh nice - I'm literally playing around with a site to detect outages for major provides (AWS/cloudflare/github) based on social media/HN posts

[Lapalux]

hehe - thank you for helping me Github - https://imgur.com/a/0KqmKpU

[TuxPowered]

It should always be at 0, because GitHub is unreachable over IPv6, which in 2025 should be considered an incident.

[tonymet]

mobile adoption is high, desktop (residential and corporate) is still quite low.

I'm a big advocate for github to add ipv6 support , but let's not pretend it's critical for their business.

[doubled112]

I haven’t had a residential ISP that provided IPv6 yet.

[fragmede]

"yeah but when I turn on ipv6 everything breaks"

[locusofself]

The Primagen video about the bash scripts underpinning github actions runner was crazy. I'm a half-assed programmer at best and I don't even think I would make some of those mistakes.

[_def]

woah this time i even caught it before the status page reported something - i thought they were rate-limiting me.

[rienbdj]

If GitHub actions break I now assume it’s them and not me. GitHub needs to work on stability ahead of AI features.

[laurmaedje]

It seems to have started slowly. For me, Github releases have failed to serve requests for hours already.

[doublerabbit]

I've not been able to browse any repo sources without the Unicorn for the past few hours.

[lawlessone]

The amazing part about this is the page even works when i'm offline.

[Ozymandias-9]

When I opened the link, I just laughed for 5 mins straight

[queuebert]

At this point, is there any downside to switching to GitLab?

[loloquwowndueo]

What’s gitlab?

(Snarky way of saying: GitHub still has huge mindshare and networking effects, dealing with another forge is probably too much friction for a lot of projects)

Not that GitHub doesn’t suck…

[JackSlateur]

Is it any better ?

We had that last year, with the full premium stuff ("pay as much as we can" mindset)

Please see this: a basic feature, much needed by lots of people (those who are stuck on azure ..): https://gitlab.com/gitlab-org/gitlab/-/issues/360592

Please read the entire thread with a particular attention to the timeline

[bdcravens]

If escaping downtime is your goal, then you should aim for a service with less downtime than Github. (they're roughly the same, with Gitlab having a slightly higher percentage of "major" outages)

[richardwhiuk]

Is the uptime any better?

[ZeroConcerns]

This is a bit... low-effort, isn't it? I'd at least expect a video of an exasperated Github user walking up to the '# days since the last GitHub incident' board, sliding out the '1' or '2' card, and replacing it with a '0'.

I mean, that joke is as old as the universe (heck, in the brief period that I worked in an office, decades ago, I had a "# days since the last person asked a stupid question" sign to enact the exact same gag)...

[old_bayes]

How can we add AI to this perfectly functional product?

[rob74]

Or an octocat standing in front of the board, holding cards from 0 to 7 in its tentacles (with the rest lying on the ground) and looking at them quizzically?

[venturecruelty]

I used to have a magic 8-ball that people could use when they wanted me to debug their code for them. I think it was broken, though; it kept saying "Outlook good". Must've been a Microsoft magic 8-ball.

[Tade0]

I was half-expecting the "days since last" meme - the one with a person smiling awkwardly while clapping with a large four-digit counter in the background showing only zeroes.

[llbbdd]

I don't use Github Pages so I might be wrong but IMO I think at least part of the joke is that its URL betrays that it's a completely static site.

[asplake]

Very low effort. Couldn’t read the text on my iPhone without zooming in. I nearly mistook it for a blank page!

[udev4096]

[flagged]

[GaryBluto]

Friendly reminder to stop saying friendly reminder when what you're saying isn't a reminder.

[llbbdd]

I don't work at Github but I'd read here recently that they've been undergoing a herculean migration from whichever cloud provider they were on to Azure since their Microsoft acquisition, and that it coincides with an increase in outages. I'm guessing that the solution here was probably just to not do that and it's too late.

[bob1029]

GitHub was already done years ago. The ideal solution was in hand as of ~2020. Nearly every release since then has brought some kind of regression.

[bdcravens]

It's just a metric, not "whining". Besides, if complaining about companies (whether it's Github/Microsoft, Anthropic, Google, etc) without offering a solution is out-of-bounds, that probably knocks out 50% of the posts and comments on HN.

[blibble]

because Microsoft are known for listening to their customers?

this trivial bug fix took more than a year to be merged:

https://github.com/actions/runner/pull/3157

that bug likely ended up costing customers millions

[jpitz]

$Work pays for GitHub, so the implicit solution offered is "take my money and make your service reliable"

[tonymet]

This is pretty dishonest because some trivial service no one cares about will reset this counter .

[guywithabike]

Your definition of "trivial" is not everyone's definition of trivial.

[ferguess_k]

I guess none of us really needs those 9s, and even two 9s are just good enough. I even doubt whether *SOME* of the banking transactions really really really need those 9s too -- like, I don't really mind if 1 out of 100 credit payment doesn't go through so I have to do it again -- it does happen once for a while and I just swiped it again.

[tormeh]

GitHub has a container registry. That going down can cause pod start failure. I agree the source code probably doesn't need infinite nines, but the container registry is different.

[dec0dedab0de]

I had an ATM glitch out on me a few months ago, I tried again and it confiscated my card. I called, and they explained that it is the failure mode to prevent people modifying them while they're offline.

[0xdeafbeef]

Retry is fine, but imagine being unable to pay for something within 10 minutes in month. And 10m in 1M is 99.98% sla. So it depends