GitHub Incident

[bakje]

Perhaps the gemini-cli bot arguing with itself is taking its toll

https://github.com/google-gemini/gemini-cli/issues/16750

[MattIPv4]

https://github.com/google-gemini/gemini-cli/issues/16723 is even worse, GitHub shows `5195 remaining items` in the collapsed timeline.

[lol768]

Jeez, what a mess. Some of those issues have over 5000 events on them.

I really hope that didn't send emails out to people.

[pdimitar]

I could not resist to put my sarcastic comment about RAM price increases serving a good cause in there.

[omoikane]

Maybe the bots need rule of ko.

https://en.wikipedia.org/wiki/Rules_of_Go#Ko

[embedding-shape]

Haha, reminds me off bringing down office mail servers by accidentally creating loops of emails back in the day... What is old is new again, but this time with probabilities :)

[johnisgood]

Wonderful, lmao.

[nullfish]

I suspect the migration to Azure is continuing to go well

[rvz]

Yes indeed. 6 years of non-stop outages across the platform every month.

Even self-hosting would have been more stable than sitting on GitHub as predicted more than half a decade ago. [0]

Now there is no 'CEO of GitHub' to contact this time (Satya does not care).

[0] https://news.ycombinator.com/item?id=22867803

[ascendantlogic]

This feels more like Copilot-as-platform-engineer to me

[someguyiguess]

I did not come to hacker news expecting comedy gold but you have done it my friend!

[corvad]

Github's recent reliability has honestly been abysmal. Not surprised.

[ferguess_k]

Unless some major customers are moving away, I don't think they are going to seriously care about it.

[jbverschoor]

Good thing git is a distributed system

[dgxyz]

Virtually no one knows how to do anything with it outside of github.

[TZubiri]

True, workers can still commit to their local git.

I've been looking into having a separate git server that we can commit to and add plain ole git hooks to, and just having it be synced with github as a clone.

[nine_k]

Git is!

PRs and code review are not. CI/CD is not.

I mean, there are solutions, but none of them seems to have a large enough mindshare and efficiency. (Even though Github's code review tools are pretty spartan.)

[howToTestFE]

If GH has an issue, it seems to always be around 4pm or 5pm GMT. I'm starting to think that i should avoid any planned production releases around this time.

[tapoxi]

helm repo add gitlab https://charts.gitlab.io/ && helm upgrade --install gitlab gitlab/gitlab

I did this in 2019, it avoided so many headaches. CI is better too since there's a nice clean mapping of build -> pod for everything and I can just exec in if something's borked.

[odie5533]

Things would have to get really bad before I considered managing my own repositories. Trading someone else's headaches for my own.

[nottimbo]

Microsoft, it's time to hire some SREs.

[arm32]

We did hire some, boss! Soshie, Vizzy and Dexter. They're AI, but they're supposed to be way better than a human SRE. At least that's what the Sintra salesguy told us.

[lenerdenator]

Why hire anyone to fix a problem when you can make an AI agent to "fix" it, tell investors about it to pump the price, and not fix anything knowing that you have a monopoly?

[VirusNewbie]

Microsoft doesn't pay well enough to attract good SRE talent.

[aruggirello]

Clippy to the rescue! :-)

[ferguess_k]

Yes we did hire SREs, unfortunately they are in another continent and they only know how to pull others into the chat. We also have some AI too, do you want to try them? They are pretty good SREs, one of them wrote 100K lines of code in a week while another one reviews every line along the way. It was fantastic! Fantastic!! FANTASTIC!!!

OK I have no idea about MSFT SREs, just to be /s.

[andrewinardeer]

Days since last GitHub incident: 0.2

[imglorp]

14 incidents this month. So far.

[postexitus]

I believe it is an Azure outage or some type of MS service - everything on Azure is down.

[ctxc]

My az services seem to be up.

[zxcvasd]

having no issues on azure here, seeing no azure incidents on the status page or any of my admin panels

[toephu2]

This is why companies should host their own source code on-prem.

[MadameMinty]

Angry unicorns seem to be over.

[phtrivier]

Fixed in about 30m to an hour.

Definitely annoying, but I'll try the hot take that, contrary to popular belief, GH is not critical infrastructure - or so I hope.

Please tell me no part of the Ukrainian air defense system depends on a gh action hook.

[eddd-ddde]

You've heard of infrastructure as code, now presenting air strikes as code!

Need a new secret offensive operation? Create a new JSON file with the coordinates, make a merge request and get Commander approval, merge it, and our new proprietary GitHub action runner will deploy a drone in seconds!

[ares623]

When millions of man-hours are lost waiting for your service to be back up, I think that deserves a bit of resiliency.

[vaylian]

It's not critical, but there's still a lot of reliance on it.

It's also the only reason why I still need IPv4.

[NewJazz]

The status page says things are still not fixed.