GitHub partial outage

[theletterf]

I was getting crazy thinking that there was something wrong with my SSH keys all of a sudden. Thanks $DEITY it's just GitHub.

[no_wizard]

Same. I reflex replaced mine thinking it needed to be. Glad its working now though

[arnvald]

I’m old enough to remember when GitHub was on main page due to a cool feature they added, now they just end up here when it stops working

[alentred]

If I remember it well, every once in a while a new cool feature was also breaking stuff, doubling the chances of getting to the top page here. But truth being told, GitHub was fixing those at light speed too and it was very interesting to follow their progress. Their delivery pipeline (per branch, deliver when ready, etc.) sounded very much innovative by then and I think inspired many people.

[numbsafari]

Anyone using GitLab have any insight on how well their operations are running these days?

We originally left GitLab for GitHub after being bit by a major outage that resulted in data loss. Our code was saved, but we lost everything else.

But that was almost 10 years ago at this point.

[kaishiro]

We use GitLab on the daily. Roughly 200 repos pushing to ~20 on any given day. There have been a few small, unpublished outages that we determined were server side since we have a geo-distributed team, but as a platform seems far more stable than 5-6 years ago.

My only real current complaint is that the webhooks that are supposed to fire in repo activity have been a little flaky for us over the past 6-8 months. We have a pretty robust chatops system in play, so these things are highly noticeable to our team. It’s generally consistent, but we’ve had hooks fail to post to our systems on a few different occasions which forced us to chase up threads until we determined our operator ingestion service never even received the hooks.

That aside, we’re relatively happy customers.

[boilerupnc]

Not sure what specific operational services are of interest - but here's a link to their historical service status [0]

[0] https://status.gitlab.com/pages/history/5b36dc6502d06804c083...

[geoffbp]

We’re using gitlab, loads of issues and outages, we want to go to github

[colesantiago]

No issues on GitLab.

Haven't seen any outage from GitLab in like, ever.

[ctkhn]

My org hosts it on prem, and while I don't like the way pages are organized for projects, I only really interact with the PR page and that is laid out well. Most of my interaction with git is happening from my terminal anyway so ¯\_ (ツ)_/¯

[arccy]

Your weekly reminder to take a break

[hoherd]

I still can't pull new branches even though the incident says it's resolved. I don't think my boss would be happy with me taking a break this long... but what else can I do when our business uses GH?

[ecshafer]

Github is owned by Microsoft, so this is a pretty small time indie operation, you need to give them a break.

[isodev]

I bet Microsoft is sad not because people can’t push, but because the training data for Copilot has slowed down.

PS: None of our 40+ engineers felt anything, our self hosted Forgejo is as snappy as ever.

[cube00]

Not replacing the CEO suggests they aren't focusing on it as much as they were.

[lysace]

Just your casual $3.8T company.

There were so many severe Github Actions outages (10+ ?) in the past year. Cause: Migration to the disaster zone also known as Azure, I assume. Most of them happened during (morning) CET working hours, as to not inconvenience the americans and/or make headlines.

Money doesn't buy competency. It's a long-term culture thing. You can never let go on maintaining competency in your organization. It rots if you do. I guess Microsoft did let go.

[JLCarveth]

This sure does seem to happen a lot

[prymitive]

Speaking of GitHub issues if you go to Insight->Traffic in your repo you’ll most likely see this banner:

“ Referring sites and popular content are temporarily unavailable or may not display accurately. We're actively working to resolve the issue.”

It’s been like that for months now with no sign of anyone working on it. They just don’t care about user experience anymore.

https://github.com/orgs/community/discussions/173494

[vaindil]

Hey, I'm the GitHub employee who's working on fixing that right now. The service powering those stats is _ancient_, and it fell over back in September. It's taken longer than I hoped to get a replacement working, but it should be fixed within the next couple of weeks, fingers crossed.

[embedding-shape]

Speaking of "temporarily unavailable but it's actually forever", I've been wanting to get into Fallout and Starfield modding, so been waiting for their official wiki to come out of maintenance mode. I think I first tried to access it when Starfield launched (September 2023), and still today it is "currently down for backend maintenance". https://wiki.bethesda.net/

[_heimdall]

I really do feel for those hubbers that are still working on this years into the Microsoft era. GitHub was an excellent product and from what I hear it was an excellent culture too. I can only assume the culture has eroded similarly to the product itself as Microsoft has finally begun integrating the org into the slower moving machine that is MS.

[contravariant]

Ah that was why. Oh well, I just needed to get the code to the server, so I didn't really need Github anyway.

[pfyra]

Coincidentally, Azure Devops was also missing the ssh keys earlier today, both in the web ui and for ssh login.

[spockz]

Well, github is moving to Azure and they are consolidating systems. No surprise there.

[danfritz]

Related to the recent announcement they are moving to Azure?

[drcongo]

Oh no. I look forward to watching my browser redirect 40 times on every attempted page load.

[the_af]

Wow. It wasn't already running on Azure? What was it (or is it) running on?

[bob1029]

https://news.ycombinator.com/item?id=45517173

[stackskipton]

Doubt it. I'm Ops person on Azure, while they just had terrible outage recently, they tend to be as stable as any other cloud provider and I haven't had many issues with Azure itself compared to whatever slop the devs are chucking into production.

[Fokamul]

Not Sharepoint? What a bummer.

[nkzd]

I thought my SSH keys were revoked, whew.

[coffeebeqn]

Just started to replace mine when I saw someone post a message about GitHub

[dustfinger]

Why does the main page show all green when there is an ongoing incident? All green here -> https://www.githubstatus.com/

[zamalek]

This is normal for Microsoft. It's as though status is owned and controlled by either marketing or accounting, not engineering.

[FrostKiwi]

Very Microsofty. I'm still furious with how they handled the global ban on updating CDN (Frontdoor) files until last week and not properly communicating it. https://www.reddit.com/r/AZURE/comments/1on0ung/azure_status...

[gkoberger]

It's marked as resolved for some reason

[dustfinger]

ahh, you are right. I am blind.

[gunalx]

Yep. Was using github for oauth on a petproject of mine. Got the unicorn, and was considering takingthe break, or just etting up something else. Seems to be running again for me now though.

[marak830]

Huh could this be why I can't login and pushing packages says my account is banned?

[rvz]

Looking forward to the postmortem.

Are they using AI agents this time to resolve the outage? Probably not.

But this time, there is no CEO of GitHub to contact and good luck contacting Satya to solve the outage.

[stuffn]

The postmortem will be simple since Github goes down so consistently every week you can almost use it as an alternative timekeeping system.

[carlyai]

thought i was going crazy

[whoknowsidont]

Another outage brought to you by Azure.

[fishgoesblub]

Must be a day ending in Y.

[wavemode]

It's possible that Microsoft buying GitHub was a large-scale psyop intended to reduce the productivity of the competition.

Any time their startup competitors are making too much progress they can just push the "GitHub incident" button and slow everyone down.

[grepfru_it]

We used to obsessively care about 500s. Like I would make a change that caused a 0.1% spike in 500s and I would silently say I'm sorry to the folks who got the unicorn page.

I'm not sure the new school cares nearly as much. But then again this is how companies change as they mature. I saw this with StubHub as well.. The people who care the most are the initial employees, employee #7291 usually dgaf