GitHub Outage Currently Ongoing – Issues, PRs, Notifications

[yjftsjthsd-h]

Alright, cue the usual:

Monoculture bad

Microsoft bad

Git is distributed but all the things around it that we really need aren't

You can set up git to push to multiple remotes automatically

Nobody is actually using git in distributed mode

Did I forget anything?

[jddj]

The conspiracy about how much infrastructure has been going down this summer, and how it's probably interns / a state actor / people pushing to get things in production before performance reviews / people being on holiday / that we've reached peak complexity.

[davestephens]

The usual fluff about migrating away to Gitlab?

[HelloNurse]

I hope to see a trend towards "you can host a Git repository yourself, why do you depend on GitHub?".

[foobiekr]

Publicly visible outages on "pro" internet properties are actually a good thing because they help correct the views so commonly held by low-information executives about 5-nines and uptime. The more ammo people have, in aggregate, that "XXX never goes down" is a lie, the better off we will be as an industry.

[techntoke]

If GitHub actually had an open source version, then I could selfhost on infrastructure that I trust.

[SilasX]

What about "rookie mistakes by code artisans who are making it up as they go along"?

[jolmg]

> Git is distributed but all the things around it that we really need aren't

Can anyone recommend an issue tracker that is distributed with the repo?

[maxerickson]

The knowing meta-post.

And the meta-meta follow on.

[jmisavage]

I kind of wish HN wouldn't show these outages unless they're going on for multiple hours or at least hide them after they're back up. Usually by the time I see them and then check myself the site is already back up. Maybe I just need to read HN more often.

[djsumdog]

Yea, there are a lot of outage posts. It does kinda illustrate how many people depend on these core services though; and how little distributeiness or elasticity is built into the system.

Linux repos and ISO download sites have TONS of mirrors, sometimes contributed by companies, but also a lot of universities, government and public institutions.

Why is one of the largest core open source repositories centralized and private? Why are we so dependent on AWS, Google and others for large chunks of Internet services to be accessible?

There is a deeper question here than just the outages; but rather why they affect so many people.

[vallismortis]

I find the post-mortems helpful. I don't want to make the same mistakes with my infrastructure, and it gives me something to reference when, for instance, the upper management decides they want to migrate to one of these services.

[k_]

I like those, even when it's been down for one or two hours only, but the title should be edited so that we know the outage is over. I'd still prefer this thread to be replaced by a postmortem but I don't think there's one yet.

[jammygit]

> Maybe I just need to read HN more often.

Last night I wrote a little spreadsheet for myself showing how much time I spend doing various things. In particular, I for one do NOT need to read HN more often

[krzkaczor]

Events like this make me realize how much I rely on my CI/CD pipeline for working on new features, deploying etc. I am often too lazy to run E2E tests on my machine if I know that on CI it will take only a few minutes to be done.

[prepend]

I feel guilty because I’ve never installed GitLab’s CI process locally and have only run it for the past few years through git commits. I keep expecting someone to complain, but so far, so good.

I expect similar poor behavior testing GitHub’s actions because it’s so convenient to only run through the service.

It is important though that it’s possible to run without CI/cd. If it wasn’t available I would complain because of the risk of lock-in.

[olah_1]

A peer-to-peer (IPFS) stack for code collaboration.

https://radicle.xyz/

A Radicle project contains a git repository, plus the associated issues and proposals.

^ Neat project

[alexeiz]

Funny thing, I didn't even notice. In fact, being heavily reliant on Git and Github, I encountered maybe one or two issues a year when I couldn't push to Github because it was unavailable. In those cases, the progress can still be made locally, my workflow was barely affected. Of course, the deployment was impossible, but the downtime was never significant enough to worry or do anything about.

[smartmic]

Git is designed as distributed VCS, so an outage of a centralized server should not matter too much. So far, the theory. Using Github excessively is the way back to centralization, nowadays handing control over to Microsoft. Everybody who is concerned (probably not too much) should have a look into alternatives.

[dspillett]

Git itself is decentralised, but many (most? (all?)) of the workflows people build around it are not.

Individuals and teams who already have what they need in the local repos can continue to work through an outage of the VCS part of github, but at the points where they need to collaborate (merging each other's changes, issue tracking, etc.) the workflows break down. Yes you could share changes in a more distributed manner, or workaround the outage in other ways, but in reality people will stop and wait for the central repo to be available again. Also, pulling changes from my repo to yours directly to avoid that part of the downed service doesn't solve the issue tracker or CI manager also being down. The clue is in the name: git HUB.

That said, the issue is similar to how people see the results of aviation accidents: a jumbo going down takes out a lot of people but statistically, considering all air journeys, that is a lot less than the deaths that would happen due to equivalent car journeys. I suspect that everyone being a bit inconvenienced for a while when github/gitlab/other-centralised-service has issues like today doesn't add up to as much as the many small individual inconveniences they'd experience over time with their own local instances of, for example, gitlab, especially if including the routine maintenance time required which doesn't exist as much when using github, or gitlab managed instead of self-hosted - the outage just feels bigger than all the little problems because everyone is affected at once.

[reacweb]

There are so many ways we become dependent of github, that nobody will shame on us because we are impacted by a github outage. In this context, is it useful to protect against outages ? It may be cheaper to do nothing.

[csomar]

Yea because the alternative (hosting git yourself on your own servers) will have better availability and few issues (UX just came to mind!)

[scandox]

We're self hosting Gogs on a Hetzner VM and it's been a great experience. Of course it has less features. But it's simple and fast and I do about 10 minutes sys admin per quarter.

[enlyth]

Resolved, all systems operational, that was quick.

[fortran77]

If you don't the the "community" features of GitHub, we have had very good results with Amazon Code Commit. I've never experienced an outage, it's easy to manage, and very inexpensive.

https://aws.amazon.com/codecommit/

[u801e]

I see posts like this make the front page of hacker news every so often, but I don't recall seeing posts about things like making list outages for something like the LKML (Linux Kernel Mailing List) or an IRC network like Freenode.

[objclxt]

Perhaps because Freenode has about 0.25% the number of users of Github - ~100,000 vs 40 million?

[johnisgood]

> IRC network like Freenode

When was the last time ALL freenode servers were offline? I know about occasional netsplits but those do not affect all of their servers and freenode itself is still operational.

[xchaotic]

Justyna yesterday I was complaining about GitHub monoculture in the context of Atlassian sunsetting Mercurial. Outages like this are normal, but the problem is lack of good competitors.

[nickpsecurity]

The problem is bad design and/or integration of components. VMS clusters over leased lines went years without outages. Record was 17 years at a rail yard. They also did rolling upgrades across both different OS versions and CPU architectures. The methods are public if any competitors want to match or exceed them:

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na...

These modern cloud services using Linux ecosystem are doing rolling outages instead. I'm amazed people keep making them dependencies. Maybe their local systems using similar technology went down more often. Maybe this is an improvement. I'd still put mine on an OpenBSD or OpenVMS cluster if it was business-critical with lots of money on the line. I want it staying up, up, up. :)

[arghwhat]

There are quite a decent list of competitors, all of which have similar or sometimes better offerings. There are also plenty of fully free and open source solutions, should you want to do things yourself.

The Github monopoly is driven by its users.

[consolenaut]

I think you're right about a general move towards a Git monoculture (as opposed to a GitHub one), but there are dozens of great Git repo hosting/collaboration companies - GitLab, Bitbucket & SourceForge to name a few of the bigger ones

[cproctor]

I've been having trouble getting GitHub to send me a verification email (after adding a new email address) for over a week. Anybody else having this problem? I wonder if it's connected.

[knorker]

Also still ongoing: They don't have IPv6.

[kome]

it's time to switch to SourceForge.

[sergiotapia]

I've migrated all of our repos to codeplex.

[justinator]

What's a, "PR". (I'm being serious)

[janpot]

https://help.github.com/en/articles/about-pull-requests