GitHub Status – Incident on 2019-03-12

[msghacq]

I’ve been hearing from friends at GitHub and MS that things are on fire internally. The GitHub employees are in semi-revolt because they think Nat is integrating too fast with MS. Many people have quit or are being pushed out. It may actually be for the best because the post-founder/pre-buyout team was pretty mediocre. Nat is also trying to crack down on some of the more overt activism in the company. They do “eq” screenings for political fit during the interview process and he’s trying to get them to stop. They also have a brand new head of product and engineering. Product was taken from Jason Warner about two months ago and placed directly under Nat. Sounds really bad and I don’t expect much turn around.

[ambulancechaser]

> They do “eq” screenings for political fit during the interview process and he’s trying to get them to stop.

This sounds crazy to me. My last company would freak out if the dev team got together and discussed the person's politics and if we thought they should be hired or not based on that. I hope they do stop this.

The more i think about this the crazier it sounds. Imagine a Houston oil consultancy firm discussing a candidate who applied and asking "when we talked about lowering taxes, did the inflection in his voice sound real or feigned?". This is what an "old boys club" looks like.

[tenderlove]

Hi. I don't know of any fires, revolts, or crackdowns. GitHub is independent of MS, and Nat has worked hard to keep it that way. We do have org changes, but I don't things get "taken" so much as "we work together to figure out an org structure that will make us most productive and build a great product".

FWIW I'm very happy at work

[dgellow]

> They do “eq” screenings for political fit during the interview process

Wait, what? How is that even legal? Not sure what's the exact meaning of "eq screenings" but anything related to political affiliation should be out of any interview process. At least in Europe discrimination on political views or other belief is completely illegal (with some exception for the church in some countries, e.g: Germany).

[amaccuish]

Sorry but this a 2 hour old account, I think I'll take this with a pinch of salt and advise others to do the same.

[pojntfx]

Use Gitlab.

[justinjlynn]

Update 1: Info.

Update 2: Same info Again.

Update 3: The same info again phrased differently.

What's the point of giving regular "updates" when you don't add any information? Is it just to tick a managerial box somewhere?

[jressey]

If you're on a flight, do you not appreciate when the pilot updates you, even if he says "We are still investigating the maintenance issue?"

[Operyl]

Some clients will feel ignored, even if you don't regurgitate the same stuff over and over again while there's no new useful information to disseminate (or not had the time to properly write it up).

At my job, I wrote a small slack bot that'll ping the person who is dealing with the status page/customer impact, it'll ask "I'm posting an automated update in 15 minutes, would you like that or do you have a full size one?" That way, in the heat of a fire, nobody forgets about the client's status page updates. It works rather well.

[abathur]

This is probably SLA compliance stuff (stipulations for how quickly they have to provide notice of issues, how often they're obliged to provide updates, and how quickly they'll fix an issue). These stipulations, obviously, don't say much about the quality of the notice or updates.

So, yes--ticking a box (but a box that will matter for securing and maintaining contracts with larger organizations).

[Legogris]

This could be just a coincidence, but just now I also noticed HN was down as well as several global services (non-tech related stuff) we integrate with that are all timing out. Some deeper infrastructure issues at hand?

[mcrider]

Odd side effect -- I've received three duplicate 'Github Explore' emails about every hour this morning.

[rinchik]

Yep. Can't view newly open PRs, also random GH related failures in CI

[Vervious]

Travis CI git clone over http is borked as a result. https://www.traviscistatus.com/

[honopu]

I wasn't getting 2fa notifications either, haven't tried lately but switched to google authenticator as a precaution against this happening in the future.

[MuffinFlavored]

Does anybody else feel GH has a much higher than normal incident rate?

[yjftsjthsd-h]

Unless they under-report, we shouldn't need to feel anything; we can just check https://www.githubstatus.com/history and objectively look at frequency. Of course, some services do under-report; dunno if GH does or not.

[colinbartlett]

Depends on what you mean by normal but compared to many of the status pages I am tracking, GitHub has a pretty low incident rate (though an uptick very recently): https://statusgator.com/services/github

[ashelmire]

Nah, seems pretty normal. Compare to Netflix, Hulu, Heroku, etc, which have some outages nearly daily. This was just degraded performance for two hours, and the only more serious one I can recall was pretty recent.

[zilian]

Deploy to github pages delayed for more than an hour here

[elesbao]

that's a shitty incident report and a sad thread if there's this kind of activism internally while shit is on fire.

[lhorak]

Seem working alright... hmmm