[itzjacki]

A colleague of mine just came bursting through my office door in a panic, thinking he brought our site down since this happened just as he made some changes to our Cloudflare config. He was pretty relieved to see this post.

[arbuge]

Tell him it's worse than he thinks. He obviously brought the entire Cloudflare system down.

[mlrtime]

You joke and I think its funny, but as a junior engineer I would be quite proud if some small change I made was able to take down the mighty Cloudflare.

[sakisv]

Well, you can never be sure that he didn't:

https://www.fastly.com/blog/summary-of-june-8-outage

[nevf1]

It's also what was the cause of the Azure Front Doors global outage two weeks ago - https://aka.ms/air/YKYN-BWZ

"A specific sequence of customer configuration changes, performed across two different control plane build versions, resulted in incompatible customer configuration metadata being generated. These customer configuration changes themselves were valid and non-malicious – however they produced metadata that, when deployed to edge site servers, exposed a latent bug in the data plane. This incompatibility triggered a crash during asynchronous processing within the data plane service. This defect escaped detection due to a gap in our pre-production validation, since not all features are validated across different control plane build versions."

[itzjacki]

Oh don't you worry. We are very much talking about the global outage as if he was the root cause. Like good colleagues :)

[srmarm]

> May 12, we began a software deployment that introduced a bug that could be triggered by a specific customer configuration under specific circumstances.

I'd love to know more about what those specific circumstances were!

[Bloomy22]

I'm pretty sure I crashed Gmail using something weird in its filters. It was a few years ago. Every time I did something specific (I don't remember what), it would freeze and then display a 502 error for a while.

[CableNinja]

Damn, imagine being the customer responsible for that, oof

[Freak_NL]

Is there a word for that feeling of relief when someone else fucked up after initially thinking it was you?

[spamizbad]

What’s funny is as I get older this feeling of relief turns more like a feeling of dread. The nice thing about problems that you cause is that you have considerable autonomy to fix them. Cloudflare goes down you’re sitting and waiting for a 3 party to fix something.

[jspash]

The problem is, I still get the wrong end of the stick when AWS or CF go down! Management doesn't care, understandably. They just want the money to keep coming in. It's hard to convince them that this is a pretty big problem. The only thing that will calm them down a bit is to tell them Twitter is also down. If that doesn't get them, I say ChatGPT is also down. Now NOBODY will get any work done! lol.

[shortrounddev2]

When I'm debugging something, I'm not usually looking for the solution to the problem; I'm looking for sufficient evidence that I didn't cause the problem. Once I have that, the velocity at which I work slows down

[jpmonette]

phewphoria

[mcphage]

Maybe this isn’t great, but I get a hint of that feeling when I’m on an airplane and hear a baby crying. For a number of years, if I heard a baby crying, it was probably my baby and I had to deal with it. But now my kids are past that phase, so when I hear the crying, after that initial jolt of panic I realize that it isn’t my problem, and that does give me the warm fuzzies. Even though I do feel bad for the baby and their parents.

[bookofjoe]

The German word “schadenfreude” means taking pleasure in someone else’s misfortune; enjoyment rather than relief.

[Rooster61]

Schadenfriend?

You gain relief, but you don't exactly derive pleasure as it's someone you know that's getting the ass end of the deal

[stonecharioteer]

It's close enough to Schadenfreude but not really.

[StanAngeloff]

Schadenfreude

[cromka]

vindication?

[hoistbypetard]

schadenfuckup

[nrhrjrjrjtntbt]

The company where this colleague works? Cloudflare.

[sefke]

I woke up getting bombarded by multiple clients messages of sites not working, I shitted my pants because I've changed the config just yesterday. When I saw the status message "cloudflare down" I was so relieved.

[disconnection]

Good that he worked it out so quick. I recently spent a day debugging email problems on Railway PaaS, because they silently closed an SMTP port without telling anyone.

[bamboozled]

How do we know your colleagues changes didn't take down Cloudflare though?

[itzjacki]

Good point. We should probably assume they did, until proven otherwise.

[0xblinq]

Do you guys work at Cloudflare? Do you mind reverting that change just in case?

[ants_everywhere]

Chances are still good that somewhere within Cloudflare someone really did do a global configuration push that brought down the internet.

When aliens study humans from this period, their book of fairy tales will include several where a terrible evil was triggered by a config push.

[0xblinq]

Plot twist: They work at Cloudflare

[dcjdfvk]

Even pornhub is down becuase it uses clouflare.

[carlos_rpn]

Is Cloudflare being down the work of conservative hackers and the rest of the internet is just collateral damage?

[belter]

Wait for the post mortem ... It is a technical possibility, race condition propagates one customer config to all nodes... :-)

[raxxorraxor]

Did your colleague perhaps change the Cloudflare config again right now? Seems to be down again.

[theoldgreybeard]

You should tell him his config change took down half the internet.

[NitpickLawyer]

You missed a great opportunity to dead-pan him with something like "No, Bob, not just our site, you brought down the entire Internet, look at this post!"