[johncessna]

Why did they update everything all at once?

[afavour]

I assume Crowdstrike is software you usually want to update quickly, given it is (ironically) designed to counter threats to your system.

Very easy for us to second guess today of course. But in another scenario a manager is being torn a new one because they fell victim to a ransomware attack via a zero day systems were left vulnerable to because Crowdstrike wasn’t updated in a timely manner.

[SoftTalker]

Maybe, if there's a new zero-day major exploit that is spreading like wildfire. That's not the normal case. Most successful exploits and ransom attacks are using old vulnerabilites against unpatched and unprotected systems.

Mostly, if you are reasonably timely about keeping updates applied, you're fine.

[johncessna]

In the context of the GP where they were actively treating a heart attack, the act of restarting the computer (let alone it never come back) in of itself seems like an issue.

[anonymous8888]

tfw you need to start staggering your virus updates in case your anti-virus software screws you over instead

[nikau]

Maybe those old boomer IT people were on to something by using different Citrix clusters and firewalling off the ones that run essential software...

[jmcgough]

Crowdstrike pushed a configuration change that was a malformed file, which was picked up by every computer running a the agent (millions of computers across the globe). It's not like hospitals and IT systems are manually running this update and can roll it back.

As to why they didn't catch this during tests or why they don't use perform gradual change rollouts to hosts, your guess is as good as mine. I hope we get a public postmortem for this.

[jboy55]

Considering Crowdstrike mentioned in their blog that systems that had their 'falcon sensor' installed weren't affected [1], and the update is falcon content, I'm not sure it was a malformed file, but just software that required this sensor to be installed. Perhaps their QA only checked if the update broke systems with this sensor installed, and didn't do a regression check on windows systems without it.

[1]https://www.crowdstrike.com/blog/statement-on-falcon-content...

[lr1970]

> Why did they update everything all at once?

This is beyond hospital IT control. Clownstrike (sorry, Crowdstrike) unconditionally force-updates the hosts.

[cyanydeez]

Likely because staggered updates would harm their overall security services. I'm guessing these software offer telemetry that gets shared across their clientele, so that gets hampered if you have a thousand different software versions.

[whydoyoucare]

My guess is this was an auto-update pushed out by whatever central management server they use. Given CS is supposed to protect your from malware, IT may have staged and pushed the update in one go.

[Groxx]

Auto-updates are the only reason something like this gets so widespread so fast.