Really interesting to me that none of the commentators I've seen in the press have even hinted that maybe an OS that requires frequent security patches shouldn't be used for infrastructure in the first place. For just one example, I've seen photos of BSODs on airport monitors that show flight lists -- why aren't those built on Linux or even OpenBSD?
Security is not a feature that can be layered on. It has to be built in. We now have an entire industry dedicated to trying to layer security onto Windows -- but it still doesn't work.
The vendor who makes the software has always written for Windows (or in reality, wrote for either DOS or OS/2 then transitioned to NT4). History, momentum, familiarity, cost, and ease of support all are factors (among others, I'm sure).
Security is a process, not a product.
And yes, distros require frequent updates, though more to your point, you can limit the scope of installed software. I'm sure airport displays don't need MPEG2, VP1 and so on codecs, for instance.
It's also important to remember that there is a lot of 'garageware' out there with these specialized systems. Want SAML/OIDC support? We only support LDAP over cleartext, or Active Directory at best. Want the latest and greatest version of Apache Tomcat? Sorry, the vendor doesn't know how to troubleshoot either, so they only "support" a three year old vulnerable version.
Ran into that more than a few times.
Given the hypothesis of what caused the BSOD with Crowdstrike (NUL pointer), using a safe language would have been appropriate -- it's fairly easy in this case to lay the blame with CS.
Microsoft supplies the shotgun. It's the vendors responsibility to point it away from themselves.
> an OS that requires frequent security patches
> Security is not a feature that can be layered on. It has to be built in
This is a common misunderstanding, an OS that receives frequent security updates is a very good thing. That means attention is being paid to issues being raised, and risks are being mitigated. Security is not a 'checkbox' it's more of a neverending process because the environment is always in a state of flux.
So to flip it, if an OS is not receiving updates, or not being updated frequently, that's not great.
What you want is updates that don't destabilize an OS, and behind that is a huge history and layers of decisions at each 'shop' that runs these machines.
Security is meant to be in layers and needs to be built in.
> but it still doesn't work.
It does work because the 'scene' has been silent for so long, but what we as humans notice is the incident where it didn't.
Airport staff need to be able to support them. Not HN types.
Most people know how to use a windows computer.
Most IT desktop support knows how to use and manage windows. Even building facilities folks can help support them.
Microsoft makes it easy to manage a fleet of computers. They also provide first party (along with thousands of 3rd parties) training and certifications for it.
For many CTO/CISO it is more important to have a good target to shift responsibility when things go awry than to have a reliable/secure system. A Big Brand is a good target, an open-source project like OpenBSD is not. I doubt any CTO will be fired for choosing Widnows+CrowdStrike (instead of Linux/BSD) despite many million losses.
"Nobody ever gets fired for buying IBM" is as true as ever at least in the corporate world.
Or don't use an OS at all. We need to think about minimizing the use of software in critical infrastructure. If that means less efficiency because you have to be near something to maintain it then so be it. That would be good for jobs anyway.
To pick on your airport example a bit… all of the times I’ve gotten to enjoy a busted in-seat entertainment system, I’ve found myself staring at a stuck Linux boot process. This goes well beyond the OS.
Those sorts of things just need to boot to a web browser in full screen with some watchdog software in the background, launching from a read only disk (or network image). Get a problem, just unplug it and plug it back in. Make it POE based so you can easily do it automatically, stick them on a couple of distros (maybe even half on bsd, half on linux, half using chrome, half on firefox)
I'm sure we've all heard the phrase "We're a Windows shop" in some variation.
I understand the reasons for it, and why large, billion dollar companies try to create some sort of efficiency by centralising on one "vendor", but, then this happens.
I don't know how to fix the problem of following "Industry Trends" when every layer above me in the organisation is telling me not to spend the time (money) to investigate alternative software choices which don't fit into their nice box.
I read the T&C of this CrowdStroke garbage and they have the usual blurb about not using it in critical industry. Maybe we just charge & arrest the people that put it there and this checkbox-software mess stops real quick.
>Really interesting to me that none of the commentators I've seen in the press have even hinted that maybe an OS that requires frequent security patches shouldn't be used for infrastructure in the first place.
Nobody's commenting on that because it's the wrong thing to focus on.
1) This fuckup was on CrowdStrike's Falcon tool (basically a rootkit) bricking Windows due to a bad kernel driver they pushed out without proper hygiene, not on Windows's security patches being bad.
2) Linux also needs to get patches all the time to be secure (remember XZ?) It's not just magically secure by default because of the chubby penguin but is only as secure as it's most vulnerable component, and XZ proved it has a lot of components. I'd be scared if a long period goes by and I see no security patches being pushed to my OS. Modern software is complex and vulnerabilities are everywhere. No OS is ever bug-free and fully bullet proof in order to believe it can be secure without regular patches. Other than TempleOS of course.
The lesson is whichever OS you use, don't surrender your security to a single third party vendor who you now have to trust with the keys of your kingdom as that now becomes your single point of failure. Or if you do be sure you can sue them for the damages.
every year multiple times per year there's reports of Microsoft Windows systems having either mass downtime or exploitation.... it's kind of amazing that critical systems would rely on something that causes so much frustration on a regular basis.... I've been running systems under Linux and Unix for decades and never had any down time... so I don't know I mean it's nice to know that Linux is pretty solid and always has been the worst that's ever happened has been like a process that might go down during an upgrade, but never the whole system.
Linux is vulnerable too (but not as vulnerable as windows of course) it’s just not targeted by hackers because it’s market share is so small. That wouldn’t be the case if, say, half of all users ran Linux.
I've never managed linux IT departments--how well are the management tools compared to what Microsoft offers such as tooling for managing thousands of computers across hundreds of offices.
Layering is absolutely possible, but more at the network layer than the individual computer layer.
Minimal software and OS running on linux as a layer between any windows/whatever and internet connectivity. Minimize and control the exact information that gets to the less hardened and trustworthy/complicated computers
I'm sorry but even Linux requires frequent security updates due it's large ecosystem of dependencies. It's more or less required by every cyber security standard to update them just like windows.
Was there ever such a time? If so then tell me when it was.
"The latest chaos wasn’t caused by an adversary, but it provided a road map of American vulnerabilities at a critical moment."
I've no doubt that road maps of American vulnerabilities are currently being planned, roadmaped and stockpiled for future use by those who aren't on the best terms with the US.
In one way I'm amazed at how laxadasical the US and others are towards these threats and that they have not done more to harden the vulnerabilities. On the other hand, it's obvious: cost is one factor but I reckon another bigger one is 'convenience'. Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that.
Remember, this happened big-time when Microsoft introduced Windows especially Windows 95. To capture the market Microsoft made everything as easy as possible for nontechnical users—just click on something and it'd happen, things would happen with ease. And all this happened without due consideration to security.
When viruses, vulnerabilities, breaches got out of hand restrictions were introduced which meant users had less freedom to do what they'd gotten used to doing. What Microsoft did was to get the world used to slack operating procedures and efforts reign this in has met with user resistance ever since.
We're now stuck with a major problem that was easily foreseeable even before Microsoft launched Windows 95. Fixing it will be extremely difficult.
> In one way I'm amazed at how laxadasical the US and others are towards these threats and that they have not done more to harden the vulnerabilities. On the other hand, it's obvious: cost is one factor but I reckon another bigger one is 'convenience'. Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that.
"Show me the incentives, and I'll show you the outcomes." - Charlie Munger.
We do not incentivize companies to operate secure, redundant, reliable computer systems. We incentivize companies to make the number at the bottom of the spreadsheet beat the expectations some analyst in Lower Manhattan set 90 days prior. And since companies handle the majority of societal work in the United States, that's how most critical systems are designed.
Now, there's a chance that this will play out in court, and that Crowdstrike will have to be bought out to make up for the damages their customers suffered starting on July 19th. However, that will take years, and the outcome could very well be that the plaintiffs will receive symbolic or even no damages. By then, the market will have hedged, captured regulatory authorities, cut its losses, and just altogether moved on. The assets will be purchased in a firesale by people who see this as "creative destruction" and won't care that peoples' lives were put at risk because of this.
This is an area where studying Ukraine's experience will be very useful (and probably has already been useful)
There were years of cyberattacks against pretty much every peice of critical infrastructure they have. Things went down, there were disruptions, but they adapted. Sometimes by falling back to low-tech solutions, sometimes by developing new systems with robustness into new systems and purging the old (much easier to politically justify when the problem is tangible and immediate).
I seem to recall that one of the first things we did when tensions started ramping up was sending teams of cyber security experts from the NSA to help them lock down and root out infiltrations.
The "cyber agencies" focus on offence, because that's easy to score points with and appear to be doing something, whereas defence is a very boring job of securing a zillion outdated endpoints. Or trying to get profitable megacorps to do something less vulnerable and less profitable.
>Was there ever such a time? If so then tell me when it was.
The 90s and into the early 2000s at least. You would get laughed out the room and then fucking fired if you hooked anything critical up to the internet.
> Was there ever such a time? If so then tell me when it was.
It was a goal for a long time, and I'd say we use to be more resilient pre-cloud SaaS auto-update everything. When every software solution installation is on private networks, with fundamentally different architectures (both machine and topology), along with a wide selection of even very poor quality software, was a lot more resilient than what we have today.
Today a single outage in a single service (say AWS) can grind a large number of companies to a halt. A bad update like this one immediately impacts everyone all at once and has a domino effect. That didn't use to happen.
We've been concentrating our collective architecture into a few best practice tools but that all become single points of failure for not only digital attacks, but misconfigurations, mismanagement, company failures, exhausted underpaid engineers, optimizations, etc.
> Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that.
This isn't necessarily true, and I'd argue quite the opposite direction has been happening in the security industry over the past decade or so. People realized that hard security would only cause users to find simple predictable bypasses that would overall _weaken_ the security posture. You just have to look at the evolution of NIST recommendations around passwords to see this happening.
Must change a password every 90 days that can't be the same as your last 10 passwords and complex password requirements? Well users are going to use the minimum size in predictable patterns and just increment a number at the end. Those old password hashes you have to keep around to check if the user is reusing the password? Those are a liability that, when broken, tell the attacker which pattern each user is using. Not the case anymore and there is a lot more usable security rolled that is entirely transparent to end users or almost entirely transparent.
Think about how prevalent and bad captchas used to be on the website and how easy they were to circumvent. Cloudflare's and Google's captcha solution are pretty transparent and has much greater efficacy than the old ones.
Did Microsoft's general and on-going laxness contribute to bad security practices? Absolutely, but that is one ecosystem that had weird other by the nature of how inherently unstable that environment was and is not and hasn't except for maybe a brief peak ever been a core foundation of the internet infrastructure, just enterprise infrastructure unfortunately. They definitely never got the memo about usable or transparent security. I hope they're at least trying behind the scenes now.
> Was there ever such a time? If so then tell me when it was
It seems very plausible that "digital resilience" that this has been buzz phrase repeated often enough in meetings of security-adjacent corporate bureaucrats that some number of people convinced themselves it was a real thing.
And the same divorced-from-specifics approach allows these decision makers to paper over any and all choices that inherently weakened security 'cause the triage needed to partially protect the resulting structurally insecure system can be presented with similar glowing buzz phrases.
In a twisted way, Crowdstrike just gave western civilization a disaster recovery and resilience forced test. an actual attack won't be rolled back within an hour.
In case you don't know, Crowdstrike is hardly the only company with large scale access to this many companies,governments and resources. It takes one rogue employee to deploy a disk wiper that destroys every computer (including linux and macos) and affected systems won't recover at all. it would be months before critical systems are back online, the global economy would come to a halt worse than how it did with COVID in such a scenario.
It isn't "why didn't Crowdstrike do better" (although they should have), it is more, why isn't technology in critical systems more resilient to one vendor screwing up or getting hacked?
For example, let's say it wasn't just a boot loop but a disk wiper erased every boot disk, is there any reason pxe booting a recovery image or a backup image configured already on servers, atms, kiosks, point of sale systems,etc...? even if UEFI and bios were erased, it is technically not impossible to have an auto-recovery mechanism implemented right?
If you have never been in an incident response (IT and security incidents) root cause analysis, I don't blame you for not thinking deeper about the root cause, but that is the type of root cause analysis that has been missing despite over a decade of rampant ransomware, disk wipers, and supply chain risks.
Finding someone to blame and be angry at is easy and doesn't solve the root cause. Making hard technical decisions and not wasting this opportunity (never waste a good crisis) to push for resilient technology investments actually solves the root cause behind this and other repeating problems.
This has been an open secret for decades. Just a handful of major OS and browser vendors, constantly shipping patches to their systems and most software having such vast software supply chains that it's effectively impossible to audit anything, let alone truly certify anything as safe, and "security" software just expands the attack surface.
If you are a non-US company you have to be insane to use this CrowdStrike service. The FBI can legally use a secret warrant[1] and force CrowdStrike to inject a DLL into your infrastructure!
Just told my family yesterday that if we are ever in a real war expect everything to stop working within 8 hours. We will go back to cash and paperwork but it will be painful and slow.
Massive computer outage, worldwide affecting enterprises with Windows machines running CrowdStrike, a very popular software that is sold as hacking protection but which is, in reality, used by C-suite execs to spy on employee behavior. It is installed with extraordinary permissions and is difficult to fix or remove by design.
I wonder if this will teach absolutely anyone a lesson about anything.
> “We are optimistic that A.I. is actually allowing us to make significant — not transformative yet, but significant — progress in being able to identify vulnerabilities, patch holes, improve the quality of coding,” Kent Walker, the president for global affairs at Google, said at the Aspen forum.
I disagree. If the only hope is some vague promise of bs AI, there is no hope indeed.
There are some point where you should redefine what it mean to be an adversary. To be practically forced into a position that lead to this level of harm, by actors that you don't want to perceive, is something that you may want to analyze.
The purpose of a system is what it actually do, not what it claims to do but fails every time at that. Turning everything to vulnerable as fragile with some big strategic and global plan ahead makes you into a disposable asset, a sacrificial victim in some higher level chess game. And you can agree with that with your decisions.
Here's an interesting exercise: what's the minimum quantity of explosives that would lead to 1% drop in western GDP? would doubling it lead to 2% or 4%? is the relationship linear?
I don't have an answer, but thinking about it makes one understand how incredible fragile our complex logistic chains (and indeed our economy) are. One day all this complexity will collapse upon itself and we'll wonder what happened.
How long before our evident incompetence as a profession comes back to bite us in the form of more draconian regulation about who and what is allowed to run in kernel space, or other privileged contexts, on critical infrastructure?
There is no "Digital Resilience" because that is perceived as too expensive, a cost center with hard to quantify value. So it's easier to try and carve out everything that doesn't fit into a spreadsheet, everything that isn't core business, and everything that is not able to present what value it generated.
If general IT had the abilities of sales, marketing, or insurance, there might be a chance that the business would take the responsibility to have the internal knowledge and capabilities to assert control over their systems. But they don't, and as such they won't and instead shove that responsibility over to a third party generalist elsewhere with enough paperwork to have both parties feel their asses are covered.
As long as everything seems to be working, the signals that are still getting through is project failures, be it complete failures or just time and/or money being consumed more than planned and maybe some requirements getting cut. But as soon as enough stuff breaks at the same time, we get news outlets writing articles about resilience and the greater public suddenly no longer agreeing with that is effectively just the result of the status quo because it impacts them directly.
externalizing a threat, from a national news source.. Thought experiment -- a healthy society has plural viewpoints, and plural economic strengths. What if a core and entitled group of groups imposed their "security" on a plural society, for their own profit at the expense of the majority? What if their security is monoculture and internally inconsistent, without the ability to admit error ? What if there is a reflex to blame external groups specifically to divert attention from an internal and unbalanced chain of actions, controls and monetary flows?
What is the response of a Free Press to news stories exercising reflexive blame-game from allied core groups with major monetary interests in the outcomes?
If CrowdStrike's system wasn't able to prevent a kernel driver thats all zeros from getting by, you can be sure a malicious payload would have breezed right through.
The fire more deadly than enemy fire is friendly fire. For adversaries, they cannot do any harm unless they get in, even if they get in, the damage is limited to the access of the account they run on. But for AVs, they are invited in, which renders the 1st line of defense useless. Making it worse, they are running with SYSTEM privileges, which is higher than Admin privileges. And we just witnessed what could happen if AVs went rogue.
The only vulnerability here was CrowdStrike's EDR product that runs exclusively in ring 0 and the entire corporate & technical class that lazily relied on this flawed security model and centalized this incompetence.
As much as some people want to believe that Microsoft is blameless here, I hold them partly responsible. They need to create a stable API in their kernel and force third party security vendors to use it.
I haven't worked in a Windows environment for a long time so was a little surprised how much of the online commentary suggests people in that environment are comfortable or at least resigned to the necessity of unattended live third party updates on critical infrastructure. I can't see any justification for that on the *nix side of things and hope that culture never transfers over.
[+] [-] neonate|1 year ago|reply
[+] [-] ScottBurson|1 year ago|reply
Security is not a feature that can be layered on. It has to be built in. We now have an entire industry dedicated to trying to layer security onto Windows -- but it still doesn't work.
[+] [-] nullindividual|1 year ago|reply
The vendor who makes the software has always written for Windows (or in reality, wrote for either DOS or OS/2 then transitioned to NT4). History, momentum, familiarity, cost, and ease of support all are factors (among others, I'm sure).
Security is a process, not a product.
And yes, distros require frequent updates, though more to your point, you can limit the scope of installed software. I'm sure airport displays don't need MPEG2, VP1 and so on codecs, for instance.
It's also important to remember that there is a lot of 'garageware' out there with these specialized systems. Want SAML/OIDC support? We only support LDAP over cleartext, or Active Directory at best. Want the latest and greatest version of Apache Tomcat? Sorry, the vendor doesn't know how to troubleshoot either, so they only "support" a three year old vulnerable version.
Ran into that more than a few times.
Given the hypothesis of what caused the BSOD with Crowdstrike (NUL pointer), using a safe language would have been appropriate -- it's fairly easy in this case to lay the blame with CS.
Microsoft supplies the shotgun. It's the vendors responsibility to point it away from themselves.
[+] [-] V__|1 year ago|reply
[1] https://news.ycombinator.com/item?id=41018029
[+] [-] politelemon|1 year ago|reply
This is a common misunderstanding, an OS that receives frequent security updates is a very good thing. That means attention is being paid to issues being raised, and risks are being mitigated. Security is not a 'checkbox' it's more of a neverending process because the environment is always in a state of flux.
So to flip it, if an OS is not receiving updates, or not being updated frequently, that's not great.
What you want is updates that don't destabilize an OS, and behind that is a huge history and layers of decisions at each 'shop' that runs these machines.
Security is meant to be in layers and needs to be built in.
> but it still doesn't work.
It does work because the 'scene' has been silent for so long, but what we as humans notice is the incident where it didn't.
[+] [-] wil421|1 year ago|reply
Most people know how to use a windows computer.
Most IT desktop support knows how to use and manage windows. Even building facilities folks can help support them.
Microsoft makes it easy to manage a fleet of computers. They also provide first party (along with thousands of 3rd parties) training and certifications for it.
Windows are the de facto Business Machines.
Most signage companies use windows.
Finding someone who knows a BSD is not easy.
[+] [-] citrin_ru|1 year ago|reply
"Nobody ever gets fired for buying IBM" is as true as ever at least in the corporate world.
[+] [-] dopylitty|1 year ago|reply
[+] [-] LVB|1 year ago|reply
[+] [-] ta1243|1 year ago|reply
[+] [-] tester756|1 year ago|reply
What makes you think so?
How is Linux better in that area?
[+] [-] hi_hi|1 year ago|reply
I understand the reasons for it, and why large, billion dollar companies try to create some sort of efficiency by centralising on one "vendor", but, then this happens.
I don't know how to fix the problem of following "Industry Trends" when every layer above me in the organisation is telling me not to spend the time (money) to investigate alternative software choices which don't fit into their nice box.
[+] [-] stefan_|1 year ago|reply
[+] [-] Rinzler89|1 year ago|reply
Nobody's commenting on that because it's the wrong thing to focus on.
1) This fuckup was on CrowdStrike's Falcon tool (basically a rootkit) bricking Windows due to a bad kernel driver they pushed out without proper hygiene, not on Windows's security patches being bad.
2) Linux also needs to get patches all the time to be secure (remember XZ?) It's not just magically secure by default because of the chubby penguin but is only as secure as it's most vulnerable component, and XZ proved it has a lot of components. I'd be scared if a long period goes by and I see no security patches being pushed to my OS. Modern software is complex and vulnerabilities are everywhere. No OS is ever bug-free and fully bullet proof in order to believe it can be secure without regular patches. Other than TempleOS of course.
The lesson is whichever OS you use, don't surrender your security to a single third party vendor who you now have to trust with the keys of your kingdom as that now becomes your single point of failure. Or if you do be sure you can sue them for the damages.
[+] [-] jijji|1 year ago|reply
[+] [-] giancarlostoro|1 year ago|reply
Or even ChromeOS which has insane security.
> but it still doesn't work.
It works momentarily but there will always be 0-days the people who make the exploits intimately know the windows API internals.
[+] [-] Drygord|1 year ago|reply
[+] [-] balls187|1 year ago|reply
[+] [-] pjmlp|1 year ago|reply
[+] [-] tinytime|1 year ago|reply
[+] [-] beefnugs|1 year ago|reply
Minimal software and OS running on linux as a layer between any windows/whatever and internet connectivity. Minimize and control the exact information that gets to the less hardened and trustworthy/complicated computers
[+] [-] Osiris|1 year ago|reply
We moved to a more frequent update cycle because when a critical vulnerability was found, no one wanted to wait 6-12 months for the service pack.
[+] [-] delfinom|1 year ago|reply
[+] [-] hilbert42|1 year ago|reply
Was there ever such a time? If so then tell me when it was.
"The latest chaos wasn’t caused by an adversary, but it provided a road map of American vulnerabilities at a critical moment."
I've no doubt that road maps of American vulnerabilities are currently being planned, roadmaped and stockpiled for future use by those who aren't on the best terms with the US.
In one way I'm amazed at how laxadasical the US and others are towards these threats and that they have not done more to harden the vulnerabilities. On the other hand, it's obvious: cost is one factor but I reckon another bigger one is 'convenience'. Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that.
Remember, this happened big-time when Microsoft introduced Windows especially Windows 95. To capture the market Microsoft made everything as easy as possible for nontechnical users—just click on something and it'd happen, things would happen with ease. And all this happened without due consideration to security.
When viruses, vulnerabilities, breaches got out of hand restrictions were introduced which meant users had less freedom to do what they'd gotten used to doing. What Microsoft did was to get the world used to slack operating procedures and efforts reign this in has met with user resistance ever since.
We're now stuck with a major problem that was easily foreseeable even before Microsoft launched Windows 95. Fixing it will be extremely difficult.
[+] [-] lenerdenator|1 year ago|reply
"Show me the incentives, and I'll show you the outcomes." - Charlie Munger.
We do not incentivize companies to operate secure, redundant, reliable computer systems. We incentivize companies to make the number at the bottom of the spreadsheet beat the expectations some analyst in Lower Manhattan set 90 days prior. And since companies handle the majority of societal work in the United States, that's how most critical systems are designed.
Now, there's a chance that this will play out in court, and that Crowdstrike will have to be bought out to make up for the damages their customers suffered starting on July 19th. However, that will take years, and the outcome could very well be that the plaintiffs will receive symbolic or even no damages. By then, the market will have hedged, captured regulatory authorities, cut its losses, and just altogether moved on. The assets will be purchased in a firesale by people who see this as "creative destruction" and won't care that peoples' lives were put at risk because of this.
And the cycle will continue.
[+] [-] dralley|1 year ago|reply
There were years of cyberattacks against pretty much every peice of critical infrastructure they have. Things went down, there were disruptions, but they adapted. Sometimes by falling back to low-tech solutions, sometimes by developing new systems with robustness into new systems and purging the old (much easier to politically justify when the problem is tangible and immediate).
I seem to recall that one of the first things we did when tensions started ramping up was sending teams of cyber security experts from the NSA to help them lock down and root out infiltrations.
[+] [-] pjc50|1 year ago|reply
[+] [-] kortilla|1 year ago|reply
The 90s and into the early 2000s at least. You would get laughed out the room and then fucking fired if you hooked anything critical up to the internet.
[+] [-] TrueDuality|1 year ago|reply
It was a goal for a long time, and I'd say we use to be more resilient pre-cloud SaaS auto-update everything. When every software solution installation is on private networks, with fundamentally different architectures (both machine and topology), along with a wide selection of even very poor quality software, was a lot more resilient than what we have today.
Today a single outage in a single service (say AWS) can grind a large number of companies to a halt. A bad update like this one immediately impacts everyone all at once and has a domino effect. That didn't use to happen.
We've been concentrating our collective architecture into a few best practice tools but that all become single points of failure for not only digital attacks, but misconfigurations, mismanagement, company failures, exhausted underpaid engineers, optimizations, etc.
> Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that.
This isn't necessarily true, and I'd argue quite the opposite direction has been happening in the security industry over the past decade or so. People realized that hard security would only cause users to find simple predictable bypasses that would overall _weaken_ the security posture. You just have to look at the evolution of NIST recommendations around passwords to see this happening.
Must change a password every 90 days that can't be the same as your last 10 passwords and complex password requirements? Well users are going to use the minimum size in predictable patterns and just increment a number at the end. Those old password hashes you have to keep around to check if the user is reusing the password? Those are a liability that, when broken, tell the attacker which pattern each user is using. Not the case anymore and there is a lot more usable security rolled that is entirely transparent to end users or almost entirely transparent.
Think about how prevalent and bad captchas used to be on the website and how easy they were to circumvent. Cloudflare's and Google's captcha solution are pretty transparent and has much greater efficacy than the old ones.
Did Microsoft's general and on-going laxness contribute to bad security practices? Absolutely, but that is one ecosystem that had weird other by the nature of how inherently unstable that environment was and is not and hasn't except for maybe a brief peak ever been a core foundation of the internet infrastructure, just enterprise infrastructure unfortunately. They definitely never got the memo about usable or transparent security. I hope they're at least trying behind the scenes now.
[+] [-] joe_the_user|1 year ago|reply
> Was there ever such a time? If so then tell me when it was
It seems very plausible that "digital resilience" that this has been buzz phrase repeated often enough in meetings of security-adjacent corporate bureaucrats that some number of people convinced themselves it was a real thing.
And the same divorced-from-specifics approach allows these decision makers to paper over any and all choices that inherently weakened security 'cause the triage needed to partially protect the resulting structurally insecure system can be presented with similar glowing buzz phrases.
[+] [-] binary132|1 year ago|reply
[+] [-] notepad0x90|1 year ago|reply
In case you don't know, Crowdstrike is hardly the only company with large scale access to this many companies,governments and resources. It takes one rogue employee to deploy a disk wiper that destroys every computer (including linux and macos) and affected systems won't recover at all. it would be months before critical systems are back online, the global economy would come to a halt worse than how it did with COVID in such a scenario.
It isn't "why didn't Crowdstrike do better" (although they should have), it is more, why isn't technology in critical systems more resilient to one vendor screwing up or getting hacked?
For example, let's say it wasn't just a boot loop but a disk wiper erased every boot disk, is there any reason pxe booting a recovery image or a backup image configured already on servers, atms, kiosks, point of sale systems,etc...? even if UEFI and bios were erased, it is technically not impossible to have an auto-recovery mechanism implemented right?
If you have never been in an incident response (IT and security incidents) root cause analysis, I don't blame you for not thinking deeper about the root cause, but that is the type of root cause analysis that has been missing despite over a decade of rampant ransomware, disk wipers, and supply chain risks.
Finding someone to blame and be angry at is easy and doesn't solve the root cause. Making hard technical decisions and not wasting this opportunity (never waste a good crisis) to push for resilient technology investments actually solves the root cause behind this and other repeating problems.
[+] [-] lambdaone|1 year ago|reply
Everyone in the industry knows this.
Interesting to see the NYT just catching up.
[+] [-] newzisforsukas|1 year ago|reply
Maybe it has to do with some major incident that happened yesterday, and the fact they are a news company?
[+] [-] unknown|1 year ago|reply
[deleted]
[+] [-] sschueller|1 year ago|reply
[1] https://en.wikipedia.org/wiki/United_States_Foreign_Intellig...
[+] [-] encoderer|1 year ago|reply
[+] [-] AlbertCory|1 year ago|reply
If more of the critical machines were running different OS's, the damage would be contained.
When we talk about the dangers of "monoculture" it's usually about plants. The same danger applies to computing infrastructure.
[+] [-] simpaticoder|1 year ago|reply
I wonder if this will teach absolutely anyone a lesson about anything.
[+] [-] pm90|1 year ago|reply
> “We are optimistic that A.I. is actually allowing us to make significant — not transformative yet, but significant — progress in being able to identify vulnerabilities, patch holes, improve the quality of coding,” Kent Walker, the president for global affairs at Google, said at the Aspen forum.
I disagree. If the only hope is some vague promise of bs AI, there is no hope indeed.
[+] [-] gmuslera|1 year ago|reply
The purpose of a system is what it actually do, not what it claims to do but fails every time at that. Turning everything to vulnerable as fragile with some big strategic and global plan ahead makes you into a disposable asset, a sacrificial victim in some higher level chess game. And you can agree with that with your decisions.
[+] [-] GeoAtreides|1 year ago|reply
I don't have an answer, but thinking about it makes one understand how incredible fragile our complex logistic chains (and indeed our economy) are. One day all this complexity will collapse upon itself and we'll wonder what happened.
[+] [-] cmrdporcupine|1 year ago|reply
[+] [-] thedataexchange|1 year ago|reply
2. Also a great time to start prepping for AI Incidents => https://thedataexchange.media/ai-incident-response/
[+] [-] oneplane|1 year ago|reply
If general IT had the abilities of sales, marketing, or insurance, there might be a chance that the business would take the responsibility to have the internal knowledge and capabilities to assert control over their systems. But they don't, and as such they won't and instead shove that responsibility over to a third party generalist elsewhere with enough paperwork to have both parties feel their asses are covered.
As long as everything seems to be working, the signals that are still getting through is project failures, be it complete failures or just time and/or money being consumed more than planned and maybe some requirements getting cut. But as soon as enough stuff breaks at the same time, we get news outlets writing articles about resilience and the greater public suddenly no longer agreeing with that is effectively just the result of the status quo because it impacts them directly.
[+] [-] mistrial9|1 year ago|reply
What is the response of a Free Press to news stories exercising reflexive blame-game from allied core groups with major monetary interests in the outcomes?
[+] [-] cdchn|1 year ago|reply
[+] [-] mrjin|1 year ago|reply
[+] [-] Timber-6539|1 year ago|reply
As much as some people want to believe that Microsoft is blameless here, I hold them partly responsible. They need to create a stable API in their kernel and force third party security vendors to use it.
[+] [-] shirro|1 year ago|reply