WingNews logo WingNews GitHub [2]
top | item 17879444

Linux Kernel Developer Criticizes Intel for Meltdown, Spectre Response

454 points| CrankyBear | 7 years ago |eweek.com | reply

145 comments

order
[+] hansendc|7 years ago|reply
I worked on the Meltdown mitigations, at Intel, during the embargo. I still work on Linux at Intel.

Intel told quite a few members of the Linux community about this well before late December. Some got told because they work for traditional distributions and others were contacted directly because they are community maintainers or subject matter experts.

Unfortunately, Greg was not one of those folks that was told early. It's pretty clear at this point that it would have been a lot better had he and folks like him been involved earlier. This is especially true for Greg since he plays such a crucial role in stable kernel maintenance, which is how a lot of the world consumes their kernels (including distros like Debian).

I'm glad Greg thinks "Intel has gotten better at this." I'd like to think so too.

[+] 2trill2spill|7 years ago|reply
Shouldn't Greq and Linus be some of the very first people to be informed considering the amount of impact they both have on the entire Linux ecosystem? I find it alarming that they were not contacted immediately considering how much of the world/internet relies on Linux.
[+] ibotty|7 years ago|reply
That does not address the stupid embargo on involved linux developers to speak about it to each other.
[+] geezerjay|7 years ago|reply
> Unfortunately, Greg was not one of those folks that was told early.

Why wasn't he one of the first persons informed by Intel?

[+] cthalupa|7 years ago|reply
>"The majority of the world runs Debian or they run their own kernel," Kroah-Hartman said. "Debian was not allowed to be part of the disclosure, so the majority of the world was caught with their pants down, and that's not good."

Is there any actual statistics to back this up? I feel like RHEL and to a lesser extent CentOS have a stranglehold on the big enterprise-y environments, and I see Ubuntu basically everywhere else, and Canonical does their own kernels.

Edit: To be clear, I am aware Ubunbtu is a Debian derivative, but since we're talking specifically about who was informed for kernel level mitigations, and Canonical does their own kernels, it seems weird to talk about how Debian wasn't informed and thus people were affected, when Ubuntu being updated wasn't reliant on Debian being updated.

[+] stefan_|7 years ago|reply
The majority of Linux installations is probably Android smartphones, and well Google was informed alright, but still 95% of those phones were caught with their pants down because well their pants had fallen down a long long time ago given they last received security updates years ago.
[+] dijit|7 years ago|reply
I can speak to the companies I've worked for: US companies (and, this includes the UK because apparently we love to be like america) use CentOS predominantly, but mainland europe (Sweden, Germany, Finland) seem to prefer Debian.

I can say that I've personally administered roughly a 1:1 ratio of CentOS:Debian despite coming from a country that's servers tend to be CentOS.

Of course, this is anecdotal, but don't undersell debian.

[+] jack12|7 years ago|reply
https://youtu.be/lQZzm9z8g_U?t=1565 seems to be the same talk delivered at another instance of this conference. He does claim one statistic that an unnamed "top 3" cloud provider told him that fewer than 10% of their customers install company-based kernels and over 90% install community-based kernels like debian or kernel.org code.

He is not limiting "the majority" to just Debian, though. He is comparing installs of distributions backed by companies (like Red Hat+Canonical+SuSE) as "the minority" vs. every possible non-corporate-backed distribution or source.

[+] curveship|7 years ago|reply
By "majority" he may have meant majority of Linux distributions, not majority of Linux users. There are a ton of distributions derived from Debian: https://wiki.debian.org/Derivatives/Census . All of them would have to update their distros to respond to Meltdown.
[+] sounds|7 years ago|reply
Perhaps Greg-KH was implicitly including Ubuntu under Debian?
[+] xchaotic|7 years ago|reply
I'm sure he cares a lot about Debian and a lot of my early learning was on Debian derivatives but I feel that the production server side world has largely standardised on RedHat. Non RedHat/CentOS, non-Ubuntu debian distros are probably a distant fifth in terms of business critical production server so no wonder they were not included in the disclosure early on...
[+] crankylinuxuser|7 years ago|reply
I run centos in production. That's because of the following:

     1. SELinux 
     2. OpenSCAP
     3. 99% fit with RedHat
None of the other Linux variants have those tools, or the time put in for compliance goodies. My time == $$$
[+] shakna|7 years ago|reply
Why would an off-the-record statement, which is what he said this was, need to be 100% without exaggeration or hyperbole?
[+] l0b0|7 years ago|reply
Translation: s/The majority of the world runs/My fellow Debian kernel developers run/
[+] rurban|7 years ago|reply
Desktop and personal PC has Ubuntu dominant, but servers are 70% CentOS. Nobody trusts Debian's security record and gcc/kernel abilities.

He is right with the affected number in this case because this time bugs are affected by simple webpage drive-by Javascript attacks, like a common virus. So it affects Debian and Android more than servers and cloud services. And those are the ones which are not regularly updated so the security impact is bigger than on a server.

[+] 2trill2spill|7 years ago|reply
> "That's a long time, and we only heard rumors because another very large operating system vendor told Intel to get off their tails and tell us about it."

I wonder which operating system vendor pressured Intel to tell the Linux dev community, especially because it sounds like it was a non Linux OS vendor. Whomever it was, good job!

But it seems like Intel has angered the Linux community as well as the various BSD operating systems. You would think Intel would be doing whatever it can to please all operating system vendors especially now that AMD is getting competitive again.

[+] judge2020|7 years ago|reply
My guess is Microsoft seeing how recently they're pouring a lot of resources into the Linux subsystem and how they're trying to seem more "developer-oriented" overall in recent months.
[+] cthalupa|7 years ago|reply
Azure has a sizable customer base running Linux, so I imagine it was Microsoft.
[+] snaky|7 years ago|reply
BTW

> Experts called for a new generation of secure-by-design computers at the Hot Chips conference here. In small steps in that direction, Microsoft and Google described their separate but similar hardware security architectures.

https://www.eetimes.com/document.asp?doc_id=1333616

[+] y-c-o-m-b|7 years ago|reply
I wonder how many NSA back-doors will go into those.
[+] AdmiralAsshat|7 years ago|reply
Greg's response is understandably frustrated, though seemingly less so than the OpenBSD devs. Why are they being repeatedly left out of the loop?
[+] amaranth|7 years ago|reply
OpenBSD has been accused of breaking embargoes in the past. They are pretty open about their policy of pushing their fix as soon as it's ready and not doing anything to obfuscate what they're fixing and why.
[+] rrix2|7 years ago|reply
> "Intel has gotten better at this," he said.

Someone should let the BSD folks know and see what they think...

[+] cperciva|7 years ago|reply
My understanding is that they've gotten better. FreeBSD has had advance notice of some issues. Last I heard they offered to let OpenBSD in too, but hadn't found anyone willing to sign an NDA.
[+] cperciva|7 years ago|reply
While I agree that Intel's response was far from ideal, I find it a bit rich for Linux kernel developers to be criticizing them. Remember, the completely uncoordinated disclosure happened because Linux kernel developers started discussing the vulnerability -- while under NDA -- on a public mailing list.
[+] dschuetz|7 years ago|reply
Because Intel is that good in security and hardening, why not use Intel's newly minted own special secure Linux distribution?

https://01.org/blogs/imad/2018/letter-industry

This is almost like satire.

[+] smhost|7 years ago|reply
Not sure what you mean. This just looks to me like they're admitting they're awful at security and they're pleading/threatening the global community to help secure their ecosystem or else you're all going to be in a ton of trouble because the robots/cars are going to kill you and your workers and what'll you do then?
[+] hornetblack|7 years ago|reply
I think even a Intel contractor was indirectly responsible for GRSecurity going behind a paywall.

(tl;dr intel contractor were shipping an os that was using grsec tradmark in marketing for a non-grsec approved kernel)

[+] 0xFFC|7 years ago|reply
I was in the room. Greg specifically said “off the record”.
[+] docker_up|7 years ago|reply
What is the current status of this? The last time I heard, the OS fixes would impact CPU performance by 30%. Is this still the case? Will new iterations of Intel CPUs be immune to this, or is this an ongoing issue going forward because it's inherent with the architecture?
[+] hansendc|7 years ago|reply
Disclaimer: I work on Linux at Intel.

Future hardware will be mitigated against side-channel issues. There's a nice table showing how things are mitigated on future processors here:

https://www.tomshardware.com/news/intel-cascade-lake-details...

But, not everything is mitigated in the silicon or microcode. The mitigation for Spectre / Variant 1 / Bounds Check Bypass, for instance, will continue to be in software.

[+] simcop2387|7 years ago|reply
Based on my own experience with things and seeing other benchmarks, it looks like there's some synthetic benchmarks and very specific loads that will see up to 30% losses, but most things are really around 10-15% if they've got any significant losses at all. That's still a big impact but not nearly as "sky is falling" as it seemed like it might be. No idea how this'll be going forward with hardware changes since those haven't happened yet, and even when they do, how will we benchmark them properly?
[+] tracker1|7 years ago|reply
Everything I've read has said real workload differences can be as little as 3% or as much as 30%, but mostly somewhere in between (closer to the 3% side). Depends on what you're doing I guess.
[+] pmontra|7 years ago|reply
> While there have been many patches made in Linux, he strongly advised users to update with Intel's microcode fixes as well, as they provide an additional layer of protection beyond what an operating system can provide.

I found this page explaining how to do it https://www.cyberciti.biz/faq/install-update-intel-microcode...

I checked and my microcode is from 2018-01-21. Being on a 2014 laptop it means it got updated by the package manager.

[+] 2bluesc|7 years ago|reply
Any videos of the presentation?
[+] mikeyouse|7 years ago|reply
No video as far as I know, but he did post the slides with some backup materials on his GitHub:

https://github.com/gregkh/presentation-spectre

One note from his talk though, his slides say that Foreshadow was fixed in April -- but it's clearly supposed to be August. He mentioned he was fixing the updates even on the flight out to OSSNA.

[+] bigDeal|7 years ago|reply
Does anyone honestly buy into the idea that this is something other than a back door, carefully plotted by certain nation-state actors?

For real.

[+] bArray|7 years ago|reply
The creators aren't always aware of the monsters they create. I think Intel was caught with their trousers down - AMD and ARM in many respects too. Hopefully this will lead to automated testing procedures for Intel, other processor manufacturers and security researchers.

The reason these flaws were found in the first place were because of security researchers, hopefully they can now make their cases to get large research grants and bring about a more secure future.

On the other hand, if you want to talk about tin-foil hat "CPUs working against users", only look so far as Intel ME and the equivalents on other processors. Closed-source and highly privileged code running all the time inside the CPU - it's not unthinkable that Intel would bend the knee to some state actors, especially if that leads to higher profits (i.e. a tax break or entry into a new market (cough China)). It's enough of a concern that a lot of reverse engineering effort has gone into preventing it from running.

[+] adiusmus|7 years ago|reply
Applying Occam’s razor to Intel ME and similar security issues: conspiracy with some evil agency is possible. Maybe. But not the only explanation.

Following the money suggests Intel was chasing an enterprise market and didn’t bother to think through the consequences (unlikely) or made a car-manufacturing style decision where the costs of failure were deemed to be overshadowed by the profits on their balance sheets.

For seven years the Ford Motor Company sold Pinto cars in which it knew hundreds of people would needlessly burn to death. It had a better design but it would have cost production delays to implement it. Less money to have people die than fix the issue when it was noticed in testing.

While I’m not equating Intel’s actions to Ford’s in direct comparison, I would say the bean counter thought process is far more likely than conspiracy.

[+] froogie|7 years ago|reply
Why would Intel be complicit in it?

Suppose they did this. Now, that the vulnerabilities are public, Intel takes the fall. Oopsie, huh?

[+] apetresc|7 years ago|reply
Yes, literally every rational person.
[+] cmurf|7 years ago|reply
Yes. But then, you're asking for an opinion. If you were asking the reader for evidence, you wouldn't have started out with a premise depending on paranoia and conspiracy.