> 8/14/18 – Updated: As in the case with Meltdown, we believe our processors are not susceptible to these new speculative execution attack variants: L1 Terminal Fault – SGX (also known as Foreshadow) CVE 2018-3615, L1 Terminal Fault – OS/SMM (also known as Foreshadow-NG) CVE 2018-3620, and L1 Terminal Fault – VMM (also known as Foreshadow-NG) CVE 2018-3646, due to our hardware paging architecture protections. We are advising customers running AMD EPYC™ processors in their data centers, including in virtualized environments, to not implement Foreshadow-related software mitigations for their AMD platforms.
For those on AMD platforms, how do you disable software mitigations for Foreshadow? Is this automatically done by browsers, operating systems and hypervisors?
With the newer AMD processors having as many real cores as they do, does the cost-benefit analysis of HT/SMT change? I read in a comment here a few weeks ago that turning it off on the newer AMD CPUs can yield better performance because of improved cache-coherency on some workloads (My memory of what I read might be totally wrong).
For anyone (understandably) confused about the attacks and mitigations related to L1TF, I've found the linux kernel documentation on the mitigations[0] to be a great resource.
One interesting thing is that to mitigate L1TF hyperthreads only need to be disabled if you are running VMs, the userspace mitigations are effective regardless of HT status. However, there's a catch, you can leave hyperthreading enabled if you disable the Extended/Nested Page Table virtualization feature. However it is noted that this will result in a significant performance impact.
However this does not mean that HT with VMs is totally secure, as there may be more vulnerabilities relating to HT yet to be disclosed/released as alluded to by Theo. (For context, see the previous discussion [1] around the Lazy FPU switching vulnerability where Theo made the decision to enable mitigations in OpenBSD[2] prior to the public disclosure of the bug based (Theo/OpenBSD was _not_ party to the embargo))
We have disabled Hyper Threading(HT) on all public facing servers(running OpenBSD). However, our compute nodes running Linux kernel are benefiting about 80 to near 100% boost for specific scientific workloads. So, we run our INTERNAL NETWORK ONLY compute nodes with HT on. In places where security is not primary concern, why not make use of HT for extra efficiency?
Think and plan before you blanket disable HT on all servers running intel CPUs...
If an attacker is able to run any code on these private servers, I have bigger problems to deal with than HT as attack vector..
If you fully trust the software you're running, I see no reason to disable HT. At this point, I don't think I'd have it running on anything publicly facing, though. That said, I still have it enabled on my work PC & home PC.
Agree. A personal computer could probably even risk it as long as they don't run untrusted javascript (which they shouldn't do anyways, or only under sandboxed/careful conditions).
You can give both hyperthreads in a physical core to the same tenant, no?
Scheduling different VMs to run on the same hyperthreaded core at once seems like it can't be good for either VM's performance, even if there were no security concerns. Hyperthreading is much more useful for running multiple threads of the same app, accessing similar instruction caches etc.
(There's also a question of safety within the VM, but a huge number of cloud users are running effectively one user within their VM.)
Vmware have disclaimers in the mitigation options that preclude turning off HT, meaning, use at your own risk. [1]
I am still waiting on a comment from Linode [2]
Openstack have some knobs you can adjust, but it really depends on your workloads and what risk you are willing to accept. [3]
AWS have their own custom hypervisor and are said to have worked around the issue. [4] Amazon had info on this before others. It appears they have a special relationship with Intel?
I have not found any hardware or OS vendors that are willing to say that you can leave HT enabled. It is a very heated topic because folks will have to increase their VM infrastructure anywhere from 5% to 50% depending on their workload profiles. For public clouds, you can't predict workload profiles.
Is it a feasible solution to enable hyperthreading only for threads or forks of the same process? Then they can use this ability, but other processes cannot do timing attacks on this process in this core... I think
Thinking about this, they're probably gonna introduce "insecure but cheap" instances for customers that don't mind the chance of data leaks and takeovers...
Question: If I rent 4 core AWS instance, does it mean 4 physical cores or 4 hyper threaded cores? Is there a standard to this definition of “cores” across GCP, DO, Linode, etc. I don’t have the experience or knowledge about cloud computing but just have a DO instance running a web server. I’m curious.
> SMT is fundamentally broken because it shares resources between the two cpu instances and those shared resources lack security differentiators.
I thought the root of one of the Foreshadow problems was that caches are shared across cores, and therefore even with hyperthreading disabled, you still gain information about a process on another core. Am I misinterpreting it?
It does seem like the paranoid thing to do is that each socket gets to be used by only a single user. (I half-jokingly suggested at work that we replace our internal cloud with a Beowulf cluster of Raspberry Pis...)
It also seems like you could design OSes in a way which is more robust to this, e.g., certain cores are only for the kernel and processes running as root, and system calls are inter-processor interrupts, so privileged kernel (or userspace root) data doesn't go into untrusted caches at all.
Scary looking headline on a discussion forum with instructions to perform a task that the average user would not really understand, with no explanation of attack vector or even consequences for any user who doesn't want to take the time (and energy, frankly at this point) to follow security news.
I'm pretty close to not caring anymore. I hope somebody figures out how to at least fix the security news infrastructure, if fixing security is still a ways off.
EDIT: Scratch that, I assume attack vector is a browser since they mentioned JavaScript.
This is not security news at all. This is Theo de Raadt's personal E-mail sent to the OpenBSD development mailing list for system developers. It is never intended for the consumption by the general public.
Maybe this is what finally gets me to upgrade from my ~2012 i7-3770. Not because of performance improvements, but to avoid performance degradation from all these security patches...
I'm not so sure I'd bother, more of these attacks keep coming out, odds are you'll just buy a CPU that'll be vulnerable to the next one. Maybe AMD would save you from some.
I'm in this exact scenario. I am thinking I might just go with AMD this time around, even if it is mostly an illusory short-term strength over Intel. In the long run I will undoubtedly have to refresh my hardware as new exploits come out, but at least I can take solace in the fact that I'm only worried about a single machine and not a datacenter.
I feel the same, maybe I will switch to AMD next time I upgrade.
On the other end, I am happy with my budget dedicated server: its Intel Atom N2800 is so "rustic" that it did not get affected by any of Intel vulns (yet).
>>> We are having to do research by reading other operating systems.
Si Intel cooperates with business partners like apple/windows and not with open source. Does it mean that Apple and Windows can claim to be more secure because they have access to the information needed to fix Intel's issues ?
In the end, the conclusion is:
"Long story short, Hyper Threading is still very much relevant in 2018 with current-generation Intel CPUs. In the threaded workloads that could scale past a few threads, HT/SMT on this Core i7 8700K processor yielded about a 30% performance improvement in many of these real-world test cases."
Will be interesting to see what Apple does about this in their next software update. I can’t imagine many people will be happy if the next software update forcibly disables hyper threading.
(For those who aren’t familiar with Apple devices, Apple don’t expose settings like this to a user, which are usually available in the BIOS on a PC)
It's not clear that he's using "SMT" to refer to AMD specifically as he goes on to talk about "Intel CPUs" and disabling it in "Intel BIOS". Does the Zen architecture have the same issue?
SMT is the generic name? Hyper-Threading is the name of Intel's 2 way SMT implementation. I don't think they call the Xeon Phi 4-way SMT Hyper-Threading, but I could be wrong there.
Title: Disable SMT/Hyperthreading in all Intel BIOSes
Posted by Theo de Raadt-2 on Aug 23, 2018; 11:35am
Two recently disclosed hardware bugs affected Intel cpus:
- TLBleed
- T1TF (the name "Foreshadow" refers to 1 of 3 aspects of this bug, more aspects are surely on the way)
Solving these bugs requires new cpu microcode, a coding workaround, AND the disabling of SMT / Hyperthreading.
SMT is fundamentally broken because it shares resources between the two cpu instances and those shared resources lack security differentiators. Some of these side channel attacks aren't trivial, but we can expect most of them to eventually work and leak kernel or cross-VM memory in common usage circumstances, even such as javascript directly in a browser.
There will be more hardware bugs and artifacts disclosed. Due to the way SMT interacts with speculative execution on Intel cpus, I expect SMT to exacerbate most of the future problems.
A few months back, I urged people to disable hyperthreading on all Intel cpus. I need to repeat that:
DISABLE HYPERTHREADING ON ALL YOUR INTEL MACHINES IN THE BIOS.
Also, update your BIOS firmware, if you can.
OpenBSD -current (and therefore 6.4) will not use hyperthreading if it is enabled, and will update the cpu microcode if possible.
But what about 6.2 and 6.3?
The situation is very complex, continually evolving, and is taking too much manpower away from other tasks. Furthermore, Intel isn't telling us what is coming next, and are doing a terrible job by not publically
documenting what operating systems must do to resolve the problems. We are having to do research by reading other operating systems. There is no time left to backport the changes -- we will not be issuing a complete set of errata and syspatches against 6.2 and 6.3 because it is turning into a distraction.
Rather than working on every required patch for 6.2/6.3, we will re-focus manpower and make sure 6.4 contains the best solutions possible.
So please try take responsibility for your own machines: Disable SMT in the BIOS menu, and upgrade your BIOS if you can.
I'm going to spend my money at a more trustworthy vendor in the future.
Okay, I found it. A SMT knob was added alongside in the L1TF fixes.
/sys/devices/system/cpu/smt
/sys/devices/system/cpu/smt/active
/sys/devices/system/cpu/smt/control
active: Tells whether SMT is active (enabled and siblings online)
control: Read/write interface to control SMT. Possible
values:
"on" SMT is enabled
"off" SMT is disabled
"forceoff" SMT is force disabled. Cannot be changed.
"notsupported" SMT is not supported by the CPU
If control status is "forceoff" or "notsupported" writes are rejected.
I don't think you can answer that with a single question. It depends on how CPU-intensive the work that server was handling was.
I think, though, that if you'd be particularly willing to knowingly allow these kinds of vulnerabilities in exchange for some performance, OpenBSD probably isn't a good fit for you in the first place.
[+] [-] walterbell|7 years ago|reply
Edit: https://www.amd.com/en/corporate/security-updates
> 8/14/18 – Updated: As in the case with Meltdown, we believe our processors are not susceptible to these new speculative execution attack variants: L1 Terminal Fault – SGX (also known as Foreshadow) CVE 2018-3615, L1 Terminal Fault – OS/SMM (also known as Foreshadow-NG) CVE 2018-3620, and L1 Terminal Fault – VMM (also known as Foreshadow-NG) CVE 2018-3646, due to our hardware paging architecture protections. We are advising customers running AMD EPYC™ processors in their data centers, including in virtualized environments, to not implement Foreshadow-related software mitigations for their AMD platforms.
For those on AMD platforms, how do you disable software mitigations for Foreshadow? Is this automatically done by browsers, operating systems and hypervisors?
[+] [-] djsumdog|7 years ago|reply
I haven't really ready anything on this most recent set of vulnerabilities and AMD.
[+] [-] tareqak|7 years ago|reply
[+] [-] gameswithgo|7 years ago|reply
[+] [-] bonzini|7 years ago|reply
[+] [-] morsma|7 years ago|reply
[+] [-] exrook|7 years ago|reply
One interesting thing is that to mitigate L1TF hyperthreads only need to be disabled if you are running VMs, the userspace mitigations are effective regardless of HT status. However, there's a catch, you can leave hyperthreading enabled if you disable the Extended/Nested Page Table virtualization feature. However it is noted that this will result in a significant performance impact.
[0] https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html
However this does not mean that HT with VMs is totally secure, as there may be more vulnerabilities relating to HT yet to be disclosed/released as alluded to by Theo. (For context, see the previous discussion [1] around the Lazy FPU switching vulnerability where Theo made the decision to enable mitigations in OpenBSD[2] prior to the public disclosure of the bug based (Theo/OpenBSD was _not_ party to the embargo))
[1] https://news.ycombinator.com/item?id=17304233
[2] https://marc.info/?l=openbsd-cvs&m=152818076013158&w=2
[+] [-] reacharavindh|7 years ago|reply
Think and plan before you blanket disable HT on all servers running intel CPUs...
If an attacker is able to run any code on these private servers, I have bigger problems to deal with than HT as attack vector..
[+] [-] hermitdev|7 years ago|reply
[+] [-] spydum|7 years ago|reply
[+] [-] bo1024|7 years ago|reply
[+] [-] Roritharr|7 years ago|reply
Cloud Providers are gonna have a bad time if this is true.
[+] [-] geofft|7 years ago|reply
Scheduling different VMs to run on the same hyperthreaded core at once seems like it can't be good for either VM's performance, even if there were no security concerns. Hyperthreading is much more useful for running multiple threads of the same app, accessing similar instruction caches etc.
(There's also a question of safety within the VM, but a huge number of cloud users are running effectively one user within their VM.)
[+] [-] LinuxBender|7 years ago|reply
Vmware have disclaimers in the mitigation options that preclude turning off HT, meaning, use at your own risk. [1]
I am still waiting on a comment from Linode [2]
Openstack have some knobs you can adjust, but it really depends on your workloads and what risk you are willing to accept. [3]
AWS have their own custom hypervisor and are said to have worked around the issue. [4] Amazon had info on this before others. It appears they have a special relationship with Intel?
I have not found any hardware or OS vendors that are willing to say that you can leave HT enabled. It is a very heated topic because folks will have to increase their VM infrastructure anywhere from 5% to 50% depending on their workload profiles. For public clouds, you can't predict workload profiles.
Edit: Oops I left out the main site for L1TF [5]
[1] - https://kb.vmware.com/s/article/55806
[2] - https://blog.linode.com/2018/08/16/intels-l1tf-cpu-vulnerabi...
[3] - https://access.redhat.com/articles/3569281
[4] - https://aws.amazon.com/security/security-bulletins/AWS-2018-...
[5] - https://foreshadowattack.eu/
[+] [-] Aardwolf|7 years ago|reply
[+] [-] Roritharr|7 years ago|reply
[+] [-] fermienrico|7 years ago|reply
[+] [-] _JamesA_|7 years ago|reply
[+] [-] shittyadmin|7 years ago|reply
[+] [-] crunchlibrarian|7 years ago|reply
Turn it off or sell your Intel chips.
[+] [-] geofft|7 years ago|reply
I thought the root of one of the Foreshadow problems was that caches are shared across cores, and therefore even with hyperthreading disabled, you still gain information about a process on another core. Am I misinterpreting it?
It does seem like the paranoid thing to do is that each socket gets to be used by only a single user. (I half-jokingly suggested at work that we replace our internal cloud with a Beowulf cluster of Raspberry Pis...)
It also seems like you could design OSes in a way which is more robust to this, e.g., certain cores are only for the kernel and processes running as root, and system calls are inter-processor interrupts, so privileged kernel (or userspace root) data doesn't go into untrusted caches at all.
[+] [-] orblivion|7 years ago|reply
I'm pretty close to not caring anymore. I hope somebody figures out how to at least fix the security news infrastructure, if fixing security is still a ways off.
EDIT: Scratch that, I assume attack vector is a browser since they mentioned JavaScript.
[+] [-] bcaa7f3a8bbc|7 years ago|reply
[+] [-] jonny_eh|7 years ago|reply
[+] [-] seiferteric|7 years ago|reply
[+] [-] shittyadmin|7 years ago|reply
[+] [-] mentalpiracy|7 years ago|reply
[+] [-] kleim|7 years ago|reply
[+] [-] wiz21c|7 years ago|reply
>>> We are having to do research by reading other operating systems.
Si Intel cooperates with business partners like apple/windows and not with open source. Does it mean that Apple and Windows can claim to be more secure because they have access to the information needed to fix Intel's issues ?
[+] [-] bonsai80|7 years ago|reply
In the end, the conclusion is: "Long story short, Hyper Threading is still very much relevant in 2018 with current-generation Intel CPUs. In the threaded workloads that could scale past a few threads, HT/SMT on this Core i7 8700K processor yielded about a 30% performance improvement in many of these real-world test cases."
[+] [-] fyfy18|7 years ago|reply
(For those who aren’t familiar with Apple devices, Apple don’t expose settings like this to a user, which are usually available in the BIOS on a PC)
[+] [-] kevin_thibedeau|7 years ago|reply
[+] [-] gaius|7 years ago|reply
[+] [-] mackal|7 years ago|reply
[+] [-] Tuna-Fish|7 years ago|reply
[+] [-] swixmix|7 years ago|reply
URL: http://openbsd-archive.7691.n7.nabble.com/Disable-SMT-Hypert...
Here it is: ]
---
Title: Disable SMT/Hyperthreading in all Intel BIOSes
Posted by Theo de Raadt-2 on Aug 23, 2018; 11:35am
Two recently disclosed hardware bugs affected Intel cpus:
- TLBleed
- T1TF (the name "Foreshadow" refers to 1 of 3 aspects of this bug, more aspects are surely on the way)
Solving these bugs requires new cpu microcode, a coding workaround, AND the disabling of SMT / Hyperthreading.
SMT is fundamentally broken because it shares resources between the two cpu instances and those shared resources lack security differentiators. Some of these side channel attacks aren't trivial, but we can expect most of them to eventually work and leak kernel or cross-VM memory in common usage circumstances, even such as javascript directly in a browser.
There will be more hardware bugs and artifacts disclosed. Due to the way SMT interacts with speculative execution on Intel cpus, I expect SMT to exacerbate most of the future problems.
A few months back, I urged people to disable hyperthreading on all Intel cpus. I need to repeat that:
DISABLE HYPERTHREADING ON ALL YOUR INTEL MACHINES IN THE BIOS.
Also, update your BIOS firmware, if you can.
OpenBSD -current (and therefore 6.4) will not use hyperthreading if it is enabled, and will update the cpu microcode if possible.
But what about 6.2 and 6.3?
The situation is very complex, continually evolving, and is taking too much manpower away from other tasks. Furthermore, Intel isn't telling us what is coming next, and are doing a terrible job by not publically documenting what operating systems must do to resolve the problems. We are having to do research by reading other operating systems. There is no time left to backport the changes -- we will not be issuing a complete set of errata and syspatches against 6.2 and 6.3 because it is turning into a distraction.
Rather than working on every required patch for 6.2/6.3, we will re-focus manpower and make sure 6.4 contains the best solutions possible.
So please try take responsibility for your own machines: Disable SMT in the BIOS menu, and upgrade your BIOS if you can.
I'm going to spend my money at a more trustworthy vendor in the future.
[+] [-] ianai|7 years ago|reply
[+] [-] bcaa7f3a8bbc|7 years ago|reply
[+] [-] bcaa7f3a8bbc|7 years ago|reply
[+] [-] scns|7 years ago|reply
[+] [-] mehrdadn|7 years ago|reply
[+] [-] AdmiralAsshat|7 years ago|reply
[+] [-] ams6110|7 years ago|reply
If you read https://www.openbsd.org/goals.html, the word "performance" does not appear.
[+] [-] bunderbunder|7 years ago|reply
I think, though, that if you'd be particularly willing to knowingly allow these kinds of vulnerabilities in exchange for some performance, OpenBSD probably isn't a good fit for you in the first place.
[+] [-] messe|7 years ago|reply
[+] [-] fredch|7 years ago|reply
[deleted]
[+] [-] codedokode|7 years ago|reply