EDIT: Here's some more RE work on the matter. Has some symbol remapping information that was extracted from the prefix trie the backdoor used to hide strings. Looks like it tried to hide itself even from RE/analysis, too.
The back door pulls this from the certificate received from a remote attacker, attempts to decrypt it with ChaCha20, and if it decrypts successfully, passed to `system()`, which is essentially a simple wrapper that executes a line of shellscript under whichever user the process is currently executing.
If I'm understanding things correctly, this is worse than a public key bypass (which myself and I think a number of others presumed it might be) - a public key bypass would, in theory, only allow you access as the user you're logging in with. Assumedly, hardened SSH configurations would disallow root access.
However, since this is an RCE in the context of e.g. an sshd process itself, this means that sshd running as root would allow the payload to itself run as root.
Wild. This is about as bad as a widespread RCE can realistically get.
> However, since this is an RCE in the context of e.g. an sshd process itself, this means that sshd running as root would allow the payload to itself run as root.
With the right sandboxing techniques, SELinux and mitigations could prevent the attacker from doing anything with root permissions. However, applying a sandbox to an SSH daemon effectively is very difficult.
One have question on this is, if the backdoor would not been discovered due to performance issue (which was as I understood it purely an oversight/fixable deficiency in the code), what are the chances of discovering this backdoor later, or are there tools that would have picked it up? Those questions are IMO relevant to understand if this kind of backdoor is the first one of the kind, or the first one that was uncovered.
Make absolutely sure to include `-a` so it doesn't nuke your env file, and generally speaking, one should upgrade to a version without the malicious code and restart, of course.
Can someone explain succinctly what the backdoor does? Do we even know yet? The backdoor itself is not a payload, right? Does it need a malicious archive to exploit it? Or does it hook into the sshd process to listen for malicious packets from a remote attacker?
The OP makes it sound like an attacker can send a malicious payload in the pre-auth phase of an SSH session - but why does he say that an exploit might never be available? Surely if we can reverse the code we can write a PoC?
Basically, how does an attacker control a machine with this backdoor on it?
It baffles me how such an important package that so many Linux servers use every day is unmaintained by the original author due to insufficient funds. Something gotta change in OSS. I think one solution could be in licenses that force companies/business of certain sizes to pay maintenance fees. One idea from the top of my head.
Currently if you visit the xz repository it is disabled for violating github's TOS.
While it should clearly be disabled, I feel like github should leave the code and history up, while displaying a banner (and disabled any features that could be exploited), so that researchers and others can learn about the exploit.
In more minor situations when a library is hosting malicious code, if I found the repo to be down I might not think anything of it.
If you are interested in the source code that is easy to find. This code and git repo are linked all over the world, in many git repos, and the source is bundled many times
in releases as well.
As a de facto maintainer of an obscure open source game, I see devs come and go. I just merge all the worthwhile contributions. Some collaborators go pretty deep with their features, with a variety of coding styles, in a mishmash of C and C++. I'm not always across the implementation details, but in the back of my mind I'm thinking, man, anyone could just code up some real nasty backdoor and the project would be screwed. Lucky the game is so obscure and the attack surface minuscule, but it did stop me from any temptation to sign Windows binaries out of any sense of munificence.
This xz backdoor is just the most massive nightmare, and I really feel for the og devs, and anyone who got sucked in by this.
- what other ones exist by this same team or similar teams?
- how many such teams are operating?
- how many such dependencies are vulnerable to such infiltration attacks? what is our industry’s attack surface for such covert operations?
I think making a graph of all major network services (apache httpd, postgres, mysql, nginx, openssh, dropbear ssh, haproxy, varnish, caddy, squid, postfix, etc) and all of their dependencies and all of the committers to all of those dependencies might be the first step in seeing which parts are the most high value and have attracted the least scrutiny.
This can’t be the first time someone attempted this - this is just the first unsuccessful time. (Yes, I know about the attempted/discovered backdoor in the linux kernel - this is remote and is a horse of a different color).
Why did they decide to create a backdoor, instead of using a zeroday like everyone else?
Why did they implement a fully-featured backdoor and attempted to hide the way it is deployed, instead of deploying something innocent-looking that might as well be a bug if detected?
These must have been conscious decisions. The reasons might provide a hint what the goals might have been.
So this basically means to scan for this exploit remotely we'd need the private key of the attacker which we don't have. Only other option is to run detection scripts locally. Yikes.
One completely awful thing some scanners might choose to do is if you're offering RSA auth (which most SSH servers are and indeed the SecSH RFC says this is Mandatory To Implement) then you're "potentially vulnerable" which would encourage people to do password auth instead.
Unless we find that this problem has somehow infested a lot of real world systems that seems to me even worse than the time similar "experts" decided that it was best to demand people rotate their passwords every year or so thereby ensuring the real security is reduced while on paper you claim you improved it.
It might be possible to use timing information to detect this, since the signature verification code appears to only run if the client public key matches a specific fingerprint.
The backdoor's signature verification should cost around 100us, so keys matching the fingerprint should take that much longer to process than keys that do not match it. Detecting this timing difference should at least be realistic over LAN, perhaps even over the internet, especially if the scanner runs from a location close to the target. Systems that ban the client's IP after repeated authentication failures will probably be harder to scan.
The tweet says "unreplayable". Can someone explain how it's not replayable? Does the backdoored sshd issue some challenge that the attacker is required to sign?
Unpopular opinion, but I cannot but admire the whole operation. Condemn it of course, but still admire it. It was a piece of art! From conception to execution, masterful! We got extremely lucky that it was caught so early.
I'm assuming nation states and similar actors monitor mailing lists for phrases like "I'm feeling burnt out" or "not enough bandwidth, can you open a PR?"
So I imagine major actors already have other assets in at-risk open source projects, either for the source code or distro patch/packaging level. Is that too tinfoil hat? I only know enough about secops to be dangerous to myself and everyone around me.
Imagine a future where state actors have hundreds of AI agents fixing bugs, gaining reputation while they slowly introduce backdoors. I really hope open source models succeed.
I work for a large closed-source software company and I can tell you with 100% that it is full of domestic and foreign agents. Being open source means that more eyes can and will look at something. That only increases the chance of malicious actions being found out ... just like this supply-chain attack.
... if we want security it needs trust anyway. it doesn't matter if it's amazing Code GPT or Chad NSA, the PR needs to be reviewed by someone we trust.
it's the trust that's the problem.
web of trust purists were right just ahead of the time.
> OpenSSH certs are weird in that they include the signer's public key.
OpenSSH signatures in general contain signer's public key, which I personally think it's not weird but rather cool since it allows verifying the signature without out of the band key delivery (like in OpenPGP). The authentication of the public key is a separate subject but at least some basic checks can be done with an OpenSSH signature only.
> cool since it allows verifying the signature without out of the band key delivery
hope you do key selection sanitization instead of the default (nobody does). otherwise you're accepting random keys you have laying around (like github) when logging to secret.example.com
Lucky the XZ license switched from "Public Domain" to 0BSD in February (just before these 5.6.0 and 5.6.1 releases)!
0BSD has no clauses, but it does have this:
> IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
If I'm reading this right, would there be any persistent evidence of the executed payload? I can't think of a reason anything would have to go to disk in this situation, so a compromise could easily look like an Auth failure in the logs .... maybe a difference in timings .. but that's about it ...
Unless the payload did something that produced evidence, or if the program using openssl that was affected was, for some reason, having all of its actions logged, then no, probably not.
git.tukaani.org runs sshd. If that sshd was upgraded with the xz backdoor, we cannot exclude that the host was compromised as it could be have been a obvious target for the backdoor author.
And it's kind of smart to attack a compression library - you have plausible deniability for these opaque binary blobs - they are supposedly test cases, but in reality encode parts of the backdoor.
God the amount of damage this would've caused, nightmarish, we are so unbelievably lucky. In a few months it would've been in every deb&rpm distribution. Thank God we found it early!
I found the backdoor on five of my Vultr servers as well as my MacBook Pro this evening. I certainly didn’t catch it early.
So if that’s the state of it, it could very well be too late for many many companies. Not to mention folks who rely on TOR for their safety - there could be entire chains of backdoored entry, middle and exit nodes exposing vast numbers of TOR users over the past month or so (spies included!).
Does anyone know if a honeypot has been set up for this?
In the event that exploit code has already been deployed, while these exploit attempts should now thankfully be futile, is there any valuable information that can be gained about the network sources of these exploit attempts?
We might assume that since this attack was foiled, exploit attempts won't happen, but if this is an automated botnet project, there may already be other operational elements in the wild that are knocking?
It's possible that there is an infection detection component to the project which is already measuring baseline accessibility, possibly using this and several alternate vectors. After all, effort was made to evade exploit attempt detection which would enable valuable active monitoring since it shouldn't trigger suspicions.
[+] [-] junon|1 year ago|reply
https://gist.github.com/smx-smx/a6112d54777845d389bd7126d6e9...
Full list of decoded strings here:
https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01
--
For someone unfamiliar with openssl's internals (like me): The N value, I presume, is pulled from the `n` field of `rsa_st`:
https://github.com/openssl/openssl/blob/56e63f570bd5a479439b...
Which is a `BIGNUM`:
https://github.com/openssl/openssl/blob/56e63f570bd5a479439b...
Which appears to be a variable length type.
The back door pulls this from the certificate received from a remote attacker, attempts to decrypt it with ChaCha20, and if it decrypts successfully, passed to `system()`, which is essentially a simple wrapper that executes a line of shellscript under whichever user the process is currently executing.
If I'm understanding things correctly, this is worse than a public key bypass (which myself and I think a number of others presumed it might be) - a public key bypass would, in theory, only allow you access as the user you're logging in with. Assumedly, hardened SSH configurations would disallow root access.
However, since this is an RCE in the context of e.g. an sshd process itself, this means that sshd running as root would allow the payload to itself run as root.
Wild. This is about as bad as a widespread RCE can realistically get.
[+] [-] jeroenhd|1 year ago|reply
With the right sandboxing techniques, SELinux and mitigations could prevent the attacker from doing anything with root permissions. However, applying a sandbox to an SSH daemon effectively is very difficult.
[+] [-] tootie|1 year ago|reply
[+] [-] sega_sai|1 year ago|reply
[+] [-] sgammon|1 year ago|reply
https://gist.github.com/q3k/af3d93b6a1f399de28fe194add452d01...
Which functions as a killswitch:
https://piaille.fr/@zeno/112185928685603910
If that is indeed the case, one mitigation might be
```
echo "yolAbejyiejuvnup=Evjtgvsh5okmkAvj" | sudo tee -a /etc/environment
```
[+] [-] sgammon|1 year ago|reply
[+] [-] dannyw|1 year ago|reply
[+] [-] chatmasta|1 year ago|reply
The OP makes it sound like an attacker can send a malicious payload in the pre-auth phase of an SSH session - but why does he say that an exploit might never be available? Surely if we can reverse the code we can write a PoC?
Basically, how does an attacker control a machine with this backdoor on it?
[+] [-] rubymamis|1 year ago|reply
[+] [-] tevon|1 year ago|reply
While it should clearly be disabled, I feel like github should leave the code and history up, while displaying a banner (and disabled any features that could be exploited), so that researchers and others can learn about the exploit.
In more minor situations when a library is hosting malicious code, if I found the repo to be down I might not think anything of it.
[+] [-] jhfdbkofdchk|1 year ago|reply
[+] [-] Randalthorro|1 year ago|reply
If you are interested in the source code that is easy to find. This code and git repo are linked all over the world, in many git repos, and the source is bundled many times in releases as well.
[+] [-] shp0ngle|1 year ago|reply
[+] [-] tigerlily|1 year ago|reply
This xz backdoor is just the most massive nightmare, and I really feel for the og devs, and anyone who got sucked in by this.
[+] [-] sneak|1 year ago|reply
- what other ones exist by this same team or similar teams?
- how many such teams are operating?
- how many such dependencies are vulnerable to such infiltration attacks? what is our industry’s attack surface for such covert operations?
I think making a graph of all major network services (apache httpd, postgres, mysql, nginx, openssh, dropbear ssh, haproxy, varnish, caddy, squid, postfix, etc) and all of their dependencies and all of the committers to all of those dependencies might be the first step in seeing which parts are the most high value and have attracted the least scrutiny.
This can’t be the first time someone attempted this - this is just the first unsuccessful time. (Yes, I know about the attempted/discovered backdoor in the linux kernel - this is remote and is a horse of a different color).
[+] [-] almostnormal|1 year ago|reply
Why did they implement a fully-featured backdoor and attempted to hide the way it is deployed, instead of deploying something innocent-looking that might as well be a bug if detected?
These must have been conscious decisions. The reasons might provide a hint what the goals might have been.
[+] [-] mdavidn|1 year ago|reply
[+] [-] jtchang|1 year ago|reply
[+] [-] tialaramex|1 year ago|reply
Unless we find that this problem has somehow infested a lot of real world systems that seems to me even worse than the time similar "experts" decided that it was best to demand people rotate their passwords every year or so thereby ensuring the real security is reduced while on paper you claim you improved it.
[+] [-] CodesInChaos|1 year ago|reply
The backdoor's signature verification should cost around 100us, so keys matching the fingerprint should take that much longer to process than keys that do not match it. Detecting this timing difference should at least be realistic over LAN, perhaps even over the internet, especially if the scanner runs from a location close to the target. Systems that ban the client's IP after repeated authentication failures will probably be harder to scan.
(https://bench.cr.yp.to/results-sign.html lists Ed448 verification at around 400k cycles, which at 4GHz amounts to 100us)
[+] [-] Thorrez|1 year ago|reply
[+] [-] noncoml|1 year ago|reply
[+] [-] grepLeigh|1 year ago|reply
According to the timeline here, trust was established in "only" a few years. https://boehs.org/node/everything-i-know-about-the-xz-backdo...
So I imagine major actors already have other assets in at-risk open source projects, either for the source code or distro patch/packaging level. Is that too tinfoil hat? I only know enough about secops to be dangerous to myself and everyone around me.
[+] [-] jmward01|1 year ago|reply
[+] [-] martinohansen|1 year ago|reply
[+] [-] exitzer0|1 year ago|reply
[+] [-] legobmw99|1 year ago|reply
[+] [-] pas|1 year ago|reply
it's the trust that's the problem.
web of trust purists were right just ahead of the time.
[+] [-] wiktor-k|1 year ago|reply
OpenSSH signatures in general contain signer's public key, which I personally think it's not weird but rather cool since it allows verifying the signature without out of the band key delivery (like in OpenPGP). The authentication of the public key is a separate subject but at least some basic checks can be done with an OpenSSH signature only.
[+] [-] 1oooqooq|1 year ago|reply
hope you do key selection sanitization instead of the default (nobody does). otherwise you're accepting random keys you have laying around (like github) when logging to secret.example.com
[+] [-] juliusdavies|1 year ago|reply
0BSD has no clauses, but it does have this:
> IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
[+] [-] emeraldd|1 year ago|reply
[+] [-] junon|1 year ago|reply
[+] [-] dolmen|1 year ago|reply
[+] [-] bostik|1 year ago|reply
[+] [-] kzrdude|1 year ago|reply
Listed among the most recent commits there is "update two test files" https://git.tukaani.org/?p=xz.git;a=commitdiff;h=6e636819e8f...
And it's kind of smart to attack a compression library - you have plausible deniability for these opaque binary blobs - they are supposedly test cases, but in reality encode parts of the backdoor.
[+] [-] lyu07282|1 year ago|reply
[+] [-] supposemaybe|1 year ago|reply
I found the backdoor on five of my Vultr servers as well as my MacBook Pro this evening. I certainly didn’t catch it early.
So if that’s the state of it, it could very well be too late for many many companies. Not to mention folks who rely on TOR for their safety - there could be entire chains of backdoored entry, middle and exit nodes exposing vast numbers of TOR users over the past month or so (spies included!).
[+] [-] christo|1 year ago|reply
In the event that exploit code has already been deployed, while these exploit attempts should now thankfully be futile, is there any valuable information that can be gained about the network sources of these exploit attempts?
We might assume that since this attack was foiled, exploit attempts won't happen, but if this is an automated botnet project, there may already be other operational elements in the wild that are knocking?
It's possible that there is an infection detection component to the project which is already measuring baseline accessibility, possibly using this and several alternate vectors. After all, effort was made to evade exploit attempt detection which would enable valuable active monitoring since it shouldn't trigger suspicions.