OpenSSH: client bug CVE-2016-0777

[patio11]

Use the workaround immediately and patch ASAP, both on your workstation and across your infrastructure where SSH is used, which can be in surprising places.

SSH is designed so that, even if you connect to an evil host, the host only learns your public key, not your private key. This leaks your private key to an evil host. You might think "I only SSH into boxes that I own, so I'm good, right?", but if you in the future lose a box to the enemy, they can then use that box to grab your private key, and then use that key to get into every other box you can access with that keypair.

This turns your sysadmin's photo blog hosted on Digital Ocean into a potential vector into your entire infrastructure, since there is a high likelihood they use the same private key for both.

If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive.

You should expect adversaries to add an exploit for this to their rootkits. That would be bad enough since attackers targeting your infrastructure may exist, but this is also plausibly exploitable in a spray-and-pray fashion. Root any server on the Internet, using e.g. a WordPress vulnerability. Install key-stealing malicious SSH server. Forward all keys to your C&C infrastructure. Your infra does a reverse lookup from public key to Github account (trivial -- try "ssh whoami.filippo.io" if you don't believe me) and immediately optimistically tries to use the stolen key to log into the web tier of any site listed in their email/bio/Twitter profile/etc, pinging the attacker when it succeeds.

[agwa]

> If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive.

A better mandate would be to use ssh-agent. Using separate key pairs without an agent doesn't prevent key theft due to memory disclosure vulnerabilities like this one; it merely compartmentalizes the damage. But if you use an agent, the client never loads the private key, so it's safe from memory disclosure vulnerabilities. Stealing a private key would require filesystem access or remote code execution, at which point all of your private keys are at risk, so you might as well just use a single private key per device.

For the best security, store your private key on a smartcard, so even an attacker with filesystem access or remote code execution can't steal it.

[bisby]

Ironically, ssh'ing into a random server is the exact thing you shouldn't do if you aren't patched.

Make sure you patch BEFORE doing something like ssh'ing into whoami.filippo.io "just because someone on the internet told me to"

This site is (from what I can tell) not malicicous, but copy pasting scripts and blindly following advice from the internet is a prime example of how an exploit would occur.

[moviuro]

This may be a start:

  Host *
  IdentityFile ~/.ssh/%h

Comment: One key per host, named after the host you connect to

[rchiniquy]

If you were to mandate a key rotation across all employees and mandate all employees use a keypair prepared for work and no other purpose I would rate those mandates as not being excessive.

As patio11 notes, this would not at all be excessive as a response to the threat model, but even with an aggressive key rotation mandate, this still doesn't make your production infrastructure as secure as your corporate Confluence wiki with SSO. Your employee's private key is still a static credential, and any rotation policy (14 days? 30 days? 90 days?) will leave a significant window for an attacker to use a stolen credential. Additionally, using a single key per employee for all infrastructure access magnifies the attack surface of a stolen credential to everything you operate.

Full disclosure, I'm a co-founder at ScaleFT, a startup focused on solving these sorts of problems. We're releasing a patch to fully mitigate this for our users this morning.

[robertcope]

I really hope that sysadmins/people are not using the same private key for work and pleasure. And for that matter, the same key at work for DEV/TEST/PROD. But you're probably right that a lot of them are.

[StefanKarpinski]

Does it strike anyone else as bizarre / poor form for an experimental feature to be enabled by default in OpenSSH, which is normally very conservative with option defaults?

[agwa]

What's more, it was not documented in the ssh_config(5) man page. If it had been, I probably would have disabled it long ago when hardening my SSH config.

Time to head to the source to look for other undocumented options...

Update: my findings are here (scroll to bottom for the upshot): https://gist.github.com/AGWA/e92d4f5343be1f7a941d

UseRoaming is the only one to be concerned about. There are many other undocumented options, but they're all aliases for a documented option or are deprecated/unsupported.

[drzaiusapelord]

It just sounds like a big code-base fuckup. They more or less admit it: "Server side was disabled/gutted for years already, but this aspect was surprisingly forgotten."

Sounds like this was put in at one time, forgotten about, and the code lingered for a long time until someone pointed it out. SSH as a protocol is pretty crazy. Everyone loves it, but its a lot of things in one, which ironically goes against the unix philosophy. Its a remote terminal, a file transfer server, a network tunnel server, a socks server, etc. There's a lot of stuff in there and I imagine difficult to work with sometimes.

No word if this is enabled in Putty, but I imagine it is if its using openssh libraries.

[jlgaddis]

Server-side you can see the "full config" sshd uses:

  $ /usr/sbin/sshd -T -f /etc/ssh/sshd_config

Modify, if necessary, to point to the location of your `sshd` and configuration file.

[JimmaDaRustla]

I came here to ask exactly this, and also why this feature isn't documented. Sounds like a nice little exploit.

[ryanlol]

Someone tweeted this, the roaming code doesn't seem to be very well managed: https://twitter.com/marver/status/687644904575627264

Edit 1: Here's some relevant commits too https://marc.info/?l=openbsd-cvs&m=145278217421101&w=2

Edit 2: This mailing list post seems to discuss the vulnerable feature http://www.gossamer-threads.com/lists/openssh/dev/49018?do=p...

Edit 3: Got a better description of actual impact of the bugs:

>Experimental roaming code in the ssh client could be tricked by a hostile sshd

>server, potentially leaking key material. CVE-2016-077 and CVE-0216-078.

>Prevent this problem immediately by adding the line "UseRoaming no" to

>/etc/ssh/ssh_config.

[masklinn]

> Someone tweeted this, the roaming code doesn't seem to be very well managed: https://twitter.com/marver/status/687644904575627264

The subsystem was apparently experimentally (hence the undocumented option) introduced 6 years ago and essentially never used, the server support was never implemented: http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/roa...

[kardos]

So the real question is, can a MITM intercept connections to boxen you frequent to exploit this? Or is it limited to connecting to hostile honeypots?

[jlgaddis]

Workaround (yes, it's client-side):

  # echo -e "Host *\n\tUseRoaming no\n" >> /etc/ssh/ssh_config

Disclaimer: won't work on all operating systems, shells, etc. YMMV. Consult a doctor before following any advice you get from the Internet. Void where prohibited. Restrictions may apply.

Edited per comments below

[ralphm]

Don't ever fix your `ssh_config` by appending stuff to the end of the file. The configuration syntax allows for block constructs without an explicit end marker (like `Match` and `Host`). Appending will cause all kinds of sadness.

[doogle88]

Presumably you would have to connect to a malicious host to be effected? Or perhaps a MITM on your connection to a legit host can exploit you somehow.

[pmoriarty]

Would this work too?

  $ echo "UseRoaming no" >> ~/.ssh/config

[narsil]

I'm emailing the one-liner to folks and found it helpful to just include the root subshell:

    sudo bash -c 'echo -e "Host *\n\tUseRoaming no\n" >> /etc/ssh/ssh_config'

[nulltype]

If you have a mac (Yosemite), it looks like you want to add " UseRoaming no" under the "Host *" line in /etc/ssh_config (as root). You can test it before and after with this:

ssh -v -T [email protected] 2>&1 | grep Roaming

debug1: Roaming not allowed by server <- bad

ssh -v -T [email protected] 2>&1 | grep Roaming

(no output is good)

[super_mario]

On El Capitan the file is /etc/ssh/ssh_config

[metavida]

A few of my friends found multiple copies of ssh_config lying around, so I wrote this one-liner that looks at 3 separate known ssh_config files & patches any vulnerable ones that it finds.

$ for SSH_CONF in /etc/ssh/ssh_config /etc/ssh_config /private/etc/ssh_config; do [ -f $SSH_CONF ] && ! grep -q 'UseRoaming no' $SSH_CONF && echo "Patching $SSH_CONF" && echo -e '\nHost *\n UseRoaming no' | sudo tee -a $SSH_CONF > /dev/null; done

[noterminal]

Here is a user-friendly script that will disable UseRoaming on your os x computer. If you want to send this out to your coworkers, remind them that they have to right-click and choose open to execute a .command script (unless they have "allow apps downloaded from anywhere" enabled)

https://gist.github.com/logicalmethods/49f42190406667cbbbee

[lftl]

Interestingly I haven't been able to get the Roaming not allowed message to show on my client with or without the config option set.

[joe_bleau]

Testing the ssh client config workaround:

   ssh -v user@localhost 2>&1 >/dev/null | grep -i 'roaming'

returns "debug1: Roaming not allowed by server" when vulnerable, and nothing when not. YMMV, only tested on a few machines, etc.

[clort]

does this require a ssh server running on localhost?

[sharjeel]

Since this is a client side issue, can this be used to exploit those automated scanners who try to break into your SSH machine?

[Aissen]

Release is here, with many more details, and other security issues fixed: https://lists.mindrot.org/pipermail/openssh-unix-dev/2016-Ja...

[kondbg]

Details: http://www.openwall.com/lists/oss-security/2016/01/14/7

[aexaey]

Does anybody else find this eerily reminiscent of heartbleed?

Obscure broken feature that nobody uses (or needs) is enabled by default and allow private keys to leak.

[tomjen3]

It is OpenSSH, the clown car of bugs in obscure features nobody uses.

Does anybody know if LibraSSL has released an SSH client yet? I can't code well enough to make security code safe, so I won't be able to help them out on it, but it sounds more and more like OpenSSH can't either.

[slasaus]

My hardened ~/.ssh/config on OS X 10.11:

  Host *
    Ciphers [email protected],[email protected],[email protected],aes256-ctr
    MACs [email protected],[email protected],hmac-sha2-512,hmac-sha2-256
    KexAlgorithms [email protected],diffie-hellman-group-exchange-sha256
    HostKeyAlgorithms ssh-ed25519,ssh-rsa
    ChallengeResponseAuthentication no
    UseRoaming no

If you only connect to newer servers you can further restrict ciphers to only use AEADs (only list the chacha20-poly1305 and aes-gcm ciphers). I assume using AEADs-only makes the MACs keyword obsolete, is this correct?

Config is based on tips from https://stribika.github.io/2015/01/04/secure-secure-shell.ht...

[zackelan]

> I assume using AEADs-only makes the MACs keyword obsolete, is this correct?

Correct. You can test this by running SSH with -v/--verbose and observing these log lines:

    debug1: kex: server->client [email protected] <implicit> none
    debug1: kex: client->server [email protected] <implicit> none

When using a cipher that does not support AEAD, the 2nd field will include the MAC instead of <implicit>

    debug1: kex: server->client aes256-ctr [email protected] none
    debug1: kex: client->server aes256-ctr [email protected] none

I wasn't sure what the 3rd "none" field is, but after some digging it appears to be the compression algorithm: https://github.com/openssh/openssh-portable/blob/master/kex....

And I confirmed this with ssh -C:

    debug1: kex: server->client aes256-ctr [email protected] [email protected]
    debug1: kex: client->server aes256-ctr [email protected] [email protected]

[ehPReth]

Mozilla also maintains a SSH guide for both clients and servers: https://wiki.mozilla.org/Security/Guidelines/OpenSSH

[grawlinson]

Funny story, I erroneously threw `UseRoaming no` on my server config and freaked the fuck out when I couldn't get into via SSH.

Six rescue sessions later, I figured out it shouldn't have been there ...

In the future, I'm definitely going to read documentation & test config files (sshd -t -f /path/to/sshd_config, in this case)

EDIT: Moral of the story, I am not a smart person sometimes.

[Jadestuart1]

Having been through some difficult times when i had been let down by everyone around me, i came across Brandon Reid who is a hacker its actually impossible to put in words how much of a Genius he is and also cant stop thanking him for helping me through my divorce, and also my Nephews grades. He is a Blackhat hacker and very capable of almost and everything, he is actually one of the best out there and also shows you proof before charging you, in my own case the money wasnt the problem and i can gladly say every money spent was so worth it. I have reffered a number of people who will prefer not to be named to Brandon and all have been immensly satisfied with the Top Notch hacking service Brandon offers.To whoever who is lucky enough to read this its a new year and i am only doing this for those genuine people out there who would want the services of a hacker please as i said before be careful how you speak to him his a BigFish. He can be contacted on his services email [email protected] or text [phone] . Do mention the name Jade

[Aissen]

Roaming feature has been in OpennSSH code for a while, but is undocumented: http://superuser.com/a/826734/47771

[DrRobinson]

"Affects all OpenSSH 5.4 - 7.1". 5.4 was released March 2010 it seems

[ryanlol]

AFAIK the roaming feature has been disabled on the server side for a while, but mistakenly left enabled on the client.

[hamburglar]

How does a client bug end up leaking private key material to the server if the private key is in an entirely different process (the agent)? Or is this only an issue if you are loading your identity directly in the client?

[peterwwillis]

So, if there was forgotten code in OpenSSH (which, how the fuck does that happen?), where else might there be forgotten code?

Also: this is a really great example of why you should never have undocumented features in your code.

[ygjb-dupe]

While the bug doesn't expose a user to mitm attacks in general, am I correct in thinking that folks using "-o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" have basically opened the door for that?

https://github.com/search?q=-o+UserKnownHostsFile%3D%2Fdev%2...

[tobiasu]

Ughhh, that's insane.

[merqurio]

As far as I know, this will affect any OSX, am I right ?

[moviuro]

The most terrible thing here is that OpenBSD usually does some full disclosure. This is sure going to be a nasty bug.

[noir_lord]

http://ftp.openbsd.org/pub/OpenBSD/patches/5.8/common/010_ss...

[fulafel]

Leaking private keys seems to be all the rage these days.

[mag00]

I'm curious how typo and bit squatting would come into play here, and if attacks leveraging them could collect private keys at a dangerously high rate before people can patch their clients.

Products like heroku, or the stripe CTF, or other things that come to mind that operate over SSH going rogue a bit scarier. If one were to be compromised it would be a case where mass amounts of private keys could leak. AWS, github, all cloud VPS providers, etc.

Multifactor is relevant as a defense with a vulnerability like this.