top | item 40610621

OpenSSH introduces options to penalize undesirable behavior

401 points| zdw | 1 year ago |undeadly.org | reply

287 comments

order
[+] janosdebugs|1 year ago|reply
Having written an SSH server that is used in a few larger places, I find the perspective of enabling these features on a per-address basis by default in the future troubling. First, with IPv4 this will have the potential to increasingly penalize innocent bystanders as CGNs are deployed. Worst case, this will give bad actors the option to lock the original owner out of their own server if they have a botnet host in the same network. With IPv6 on the other hand, it is trivially easy to get a new IP, so the protection method described here will be completely ineffective.

From my experiments with several honeypots over a longer period of time, most of these attacks are dumb dictionary attacks. Unless you are using default everything (user, port, password), these attacks don't represent a significant threat and more targeted attacks won't be caught by this. (Please use SSH keys.)

I have seen experienced sysadmins create the test user with the password of "test" on a live server on port 22 because they were having an "autopilot moment". It got hacked within 20 minutes of going online and these mechanisms wouldn't have saved it, the attacker got in on the second or third try.

If you want to have a read about unsolved problems around SSH that should be addressed, Tatu Ylonen (the inventor of SSH) has written a paper about it in 2019: https://helda.helsinki.fi/server/api/core/bitstreams/471f0ff...

[+] crote|1 year ago|reply
> With IPv6 on the other hand, it is trivially easy to get a new IP

OpenSSH already seems to take that into account by allowing you to penalize not just a single IP, but also an entire subnet. Enable that to penalize an entire /64 for IPv6, and you're in pretty much the same scenario as "single IPv4 address".

I think there's some limited value in it. It could be a neat alternative to allowlisting your own IP which doesn't completely block you from accessing it from other locations. Block larger subnets at once if you don't care about access from residential connections, and it would act as a very basic filter to make annoying attacks stop. Not providing any real security, but at least you're not spending any CPU cycles on them.

On the other hand, I can definitely see CGNAT resulting in accidental or intentional lockouts for the real owner. Enabling it by default on all installations probably isn't the best choice.

[+] overstay8930|1 year ago|reply
> With IPv6 on the other hand, it is trivially easy to get a new IP, so the protection method described here will be completely ineffective.

I’m sure this will be fixed by just telling everyone to disable IPv6, par for the course.

[+] solatic|1 year ago|reply
Serious question: why doesn't OpenSSH declare, with about a year's notice ahead of time, the intent to cut a new major release that drops support for password-based authentication?
[+] skissane|1 year ago|reply
> I have seen experienced sysadmins create the test user with the password of "test" on a live server on port 22 because they were having an "autopilot moment". It got hacked within 20 minutes of going online and these mechanisms wouldn't have saved it, the attacker got in on the second or third try.

Is it possible to create some kind of reverse proxy for SSH which blocks password-based authentication, and furthermore only allows authentication by a known list of public keys?

The idea would be SSH to the reverse proxy, if you authenticate with an authorised public key (or certificate or whatever) it forwards your connection to the backend SSH server; all attempts to authenticate with a password are automatically rejected and never reach the backend.

In some ways what I'm describing here is a "bastion" or "jumphost", but in implementations of that idea I've seen, you SSH to the bastion/jumphost, get a shell, and then SSH again to the backend SSH – whereas I am talking about a proxy which automatically connects to the backend SSH using the same credentials once you have authenticated to it.

Furthermore, using a generic Linux box as a bastion/jumphost, you run the same risk that someone might create a weak password account–you can disable password authentication in the sshd config but what if someone turns it on? With this "intercepting proxy" idea, the proxy wouldn't even have any code to support password authentication, so you couldn't ever turn it on.

[+] mardifoufs|1 year ago|reply
Wait, how often do you connect to a ssh remote that isn't controlled by you or say, your workplace? Genuinely asking, I have not seen a use case for something like that in recent years so I'm curious!
[+] TacticalCoder|1 year ago|reply
> First, with IPv4 this will have the potential to increasingly penalize innocent bystanders... Worst case, this will give bad actors the option to lock the original owner out of their own server if they have a botnet host in the same network.

So instead of looking, like the author of these new options, for ways to make life for the bad guys harder we do nothing?

Your concerned are addressed in TFA:

> ... and to shield specific clients from penalty

> A PerSourcePenaltyExemptList option allows certain address ranges to be exempt from all penalties.

It's easy for the original owner to find the list of all the IP blocks the three or four ISPs he's legitimately be connecting from to that exemption list.

I don't buy your argument nor all the variation on the same theme: "There's a minuscule risk of X, so we absolutely nothing but saying there's nothing to do and we let bad guys roam free!".

There's nothing more depressing than that approach.

Kudos to the author of that new functionality: there may be issues, it may not be the panacea, but at least he's trying.

[+] hot_gril|1 year ago|reply
It's not quite fair, but if you want the best service, you have to pay for your own ipv4 or, in theory, a larger ipv6 block. Only alternative is for the ISP deploying the CGN to penalize users for suspicious behavior. Classic ip-based abuse fighter, Wikipedia banned T-Mobile USA's entire ipv6 range: https://news.ycombinator.com/item?id=32038215 where someone said they will typically block a /64, and Wikipedia says they'll block up to a /19.

Unfortunately there's no other way. Security always goes back to economics; you must make the abuse cost more than it's worth. Phone-based 2FA is also an anti-spam measure, cause clean phone numbers cost $. When trying to purchase sketchy proxies or VPNs, it basically costs more to have a cleaner ip.

[+] jimmaswell|1 year ago|reply
I like being able to log into my server from anywhere without having to scrounge for my key file, so I end up enabling both methods. Never quite saw how a password you save on your disk and call a key is so much more secure than another password.
[+] Latty|1 year ago|reply
And even with IPv4, botnets are a common attack source, so hitting from many endpoints isn't that hard.

I'd say "well, it might catch the lowest effort attacks", but when SSH keys exist and solve many more problems in a much better way, it really does feel pointless.

Maybe in an era where USB keys weren't so trivial, I'd buy the argument of "what if I need to access from another machine", but if you really worry about that, put your (password protected) keys on a USB stick and shove it in your wallet or on your keyring or whatever. (Are there security concerns there? Of course, but no more than typing your password in on some random machine.)

[+] mananaysiempre|1 year ago|reply
> I have seen experienced sysadmins create the test user with the password of "test" on a live server on port 22 because they were having an "autopilot moment".

pam_pwnd[1], testing passwords against the Pwned Passwords database, is a(n unfortunately abandoned but credibly feature complete) thing. (It uses the HTTP service, though, not a local dump.)

[1] https://github.com/skx/pam_pwnd

[+] chuckadams|1 year ago|reply
I'd love to penalize any attempt at password auth. Not the IP addresses, just if you're dumb enough to try sending a password to my ssh server, you're going to wait a good long time for the failure response.

Actually I might even want to let them into a "shell" that really screws with them, but that's far outside of ssh's scope.

[+] andix|1 year ago|reply
I had a similar experience with a Postgres database once. It only mirrored some publicly available statistical data, and it was still in early development, so I didn't give security of the database any attention. My intention was anyway to only expose it to localhost.

Then I started noticing that the database was randomly "getting stuck" on the test system. This went on for a few times until I noticed that I exposed the database to the internet with postgres/postgres as credentials.

It might have been even some "friendly" attackers that changed the password when they were able to log in, to protect the server, maybe even the hosting provider. I should totally try that again once and observe what commands the attackers actually run. A bad actor probably wouldn't change the password, to stay unnoticed.

[+] Too|1 year ago|reply
Interesting paper from Tatu Ylonen. He seem to be quick on throwing out the idea of certificates only because there is no hardened CA available today? Wouldn’t it be better to solve that problem, rather than going in circles and making up new novel ways of using keys? Call it what you want, reduced to their bare essentials, in the end you either have delegated trust through a CA or a key administration problem. Whichever path you choose, it must be backed by a robust and widely adopted implementation to be successful.
[+] yardstick|1 year ago|reply
> Worst case, this will give bad actors the option to lock the original owner out of their own server if they have a botnet host in the same network.

According to to the article, you can exempt IPs from being blocked. So it won’t impact those coming from known IPs (statics, jump hosts, etc).

[+] waihtis|1 year ago|reply
Agreed. In addition to the problems you mentioned, this could also cause people to drop usage of SSH keys and go with a password instead, since it's now a "protected" authentication vector.
[+] qwertox|1 year ago|reply
> innocent bystanders as CGNs are deployed

SSH is not HTTPS, a resource meant for the everyday consumer. If you know that you're behind a CGN, as a developer, an admin or a tool, you can solve this by using IPv6 or a VPN.

> Worst case, this will give bad actors the option to lock the original owner out of their own server

Which is kind of good? Should you access your own server if you are compromised and don't know it? Plus you get the benefit of noticing that you have a problem in your intranet.

I understand the POV that accessing it via CGN can lead to undesirable effects, but the benefit is worth it.

Then again, what benefit does it offer over fail2ban?

[+] hartator|1 year ago|reply
Yes, I agree. This seems a naive fix.

Just silencing all the failed attempts may be better. So much noise in these logs anyway.

[+] Grimeton|1 year ago|reply
Just throw away that document and switch to kerberos.

All the problems in this document are solved immediately.

[+] Someone1234|1 year ago|reply
This is great, and helps solve several problems at once.

I would like to remind everyone that an internet facing SSH with a password is very unwise. I would argue you need to be able to articulate the justification for it, using keys is actually more convenient and significantly more secure.

Aside from initial boot, I cannot think of the last time I used a password for SSH instead of a key even on a LAN. Support for keys is universal and has been for most of my lifespan.

[+] _JamesA_|1 year ago|reply
The number of expect scripts I find in production that are used to automate ssh password authentication is ridiculous.
[+] im3w1l|1 year ago|reply
I resent that every application needs its own special snowflake auth method. One uses a certain 2fa app. Another uses another 2fa app. Another uses emailed code. Another uses text code. Another uses special ssh keys. Another opens a prompt in the browser where I have to confirm. Another uses special scoped tokens.

Yes there are good reasons. But it is quite a hassle to manage too.

[+] joelthelion|1 year ago|reply
> internet facing SSH with a password is very unwise

If your password is strong, it's not.

[+] TacticalCoder|1 year ago|reply
> ... using keys is actually more convenient and significantly more secure.

And for those for whom it's an option, using U2F keys (like Yubikeys) is now easily doable with SSH.

So unless the attacker can hack the HSM inside your Yubikey, he's simply not getting your private SSH keys.

[+] KennyBlanken|1 year ago|reply
> I would like to remind everyone that an internet facing SSH with a password is very unwise.

Bullshit. You can have a terrible password and your system will still be nearly impossible to get into. Also, these attackers are usually looking for already exploited systems that have backdoor account/password combos, unless they are specifically attacking your organization.

Repeat after me: dictionary attack concerns have nothing to do with remote access authentication concerns.

Let's say my password is two common-use English words (100k-200k.) That's ten billion possibilities. Assume you hit on my password half-way through. That would be fifteen years of continuous, 24x7x365 testing at 10 password attempts per second....and then there's the small matter of you not knowing what my username is, or even whether you've got the right username or not, unless the ssh server has a timing-based detection attack.

The only argument for putting this functionality in the daemon itself is that by locating it in the daemon, it can offer advanced application-layer capabilities, such as failing auth attempts no matter what after the limit is tripped so that brute-forcing becomes more pointless - unless you get it right within the first few attempts, you could hit the right password and never know it. If they intend to implement features like that in the future, great - but if it's just going to do what fail2ban does, then...just run fail2ban.

Fail2ban has a higher overview of auth on the system, is completely disconnected from the ssh daemon in terms of monitoring and blocking, and the blocking it does happens at the kernel level in the networking stack instead of in userspace with much more overhead and in a proprietary system specific to SSH.

As a sysadmin, this is 'yet another place you have to look' to see why something isn't working.

[+] GordonS|1 year ago|reply
Another good option is making SSH only accessible over Tailscale or a VPN.
[+] timw4mail|1 year ago|reply
Any time you access an SSH connection from a different computer, you basically need the password.
[+] kelnos|1 year ago|reply
This seems like something I wouldn't want. I already use fail2ban, which does exactly the same thing, in a more generic manner. sshd is a security-sensitive piece of software, so ideally I want less code running in that process, not more.
[+] akvadrako|1 year ago|reply
The security sensitive parts of SSH run in a separate process. I would assume that most of the new code would be in the unprivileged part.
[+] idoubtit|1 year ago|reply
I've read the commit message in the post, and read it again, but I did not understand how it would be configured. The penalty system seems complex but only 2 parameters are mentioned.

From the documentation, one of these parameters is in fact a group of 8 parameters. I guess the separator is space, so one could write:

    PerSourcePenalties authfail:1m noauth:5m grace-exceeded:5m min:2m
See https://man.openbsd.org/sshd_config.5#PerSourcePenalties

Unfortunately, the default values are undocumented. So `PerSourcePenalties yes` (which will be the default value, according to the blog post) will apply some penalties. I did attempt to read the source code, but I'm reluctant to install a CVS client, two decades after dropping that versioning system.

[+] mananaysiempre|1 year ago|reply
The OpenBSD project provides a CVSWeb interface[1] and a GitHub mirror[2]. The portable OpenSSH project[2] that most of the world gets their OpenSSH from uses a Git repo[4] that also has a Web interface (at the same address) and a GitHub mirror[5]. Per the code there[6], the default configuration seems to be

  PerSourcePenalties crash:90 authfail:5 noauth:1 grace-exceeded:20 max:600 min:15 max-sources:65536 overflow:permissive
[1] https://cvsweb.openbsd.org/

[2] https://github.com/openbsd/src

[3] https://www.openssh.com/portable.html

[4] https://anongit.mindrot.org/openssh.git

[5] https://github.com/openssh/openssh-portable

[6] https://anongit.mindrot.org/openssh.git/tree/servconf.c?id=0...

[+] yjftsjthsd-h|1 year ago|reply
I've seen MaxAuthTries used for similar reasons, and of course fail2ban, but this seems like a nice improvement and it's built in which is probably a win in this case.
[+] ComodoHacker|1 year ago|reply
Will it really help today, when anyone with any serious intent doesn't launch their attacks from one or two standalone hosts, but buys botnet capacity?
[+] verandaguy|1 year ago|reply
I don't think this attempts to address botnet attacks, but to be fair, there are very few tools that you can just run on a single physical or VPS host that can effectively defend against a botnet. Frankly, most things that aren't Cloudflare (or in the same ballpark) will be ineffective against well-crafted botnet attacks.

This is useful in a defence-in-depth scenario, same as fail2ban. You might be able to defeat the odd hacker or researcher doing recon on your host, and sometimes that's good enough for you.

If you need botnet protection, you shop around for botnet protection providers, and you get a botnet protection solution. Easy as.

[+] sleepydog|1 year ago|reply
I'm not a fan of this feature. First, I don't think it's going to help all that much for the reasons other people have stated (it's easy to obtain a new IP, and using ssh key-only remote login nullifies most attacks anyway).

More importantly, though, is that it is difficult to debug why you can't login to a remote system, unless you've been diligent enough to setup remote logging and some kind of backdoor you can use in a pinch. I imagine many companies have some unimportant script running in the background that logs into a remote system over ssh, and the person who set it up left the company years ago. One password change/key rotation later, and suddenly 25% of employees cannot login to that remote system because the script got one of the office's 4 public IPv4 addresses blocked on the remote server.

It's very easy to say "you should manage your systems better, you should separate your networks better", and so on. But in my line of work (customer support), I only hear about the issue after people are already locked out. And I've been on many phone calls where users locked themselves out of their server that had fail2ban setup (ubuntu setup fail2ban by default in one of its releases).

[+] enasterosophes|1 year ago|reply
People keep mentioning fail2ban. I claim that both this new behavior in sshd, and fail2ban, are unprincipled approaches to security. Now, I know fail2ban is a crowd favorite, so let me explain what I mean by unprincipled.

This is the problem fail2ban (and now sshd) try to solve: I want a few people to log into my computer, so I open my computer to billions of other computers around the world and allow anyone to make a login attempt, and then I want to stop all the illegitimate attempts, after they were already able to access port 22.

It's simple Bayesian probability that any attempt to head off all those illegitimate accesses will routinely result in situations where legitimate users are blocked just due to random mistakes rather than malicious intent. Meanwhile, illegitimate attempts continue to come en masse thanks to botnets, allowing anyone with an ssh exploit the chance to try their luck against your server.

A more principled approach to security is to not roll out the welcome mat in the first place. Instead of opening up sshd to the world, allowing anyone to try, and then blocking them, instead don't open up sshd to the world in the first place.

1. If possible, only permit logins from known and (relatively) trusted networks, or at least networks where you have some recourse if someone on the same network tries to attack you.

2. If access is needed from an untrusted network, use wireguard or similar, so sshd only needs to trust the wireguard connection. Any attempt at illegitimate access needs to crack both wireguard and ssh.

With those one or two simple measures in place, have another look at your sshd auth logs and marvel at the silence of no one trying to attack you a million times per day, while also having confidence that you will never accidentally lock yourself out.

[+] kazinator|1 year ago|reply
If these people don't know what to do with themselves next that much, they should so something useful, like learn git, instead of implementing fail2ban-style features that nobody needs or wants in the software itself.

People who want this sort of thing and already have a single solution that handles multiple services have to complicate their setup in order to integrate this. They keep their existing solution for monitoring their web and mail server logs or whatever and then have this separate config to deal with for OpenSSH.

What if you don't want to refuse connections that exhibit "undesirable behavior" but do something else, like become a black hole to that IP address, and perhaps others in the IP range?

You want the flexibility to script arbitrary actions when arbitrary events are observed.

In my log monitoring system (home grown), the rules are sensitive to whether the account being targeted is the superuser or not.

[+] gnufx|1 year ago|reply
Do the people who are going on about fail2ban know whether that's even ported to, and included in, OpenBSD? I suspect not.
[+] Grimeton|1 year ago|reply
pam-script with xt_recent works just fine.

Everytime when an authentication fails, you add the ip address to the xt_recent list in /proc and in iptables you just check via --hits and --seconds and then reject the connection attempt the next time.

[+] password4321|1 year ago|reply
I would like to see support for blocking by client identifier, though if it were a default all the bots would recompile libssh.

Until then this has been a great differentiator for Bitvise SSH.

[+] nazgu1|1 year ago|reply
Is it something that can replace fail2ban or sshguard?
[+] textninja|1 year ago|reply
A “SSHal credit score” tied to a pooled resource, yes, that will work out well! Kind of like how a used car purchase should come with all its tickets!

EDIT: To this feature’s credit, it’s not federated centrally, so a DDOS to nuke IP reputation would have its blast radius limited to the server(s) under attack.

[+] semi|1 year ago|reply
This is interesting but something i feel like id disable on most of my ssh servers as they are only exposed through a shared jump host, and I don't want users that have too many keys in their agent to cause the jump host IP to be penalized.

On the jump host itself it makes sense though

[+] a-dub|1 year ago|reply
ip addresses are kinda meaningless these days, and address based accounting and penalization can penalize legitimate users. (bitcoind has a banscore system, it's kinda cute but these kinds of things tend to be bandaidy)

it's a hard problem. wireguard has a pretty good attempt at it built into its handshaking protocol, but like all of these things, it's not perfect.

could maybe do something interesting with hashcash stamps for client identity assertion (with some kind of temporal validity window). so a client creates a hashcash stamped cookie that identifies itself for 30 minutes, and servers can do ban accounting based on said cookie.

[+] tonymet|1 year ago|reply
Can you trigger a command when they are in the "penalty box" ? it would be nice to firewall those sources so they stop consuming sshd cpu
[+] olooney|1 year ago|reply
This reminds me of Zed Shaw's Utu protocol from back in the day:

https://weblog.masukomi.org/2018/03/25/zed-shaws-utu-saving-...

I am not a crypto guy, but my understanding is that users can downvote each other, and the more downvotes a user gets the harder the proof-of-work problem they had to solve before they post. If you received enough hate, your CPU would spike for a couple of minutes each time you tried to post, thus disincentivizing bad behavior.

I see on github the project is officially dead now:

https://github.com/zedshaw/utu

[+] WhatIsDukkha|1 year ago|reply
This seems like a bad fix to the problem of insisting that ssh continue to only use TCP.

Wireguard only responds to a complete useful key from incoming UDP (as I understand). Probe resistant.

I get the legacy argument here but it seems like almost two decades of "this tcp thing has some downsides for this job"?