One thing I've found amusing is you can cut down SSH hacking attempts to almost zero (at the cost of compatibility with legacy systems) by using very progressive encryption. If you force use of ChaCha20/Poly1305, ED25519 and so forth in sshd_config, you'll see these attempts almost disappear. There's probably an observation to be made about just how overwhelmingly many of these attempts come from fairly old software setups.
I did that and it bit back when it prevented Transmit from connecting to my servers, with only a generic "Unable to connect" message popping up. This was fixed fairly recently[0], but it's a reminder that using a restricted set of ciphers may not only prevent illegitimate connections from going through.
I can confirm from the same experience, however, that it prevents most hacking attempts from ever going to the authentication stage.
- Have a stand alone SSH server that is something like a R-Pi[type B] running an OS that gets patches regularly via unattended-upgrades and reboots itself at least once a week.[could also be a minimal VM like AlpineOS if you need Gbps+ line speed]
- Have this R-Pi and your network gear plugged into a UPS that can withstand at least a couple of hours of power outage.
- Use non-standard ports on your perimeter FW for forwarding to the R-Pi.
- Run fail2ban or something similar. Set a bantime of at least an hour[3600 sec] after no more than 5 attempts in 600 sec. This is mostly to discourage anyone who stumbles across your perimeter listening port.[which enough will] Password login will be disabled anyway, but a bot might not check for allowed auth types.[nmap -Pn -p 22 --script ssh-auth-methods <target>]
- No passwords allowed in /etc/ssh/sshd_config[PasswordAuthentication no]. Use public key based authentication.[PubkeyAuthentication yes] Put the public SSH key[id_rsa.pub] of any machine that you wish to authorize into ~/.ssh/authorized_keys on your R-Pi. This way you can authorize and de-authorize remote machines[with particular users] at will.
- add 'AllowUsers xxxx yyyy' to your /etc/ssh/sshd_config
- No root logins allowed![PermitRootLogin no]
- Maybe limit via perimeter or on the R-Pi which source IP ranges are allowed to access to SSH.[whitelisting]
- Periodically review your logs of perimeter FW, R-Pi FW and /var/log/auth.log to see what's going on. It should be pretty quiet, but look at them anyway once a week.
- Sleep well knowing that you have a simple, layered and robust defense strategy that also has power-event survivability.
Others have mentioned port-knocking which is a cool trick but not something that I typically use in an actual daily defense strategy because I'm not sure how much value it really adds.
Don't use your perimeter device to host SSH services. It will not get patches fast enough when vulns come up.
While I completely agree and encourage people to use all of your suggestions, this single step - not being on port 22 - has easily provided the most benefit. When sshd listened on port 22, I regularly saw attacks that would run through thousands of common passwords at high speed[1]. After moving to a random port, it was over a year before I saw another hostile login attempt.
It may be easy to discover the server with a simple port scan, but in practice the attacks seem to focus on the easy targets.
> fail2ban
Fail2ban is great and highly configurable. I could accept having to manually reset occasional false positives, so my fail2ban will immediately (no retries) ban on the first use of anything that is known to be malicious such as any attempt to login as root with a password, asking to login with an account name like [admin, mysql, squid], or any apache access log matching /\/phpMyAdmin/, etc.
[1] Several logins per second over ADSL could use a large fraction of the upload bandwidth.
"Others have mentioned port-knocking which is a cool trick but not something that I typically use in an actual daily defense strategy because I'm not sure how much value it really adds."
I don't know how much it adds, but it is non-zero.
The knockd daemon is rock solid[1] and your ssh port traffic goes down to zero (other than your own use).
Port knocking has no place in security by itself but I think it's a wonderful addition to a layered defense - my favorite one, in fact.
Fairly standard stuff, definitely interesting to see all the IoT credentials attempred.
I'd recommend SSHGuard over fail2ban though, I seem to remember the version of fail2ban in the Debian repos completely choking on IPv6 and failing open which is obviously undesirable.
I feel like fail2ban failing open is a good thing. If it fails closed you can't SSH into the server. If it fails open it's just as though you aren't using fail2ban.
Actually, my biggest fear with fail2ban is it failing and locking me out of a remote server. If it indeed always fails open then there's no reason for me not to use it.
Though, I don't know how it could be guaranteed to always fail open.
My experience with SSHGuard was it treated monitoring events (specifically from monit) as failed attempts and blocked further attempts marking the service as unreachable.
I'm sure this can be configured but fail2ban does not block such attempts.
I wonder why the author set up a dedicated SSH python server which mimics some unix userspace as a honeypot. Why not a VM with a real standard widespread linux distribution to see what attackers actually do? Obviously their codes are not very sophisticated that they stop when "uname" fails with certain arguments.
Changing the SSH port proved to be most successful. Yes, of course you can find the port with a port scan. But it keeps the logs clean. And I don't want to waste CPU cycles on some brute-forcing idiots.
Is using non standard ports a reasonable approach? On one hand, it's kind of a security-by-obscurity measure, and it's also a (very minor) inconvenience to real users. However if it's not being used in place of other reasonable security measures, I'm not really sure what's bad about it, but it does feel a bit janky to me.
Same goes for other non-public services such as VPN.
What ports should be used? I think clearly you don't want to use other well known services (eg, running ssh on port 25 is just going to fill your logs with failed connection errors). Should you use ports >1024? >10000?
Years ago (actually probably more than 10 years ago now) I actually switched from port 22 to 622 on my server because I wanted to get rid of the noise of people attempting to bruteforce my SSH regularly. It worked perfectly for years but surprisingly I've had a handful of bruteforce attempts on port 622 in the past year or so. Nothing particularly targeted either, always the same typical default users (root, admin, oracle, mysql etc...). I wonder who bothers doing that.
Since I disabled password logging on all accounts and my key is stored on an HSM I'm not too worried about it though.
I never understood why people would use fail2ban or similar approaches. Just set proper password and move on. SSH traffic is tiny, it's not a problem. You'll never be hacked with a proper password. Let them waste their time.
You aren't infallible. While it's easy enough to secure a minimal server with only SSH access, every time you add another service, provide access to a new user, or change a configuration, there's a chance you'll miss something or make a dumb mistake. Running Fail2Ban is a very cheap and easy method to ensure that, even if you do fuck up somewhere, there's another layer of security that should prevent malicious access in most cases (nothing will save you if you set the pass to admin/admin, e.g.).
The failed attempts make it harder to monitor for other attacks because of the noise in log files, network traffic, etc. and if an attacker IP is blocked early they can't try more effective attacks.
I originally set up fail2ban because I got fed up with the noise of the hard disk going tick...tick...tick every 15 seconds or so, as another password guessing bot login attempt got logged and the disk heads had to seek as the logfile was written out to disk...
Do you ever examine logs? Fail2ban (or even better, using a nonstandard port or both) keep the noise out of lots to the point that you can still examine them manually.
Without it, 99.99% of the average ssh log is failed back attempts.
I have SSH with key auth everywhere, so I have no security worries.
But when sshd ends up pegging an entire core, and I can't login myself anymore, then fail2ban (or even whitelisting IP ranges for ssh) becomes necessary.
Nothing wrong with this, but a strong password for all Unix accounts (or, ideally, only allowing SSH with public key authentication) is essentially all you need. Unless you're using a dumb password, you're far more likely to be compromised by an application or OPSEC flaw (like someone getting into the email you used to register your server hosting account) than some crazy SSH compromise.
I wrote a little utility to keep ssh ports off unless a totp packet is sent to a UDP port. I haven't worked on it much since the initial proof of concept. There are several things that could be done to improve it (open only for the address that sent the packet, symmetric key for safer secret distribution, etc). Shared key would involve encrypting time and only open if the decrypted time matches current.
The totp packet is a full hash of time + shared code so brute force is completely infeasible.
It's currently in python, it would be much better in something like go with daemon monitoring. It is very simple, and it practically eliminates attempts. If it became widely known there would probably be attempts on the udp port. An additional improvement would be randomize the listening port based on the time+hash:
This all is easy if you only get a handful of login attempts. But when you get more, and more, it becomes an issue. On one of my servers it got so bad that SSH was at the end constantly using an entire CPU core.
I don’t have logs from that server, but here are logs – just the failed attempts – of 2 months from a server that was less severely affected: https://s3.kuschku.de/public/failed_ssh [327M]
And that was despite already blocking massive areas of IP space already.
My servers generally have an IPv4 address and a whole IPv6 subnet. I like to disable SSH except on a specific IPv6 address.
Many botnets just focus on IPv4 and there is a lot more territory to scan on IPv6. I usually couple that with moving to a non-standard port, enforcing modern encryption (ed25519 or ChaCha20, etcc), fail2ban, require SSH keys, etc
I think that by using obviously easy passwords anyone who really is looking for something is going to go to the next target. After all impossible to believe anything of value is behind 1234567 or changeme as a password. Even 'baseball' makes slightly more sense. Besides someone running an ssh server is not exactly mr computer newbie. Maybe a desktop you pick out of the trash might contain something interesting if that was the password.
In a classic CIA honeypot I would imagine the women are carefully matched to the target. If you are "James Bond" you get a better looking woman to try and trap you. If you are "Wallace Shawn" (in looks) you would probably figure out really quickly it's a trap if the woman looked like Angelina Jolie.
I found fail2ban is necessary but not sufficient. It requires some initial number of failed attempts before the attacker is blocked.
To prevent even those initial attempts, I use multiple layers of defenses such as country based blocking or blacklist based blocking. If an attacker slips through despite all this, I blacklist them manually in /etc/hosts.deny.
> It absolutely shocked me that it is worth the time of these drive-by attacks to try combinations like root/password, root/root, or root/admin. Apparently they have enough successes using those incredibly insecure combinations that it is worth their time.
It is fairly evident to anyone that has looked at the traffic that a honeypot gets that most of the activity is automated.
Most of it is purely botnets, some of it is automated password guessing followed by manual login. I think the amount of people manually trying passwords is insignificantly small.
That's just obviously nonsense? Closing the port does not change anything about the attempts. Nor about the success rate of the attempts, if you aren't being an idiot with insecure passwords.
Someday I'm going to put all the different pieces together and have things like SMTP & SSH auth failures, hosts hitting my spamtraps, bots scanning for open 23/TCP, and so on, to where I end up with one central host collecting all those events and injecting /32 routes for those IPs towards null0 in my core.
How configurable is fail2ban? Is it service based? Am I in danger of locking out a legitimate user like someone who makes 25 ajax requests within the span of 3 seconds? Can I make it work for authenticated HTTP endpoints as well?
Looks like the graph was generated from the Top 10 only. Obviously every item appears exactly once, so they all have the same 10%. The presentation as a graph is useless.
[+] [-] ejrv|8 years ago|reply
[+] [-] asclepi|8 years ago|reply
I can confirm from the same experience, however, that it prevents most hacking attempts from ever going to the authentication stage.
[0] https://library.panic.com/releasenotes/transmit5/#5-1
[+] [-] maxk42|8 years ago|reply
[+] [-] anonu|8 years ago|reply
[+] [-] ethbro|8 years ago|reply
[+] [-] tempfs|8 years ago|reply
- Have a stand alone SSH server that is something like a R-Pi[type B] running an OS that gets patches regularly via unattended-upgrades and reboots itself at least once a week.[could also be a minimal VM like AlpineOS if you need Gbps+ line speed]
- Have this R-Pi and your network gear plugged into a UPS that can withstand at least a couple of hours of power outage.
- Use non-standard ports on your perimeter FW for forwarding to the R-Pi.
- Run fail2ban or something similar. Set a bantime of at least an hour[3600 sec] after no more than 5 attempts in 600 sec. This is mostly to discourage anyone who stumbles across your perimeter listening port.[which enough will] Password login will be disabled anyway, but a bot might not check for allowed auth types.[nmap -Pn -p 22 --script ssh-auth-methods <target>]
- No passwords allowed in /etc/ssh/sshd_config[PasswordAuthentication no]. Use public key based authentication.[PubkeyAuthentication yes] Put the public SSH key[id_rsa.pub] of any machine that you wish to authorize into ~/.ssh/authorized_keys on your R-Pi. This way you can authorize and de-authorize remote machines[with particular users] at will.
- add 'AllowUsers xxxx yyyy' to your /etc/ssh/sshd_config
- No root logins allowed![PermitRootLogin no]
- Maybe limit via perimeter or on the R-Pi which source IP ranges are allowed to access to SSH.[whitelisting]
- Periodically review your logs of perimeter FW, R-Pi FW and /var/log/auth.log to see what's going on. It should be pretty quiet, but look at them anyway once a week.
- Sleep well knowing that you have a simple, layered and robust defense strategy that also has power-event survivability.
Others have mentioned port-knocking which is a cool trick but not something that I typically use in an actual daily defense strategy because I'm not sure how much value it really adds.
Don't use your perimeter device to host SSH services. It will not get patches fast enough when vulns come up.
[+] [-] pdkl95|8 years ago|reply
While I completely agree and encourage people to use all of your suggestions, this single step - not being on port 22 - has easily provided the most benefit. When sshd listened on port 22, I regularly saw attacks that would run through thousands of common passwords at high speed[1]. After moving to a random port, it was over a year before I saw another hostile login attempt.
It may be easy to discover the server with a simple port scan, but in practice the attacks seem to focus on the easy targets.
> fail2ban
Fail2ban is great and highly configurable. I could accept having to manually reset occasional false positives, so my fail2ban will immediately (no retries) ban on the first use of anything that is known to be malicious such as any attempt to login as root with a password, asking to login with an account name like [admin, mysql, squid], or any apache access log matching /\/phpMyAdmin/, etc.
[1] Several logins per second over ADSL could use a large fraction of the upload bandwidth.
[+] [-] rsync|8 years ago|reply
I don't know how much it adds, but it is non-zero.
The knockd daemon is rock solid[1] and your ssh port traffic goes down to zero (other than your own use).
Port knocking has no place in security by itself but I think it's a wonderful addition to a layered defense - my favorite one, in fact.
[+] [-] ransom1538|8 years ago|reply
PasswordAuthentication no
^ it should return that. The end.
[+] [-] lol768|8 years ago|reply
I'd recommend SSHGuard over fail2ban though, I seem to remember the version of fail2ban in the Debian repos completely choking on IPv6 and failing open which is obviously undesirable.
[+] [-] jazoom|8 years ago|reply
Actually, my biggest fear with fail2ban is it failing and locking me out of a remote server. If it indeed always fails open then there's no reason for me not to use it.
Though, I don't know how it could be guaranteed to always fail open.
[+] [-] h1d|8 years ago|reply
[+] [-] sametmax|8 years ago|reply
[+] [-] ktpsns|8 years ago|reply
[+] [-] gsich|8 years ago|reply
[+] [-] gregmac|8 years ago|reply
Is using non standard ports a reasonable approach? On one hand, it's kind of a security-by-obscurity measure, and it's also a (very minor) inconvenience to real users. However if it's not being used in place of other reasonable security measures, I'm not really sure what's bad about it, but it does feel a bit janky to me.
Same goes for other non-public services such as VPN.
What ports should be used? I think clearly you don't want to use other well known services (eg, running ssh on port 25 is just going to fill your logs with failed connection errors). Should you use ports >1024? >10000?
[+] [-] dingaling|8 years ago|reply
It's a big quiet void out there in sixspace.
[+] [-] simias|8 years ago|reply
Since I disabled password logging on all accounts and my key is stored on an HSM I'm not too worried about it though.
[+] [-] VilniusHacker|8 years ago|reply
[deleted]
[+] [-] vbezhenar|8 years ago|reply
[+] [-] daenz|8 years ago|reply
[+] [-] larkeith|8 years ago|reply
[+] [-] vxl|8 years ago|reply
[+] [-] pm215|8 years ago|reply
[+] [-] beagle3|8 years ago|reply
Without it, 99.99% of the average ssh log is failed back attempts.
[+] [-] nhooyr|8 years ago|reply
[+] [-] vuln|8 years ago|reply
[+] [-] kuschku|8 years ago|reply
But when sshd ends up pegging an entire core, and I can't login myself anymore, then fail2ban (or even whitelisting IP ranges for ssh) becomes necessary.
[+] [-] cup-of-tea|8 years ago|reply
[+] [-] arca_vorago|8 years ago|reply
nftables instead of iptables
port-knocking
non-standard port
key+pass access/auth
ip whitelist
good logging
ED25519 wherever possible!!!
[+] [-] Artemis2|8 years ago|reply
[+] [-] meowface|8 years ago|reply
[+] [-] thickice|8 years ago|reply
[+] [-] cbzbc|8 years ago|reply
[+] [-] nerdponx|8 years ago|reply
[+] [-] daveguy|8 years ago|reply
The totp packet is a full hash of time + shared code so brute force is completely infeasible.
It's currently in python, it would be much better in something like go with daemon monitoring. It is very simple, and it practically eliminates attempts. If it became widely known there would probably be attempts on the udp port. An additional improvement would be randomize the listening port based on the time+hash:
https://dhj.io/posts/2017/05/16/totp-knock/
[+] [-] kuschku|8 years ago|reply
I don’t have logs from that server, but here are logs – just the failed attempts – of 2 months from a server that was less severely affected: https://s3.kuschku.de/public/failed_ssh [327M]
And that was despite already blocking massive areas of IP space already.
[+] [-] m0nty|8 years ago|reply
http://denyhosts.sourceforge.net/
[+] [-] deftnerd|8 years ago|reply
Many botnets just focus on IPv4 and there is a lot more territory to scan on IPv6. I usually couple that with moving to a non-standard port, enforcing modern encryption (ed25519 or ChaCha20, etcc), fail2ban, require SSH keys, etc
[+] [-] gist|8 years ago|reply
In a classic CIA honeypot I would imagine the women are carefully matched to the target. If you are "James Bond" you get a better looking woman to try and trap you. If you are "Wallace Shawn" (in looks) you would probably figure out really quickly it's a trap if the woman looked like Angelina Jolie.
[+] [-] kamaraju|8 years ago|reply
To prevent even those initial attempts, I use multiple layers of defenses such as country based blocking or blacklist based blocking. If an attacker slips through despite all this, I blacklist them manually in /etc/hosts.deny.
If there is any interest, my scripts are in https://github.com/KamarajuKusumanchi/hosts.deny/
[+] [-] codetrotter|8 years ago|reply
It is fairly evident to anyone that has looked at the traffic that a honeypot gets that most of the activity is automated.
Most of it is purely botnets, some of it is automated password guessing followed by manual login. I think the amount of people manually trying passwords is insignificantly small.
[+] [-] znpy|8 years ago|reply
No open port = no hacking attempts.
Investing a week of on-and-off studying and tinkering with openvpn is really paying off.
[+] [-] zAy0LfpBZLC8mAC|8 years ago|reply
That's just obviously nonsense? Closing the port does not change anything about the attempts. Nor about the success rate of the attempts, if you aren't being an idiot with insecure passwords.
[+] [-] TheArcane|8 years ago|reply
[+] [-] jlgaddis|8 years ago|reply
[+] [-] fareesh|8 years ago|reply
[+] [-] WillieStevenson|8 years ago|reply
https://livesshattack.net
[+] [-] jwilk|8 years ago|reply
[+] [-] raimue|8 years ago|reply