I'm still seeing the same number of attempts on my public SFTP servers. As a funny side note I found that by going through the hardening steps on ssh-audit [1], most of the bots can't even negotiate a connection. I only see them because I configured verbose logging. They seem to be using really old ssh libraries in the bot code that severely limit the ciphers available to them. Another interesting side effect is that my ssh negotiation is much faster on high latency connections after the hardening.
[Edit]: I forgot to add that if testing the ssh hardening from that site, be sure to do this locally where one has console access and the same version of sshd or if on remote nodes make sure that remote consoles are working first to avoid being locked out of the remote nodes. Modifying the client ciphers first and testing remote connections prior to modifying the servers is the lowest risk.
I found disabling aes ciphers alone killed 90% of the bots with an occasional ECC only bot sneaking through. I only have 2-3 options for MAC, KEX and cipher. Certain mobile clients won't work (older embedded ssh library) but others support more modern configs. But I started my journey with [1] and have slowly tweaked and slimmed it down over the years.
> [after] hardening steps [...] most of the bots can't even negotiate a connection
Yep, same here, except I'm using [tinyssh], which organically does not support password-based auth, or any ciphers other than ssh-ed25519, curve25519-sha256, and [email protected].
I'm hoping that the folks running the bots use their bots' failure to negotiate a connection as a filter in a similar, but opposite, way that Nigerian scammers use ridiculously scammy, all caps, emails as a filter. It would significantly reduce log spam if each bot only ever connected once.
Anyone who responds to a Nigerian scam email is much more likely to be able to be successfully scammed. It is more efficient, for the scammer, that less credulous people get filtered out in the first step, so as not to waste time on them.
Similarly, any site where an ssh bot fails connection because the site operator removed e.g., weaker default ciphers is much less likely to support password auth for it to even be possible to have a password brute forced. It is more efficient for the bot operator to just move along to a more likely target.
Same for www. Ratchet up the TLS/SSL - https://ssl-config.mozilla.org/ - go for modern and you'll see a lot of failed connections from bots and scanners.
Also, if you don't use any other IP block list, do use DROP from Spamhaus: https://www.spamhaus.org/drop/ - that is small enough that you can run it on the webserver if you don't have much control over your connection to the outside world.
...be sure to do this locally where one has console access...
I found I was able to restart sshd without bouncing my ssh connection. I think this is because sshd forks off children? Of course I made sure everything was working before killing the existing connection!
Thanks for the link to the audit service! We went from "F" to "A+"...
You are correct about the libraries not supporting modern ssh :)
I ran a bunch of honeypots for a while and grabbed hassh (like ja3 but for ssh) signatures, most bots are using old fucking libssh/libssh2/paramiko that simply can't talk to modern hosts.
Fail2ban blocked 1087 ip addresses in the last week, which seems normal.
I reset it and it has blocked eleven ip addresses in the last hour, mainly China and Digital Ocean as usual.
Just to see what happens, I'v tried sending abuse reports about ssh brute force, vnc brute force and phishing sites, by the standard method of doing a whois lookup on the ip for the abuse email address.
Some server and web hosting companies take the inconvenient approach of having an email auto-reply that says "we ignore all emailed abuse reports, you must use this web form", sometimes requiring a captcha.
When I reported a load of boxes attempting brute force logins, most complaints disappeared into the void.
I got a few responses from virtual server providers saying "no response from the customer after two weeks so we shut down the box" and one CC:ed email that appeared to be from an end user saying "we have reinstalled the box and changed the password."
I had a DigitalOcean droplet running Selenium grid that I didn't secure properly. I got an email that Sony Entertainment had reported my IP for botting login attempts in the Playstation store. I think I had 48 hours to respond to DigitalOcean.
Shut down Selenium right away, checked the logs, yep someone had taken over my selenium grid. It was just a small personal project on a sever I had forgot was still running, but at least they took reports seriously when my account was causing things.
I am genuinely surprised that the virtual server providers responded meaningfully to the abuse complaints and took action. Good job reporting the scum.
On a related note, set up a website a few days ago. Brand new domain, newly opened port. But I've rented this VPS for a while so I guess its IP is already on a few lists.
But anyways, it's interesting watching the logs for what I assume are tests of known exploits.
GET /.gitconfig HTTP/1.0" 422 Unprocessable Entity
GET /.git/config HTTP/1.0" 404 Not Found
GET /owa/auth/x.js HTTP/1.0" 404 Not Found
GET /ecp/Current/exporttool/microsoft.exchange.ediscovery.exporttool.application HTTP/1.0" 404 Not Found
GET /owa/auth/logon.aspx?url=https%3a%2f%2f1%2fecp%2f HTTP/1.0" 404 Not Found
GET /system_api.php HTTP/1.0" 404 Not Found
GET /c/version.js HTTP/1.0" 404 Not Found
GET /streaming/clients_live.php HTTP/1.0" 404 Not Found
GET /stalker_portal/c/version.js HTTP/1.0" 404 Not Found
GET /stream/live.php HTTP/1.0" 404 Not Found
GET /flu/403.html HTTP/1.0" 404 Not Found
POST /vendor/phpunit/phpunit/src/Util/PHP/eval-stdin.php HTTP/1.0" 404 Not Found
GET /?XDEBUG_SESSION_START=phpstorm HTTP/1.0" 404 Not Found
GET /backups-dup-lite/dup-installer/main.installer.php HTTP/1.0" 404 Not Found
GET /%24%7B%28%23a%3D%40org.apache.commons.io.IOUtils%40toString%28%40java.lang.Runtime%40getRuntime%28%29.exec%28%22whoami%22%29.getInputStream%28%29%2C%22utf-8%22%29%29.%28%40com.opensymphony.webwork.ServletActionContext%40getResponse%28%29.setHeader%28%22X-Cmd-Response%22%2C%23a%29%29%7D/
That's why I recommend always pointing the default virtual host in the apache or nginx configuration to an empty static site, and making the real site visible only as a named virtual host (requiring the correct Host header), even when the server will be used only for a single site. Most of these automated exploit attempts will never send the correct Host header, and therefore will only see the default virtual host.
I run a honeypot network to produce a threatfeed. Over the last month I saw an increase in daily numbers from Aug 7th to 19th. About 30% increase. From the 19th until yesterday I saw a 20% decrease from the peak. Still above average right now slightly.
In fact, it's interesting. The time it takes to open a port and wait for a random attacker isn't measured in minutes. It's seconds. You can do this at home.
>For some numbers, over the past 7 days we had 24,000 attempts against 'root' and only 749 against the next most popular target, which is a login name ('admin') that doesn't even exist here. Just over 10,000 of those attempts came from a single IP address, and just four IPs made 1,000 or more attempts against anything. Besides root, only five login names had more than 100 attempts (and none of them exist here): 'admin', 'user', 'ubuntu', 'debian', and 'pi'. And only three machines saw more than 1,000 attempts (across all targeted login names).
I've been interested in showing others this type of data related to SSH honeypots. I'm currently running a cowrie honeypot and I put together this Flask web app wrapped over cowrie's SQLite db to display the data it collects.
So far, ~1/4 of the 85,000 or so SSH attempts have been from the same address in Brazil.
What happened is they have gotten better at identifying worthy targets and they spray instead of bruteforce. I have dealt with compromised cloud VMs recently enough due to bad ssh password, it was a crypto mining worm, the cloud indeed is a better place to spend your resources as an attacker because most people are lazy enough to allow entire /8s of their cloud provider and because the internet can't reach it, who cares?
We have take the approach that the only ports visible to internet traffic are un-authenticated ones... everything else must traverse ipsec or wireguard. This gives us completely silent ssh ports.
We found strongswan examples pretty useful... it's basically pick your config and copy/paste. Wireguard only has 'one way' to set it up which is a whole 'nother level of simplicity.
This has completely silenced our firewall and ssh logs. Nobody is out blasting ipsec or wireguard connection probes around the internet (for now).
Alternatively, you can hide your ssh port behind port knocking… which I find to be simple and elegant - and requires almost no configuration or tooling.
Funny - I had noticed something similar in the last couple of months, but hadn't though to look in to it more. I was seeing 30,000 to 50,000 attempts a day, but over the last two weeks, it's around 10,000.
Perhaps it's like mining cryptocurrency - all the low hanging fruit has been found, so there's hardly much sense in continuing to waste resources looking for more.
Is it possible the majority of attempts were actually being run by a single botnet, and it either shut down or the owner changed the attack payload? Perhaps it's just been running for the last few years, with the botmaster apathetic, forgetting about it or maybe even had lost control.
I think you're right about the low-hanging fruit part: it seems unlikely anyone would bother to put serious effort or money (eg: the resources of a botnet) into this type of attack anymore, and the OP seems to support this idea:
> For some numbers, over the past 7 days we had 24,000 attempts against 'root' and only 749 against the next most popular target, which is a login name ('admin') that doesn't even exist here. Just over 10,000 of those attempts came from a single IP address, and just four IPs made 1,000 or more attempts against anything.
If you setup Fail2Ban and/or switch SSH from the default port 22 to something in the low 5 digits, then you should get very few attacks.
Someone above talked about using layered/defense-in-depth approach to avoid zero-days and that's also a good idea, but it's more involved / sometimes impractical.
I did a small study on brute force ssh attacks a couple of years back. Switching the port does a really good job dodging brute force attacks, as long as you don't use 2222. Paradoxically, port 2200 appeared to be a safe choice.
> [...] blocking of only a few IPs is disproportionately effective at stopping brute force SSH attacks here. Also, since we already block Internet logins to 'root', we're in almost no danger. No matter how many times they try, they have literally no chance of success.
This assumes OpenSSH, OpenSSL and other things in between the client and the server don't have ready to exploit security vulnerabilities. These large-scale scanners are a long term investment when there's a zero-day vulnerability to exploit in OpenSSL etc. That's why a layered/defense-in-depth approach is preferable.
The recall for these is also terrible. it's like going after a mosquito with a bazooka. case in point, we've accidentally blocked entire universities on a shared IP and the entire AWS West region doing this
I still see a good amount, but I think I am better at blocking them through distributed firewall sensors and such. Also, I think there is an opportunity cost. When you hear how lucrative other hacking avenues are, why bother with crappy little accounts that probably dont monetize for that much vs activities that are far more lucrative
This isn't my experience at all. Its a constant stream, people try common first names, and they're from lots of different people. Certain cloud websites in particular host the bots. A weird amount of requests come from IP addresses associated with security companies - just enough to be a nuissance for a website that doesn't actually get any traffic and is just for me to test things out on.
[+] [-] LinuxBender|3 years ago|reply
[Edit]: I forgot to add that if testing the ssh hardening from that site, be sure to do this locally where one has console access and the same version of sshd or if on remote nodes make sure that remote consoles are working first to avoid being locked out of the remote nodes. Modifying the client ciphers first and testing remote connections prior to modifying the servers is the lowest risk.
[1] - https://www.ssh-audit.com/
[+] [-] technonerd|3 years ago|reply
https://stribika.github.io/2015/01/04/secure-secure-shell.ht...
[+] [-] nousermane|3 years ago|reply
Yep, same here, except I'm using [tinyssh], which organically does not support password-based auth, or any ciphers other than ssh-ed25519, curve25519-sha256, and [email protected].
[tinyssh] https://tinyssh.org/
[+] [-] sillystuff|3 years ago|reply
Anyone who responds to a Nigerian scam email is much more likely to be able to be successfully scammed. It is more efficient, for the scammer, that less credulous people get filtered out in the first step, so as not to waste time on them.
Similarly, any site where an ssh bot fails connection because the site operator removed e.g., weaker default ciphers is much less likely to support password auth for it to even be possible to have a password brute forced. It is more efficient for the bot operator to just move along to a more likely target.
[+] [-] gerdesj|3 years ago|reply
Also, if you don't use any other IP block list, do use DROP from Spamhaus: https://www.spamhaus.org/drop/ - that is small enough that you can run it on the webserver if you don't have much control over your connection to the outside world.
[+] [-] password4321|3 years ago|reply
[+] [-] jessaustin|3 years ago|reply
I found I was able to restart sshd without bouncing my ssh connection. I think this is because sshd forks off children? Of course I made sure everything was working before killing the existing connection!
Thanks for the link to the audit service! We went from "F" to "A+"...
[+] [-] mmsnberbar66|3 years ago|reply
why?
[+] [-] nibbleshifter|3 years ago|reply
I ran a bunch of honeypots for a while and grabbed hassh (like ja3 but for ssh) signatures, most bots are using old fucking libssh/libssh2/paramiko that simply can't talk to modern hosts.
[+] [-] MrRadar|3 years ago|reply
> NIST P-curves are possibly back-doored by the U.S. National Security Agency
Is there any evidence of this? This seems extremely paranoid.
[+] [-] ajp11|3 years ago|reply
I reset it and it has blocked eleven ip addresses in the last hour, mainly China and Digital Ocean as usual.
Just to see what happens, I'v tried sending abuse reports about ssh brute force, vnc brute force and phishing sites, by the standard method of doing a whois lookup on the ip for the abuse email address.
Some server and web hosting companies take the inconvenient approach of having an email auto-reply that says "we ignore all emailed abuse reports, you must use this web form", sometimes requiring a captcha.
When I reported a load of boxes attempting brute force logins, most complaints disappeared into the void.
I got a few responses from virtual server providers saying "no response from the customer after two weeks so we shut down the box" and one CC:ed email that appeared to be from an end user saying "we have reinstalled the box and changed the password."
[+] [-] squeaky-clean|3 years ago|reply
Shut down Selenium right away, checked the logs, yep someone had taken over my selenium grid. It was just a small personal project on a sever I had forgot was still running, but at least they took reports seriously when my account was causing things.
[+] [-] nousermane|3 years ago|reply
It is trivial to record netflow data (and most networks do that already), and then verify incoming abuse reports against those records.
[+] [-] orangepurple|3 years ago|reply
[+] [-] unknown|3 years ago|reply
[deleted]
[+] [-] m4jor|3 years ago|reply
[+] [-] zimpenfish|3 years ago|reply
[+] [-] Frotag|3 years ago|reply
But anyways, it's interesting watching the logs for what I assume are tests of known exploits.
...and the list goes on.[+] [-] cesarb|3 years ago|reply
[+] [-] kgeist|3 years ago|reply
[+] [-] banana_giraffe|3 years ago|reply
https://gist.github.com/Q726kbXuN/85c947a5d37cb01f72f82318d0...
This is two weeks of 404s
[+] [-] technonerd|3 years ago|reply
https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blo...
[+] [-] bestes|3 years ago|reply
[+] [-] technion|3 years ago|reply
[+] [-] incomingpain|3 years ago|reply
In fact, it's interesting. The time it takes to open a port and wait for a random attacker isn't measured in minutes. It's seconds. You can do this at home.
>For some numbers, over the past 7 days we had 24,000 attempts against 'root' and only 749 against the next most popular target, which is a login name ('admin') that doesn't even exist here. Just over 10,000 of those attempts came from a single IP address, and just four IPs made 1,000 or more attempts against anything. Besides root, only five login names had more than 100 attempts (and none of them exist here): 'admin', 'user', 'ubuntu', 'debian', and 'pi'. And only three machines saw more than 1,000 attempts (across all targeted login names).
Here's a public threatfeed. http://charles.the-haleys.org/ssh_dico_attack_with_timestamp...
He has had a fair number of attackers in the last week.
I feel like 'stopped' isn't the right word.
[+] [-] twalla|3 years ago|reply
[+] [-] protomyth|3 years ago|reply
CISCO, User, adempiere, admin1, adminstrator, alarm, alem, amanda, ansible, apache2, apc, arma3server, as, assembla, azure, azureuser, bamboo, bilbomeakine, bill, blackvoid, carlos, cds, centos, chinochan, cloud, cloudera, codeship, contributor, csgo, csgoserver, csserver, debian, default, demo, deployer, dev, device, devops, docker, dominion, ec2, ec2-user, ecs, elastic, elasticsearch, elsearch, engineer, es, esuser, eurek, for, ftp, ftp_admin, ftp_user, ftpadmin, ftpserver, ftptest, ftpuser, git, gitlab, glassfish, gmodserver, gpadmin, grav, grid, guest, hadoop, hduser, hostmetrics, jboss, jenkins, jira, john, joomla, junkbust, kafka, kevin, kibana, kubernetes, lighthouse, linkl, linkxess, localadmin, mail, marketing, mc, mcserv, mcserver, michael, mike, minecraft, momo, mongodb, netgear, netscreen, nexus, odoo, office, opc, oper, oracle, osm, osmc, pi, port, postgres, pvm, r00t, redmine, rust, rustserver, sanlang, secscan, service, spark, sphinx, squid, steam, steve, suhelper, super, support, svn, svpilot, systemd, systems, systemx, tbnet, teamspeak3, telecomadmin, test, test2, test3, test4, test6, test7, testftp, testuser, tomcat, ts3, ts3bot, ts3server, ubnt, ubuntu, uftp, upload, uploader, user, usuario, uucp, vagrant, vpn, vpnssh, web, webadmin, weblogic, wordpress, wp, wy, xbmc, xinyi, z, zabbix, zerotier-one, zyfwp
I get the basic service names, but what the heck is going on with "z" and "kevin"?
[+] [-] ackatz|3 years ago|reply
So far, ~1/4 of the 85,000 or so SSH attempts have been from the same address in Brazil.
If anyone's interested -- https://live-honeypot.com
[+] [-] badrabbit|3 years ago|reply
[+] [-] exabrial|3 years ago|reply
We found strongswan examples pretty useful... it's basically pick your config and copy/paste. Wireguard only has 'one way' to set it up which is a whole 'nother level of simplicity.
This has completely silenced our firewall and ssh logs. Nobody is out blasting ipsec or wireguard connection probes around the internet (for now).
[+] [-] rsync|3 years ago|reply
[+] [-] johnklos|3 years ago|reply
Perhaps it's like mining cryptocurrency - all the low hanging fruit has been found, so there's hardly much sense in continuing to waste resources looking for more.
[+] [-] gregmac|3 years ago|reply
I think you're right about the low-hanging fruit part: it seems unlikely anyone would bother to put serious effort or money (eg: the resources of a botnet) into this type of attack anymore, and the OP seems to support this idea:
> For some numbers, over the past 7 days we had 24,000 attempts against 'root' and only 749 against the next most popular target, which is a login name ('admin') that doesn't even exist here. Just over 10,000 of those attempts came from a single IP address, and just four IPs made 1,000 or more attempts against anything.
[+] [-] m4jor|3 years ago|reply
[+] [-] AtlasBarfed|3 years ago|reply
Amazon and other cloud providers using key-based ssh as the entrance to their instances probably pushed it to mainstream.
[+] [-] i_have_an_idea|3 years ago|reply
Someone above talked about using layered/defense-in-depth approach to avoid zero-days and that's also a good idea, but it's more involved / sometimes impractical.
[+] [-] m000|3 years ago|reply
[+] [-] kQq9oHeAz6wLLS|3 years ago|reply
[+] [-] _wldu|3 years ago|reply
[+] [-] _iven_|3 years ago|reply
[+] [-] alpb|3 years ago|reply
This assumes OpenSSH, OpenSSL and other things in between the client and the server don't have ready to exploit security vulnerabilities. These large-scale scanners are a long term investment when there's a zero-day vulnerability to exploit in OpenSSL etc. That's why a layered/defense-in-depth approach is preferable.
[+] [-] jms703|3 years ago|reply
[+] [-] gxt|3 years ago|reply
[+] [-] Gasp0de|3 years ago|reply
[+] [-] attentive|3 years ago|reply
No reason to expose ssh to the internet.
[+] [-] throwaway888abc|3 years ago|reply
"One of the things I've learned from this is that targeted blocking of only a few IPs is disproportionately effective at stopping brute force"
This is very much true also for other types of attacks/scan/etc.
Are you blocking Azure/OVH range on firewall ?
[+] [-] hackernewds|3 years ago|reply
[+] [-] mdtancsa|3 years ago|reply
https://thecyberwire.com/podcasts/research-saturday/247/note...
The numbers in that story (even at the low end) are eye popping :(
[+] [-] c3534l|3 years ago|reply
[+] [-] senorqa|3 years ago|reply