M64.pl – how I learned that the default settings are not production settings

[buro9]

On the cheap Linode instances I setup for others, I always install this: http://www.configserver.com/cp/csf.html

It includes the equivalent of fail2ban, manages iptables, watches log files, can email on failed logins, processes gone mad, and a few other things through a very simple config file.

It's that simplicity of the config file that I like, it means I can hand it to someone and know that they're unlikely to break something due to misconfiguration or failure to actually apply something.

This is on top of obvious things like running ssh on a non-default port, turning off password auth and restricting ssh to named users only (never root).

The install script is easy enough: http://www.configserver.com/free/csf/install.txt so it's really the case that I can do whatever work I'm doing for someone (hello Wordpress) and this can be my simple security solution which at least means I'm sure that I haven't left every window open.

[aragot]

If you disable logging in with the root user, ...do you allow Sudo for another user? Then, what's the point?

[deoxxa]

Why do you run ssh on a non-default port?

[kybernetyk]

I hope you have killed and reinstalled the box because once a system has been compromised you can't ever trust it again.

[jnbiche]

I'm always amazed at the number of people who simply re-use servers that have been owned. It explains a lot.

[artursapek]

I know. I'm taking the risk with it for a few more days because I was already in the process of moving those apps to fresh boxes (the other app is http://mondrian.io). I'd rather not take them down because they get considerable traffic. At this point I'm just keeping the server alive until I can fully replace it - not using it to do anything new.

I did immediately change www's password again and de-privilege the ssh key that that box had been equipped with. In fact I've recently started always ssh-ing into servers with the -A flag so I don't have to give them keys at all.

So far I don't see any more mysterious activity. Thanks for the word of caution.

[michh]

In this case, if I read the article correctly, only a single account was compromised. I'm assuming the www user doesn't have sudo privileges (though I might be wrong as it shouldn't have a password either and it clearly did..).

Sure there might have also been a local privilege escalation vulnerability, so rootkits are definitely something to check for. Depending on the situation, re-installing might be less work, so then it'd make sense.

But especially if you're not the only one using the machine, having to reinstall the OS every time a single user fucks up and has 'password' for a password, it's going to get really tedious really quickly.

[scott_karana]

I don't think the www user has a password in any distro, by default.

Meaning one of his administrators must have set one (likely for temporary troubleshooting), kicking off the whole issue...

edit: if SSH access was indeed the first cause. Running any upload-and-run script as `www` would let them set the password themselves, I think.

[euank]

For anyone wondering how to properly troubleshoot in this manner without breaking things:

Run 'sudo su www -s /bin/bash'. using '-s /bin/bash' will override the usual nologin shell, and running 'su' as root will mean a passwordless account can be su'd too.

This will allow you to try accessing files and directories as if you had the user 'www's privileges without having to make the 'www' account regularly usable.

You should never set a real shell or password for any accounts that a real user will not be using.

[Maxious]

Probably an administrator troubleshooting, www user should also have a nologin shell by default which cannot be changed by a normal user via an uploaded script.

Maybe default settings are production settings after all!

[eli]

Disabling password logins is good, but there are other steps you could/should take. These are all pretty simple: https://library.linode.com/securing-your-server

Also, I would guess the attack is fully automated.

[hackerboos]

I'm scared noobs are going to blindly turn off passwords in their SSH configs without correctly setting up keys.

[nodesocket]

Also, heads up. Don't ever run Elasticsearch open to the world in iptables. This is actually very common because users want to use Kibana (http://www.elasticsearch.org/overview/kibana/). Dynamic scripting is allowed by default (< version 1.2.0) and be exploited easily. It is especially nasty if you run Elasticsearch as root (also very common).

[x0x0]

yeah, "special" people at my work did this. Along with installing / automatically starting elasticsearch on every ami so that every box could be rooted, not just the search cluster. And putting the fucking admin pam on the boxes instead of learning about agent forwarding. And not securing processes in any way -- why would you use chroot or docker, that would throw up obstacles in front of the nice chinese people who want to use your ec2 boxes.

it was a very long week.

[stiff]

I once had a box "bot-netted" by some automatic scanner after I set up postgres on it, being in a hurry after another box crashed unexpectedly. It is stunningly idiotic to me that SSH seems to be enabled by default for all users (at least on Ubuntu server).

[zaroth]

All users with a password set. So 'setting the password' has perhaps an unclear side effect.

I guess if you disable password logins, someone would need to get a pubkey into a user's ~/.ssh directory.

[xtrumanx]

I've been considering moving an app off of Heroku and to Digital Ocean so I could learn the ropes of administrating a Linux server but I feel there's so much I don't know that I should know if I wanted to effectively manage my own server.

If I got hacked with m64.pl, I'd get as far as running top and killing the process. His immediate assumption was that someone got a file that file on to his server and so he went on to check the logs and found all the failed login attempts. I would have wondered what the file was, googled around and sat around confused.

Luckily deploying a new instance on Digital Ocean sounds simple enough so I would probably just do that but that approach would leave me with a sense that I don't know what I'm doing at all. There was a time when my solutions to problems on Windows machines was to restart and/or format but I don't do that anymore but still meet people who do. I'd rather not start from square 1 when I make the leap to Linux.

What are the things one should know to legitimately say they can manage their own Linux server? I could try to just wing it and solve every problem as I encounter them but for right now I'd just like to identify the gaps in my knowledge.

[cel1ne]

Beware of this: http://unix.stackexchange.com/a/8886/11172 To fully lock down SSH you have to check PAM config as well.

My 2 cents about defending against scripters:

* SSH: non-standard port, no passwords, special user-group who is allowed to login.

* WWW: redirect everything to https. For some reason bot-attacks dropped from hundreds to none over night. :D

[mistaken]

The m64.pl executable seems to be a vanilla version of cpuminerd. By default it connects to http://127.0.0.1:9332 and mines using the scrypt algorithm. It could have been started with different parameters, or the port was tunneled to a pool. Would be interesting to know the ip.

off: does anyone know which font he uses in the terminal? Looks awesome.

[euank]

Assuming you're referring to the font of the blog's text...

Firefox: Right click, inspect element, click the "fonts" tab on the right.

Chrome: Right click, inspect elmeent, look at "Computed" styles on the right, find Rendered Fonts.

Ironically, the answer is he's not using a font. It uses Times New Roman on my Chrome and DejaVu Serif on my firefox because those are my default system fonts, not because he specified them. If you look at the source of the webpage, you'll see he has zero stylesheet links, all his styles are inline in <style> blocks... So searching font in that one source page is sufficient to find he only styles code blocks, the rest is default.

If you're referring to some other font, I have no clue, sorry!

[viraptor]

While someone could set the password on the www user manually, I get the impression that the author is the only admin and would remember that. Since the only authenticated user is www, it's likely someone just exploited his application / webserver and set the password on www for later ssh logins.

The author will soon know that if the situation repeats without the ssh access.

[dbalan]

Using a tool like fail2ban is also favourable. Most of the bots will give up and look for other targets even if you ban them for a hour.

[microtonal]

The problem with fail2ban is that you have another possible vulnerable program that an external user can feed information to (although very restricted).

I have always been surprised that fail2ban is so popular, since iptables can do rate limiting, etc. So, it's easy to block most attacks with the in-kernel firewall:

http://www.debian-administration.org/articles/187

[artursapek]

I actually did find fail2ban in my Googling. Seems like a decent tool, have to do more research before I decide whether to use it from now on.

[kbar13]

fail2ban is a log prettifier, doesn't actually add anything to security.

[mqzaidi]

You should also run logwatch - it will show you all the attempts that keep happening on any public servers running ssh. I once had to look at a box compromised, which was then used to brute force other servers, and it contained a file with 100-200 passwords for other hosts it brute-forced.

[Fuxy]

I was hoping for a more detailed account of how he managed to log in as www.

That looked like a pretty amateur attempt yet he managed to log in as www.

Was there no password for www?

Why was it even available to SSH into?

This leaves a lot of questions unanswered.

[zaroth]

A couple funny things; the intruder changing the www password, and running a mining script which is going to generate basically zero satoshi before getting shut down.

I guess change the password so someone else can't break in after you?

I can't explain the mining script. Maybe they want to alert you the box is owned, as a means to find only targets who don't wipe their system after being infected? What's the point...?

[Kiro]

Even if the script only manages to mine very little it would still be worthwhile considering this is probably 100% automated.

[adders]

The script appears to be be at least part minerd, a program for bitcoin/litcoin mining.

Details of minerd at https://bitcointalk.org/index.php?topic=55038.0

[andyhmltn]

Seeing a bruteforce in the logs isn't at all uncommon. I see it every time I have a new server. Turn off password authentication and disable root login via SSH. That way, you're golden!

[nathell]

A similar if not same thing: http://ubuntuforums.org/showthread.php?t=2203663

[keeperofdakeys]

And if you do want to use password authentication, simply whitelist the allowed logins. This assumes you are using a strong password.

[deckiedan]

I have an ssh-access group, then the ssh server config is limited to that group. It makes it very easy to keep track of, and also works pretty well for maintenance with ansible.

[threeseed]

Other options: Run it on a different port, use certificates or YubiKey for authentication, firewall it to your home computer, use Fail2ban or Denyhosts.

[edoceo]

Duh. Always lock SSH down