How to destroy someone who hosts stuff at Hetzner dedicated server

[turshija]

I'm using Hetzner services for several years so far (luckily only for personal stuff and friends minecraft server), and had this problem few times. Every time I said to myself "I will get away from Hetzner ASAP", but I always stay there. I would NEVER imagine to run a business hosted there at all, and here is why...

DDoS is a common problem many companies are facing, but Hetzner's policy on that is really crap. If someone starts DDoS on your dedicated, after several minutes they just shut down your dedicated from network, and send you an email like "We disabled your network because you have DDoS attack on your server. Write us an email to reenable your network". And of course, several hours later I saw that email and tell them "Okay, please enable my network", but boom, I will have to wait Monday, because their support that can ACTIVATE network on a dedicated works only from Mondays to Fridays ... And then the person who attacked me sends me anonymous email like "lol, I bought 5$ packet at [some random booter/network stresser website], and I have put you offline for few days for only 15 minutes of DDoS, HAHAHAHA"

So basically yea, start small flood from random VPS/dedicated or whatever that is 100mbit or more, leave it on for several minutes until Hetzners system automatically disable network from person you are attacking, and look at them being offline for few days :) I'm ordering a new dedicated from someone else now, no more Hetzner...

[MehdiEG]

It's worth putting this in context. Hetzner provides really beefy dedicated servers for ridiculously low prices [1].

You get great support (always had phone calls answered pretty much instantly and emails answered within a few minutes and all the techs I've dealt with knew what they were doing).

You can issue automated hardware resets and even get a remotely-controlled KVM attached to tweak the BIOS or regain access to your machine if you messed up the networking config (usually only takes a few minutes to get the KVM attached).

Orders for new hardware are also really fast - dealt with within the hour and often in under 15 minutes.

But there's no such thing as a free lunch. If you host at Hetnzer, you have to be aware of the reasons why they're so cheap, namely:

1) The servers are 100% unmanaged. They'll install new hardware for you if you ask them but everything else is up to you.

2) A lot of their hardware is desktop-grade, e.g. Intel Core i7 CPUs and non-ECC RAM. They do have some server-grade hardware in their high-end range however.

3) Their servers are in Germany. So you get quite a bit of latency if accessed from Asia or the West Coast of the US (see [2]).

4) They don't have any DDoS protection. In case of a DDoS, your server will get null-routed (but they tell you first). Again: 100% unmanaged. Up to you to deal with it. I've been lucky enough to not have to deal with a DDoS but my first port of call would probably be CloudFlare it it happened.

Provided that you're happy to do some sys admin, Hetzner is brilliant for a personal server, a CI server or even a prod server for a bootstrapped startup.

For literally next to nothing, you get a really powerful machine that will easily handle big traffic spikes without a breaking a sweat. And dedicated machine means that you get excellent and consistent CPU performance and disk I/O. If and when your startup takes off and you get funding, you can then choose between hiring a sys admin or moving to a more expensive host that offers a more managed setup.

[1] http://www.hetzner.de/en/hosting/produktmatrix/rootserver-pr...

[2] https://news.ycombinator.com/item?id=3898714

[wazoox]

ovh.com and online.net provides even cheaper servers, but with a DDoS filter by default. I'm very happy with both.

[Duckeh]

A lot of the people commenting don't seem to understand how hard it is to fend off such DDoS attacks. You either need some serious infrastructure (cloudflare style) or you need to buy equipment to mitigate attacks (like radware devices) or route it via a DDoS mitigation service (prolexic style). The one thing all these solutions have in common is that they are insanely expensive. People can buy a 1 gigabit DDoS for only a few bucks, whereas mitigating a 1 gigabit DDoS will cost you either $20K+ dollars for a mitigation device or some stupid amount of money to have a service like prolexic mitigate it for you. Services like cloudflare are a whole load cheaper but only provide basic reverse proxy protection and still leave your server vulnerable for attacks directed at it's IP instead of DNS name.

I can't say I've ever heard of Hetzner, but from the comments I'm reading they apparently offer servers for cheap. Bearing in mind how much money DDoS mitigation costs I don't see how they could handle this any other way without having to make some pretty serious investments (which in turn would make their hosting less cheap as the money has to come from somewhere, right?)

[patrickg_zill]

You can do some of it via BGP, which is a standard method for handling routing once you become any sort of server provider with multiple bandwidth providers. It is builtin to some Juniper devices already, for instance: http://njetwork.wordpress.com/2013/04/30/mitigating-ddos-att...

There are other ways to do it via BGP also. Plus there is null-routing, bandwidth limiting, etc.

[metabrew]

IRCCloud had to move off hetzner for this reason. We were continually getting ddos'ed, and hetzner showed no interest in working with us to try and mitigate.

At one point they just suggested we "ask the responsible parties to stop", and closed the ticket.

Now we are on Black Lotus. Expensive, but the regular 50mb-10gbit ddos attacks are mitigated just fine.

[metabrew]

...however, if you aren't concerned about ddos, I still recommend hetzner.

Excellent value for money dedicated servers, with good automated systems. You can remotely reboot a dedicated server into a recovery image and fix problems yourself. You can run the install process yourself too, so you get exactly what you want... except ddos mitigation.

[revelation]

That said, almost every (cheap) server hoster will have something in their TOS forbidding IRC use.

I have no particular insight into why IRC is so troubling, it used to be popular for malware botnets (C&C) and it attracts its fair share of script kiddies.

[tribaal]

Wow, thank you so much for irccloud, it's amazing! Paying customer since about 5 minutes into the beta here :) I didn't realise you founded last.fm too!

[glomph]

Was that not also because of efnet klining hetzner?

[Lazare]

Offtopic, but: IRCCloud is amazing.

[sillysaurus2]

Wow, you founded Last.fm and IRCCloud.

Thanks for both!

[spindritf]

Yup, pretty much. Those attacks have become a real problem because they can be ordered so cheaply and easily that even kids use them in Minecraft feuds. The channel takeovers of the 21st century.

OVH's much more tolerant in that regard (ie. they keep your server online if battered) and all their servers now include a mandatory anti-ddos protection[1]. Unfortunately, they're fighting turn-over and don't accept new orders.

[1] http://forum.ovh.co.uk/showthread.php?t=6661

[oellegaard]

So I manage quite a few servers at Hetzner and we were DDOS'ed quite a few times. First, they warn you and if you don't get back to them in 12-24 hours, then they will shut down your server.

Sounds like you were unfortunate, but this is not generally what they do.

[leokun]

What would you do after they warn you? It's not really under your control to fix is it.

[turshija]

I had that thing before (several months ago, also during DDoS), I received an email where they told me I have DDoS attack and if it continues, they will have to block my server. In this case however, I didn't get any warning email, just the one where they are telling me my dedicated is disconnected from network ...

[level09]

That sucks. I have moved many websites recently from EC2 to Hetzner. what they offer is really impressive and the difference is clear (probably 5x more resources/power for 25% of the Amazon price).

I guess I will still keep the server, but will have to work on a quick migration/failover plan in case I encounter something similar.

I have also started using cloudflare as my default DNS host, so that could also be a possible solution.

[turshija]

Cloudflare doesn't help if they DDoS your server's IP directly ... You can also "hide" your IP by activating CF on all subdomains (the orange cloud thingy), but people always find a way to find server's IP and attack it (the CF doesn't help there at all, they only filter packets that are going through their servers which your domains resolve to).

[Qantourisc]

Here is a simple solution and everybody is happy: re-enable it every hour, if DDoS continues, disable again.

Everybody is probably "happy" then: Customer-> their unusable DDoSed server is disconnected, but wasn't reachable anyway. But once the DDoS is over, it's back online. Provider -> they have their traffic routed to null. However, they will have to do some more work to get this working too. And not to mention happier customers.

[codexon]

Here is a forum that sells DDoS attacks. Attacks are much cheaper than protection.

http://www.hackforums.net/forumdisplay.php?fid=232

[verroq]

You are kidding me right, nobody takes skidforms seriously.

[andrew_wc_brown]

I had to do deal with DDOS attacks in the past and DDOSArrest worked like a charm to mitigate the problem.

[qohen]

Link for convenience:

http://www.dosarrest.com/

BTW, does anyone know how what their prices are like?

(Their site doesn't have seem to have pricing info, just "Get a Free Quote" forms.)

[csense]

How can DDoS mitigation devices distinguish between legit and malicious traffic? I'm not a networking expert, but it seems to me that if you're a website hosting a big file like the latest Ubuntu release, a legitimate client will say:

    GET /ubuntu-13.10-server-amd64.iso

and cost you 500 MB of traffic (or however big the ISO file is).

A DDoS is nothing more than thousands or millions of machines saying:

    GET /ubuntu-13.10-server-amd64.iso

How do the solutions others are talking about in this thread (DDoS mitigation provider or specialized hardware) tell the difference between DDoS traffic and legitimate requests?

[turshija]

That is something different, it is only used to waste bandwidth from someone (or potentially clogging server's upload, but its easy solvable), but in big DDoS attacks the attacker usually has several hundred thousands of zombies infected in his botnet, and then he orders all those zombies to spam packets at an IP he orders ... Every infected PC uses his maximum upload to target IP, resulting into something like this: http://d.pr/i/kmAn

If I'm online during the attack and check iptraf or tcpdump, I can see literally hundreds of different IPs spamming random stuff at me, completely overflowing my download until I get totally disconnected from server (time out), and I can do nothing about it, just watch it being offline ...

[lb0]

Wow, they detect the DDoS, but instead of blocking this they take off the servers?? Sounds ingenious..

Or are they unable to properly detect a DDoS and would also take off a server that hosts a web page mentioned on Hacker News?

How do other hosters handle this situation?

[4hthth4]

From what other people are saying, it sounds like Hetzner is the Walmart of service providers. You wouldn't see a traffic jam in the Walmart parking lot and then become indignant that they didn't have valet parking automatically start up to clear the parking lot traffic.

[onestone]

Rackspace basically do the same. I've had a Rackspace Cloud server null routed due to a plain vanilla SYN flood DoS attack (note the single D).

[devicenull]

Detecting these is trivial. Actually blocking them requires significant bandwidth capacity and equipment. Do you expect them to make that investment? You have the choice between cheap hosting and DDOS protected hosting. Buying cheap hosting then complaining that your host is not providing expensive services is silly.

[_s]

Use cloudflare or a similar service provider to mitigate such attacks?

[swinglock]

Not everything on the Internet is a web server.

[linas]

We had the same problem at Hetzner, the server was attacked on Saturday. We moved out. Hetzner is very cheap and you get what you pay for.

[turshija]

Yep, I would like to move out my files from Hetzner at this moment, but my server is locked, and I will have to wait Monday to get access to it ... Luckily I'm not hosting anything important on it and my business doesn't rely on them, or else I would be screwed very hard ...

[ianhawes]

Great tip. Does anyone know who Hetzner's largest customers are? Or at least major web services that host with Hetzner?

[aroch]

They have a "notable customers" page: http://www.hetzner.de/en/hosting/unternehmen/referenzkunden

There are a couple NBA.com subdomains and Der Speigel appears to be hosting 'local' CDNable content for their Germany-based readers

[AznHisoka]

Does this apply to servers that do NOT host websites? I host databases in Hetzner that aren't hosted in the same server as the website(they're in another provider)

[vertis]

In theory if they can find it, it can be DDOS'd. But not hosting public facing servers makes it much less likely to be a problem.

[Demiurge]

well this is good timing, just moved to hetzner last month and server mysteriously went awol yesterday until a reset...

[turshija]

Google this: hetzner hard disk failure They are putting faulty hard drives in their servers, and if you notice its faulty and tell them, they replace it with another faulty one (less faulty if you are lucky) ... Make regular backups to servers outside Hetzner network...

[na8ur]

Consider somthing else than a ddos - attack. I realized probs with the hardware (RAM and Bios - Update was done in the middle of the night in 10min after telling them my insights)

[bolder88]

FWIW, This is fairly standard.

Linode for example will null-route your linode for 24 hours if it's attacked.

It's quite irritating that hosting companies seem to see null-routing as a solution to a DDoS attack.

[zzzcpan]

   > This is fairly standard.

True.

   > It's quite irritating that hosting companies
   > seem to see null-routing as a solution to 
   > a DDoS attack.

Not everyone can afford a proper solution. In fact, I don't think anyone, except OVH, is able to offer a server for less than $100/month and include proper DDoS protection.

[devicenull]

What alternative would you propose? With a virtual machine an attack on one instance can effect everyone on the same machine. Also actually blocking the attack is very expensive.

[patrickg_zill]

If they can detect the DDOS, they should be able to mitigate it, right?

(EDIT: of course Hetzner could choose to mitigate the DDOS by any number of methods - but they choose not to, because they have made a conscious decision based on cost.)

[lucb1e]

It's like a traffic jam. You can solve it by stopping the incoming cars at the ramp or by making the road wider, but both are outside of your control when you're just running the toll gates at some point ahead. You can only ask your "host" to do that (in this example, the owner of the road).

[devicenull]

No. Detecting some types of DDOS attacks is pretty trivial. Just parse Netflow output and look for big bandwidth spikes. Actually blocking these requires significant effort to classify the attack to be able to block it without also blocking normal traffic.

[turshija]

when my dedicated was offline due to DDoS, I asked them to PAY for any kind of anti ddos protection, just to make my server online, and they refused that and told me they don't offer DDoS protection ...

[zzzcpan]

I don't think they can detect DDoS attacks, just an unusually high PPS.

[ye]

No.