top | item 8692537

Incident Report – DDoS Attack

146 points| alainmeier | 11 years ago |blog.dnsimple.com | reply

38 comments

order
[+] latch|11 years ago|reply
I need to learn to let things go, but: https://news.ycombinator.com/item?id=4280515

I've been a DnsMadeEasy customer for a while (they had an outage ~4 years ago from a 50Gbps attack), but once my year is up, I'm switching to Route53. The addition of the Geo DNS Queries was key for me. It isn't clear to me why I shouldn't pick Route53. DnsSimple's unlimited queries seems nice, but I kinda like having actual scaling costs forwarded to customers.

[+] kyledrake|11 years ago|reply
I've had a similar thought RE using Route53 for Neocities. Here's the problem with Route53 though. If you get a DDoS attack using it, it's quite plausible that you would be charged for resources used in the DDoS attack. A recent Vice article discussed this: http://motherboard.vice.com/read/inside-the-unending-cyber-s...

DDoS is a nasty problem. We've received a DDoS attack that shut the entire site down for days. We can't use Cloudflare because they don't support wildcard domains without their very expensive plan. I've also heard stories from people using Cloudflare that have still not been able to resolve DDoS issues (I'm not knocking Cloudflare, they're a great company that does a really good job fighting this very hard problem, but sometimes even they have trouble with it).

I'll be completely honest and say that I have no idea how to solve this problem. It's really, really, really hard. Switching to different service providers won't get you very far against the monster DDoS attacks that some people can execute.

[+] b0k|11 years ago|reply
I really don't understand why some of these low-grade DNS hosting services are so popular when Route53 is available. With Route53 you get a top-grade DNS service that is equivalent, if not better, than the enterprise hosted DNS solutions but at the price of the low-end consumer style services.

I swear by Route53, it is the only service I use on AWS and I have moved a lot of my clients over to it.

[+] aeden|11 years ago|reply
"and even then you can still be screwed if your bandwidth is saturated"

Which is exactly what happened in this case. It sucks to be on the receiving end of this. We couldn't defend against it and let our customers down, and that hurts me deeply. We choose one approach to defense, which was internal, and that was a mistake. We're going to work on rectifying that now.

[+] kator|11 years ago|reply
> A new customer signed up for our service and brought in multiple domains that were already facing a DDoS attack. The customer had already tried at least 2 other providers before DNSimple. Once the domains were delegated to us, we began receiving the traffic from the DDoS.

I'm curious did they know this in advance or discovered it after the fact?

I often wonder about business models where the core expense is "unlimited and free". The reality is there is nothing unlimited or free for the service provider. It seems with a business model like this you open yourself to people abusing your service either by accident or by choice. Imagine poor Mr. Customer here who most likely was having horrible problems thinking to themselves "These guys can do it and for free, if I go to X service they'll cost me a lot of money".

I'm a big believer in business models that incentivize both parties properly. I'm sure in general this service provider is arbitraging the 99.9% of domains that barely need any services. That said it only takes a couple of "opps" customers to drive your operational costs through the roof.

[+] aeden|11 years ago|reply
Anthony from DNSimple here. We discovered it after the fact, via a tip from other DNS providers.
[+] b0k|11 years ago|reply
"unlimited" plans are subsidised by low utilization users who are getting less than what they paid for.

to pull it off properly as a service provider, you really need to have a solid understanding of user usage patterns.

one of the big problems that tips the low/high utilization ratio unfavorably is that unlimited plans that are primarily marketed for being unlimited tend to attract users in the high utilization bracket.

so the challenge for service providers is not just understanding users and understanding that ratio but figuring out how you are going to market to, and signup, those users who will be in the low utilization bracket and will essentially be paying for something they won't be user (which is hard to do)

it isn't hard to find case studies of companies that launch optimistically with one pricing plan around unlimited, to then only go back and revise their pricing and break promises because they didn't understand their users and were unable to market to and signup low utilization users.

one recent example is Bitcasa

[+] stephenr|11 years ago|reply
The solution here is one for customers, not providers.

Manage your DNS at one location on "master" (potentially a "private" server with IP restricted access and zone transfer ACLs).

Setup 2+ accounts with "DNS providers" that support incoming zone transfers - that is, they can operate as "slave" DNS servers, pulling records automatically from your "master" (once access rules are set of course) and returning results directly to clients making DNS queries.

Most "Secondary DNS" packages are < $50 year, so use a few, and don't worry about individual DNS networks being burnt to the ground.

[+] jhealy|11 years ago|reply
It seems like inbound and outbound zone transfers aren't offered by a number of providers (like AWS). Do you know of a list of DNS providers that support either option?
[+] abalone|11 years ago|reply
So who do you think the "well-known third-party service that provides external DDoS protection using reverse DNS proxies" is they're going to use now?

CloudFlare?

[+] EwanToo|11 years ago|reply
I would assume Prolexic or Incapsula, assuming they're using a high end provider (which they should, DDOS attacks against smaller DNS providers being so easy to carry out).
[+] crystaln|11 years ago|reply
Hopefully not. CloudFlare is remarkably unreliable for a service that claims to improve uptime.
[+] cm2187|11 years ago|reply
Out of curiosity, what are the follow ups of an attack like that? The perpetrators are probably using their own servers or compromised clients or servers. Would DNS Simple follow up on this with the abuse/complaint dept of the ISP of the attackers? Are ISP typically responsive to abuse and complaints? If they are not is there any way to black list blocks of IPs assigned to ISP who do not care about being the source of DDoS attacks?

Investing in anti DDoS devices is important but even more important is for the perpetrators to face the consequences of their acts (or anyone who lets his machine being used by pirates - terminating or suspending their contract would be a fair response).

[+] milos_cohagen|11 years ago|reply
What was the overall makeup of the attack traffic? For example, 50% tcp syn, etc.