I strongly believe it's not possible to safely run a site without DDoS protection for all servers anymore. Anyone with $20 can take down anything on Digital Ocean, Linode, Hetzner, and many others. Or they can run up a huge bill for you on AWS. I would love to use Cloudflare but I can't afford $6000/mo for DDoS protection on my servers with the wildcard requirements we need. Linode may have solved their DDoS problems with their own stuff, but what about their customers' VPSes?
I really wish people would start taking DDoS more seriously. It's really not something we can just null route servers for anymore. It's becoming a very serious problem. It's not going away, it's amplifying and getting far worse.
I'm also not sure how effective it would be, but it would be nice to see the FBI, NSA or whomever spend at least as much time fighting these DDoS warlords as they did persecuting whistleblowers and trying to shove backdoors into cryptography.
I think that effort would be better spent encouraging (see Forcing) ISPs to start dropping forged traffic at their borders.
IMO there should be significant penalties for network operators who do not drop obviously I forged traffic. How long has that rfc been around now and how little adoption has it seen?
Of all the DDoS protection solutions you could spend $6000/mo on CloudFlare is probably the worst choice, hell Voxility will give you better service for $600 a month (And on top of that, they don't give the NSA unfettered access to your all of your traffic).
Don't buy CloudFlare for just DDoS protection, their pricing and products are far from competitive.
Layer 7 attacks are the new hotness in DDoS. If you have a big enough botnet (either conventional botnet, or hijacked browsers), you can do them, and they're often quite effective.
Fundamentally, layer 3/4 are usually amplification. Those are still effective, and very efficient for the attacker, but they will someday (5y? 10y?) be blocked by closing up sources amplification. Address spoofing address at layer 3/4 might get addressed by BCP 38, Vixie's good fight, etc., but not holding my breath.
By the time all that happens, attackers will have moved on to layer 7 attacks. Those can target the weakest parts of your stack, and with a large botnet, even the act of blocking the IPs in the wrong place can add enough overhead to hurt. With a huge botnet of hijacked browsers, blocking everyone affected becomes a DoS vector in itself, since some of those are your own legitimate attacks.
The big problem for DDoS mitigation is that this requires much deeper knowledge of the protected application. It's hard to just put a box inline, or an unmodified cloud service, and have it block the attacks. There's both good science and great engineering to be done, by developers, platform vendors, and specialty anti-DDoS providers, to block this emerging kind of attack.
Wow, still a lot of people fighting over whether or not Linode is a good company. It's a shame we don't get to see how <hipster hosting company of the month> copes with 80gbps of DDoS on a single DC.
I'm personally happy with Linode. They have a seriously tough technical issue to deal with —as much working out what's happening as how to stop it— and they seem to be doing a fairly top job at staying afloat. My servers haven't gone down. Any downtime in the last four years has been my fault.
So even if they are targets of some ludicrously powerful botnet, I'd rather stay with them than let the bastards doing this win. The attack isn't hurting my business or my clients and each incident we go through, the lower the chances of it ever being a problem in the future.
On a more serious note, governments keep moaning on about encryption but botnets are still a much greater direct threat to national security.
Where's the Linode founder(s) in all this, and why couldn't they have kept customers informed? It seems like a lone network engineer has been left to deal with a potentially company destroying event.
I was personally in the room, and in agreement, when running a real grown-up AS with carrier transit was proposed to Chris Aker as early as 2010, maybe 2009, to avoid this very scenario and many others like it. It's not really news. I have tremendous respect for the engineer who proposed it and fully believe he could have executed on this when Linode still had four facilities and 360 MB Linodes were the norm. I'm not saying that to toot my own horn (really, I'm not "I told you so" or arrogant like that), but there are very specific reasons that this wasn't done for as long as possible. I lack recent context, but the Linode decision-making culture was for many years completely driven by one individual who worked to spend as little on infrastructure as humanly possible.
Even once growth really took off and revenue started making these big shifts in strategy viable, the mindset was still to be lean and scrappy. The minimal capital expenditure strategy had benefits early on and allowed Linode to maintain an incredible margin and support explosive growth, but they were too slow to start thinking like a grown-up company when it started to matter, and it's coming back to bite both on security (with almost zero investment; just enough to pass PCI-DSS) and things like this.
When I heard they bought the Philadelphia building, for example, I was very surprised because that's not the Chris I knew. We lobbied for a Philadelphia office for years. Could be a good sign regarding decision-making culture for the future, but hard to say.
Don't read me as bad blood or anything, as I wish Linode no ill will (I actually hope they can turn this perceived slump around), it's just educational to see the consequences of choices and mindset catch up with a company. I learned a lot about management style while working there and contrasting with subsequent employers.
> Our longest outage by far... can be directly attributed to frequent breakdowns in communication
I have direct experience with Linode staff breakdowns in communication because of a security problem before the December attacks.
The problem affected many Linode customers and included risks to confidential information such as billing.
The Linode staff communication was terrible. The problem was severe and ended up with Linode on a blacklist of companies that are not suitable for hosting.
I have to agree with tptacek: do not use Linode for anything, and if you do now, make plans to switch to a new provider.
To end on a happy note, I migrated the project to Rackspace, and the Rackspace staff communication is excellent.
I'd like to learn more about these blacklists so I can factor that in when choosing a vendor, do you have a link? Google is just showing me pages of vendors trying to sell me hosting when I search.
As hard as this may be to believe, tptacek's opinions are not the gospel of technology. I've been with Linode for several years and their staff has been top notch.
>Layer 7 (“400 Bad Request”) attacks toward our public-facing websites
I really wonder what that is supposed to mean, Linode has mentioned it multiple times but not elaborated on what sort of an attack this is.
I personally haven't ever head of a "400 bad request"-attack.
Edit: Yeah, I know what Layer 7 floods are :), but I'm pretty sure "400 bad request" floods are something Linode came up with, so that could use some elaboration by them.
My guess is strategies like this but it isn't clear. [e.g. Requests designed to slow/increase the processing per-request to create a log jam at the web application level ]
Perhaps they're saying: an attack on OSI Layer 7 (web apps themselves) that result in users receiving HTTP 400 status codes in response to their requests.
OSI Layer 7 would actually refer to all "applications", but the mention of "400 Bad Request" implies specifically web applications were affected.
I am pretty amazed Linode didn't have their own IP Transit up to this point. Their colo provider is Newark charges some pretty high prices from what i've seen.
My plan is to keep saying this on Linode threads, just in case there are people who have missed it. Take my advice or leave it:
Please don't use Linode. If you are using it now, make immediate plans to switch. If you have friends who have things built on Linode servers, tell them to switch.
My plan is to keep saying this on Starfighter threads, just in case there are people who have missed it. Take my advice or leave it:
Please don't use Starfighter. If you are using it now, make immediate plans to switch. If you have friends who have used Starfighter, tell them to switch.
Feel how dickish this sounds without giving any reason whatsoever?
Yeah, unless you're willing to elaborate on the reasons behind this statement, it's hard to give it a lot of credence. In fact, your reputation is the only reason I'd give it any credence at all, without some explanation.
We've been happy Linode customers for a while now, and definitely prefer Linode to where we were before (Rackspace, via the Slicehost acquisition). I'm not opposed to moving to something like DigitalOcean or other but right now I'm seeing any compelling reason to make a move.
Just a comment on my experience: the last time one of these threads appeared (after reading several prior), I followed tptacek's advice and shut down the few remaining Linode instances I had. Linode handled it professionally: I removed my nodes, canceled my account online in a couple clicks, and requested a refund of the credit on my account (I had pre-paid for a year, and Linode refunded the time remaining). The payment was issued quickly and without confrontation (though they did ask, by email, for the last six digits of my credit card for "verification") I then deleted my account.
For all those who want some sort of backup: There is a search function here. If you search for Linode, you'll find a long string of security blunders. You really won't have to try hard. (PS. I'm a current Linode customer)
Could you elaborate? We've been their happy customer for 4 years now and while this incident did hit us with several hours of downtime, we are definitively not gonna switch just because of this.
What are we supposed to use? How do I know the host I pick has any better security practices or that they just haven't been owned so far? Which other VPS providers have essential features like private networking?
Please don't say AWS, I have no interest in learning that overcomplicated mess.
They mention segregating their customers into separate /24s, and consequently having to assign an IP from every one of these subnets to the router for use by the customer as a gateway.
Is there any reason why they couldn't get rid of these by having customers set up a static route to the "primary" IP of the router (migration / configuration issues aside)?
No guess at motive? Did someone ask for ransom before these started? Is one of the Linode subscribers hosting censorship-evasion technologies? Or is this one just some very determined kids having fun over holiday break?
How? I thought CloudFlare only protected HTTP? Can you have it reverse proxy a DNS server or is Linode using CloudFlare as the host for ns1.linode.com now?
Yeah, it's called Virtual DNS (vDNS); essentially a DNS application proxy.
(email me if you want more info; it's not really ideal for small sites, it's better to just use cf for hosted DNS then, since it's free, but we're happy to do vDNS for people who can't do hosted. Mainly providers, but also some enterprise customers with special DNS needs. It's a pretty cool technology.)
> after some stubborn transit providers finally acknowledged that their infrastructure was under attack and successfully put measures in place to stop the attacks.
Care to elaborate why it took them so long to ack? And name them so I know who to avoid in the future (or route around)!
> blackholing is a blunt but crucial weapon in our arsenal, giving us the ability to ‘cut off a finger to save the hand’ – that is, to sacrifice the customer who is being attacked in order to keep the others online
There is something very ironic about this. They have a policy which instead of addressing the problem actively assists anyone wanting to attack their customers. No surprise that these customers have been complaining about this practice for a long time. But until now it was Somebody Else's Problem so they didn't bother figuring out some proper (or at least less terrible) solution. Now this lack of preparedness bit them in the ass...
I'd posit that 98% of providers from whom you can acquire budget VPS will do the same thing. The practice is not unique to Linode; why should a network you're paying $20-$100 do everything they can to keep a target online and threaten other customers?
Contrary to popular opinion, if you're getting DoS attacked, you're either (a) popular enough to start thinking about adult-size pants for your transit strategy or (b) inviting the attention by your choice of content or activities. In years of hosting, I started to know the targets of DoS attacks by name. You have to own at least a little bit of responsibility, and mitigate on your own end if you're going to be inviting that kind of attention; IRC and controversial blogs are the usual suspects here, but that's probably changed recently as I've been out of the hosting game for a while.
Linode has few options for reacting other than the one they use. I know that sucks, but it's how it is.
Riding his motorcycle across Europe, occasionally sharing photos from vacation with his plebeian workers in #linode-staff who are earning $38k and can't afford to take any vacation.
It wouldn't work. That's not what CloudFlare does (right? they didn't do BGP last I heard). You'd need something like Black Lotus, now owned by Level3, for that.
[+] [-] kyledrake|10 years ago|reply
I really wish people would start taking DDoS more seriously. It's really not something we can just null route servers for anymore. It's becoming a very serious problem. It's not going away, it's amplifying and getting far worse.
I'm also not sure how effective it would be, but it would be nice to see the FBI, NSA or whomever spend at least as much time fighting these DDoS warlords as they did persecuting whistleblowers and trying to shove backdoors into cryptography.
[+] [-] click170|10 years ago|reply
IMO there should be significant penalties for network operators who do not drop obviously I forged traffic. How long has that rfc been around now and how little adoption has it seen?
[+] [-] ryanlol|10 years ago|reply
Don't buy CloudFlare for just DDoS protection, their pricing and products are far from competitive.
[+] [-] rmdoss|10 years ago|reply
https://incapsula.com
https://sucuri.net/website-firewall/
Both a lot cheaper and do a great job protecting against ddos.
[+] [-] rdl|10 years ago|reply
[+] [-] rdl|10 years ago|reply
Fundamentally, layer 3/4 are usually amplification. Those are still effective, and very efficient for the attacker, but they will someday (5y? 10y?) be blocked by closing up sources amplification. Address spoofing address at layer 3/4 might get addressed by BCP 38, Vixie's good fight, etc., but not holding my breath.
By the time all that happens, attackers will have moved on to layer 7 attacks. Those can target the weakest parts of your stack, and with a large botnet, even the act of blocking the IPs in the wrong place can add enough overhead to hurt. With a huge botnet of hijacked browsers, blocking everyone affected becomes a DoS vector in itself, since some of those are your own legitimate attacks.
The big problem for DDoS mitigation is that this requires much deeper knowledge of the protected application. It's hard to just put a box inline, or an unmodified cloud service, and have it block the attacks. There's both good science and great engineering to be done, by developers, platform vendors, and specialty anti-DDoS providers, to block this emerging kind of attack.
[+] [-] dantiberian|10 years ago|reply
[+] [-] oliwarner|10 years ago|reply
I'm personally happy with Linode. They have a seriously tough technical issue to deal with —as much working out what's happening as how to stop it— and they seem to be doing a fairly top job at staying afloat. My servers haven't gone down. Any downtime in the last four years has been my fault.
So even if they are targets of some ludicrously powerful botnet, I'd rather stay with them than let the bastards doing this win. The attack isn't hurting my business or my clients and each incident we go through, the lower the chances of it ever being a problem in the future.
On a more serious note, governments keep moaning on about encryption but botnets are still a much greater direct threat to national security.
[+] [-] larrymcp|10 years ago|reply
http://status.linode.com/incidents/mkcgnmjmnnln
[+] [-] jsmthrowaway|10 years ago|reply
[+] [-] staunch|10 years ago|reply
[+] [-] jsmthrowaway|10 years ago|reply
Even once growth really took off and revenue started making these big shifts in strategy viable, the mindset was still to be lean and scrappy. The minimal capital expenditure strategy had benefits early on and allowed Linode to maintain an incredible margin and support explosive growth, but they were too slow to start thinking like a grown-up company when it started to matter, and it's coming back to bite both on security (with almost zero investment; just enough to pass PCI-DSS) and things like this.
When I heard they bought the Philadelphia building, for example, I was very surprised because that's not the Chris I knew. We lobbied for a Philadelphia office for years. Could be a good sign regarding decision-making culture for the future, but hard to say.
Don't read me as bad blood or anything, as I wish Linode no ill will (I actually hope they can turn this perceived slump around), it's just educational to see the consequences of choices and mindset catch up with a company. I learned a lot about management style while working there and contrasting with subsequent employers.
[+] [-] jph|10 years ago|reply
I have direct experience with Linode staff breakdowns in communication because of a security problem before the December attacks.
The problem affected many Linode customers and included risks to confidential information such as billing.
The Linode staff communication was terrible. The problem was severe and ended up with Linode on a blacklist of companies that are not suitable for hosting.
I have to agree with tptacek: do not use Linode for anything, and if you do now, make plans to switch to a new provider.
To end on a happy note, I migrated the project to Rackspace, and the Rackspace staff communication is excellent.
[+] [-] click170|10 years ago|reply
[+] [-] workitout|10 years ago|reply
[+] [-] ryanlol|10 years ago|reply
I really wonder what that is supposed to mean, Linode has mentioned it multiple times but not elaborated on what sort of an attack this is.
I personally haven't ever head of a "400 bad request"-attack.
Edit: Yeah, I know what Layer 7 floods are :), but I'm pretty sure "400 bad request" floods are something Linode came up with, so that could use some elaboration by them.
[+] [-] sandstrom|10 years ago|reply
[+] [-] fweespeech|10 years ago|reply
My guess is strategies like this but it isn't clear. [e.g. Requests designed to slow/increase the processing per-request to create a log jam at the web application level ]
[+] [-] spdustin|10 years ago|reply
OSI Layer 7 would actually refer to all "applications", but the mention of "400 Bad Request" implies specifically web applications were affected.
[+] [-] virtuallynathan|10 years ago|reply
[+] [-] tptacek|10 years ago|reply
Please don't use Linode. If you are using it now, make immediate plans to switch. If you have friends who have things built on Linode servers, tell them to switch.
[+] [-] yomism|10 years ago|reply
Feel how dickish this sounds without giving any reason whatsoever?
Please explain why.
[+] [-] mindcrime|10 years ago|reply
We've been happy Linode customers for a while now, and definitely prefer Linode to where we were before (Rackspace, via the Slicehost acquisition). I'm not opposed to moving to something like DigitalOcean or other but right now I'm seeing any compelling reason to make a move.
[+] [-] tomkinstinch|10 years ago|reply
All told, it took a few minutes one evening.
[+] [-] minsight|10 years ago|reply
[+] [-] Mahn|10 years ago|reply
[+] [-] mikeash|10 years ago|reply
[+] [-] xaemusa|10 years ago|reply
Please don't say AWS, I have no interest in learning that overcomplicated mess.
[+] [-] corobo|10 years ago|reply
[+] [-] ancarda|10 years ago|reply
[+] [-] fweespeech|10 years ago|reply
Every company around that price point has these problems [and if they don't, they are either burning VC money or lying].
[+] [-] workitout|10 years ago|reply
[+] [-] radialbrain|10 years ago|reply
They mention segregating their customers into separate /24s, and consequently having to assign an IP from every one of these subnets to the router for use by the customer as a gateway.
Is there any reason why they couldn't get rid of these by having customers set up a static route to the "primary" IP of the router (migration / configuration issues aside)?
[+] [-] thomaslutz|10 years ago|reply
[+] [-] ryanlol|10 years ago|reply
Not sure why you'd choose a budget provider for production infra anyway.
[+] [-] brownbat|10 years ago|reply
[+] [-] nodamage|10 years ago|reply
[+] [-] ancarda|10 years ago|reply
How? I thought CloudFlare only protected HTTP? Can you have it reverse proxy a DNS server or is Linode using CloudFlare as the host for ns1.linode.com now?
[+] [-] rdl|10 years ago|reply
(email me if you want more info; it's not really ideal for small sites, it's better to just use cf for hosted DNS then, since it's free, but we're happy to do vDNS for people who can't do hosted. Mainly providers, but also some enterprise customers with special DNS needs. It's a pretty cool technology.)
https://www.cloudflare.com/virtual-dns/
[+] [-] devicenull|10 years ago|reply
[+] [-] wereHamster|10 years ago|reply
Care to elaborate why it took them so long to ack? And name them so I know who to avoid in the future (or route around)!
[+] [-] tim333|10 years ago|reply
Is it really $100bn+ ? If so we could do with some government funded research / countermeasures.
[+] [-] jakeogh|10 years ago|reply
[+] [-] bjano|10 years ago|reply
There is something very ironic about this. They have a policy which instead of addressing the problem actively assists anyone wanting to attack their customers. No surprise that these customers have been complaining about this practice for a long time. But until now it was Somebody Else's Problem so they didn't bother figuring out some proper (or at least less terrible) solution. Now this lack of preparedness bit them in the ass...
[+] [-] jsmthrowaway|10 years ago|reply
Contrary to popular opinion, if you're getting DoS attacked, you're either (a) popular enough to start thinking about adult-size pants for your transit strategy or (b) inviting the attention by your choice of content or activities. In years of hosting, I started to know the targets of DoS attacks by name. You have to own at least a little bit of responsibility, and mitigate on your own end if you're going to be inviting that kind of attention; IRC and controversial blogs are the usual suspects here, but that's probably changed recently as I've been out of the hosting game for a while.
Linode has few options for reacting other than the one they use. I know that sucks, but it's how it is.
[+] [-] brandon272|10 years ago|reply
[+] [-] db7a11196|10 years ago|reply
[+] [-] gauravphoenix|10 years ago|reply
[+] [-] jsmthrowaway|10 years ago|reply