We used to run a game server for a small community of around 400-500 people and DDos attacks were something we had to face almost every week, whenever someone got upset with the admin team, the go to solution was was to DDos, you get scammed by another player? DDos. Got banned for saying racist things ingame? DDos. You figured out a new way to cheat in game and the admins fixed it? DDos.
We were kids back then and those were kids that were attacking us with just a 5-10usd budget. Yes they were relatively small (ranging from 10-60Gbps) attacks compared to the Tbps attacks that are happening to some companies, but good god it was so annoying when all it took was just 5 usd from some idiot to take down your server.
We moved to gcp got null routed (or reduced network bandwitch to the node under attack) every-time there was an attack. Bought azure's 3000usd a month anti DDos protection, was worthless for a tcp/udp service. Tried to have a network load balancer in the cloud that auto-scaled, still some players got effected when an attack came in.
Finally we moved over to OVH and placed a few really powerful servers in-front of the game server and applied some ipfilter rules to reduce common attacks. That ended up being the cheapest option out of all the options. When you have a very small community its not like you have the biggest budget to work with. But it was really fun and taught all of us a lot. Looking back its kinna sad we had to end things. But it was a lot fun.
DDos attacks are one of those things that really makes me worried about the future of the internet. The only way to win it is to throw money at it and cross your fingers that the attacker will run out of resources before you do.
Definitely companies like cloudflare does an incredibly good job of stopping some insanely big attacks when it comes to http/https (I recently saw they were supporting udp and tcp based services now, never tried it).
But one thing that's weird is having to rely on some 3rd party company. Yes cloudflare so far has been a company I can trust, but, I once loved and trusted a company that said "Don't be evil".
If you are a developer for some IOT device manufacturer please do your best to makesure someone wont turn your light bulb in to a part of a botnet. When you guys fuck-up the rest of us have to suffer.
The significant thing about "script kiddy DDOS" level attacks, is that they significantly raise the effort and expense for the smallest projects. This is exactly where the most important innovations happen:
Finally we moved over to OVH and placed a few really powerful servers in-front of the game server and applied some ipfilter rules to reduce common attacks. That ended up being the cheapest option out of all the options
The cheaper attacks seem to be at the level, where machine learning could be able to counter them. Raising the bar for inexpensive attacks would be a huge boon to the internet and human progress. It wouldn't be that expensive to fund, either.
We used to run a game server for a small community of around 400-500 people and DDos attacks were something we had to face almost every week, whenever someone got upset with the admin team, the go to solution was was to DDos, you get scammed by another player? DDos. Got banned for saying racist things ingame? DDos. You figured out a new way to cheat in game and the admins fixed it? DDos.
I wonder if this sort of thing could be honeypotted? Give perpetrators a way to figure out and target a fake "edge server" of a particular user? (Which only affects about 5% of your user base, let's say.) However, that "edge server" is actually a honeypot that gathers data on the attack, and correlates that to support emails to the admin team, or flame wars in the game's forums.
This is the kind of suckage that holds back the entire network, but which can ultimately be defeated:
Slightly OT but the significant thing about "Don't be Evil" is that Google had already taken the fundamental choices that were evil. The slogan itself is blatantly self-conscious - an acknowledgement of the insane power that would inevitably accumulate as a result of the business model they were pioneering.
> Definitely companies like cloudflare does an incredibly good job of stopping some insanely big attacks when it comes to http/https (I recently saw they were supporting udp and tcp based services now, never tried it).
CF still requires an Enterprise contract for proxying arbitrary traffic via Spectrum, likely because of the abuse prevention aspect. Otherwise SSH and minecraft is offered at pay-as-you go rates, but a lot have complained about how expensive it is:
Was this a RuneScape private server (RSPS)? I remember back in the day you could find a free RSPS DDoS tool with a quick Google search, and all you had to do was enter the public IP address to start attacking. The culture for the RSPS scene was exactly what you're describing.
Also there was another kind of attack where you would start thousands of bot clients at once that would spam messages. The hopes would be that you would (a) shut down the server, (b) attract the players to the server your bots were advertising
> When you guys fuck-up the rest of us have to suffer.
Is there a lawyer here who can comment on whether the manufacturer of these horrid devices have any civil liability - either currently or possibly in the future?
My gut tells me the only way this will get better is for their to be rules of negligence applied to the realm of computer security.
I don't know what it is about gaming that attracts DDoS events more than practically anything else, but there are a lot of server hosts that will not even rent you a server if they know that there will be a game hosted on it due to this.
I have used Cloudflare Spectrum to prevent attacks. It does work incredibly well but the cost is significant.
As for 3rd party copmanies, I do hate to rely on cloudflare for this. It is the worst business relationship by far I have ever been in, but yet there are no good alternatives we found.
The magic of 3rd party anti-DDOS providers is rarely the software/methods: it's just about having bigger pipes. Everyone can figure out how to block volumetric attacks with iptables or whatever, the problem is if you have a 1 gig pipe from your transit provider, it's going to get saturated even before you can do any blocking. The 3rd parties can afford to have multiple 100g pipes with 10gbps commits in multiple DCs -- you share this cost with other customers for when you get attacked. That's kinda the entire point of 3rd party anti-DDOS providers, and not much else.
And this is why we need MaidSAFE instead of the Web. They don’t have DDOS attacks, instead you make money every time someone accesses a chunk of a resource, and the kademlia tree hides the hosts’ IP after one hop so the network and hosts can’t be taken down easily. Very different from Tor.
https://maidsafe.net is the best project to come out of the “Web3” space. If you heard of freenet, this is like freenet 2.0
PS: Why the massive silent downvotes? This platform actually solves the problem and many others HN constantly correctly complain about. But when posted, you prefer to ignore it. (Disclaimer: I am not affiliated with them in any way. In some ways they are a competitor to Qbix and Intercoin but I give credit where it is due.)
This had me scratching my head earlier today when I was debugging why renewal was taking so long. I've taken Let's Encrypt's reliability for granted. Didn't even cross my mind that it might be a service issue.
Here is something to think about. If you get ddos'd in middle of trial run by an enterprise customer and its the end of your startup. AZ , AWS , OVH almost all hosting providers will all start dropping your connections.
And DDOS protection services are expensive. Pay as you go models are awful for this as when you do get ddos'd , you bill could be quite high.
No, OVH has DDoS protection built into their service, and it's free, and your connections will not get dropped. I moved to OVH after getting a few DDoS attacks, and since then there have been no problems. I've had a few emails from OVH notifying me about attacks in progress and that they are automatically mitigating it. When it happens, it has zero effect on our service.
OVH actually does a really good job when it comes to DDoS protection. They can take some really big attacks and slow it down just enough so you firewall rules can take care of the rest.
From all the hosting providers I have used, they are the only ones who don't null route you as soon as you get a attacked, and considering the cost of their service OVH is a real life saver when you really need the help.
----
Just realized this sounds like an advert for OVH, lol I have no affiliation with them whatsoever, just a really happy old customer.
Caddy does a lot of it: https://caddyserver.com/docs/automatic-https#errors, and Caddy also falls back to ZeroSSL if it couldn't issue with Let's Encrypt. Caddy is without a doubt the most robust ACME client implementation to date.
"no one is buying our expensive SSL certs. If we show businesses how unreliable a free service is, CTOs will make their admins buy from us."
or
"domain xyz's certificate is expiring. If we pay for a ddos, their site won't be able to renew and (customers wont go to the site due to expired cert/API people use wont work/we can take advantage of a compromised cert longer)"
Possibly testing or demonstrating a botnet. "For bragging rights" is a thing, as is advertising ("my botnet took down critical infrastructure, wanna buy my DDoS service?").
Not sure. Unless this is sustained for a long time it shouldn't affect autorenewals which are done well in advance of expiry. So it _shouldn't_ affect cert expiry unless people are still manually renewing and leaving it to last minute.
[edit] Unless the attackers identified a bug in certbot (commonly used autorenewal scripts), e.g what happens when LE is unavailable when autorenew is triggered - you'd hope it would retry periodically until LE is restored, but perhaps not. If not you could time the DDoS just right to ensure a specific cert does not get renewed even after the DDoS stops, then maybe a couple weeks later it would expire... But that's relying on such a bug existing and the site owners not noticing it (LE will also email the registered email address eventually regardless of autorenewal scripts), so maybe this is too much of a stretch.
Just last week our shared hosting provider was attacked and the attacker tried to brute-force it‘s way into a management API. I cannot image another reason as „fun“ and „just because we can“ because there‘s nothing to get [besides money after encrypting all data].
So I think the attacker just attacks LE because it‘s in the internet and he can.
Often a DDoS is used as a smokescreen/cover for an actual compromise. I guess the hope is that it gets unnoticed in all the noise and whilst all hands are busy at the pumps. Hope not!
I see in their status page that OCSP endpoints are also impacted. There could be any number of motivations including interfering with someone's ability to check if a certificate has been revoked.
Given the complete absence of information could it just be an accident? I think Wikipedia recently had performance issues where it turned out that a popular app was just pulling an image in the background and the app developers fixed it once they were notified.
it's better to assume that people will simply do everything that is possible, you can't open umbrella in your butt so that can be assumed not to be done, but everything possible should be assumed done/to be done sonner or later; motivation of "because I can" is simply enough.
Just a reminder that users of Caddy (v2.3.0 or higher) are not at risk when LE gets hit like this, because it will fallback to having a certificate issued from ZeroSSL. Both issuers would need to be down for the whole last 30 days of the certificate's 90 day lifetime before Caddy would be stuck with expired certificates.
This is a very morbid thought, but I wonder if the people who run LE ever travel via the same means. If somebody took them out all at once, would the web's security essentially crumble? This is the danger of centralized services, but moreso the crap design of web PKI.
All "usable" HTTPS depends on certs, right? And "usable" certs require a domain, right? And that cert for that domain needs to have been generated by a CA, right? But it's tied to a domain, and IP space. You have to prove to a CA that you both control a domain record and some IP space it points to. Nobody has designed anything to straightforwardly prove that in an unhackable way. We have shitty hacks, like "serve this unique file on this web server that this domain record is pointing to", or "answer an e-mail on one of 20 addresses at this domain", etc.
But none of those address what we actually want to do, which is just to prove that we own/control a domain record. That's the only meaningful thing in having a cert: proving that you actually own the domain record this cert is assigned to. And we have no actual way to do this. Literally the only way to prove definitively that you own a domain is to talk to the registrar, and the only way to prove that you control a domain record is to talk to the nameserver that the registrar is pointing to. The former we don't handle at all, and the latter is highly susceptible to various attacks.
You could remove the reliance on CAs entirely with a different model. You tie a private key to domain ownership, and a private key to a domain record. Then you only have to trust registrars' keys/certs, and you can walk backward along a cryptographically-signed web of trust. Your browser trusts the registrar's key X. The registrar signs your domain key Y. The domain key Y signs a domain record key Z. Your web server generates a cert using domain key Z.
For a client to verify the web server cert, they verify it was created by key Z, and verify that key Z was signed by key Y, and that key Y was signed by key X. Then any webserver can generate its own cert for any domain record, we don't need CAs to generate certs, and we have a solid web of trust that goes back to the actual owner of the domain, but also allows split trust via the domain owner assigning keys to domain records.
This is such a well-understood problem in fact that it has a name and Wikipedia entry, called "bus factor". According to:
> The "bus factor" is the minimum number of team members that have to suddenly disappear from a project before the project stalls due to lack of knowledgeable or competent personnel
As for proving that you own a domain, I think the DNS-01 challenge that is used to grant Star-certificates does a pretty good approximation, if you can create and update TXT records in the root zone, you have at least functionally "owned" a domain even if you don't legally own the domain.
> I wonder if the people who run LE ever travel via the same means
Afaik, the LE team is distributed across the globe.
> If somebody took them out all at once, would the web's security essentially crumble?
No, there are other both free and paid CAs
> We have shitty hacks, like "serve this unique file on this web server that this domain record is pointing to", or "answer an e-mail on one of 20 addresses at this domain", etc.
Yes, but we also have certificate transparency. You can monitor all certificates issued to your domains and revoke them if needed. Not perfect but imo still reasonably safe considering you know that all the issued certs are on your servers.
> You tie a private key to domain ownership, and a private key to a domain record. Then you only have to trust registrars' keys/certs, and you can walk backward along a cryptographically-signed web of trust.
That exists and is called DNSSEC. If you haven't heard of it, you already understand: it isn't widely used. Also, it would require major rethinking of how we use the internet. Most clients do not validate DNSSEC, only public and maybe ISP resolvers do, but they can (and probably will) tamper the DNSSEC answers if they can better spy and mitm you.
> Your browser trusts the registrar's key X
Sure, we could do it in browsers, but the internet is wider than the web, and we would need to rewrite a great part or what we use every day (not saying that we can't or should not).
In the mean time, if you use a DNSSEC-compatible TLD and registrar, you can already sign your zones. That way, the current CAs will be able to cryptographically verify that the server asking for a cert also owns the domain/subdomain.
TLS cert authorities shouldn't end, but more importantly, HTTP shouldn't end. HTTP+HTTPS together are great. HTTPS only, as being pushed in modern times is quite bad.
LetsEncrypt is great and I am really glad someone stepped up to create a mostly not evil non-profit cert authority. But everyone using LE is very bad for the health of the internet. It provides nearly a single point of failure for government/political interefence, technical failure, and failure due to corruption from money and scale internally.
[+] [-] malikNF|5 years ago|reply
We were kids back then and those were kids that were attacking us with just a 5-10usd budget. Yes they were relatively small (ranging from 10-60Gbps) attacks compared to the Tbps attacks that are happening to some companies, but good god it was so annoying when all it took was just 5 usd from some idiot to take down your server.
We moved to gcp got null routed (or reduced network bandwitch to the node under attack) every-time there was an attack. Bought azure's 3000usd a month anti DDos protection, was worthless for a tcp/udp service. Tried to have a network load balancer in the cloud that auto-scaled, still some players got effected when an attack came in.
Finally we moved over to OVH and placed a few really powerful servers in-front of the game server and applied some ipfilter rules to reduce common attacks. That ended up being the cheapest option out of all the options. When you have a very small community its not like you have the biggest budget to work with. But it was really fun and taught all of us a lot. Looking back its kinna sad we had to end things. But it was a lot fun.
DDos attacks are one of those things that really makes me worried about the future of the internet. The only way to win it is to throw money at it and cross your fingers that the attacker will run out of resources before you do.
Definitely companies like cloudflare does an incredibly good job of stopping some insanely big attacks when it comes to http/https (I recently saw they were supporting udp and tcp based services now, never tried it).
But one thing that's weird is having to rely on some 3rd party company. Yes cloudflare so far has been a company I can trust, but, I once loved and trusted a company that said "Don't be evil".
If you are a developer for some IOT device manufacturer please do your best to makesure someone wont turn your light bulb in to a part of a botnet. When you guys fuck-up the rest of us have to suffer.
[+] [-] stcredzero|5 years ago|reply
http://www.paulgraham.com/marginal.html
Finally we moved over to OVH and placed a few really powerful servers in-front of the game server and applied some ipfilter rules to reduce common attacks. That ended up being the cheapest option out of all the options
The cheaper attacks seem to be at the level, where machine learning could be able to counter them. Raising the bar for inexpensive attacks would be a huge boon to the internet and human progress. It wouldn't be that expensive to fund, either.
We used to run a game server for a small community of around 400-500 people and DDos attacks were something we had to face almost every week, whenever someone got upset with the admin team, the go to solution was was to DDos, you get scammed by another player? DDos. Got banned for saying racist things ingame? DDos. You figured out a new way to cheat in game and the admins fixed it? DDos.
I wonder if this sort of thing could be honeypotted? Give perpetrators a way to figure out and target a fake "edge server" of a particular user? (Which only affects about 5% of your user base, let's say.) However, that "edge server" is actually a honeypot that gathers data on the attack, and correlates that to support emails to the admin team, or flame wars in the game's forums.
This is the kind of suckage that holds back the entire network, but which can ultimately be defeated:
http://www.paulgraham.com/spam.html
[+] [-] scandox|5 years ago|reply
[+] [-] judge2020|5 years ago|reply
CF still requires an Enterprise contract for proxying arbitrary traffic via Spectrum, likely because of the abuse prevention aspect. Otherwise SSH and minecraft is offered at pay-as-you go rates, but a lot have complained about how expensive it is:
https://community.cloudflare.com/t/what-do-you-think-about-t...
[+] [-] radihuq|5 years ago|reply
Also there was another kind of attack where you would start thousands of bot clients at once that would spam messages. The hopes would be that you would (a) shut down the server, (b) attract the players to the server your bots were advertising
[+] [-] MR4D|5 years ago|reply
Is there a lawyer here who can comment on whether the manufacturer of these horrid devices have any civil liability - either currently or possibly in the future?
My gut tells me the only way this will get better is for their to be rules of negligence applied to the realm of computer security.
[+] [-] Negitivefrags|5 years ago|reply
I have used Cloudflare Spectrum to prevent attacks. It does work incredibly well but the cost is significant.
As for 3rd party copmanies, I do hate to rely on cloudflare for this. It is the worst business relationship by far I have ever been in, but yet there are no good alternatives we found.
[+] [-] tomphoolery|5 years ago|reply
[+] [-] parliament32|5 years ago|reply
[+] [-] homero|5 years ago|reply
https://www.cloudflare.com/products/cloudflare-spectrum/
[+] [-] trinovantes|5 years ago|reply
[+] [-] s17n|5 years ago|reply
[+] [-] tester756|5 years ago|reply
edit: nvm.
[+] [-] EGreg|5 years ago|reply
https://maidsafe.net is the best project to come out of the “Web3” space. If you heard of freenet, this is like freenet 2.0
PS: Why the massive silent downvotes? This platform actually solves the problem and many others HN constantly correctly complain about. But when posted, you prefer to ignore it. (Disclaimer: I am not affiliated with them in any way. In some ways they are a competitor to Qbix and Intercoin but I give credit where it is due.)
[+] [-] mjthompson|5 years ago|reply
[+] [-] manishsharan|5 years ago|reply
[+] [-] cpncrunch|5 years ago|reply
[+] [-] malikNF|5 years ago|reply
From all the hosting providers I have used, they are the only ones who don't null route you as soon as you get a attacked, and considering the cost of their service OVH is a real life saver when you really need the help.
----
Just realized this sounds like an advert for OVH, lol I have no affiliation with them whatsoever, just a really happy old customer.
[+] [-] benlivengood|5 years ago|reply
Let's Encrypt asks for max 1 request per day per certificate: https://letsencrypt.org/docs/integration-guide/
[+] [-] francislavoie|5 years ago|reply
[+] [-] timvisee|5 years ago|reply
[+] [-] mysterydip|5 years ago|reply
or
"domain xyz's certificate is expiring. If we pay for a ddos, their site won't be able to renew and (customers wont go to the site due to expired cert/API people use wont work/we can take advantage of a compromised cert longer)"
Just some possible but implausible scenarios.
[+] [-] tgsovlerkhgsel|5 years ago|reply
[+] [-] tomxor|5 years ago|reply
[edit] Unless the attackers identified a bug in certbot (commonly used autorenewal scripts), e.g what happens when LE is unavailable when autorenew is triggered - you'd hope it would retry periodically until LE is restored, but perhaps not. If not you could time the DDoS just right to ensure a specific cert does not get renewed even after the DDoS stops, then maybe a couple weeks later it would expire... But that's relying on such a bug existing and the site owners not noticing it (LE will also email the registered email address eventually regardless of autorenewal scripts), so maybe this is too much of a stretch.
[+] [-] chrisandchris|5 years ago|reply
Just last week our shared hosting provider was attacked and the attacker tried to brute-force it‘s way into a management API. I cannot image another reason as „fun“ and „just because we can“ because there‘s nothing to get [besides money after encrypting all data].
So I think the attacker just attacks LE because it‘s in the internet and he can.
[+] [-] beermonster|5 years ago|reply
I see in their status page that OCSP endpoints are also impacted. There could be any number of motivations including interfering with someone's ability to check if a certificate has been revoked.
[+] [-] jamescun|5 years ago|reply
[+] [-] josefx|5 years ago|reply
[+] [-] iso1210|5 years ago|reply
[+] [-] bombcar|5 years ago|reply
[+] [-] scoot|5 years ago|reply
[+] [-] mirekrusin|5 years ago|reply
[+] [-] oconnor663|5 years ago|reply
[+] [-] polycaster|5 years ago|reply
[+] [-] francislavoie|5 years ago|reply
https://caddyserver.com/docs/automatic-https#errors
https://github.com/caddyserver/caddy/releases/tag/v2.3.0
[+] [-] noncoml|5 years ago|reply
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] 0xbadcafebee|5 years ago|reply
All "usable" HTTPS depends on certs, right? And "usable" certs require a domain, right? And that cert for that domain needs to have been generated by a CA, right? But it's tied to a domain, and IP space. You have to prove to a CA that you both control a domain record and some IP space it points to. Nobody has designed anything to straightforwardly prove that in an unhackable way. We have shitty hacks, like "serve this unique file on this web server that this domain record is pointing to", or "answer an e-mail on one of 20 addresses at this domain", etc.
But none of those address what we actually want to do, which is just to prove that we own/control a domain record. That's the only meaningful thing in having a cert: proving that you actually own the domain record this cert is assigned to. And we have no actual way to do this. Literally the only way to prove definitively that you own a domain is to talk to the registrar, and the only way to prove that you control a domain record is to talk to the nameserver that the registrar is pointing to. The former we don't handle at all, and the latter is highly susceptible to various attacks.
You could remove the reliance on CAs entirely with a different model. You tie a private key to domain ownership, and a private key to a domain record. Then you only have to trust registrars' keys/certs, and you can walk backward along a cryptographically-signed web of trust. Your browser trusts the registrar's key X. The registrar signs your domain key Y. The domain key Y signs a domain record key Z. Your web server generates a cert using domain key Z.
For a client to verify the web server cert, they verify it was created by key Z, and verify that key Z was signed by key Y, and that key Y was signed by key X. Then any webserver can generate its own cert for any domain record, we don't need CAs to generate certs, and we have a solid web of trust that goes back to the actual owner of the domain, but also allows split trust via the domain owner assigning keys to domain records.
[+] [-] yebyen|5 years ago|reply
This is such a well-understood problem in fact that it has a name and Wikipedia entry, called "bus factor". According to:
> The "bus factor" is the minimum number of team members that have to suddenly disappear from a project before the project stalls due to lack of knowledgeable or competent personnel
As for proving that you own a domain, I think the DNS-01 challenge that is used to grant Star-certificates does a pretty good approximation, if you can create and update TXT records in the root zone, you have at least functionally "owned" a domain even if you don't legally own the domain.
[+] [-] pastech|5 years ago|reply
Afaik, the LE team is distributed across the globe.
> If somebody took them out all at once, would the web's security essentially crumble?
No, there are other both free and paid CAs
> We have shitty hacks, like "serve this unique file on this web server that this domain record is pointing to", or "answer an e-mail on one of 20 addresses at this domain", etc.
Yes, but we also have certificate transparency. You can monitor all certificates issued to your domains and revoke them if needed. Not perfect but imo still reasonably safe considering you know that all the issued certs are on your servers.
> You tie a private key to domain ownership, and a private key to a domain record. Then you only have to trust registrars' keys/certs, and you can walk backward along a cryptographically-signed web of trust.
That exists and is called DNSSEC. If you haven't heard of it, you already understand: it isn't widely used. Also, it would require major rethinking of how we use the internet. Most clients do not validate DNSSEC, only public and maybe ISP resolvers do, but they can (and probably will) tamper the DNSSEC answers if they can better spy and mitm you.
> Your browser trusts the registrar's key X
Sure, we could do it in browsers, but the internet is wider than the web, and we would need to rewrite a great part or what we use every day (not saying that we can't or should not).
In the mean time, if you use a DNSSEC-compatible TLD and registrar, you can already sign your zones. That way, the current CAs will be able to cryptographically verify that the server asking for a cert also owns the domain/subdomain.
[+] [-] twirlock|5 years ago|reply
[deleted]
[+] [-] FreeCodeFreak|5 years ago|reply
[deleted]
[+] [-] Tenpenny99|5 years ago|reply
[+] [-] superkuh|5 years ago|reply
LetsEncrypt is great and I am really glad someone stepped up to create a mostly not evil non-profit cert authority. But everyone using LE is very bad for the health of the internet. It provides nearly a single point of failure for government/political interefence, technical failure, and failure due to corruption from money and scale internally.