This doesn't help my impression of Digital Ocean at all (even if I am a paying customer currently). A few years ago you could impersonate Digital Ocean staff on their support pages with no effort. They grabbed the username from your email, so whatever you put in front of the @ becamse your username on the forums, visible to everyone. And the avatar came from one of those email->avatar services where you can sign up and set it to anything. So when I signed up with a username like [email protected], I ended up being called "digitalocean" on the support forums, and if I had wanted I could just change the avatar to the Digital Ocean logo and impersonate DO or anyone else.
I tried reporting it but got pretty much the same answer as this guy (though I did not get banned). Luckily they fixed it like a year later.
Great write-up, and interesting problem! I wonder if more hosting providers are vulnerable to the same problem.
This same thing happens with CloudFlare & is being actively exploited. We reported it to them within the last two weeks and we were told that it's expected behaviour and that they weren't going to do anything about it.
I asked them to, at the absolute least, send an email notification to the prior-CloudFlare owner letting them know that the domain "your CF account used to control is now being controlled by a new CF account". Better yet, implement a domain ownership validation scheme.
They told us that they wouldn't be making any changes.
FWIW, on CloudFlare what happened to us was: we were moving registrars for ~100 domains, from GoDaddy to Route53. During this transition, the NS for the domains temporarily became blank; at this point CF automatically removed the domains from our CF account. The NS were then re-added to the domains on the Route53 side (<4 hours of 'no nameserver' time).
Apparently there are people out there that are looking for domains that are pointed to CF and then attempting to add them to their own CF account (automated I'm sure) -- which CF lets them do without any verification once they've been auto-removed from your [the original CF account] account.
Interestingly, the original account must be still stored in their system with the domain because we were able to re-add the domain to our original CF account without any verification; effectively "stealing the domains back" to our CF account, away from the thieve's CF account.
In this case, the "attackers" (perhaps more appropriate, I call them 'malicious actors') were able to commandeer ~100 of our domains for ~2 months, for free; they redirected them to Russian websites, torrent sites, affiliate sites, etc.
Again, this is being actively exploited on CloudFlare, at the direct expense of CF customers -- but, according to CF, it's not an issue...?!
According to CloudFlare, they are are a reverse proxy, and they are not responsible for anything. This has been their response to every issue that I've tried to bring up with them over any channel, including here on HN.
I work at CloudFlare, not in DNS or on this code, and have mentioned this incident in the all-company chat. There is a healthy conversation happening there and it turns out a fix was already in the works for the underlying issue.
adanto6840 has supplied the support ticket number (thank you), and this specific incident is also being reviewed.
I will never stop being infuriated by responses like this from companies - how many more megaleaks have to happen before they realize that they need to embrace white hats, not ban their accounts, not sue them, not swat them / have them arrested, not silence them.
Why, when the author realised this was likely to be possible, didn't they get in touch with DO? Or try with a domain they knew was OK to use? Or at least just try with one domain.
They took almost twenty thousand, sent all the requests to their own server and logged them.
I just wanted to let you know that I really appreciate your feedback, as well as the feedback from the other commenters here.
I understand that many here are concerned that banning the account seems, from this perspective, to have been an unjustified action. I do believe that there is a bit of a misunderstanding on the timeline of events here, as well as the source of the decision. To be clear, Cash supported the decision that I had made to ban the account in question, and there had been no communication between us and Matthew at this point. We began receiving a significant number of support requests to remove domains from this account, and I authorized the shutting down of this account as it was clear to me what was happening. I have been working with our engineers to see to the removal of the domains from the account as well.
I apologize if our actions seemed at any point rude or inappropriate, it was definitely not my intention. I want nothing more than to look out for the safety and wellbeing of our customers, and I chose what I believed to be the best action. I do want you to know that if I was aware that a security researcher had been working on testing a theory, I might have acted differently. That can, however, impact the reason behind a white hat test. You generally want the company to see you as normal user, so that you can see how they act in return. We do shut down users who are intentionally causing problems for other users, and I do think that was made evident here.
I do understand that opening a line of communication with Matthew may have been appropriate, and I consider that valuable feedback moving forward.
Hey Jarland thanks for the thoughtful response. I don't think your actions were rude or inappropriate (what host would see this behavior and not act similarly?). I apologize that the discussion on the topic became so negative towards DigitalOcean (this might be my fault, I was merely trying to point out why I hadn't been able to delete the domains - not say that DigitalOcean was a bad company/etc). The disclaimer I added early on I thought would mitigate some of this negativity but apparently not quite.
At the very least I'm happy that some people have seen this more as a study of this pattern of functionality being an issue instead of this being only a DigitalOcean problem. I've seen (and have been reached out by) multiple people realizing this affects their company as well which has been awesome to watch.
I think most of the "outrage" is with the complete lack of validation and the "This is a known workflow within our platform" response more than the ban. Any plans to address the issue other than reactive bans?
Linode also does not verify domain ownership before you add a domain to their DNS. And they also use the same nameservers for all accounts. So I think they're in the same boat as DO as far as this "vulnerability" is concerned.
I'm a long-time Linode customer and use them for all servers that run important services. I use DO for the less important stuff. But Linode has had a not-so-stellar record of data breaches.
Something similar happened to me a few months ago with Cloudflare. I set up a new domain to use Cloudflare's nameservers but did not immediately get around to setting it up on the admin panel. By the time I wanted to add the domain, someone else had already grabbed it and set up some sort of spam page.
Took a few emails to Cloudflare support to resolve this one. They also didn't seem to care much about the security implications when I questioned them about it.
Wonder what else might be vulnerable to this... CloudFlare seems like it may, they only have a handful of nameservers in any case.
Sort of hard to call it a vulnerability on DO's part though - more of an issue with the admins. I think most DNS services operate in this way, really, route53 may be the exception, not the rule.
I have an account with DigitalOcean (and several competitors) and I'm not going anywhere or moving any sites around because of this. Sure, they could have handled things better and the security researcher could have too. I don't see any malice or incompetence here, nor do I see a reason to make the effort to switch to another provider.
Where are you going to run off to? How is their security better over there? How many hours of work does that involve?
It's pretty understandable actually. Most DNS providers do similar, it's really just the admins fault for not switching their DNS, likely because they were abandoned and unused.
Meh, it was just mild incompetence. I expect the only providers (hell, large companies in general) who never responded this way are the ones who haven't been around long enough.
Was there a realization into how legitimate users may be affected by this action? Was there a plan to remove those domains from their account after making and disclosing their proof of concept?
Why not stop at 10 or 20, and then alert DO to the findings?
Amazon S3 has similar problems. To host static website you need use your domain name as the S3 bucket name. Amazon does not verify ownership of your domain, and bucket names use global namespace.
Someone can easily block you from using S3 static website hosting by adding a bucket with your domain name before you do. Also if you delete a bucket and do not change your DNS, someone can recreate the bucket and will be serving files from your domain.
Bye bye digitalocean - account deletion request submitted 1178917. When you have reckless people like Cashan Stine (trust & safety specialist - WTF is that title? sounds like a road safety officer?) that close accounts due to a security report then it won't win any business from me or my clients.
If you'd just straight up cancel an account that fast I don't believe you had any service or clients hosted on DigitalOcean. Idle threats belong on Facebook and Twitter, not Hacker News.
You don't need to make a deletion request, you can deactivate your account from your account settings, and it offers to delete everything for you. That's what I did when I wanted to close my account recently (I wasn't using the droplets I had, and liked my other VPS better anyway.)
I've reported multiple vulnerabilities to DigitalOcean before and they've fixed them rapidly, credited me for the effort, and gave me free time on their services.
The difference is I didn't exploit 20 thousand domains to make flashy headlines and prove a point about something that isn't even a serious bug.
I think most of the providers (e.g. DO, Linode, CloudFlare etc) do not check the authority of DNS due to the chicken-and-egg problem. The AWS way to handle this issue is definitely awesome but the infrastructure required is not worth for those companies who are providing "free DNS service" as an add-on to their existing customers. Anyway, IMO, it is your fault if you point to a nameserver but not utilizing it.
Banning his account was totally unjustified since he approached them first with the issue. A less ethical person could have tried to make money or sold this off on the back market. People like him should be rewarded not have their accounts banned. For all we know he just saved DO a lot of headache in sorting this issue had it gone wrong. I really wish the response from DO on this was different.
Adding 20k domains to your account is probably enough to flag as abuse even if you own the domains. Next time the author should probably try just the one or two. Bonus points if they're their own domains.
> The main reason I did not reach out with the theory instead of the proof-of-concept was because I believed that it would be ignored due to lack of evidence (as is my experience with past disclosures)
The security team at DigitalOcean has been working to ensure that DO is a safe place for security researchers to identify issues on the Internet as well as at DigitalOcean - security is in everyone's interest. We encourage researchers to contact us when they want to use our platform for this type of work specifically so that we can avoid the types of pain that Matthew encountered while doing his experimentation.
Feel free to reach out to [email protected] and we will be happy to help.
"I was walking down the street and I noticed your house wasn't locked very well. So I stole all your stuff and put it in my own house.
Now I'm in prison because of this so it's really hard for me to put it back."
The article writer is an idiot. He deliberately stole accounts because he could. Just because he then decided to blame the provider because he was able to do this does't make it any more defensible.
If I mug you in the street, should I then post that because I was able to do so it's all your fault? No. I'd go to prison....
I believe I was clear about it. However sometimes my writing can be unclear so perhaps it wasn't properly understood (I assume you're talking about their security team's response and not Trust & Safety?). Kind of sad about the massive amount of hate for DigitalOcean in this thread as their security team really seemed quite nice. Their support was just acting on an anomaly they had seen so shrugs.
An additional vector for this kind of attack is to create a zonefile for a subdomain off of a working, live domain administered by the same DNS server.
EG if foo.com is a working site on your DNS provider, try creating a zonefile for bar.foo.com and see if you can create an A record to point to your own server.
This used to be something shared web hosting services running CPanel/WHM were particularly susceptible to. Clearly, the risks here are both phishing/identity and cookie credential stealing.
[+] [-] flexd|9 years ago|reply
I tried reporting it but got pretty much the same answer as this guy (though I did not get banned). Luckily they fixed it like a year later.
Great write-up, and interesting problem! I wonder if more hosting providers are vulnerable to the same problem.
[+] [-] adanto6840|9 years ago|reply
I asked them to, at the absolute least, send an email notification to the prior-CloudFlare owner letting them know that the domain "your CF account used to control is now being controlled by a new CF account". Better yet, implement a domain ownership validation scheme.
They told us that they wouldn't be making any changes.
FWIW, on CloudFlare what happened to us was: we were moving registrars for ~100 domains, from GoDaddy to Route53. During this transition, the NS for the domains temporarily became blank; at this point CF automatically removed the domains from our CF account. The NS were then re-added to the domains on the Route53 side (<4 hours of 'no nameserver' time).
Apparently there are people out there that are looking for domains that are pointed to CF and then attempting to add them to their own CF account (automated I'm sure) -- which CF lets them do without any verification once they've been auto-removed from your [the original CF account] account.
Interestingly, the original account must be still stored in their system with the domain because we were able to re-add the domain to our original CF account without any verification; effectively "stealing the domains back" to our CF account, away from the thieve's CF account.
In this case, the "attackers" (perhaps more appropriate, I call them 'malicious actors') were able to commandeer ~100 of our domains for ~2 months, for free; they redirected them to Russian websites, torrent sites, affiliate sites, etc.
Again, this is being actively exploited on CloudFlare, at the direct expense of CF customers -- but, according to CF, it's not an issue...?!
[+] [-] yoo1I|9 years ago|reply
According to CloudFlare, they are are a reverse proxy, and they are not responsible for anything. This has been their response to every issue that I've tried to bring up with them over any channel, including here on HN.
CloudFlare just doesn't care.
[+] [-] buro9|9 years ago|reply
Definitely not the case.
I work at CloudFlare, not in DNS or on this code, and have mentioned this incident in the all-company chat. There is a healthy conversation happening there and it turns out a fix was already in the works for the underlying issue.
adanto6840 has supplied the support ticket number (thank you), and this specific incident is also being reviewed.
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] arkadiyt|9 years ago|reply
Great find / writeup.
[+] [-] IanCal|9 years ago|reply
They took almost twenty thousand, sent all the requests to their own server and logged them.
That's surely not your first step.
[+] [-] empath75|9 years ago|reply
[+] [-] treehau5|9 years ago|reply
[+] [-] jarland|9 years ago|reply
I just wanted to let you know that I really appreciate your feedback, as well as the feedback from the other commenters here.
I understand that many here are concerned that banning the account seems, from this perspective, to have been an unjustified action. I do believe that there is a bit of a misunderstanding on the timeline of events here, as well as the source of the decision. To be clear, Cash supported the decision that I had made to ban the account in question, and there had been no communication between us and Matthew at this point. We began receiving a significant number of support requests to remove domains from this account, and I authorized the shutting down of this account as it was clear to me what was happening. I have been working with our engineers to see to the removal of the domains from the account as well.
I apologize if our actions seemed at any point rude or inappropriate, it was definitely not my intention. I want nothing more than to look out for the safety and wellbeing of our customers, and I chose what I believed to be the best action. I do want you to know that if I was aware that a security researcher had been working on testing a theory, I might have acted differently. That can, however, impact the reason behind a white hat test. You generally want the company to see you as normal user, so that you can see how they act in return. We do shut down users who are intentionally causing problems for other users, and I do think that was made evident here.
I do understand that opening a line of communication with Matthew may have been appropriate, and I consider that valuable feedback moving forward.
<3 Jarland
[+] [-] mandatory|9 years ago|reply
At the very least I'm happy that some people have seen this more as a study of this pattern of functionality being an issue instead of this being only a DigitalOcean problem. I've seen (and have been reached out by) multiple people realizing this affects their company as well which has been awesome to watch.
[+] [-] mattaustin|9 years ago|reply
[+] [-] diegorbaquero|9 years ago|reply
Let's remember Linode offers 2x the RAM.
[+] [-] putlake|9 years ago|reply
I'm a long-time Linode customer and use them for all servers that run important services. I use DO for the less important stuff. But Linode has had a not-so-stellar record of data breaches.
[+] [-] thegeekpirate|9 years ago|reply
[+] [-] optforfon|9 years ago|reply
[+] [-] fiatpandas|9 years ago|reply
Gray hat. (Still doesn't justify the idiotic response from DO security team)
[+] [-] Twirrim|9 years ago|reply
[+] [-] DangerousPie|9 years ago|reply
Took a few emails to Cloudflare support to resolve this one. They also didn't seem to care much about the security implications when I questioned them about it.
So this is far from a DO-specific issue...
[+] [-] xxdesmus|9 years ago|reply
[+] [-] ultramancool|9 years ago|reply
Sort of hard to call it a vulnerability on DO's part though - more of an issue with the admins. I think most DNS services operate in this way, really, route53 may be the exception, not the rule.
[+] [-] anilgulecha|9 years ago|reply
This is an absolutely terrible response from DO. If I had anything hosted here, I'd move away ASAP. Seriously, do it.
[+] [-] tombrossman|9 years ago|reply
Where are you going to run off to? How is their security better over there? How many hours of work does that involve?
[+] [-] ultramancool|9 years ago|reply
[+] [-] sattoshi|9 years ago|reply
DO knows people can do this, but they don't want people to do it.
Remembering 2 recurring DNS servers is easier for bulk management than a bunch of different ones.
You don't test services like that, it can negatively affect other users.
Response wasn't perfect but it was reasonable.
[+] [-] zatkin|9 years ago|reply
[+] [-] icebraining|9 years ago|reply
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] hetfeld|9 years ago|reply
[+] [-] V8OaSsoA|9 years ago|reply
Was there a realization into how legitimate users may be affected by this action? Was there a plan to remove those domains from their account after making and disclosing their proof of concept?
Why not stop at 10 or 20, and then alert DO to the findings?
20 thousand was unnecessary.
[+] [-] mixedbit|9 years ago|reply
Someone can easily block you from using S3 static website hosting by adding a bucket with your domain name before you do. Also if you delete a bucket and do not change your DNS, someone can recreate the bucket and will be serving files from your domain.
[+] [-] athrun|9 years ago|reply
This way you are not limited by bucket names and you also avoid any SSL validation errors.
Note: you can set Cloudfront's TTL to 0 if you don't need any caching.
[+] [-] nowprovision|9 years ago|reply
[+] [-] eropple|9 years ago|reply
While his intentions might have been good (and I expect that they were!), that kind of behavior isn't.
[+] [-] corobo|9 years ago|reply
[+] [-] Gracana|9 years ago|reply
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] dumbsecreport|9 years ago|reply
The difference is I didn't exploit 20 thousand domains to make flashy headlines and prove a point about something that isn't even a serious bug.
[+] [-] tszming|9 years ago|reply
[+] [-] supersan|9 years ago|reply
[+] [-] corobo|9 years ago|reply
[+] [-] mitoyarzun|9 years ago|reply
I think this was his mistake.
[+] [-] nvigier|9 years ago|reply
The security team at DigitalOcean has been working to ensure that DO is a safe place for security researchers to identify issues on the Internet as well as at DigitalOcean - security is in everyone's interest. We encourage researchers to contact us when they want to use our platform for this type of work specifically so that we can avoid the types of pain that Matthew encountered while doing his experimentation.
Feel free to reach out to [email protected] and we will be happy to help.
Nick, DigitalOcean Security Director
[+] [-] jbb555|9 years ago|reply
Now I'm in prison because of this so it's really hard for me to put it back."
The article writer is an idiot. He deliberately stole accounts because he could. Just because he then decided to blame the provider because he was able to do this does't make it any more defensible.
If I mug you in the street, should I then post that because I was able to do so it's all your fault? No. I'd go to prison....
[+] [-] djhworld|9 years ago|reply
The first person that replied looks like he just skim read your email or didn't understand the fact you had sinkholed a lot of traffic.
[+] [-] mandatory|9 years ago|reply
[+] [-] devrelm|9 years ago|reply
http://security.stackexchange.com/questions/49612/how-does-d...
[+] [-] i_have_to_speak|9 years ago|reply
[+] [-] unknown|9 years ago|reply
[deleted]
[+] [-] dotBen|9 years ago|reply
EG if foo.com is a working site on your DNS provider, try creating a zonefile for bar.foo.com and see if you can create an A record to point to your own server.
This used to be something shared web hosting services running CPanel/WHM were particularly susceptible to. Clearly, the risks here are both phishing/identity and cookie credential stealing.