A little tip for people trying to armor themselves against this problem: If your app reaches out to do network transactions, it really ought to block localhost. However, bear in mind that localhost isn't "127.0.0.1"... it's "127.0.0.0/8" (or "127.x.x.x" if you don't casually speak CIDR). Ping 127.2.88.33 on your console now... you'll see replies.
On the flip side, if you're doing a security test like this, I've gotten mileage out of convincing apps to access local resources with things like 127.88.23.245, precisely because the developer blocked 127.0.0.1 specifically and thought they were done.
You should also usually block all internal and external IPs for your entire network, but especially in the cloud this can begin to get tricky. Still, you should.
Also worth noting: don't just regex-check the URL with (localhost|127.*) or something similar. Any hostname could point to 127.0.0.1.
iptables with --uid-owner denying traffic to local/private IP space (plus infrastructure-specific stuff like EC2's instance metadata service) would probably be the best option.
Great point. Also, possibly all reserved IPv4 spaces (see the Apache proxy mentioned in the article), not just 127/localhost (and don't forget 169.254/16 as the article points out). It's hard to do this right, since there might be use cases where local networks should be allowed, but better to err on the side of caution.
This seems a little like playing whack-a-mole security. Is there a comprehensive list of what should be blocked by default unless your specifically needs it?
The next big mistake is configuring your packet filters and application ACLs to only filter requests coming in from the internet; they can still make requests to 54.175.131.207 or whatever the IP is and the source of the request packets will be from the loopback interface.
And the IPv6 equivalent is "::1". Did they even set up basic packet filtering or ACLs for IPv6 requests? If they have no public IPv6 address, they probably never did.
Quick question. Is this a field that layman web developers/script developers should know? Or is this more of the domain of information security/network security? I don't make apps as a product, but I do like to tinker with stuff and make projects that sometimes my friends use. Not really sure what counts as a network transaction.
> Ping 127.2.88.33 on your console now... you'll see replies.
People's results here might vary depending on their operating system. For me, these pings were returned on a vagrant box running ubuntu, but dropped (not blocked) on the host system.
I don't think this is true on Windows. The hostname is tied to the IP which is tied to the loopback adapter and it defaults to the IPv6 addr anyway. The hosts file has it commented out as the literal 127.0.0.1 for those who need it on IPv4.
"Pocket does not provide monetary compensation for any identified or possible vulnerability."
Cheapskates. This could have cost them money if somebody abusive had discovered it first. He deserves a monetary award.
[edit] Should we be concerned about the massive number of people listed on that page who have found security problems with Pocket? I counted 153 separate people...
It's a freemium service that doesn't depend on ad revenue. Maybe they're not made of money as Facebook and Google are and that's the reason for not compensating.
Exploits in a proprietary cloud service like this are harser to sell, at least se acquisition programs are expressly uninterested in them. I wonder what effect that has on bug bounties
Yet another service discovered which was built/deployed with no regard for security whatsoever. I'm beginning to realize - this is the norm. Security is the least important thing for most of the IT companies.
I guess the DevOps trend (i.e. not hiring sysadmins) should take it's share of blame. Or maybe it's the other way around - you don't care for security, so there is no point in hiring security experts?
Oh Mozilla, why couldn't you resist the money. Your recent so called "services" are not welcome. You know it. But well, money makes the world go around.
How much did Telefonica pay you for the Hello integration?
When we decided to add a reading list to Firefox, we had two options: build and maintain our own service and integrations, or partner with an established player who had sane privacy and data access policies. We chose the latter.
Pocket began life as a Firefox add-on, and is now integrated into literally hundreds of applications. Embracing that is a reasonable choice in terms of sustainability and value for users.
For what it's worth, I do not believe any money changed hands in the Pocket deal, but I don't know that authoritatively.
No money changed hands. Mozilla integrated Pocket because they wanted such a feature in their browser (apparently, so did users), and Pocket already existed (better than reimplementing it from scratch)
FWIW: Reading this on Pale Moon which is basically a rebranded fork of Firefox before Aurora.
Can't say anything about PaleMoons security but I like the old design better, the team seems more responsive and they haven't yet (had the chance to) give in to hypocrites and fire "Brendan Eich".
Really interesting write up! I'm surprised they are still running in EC2-Classic. However, even if they are, security groups should still be restrictive enough to prevent some of the things discussed. For example, bypassing the load balancer shouldn't be possible. A security group applied to the back end instances should only allow HTTP/S traffic from the load balancer group. SSH security groups should only allow inbound traffic from known IPs (like the office network), etc. Unfortunately, not enough people do this, and once you can query instance meta data or obtain an SSH key, it's game over.
If you were Pocket how would you handle the vulnerability created by having internal services hit user-supplied URLs?
Some ideas:
- Move the service doing the fetching to an untrusted network. At least it would be unable to access any internal services and any compromises there would be hopefully limited. You still have the problem that the local machines there could potentially be compromised.
- Validate / verify the URL to ensure it's not hitting
anything internal. This sounds hard. Pre-resolve the name and check to see if the IP is in an internal range? Seems easy to get our of date as your network changes. Make sure to repeat for any redirects? Is there a better way to validate?
- Ensure that all internal services require authentication. This also sounds hard and easy to miss something.
This is the problem with services that store user information: it is highly probably that vulnerabilities like these exist. Security is rarely given the time and attention it requires.
I'm not trying to single out Pocket; they are just the latest evidence that even in the few cases where "you can trust us with your data" is said honestly, it isn't a promise that can be kept in practice.
One more vulnerability which is possible when you can request an instance's metadata: Any AMI roles which have been given to that instance (for example to enable S3 access or decrypt data using AWS' key service) would be visible.
These keys are rotated relatively frequently, but it opens up a whole new level of exploits against the company which runs those AWS servers.
Is there a name for this sort of attack? We were just protecting against some similar attacks earlier this week, and it would have been nice to have a short name to refer to them as instead of "that attack vector where we make unrestricted HTTP requests based on user input".
Most developers don't think (or know) about security issues like this, and running something as root avoids a whole class of deploy problems, so I understand why this can happen.
It's certainly not right, and indicative of the need for either a system administrator with an eye for deployment best practices, or an explicit security position who sets up audits to look for these kinds of flaws.
I'm really not a fan of EC2 exposing instance meta data as a RESTful HTTP API running on Local-Link IP addresses. If its only supposed to be queried locally, why aren't these just environment variables? Perhaps they are dynamic and that won't work but come on!
At the very least, run it on localhost:10101 or something. Don't give us another range to have to filter!
One of the biggest selling points of EC2 (at least for me) is that you get a real VM with a real kernel and userspace, and not some "user-friendly" thing the provider made with loads of alien services running as root. So EC2 can't define vars inside your instances + as you said, they can be dynamic.
localhost:SOMETHING will also not work for same reason - they would need to run a service inside your instance.
There is one more popular solution to the metadata problem - providing it as an emulated cdrom, which would be also vulnerable in this case. And it can't be dynamic.
That IP for the metadata is internal to Amazons AWS. Under normal circumstances it isn't exposed to the outside network. Because Pocket is remembering URLs and fetching them for us, it leaks it's own private network in the responses.
Both of those solutions require Amazon to do "something" to the instance itself and thus limit the kinds of machines you can run on EC2. Custom kernels, FreeBSD, etc..
Their current metadata approach works across OSes.
And, yes, the data are dynamic. Things like AWS access keys change over time and can be accessible via the metadata if you've given the instances IAM profiles. I'm surprised the author didn't mention this.
I agree that the approach feels uncomfortable in general but it seems like the best approach for the functionality they wanted.
If you want a laundry list of SSRF methods you should protect against, a great place to start is this slide deck from a talk at ONsec a couple of years ago:
Security researchers out there: on what side of "the line" do you view this kind of exploitation to be? It was not done for nefarious purposes, but it did involve intentionally accessing resources that were clearly not intended to be accessed, like /etc/passwd. Would you worry if you did this that the company might call the police instead of thanking you?
The one thing I don't like about this article (and indeed, much of the discourse around the Pocket integration) is its characterization of the Pocket integration itself. It calls it an “opt-out non-removable [extension]”. The truth is that you can easily disable it just as you can easily disable many other things that Firefox includes by-default. In fact, if you use Classic Theme Restorer (I use it not because I dislike australis, but because I really do not want a navigation toolbar), it has an option baked in to disable Pocket along with webrtc, et al.
Admittedly, I suppose it would be nice if Firefox actually packaged Pocket as a real extension that could be removed from the Extensions menu, but they have already integrated several things without using that schema.
I still use firefox, just with more and more things disabled, because none of the other browsers out there even come close to having what I need in a GUI browser (though, I would note that I'm evermore tempted to abandon GUI browsing altogether).
Either way, the write-up is great, and everything in the article other than that one characterization (which rubbed me a bit the wrong way in the wake of all the fevered discussions around the Pocket Integration) was a truly enjoyable read. Not to mention, it's great that the Pocket devs fixed things quickly; that's always a plus!
>you can easily disable it just as you can easily disable many other things that Firefox includes by-default
>it would be nice if Firefox actually packaged Pocket as a real extension that could be removed from the Extensions menu
These two statements you made seem to corroborate his characterization of it being "opt-out" (meaning on by default, but capable of being disabled) and "non-removable" (baked into firefox as opposed to an extension). Not sure what you find wrong with his characterization.
>Grab ssh private keys from autoprovisioned EC2 user’s home directory using 301 redirect to file URI (after all, we’re running as root, we can read them).
This is not a fair assumption to make. Maybe they are running a LSM like AppArmor.
What is more important - the existence of SSH private keys on the EC2 instances us unlikely... There is chance there are SSH private keys there, but they would most probably be SSH deploy keys for some private repo (configuration management, software).
i prefer using offline bookmark..specially the bookmark manager of Opera browser is impressive.
=============================================
and for online bookmarking there is 'raindrop.io'
[+] [-] jerf|10 years ago|reply
On the flip side, if you're doing a security test like this, I've gotten mileage out of convincing apps to access local resources with things like 127.88.23.245, precisely because the developer blocked 127.0.0.1 specifically and thought they were done.
You should also usually block all internal and external IPs for your entire network, but especially in the cloud this can begin to get tricky. Still, you should.
And don't forget IPv6.
[+] [-] pfg|10 years ago|reply
iptables with --uid-owner denying traffic to local/private IP space (plus infrastructure-specific stuff like EC2's instance metadata service) would probably be the best option.
[+] [-] jamiesonbecker|10 years ago|reply
https://en.wikipedia.org/wiki/Reserved_IP_addresses
[+] [-] rcthompson|10 years ago|reply
[+] [-] peterwwillis|10 years ago|reply
And the IPv6 equivalent is "::1". Did they even set up basic packet filtering or ACLs for IPv6 requests? If they have no public IPv6 address, they probably never did.
[+] [-] SpaceManNabs|10 years ago|reply
[+] [-] escape_goat|10 years ago|reply
People's results here might vary depending on their operating system. For me, these pings were returned on a vagrant box running ubuntu, but dropped (not blocked) on the host system.
[+] [-] stimpy77|10 years ago|reply
[+] [-] mike-cardwell|10 years ago|reply
"Pocket does not provide monetary compensation for any identified or possible vulnerability."
Cheapskates. This could have cost them money if somebody abusive had discovered it first. He deserves a monetary award.
[edit] Should we be concerned about the massive number of people listed on that page who have found security problems with Pocket? I counted 153 separate people...
[+] [-] alimbada|10 years ago|reply
[+] [-] mst|10 years ago|reply
[+] [-] eli|10 years ago|reply
[+] [-] wcummings|10 years ago|reply
[+] [-] bluedevilzn|10 years ago|reply
[+] [-] drzaiusapelord|10 years ago|reply
Heaven forbid, people submit security issues because they want to be helpful or care about the project.
[+] [-] skarap|10 years ago|reply
I guess the DevOps trend (i.e. not hiring sysadmins) should take it's share of blame. Or maybe it's the other way around - you don't care for security, so there is no point in hiring security experts?
[+] [-] BenjaminWill|10 years ago|reply
How much did Telefonica pay you for the Hello integration?
But sure, our surfing history will be secure ...
https://blog.mozilla.org/advancingcontent/2015/05/21/providi...
Did you guys acutally read your PR-bullshit here?
But soon a new small, fast, free, secure open-source browser will arrive and Mozilla will be history. But your pocket full. Well done.
[+] [-] callahad|10 years ago|reply
Pocket began life as a Firefox add-on, and is now integrated into literally hundreds of applications. Embracing that is a reasonable choice in terms of sustainability and value for users.
For what it's worth, I do not believe any money changed hands in the Pocket deal, but I don't know that authoritatively.
[+] [-] Manishearth|10 years ago|reply
No money changed hands. Mozilla integrated Pocket because they wanted such a feature in their browser (apparently, so did users), and Pocket already existed (better than reimplementing it from scratch)
[+] [-] rntz|10 years ago|reply
Mozilla almost certainly isn't getting any money out of the Pocket integration deal. Mozilla has far more money than Pocket does.
[+] [-] lucian1900|10 years ago|reply
Really? How exactly? Who will fund its development?
[+] [-] reitanqild|10 years ago|reply
Can't say anything about PaleMoons security but I like the old design better, the team seems more responsive and they haven't yet (had the chance to) give in to hypocrites and fire "Brendan Eich".
[+] [-] cddotdotslash|10 years ago|reply
[+] [-] ddlatham|10 years ago|reply
Some ideas:
- Move the service doing the fetching to an untrusted network. At least it would be unable to access any internal services and any compromises there would be hopefully limited. You still have the problem that the local machines there could potentially be compromised.
- Validate / verify the URL to ensure it's not hitting anything internal. This sounds hard. Pre-resolve the name and check to see if the IP is in an internal range? Seems easy to get our of date as your network changes. Make sure to repeat for any redirects? Is there a better way to validate?
- Ensure that all internal services require authentication. This also sounds hard and easy to miss something.
[+] [-] pdkl95|10 years ago|reply
I'm not trying to single out Pocket; they are just the latest evidence that even in the few cases where "you can trust us with your data" is said honestly, it isn't a promise that can be kept in practice.
[+] [-] falcolas|10 years ago|reply
These keys are rotated relatively frequently, but it opens up a whole new level of exploits against the company which runs those AWS servers.
[+] [-] schmichael|10 years ago|reply
[+] [-] Manishearth|10 years ago|reply
[+] [-] mrbig4545|10 years ago|reply
[+] [-] falcolas|10 years ago|reply
It's certainly not right, and indicative of the need for either a system administrator with an eye for deployment best practices, or an explicit security position who sets up audits to look for these kinds of flaws.
[+] [-] nadams|10 years ago|reply
[1] http://forums.somethingawful.com/showthread.php?noseen=0&thr...
[+] [-] billyhoffman|10 years ago|reply
At the very least, run it on localhost:10101 or something. Don't give us another range to have to filter!
[+] [-] skarap|10 years ago|reply
localhost:SOMETHING will also not work for same reason - they would need to run a service inside your instance.
There is one more popular solution to the metadata problem - providing it as an emulated cdrom, which would be also vulnerable in this case. And it can't be dynamic.
[+] [-] JamesLeonis|10 years ago|reply
[+] [-] zeendo|10 years ago|reply
Their current metadata approach works across OSes.
And, yes, the data are dynamic. Things like AWS access keys change over time and can be accessible via the metadata if you've given the instances IAM profiles. I'm surprised the author didn't mention this.
I agree that the approach feels uncomfortable in general but it seems like the best approach for the functionality they wanted.
[+] [-] robn_fastmail|10 years ago|reply
http://www.slideshare.net/d0znpp/ssrf-attacks-and-sockets-sm...
Its thrilling and terrifying :)
[+] [-] hundt|10 years ago|reply
[+] [-] halosghost|10 years ago|reply
Admittedly, I suppose it would be nice if Firefox actually packaged Pocket as a real extension that could be removed from the Extensions menu, but they have already integrated several things without using that schema.
I still use firefox, just with more and more things disabled, because none of the other browsers out there even come close to having what I need in a GUI browser (though, I would note that I'm evermore tempted to abandon GUI browsing altogether).
Either way, the write-up is great, and everything in the article other than that one characterization (which rubbed me a bit the wrong way in the wake of all the fevered discussions around the Pocket Integration) was a truly enjoyable read. Not to mention, it's great that the Pocket devs fixed things quickly; that's always a plus!
[+] [-] sigmar|10 years ago|reply
>it would be nice if Firefox actually packaged Pocket as a real extension that could be removed from the Extensions menu
These two statements you made seem to corroborate his characterization of it being "opt-out" (meaning on by default, but capable of being disabled) and "non-removable" (baked into firefox as opposed to an extension). Not sure what you find wrong with his characterization.
[+] [-] unknown|10 years ago|reply
[deleted]
[+] [-] luxoria|10 years ago|reply
This is not a fair assumption to make. Maybe they are running a LSM like AppArmor.
[+] [-] skarap|10 years ago|reply
[+] [-] _navaneethan|10 years ago|reply
[+] [-] Iuz|10 years ago|reply
[+] [-] dafrankenstein2|10 years ago|reply
[deleted]
[+] [-] dafrankenstein2|10 years ago|reply