Hey, developer of Sentry here. We're submitting a patch that prevents showing any settings on this DEBUG page.
I'd like to mention that we never suggest running Sentry in DEBUG mode, nor do we document how to do this. Sentry does use Django, so it's pretty easy to put pieces together and figure out how to do it.
So while we can help obfuscate this specific issue, running in DEBUG mode always has the potential to surface insecure information due to the nature of it.
This is why you (looking at frameworks) should never use a format that may contain code to store data, especially when the client has control over that data (even if signed). The same vulnerability has occurred in almost every language/framework that does this, including Rails and Java-based ones. Just use something like JSON, which completely avoids code execution vulnerabilities like this. Except of course for the early JavaScript JSON parsers that just used eval for parsing...
Yes, Django realized this 7 years ago [0], and JSON has been the default for the past 5 years[1], it's just that facebook was using an old version of Django, and probably not the default.
Even without the pickle-related vulnerability, exposing your secret key that is securing cookies seems pretty bad and likely to lead to other vulnerabilities, although they'd take longer to find.
And if the secret key were secure, the pickle use would not be vulnerable.
Still, multiple layers of security, yadda yadda, sure.
But this is beyond the pickle issue. I'm not sure I'm completely convinced you should not use pickle for browser cookies that are appropriately cryptographically verified. (although fwiw I believe Rails changed it's default cookie serialization to use json instead of the ruby equivalent to pickle which suffered from the same issues).
PHP has this problem too - never use PHP unserialize() to unserialize hostile (i.e. user-accessible) data. It won't end well for you. I think it's true for any sufficiently powerful serialization format...
I also don't understand why frameworks insist on returning debug data as part of the response. If you are working on something in a development cycle and need debug information, surely you have access to the server that's running in your console. I've always printed debug information there and then send the response. That prevents these kinds of leaks if the server is deployed with debug turned on.
And "something like JSON" could mean YAML, which had it's own share of RCE bugs. It's still better than these object deserialization bugs one can find in Java or Python Pickle, which seems to be even more permissive.
This is a great concrete example why you should never run debug mode on a public server. Django can only do so much for redacting private info. This is also a great example of how insecure pickle is!
I know this is going to get some jeering, but that's one nice thing about .Net's machine.config <deployment retail="true" />, you designate the machine itself as a non-development environment and tracing, debug output, and so on are disabled for all .Net/ASP.Net applications.
That might not work for all edge cases, but broadly there's a lot of machines which are only for non-development/production code, and a system-wide setting makes you a lot safer.
Exactly.. i think you often see a warning about pickle when it comes to safety.
For example, the current documentation has a big, red warning right at the top: https://docs.python.org/3/library/pickle.html
I think maybe frameworks should stop calling it debug mode and start calling it "danger mode". Because clearly people aren't paying attention to what it actually does.
Patreon's screw-up was a lot more embarassing though - they apparently left an actual Python shell exposed to the web for at least a week after someone warned them about it, and their entire user database was exfiltrated and posted on the net as a result.
Some surprising takeaways for me;
Facebook uses a Django app in their infrastructure.
That app was in debug mode, revealing server secrets.
That app was also configured to use the Pickle based session storage, leading to one of the few serious RCE vulnerabilities in Django.
I'll have to remember this next time I think an exploit scenario is too unlikely.
> Quoting the Sentry documentation, system.secret-key is “a secret key used for session signing. If this becomes compromised it’s important to regenerate it as otherwise its much easier to hijack user sessions.“; wow, it looks like it’s a sort of Django SECRET-KEY override!
One wonders why that is even there. Was Django's own session code not good enough?
Note that 5 is true though. Using a secret to sign or encrypt a cookie does normally work, and it's a common practice. Usually the impact of the secret leaking is that you can impersonate anyone, not that you can run arbitrary code, but the practice of using a session secret is common and not a bad practice nor broken inherently.
3 as well I think is unfair. That isn't something facebook implemented or is relying on; it's just the default behavior of django's debug stack. That's entirely on django to do that and lull people into a false sense of security in some cases (though it also probably helps in many cases too, so it might be okay).
The real issues are 1 (leaving debug mode on by accident), 2 (not noticing 1), and 4 (which is a django issue I think, not a facebook one).
This is a good example why you need regular pentests in big companies. Everyone (should) know that using pickle is insecure and everyone (should) know that django debug should be False in production. Still, if the numbers get large enough someone will miss something.
The use of Pickle isn't uncommon for session cookies in Python apps, from what I've seen. Pickle isn't really a problem unless you end up unserializing untrusted data... which a sign+encrypt scheme is supposed to ensure doesn't happen. You just can't leak the secret key or you're in trouble.
Though, there's no excuse for leaving Django debug on in production.
Great article. Big frameworks like Django are nice to get an app started quickly, but if you use them long-term it really pays to study how they work under the hood, read some source code, etc. Although "don't run debug mode on production" is probably in the first page of the tutorial :)
The fact that the machine has a hostname "*.thefacebook.com" doesn't imply that it also runs software of the "Facebook" social media software. So not sure how much impact this exploit would have had.
He received $5,000 for remote code execution on a server in a separate VLAN and containing no user data. Barring another vulnerability to commit something like server-side request forgery across the VLAN, that significantly reduces the actual severity of the vulnerability. You wouldn't be able to chain this execution to impact user accounts or other Facebook servers. You also wouldn't be able to exfiltrate sensitive user data. Realistically, what you'd have is a very privileged server for phishing attacks.
The severity of security vulnerabilities should be judged on their context, not on their classification or category.
I agree he deserves a higher payout, but it was on a segmented server that seems like it didn't have any customer data or other important data. If he were a real attacker, who knows where else he could've pivoted from this server (perhaps it wasn't quite as segmented as Facebook thought). But assuming it truly was pretty isolated, compromising it probably wouldn't have caused any damage.
Regardless, I feel like he deserves at least $15,000 for this, since it is full RCE.
I mean the fix is toggling a single environmental variable from True to False, on a system that isn't normally accessed by customers, so the risk is really small in rolling out the change.
> Pickle is a Python module used to serialize data, but contrarily to JSON or YAML, it allows to serialize objects properties but also methods. In most of the cases, this is not a problem, but one can also serialize an object with code in the __reduce__() method, which is called when the object is unpickled.
Of course. You can lock the process down so that it can't make unexpected system calls. If you deploy in a modern container environment, you can also use container networking to drastically limit what the application environment can talk to on the network. Though it's a less potent mitigation than seccomp and container isolation (and one you get for free once you deploy in a container), you can also limit filesystem access. If you run the application under a non-privileged uid, these mitigations combined can raise the bar somewhat for privilege escalation and pivoting after compromise.
Of course, in a sense, Facebook appears to have accomplished something simpler simply by sacrificing an instance to this application and putting it on a lonely isolated VLAN.
The other responses to your question are pretty weird, since it's obvious that there are things you can do to mitigate the possibility of your Django program literally calling execve or whatever.
There's nothing to be done at the OS level, because the OS doesn't know which things are secret and which things aren't. You can sandbox the Django process (and to a certain extent Facebook were doing this at an even higher level, by having the whole server on a separate VLAN that was (supposedly) away from the important stuff) but that only does so much because the process needs to have enough access to do whatever it was intended to do.
Ultimately the only thing you can do is not run debug mode in production, not use insecure serialization formats and not leak secret keys. If you want to be a bit more pre-emptive about it, avoid language-specific serialization formats especially in dynamic languages ("serialization" that executes arbitrary code seems more common in those) and use taint checking or, better, a type system that can avoid leaking secrets (you likely need rank-2 types to be able to have values that you can use in certain contexts but never access outside them, a la http://okmij.org/ftp/Computation/resource-aware-prog/region-... ). It sounds like Django already has some level of taint-like functionality but this plugin/addon (sentry) then didn't use it properly. So, multiple failures interacting to create this vulnerability - I guess we should take that as a sign of progress compared to the days of basic buffer overflows?
It seems unlikely. The application has a feature to use a secret key to secure everything. The same application then prints out the secret key to anyone that asks. Your OS can't do much about that.
Maybe if you tell your OS "never let any data containing this string leave the machine" you can kind of mitigate this, but it's unlikely to work. The HTTP response is probably compressed. Someone probably base64-encoded the thing and stuck it inside some JSON, which is then base64-encoded again.
Ultimately, it's about managing complexity. Django and the extension punted: Django says it will print anything and everything it has access to, and the extension has a documentation caveat about how bad that would be. One could imagine an API that tries to be resistant to this sort of thing; when the extension is initialized, it deletes the key from the environment dictionary (only partially possible), stores it in a private attribute, and only provides public EncryptAndSignCookie and DecryptAndVerifyCookie methods. This will be better than a documentation caveat, maybe, but the truly ambitious debug mode will probably get the key and print it out.
(I would also point out that I'm not a fan of storing state in encrypted+signed cookies, if only because there is no way to revoke a stolen cookie without revoking every cookie ever. If you have the state on the server to store a revocation list, you might as well just store everything there and never have this problem.)
To be clear, this isn't an issue with the Django source, as much it was a miss configured server - having debug mode on enabled the stack trace to be leaked which yielded the secret key. Debug mode shouldn't be used in production, and django probably shouldn't be responsible for snipping every possible value out of debug traces.
Depends on your threat model. For all we know chroot, docker, and SELinux could have been in active usage on this machine.
But Facebook may view a compromise on their edge network and potentially one of their trusted servers as serious even if actual damage on that server itself is limited.
> I found a Sentry service hosted on 199.201.65.36
I do not remember on top of my head now but I think there are few scanning software to find all the running apps on a remote machine. If you are aware then please share
[+] [-] mattrobenolt|7 years ago|reply
I'd like to mention that we never suggest running Sentry in DEBUG mode, nor do we document how to do this. Sentry does use Django, so it's pretty easy to put pieces together and figure out how to do it.
So while we can help obfuscate this specific issue, running in DEBUG mode always has the potential to surface insecure information due to the nature of it.
Our patch to suppress this information entirely can be found here: https://github.com/getsentry/sentry/pull/9516
[+] [-] matharmin|7 years ago|reply
[+] [-] collinmanderson|7 years ago|reply
[0] https://groups.google.com/d/msg/django-developers/YwlZ9m9k1b...
[1] https://github.com/django/django/commit/b0ce6fe656873825271b...
[+] [-] jrochkind1|7 years ago|reply
And if the secret key were secure, the pickle use would not be vulnerable.
Still, multiple layers of security, yadda yadda, sure.
But this is beyond the pickle issue. I'm not sure I'm completely convinced you should not use pickle for browser cookies that are appropriately cryptographically verified. (although fwiw I believe Rails changed it's default cookie serialization to use json instead of the ruby equivalent to pickle which suffered from the same issues).
[+] [-] smsm42|7 years ago|reply
[+] [-] vhost-|7 years ago|reply
[+] [-] rmetzler|7 years ago|reply
[+] [-] someguy101010|7 years ago|reply
[deleted]
[+] [-] Areading314|7 years ago|reply
[+] [-] ssebastianj|7 years ago|reply
[+] [-] Someone1234|7 years ago|reply
That might not work for all edge cases, but broadly there's a lot of machines which are only for non-development/production code, and a system-wide setting makes you a lot safer.
[+] [-] buster|7 years ago|reply
The Django project also has a lot of warnings about the pickle serializer: https://docs.djangoproject.com/en/2.1/topics/http/sessions/
[+] [-] rrcaptain|7 years ago|reply
[+] [-] cc-d|7 years ago|reply
[+] [-] makomk|7 years ago|reply
[+] [-] tekromancr|7 years ago|reply
I'll have to remember this next time I think an exploit scenario is too unlikely.
[+] [-] antoncohen|7 years ago|reply
https://instagram-engineering.com/web-service-efficiency-at-...
[+] [-] gandreani|7 years ago|reply
[+] [-] beefhash|7 years ago|reply
One wonders why that is even there. Was Django's own session code not good enough?
[+] [-] smsm42|7 years ago|reply
1. Enabling debug mode in production
2. Running publicly-accessible app with publicly-accessible crash screens without any monitoring system noticing it's happening
3. Relying on auto-cleaning in debug facilities to sanitize security information (never works)
4. Using over-powered serialization protocol which allows for code execution for storing user-accessible data
5. Thinking that merely signing content prevents abuse of (4)
Not bad.
[+] [-] TheDong|7 years ago|reply
3 as well I think is unfair. That isn't something facebook implemented or is relying on; it's just the default behavior of django's debug stack. That's entirely on django to do that and lull people into a false sense of security in some cases (though it also probably helps in many cases too, so it might be okay).
The real issues are 1 (leaving debug mode on by accident), 2 (not noticing 1), and 4 (which is a django issue I think, not a facebook one).
[+] [-] Grollicus|7 years ago|reply
[+] [-] arachnids|7 years ago|reply
[+] [-] Polycryptus|7 years ago|reply
Though, there's no excuse for leaving Django debug on in production.
[+] [-] QuadrupleA|7 years ago|reply
[+] [-] rhacker|7 years ago|reply
[+] [-] amelius|7 years ago|reply
[+] [-] green_on_black|7 years ago|reply
[+] [-] throwawaymath|7 years ago|reply
The severity of security vulnerabilities should be judged on their context, not on their classification or category.
[+] [-] meowface|7 years ago|reply
Regardless, I feel like he deserves at least $15,000 for this, since it is full RCE.
[+] [-] tptacek|7 years ago|reply
[+] [-] 0x8BADF00D|7 years ago|reply
[+] [-] mandeepj|7 years ago|reply
ping -4 facebook.com
results in 157.240.18.35. Maybe, author used some other way to get those IPs. Can anyone throw a light on this?
[+] [-] Cthulhu_|7 years ago|reply
[+] [-] jrowley|7 years ago|reply
[+] [-] Thaxll|7 years ago|reply
30.07.2018 00:00 CEST : initial disclosure with every details.
09.08.2018 18:10 CEST : patch in place.
[+] [-] BFatts|7 years ago|reply
One suggestion: You use "However" quite a bit. Not sure if you intended to show your thought process as it evolved, but that is the feeling I got.
[+] [-] nickthemagicman|7 years ago|reply
[+] [-] Kiro|7 years ago|reply
Why does it run __reduce__?
[+] [-] srcmap|7 years ago|reply
I am thinking something like selinux, docker or chroot - a bit like internal firewall for Django (or any other webapp).
Any suggestions on the links to latest best practices?
[+] [-] tptacek|7 years ago|reply
Of course, in a sense, Facebook appears to have accomplished something simpler simply by sacrificing an instance to this application and putting it on a lonely isolated VLAN.
The other responses to your question are pretty weird, since it's obvious that there are things you can do to mitigate the possibility of your Django program literally calling execve or whatever.
[+] [-] lmm|7 years ago|reply
Ultimately the only thing you can do is not run debug mode in production, not use insecure serialization formats and not leak secret keys. If you want to be a bit more pre-emptive about it, avoid language-specific serialization formats especially in dynamic languages ("serialization" that executes arbitrary code seems more common in those) and use taint checking or, better, a type system that can avoid leaking secrets (you likely need rank-2 types to be able to have values that you can use in certain contexts but never access outside them, a la http://okmij.org/ftp/Computation/resource-aware-prog/region-... ). It sounds like Django already has some level of taint-like functionality but this plugin/addon (sentry) then didn't use it properly. So, multiple failures interacting to create this vulnerability - I guess we should take that as a sign of progress compared to the days of basic buffer overflows?
[+] [-] jrockway|7 years ago|reply
Maybe if you tell your OS "never let any data containing this string leave the machine" you can kind of mitigate this, but it's unlikely to work. The HTTP response is probably compressed. Someone probably base64-encoded the thing and stuck it inside some JSON, which is then base64-encoded again.
Ultimately, it's about managing complexity. Django and the extension punted: Django says it will print anything and everything it has access to, and the extension has a documentation caveat about how bad that would be. One could imagine an API that tries to be resistant to this sort of thing; when the extension is initialized, it deletes the key from the environment dictionary (only partially possible), stores it in a private attribute, and only provides public EncryptAndSignCookie and DecryptAndVerifyCookie methods. This will be better than a documentation caveat, maybe, but the truly ambitious debug mode will probably get the key and print it out.
(I would also point out that I'm not a fan of storing state in encrypted+signed cookies, if only because there is no way to revoke a stolen cookie without revoking every cookie ever. If you have the state on the server to store a revocation list, you might as well just store everything there and never have this problem.)
[+] [-] jrowley|7 years ago|reply
[+] [-] Someone1234|7 years ago|reply
But Facebook may view a compromise on their edge network and potentially one of their trusted servers as serious even if actual damage on that server itself is limited.
[+] [-] C14L|7 years ago|reply
> wow, it looks like it’s a sort of Django SECRET-KEY override!
Sentry uses its "own" secret key, so Django doesn't know it should be stripped from the stacktrace.
A simple `DEBUG = False` in `settings.py` would fix the bug. You don't have Django run in debug mode in production.
[+] [-] mandeepj|7 years ago|reply
I do not remember on top of my head now but I think there are few scanning software to find all the running apps on a remote machine. If you are aware then please share
[+] [-] Jwarder|7 years ago|reply
I've also seen plenty of references to shodan.io, but I have no experience with it.
[+] [-] bluntfang|7 years ago|reply
[+] [-] ericnyamu|7 years ago|reply
[deleted]