In the abstract, i understand, why anyone would post this. I get it, and maybe i'm getting older or something, but part of me equates posting this sort of thing with pointing at an acquaintance on the street with his fly down and having a laugh with your friends rather than telling him and having a chuckle after he's zipped up.
But the whole self-righteous and incredibly passive-aggressive "man, i think these guys have a huge problem..." followed by the "man, these guys need to shape up" and "n00b mistake!" are unproductive to the extreme.
I mean, find a bug, report it, move on. Maybe i've got some unrealistic notion of karma or general human benevolence or something, but it seems hard to believe that this is such a difficult path to take especially when nearly everyone commenting has to deal with bs of this sort day-in/out.
If someones fly is unzipped, I'd point it out because that's the sort of accident that can happen to even the most competent and discerning.
If someones pants are sagged around their knees, I expect them to have noticed this themselves, and by walking around in public they've accepted the possibility of ridicule.
I consider having a non-beta site to be 'walking around in public', and revealing code and internal data (however small) when there is a malformed request to be 'walking around in public with your pants sagged at your knees'.
-----
That said, rappers don't have to buy belts and maybe the culture of the 'new economy' is such that some people do not have to value security.
This isn't a bug, it's a failure of basic security principles. Imagine if the super to your apartment complex accidentally mailed a box full of duplicate keys to a local methadone clinic. That's not an embarrassing mistake, it's a catastrophic error bordering on criminal negligence. Drawing attention to it is meant to not only deepen the embarrassment and thus encourage fixing the underlying problem so that it never happens again but also to let other people know (such as tumblr users) about the amateurishness of the tumblr operation and finally to encourage other developers out there to avoid making the same mistakes.
I agree with you, but the problem is we will never know about the times when others may have found a bug, reported it and moved on. In this kind of situation, it's only the negative stories which will get out.
Just in case… I'm really really sorry in case I shouldn't have done this. I think I already said that, but when I saw this I wasn't as lucid as I am right now and just thought about dropping it here as nobody in my twitter TL would have done anything.
Yet I have worked in a lot of different environments with PHP over the time and this never happened to me (but I was close to). It's a big, big mistake, not just a tiny error.
TL;DR: Amateurish PHP developers at Tumblr fuck up; HN developers who don't know PHP that well make wildly incorrect assumptions about PHP.
People, I know it's en vogue to bash PHP (just wait, in a few years it'll be Ruby and Python - remember, PHP was once hyped, too, and now it's going in the other direction) - but if you criticize PHP, could you at least try to sound like you've actually developed in PHP for more than a week?
Because most of the negative comments here about PHP have absolutely nothing to do with PHP as such - the Tumblr error in question has to do with incompetent programmers. If you read The Daily WTF you'll know that incompetent programmers can screw up no matter what language they're using.
Amateurish? Incompetent? It seems a bit extreme to make those generalizations because someone made a typo in a PHP file that managed to hit production.
Any programming language's flaws can be made up for by sufficient skill and discipline on the part of the developers. That does not make those flaws any less real.
We saw this from facebook few years ago. Now with tumblr. Is there something at core of php that makes this inevitable? I ask this as a concerned php dev(and not out of snark).
They've got their Twitter / Facebook / oAuth secret keys in there. Doesn't that mean everyone who sees this can act as Tumblr post to those services on behalf of users?
Nope, you need the users' token to perform that (given that they gave post to wall permission). However using Facebook secret key, you can ban users from using your app. But then again, you need user id.
Whilst I hope tumblr correct the problem rather quickly as it is a major problem, I find those jumping to blame are forgetting one small problem. No programmer is perfect, typos are easy to mistake on any keyboard and it will happen to everyone no matter how much of a ‘ninja rockstar poodle’ they think they are.
I hate to see someone else work in the clear like this. It’s like popping a zit before your first date. It’s painful and will show up for day/s afterwards. Now I know what will be today’s headline I can bypass techmeme.
Yes its a big boo boo. It’s a massive security risk and to some it may feel like the end of the world but by then it will be tomorrow. Passwords will be reset, keys will be replaced and the valley will be talking about something else. Hopefully it won’t be someone else’s mistake.
P.S Don’t forget to test your code before deploying – now you know why.
Typos are very easy to make - but that's why you need to first test your code locally, test your code in a development environment, have others test and approve your code in a staging environment before a small typo gets to production where something like this can happen.
Which is why you have test servers and never ever make live edits to deployed code. I find it exceedingly easy to say that this was kind of incompetent.
I always use a include for any hashes or passwords in a separate file. When I started learning PHP I exposed my MySQL database password more times then I could keep track of.
It does hammer home the point of staging before deploying. Also the point of making sure you vary your passwords between sites.
I know it's easy to criticize, but far out Tumbler, you guys really have to get your act together - the downtime and general laggy-ness is at least understandable, but there is no excuse for absolute newbie foul-ups like this.
Although, on the plus side, having a site that mashes up tumbler as a content provider certainly has given us plenty of opportunity to fine tune and improve our caching strategy.
I sure hope they realize they just broadcasted the pass for the "tumblr3" database user, as well as their Twitter, Facebook, Recaptcha and other secret keys.
You can clearly see all the routes in the app. 400+ routes and only 11 controllers. Most routes are concentrated on 3-4 controllers. Each of those controllers has got to be 10,000-20,000 lines apiece.
The dashboard controller alone has approximately 120 actions.
Can someone please explain this a little more? My basic understanding is that they incorrectly opened a PHP tag and exposed the code. If that's the case, wouldn't the page have appeared broken in development? Or have been found during testing?
Their site has a front-end controller (www/dispatch.php) to which all requests get routed (by a mod_rewrite rule, perhaps), rather than having .php files in the web root for each page. This file sets up the environment -- among other things, it registers a custom error handler and includes a config file (config/config.php) that defines a bunch of constants. Then it dispatches the request to the appropriate controller, based on the URL.
Someone (probably a sysadmin) edited the config file and accidentally changed the opening tag (<?php). This caused PHP to output its contents, rather than parsing and executing it. Since there was no output buffer active, those contents were sent directly to the user, which caused HTTP response headers also to be sent automatically. That's the big first line you see in the output.
Since no error had actually been triggered by this point, execution continued. It tried to set an HTTP header ("P3P: CP="P3P_CP"", whatever that means). However since HTTP headers had already been sent, this did trigger an error, which was passed to the custom error handler, which sent some debug output (the rest of the output you see) and stopped execution.
If one accepts that errors like these happen, I guess it would be a good idea to have an automated way to quickly change passwords on all the services that are used. Does anyone have some citations for literature on how to deal with that?
Just like you'd want to have a well described (preferably automated) way to restore from backups, you should also have one for resetting all passwords. Such a process is also useful for protecting against disgruntled employees.
Regardless of how it occurred, and given that this isn't a language issue, I have two questions....
Does anyone know how long was it actually in this state?
(There's a heck of a lot of entries in the Google search quoted in another comment, but then how often does Google index tumblr?)
Did no-one at least press F5 or CMD-R after making the edit, let alone run tests? Quality control is the real issue. I can easily imagine myself making this mistake, typo's are the source of the majority of my bugs, but I find it hard to imagine taking more than 10 seconds to notice it.
[+] [-] nsfmc|15 years ago|reply
But the whole self-righteous and incredibly passive-aggressive "man, i think these guys have a huge problem..." followed by the "man, these guys need to shape up" and "n00b mistake!" are unproductive to the extreme.
I mean, find a bug, report it, move on. Maybe i've got some unrealistic notion of karma or general human benevolence or something, but it seems hard to believe that this is such a difficult path to take especially when nearly everyone commenting has to deal with bs of this sort day-in/out.
[+] [-] true_religion|15 years ago|reply
If someones pants are sagged around their knees, I expect them to have noticed this themselves, and by walking around in public they've accepted the possibility of ridicule.
I consider having a non-beta site to be 'walking around in public', and revealing code and internal data (however small) when there is a malformed request to be 'walking around in public with your pants sagged at your knees'.
-----
That said, rappers don't have to buy belts and maybe the culture of the 'new economy' is such that some people do not have to value security.
[+] [-] InclinedPlane|15 years ago|reply
[+] [-] revorad|15 years ago|reply
[+] [-] adrinavarro|15 years ago|reply
Yet I have worked in a lot of different environments with PHP over the time and this never happened to me (but I was close to). It's a big, big mistake, not just a tiny error.
[+] [-] stowaway|15 years ago|reply
People, I know it's en vogue to bash PHP (just wait, in a few years it'll be Ruby and Python - remember, PHP was once hyped, too, and now it's going in the other direction) - but if you criticize PHP, could you at least try to sound like you've actually developed in PHP for more than a week?
Because most of the negative comments here about PHP have absolutely nothing to do with PHP as such - the Tumblr error in question has to do with incompetent programmers. If you read The Daily WTF you'll know that incompetent programmers can screw up no matter what language they're using.
[+] [-] nbpoole|15 years ago|reply
[+] [-] mustpax|15 years ago|reply
[+] [-] yuvadam|15 years ago|reply
Just in case.
edit: oh man. Check out what Google has on this[1]
edit: in readable form on github gist[2]
[1] - http://www.google.com/webhp?hl=en#sclient=psy&hl=en&...
[2] - https://gist.github.com/29c5c5970d1f3313abd1
[+] [-] erikpukinskis|15 years ago|reply
http://www.google.com/search?q=site%3Atumblr.com+m3MpH1C0Koh...
[+] [-] zaidf|15 years ago|reply
[+] [-] Maxious|15 years ago|reply
[+] [-] sudhirj|15 years ago|reply
I hope they've changed them.
[+] [-] xuki|15 years ago|reply
[+] [-] smallwords|15 years ago|reply
I hate to see someone else work in the clear like this. It’s like popping a zit before your first date. It’s painful and will show up for day/s afterwards. Now I know what will be today’s headline I can bypass techmeme.
Yes its a big boo boo. It’s a massive security risk and to some it may feel like the end of the world but by then it will be tomorrow. Passwords will be reset, keys will be replaced and the valley will be talking about something else. Hopefully it won’t be someone else’s mistake.
P.S Don’t forget to test your code before deploying – now you know why.
[+] [-] elliottcarlson|15 years ago|reply
[+] [-] joebananas|15 years ago|reply
[+] [-] FirstHopSystems|15 years ago|reply
It does hammer home the point of staging before deploying. Also the point of making sure you vary your passwords between sites.
[+] [-] jasonlotito|15 years ago|reply
[+] [-] mrspeaker|15 years ago|reply
Although, on the plus side, having a site that mashes up tumbler as a content provider certainly has given us plenty of opportunity to fine tune and improve our caching strategy.
[+] [-] timerickson|15 years ago|reply
[+] [-] sudhirj|15 years ago|reply
[+] [-] KevBurnsJr|15 years ago|reply
The dashboard controller alone has approximately 120 actions.
[+] [-] datasink|15 years ago|reply
[+] [-] adrinavarro|15 years ago|reply
[+] [-] radq|15 years ago|reply
[+] [-] csears|15 years ago|reply
[+] [-] angadsg|15 years ago|reply
[+] [-] oscardelben|15 years ago|reply
[+] [-] mattmight|15 years ago|reply
PHP doesn't do well on either count.
Writing secure PHP code isn't impossible, but it's tedious even for seasoned developers.
[+] [-] holdenc|15 years ago|reply
[+] [-] bluesnowmonkey|15 years ago|reply
Their site has a front-end controller (www/dispatch.php) to which all requests get routed (by a mod_rewrite rule, perhaps), rather than having .php files in the web root for each page. This file sets up the environment -- among other things, it registers a custom error handler and includes a config file (config/config.php) that defines a bunch of constants. Then it dispatches the request to the appropriate controller, based on the URL.
Someone (probably a sysadmin) edited the config file and accidentally changed the opening tag (<?php). This caused PHP to output its contents, rather than parsing and executing it. Since there was no output buffer active, those contents were sent directly to the user, which caused HTTP response headers also to be sent automatically. That's the big first line you see in the output.
Since no error had actually been triggered by this point, execution continued. It tried to set an HTTP header ("P3P: CP="P3P_CP"", whatever that means). However since HTTP headers had already been sent, this did trigger an error, which was passed to the custom error handler, which sent some debug output (the rest of the output you see) and stopped execution.
[+] [-] oscardelben|15 years ago|reply
[+] [-] unknown|15 years ago|reply
[deleted]
[+] [-] adolph|15 years ago|reply
[+] [-] troels|15 years ago|reply
Just like you'd want to have a well described (preferably automated) way to restore from backups, you should also have one for resetting all passwords. Such a process is also useful for protecting against disgruntled employees.
[+] [-] nikcub|15 years ago|reply
[+] [-] jentulman|15 years ago|reply
Does anyone know how long was it actually in this state? (There's a heck of a lot of entries in the Google search quoted in another comment, but then how often does Google index tumblr?)
Did no-one at least press F5 or CMD-R after making the edit, let alone run tests? Quality control is the real issue. I can easily imagine myself making this mistake, typo's are the source of the majority of my bugs, but I find it hard to imagine taking more than 10 seconds to notice it.