Howdy, Hacker News. I’m the CISO of Yahoo and I wanted to clear up some misconceptions.
Earlier today, we reported that we isolated a handful of servers that were detected to have been impacted by a security flaw. After investigating the situation fully, it turns out that the servers were in fact not affected by Shellshock.
Three of our Sports API servers had malicious code executed on them this weekend by attackers looking for vulnerable Shellshock servers. These attackers had mutated their exploit, likely with the goal of bypassing IDS/IDP or WAF filters. This mutation happened to exactly fit a command injection bug in a monitoring script our Sports team was using at that moment to parse and debug their web logs.
Regardless of the cause our course of action remained the same: to isolate the servers at risk and protect our users' data. The affected API servers are used to provide live game streaming data to our Sports front-end and do not store user data. At this time we have found no evidence that the attackers compromised any other machines or that any user data was affected. This flaw was specific to a small number of machines and has been fixed, and we have added this pattern to our CI/CD code scanners to catch future issues.
As you can imagine this episode caused some confusion in our team, since the servers in question had been successfully patched (twice!!) immediately after the Bash issue became public. Once we ensured that the impacted servers were isolated from the network, we conducted a comprehensive trace of the attack code through our entire stack which revealed the root cause: not Shellshock. Let this be a lesson to defenders and attackers alike: just because exploit code works doesn’t mean it triggered the bug you expected!
I also want to address another issue: Yahoo takes external security reports seriously and we strive to respond immediately to credible tips. We monitor our Bug Bounty (bugbounty.yahoo.com) and security aliases ([email protected]) 24x7, and our records show no attempt by this researcher to contact us using those means. Within an hour of our CEO being emailed directly we had isolated these systems and begun our investigation. We run one of the most successful Bug Bounty programs in the world and I hope everybody here will participate and help us keep our users safe.
We’re always looking for people who want to keep nearly a billion users safe at scale. [email protected]
Few weeks ago, I reported to your team that some of the yahoo servers' SSL cert were expired, acknowledged but no one want to fix it (until I post it here and finally get them updated...your site was showing security warning to your users for 2 weeks)
One of your awesome engineers replied the issue with expired SSL cert: "there do not appear to be any security implications as a direct result of this behavior"
I've long had experience with: exploits working for the wrong reason, and also the reverse, failing for the wrong reason.
For example, way back in the day before ISS bought my company, somebody claimed their IDS was vulnerable to an IMAP evasion. They actually weren't, but that specific test triggered a wholly separate (and much worse) bug that made it look like it was evadable. I laughed and laughed.
It looks like the guy who originally posted this has pretty much accused you of flat out lying about this[1]. What do you have to say to his comments, particularly about the sports servers being internal.
If you want hackers to report you vulnerabilities via Yahoo Bug Bounty Program, at least pay more than 50$ for a minimum bounty, 50$ is a joke https://hackerone.com/yahoo ...
Interesting I saw the Y debug tool on sports.yahoo.com . It said Y Confidential and had a web bug on it and something else in red. Also, was on the yahoo.com root domain. I saw this over a few days. Did anyone else see it? I couldn't click on anything and it was on the lower right hand corner of the screen.
My opinionated answer to the statement released by Yahoo!:
I won’t sit here and say that I think they are lying. To make such an accusation would only prove me to be a fool not having ammunition in the weapon before I fire it. I will, however, say that I believe – in my opinion – that this is a wordplay and a game of semantics. First off, there are several “shellshock” exploits. The term “shellshock” as the media has portrayed it is to execute the vulnerability or vulnerabilities recently discovered in bash by means of delivering the following payload: () { :; }; <commands>
When we look at this payload, what we are actually seeing is a function definition, not the execution of or calling of that function, with the regards to the following: () { :; };
The actual “arbitrary code” to be executed last past that point, where we’re no longer defining a function, but instead – giving instructions to be executed on the operating system via Bash. One could inevitable argue that taking the payload, and modifying it to look like: () { whatever; }; /bin/bash –c ‘id;uname –a’ could, or even could not, be identified as “shellshock” in the manner of which it was portrayed to be by the media. However, the fact still remains that this “payload” would cause the execution of the commands proceeding the {};
When we look at the other SIX (6) payloads that essentially accomplish the same exact thing: https://shellshocker.net – we see that modification of the “payload” does not stop the blatant fact that the same underlying results are achieved via the same exact vulnerable code in the Bash shell.
My response to Yahoo! : Please issue out the UNTAMPERED and UNMODIFIED apache logs, showing the payload delivered to your “sports API’s” - and other researchers, and potentially shareholders, determine what the underlying cause was. Furthermore, to state that this resulted in bypassing your “IDS/IDP” and “WAF filters” makes me wonder exactly what kind of IDS/IDP and WAF filtering you’re imploring. I’m willing to bet there were exact phrase match filters looking for what most sites have identified as the “Shellshock” vulnerability, I.e. “() { :; };” and preventing the scripts and/or bash from being executed once that string was identified. That could have been with something as simple as a wrapper for bash… And considering the IPS/IDS didn’t pick up on outbound IRC connections on port(s) 6660 and/or 6667, which NO internal server would have a reason to be connecting to, I can only say that your concept of IDS/IDP is seemingly inadequate in my professional opinion. So, once again, since the vulnerability has seemingly been “patched,” I urge you to release the details of the vulnerability in the script, and also explain why it is that the initial compromise appears to have been on a web-facing box with public access to it, and found amongst a botnet running a perl script that had self-spreading and searching capabilities based around the “shellshock” vulnerability? You are comparing apples and oranges, when in all actuality, you should be comparing “to-may-to” to “to-mah-toh."
While we are at it, let me go on a little tangent: I have Yahoo mail for android which i use as a dump of my emails, and I get perhaps 100 emails per day. After about 4-6 weeks, Yahoo mail app becomes so slow, that it is no longer possible to even scroll through emails (on Nexus 5). I have to clear Yahoo app's data cache, and reconfigure to make it fast again. Perhaps it's time to take a look at this: when you have things decaying and breaking like that, it encourages hackers to look extra hard for vulnerabilities, since it's a reasonable assumption that other things are neglected, too.
I should note that Yahoo's android mail app is probably the most viable part of the whole Yahoo business now.
This writeup doesn't really get to the point so, the tl;dr
He was looking for places to exploit shellshock by googling for cgi scripts. Most of the ones he did find had already been hit by someone using a perl script that made them join an irc channel that was being used as CnC. He also joined it and monitored it. A bunch of different yahoo boxes were in the channel and he saw some of them get rooted.
Winzip.com has been hacked as well. Do not trust their binaries.
Either this will be headline news tomorrow, or it will be suppressed in its entirety. The OP will probably go to prison, unfortunately, as they will not differentiate between this and black hat intrusion - the case will be judged by someone who saw his nephew using a computer, once, and they will go after him, because they know who he is, and will not have any joy identifying the actual intruders, and this will just go further to demonstrate that the spy agency dragnets are as useful as a chocolate teapot in preventing and acting against actual crime.
I hope he contacted a great defence attorney and the ACLU at the same time as Yahoo and the FBI.
What's the issue with Perl scripts on production web servers? Probably 90% of my (homegrown) scripts are written in Perl. What does Perl vs. PHP vs. Ruby vs. $languageoftheweek have to do with anything?
I think therein lies the problem: yapache. It's their own version of (modified) Apache. So when these bugs like shellshock come out, it's harder to patch your own home-grown version.
Am I the only one that thinks this kind of thing would be cool to see? I've seen logs of attacks, but I've never watched a botnet irc live. that would be crazy for me. Not really moving the conversation forward, but is this so commonplace that I'm the odd man for marveling?
I read that shortly after it was originally published. And I thought to myself: COOL!
I was seventeen. I had a spare Windows 95c (or was it 98se?) box laying around, and some experience with inctrl5, a linux box which could operate as a router, and some basic knowledge of tcpdump(1). Importantly, I could also script the behavior of an IRC client.
At the time I was a channel operator in a relatively popular IRC channel on EFnet... "Don't ask to ask!" :) Users would come in and request assistance with malware all the time, so I was already roughly familiar with the mechanisms of infection and CnC.
This is a long story that I must cut short: I ended up in the same CnC room as Gibson did. Not the same type--the same one. I met some of the people in the story. :D
I used to do that for fun. It was a lot easier back when SDBot and AgoBot was the shit.
The trick I used was to go on some xdcc network, change nickname to one similar to the bots and just wait. Sooner or later one of the botnet owners tried to authenticate and soon after I would exit with a ping-timeout.
Then you could just log into one of them, get a list of processes and download the one with a random name. It was pretty easy and you could get your hands on a few thousand bots in a weekend.
Oh the joy of running "!uninstall" while the owners was in the chatroom...
You might find this paper about stealing a botnet interesting [0]. Even though its five years old, the crazy stuff these researchers found is still amazing.
To be honest, it's not that interesting. If it's a well configured irc host then you will not be seeing any of the other bots, and all you will occasionally see is a command coming by from a generically named operator. Some botnet irc's are lazily configured, and will let you see all of the other bots as part of the channel, but generally will not let you speak. The bots usually have nicknames built from the host's computer name, username, country, etc.
It's interesting, but in itself is not that exciting in my opinion.
It was interesting the first few times watching them. Sometimes the commands are not even authenticated so you could do fun things like write a text file saying their computer was infected and then open it with notepad... or other things.[0] You aren't going to find many large scale botnets that still use IRC though. It is really amateur hour CnC.
[0] It probably is not really advisable to do even 'helpful' actions such as that, but when you are young you do careless things.
Expose a vulnerable linux VM to the raw internet. Wait for it to get infected. Find the process thats connected to the cnc server using lsof. use gcore to dump its memory to a file. cat that into strings and look for the irc channel and server. Or just watch it all in wireshark but thats kinda boring. Have fun and stay safe.
This guy works in the security industry and yet he couldn't google "yahoo security" to find their security contact email address (second result for me)? He was also unaware that Yahoo runs a Bug Bounty Program?
courageous? ethical probing? lol. Obviously he just wants free publicity. Pretty stupid move overall, notifying the fbi and all... unbelievable. And his crappy web server where he brags about it all is down by now too, so much for the free publicity.
Is this a new phenomenon? I always felt that Yahoo's systems weren't secure. Until I shut down my Yahoo accounts, it would be a semi-regular occurrence for both my Yahoo email and IM to send out spam to everyone in my Yahoo contacts list. Am I wrong? I've since shut down my account since I got sick of dealing with it.
That's not a Yahoo hack though. When that happens it is almost always your local machine that has been breached by a virus which simply reads the locally stored contact list. And to answer your question, no, it is not a regular occurrence for Yahoo, or any of the major players, to have their servers hacked.
A pretty interesting read, despite the occasionally challenging style. If the email to Mayer had been a little more focused then perhaps it might have punched through. But in any case, kudos to the author for doing the work and writing it up. I learned a few things. But at the same time, no kudos for using what might be the most unfortunate metaphor ever. I suggest avoiding any future attempts at picturesque description.
I wonder what best practice is for consumer websites which have domains like yahoo.com which has mostly customers, and yahoo-inc.com for corporate, for things like security@ addresses. It's reasonable someone wouldn't know about yahoo-inc.com.
[+] [-] secalex|11 years ago|reply
Earlier today, we reported that we isolated a handful of servers that were detected to have been impacted by a security flaw. After investigating the situation fully, it turns out that the servers were in fact not affected by Shellshock.
Three of our Sports API servers had malicious code executed on them this weekend by attackers looking for vulnerable Shellshock servers. These attackers had mutated their exploit, likely with the goal of bypassing IDS/IDP or WAF filters. This mutation happened to exactly fit a command injection bug in a monitoring script our Sports team was using at that moment to parse and debug their web logs.
Regardless of the cause our course of action remained the same: to isolate the servers at risk and protect our users' data. The affected API servers are used to provide live game streaming data to our Sports front-end and do not store user data. At this time we have found no evidence that the attackers compromised any other machines or that any user data was affected. This flaw was specific to a small number of machines and has been fixed, and we have added this pattern to our CI/CD code scanners to catch future issues.
As you can imagine this episode caused some confusion in our team, since the servers in question had been successfully patched (twice!!) immediately after the Bash issue became public. Once we ensured that the impacted servers were isolated from the network, we conducted a comprehensive trace of the attack code through our entire stack which revealed the root cause: not Shellshock. Let this be a lesson to defenders and attackers alike: just because exploit code works doesn’t mean it triggered the bug you expected!
I also want to address another issue: Yahoo takes external security reports seriously and we strive to respond immediately to credible tips. We monitor our Bug Bounty (bugbounty.yahoo.com) and security aliases ([email protected]) 24x7, and our records show no attempt by this researcher to contact us using those means. Within an hour of our CEO being emailed directly we had isolated these systems and begun our investigation. We run one of the most successful Bug Bounty programs in the world and I hope everybody here will participate and help us keep our users safe.
We’re always looking for people who want to keep nearly a billion users safe at scale. [email protected]
[+] [-] tszming|11 years ago|reply
Few weeks ago, I reported to your team that some of the yahoo servers' SSL cert were expired, acknowledged but no one want to fix it (until I post it here and finally get them updated...your site was showing security warning to your users for 2 weeks)
One of your awesome engineers replied the issue with expired SSL cert: "there do not appear to be any security implications as a direct result of this behavior"
[+] [-] legohead|11 years ago|reply
Not knocking on you or anything, just more interested to know if all exploits have been patched against, more than the # of patches applied.
[+] [-] robertgraham|11 years ago|reply
For example, way back in the day before ISS bought my company, somebody claimed their IDS was vulnerable to an IMAP evasion. They actually weren't, but that specific test triggered a wholly separate (and much worse) bug that made it look like it was evadable. I laughed and laughed.
[+] [-] _b8r0|11 years ago|reply
[1] - http://www.futuresouth.us/wordpress/?p=25
[+] [-] somebugbounty|11 years ago|reply
[+] [-] dude3|11 years ago|reply
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] JonathanDHall|11 years ago|reply
I won’t sit here and say that I think they are lying. To make such an accusation would only prove me to be a fool not having ammunition in the weapon before I fire it. I will, however, say that I believe – in my opinion – that this is a wordplay and a game of semantics. First off, there are several “shellshock” exploits. The term “shellshock” as the media has portrayed it is to execute the vulnerability or vulnerabilities recently discovered in bash by means of delivering the following payload: () { :; }; <commands>
When we look at this payload, what we are actually seeing is a function definition, not the execution of or calling of that function, with the regards to the following: () { :; };
The actual “arbitrary code” to be executed last past that point, where we’re no longer defining a function, but instead – giving instructions to be executed on the operating system via Bash. One could inevitable argue that taking the payload, and modifying it to look like: () { whatever; }; /bin/bash –c ‘id;uname –a’ could, or even could not, be identified as “shellshock” in the manner of which it was portrayed to be by the media. However, the fact still remains that this “payload” would cause the execution of the commands proceeding the {};
When we look at the other SIX (6) payloads that essentially accomplish the same exact thing: https://shellshocker.net – we see that modification of the “payload” does not stop the blatant fact that the same underlying results are achieved via the same exact vulnerable code in the Bash shell.
My response to Yahoo! : Please issue out the UNTAMPERED and UNMODIFIED apache logs, showing the payload delivered to your “sports API’s” - and other researchers, and potentially shareholders, determine what the underlying cause was. Furthermore, to state that this resulted in bypassing your “IDS/IDP” and “WAF filters” makes me wonder exactly what kind of IDS/IDP and WAF filtering you’re imploring. I’m willing to bet there were exact phrase match filters looking for what most sites have identified as the “Shellshock” vulnerability, I.e. “() { :; };” and preventing the scripts and/or bash from being executed once that string was identified. That could have been with something as simple as a wrapper for bash… And considering the IPS/IDS didn’t pick up on outbound IRC connections on port(s) 6660 and/or 6667, which NO internal server would have a reason to be connecting to, I can only say that your concept of IDS/IDP is seemingly inadequate in my professional opinion. So, once again, since the vulnerability has seemingly been “patched,” I urge you to release the details of the vulnerability in the script, and also explain why it is that the initial compromise appears to have been on a web-facing box with public access to it, and found amongst a botnet running a perl script that had self-spreading and searching capabilities based around the “shellshock” vulnerability? You are comparing apples and oranges, when in all actuality, you should be comparing “to-may-to” to “to-mah-toh."
[+] [-] cft|11 years ago|reply
[+] [-] mukyu|11 years ago|reply
He was looking for places to exploit shellshock by googling for cgi scripts. Most of the ones he did find had already been hit by someone using a perl script that made them join an irc channel that was being used as CnC. He also joined it and monitored it. A bunch of different yahoo boxes were in the channel and he saw some of them get rooted.
[+] [-] madaxe_again|11 years ago|reply
Winzip.com has been hacked as well. Do not trust their binaries.
Either this will be headline news tomorrow, or it will be suppressed in its entirety. The OP will probably go to prison, unfortunately, as they will not differentiate between this and black hat intrusion - the case will be judged by someone who saw his nephew using a computer, once, and they will go after him, because they know who he is, and will not have any joy identifying the actual intruders, and this will just go further to demonstrate that the spy agency dragnets are as useful as a chocolate teapot in preventing and acting against actual crime.
I hope he contacted a great defence attorney and the ACLU at the same time as Yahoo and the FBI.
[+] [-] tantalor|11 years ago|reply
> they will not differentiate between this and black hat intrusion
Should they? This reads like textbook unauthorized access to a computer system,
> A quick `ps aux` on the box yielded...
This isn't just poking at web servers to see what secrets they freely reveal, this is trespass.
[+] [-] swasheck|11 years ago|reply
[+] [-] chatmasta|11 years ago|reply
[+] [-] flashman|11 years ago|reply
[+] [-] philjohn|11 years ago|reply
[+] [-] gopalv|11 years ago|reply
All the yapache & yphp security fixes and is all undone by a a .pl with +ExeCGI.
They used to run "crack days" where all of us used to get kicks out of breaking & entering prod, whatever means available.
Was a fun way to weed through such low-hanging issues, by a highly motivated (i.e otherwise bored) crowd.
I wonder if they still have them.
[+] [-] jlgaddis|11 years ago|reply
[+] [-] discardorama|11 years ago|reply
[+] [-] mrt0mat0|11 years ago|reply
[+] [-] daveloyall|11 years ago|reply
First, read this. Note the date. http://www.crime-research.org/library/grcdos.pdf
I read that shortly after it was originally published. And I thought to myself: COOL!
I was seventeen. I had a spare Windows 95c (or was it 98se?) box laying around, and some experience with inctrl5, a linux box which could operate as a router, and some basic knowledge of tcpdump(1). Importantly, I could also script the behavior of an IRC client.
At the time I was a channel operator in a relatively popular IRC channel on EFnet... "Don't ask to ask!" :) Users would come in and request assistance with malware all the time, so I was already roughly familiar with the mechanisms of infection and CnC.
This is a long story that I must cut short: I ended up in the same CnC room as Gibson did. Not the same type--the same one. I met some of the people in the story. :D
[+] [-] ishatmypants|11 years ago|reply
The trick I used was to go on some xdcc network, change nickname to one similar to the bots and just wait. Sooner or later one of the botnet owners tried to authenticate and soon after I would exit with a ping-timeout.
Then you could just log into one of them, get a list of processes and download the one with a random name. It was pretty easy and you could get your hands on a few thousand bots in a weekend.
Oh the joy of running "!uninstall" while the owners was in the chatroom...
[+] [-] seccess|11 years ago|reply
[0] http://www.net.t-labs.tu-berlin.de/teaching/ws0910/IS_semina...
[+] [-] tokenizerrr|11 years ago|reply
It's interesting, but in itself is not that exciting in my opinion.
[+] [-] mukyu|11 years ago|reply
[0] It probably is not really advisable to do even 'helpful' actions such as that, but when you are young you do careless things.
[+] [-] JonnieCache|11 years ago|reply
[+] [-] Nyr|11 years ago|reply
http://cl.ly/image/2E3D2H2B2d2t
[+] [-] r721|11 years ago|reply
I am doubtful that FBI would share their plans and/or actions with OP.
[+] [-] onewaystreet|11 years ago|reply
[+] [-] seren|11 years ago|reply
[+] [-] daveloyall|11 years ago|reply
Which makes his signature line pretty interesting. :)
> A fool learns only from himself. A wise man will learn from the fool.
So he's got this 'honest fool' thing going for him. If he can marry that with meticulous record keeping, maybe he'll be OK.
Of course, IANAL.
But ffs, I'm sick of this world where the defense "Wait, you misunderstand--I'm the GOOD guy!" isn't good enough. Why isn't it?
[+] [-] thefreeman|11 years ago|reply
[+] [-] alvarosm|11 years ago|reply
[+] [-] benmmurphy|11 years ago|reply
[+] [-] chaostheory|11 years ago|reply
[+] [-] rll|11 years ago|reply
[+] [-] sz4kerto|11 years ago|reply
futuresouth.us got Hacker News'd this morning.
[+] [-] milankragujevic|11 years ago|reply
[+] [-] markbnj|11 years ago|reply
[+] [-] acostoss|11 years ago|reply
[1]: http://webcache.googleusercontent.com/search?q=cache:http://...
[+] [-] rdl|11 years ago|reply
[+] [-] toomuchtodo|11 years ago|reply
https://www.ietf.org/rfc/rfc2142.txt
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] primitivesuave|11 years ago|reply
[+] [-] eyeareque|11 years ago|reply
[+] [-] unknown|11 years ago|reply
[deleted]
[+] [-] meritt|11 years ago|reply