Regarding Apple's denial, there are other publications that corroborate the Bloomberg story. Previously, Apple has denied security incidents even when multiple outlets report it. For example, last year, The Information reported Apple discovered malware on Super Micro servers in their development and production environments [1]. As a result, the Information claimed that Apple ended up terminating its relationship with Super Micro.
In response to the report, an Apple spokesperson denied there was a security incident, stating: "We’re not aware of any data being transmitted to an unauthorized party nor was any infected firmware found on the servers purchased from this vendor."
However, based on sources from within Apple, Ars Technica claimed Apple employees did find compromised firmware in Apple's design lab. Super Micro SVP of Technology also reported Apple terminated its relationship with them.
It does not seem like this story is true. If it's true, it makes absolutely no sense for Apple and Amazon to attack Bloomberg. Sure, a national security letter could force them to stay quiet, or maybe even to lie to the public and say it didn't happen, but it can't make them criticize Bloomberg. Attacking Bloomberg if the story is true is only going to convince Bloomberg to dig deeper. And the story isn't even that bad for Amazon or Apple - it's much worse for US-China relations than it is for either of those companies.
The key technical detail of what these chips are allegedly doing also does not make sense. From the article:
the chips allowed the attackers to create a stealth doorway into any network that included the altered machines
How can you get around a firewall by using a compromised machine that's part of the internal network?
I don't think Bloomberg reporters are just making stuff up. But the technical confusion here makes me suspect that the government officials who leaked this story just didn't understand the details of a real incident that happened, and in the leaking the story got mangled into inaccuracy.
The actual details on the hardware are also sketchy. Based on my reading of the article, this wasn't a chip swap where one chip is replaced by a backdoored version. The article implies that this was an extra chip so either dozens (possibly hundreds) of engineers were in on the operation from the beginning to slip the chip into the design undetected or the chip was mounted without any changes to the board design. The former is much riskier than backdooring the chips and the latter, as far as I know, has not been done before with a nontrivial chip.
Fitting any chip capable of exfiltrating a nontrivial amount of data onto a modern motherboard without going through many rounds of simulation or significantly impacting performance, while also putting it in a place it is capable of intercepting valuable data is practically impossible. Hell, just getting the right power domains wired to the chip is going to be tough enough.
It depends. These chips were specifically connected to the BMC/IPMI:
>The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.
Is your OOB-management VLAN/network firewalled off from outgoing access to the internet? If not, this chip can make requests out to another host, who gives it code to run. This code can then either collect data and send it back out, or do other nefarious things.
> And the story isn't even that bad for Amazon or Apple - it's much worse for US-China relations than it is for either of those companies.
If true, it's pretty bad. For one thing customers will be more wary of buying hardware from China. If they move manufacturing elsewhere (or, for Amazon, source hardware elsewhere) they would have higher costs and lower margins, if they don't and a competitor does they lose business.
On top of that, people are already wary of the privacy implications of these companies having all your data, but most are willing to trust major US companies. If it looks like they can't keep it secure against random Chinese companies engaged in industrial espionage, more customers will conclude that operating servers in-house is the better choice.
It goes beyond that. If this story were in fact true, what we're witnessing is large companies whose entire business is built upon user trust setting fire to themselves. Who'd ever trust AWS again with anything of consequence if it turns out the denial is false?
I don't buy it as written. There's something else here.
> How can you get around a firewall by using a compromised machine that's part of the internal network?
Reverse shell depending on how you have the firewall set up. Most firewalls don't block outbound packets, and even if they do you can tunnel over an open port.
> If it's true, it makes absolutely no sense for Apple and Amazon to attack Bloomberg.
If you are a major company, are you going to use AWS if you believe their hardware is pwned?
I find the lack of real technical information quite surprising, not just on bloomberg, but here as well. The previous big thread that you can scroll endlessly has anecdotes and chit-chat but nothing on the actual technical details.
That is my question as well, without getting their hands on the machine, how do the attackers start the attacks? And in case of AWS, since everything is so virtualized, does it make sense to install a hardware backdoor if it cant even map to your true target？
It could probably make more sense as a backdoor when the hackers get hold the access to the physical device, it is however quite incredible to me that this hack is designed for a remote network access, and could go unnoticed from infosec within the company if it is truly sending packets outside the firewall...
For major major reports like this, reputed newspapers are rarely wrong. See WSJ John Crareyou of Theranos. Theranos vehemently denied it, MSM was with Theranos and at least they weren’t outright against them. Then they did more digging etc.
The Washpo digging on Roy Moore, the NYT digging on Clinton Foundation, etc. these are experienced reporters having 15-30 credible sources, evidence etc. every time this happens, everyone denies and slowly little by little the story finds more evidence and facts and it becomes true.
This is why a free press is soooo important
Why would Apple disclose that millions of their products could be hacked and you’ve lost all your privacy. Who would trust them? They’d lose billions, regulations would come etc. it’s in the best interest for every party to deny.
On the other hand, Newsweek was wrong about Satoshi.
My suspicion is that there are many cases where the Chinese government is actually trying to insert backdoors into things, and that in particular Supermicro really has been compromised by the Chinese government, but the technical details of this chip are incorrect. That explains why so many government officials are eager to leak information to Bloomberg, but at the same time the technical details don't really make sense.
It isn't really in Amazon's and Apple's best interest to lie about this. When Gmail got hacked by the Chinese government, Google was pretty honest about it. China has a lot of resources so you can't really expect companies to fend off 100% of attacks on their own; it makes sense for them to acknowledge this publicly and get help from the US government when needed.
Can National Security Letters can be used to require companies to issue outright lies to the public? The bloomberg article indicates that the investigation is not complete, so that could be on explanation for the apparent disconnect between a seemingly well-reported story and the unsually forceful denials.
Today's Money Stuff ponders another angle of this: whether those companies would be committed securities fraud by abiding by such a request:
> I do not think that the securities laws explicitly allow companies to make false statements of material fact if required for national security, but you could see giving them a pass here.
What's more likely is someone on the inside of one of these companies was privy to the discovery, but not the follow up.
I've been in such a situation myself where I was in the room during what LOOKED like a DDOS by Akamai 10 minutes after we got off the phone with them to turn down their CDN services.
In much the same way as Apple is refuting this claim, after a few weeks of internal debate above our pay grade we decided we didn't see it and it didn't exist and therefore it didn't.
Perhaps these things aren't handled at the level of "company"?
For a national security-related issue, you might just include the minimum number of people that need to know, which would naturally not include your PR team. Then when an article like this comes out, the PR team responds in exactly the way they would if it were an outright fabrication, which is what the goal would be.
No but the gov might be unhappy that the secret is out while they were trying to use the chips to infiltrate/reverse engineer wherever the chips were reporting to? Amazon has gov contracts it wants to protect so Im not surprised they would deny this at the govs request, and Apple's #1 product is privacy so having something like this show up on their servers would undermine that product strategy so I can understand their denial too (homegrown or at the govs request)
> But in my experience, Apple PR does not lie. Do they spin the truth in ways that favor the company? Of course. That’s their job. But they don’t lie, because they understand that one of Apple’s key assets is its credibility. They’d say nothing before they’d lie.
If the US files injunctions for relief, federal court (FISA) justices can probably compel you to do just about anything. Presumably the Bill of Rights still pertains and takes precedent. But it's not ludicrous to think that you could be compelled to issue a denial or not issue an affirmation of the public claims.
Bloomberg reported that the startup I worked for was for sale and the founders were pitching it to potential buyers. Internally, they vehemently denied the report and said Bloomberg completely made it up. Bloomberg was dead on the money. That's not to say every journalist/article they publish is going to be spot on, but I definitely give them enough credit that I will believe this report until proven unequivocally not true by the accused.
A CEO shopping a private startup has every incentive to lie to his employees in that scenario and little reason not to.
On the other hand, Apple and Amazon have huge reasons not to blatantly lie about this. The plaintiffs bar would be all over both companies for material false and misleading statements if this turned out to be true (see also Elon's recent experiences with the SEC).
Apple and Amazon are certainly incentivized to not admit bad facts, to spin facts, to issue misleading unclear statements that read as a denial but are not, etc. But I really don't think the legal teams at either company would let executives get away with issuing such full-throated and clear denials if they were untrue.
What if Apple or Amazon didn't find the chips but other investigators found them in Apple/Amazon servers. That would make their denial wording technically true. They didn't say no chips were found, they just said they didn't find any.
This is this is similar to all the backdoored Cisco devices the FBI found all over the government. If they are doing it at all they would have a complex plan and this would be one of the many approaches. Even scarier IMHO are the CPU fabrication hacks that add in an imperceptible backdoor directly in the chip logic. A recent report showed how CPUs can be backdoored at critical points in the fabrication process by a single operator that would be incredibly hard to detect. We are talking instruction patterns that charge capacitor buffers that allow privileged access once a threshold is reached. Amazing really.
>But they don’t lie, because they understand that one of Apple’s key assets is its credibility. They’d say nothing before they’d lie.
This is so typical of Gruber as an evangelist.
There's no way that Apple would remain silent on this even if they KNEW it were true. The only possible move is denial.
Silence is validation or uncertainty, a statement of ambiguity will tank the stock and reputation as experts, recognition of even partial truth could possibly destroy their supply lines overnight.
I honestly think the corporate denials here need to be outright ignored because they have so much to lose. A story of this magnitude is basically like pointing a gun to someone's head and asking them for permission to pull the trigger.
"But in my experience, Apple PR does not lie. Do they spin the truth in ways that favor the company? Of course. That’s their job. But they don’t lie, because they understand that one of Apple’s key assets is its credibility. They’d say nothing before they’d lie."
As a few people pointed out in the other thread, didn't they pretty explicitly deny they were involved with PRISM?
As someone on a slack server I use pointed out: a server wouldn't need to phone home, could have a planned failure and request an RMA, even if the system was wiped when it came back, it could have data stored somewhere secretly, and why you may not find anything in an audit.
I did a penetration test and security assessment for a major electronics manufacturer
whose parts are likely in every smartphone and laptop. I identified almost certain compromise
by the Chinese government with full access to modify the manufacturing specs using the
access paths I identified.
They chose to bury my findings as it would cause a huge stock hit. Sadly, NDA.
I'm not surprised in the slightest.
Hard to give an anonymous comment any weight without the slightest verification. Since we know that adversaries of freedom use social media as a disinformation vector, the only thing you can do is ignore them or encourage them to find a way to legitimately disclose the information protected under NDA -- perhaps to the press or a legislator who could help make it possible to invalidate NDAs that keep secrets that make us vulnerable.
While we would cooperate with any government investigation, we are not aware of any investigation regarding this topic nor have we been contacted by any government agency in this regard. We are not aware of any customer dropping Supermicro as a supplier for this type of issue.
Every major corporation in today’s security climate is constantly responding to threats and evolving their security posture. As part of that effort we are in regular contact with a variety of vendors, industry partners and government agencies sharing information on threats, best practices and new tools. This is standard practice in the industry today. However, we have not been in contact with any government agency regarding the issues you raised.
Furthermore, Supermicro doesn’t design or manufacture networking chips or the associated firmware and we, as well as other leading server/storage companies, procure them from the same leading networking companies.
> not much bigger than a grain of rice, that wasn’t part of the boards’ original design.
'original design' is hard to verify without help from SuperMicro.
> Amazon reported the discovery to U.S. authorities, sending a shudder through the intelligence community.
... especially for Amazon. Unless Bloomberg claims here that Amazon got parts of this SKU and compared them to newer parts of the SKU and found differences?
Is it legit to revise your design in terms of changes to passives without rev'ing the part and notifying downstream supply chain. Could the grain-of-rice 'microchip' be a different or new resistor/cap? Could it be logic masquerading as a passive?
> In one case, the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached, according to one person who saw pictures of the chips.
Like a passive in a blind/buried via?
Anyways, for everyone who claims "it can't be done", "this is implausible" -- you're probably being a little naive. US intelligence agencies do shipment interdiction and adulterate products for this purpose, [1] why couldn't/shouldn't China do the same?
I have no idea how things actually work in companies like Apple or Amazon, but would it be "normal" (at their size/scale and given their surely advanced knowledge in technology) to have inspections on the hardware they use (inspections at hardware level of the kind capable of showing these modifications)?
OK. So what's the minimum remote system management capability needed in a modern data center? The major cloud sysadmin people should figure this out, write a spec, and insist that's all that goes in. If Amazon AWS and Google wrote a spec for this, the manufacturers would fall into line.
Boards are shipping with way too much remote access capability.
It's not like you need to look at system busses via the network. You're not going to debug a broken board remotely, you're going to turn it off and replace it. Now that this is an identified problem, it's time to put IPMI and its ilk back in its cage.
Amazon bids for CIA contracts; Elemental bids for Amazon's contracts; Super Micro bids for Elemental's contracts and the PLA "bids" for Super Micro's contracts. Looks like Ouroboros, the global supply chain episode.
A little suspect when they say " giving them access to the most sensitive code even on machines that have crashed or are turned off."
When it is turned off? A magical chip that works without electricity is much more valuable than any data which could be exfiltrated from servers. A chip that works without power changes the world more than the iPhone... in the iPhone Age
The US needs to make electronics supply chain sovereignty the number one priority of the federal defense budget. Why are we fighting useless drone wars when our country is being attacked on a daily basis in the cyber realm?
If this story is true, it will be an escalation clearly and willfully by the PRC from mere state-sanctioned economic espionage to an act of war against the US.
[+] [-] tristanj|7 years ago|reply
In response to the report, an Apple spokesperson denied there was a security incident, stating: "We’re not aware of any data being transmitted to an unauthorized party nor was any infected firmware found on the servers purchased from this vendor."
However, based on sources from within Apple, Ars Technica claimed Apple employees did find compromised firmware in Apple's design lab. Super Micro SVP of Technology also reported Apple terminated its relationship with them.
I believe we are seeing the same situation here.
[1] https://arstechnica.com/information-technology/2017/02/apple...
[+] [-] IBM|7 years ago|reply
[+] [-] lacker|7 years ago|reply
The key technical detail of what these chips are allegedly doing also does not make sense. From the article:
the chips allowed the attackers to create a stealth doorway into any network that included the altered machines
How can you get around a firewall by using a compromised machine that's part of the internal network?
I don't think Bloomberg reporters are just making stuff up. But the technical confusion here makes me suspect that the government officials who leaked this story just didn't understand the details of a real incident that happened, and in the leaking the story got mangled into inaccuracy.
[+] [-] akiselev|7 years ago|reply
Fitting any chip capable of exfiltrating a nontrivial amount of data onto a modern motherboard without going through many rounds of simulation or significantly impacting performance, while also putting it in a place it is capable of intercepting valuable data is practically impossible. Hell, just getting the right power domains wired to the chip is going to be tough enough.
[+] [-] parliament32|7 years ago|reply
>The illicit chips could do all this because they were connected to the baseboard management controller, a kind of superchip that administrators use to remotely log in to problematic servers, giving them access to the most sensitive code even on machines that have crashed or are turned off.
Is your OOB-management VLAN/network firewalled off from outgoing access to the internet? If not, this chip can make requests out to another host, who gives it code to run. This code can then either collect data and send it back out, or do other nefarious things.
[+] [-] AnthonyMouse|7 years ago|reply
If true, it's pretty bad. For one thing customers will be more wary of buying hardware from China. If they move manufacturing elsewhere (or, for Amazon, source hardware elsewhere) they would have higher costs and lower margins, if they don't and a competitor does they lose business.
On top of that, people are already wary of the privacy implications of these companies having all your data, but most are willing to trust major US companies. If it looks like they can't keep it secure against random Chinese companies engaged in industrial espionage, more customers will conclude that operating servers in-house is the better choice.
[+] [-] QuinnyPig|7 years ago|reply
I don't buy it as written. There's something else here.
[+] [-] Thriptic|7 years ago|reply
Reverse shell depending on how you have the firewall set up. Most firewalls don't block outbound packets, and even if they do you can tunnel over an open port.
> If it's true, it makes absolutely no sense for Apple and Amazon to attack Bloomberg.
If you are a major company, are you going to use AWS if you believe their hardware is pwned?
[+] [-] bubblethink|7 years ago|reply
[+] [-] wetpaws|7 years ago|reply
It makes a perfect sense to protect from a potential devastating blow to stock price.
[+] [-] jcmeyrignac|7 years ago|reply
And yes, I had the problem with my Lenovo desktop.
[+] [-] tanilama|7 years ago|reply
It could probably make more sense as a backdoor when the hackers get hold the access to the physical device, it is however quite incredible to me that this hack is designed for a remote network access, and could go unnoticed from infosec within the company if it is truly sending packets outside the firewall...
[+] [-] monksy|7 years ago|reply
[+] [-] Hikikomori|7 years ago|reply
[+] [-] propman|7 years ago|reply
The Washpo digging on Roy Moore, the NYT digging on Clinton Foundation, etc. these are experienced reporters having 15-30 credible sources, evidence etc. every time this happens, everyone denies and slowly little by little the story finds more evidence and facts and it becomes true.
This is why a free press is soooo important
Why would Apple disclose that millions of their products could be hacked and you’ve lost all your privacy. Who would trust them? They’d lose billions, regulations would come etc. it’s in the best interest for every party to deny.
[+] [-] lacker|7 years ago|reply
My suspicion is that there are many cases where the Chinese government is actually trying to insert backdoors into things, and that in particular Supermicro really has been compromised by the Chinese government, but the technical details of this chip are incorrect. That explains why so many government officials are eager to leak information to Bloomberg, but at the same time the technical details don't really make sense.
It isn't really in Amazon's and Apple's best interest to lie about this. When Gmail got hacked by the Chinese government, Google was pretty honest about it. China has a lot of resources so you can't really expect companies to fend off 100% of attacks on their own; it makes sense for them to acknowledge this publicly and get help from the US government when needed.
[+] [-] notatoad|7 years ago|reply
[+] [-] paulgb|7 years ago|reply
> I do not think that the securities laws explicitly allow companies to make false statements of material fact if required for national security, but you could see giving them a pass here.
https://www.bloomberg.com/view/articles/2018-10-04/computer-...
[+] [-] Gokenstein|7 years ago|reply
I've been in such a situation myself where I was in the room during what LOOKED like a DDOS by Akamai 10 minutes after we got off the phone with them to turn down their CDN services.
In much the same way as Apple is refuting this claim, after a few weeks of internal debate above our pay grade we decided we didn't see it and it didn't exist and therefore it didn't.
[+] [-] moduspol|7 years ago|reply
For a national security-related issue, you might just include the minimum number of people that need to know, which would naturally not include your PR team. Then when an article like this comes out, the PR team responds in exactly the way they would if it were an outright fabrication, which is what the goal would be.
[+] [-] ProAm|7 years ago|reply
[+] [-] close04|7 years ago|reply
What's stopping someone to sue after buying stock under the assumption that the companies are sound? I assume they have a duty to the shareholders.
[+] [-] heartbreak|7 years ago|reply
[+] [-] wyldfire|7 years ago|reply
[+] [-] burlesona|7 years ago|reply
[+] [-] whydoineedthis|7 years ago|reply
edit: spelling.
[+] [-] berberous|7 years ago|reply
On the other hand, Apple and Amazon have huge reasons not to blatantly lie about this. The plaintiffs bar would be all over both companies for material false and misleading statements if this turned out to be true (see also Elon's recent experiences with the SEC).
Apple and Amazon are certainly incentivized to not admit bad facts, to spin facts, to issue misleading unclear statements that read as a denial but are not, etc. But I really don't think the legal teams at either company would let executives get away with issuing such full-throated and clear denials if they were untrue.
[+] [-] Jackim|7 years ago|reply
[+] [-] sgwealti|7 years ago|reply
[+] [-] edoo|7 years ago|reply
[+] [-] saudioger|7 years ago|reply
This is so typical of Gruber as an evangelist.
There's no way that Apple would remain silent on this even if they KNEW it were true. The only possible move is denial.
Silence is validation or uncertainty, a statement of ambiguity will tank the stock and reputation as experts, recognition of even partial truth could possibly destroy their supply lines overnight.
I honestly think the corporate denials here need to be outright ignored because they have so much to lose. A story of this magnitude is basically like pointing a gun to someone's head and asking them for permission to pull the trigger.
[+] [-] fermienrico|7 years ago|reply
What incentive do you see for Bloomberg to report a major story and lie!? Why would they do that?
[+] [-] wgerard|7 years ago|reply
As a few people pointed out in the other thread, didn't they pretty explicitly deny they were involved with PRISM?
[+] [-] neom|7 years ago|reply
[+] [-] baq|7 years ago|reply
https://www.reddit.com/r/sysadmin/comments/9layb7/from_bloom...
[+] [-] wyldfire|7 years ago|reply
[+] [-] peignoir|7 years ago|reply
[+] [-] _trampeltier|7 years ago|reply
[+] [-] jedberg|7 years ago|reply
While we would cooperate with any government investigation, we are not aware of any investigation regarding this topic nor have we been contacted by any government agency in this regard. We are not aware of any customer dropping Supermicro as a supplier for this type of issue. Every major corporation in today’s security climate is constantly responding to threats and evolving their security posture. As part of that effort we are in regular contact with a variety of vendors, industry partners and government agencies sharing information on threats, best practices and new tools. This is standard practice in the industry today. However, we have not been in contact with any government agency regarding the issues you raised.
Furthermore, Supermicro doesn’t design or manufacture networking chips or the associated firmware and we, as well as other leading server/storage companies, procure them from the same leading networking companies.
[+] [-] excalibur|7 years ago|reply
https://www.bloomberg.com/news/articles/2018-10-04/the-big-h...
[+] [-] notsupermicro|7 years ago|reply
[+] [-] Rafuino|7 years ago|reply
[+] [-] fuddle|7 years ago|reply
[+] [-] space00|7 years ago|reply
https://www.cnbc.com/2018/08/30/google-titan-made-by-chinese...
[+] [-] wyldfire|7 years ago|reply
'original design' is hard to verify without help from SuperMicro.
> Amazon reported the discovery to U.S. authorities, sending a shudder through the intelligence community.
... especially for Amazon. Unless Bloomberg claims here that Amazon got parts of this SKU and compared them to newer parts of the SKU and found differences?
Is it legit to revise your design in terms of changes to passives without rev'ing the part and notifying downstream supply chain. Could the grain-of-rice 'microchip' be a different or new resistor/cap? Could it be logic masquerading as a passive?
> In one case, the malicious chips were thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached, according to one person who saw pictures of the chips.
Like a passive in a blind/buried via?
Anyways, for everyone who claims "it can't be done", "this is implausible" -- you're probably being a little naive. US intelligence agencies do shipment interdiction and adulterate products for this purpose, [1] why couldn't/shouldn't China do the same?
[1] https://www.theguardian.com/books/2014/may/12/glenn-greenwal...
[+] [-] jaclaz|7 years ago|reply
I have no idea how things actually work in companies like Apple or Amazon, but would it be "normal" (at their size/scale and given their surely advanced knowledge in technology) to have inspections on the hardware they use (inspections at hardware level of the kind capable of showing these modifications)?
I mean do they routinely do these checks?
[+] [-] Animats|7 years ago|reply
Boards are shipping with way too much remote access capability. It's not like you need to look at system busses via the network. You're not going to debug a broken board remotely, you're going to turn it off and replace it. Now that this is an identified problem, it's time to put IPMI and its ilk back in its cage.
[+] [-] rajekas|7 years ago|reply
[+] [-] an0n404|7 years ago|reply
When it is turned off? A magical chip that works without electricity is much more valuable than any data which could be exfiltrated from servers. A chip that works without power changes the world more than the iPhone... in the iPhone Age
[+] [-] morpheuskafka|7 years ago|reply
If this story is true, it will be an escalation clearly and willfully by the PRC from mere state-sanctioned economic espionage to an act of war against the US.