The full story of the RSA hack can finally be told

[NelsonMinar]

This story was big news when it happened and I'm grateful to Andy Greenberg for writing this retrospective. Pretty much nothing has changed about security at companies in the last ten years that would foil this kind of attack. I mean maybe folks are a little smarter about catching spearfishing Office docs, and Flash exploits are now a thing of the past, but those are replaced by contemporary equivalents. And I'm sure conveniences like the not-quite-airgapped crucial equipment still persist. We truly have no idea how to secure systems on the Internet.

As for RSA, it came out a couple years later their products were compromised various ways by the NSA. Then a couple years later the NSA lost control of its own hacking tools with the infamous Shadow Brokers release. Not only is building secure systems hard, but the US government actively works to undermine its own companies' security.

https://www.reuters.com/article/idUSBRE9BJ1C220131220?irpc=9... https://arstechnica.com/information-technology/2014/01/how-t...

[chinathrow]

> RSA executives told me that the part of their network responsible for manufacturing the SecurID hardware tokens was protected by an “air gap”—a total disconnection of computers from any machine that touches the internet. But in fact, Leetham says, one server on RSA’s internet-connected network was linked, through a firewall that allowed no other connections, to the seed warehouse on the manufacturing side.

That's not really an air gap, isn't it?

[PeterWhittaker]

As others have noted, no, it is not. But that doesn't boggle my mind...

What boggles my mind is that the seed machine and the intervening network and the firewall did not appear to have "scream loudly then shutdown when this threshold is exceeded" mitigations in place.

They were wise enough to have a single connection from the seed host to the seed requester. They were wise enough to limit the requester to one request every 15 minutes.

They only discovered that threshold was being exceeded when they logged in to that machine.

The firewall itself should have had detection and response capabilities to notice when calls were being made faster than that, and it should have had a third, dedicated warning connection to alert humans to the fact. The seed host should have had detection and response capabilities.

And, given the value of the asset, it would have been entirely reasonable to have a transparent bit of network gear doing the same, like a custom switch invisible to the request host.

Since the article didn't mention any of these things, and since it said that the high request rate was detected only by humans on the box, I'm going to assume they didn't have these, for reasons mysterious.

EDIT: Come to think of it, since that machine was being used to burn CDs, there should also have been strict limits with appropriate detection mitigations on what that machine could do outbound.

[dredmorbius]

That configuration is more typically referred to as a bastion server (or bastion host, per Wikipedia).

Access between network segments, or to a protected host, is through a single specifically-hardened host. Through network traffic (natting or bridging) is typically disabled or at least not provided by default, though in practice, it's challenging to entirely prevent tunelling.

But no, it is not an air-gapped system. Likely a journalistic compromise as "bastion host" is a less familiar term to the public.

https://en.wikipedia.org/wiki/Bastion_host

[jaywalk]

"A firewall that only allows connections from one Internet-connected server" is quite literally not an air gap. It's one of those things that you look back on and wonder how in the world that decision was made.

[jarym]

They must have meant the executives had an air gap between their ears.

[amackera]

Seems more like an attempted air gap :facepalm:

[tyingq]

"Moments later, his computer’s command line came back with a response: “File not found.” He examined the Rackspace server’s contents again. It was empty. Leetham’s heart fell through the floor: The hackers had pulled the seed database off the server seconds before he was able to delete it."

I get the compulsion to delete it, but deleting it wouldn't have provided any real comfort. You would have no idea if that was the only copy. So, delete it just in case it is, but it doesn't change what you would have to do afterwards...the master keys have to be assumed leaked.

[PUSH_AX]

I mean you're never not going to delete it when given the option. As tiny as the chances are, it could mean your downstream consumers end up safe. Although yes, it should not stop you going into full damage control mode.

[ddlatham]

> In the hours that followed, RSA’s executives debated how to go public. One person in legal suggested they didn’t actually need to tell their customers, Sam Curry remembers. Coviello slammed a fist on the table: They would not only admit to the breach, he insisted, but get on the phone with every single customer to discuss how those companies could protect themselves. Joe Tucci, the CEO of parent company EMC, quickly suggested they bite the bullet and replace all 40 million-plus SecurID tokens. But RSA didn’t have nearly that many tokens available—in fact, the breach would force it to shut down manufacturing. For weeks after the hack, the company would only be able to restart production in a diminished capacity.

> As the recovery effort got under way, one executive suggested they call it Project Phoenix. Coviello immediately nixed the name. “Bullshit,” he remembers saying. “We're not rising from the ashes. We're going to call this project Apollo 13. We're going to land the ship without injury.”

This is the sort of response that would increase my trust to choose this company in the future. This is not easy. Our human instincts naturally kick in to minimize our faults and protect ourselves. Choosing to put the customer first at risk of your own reputation is hard, but the right choice, even for your reputation in the end.

[YetAnotherNick]

I don't know the contract with clients or anything but "didn’t actually need to tell their customers" seems totally illegal to me if the RSA is the provider of one of the most fundamental layer of security to large companies. It is a PR piece. Not saying that RSA likely did the best they could do to be secured from the hack, but this story is a pr piece.

[m463]

There's plenty of examples supporting that. The earliest I remember was the Tylenol murders where 31 million bottles of tylenol were taken off the shelves:

https://en.wikipedia.org/wiki/Tylenol_(brand)#1982_Chicago_T...

In the end, their brand got stronger and more trustworthy.

(and I believe we got sealed bottles)

[eganist]

If anyone else is wondering as to why there's so much human drama written in what should otherwise probably have been a normal retrospective, it's (at least by my judgment) to increase the likelihood that the article gets optioned for a film or tv series, etc.

This isn't a first for Andy Greenberg, either. https://www.imdb.com/name/nm5200697/

(My comment isn't critical of his writing; it's merely an effort at explaining it.)

[munificent]

I mean, I also found the story much more engaging because it involved actual humans with human feelings and reactions.

[zinglersen]

It was a bit long-winded but I was thinking that it would make a great movie already in the first half.

Now I wonder if there are any cool SecOps movies or tv shows out there(?)

[btbuilder]

I would think it would be better to have a provisioning design that did not require that the company retain the seed data for every fob they had sold.

[briffle]

Or at least auto-delete them after 30 days, in case a customer didn't get theirs, and needed it resent. Retention policies limit the blast radius when there is a problem.

[ianhawes]

And a wise business decision since you would also benefit from having to replace the client's fobs in the event that they lost their seeds.

[benlivengood]

Unlike U2F and similar specs there was no direct communication between the SecurID tokens and any other device, limiting the bandwidth to less entropy than necessary to validate public key signatures. That necessitated having a shared secret between the token and auth server.

[wolverine876]

I would lose trust if I found out that they retained copies of my private cryptographic data. Isn't that shocking in a company as sophisticated as RSA?

[neilv]

> Now, staring at the network logs on his screen, it looked to Leetham like these keys to RSA’s global kingdom had already been stolen.

That must've been a sickening feeling.

[ianhawes]

Good on the author for noting that while common now, the practice of dumping passwords in memory (a la Mimikatz) was not common until some time after this attack.

[Jaymoon85]

https://outline.com/uE6PA4

[tptacek]

Since when were NDAs routinely limited to 10 years?

[eganist]

> Since when were NDAs routinely limited to 10 years?

I'm not seeing an indication that it was routine, just that all the people involved happened to have 10 year NDAs in place. Might've been RSA-specific, potentially as a consequence of the breach or just an artifact of RSA's own policies; it's not actually mentioned. I'm also only familiar with 5 year NDAs.

[ianhawes]

My thoughts exactly. I'm curious if these people thought their NDAs expired after 10 years.

[drdavid]

Some searching tells me 1 to 5 years is considered normal.

[nyokodo]

> Multiple executives insisted that they did find hidden listening devices—though some were so old that their batteries were dead. It was never clear if those bugs had any relation to the breach.

Well that’s not exactly comforting. Who else might have had the keys to the kingdom?

[neonate]

https://archive.is/tqdS9

[BombNullIsland]

This story would be many times more interesting with more technical details and less "human drama".

[eganist]

The story was almost certainly written with media optioning in mind, hence the human drama. It's common with longform journalism, and it's common for the author as well. https://www.imdb.com/name/nm5200697/

[asdfthrowawayyy]

Wired is completely full of shit, just like EFF. FBI fronts, nothing but lightning rod ops.

RSA SecureID has been compromised for decades, way before this puff piece of sensationalist sell newspapers bit.

[sgt101]

Are the seeds large primes?

[timdierks]

No, it's a symmetric system, they're just random values.

[znpy]

non-paywalled link?

[gvb]

Disable javascript

[neonate]

https://archive.is/tqdS9

[mywacaday]

Open link in duckduckgo

[haecceity]

Should have just turned off their computers

[criddell]

I don't have an ad-blocker on this machine and I couldn't finish reading the page. The ads are ridiculously obnoxious.

It wasn't that long ago when magazines like Wired cared a great deal about the page. What a mess.

[dang]

"Please don't complain about website formatting, back-button breakage, and similar annoyances. They're too common to be interesting. Exception: when the author is present. Then friendly feedback might be helpful."

https://news.ycombinator.com/newsguidelines.html

(This guideline isn't there because such complaints are wrong or inaccurate; just the opposite.)

[bigbillheck]

> It wasn't that long ago when magazines like Wired cared a great deal about the page

Do you remember the super-early Wired print editions, because those colors made it barely readable.

[pwg]

Install uBlock origin and block the javascript running on the page.

You still get to read the article, but no "ridiculously obnoxious" ads appear anywhere.

[i_like_kotlin]

They still do, if you pay.

[wussboy]

Man do I ever hate the word "stunning" in headlines.

[dang]

I've clubbed it out of the title above, and added it to HN's debaiting software, so in the future it will get automatically dropped. (Most of the time.)

[asdfthrowawayyy]

fd