top | item 28825484

(no title)

implying | 4 years ago

This is a bizarre analysis. Public legal risk is absolutely the last imaginable threat to us-east-1, short of aliens abducting it. The U.S. security apparatus depends on AWS and would never allow it, Wall Street would never allow it, never mind the fact that Amazon itself would leverage every tool at it's disposal to protect its reputation for reliability. The politicians involved in this scenario might seek to remove Amazon's competitive advantages, or fine them, but the people who understand what AWS even is would never consider a move to shut down a datacenter.

Both the "enemy action" and "operation failure" scenarios are much bigger risks than this article makes out to be. Every non-aligned nation-state offensive cyber team has a knockout of us-east-1 at the top of their desired capabilities. I'm sure efforts range from recruiting Amazon employees to preparing physical sabotage to hoarding 0days in the infrastructure. There's no reason to think one of them wouldnt rock the boat if geopolitics dictated.

Operational failure is probably the most likely. AWS might have a decade of experience building resilience, but some events happen on longer timescales. A bug that silently corrupts data before checksums and duplication and doesn't get noticed until almost every customer is borked, a vendor gives bad ECC ram that fails after 6 months in the field and is already deployed to 10,000 servers, etc. Networking is hard and an extended outage on the order of a week isn't completely impossible. How many customer systems can survive a week of downtime? How many customer businesses can?

discuss

order

dijit|4 years ago

> Amazon itself would leverage every tool at it's disposal to protect its reputation for reliability.

This is a joke, right? The _real_ degradation map of us-east-1 of the last 5 years looks significantly worse than my non-UPS backed Home PC in Sweden.

Personally I'm not looking at us-east-1 as reliable at all; they even suffered a "harddrive crash" https://www.bleepingcomputer.com/news/technology/amazon-aws-...

michaelt|4 years ago

I assume the author isn't asking "What if there was a 30 minute outage of us-east-1" (just wait for it to come back) or "What if there was an outage of a single AZ in us-east-1" (just spin things up in a different AZ)

Rather, they're asking "What if there was a 30 day outage of us-east-1" - so anyone who isn't multi-region or multi-cloud loses everything, including backups, AMIs, and control plane access.

(FWIW I agree with people disagreeing with the worry levels in the article - a solar storm last seen in 1859 is more likely than a software bug? Ha!)

praptak|4 years ago

Yeah, the relative probability of scenarios may be wrong. Maybe even some of them are totally bogus. This doesn't undermine the main point of the article though.

I think the main point is that there's quite a lot of eggs in that basket and we should see this as a problem. Any single organization can think they have contingency plans for a big cloud region going permanently down. The problem is that when all of them try to execute their plans at once it won't work.

deepstack|4 years ago

While your analysis is sound, I wouldn't disregard a Carrington type of event. It can happen again and we never know where. None of our current electronic infrastructures are hardened to handle this kind solar storm/EMP. IPFS is a good direction in mitigate these kinds of centralised data risk.

Diversifying one's cloud/server provider is a good thing! Or simply don't rely only on the cloud. Storage devices are cheap now days, just have local backup and/or in different geographical location.

fnord123|4 years ago

It needs to be put in context. If a Carrington type event takes place then much of the tech we take for granted will be offline. If you do GPS navigation it doesn't matter if route planning is offline because the satellites themselves might be broken. It doesn't matter if you sell makeup because the delivery drivers, planes, and boats will be unable to navigate adequately to delivery anything. If doesn't matter if you stream videogames because many ISPs may well be offline.

tuyiown|4 years ago

Off-topic (and I'm totally not expert, please please correct me if i'm woring) but the the real risk of a Carrington type of event is for the power grid, especially the destruction of equipment at end of power lines.

The idea is that those large transformer cannot be mass produced and could be completely destroyed.

The cool hack is that physical disconnection in time avoids those damages.

At small scale the difference of potential is not enough to fear much physical damage.

VHRanger|4 years ago

Uh IPFS doesn't have replication? It's garbage in terms of reliability as far as I see it

Even 20 year old BitTorrent is a better option if that's the risk you're considering

MattGaiser|4 years ago

> Bear in mind that Republicans hate Amazon because of Bezos’s Washington Post and because the whole tech industry is (somewhat correctly) perceived as progressive

Do Republicans actually hate Amazon all that much or do they just go on Fox News or Twitter and proclaim that they do? As much as they might complain about certain corporations, they don't seem to be at the top of the hit list.

listless|4 years ago

I don’t think they hate amazon, but they certainly don’t appreciate WaPo or NYT or fill-in-the-blank left leaning media (read most). How that translates to shutting down an Amazon data center just kind of shows how deranged people get when it comes to politics.

javajosh|4 years ago

Trump seems to at once despise Bezos for bailing out the WP, but also admires and fear him, since Bezos' money makes Trump look poor in comparison, and the working conditions in Amazon warehouses and anti-union activity reflects the kind of autocratic values that Trump admires. It's pretty clear that Trump would love to be Jeff's friend, which gives Amazon even more leverage to survive any right-wing populist attack.

paul_f|4 years ago

This section destroyed the authors credibility altogether in my mind. Seems the entire article was a set up to make a political attack.

condiment|4 years ago

The best case scenario for an operational failure is the loss of a single AZ, but many of the scenarios you described are things that could impact an entire cloud vendor simultaneously. As others have pointed out (in discussing the Carrington event and the recent Facebook outage), it's not a matter of if it will happen, but a matter of when. And then it's just a question of duration and scope of the impact.

At this point around half of the world's leasable compute is concentrated in fewer than 100 facilities, the locations of which can easily be found with a google search. Using public satellite imagery you can identify network connection points as well as follow power transmission lines. In a wartime scenario, these are industrial targets with astounding strategic value, a tiny geographic footprint, and limited collateral damage in terms of human life. The Nagasaki and Hiroshima of the future could simply be kinetic attacks against a couple of datacenters. I'm alarmed that nobody is prepared for this and the industry zeitgeist seems to be to continue the consolidation of our economies into the cloud.

godtoldmetodoit|4 years ago

I agree datacenters would clearly be targets, but our IT infrastructure is hardly alone in being a relatively concentrated strategic target.

If kinetics are in play, said actor could also destroy our oil refining and pipeline systems. Taking out a few dozen large baseload power generation facilities would have a massive impact on the grid.

chihuahua|4 years ago

Regarding "enemy action" - I wonder what percentage of AWS/GCP/Azure employees are also employed by foreign security services? Is it 0%, 1%, 5% ?

snotrockets|4 years ago

You have to assume it's >0%, for both foreign and domestic, and I'm certain the cloud providers assume that too.

lenkite|4 years ago

I would love to see Wall Street prevent a Coronal Mass Ejection.