Apple iCloud Experiencing Issues

[Meekro]

For those out of the loop, this week has seen an outage from a bunch of major US internet companies: Cloudflare, Slack, Google Cloud, Azure, Facebook (including Instagram and WhatsApp), and now iCloud.

[mcqueenjordan]

Interns join at the beginning of summer. \o/

[walrus01]

There's been a few major office365 outages that coincided with the Azure ones, we've taken to calling it office364 internally.

[tootie]

Have any of them issues post mortems yet? Anything at all to indicate an attack of some sort?

[flylib]

this is somewhat common

[OrgNet]

They needed to connect NSA's new servers... /s

[saltedshiv]

Attackers and state actors sometimes appear to behave indiscriminately

[basch]

why is slack on that list, they use AWS. They arent an infrastructure provider.

[techrich]

Its probably some back end access for five eyes, and needs a re-start of everything. Its too fishy for it just to a coincidence.

[aristophenes]

AMD has been shipping their new Epyc Rome server CPUs to hyperscalars for a few weeks now [1]. It seems to be a very appealing chip that a lot of them are adopting. They won't be the same as whatever they were using before, so it could be that major updates are needed to include them, meaning that it's just a risky time right now?

[1] Second question in https://www.anandtech.com/show/14568/an-interview-with-amds-...

[pilif]

After reading the ZIP bomb article also linked on HN, I have copied the largest file to my iCloud drive for fun.

And an hour later, I'm reading that most of iCloud is down. I really hope this is a funny coincidence. And if not, then I'm terribly sorry.

[therein]

As far as my memory goes, iCloud features at least 99 partitions. The 99th of which is the "staging" partition where the iCloud employees' data lives.

All partitions are more or less separated and shouldn't affect one another. For that reason, it probably wasn't you that broke the whole system.

Even the Cassandra they use for metadata storage is partitioned and separated.

[postmortimus]

Nope, just a typical network hardware failure in a west coast datacenter, which triggered a cascading BGP failure disrupting much of the network connectivity in that DC.

[jokoon]

Can you try again to make sure?

[mandeepj]

Can you try again a few times? Let's see if it's reproducible. I think you are not at fault :-)

[mr_sturd]

Think they tried to scan it for malware?

[OrgNet]

upload a different sized one next time, with a new filename

[your_bully]

Don't give yourself too much credit

[envolt]

I'm going to read all the conspiracy theories on this post.

https://news.ycombinator.com/item?id=20345060 (Facebook, Instagram, and WhatsApp outages)

[3JPLW]

You know the old adage that celebrities die in threes? It's actually mathematically supported... or, well, it's supported that they die in 2.718s. Same principle would apply to cloud service outages if all the services and their failures were actually independent. We'd expect them to happen in "clusters" of e:

http://ssp.impulsetrain.com/celebrities.html

I still love me a good conspiracy theory, but clustering of random (poisson) events is much more likely than you'd expect.

[the-dude]

Actually there are not many actual conspiracies listed there.

I will start :

- All the companies in the list are getting rid of their Huawei dependencies, replacing routers by domestic ones.

- China is flexing its muscles in the trade war : look, we are creating some instability in your e-commerce. Wait until we attack your infra ( electricity, water )

[cantbecool]

Maybe some submarines are cutting some deep sea cables? In actuality it's probably massive backbone upgrades that are classified, the NSA has to obtain real-time data from somewhere.

[ksec]

Once is happenstance. Twice is coincidence. The third time it’s enemy action.

I don't even know if fourth of Fifth is confirmation.

[the-dude]

Dude, I wrote that top comment. And I was like iCloud WTF?

[johnnycab]

Just wondering out aloud rather than floating any theories, if it is in any way related to the 6.4 magnitude earthquake and/or geological events leading upto it. Also inter/un-related, Google provides Apple with the infrastructure for iCloud; they also experienced some downtime in the last few days, along with others i.e. tremors causing some sensitive/critical servers to misbehave.

However, I doubt that it is down to these factors, as there will likely be significant amount of distributed fault tolerance, failover and contingency plans in place.

https://earthquake.usgs.gov/earthquakes/eventpage/ci38443183...

[Raphmedia]

Take with a grain of salt. This is pseudoscience at best.

There were some reports of earthquake lights[1][2][3] and earthquake clouds[4][5][6] recently.

[1] https://en.wikipedia.org/wiki/Earthquake_light

[2] https://www.mirror.co.uk/news/weird-news/men-film-glowing-sn...

[3] https://www.dailystar.co.uk/news/weird-news/789760/Colorado-...

[4] https://en.wikipedia.org/wiki/Earthquake_cloud

[5] https://weather.com/vertical/video/strange-clouds-seen-in-co...

[6] https://www.denverpost.com/2019/06/20/boulder-rare-helix-clo...

[luckydata]

It's definitely the end times. Locusts coming anytime now.

[jondwillis]

The outages occurred long before the earthquake but my mind originally wandered to the same place.

What earthquake related causes could have had effects 24h or more ahead of time?

[cameronbrown]

Apple runs their own bare-metal services - only some of which are hosted on Google Cloud. You could see which ones went down when Google did the other week.

[pcora]

So, when is Amazon / AWS outage day? Tomorrow?

[Buge]

How big of an outage do you want? Here's one from 7 days ago: https://news.ycombinator.com/item?id=20298719

[mehrshad]

If there's any credibility behind the notion of these being deliberate in any way, I'd wager July 15/16.

[samcday]

I'd like to think the simplest explanation to this recent spate of outages is that all the engineers at each org spent too long reading the HN comment thread on the previous company's outage, and didn't notice their own servers catching fire ;)

[ryanmarsh]

Imagine all the ops folks gloating earlier in the week (cough CloudFlare) having to eat crow mere days later.

[SpaceManNabs]

Imagine being on call this week. My best wishes to anyone doing DevOps or Ops.

[ohnope]

Any internet engineers privy to what's going on the last few days, at a global infrastructure level?

[kemals]

Here is how ThousandEyes viewed the impact: https://twitter.com/thousandeyes/status/1146862826566250499

[DavideNL]

Little off-topic, but lately i often find myself Airdropping files between my iPad & Macbook because iCloud isn't syncing newly added files for some reason (which can't be forced/refreshed on the iPad as far as i know.)

[snazz]

Are there any graphs anywhere showing Internet traffic at different ISPs and networks? Such a graph (especially over a world map) would make it obvious if there was any DDoS, SSH bruteforcing, or other monkey business going on.

[ceejayoz]

I wish there was a historical view. Very curious if this is common, or if it's unusual (which would make it especially unusual with the Slack, Facebook, etc. outages recently).

[jaredcwhite]

I'm actually relieved to hear there were a bunch of known problems…earlier today I was wondering why none of my notes and photos were syncing and thought iCloud was loosing its marbles. It used to happen a lot to me, but in the last year or so it's been pretty stable. I was afraid sync reliability was regressing!

I suspect Apple's making upgrades as part of the push towards iOS 13 and macOS Catalina, and ran into some rollout glitches.

[thekyle]

One of the services listed is Screen Time. Maybe I'm mistaken but isn't that the feature on iPhones where you can see how much time you spend in each app? Why would that require Apple's servers.

[3xblah]

https://en.m.wikipedia.org/wiki/Flag_day_(computing)

[kburman]

In India, I'm experiencing issues with online payment from last few days. Majorly with UPI payments. Not sure if these are related but hard to ignore it.

[fc_barnes]

I was noticing some issues with mega sync earlier, maybe these orgs are sharing infrastructure at some level... I guess I won't rage quit just yet

[jwr]

And I've been wondering why my kid could play games so long today, in spite of Screen Time limits :-) I guess that explains it.

[mlosapio]

iOS Walkie Talkie has been down since Wednesday

[rahuldottech]

Holy crap this has been a turbulent couple weeks for internet companies and services, eh?