WingNews logo WingNews
top | item 30756636

Ask HN: Is Apple down?

253 points| crgt | 4 years ago

https://developer.apple.com doesn't work App Store doesn't work iMessage doesn't work. Not just me - coworkers also struggling.

Any idea what's going on?

106 comments

order

Animats|4 years ago

    nslookup
    > server a.ns.apple.com
    Default server: a.ns.apple.com
    Address: 2620:149:ae0::53#53
    Default server: a.ns.apple.com
    Address: 17.253.200.1#53
    > developer.apple.com
    Server:  a.ns.apple.com
    Address: 2620:149:ae0::53#53

    developer.apple.com canonical name = developer-cdn.apple.com.akadns.net.
    ** server can't find developer-cdn.apple.com.akadns.net: REFUSED
Ah. So Apple's own DNS servers are redirecting developer.apple.com to something on "akadns.net", which is operated by Akamai. But Apple's own DNS servers refuse to resolve that, probably because it's not in the apple.com zone.

More:

    nslookup
    > developer-cdn.apple.com.akadns.net
    Server:  127.0.0.53
    Address: 127.0.0.53#53

    Non-authoritative answer:
    developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
    world-gen.g.aaplimg.com canonical name = apple-c.g.aaplimg.com.
    apple-c.g.aaplimg.com canonical name = apple-cf.g.aaplimg.com.
    apple-cf.g.aaplimg.com canonical name = apple-lr.g.aaplimg.com.
    > server a.ns.apple.com
    Default server: a.ns.apple.com
    Address: 2620:149:ae0::53#53
    Default server: a.ns.apple.com
    Address: 17.253.200.1#53
    > developer-cdn.apple.com.akadns.net
    Server:  a.ns.apple.com
    Address: 2620:149:ae0::53#53

    ** server can't find developer-cdn.apple.com.akadns.net: REFUSED
It's clearly a botched DNS configuration. Not clear what the intent was. Did they really want to point "developer.apple.com", a web site, to "developer-cdn.apple.com.akadns.net", which is a DNS server? Or am I misreading that?

It's generally considered bad form to have all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.

Anyway, this looks like an attempt to outsource something to Akamai that went badly wrong.

lima|4 years ago

> Or am I misreading that.

Yes:

    developer.apple.com. 73 IN CNAME developer-cdn.apple.com.akadns.net.
    developer-cdn.apple.com.akadns.net. 73 IN CNAME world-gen.g.aaplimg.com.
    world-gen.g.aaplimg.com. 13 IN CNAME apple-c.g.aaplimg.com.
    apple-c.g.aaplimg.com. 8 IN CNAME apple-cf.g.aaplimg.com.
    apple-cf.g.aaplimg.com. 8 IN CNAME apple-lr.g.aaplimg.com.
    apple-lr.g.aaplimg.com. 14400 IN NS b.gslb.aaplimg.com.
    apple-lr.g.aaplimg.com. 14400 IN NS a.gslb.aaplimg.com.
The Akamai CNAME just points to a series of aaplimg.com CNAME (eventually ending up with apple-lr.g.aaplimg.com), which is Apple's own CDN domain. The CDN's resolvers (a.gslb.aaplimg.com and b.gslb.aaplimg.com) refused to serve A records for apple-lr.g.aaplimg.com.

They fixed that and now it's back up.

This kind of setup is typically done for flexibility reasons (geographical DNS load balancing or similar, where the Akamai DNS servers serve as the geo LB).

> It's generally considered bad form to have the all the DNS servers for "example.com" under "example.com", by the way. If you mess up "example.com", or it goes down, getting to it to fix it can be difficult.

Not necessarily - this is what glue records[1] are for. Many large companies host their authoritative DNS on the same domain, it's not a bad practice when done carefully.

[1]: https://ns1.com/blog/glue-records-and-dedicated-dns

silisili|4 years ago

> Did they really want to point "developer.apple.com", a web site, to "developer-cdn.apple.com.akadns.net", which is a DNS server.

It's just a CNAME, meaning go look that up. It does not indicate that developer-cdn.apple.com.akadns.net is a DNS server.

The above seems to indicate that somewhere in the chain of resolving developer-cdn.apple.com.akadns.net, a DNS server refused the query. A dig +trace should indicate which.

frays|4 years ago

Works with other DNS servers.

  $ nslookup developer-cdn.apple.com.akadns.net a.ns.apple.com
  Server:  a.ns.apple.com
  Address: 17.253.200.1#53

  ** server can't find developer-cdn.apple.com.akadns.net: REFUSED

  $ nslookup developer-cdn.apple.com.akadns.net 1.1.1.1
  Server:  1.1.1.1
  Address: 1.1.1.1#53

  Non-authoritative answer:
  developer-cdn.apple.com.akadns.net canonical name = world-gen.g.aaplimg.com.
  Name: world-gen.g.aaplimg.com
  Address: 17.253.121.201
  Name: world-gen.g.aaplimg.com
  Address: 17.253.121.202

jonfw|4 years ago

This looks like an Akamai DNS load balancing solution. It will route a user to an endpoint based on a bunch of statistics (think location, availability, latency, and/or load), and will often handle caching and DDOS protection as well

mnd999|4 years ago

Can we refer to this as “Doing a Facebook?”

tshaddox|4 years ago

Yep.

Wife: My Apple Maps isn't working.

Me: Hmm, it's not working for me either. They must be having server problems. You should use Google Maps for now.

Wife: I can't download Google Maps either, the App Store doesn't seem to be working.

moepstar|4 years ago

Maps, App Store, iMessage on macOS works

They work on iOS as well - so it seems to be a regional thing?

(Location: Germany)

donarb|4 years ago

Both Apple Maps and Google Maps work in the browser, no need for an app.

divbzero|4 years ago

Same with my Apple Maps over the course of an hour this morning.

Rough order of events:

1. Not working (could not find server)

2. Not working (request timeout)

3. Restart app

4. Working

Perhaps DNS was broken for awhile and restarting the app cleared the DNS cache and forced a fresh IP lookup?

asvitkine|4 years ago

Can you use the web version of Google maps?

tormock|4 years ago

[deleted]

donohoe|4 years ago

You wouldn't think it if you went by this:

https://www.apple.com/support/systemstatus/

adwi|4 years ago

I’m sure it wasn’t when you posted 10 minutes prior, but FWIW currently listing 11 outages:

> App Store - Outage Today, 12:32 PM - ongoing Some users are affected Users may be experiencing intermittent issues with this service.

Apple Arcade - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.

Apple Music - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.

Apple TV+ - Outage Today, 12:32 PM - ongoing Some users are affected Users may be experiencing a problem with Apple TV+. We are investigating this issue.

iTunes Store - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.

Podcasts - Outage Today, 12:32 PM - ongoing Some users are affected Users are experiencing a problem with this service. We are investigating and will update the status as more information becomes available.

Radio - Outage Today, 12:32 PM - ongoing Some users are affected This service may be slow or unavailable.

Apple Business Manager - Outage Today, 1:14 PM - ongoing Some users are affected Users may be unable to sign in.

Apple School Manager - Outage Today, 1:14 PM - ongoing Some users are affected Users may be unable to sign in.

Device Enrollment Program - Outage Today, 1:14 PM - ongoing Some users are affected Users are experiencing a problem with this service. We are investigating this issue.

Schoolwork - Outage Today, 1:14 PM - ongoing Some users are affected This service may be slow or unavailable.

oxplot|4 years ago

A lot of system status pages are updated by humans who will verify issues before reporting them. Main reason is to avoid overly surface every minor and transitory issue to public view.

nneonneo|4 years ago

Looks like it’s been updated. Currently showing 11 services down, some of which have been down for over an hour.

ransom1538|4 years ago

That is just a static github page with html. These are just green dots on a screen.

samwillis|4 years ago

I chose the perfect time to restore a repaired iPhone, don’t seem to be able to fully login to iCloud, it’s hanging on the login screen…

Edit: It’s also refusing to download any apps, doesn’t even show the progress circle. Just a download icon next to the app name on the Home Screen and errors out when you click it.

Edit: Login and app downloads now working as of 6.00GMT

spansoa|4 years ago

It's times like this that force us to remind ourselves how reliant we are on critical services like these. On one hand, we can celebrate (Internet snow-day!) but on the other we are forced to shop around for alternatives too.

I often wondered how medieval the world would become if there was a huge sun flare ejection that breached the magnetic field and destroyed a bunch of data-centers. Think of the mess we'd be in!

jcims|4 years ago

I’m sure it’s happens more than I’m aware but i have to say that i can’t recall an App Store outage since i got back in the platform 3-4 years ago. Not bad!

rvieira|4 years ago

I picked a terrific time to lose my temper and do a `rm -rf /Library/Developer/CommandLineTools ; xcode-select --install` /facepalm

lima|4 years ago

Looks like their DNS servers are responsive, but refuse to serve records:

    $ dig -t NS developer.apple.com
    [...]
    apple-lr.g.aaplimg.com.     14400   IN      NS      b.gslb.aaplimg.com.
    apple-lr.g.aaplimg.com.     14400   IN      NS      a.gslb.aaplimg.com.

    $ dig @a.gslb.aaplimg.com developer.apple.com
    [...]
    ;; ->>HEADER<<- opcode: QUERY, status: REFUSED
    ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
    ;; WARNING: recursion requested but not available
    ;; WARNING: recursion requested but not available
Most likely a configuration mistake that'll be undone as soon as they figured out how to re-deploy their DNS servers while DNS is down.

Unlikely to be BGP shenanigans as some people on Twitter claim. My network has direct peerings to Apple's AS714.

chewmieser|4 years ago

Definitely. Downdetector shows a bunch of reports too (e.g. https://downdetector.com/status/apple-music/). I noticed issues with Music and News, seems like a ton of their services are down

callalex|4 years ago

Downdetector has predicted about 50 of the last 3 outages, and linking to them here just makes the self-fueling cycle even worse.

synaesthesisx|4 years ago

Yes. Even developer.apple.com won't load at all for me. Who wants to take bets on DNS as the culprit?

chuinard|4 years ago

My app update was rejected because my Upgrade screen was unable to fetch prices from their servers and instead showed an infinite spinner.

thih9|4 years ago

Would an infinite spinner also show up if the server was up but the connection was problematic? If yes, this would be about not handling network errors, which sounds like a decent rejection reason to me.

donatj|4 years ago

My Apple Music stopped working mid song and is being weird now. Everything seems to be working fine for my wife. Weirdly spotty.

oxplot|4 years ago

iCloud Private Relay is shown as affected as well. This is an interesting case when it comes to failure behavior. From security perspective, you want your connection to stop working instead of falling back to insecure. Is this the case? Can anyone confirm?

mathieuh|4 years ago

It fell back to insecure for me, for about 30 seconds (maybe longer before I noticed) I couldn’t connect to the Internet from my iPhone, then I got a notification saying private relay was unavailable and I was able to connect again.

A few minutes later it gave me another notification saying private relay was working again.

nyuszika7h|4 years ago

iCloud Private Relay is not designed to be a full-fledged VPN anyway. HTTPS traffic in apps (other than browsers) bypasses it AFAIK.

ChrisMarshallNY|4 years ago

They seem to have been having a bit of a lie-down, today. I can't submit TestFlight builds, but now, it is taking longer, before the server throws a nutty, so I guess the fix is on its way.

SlimyHog|4 years ago

Yeah, I'm seeing anecdotal reports of a bunch of services out

rateofclimb|4 years ago

App Store Connect was down for me but appears to be up again now.

selimthegrim|4 years ago

I haven’t been able to cancel subscriptions lately. I filed a refund request and complaint to Apple, maybe it didn’t get through because of this?

mariojv|4 years ago

It's a partial outage for me. I was just able to send an iMessage, but directions on Maps are not working. I live in central Texas.

leviathan|4 years ago

I've been struggling with a DNS downtime at Mediatemple all day. Is there a possible more global DNS issue?

SalimoS|4 years ago

Yes, got a notification that Apple private relay is unavailable

And another notification that it’s back online 40min later

antupis|4 years ago

Yeah had to close private reley because websites didn’t load.

teeray|4 years ago

I had abnormal trouble pulling video I uploaded to iCloud yesterday. Something is up.

wanderer_|4 years ago

I noticed a blip in iMessage earlier, but it sorted itself out before too long.

brown9-2|4 years ago

the domain name developer.apple.com resolves through a series of CNAMEs to Apple's CDN (applimg.com), which if it was down would explain other things like iMessage also being unavailable.

1023bytes|4 years ago

Yeah, for me the CNAME chain ends with apple-lr.g.aaplimg.com, which doesn't resolve to anything

variant|4 years ago

Some reports that there were DNSSEC validation issues w/ proxy.safebrowsing.apple which CNAMEs to aaplimg.com.

sys_64738|4 years ago

AAPL is down too, today.

hit8run|4 years ago

For me in Germany: iMessage up App Store up Developer site down

ComputerGuru|4 years ago

It’s coinciding with an AWS outage. Probably not unrelated.

gunzor|4 years ago

Can't upload an ipa to App Store Connect for an hour.

jhgb|4 years ago

> Any idea what's going on?

Must be gravity. (Sorry, I had to.)

alopes|4 years ago

Had a few issues with the App Store with OS 12.3

traceroute66|4 years ago

Must be regionalised. Nothing wrong here.

tiahura|4 years ago

Down in French Polynesia.

spike021|4 years ago

iMessage texts are working fine for me but an image I sent to a friend is stuck. Music is also down for me.

novateg|4 years ago

Down for me via CloudFlare WARP

fargle|4 years ago

It's always DNS.

ragnot|4 years ago

Down for me.

windex|4 years ago

fedora is down.