top | item 4785835

iOS 6.0 Causes CDN Overages

157 points| woodhull | 13 years ago |labs.prx.org | reply

92 comments

order
[+] robomartin|13 years ago|reply
I have never considered Apple to be a solid software company. They are a marketing company in the tech sector. If they made washing machines they'd be sexy as hell. In my years of exposure to Apple tech I have always been in awe of their industrial design, mechanical design and manufacturing prowess. Their software, however, often leaves much to be desired. I say this both as a developer and a user. From OSX to iOS there are amazing patches of software manure in plain sight. And it isn't getting any better.
[+] zizee|13 years ago|reply
addlepate wrote 37 minutes ago | link [dead]

OK. I bet you use a WebKit browser.

addlepate: you were hell-banned 100 days ago (unfairly in my mind).

[+] huhtenberg|13 years ago|reply
You don't say.

All that effort that went into humanizing BSD is absolutely laughable.

[+] pretoriusB|13 years ago|reply
>I have never considered Apple to be a solid software company.

What does that even mean? That they make bad software? For one, they make the most popular certified UNIX OS in the world, and one of the two most popular mobile OSs. Their professional offerings are great too: Logic Pro, Aperture, Final Cut Pro, etc. In fact, FCPX aside, they are professional staples, with few competitors.

And huge number of desktop and the majority of mobile users use a browser they developed to the best game in town starting from humble open source origins (khtml -> webkit).

And that Clang thing, that FreeBSD recently adopted? Their work too, along with other LLVM infrastructure.

>Their software, however, often leaves much to be desired. I say this both as a developer and a user.

Care to mention any substantially better mobile OS than iOS?

(One might argue that Android is better. But substantially better, no way in hell).

[+] untog|13 years ago|reply
This is where Apple's closed nature is very annoying. Assuming the article is correct, it was bug introduced in 6.0, and fixed in 6.0.1. So Apple knew about it, investigated it and published a fix.

Yet they didn't tell anyone about it. Or the crippling effect this bug could have on their data allowances. Why not? Are they really that terrified of admitting to making a mistake?

[+] rachelbythebay|13 years ago|reply
This same behavior has been happening to me since before iOS 6 came out. I can show the same "206 storm" happening as far back as August of last year with iOS 4.3.5.
[+] michael_miller|13 years ago|reply
To play devil's advocate, why would it matter? Is there any fix app developers could implement, short of blocking requests from iOS 6 users?
[+] Ensorceled|13 years ago|reply
This is the reason Forstall has left the building, iOS 6 is a disaster:

    * Maps was not ready for prime time (so bad it required a public apology)
    * Podcast was not ready for prime time.
    * Passbook was not ready for prime time.
    * App store is a step backwards.
    * Battery and WIFI issues.
    * Soooo many bugs.
There are usually problems but this is the first time I've seen so much utter crap shipped including tons of minor bugs and issues (well Mobile Me was pretty bad :-)

I'm using Audible, Downcast and the built in Music app where I used to just use the Music app. iOS 6 was a huge step backwards for me.

Rather than waiting until it's ready (think copy/paste on iOS) Forstall was responsible for shipping a mess. Hopefully this will fix itself with new leadership.

[+] gurkendoktor|13 years ago|reply
How do you know that Forstall was the person advocating a yearly OS upgrade cycle? It seems just as likely to me that this rigid schedule was forced onto him and he couldn't deliver. Things like OS X Lion (a huge change under the hood) and the Maps places database just aren't things that can be built to a deadline.
[+] dageshi|13 years ago|reply
Is it just me or are apple making more dumb mistakes lately? There's this and that ajax caching issue from a month or two ago.

Perhaps I'm just having selective memory?

[+] publicfig|13 years ago|reply
I think Apple has always had the tendency to make dumb mistakes, but I will say that I think reporting on it has changed. Remember all of the jokes about "It's not a bug, it's a feature" back when Jobs was there? Apple is by no means immune from mistakes and I don't think they ever were.
[+] stevoski|13 years ago|reply
Ever since I started using Apple gear, I've found apple's software quality is way below hardware. iTunes on Windows was always a disappointment.
[+] hesdeadjim|13 years ago|reply
I've had the exact same feeling. It seems like their quality control has gotten worse over the last year or so.
[+] danso|13 years ago|reply
I dunno, how crummy the App Store has become is a perfect example of this. The App Store is one prominent way they make loads of money and build loyalty to the ecosystem, and yet it is a shitshow on IOS6...the responsiveness is as bad as I've seen on non-iOS software, but for me, the two really inexplicable failures are:

1. the use of modals to show individual apps on the iPad. So if you get to an app page and accidentally touch outside the modal, the app page disappears and you have to go back to Safari to find the link that brought you there.

2. On my brand new iPod, the Genius View has never NOt crashed. I know Genius (and most of Apple's search services) are poor to begin with, but it's a prominent button on the nav. How does it just not work on a brand new device?

3. I'm also guessing the lack of a light sensor on the new iPod was the result of a screwup in the dev cycle...I would've never guessed that its absence would be such a deficiency but when you can't see a single thing on the iPod merely because you stepped from indoors to outdoors...to the point where you have to go back inside to adjust the brightness...that's just infuriating.

I can't remember the last time I've been infuriated by an Apple device...I feel infuriated all the time with my expensive Sony camera and how its video button is right where you can accidentally bump into it when shooting a pic...which prevents you from shooting a photo. The last few Apple devices I've had (two laptops, 2 phones, 2 iPads, countless iPods) have always been satisfactory in terms of polish...not iOS6 nor the new iPod touch

[+] pretoriusB|13 years ago|reply
>Is it just me or are apple making more dumb mistakes lately? There's this and that ajax caching issue from a month or two ago. Perhaps I'm just having selective memory?

No, it's not just you. It's the whole tech pop culture sector that has selective memory.

Apple did as much mistakes on the old times. It's just that it's not the underdog anymore, and all the hip kids are looking to fry Apple for those, instead of understanding that, heck, mistakes happen in every software/hardware offering.

The iPhone 4 antenna issue? Not even a blip compared to the iBook G3 logic board fiasco. But everybody forgets those things and adds it to some all-encompassing theory of decline...

Same with the notion of the "non-innovative update" for iPhone 5, etc.

The forgotten that less than a decade ago Apple did even far more minor incremental updates to the iPod line (a new click wheel, some more storage, a little thinner, now with a color screen, now it plays video, etc) and everybody was overwhelmed everytime a new device came out.

Suddenly they expect miracles out of every Keynote.

[+] chrisrhoden|13 years ago|reply
Author of the post here, happy to answer any questions. I'm pretty confident in our findings here, and some of the behavior is just bizarre.
[+] owenfi|13 years ago|reply
I'm seeing this line in my log as well, this person downloaded about 750MB and they don't seem to be on iOS 6.0.

  Reqs  User Agent
  5,378 Mozilla/5.​0 (Macintosh​; Intel Mac OS X 10_​8_​2) AppleWebKi​t/536.​26.​14 (KHTML, like Gecko) Version/6.​0.​1 Safari/536​.​26.​14
[+] dsl|13 years ago|reply
I saw similar behavior when I was working on the backend for an IOS app.

We were able to track it down to a retry mechanism for streaming content when the server was responding too slowly. Maybe you are running into something similar.

[+] sigzero|13 years ago|reply
Does 6.0.1 fix it?
[+] riobard|13 years ago|reply
Hmm, so this explained my 3G data overage last month. I have 6GB/month of data, and usually I'd be using around only 700MB. But last month my iPhone 4S (running iOS 6.0) consumed nearly 8GB in total, resulted in quite some overage fee.

Should I ask Apple for a refund for this? :|

[+] malyk|13 years ago|reply
Last month I moved and didn't have an ISP at home for the first 3 weeks or so. I set up tethering on my phone and a 5GB data plan. Yesterday I got a notice that I was over my data limit. I haven't done a whole lot of web surfing in the new house, but I have listened to a ton of podcasts using the Podcast app which I have never done before.

So if I just blew through 5 GB of data in 2ish weeks and the only thing different was using the podcasts app...

(I did watch 3 episodes of walking dead on Netflix which obviously contributed to the data use, but a 45 minute episode can't be that big)

[+] pk2200|13 years ago|reply
Same problem here. My monthly data usage increased by 10x after I upgraded to iOS 6.0. It's annoying that there's (apparently?) no mention of this issue in the 6.0.1 release notes.
[+] gigamike|13 years ago|reply
I had a similar experience. I normally consume 750-800 GB a month and on my first full month of iOS 6, Verizon said I consumed 8.89 GB. Luckily i'm grandfathered into an unlimited plan.
[+] frogpelt|13 years ago|reply
Maybe Apple was trying to stimulate the economy.
[+] mintplant|13 years ago|reply
Try your data carrier, rather than Apple.
[+] noinput|13 years ago|reply
Same here, noticed a huge spike in bandwidth for my startup's shows hosted on CloudFront. 206 responses are flooding my logs and I moved everything off of CloudFront for my users to triage the cost. This breakdown explains it all: http://cl.ly/image/2H3u3u2O333g

Talk about a pain, not to mention the 10x bill from Amazon that came as a surprise.

[+] donavanm|13 years ago|reply
"I moved everything off of CloudFront for my users to triage the cost."

Can you explain more? Was there some cloudfront behavior exacerbating the issue? AFAIK this is just a client (mis)using range requests and CloudFront replying as requested.

[+] kookster|13 years ago|reply
Exactly - we see this causing a crazy number of 206 calls.
[+] owenfi|13 years ago|reply
Our very small outfit at verbmill.net is seeing similar signs. For instance, 700MB+ downloaded by a single client when an individual podcast is a maximum of ~50MB. Checking my logs now to see if I can narrow down device/version number. All of our files are hosted on S3.
[+] owenfi|13 years ago|reply
Ominous example.

  Reqs	Total	Avg	B/s	Client
  666	2G	3M	127K	mobile-​198-​xxx-xxx-​088.​mycingular​.​net.​
  666	AppleCoreMe​dia/1.​0.​0.​10A403 (iPhone; U; CPU OS 6_​0 like Mac OS X; en_​us)
I hope this user is on an unlimited plan :(
[+] erickhill|13 years ago|reply
I believe this is directly related to phone data usage overages as well. Having used an iPhone 4 on ATT for 2 years, I could almost guess my data usage perfectly each month.

By the 2nd day of using my new Verizon iP5, I was receiving warning texts from Verizon that I was about to exceed my data limits.

Thinking it was me, I increased my plan. That only stopped the bleeding for another few days.

Oh, and I installed the "fix" that was sent out to supposedly repair the known issue. (I was always on Wi-Fi networks but still being charged for data.)

I recently called Verizon to dispute the data. They quickly erased all of my data overages. Snap, like that. The man I spoke with called a technician on my behalf.

The people who can erase overages do not officially have this issue in their database as something to be aware of. But the technician he spoke with said there was absolutely a problem, and both Verizon and Apple knew about it and were working on it.

Then, I was given a "diamond ticket" to absolve me of any future overages until a fix is found.

[+] fernandezpablo|13 years ago|reply
Makes you think twice about those graphics where iOS devices take a big piece of the pie of the "internet usage"
[+] eridius|13 years ago|reply
No it doesn't. Those aren't tracking bytes streaming over the internet, but rather web browsers visiting pages.
[+] andrewcastmate|13 years ago|reply
I don't have anything wildly new to add, but I'm part of a podcast hosting site (http://Castmate.fm) and we've seen this bigtime, it's crazy.

We host some high-traffic podcasts with a huge range of listeners, and I've gone through the logs and regularily see stuff like NUMEROUS listeners making 4000+ http requests in 30 minutes, all the 206 calls everyone here is reporting, it's just nuts, so bizarre.

I feel terrible for all the people getting hit with overages, it's so gross. I did some gorilla math the other day and feel pretty confident that this has boosted our network usage by ~25%, I should have narrowed it down to just the affected iOS 6.0 devices, must be astronomical.

[+] rachelbythebay|13 years ago|reply
This just seems to be how they handle audio media. It's not even anything specific to podcasts. My own scanner stuff is just a web page, and it gets hit by this from time to time. Each call is a single file, it's static, and there's nothing funny going on with it. It looks like the playback stuff is jumping around in some attempt to figure out exactly what it is first. It seems like it would be far better if they always downloaded at least some reasonable quantity and then decided to bail out later if it seemed overly huge, but this is where we are.

To me, it feels like someone who's trying to be fancy and do things like seek() in a stream. Just because HTTP has progressed to the point where you can effectively do that with Range: requests doesn't mean it's a good idea. Offering seek() type behavior for a HTTP resource seems like a great way to get people to use it and not even realize what sort of messes they may be creating. Leaky abstractions, eh?

Here's just one example from last week. Note: the file is 7001 bytes. There's no reason for this insanity, and yet here it is.

    x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:47 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 2 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
    x.x.x.x - - [04/Nov/2012:16:02:48 -0800] "GET /audio/16-1352070236.mp3 HTTP/1.1" 206 7001 "-" "AppleCoreMedia/1.0.0.10A403 (iPhone; U; CPU OS 6_0 like Mac OS X; en_us)" "scanner.rachelbythebay.com" "-"
[+] chrisrhoden|13 years ago|reply
Yep, as I mentioned in the article this is the AV Foundation framework - presumably used in all Apple apps since it's the official API they recommend. Doesn't surprise me that the browser would be affected.

The weird thing is that, when we first started doing the tests, it would happen immediately on pressing play. After some period of time running the tests, it started happening later and later in the process, indicating - what? No clue.

[+] kemiller|13 years ago|reply
This is exactly what I saw with the Podcast app last week. Used up 2GB in about 6 hours and put me into overage. I bet the class action lawyers are already circling.
[+] kokey|13 years ago|reply
I am anxiously awaiting an update to the Podcast app which fixes the issues that I am having, but perhaps I'm too optimistic. I listen to some podcasts from the BBC and the app doesn't recognise that a new daily show with the same name is new, the only way around it I have found is to delete my subscription and manually find the podcast every morning. Apart from that the thing crashes and runs very slowly at times.
[+] kannan|13 years ago|reply
Now that explains the sudden spurt in my 3G data usage in the last 3 weeks. This forced me to switch off my 3G :(. Hope Apple releases a fix soon
[+] farski|13 years ago|reply
We (PRX) haven't seen any evidence to show that it is still an issue in iOS 6.0.1, so you should be all set if you've upgraded, or an upgrade should fix it. If you are upgrade already and seeing overuse, please comment.
[+] meej|13 years ago|reply
Hmmm, I had to turn off the LTE on my iPad two days ago because I hit my limit 10 days prior to my monthly renewal. I upgraded to 6.0.1 last week, but I wonder if this bug explains how I managed to chew through my data plan faster than ever before.
[+] kevingadd|13 years ago|reply
The iOS 6.0 podcast bug apparently caused huge overages for GiantBomb as well. They're discussing it right now on one of their live streams; it apparently was so bad that their corporate/finance people are upset with them.
[+] bquarant|13 years ago|reply
NoAgendaShow.com has been discussing this since iOS 6 launched. Don't upgrade.
[+] berberich|13 years ago|reply
As pointed out above, 6.0.1 fixes this, so upgrading shouldn't be an issue.
[+] Mizza|13 years ago|reply
Ouch, elsewhere on the internet it seems like people are actually getting hit by this. Hopefully damage won't be too bad. Good research, PRX!
[+] blinkingled|13 years ago|reply
Wonder if the CDNs don't have any safeguards in place to avoid single client downloading same resource multiple times?
[+] chc|13 years ago|reply
That doesn't seem practical. How would they know if it's the same client? Just watching for multiple requests from the same IP will result in a lot false positives (e.g. most college students will be SOL). And what would they do if they could detect it but the client didn't realize it already had the file? Just drop the request?