Lots of people are focussing on Marco's particular use case in the comments, and I think it's a valid one, but this extends beyond simple documents.
There is a category of data that is aimed at offline use. Streaming apps like Spotify, that let you download playlists for offline use. GPS apps that download hundreds of MB of map data. You get the idea.
On one hand, this data is a form of cache. The data is always available elsewhere (on the content provider servers) and it can be restored if necessary in a worst case scenario. But the key word here is "offline". This is the kind of data that, by definition depends on being around if the user is offline and therefore cannot be easily restored on demand, when the user needs it.
Obviously, having all of this stuff backed up to iCloud and using up GBs of people's capacity is not feasible or even logical. So this kind of data does not belong anywhere that iCloud will back up. But it must be stored somewhere that is safe from being purged.
Yes, a users GPS maps can be restored eventually but that doesn't help them when they are stranded in the middle of nowhere with nothing but a weak GPRS signal and all of their maps gone.
Apple have made an almighty cockup in overlooking the "offline data" use case.
In Marco's case, I'd agree that the articles represent user data that should be stored somewhere like Application Support, which will be backed by iCloud but I think that's probably fine in this case.
I entirely agree. Apple forgot an important use case? I hate it when software is built to think in place of the user. There are two culprits here:
* the lack of persistent, non backable offline storage
* the addition of a auto-cleaning procedure
The auto-cleanup without a storage alternative is a blunder. iOS 5 decides for the user what should be deleted to free up space. An alternative would be to prompt what application data you would like to remove instead of a blind decision. Otherwise, that's the kind of programming that will one day lead robots to cleanup the human race! ;)
Of course, having permanent local storage that's not backed up would be a more user-friendly solution. However, it would probably lead people to run out of space when developers begin abusing it like some did with temp or cache storage.
It seems like a trend at Apple these days. With Lion, the OS now decides when an application should quit when unused.
I'm sympathetic to Marco's blog post here, but I wonder if he's failing to interpret the two paragraphs from the documentation within the context of Instapaper's purpose.
With respect to point #1, and in the context of what Instapaper is for, the user is intending to read an article offline, and the local copy of the article was generated by the user's intent (and can, therefore, be considered user-generated even though the user is not the author of the article.)
With respect to point #2, the articles fail the "can be downloaded again" test in light of the app's purpose of making the articles available to the user when the network is not available. When the network is unavailable, the articles cannot be downloaded again. Edit: JeffDClark makes another excellent point here that the article may have originally been behind a paywall and therefore cannot be guaranteed to be re-downloadable. The same is true if the author removed the original article from their webserver.
Ergo, put them in the Documents folder. Whether this would satisfy the app reviewer is another question, but it's worth a shot to carefully explain to them how your app is not violating the letter of the law.
I write several offline mapping apps, and this is totally throwing us for a loop. We're recommending our power users not upgrade to iOS 5. Users download gigabytes of maps to their cache directory, they don't want to eat their iCloud allotment with that, or their slow their iTunes sync. But they also don't want to have to download those maps again, or find themselves in the middle of the woods without the maps they downloaded.
iCloud is seamlessly integrated into your apps, so you can access your content on all your devices.
This is Apple's definition for the iCloud service. It doesn't matter what the data is, it's your data and Apple is promising to sync it between your devices, to preserve your experience.
In the case of Instapaper, the solution is obvious: Put the files in Documents. That is Instapaper's content and part of the experience that users want synced between devices.
If Apple penalizes developers and undermines the promise it is making to users because it decides to be miserly about bandwidth, then it has to admit it launched iCloud before it was ready.
Their marketing is definitely hyperbolic. Even in terms of Apple's own services. The supposed data syncing in iWork actually doesn't work on the Mac versions. It only works between iOS devices. So if I edit a paper on my Mac, and I want to use it on my iPad, I still have to go through an annoying uploading/downloading phase. There's nothing seamless about it.
I know this is a tangent to the OP but I was very annoyed to discover this last night, I feel like Apple was pretty misleading here. There are clearly technical difficulties that still need to be surmounted.
"If Apple penalizes developers and undermines the promise it is making to users because it decides to be miserly about bandwidth, then it has to admit it launched iCloud before it was ready."
That's the key right there. The size of the Instapaper database is determined by the user. It will only fill up a user's iCloud space if they decide to save a lot of documents. Plus, they always have the option to buy additional space!
Aside from making sure the documents are reasonably compressed, the size of the Instapaper database should not be Marco's concern.
But it's not clear that Instapaper is allowed to put the files in Documents. According to the Data Storage Guidelines cited in the article:
1. Only documents and other data that is user-generated, or that cannot otherwise be recreated by your application, should be stored in the <Application_Home>/Documents directory and will be automatically backed up by iCloud.
2. Data that can be downloaded again or regenerated should be stored in the <Application_Home>/Library/Caches directory. Examples of files you should put in the Caches directory include database cache files and downloadable content, such as that used by magazine, newspaper, and map applications.
I don't know if this is something enforced in the approval process, but it seems like Apple's intention is for redownloadable content to all be stored in temporary, wipeable locations.
Apple's in a tight spot here - either they leave things as they are and piss off developers (and, by fiat, piss off users), or they have to start forcing users to more proactively manage their remaining space.
One can probably easily imagine an interface for showing the user how much data a particular app is using and allow them to nuke the temporary stuff. It might even look beautiful. It might even be fun to use. But it's going to introduce a lot of hand-wringing and micro-managing and lots of mental overhead that Apple really, really wants to avoid.
This general problem is a symptom of a major issue regarding the use of software that does not respect the freedom of its users. Every non-apple developer and user is required to bend to the will of the programmers who develop the proprietary iOS. Any negative choices or limitations imposed by Apple Inc. are virtually uncircumventable.
On the other hand, if users and developers in a community are free to view, modify and share the source code of the operating system and programs they are using, then that creates a dynamic where software cannot remain defective by design, something which cannot be said about Apple's hardware or software.
If we were talking about free (as in speech) software, users would have control of their data, and their choice of how to manage it; not some distant programmers working for a corporation's bottom line.
iOS 5 now lets you see per app space usage. (it's under Settings > General > Usage) I don't see anyway to delete that temp data, however, only the app itself (for non-apple apps).
There are a number of possible scenarios for file storage, the problem is a lack of clarity or documentation about the properties of the various locations as they stand now. As a developer, I could imagine desiring the following choices:
1. Temp: No backup, cleared regularly
2. Cache: No backup, cleared when space is tight
3. Local: Local backup only, never cleared
4. Documents: Local/cloud backup, never cleared
5. Cloud: Cloud backup, cleared when space is tight
The problem seems to be that #3 doesn't exist. Yet you'd think it would be a common requirement for stuff like in-app purchases of large and essential content packs, for example, turn-by-turn navigation maps.
I'd hate to be on holiday and have a 10 megabyte podcast download automatically trigger the erasure of 1000 megabytes of navigation data.
I don't understand Apple's logic here. You can't reconcile the idea of "cleaning because you can redownload" with "available offline". As soon as you clean up, you are going to break offline uses.
I would guess the idea is to help enable these 500 MB per issue magazine downloads. You download a new issue, you nuke some old issue, no one cares. As long as that wasn't an issue you cared about.
I had the exact same question and your idea as to the answer is interesting. Perhaps this new behavior is in response to Newsstand's ability to automatically download new issues--users may allow those to stack up without any thought and run out of space. The question then becomes: why not just enforce this behavior for Newsstand?
The whole idea of an app like Instapaper (or any of the other examples presented) is that the "stuff" that is saved for later is all user-generated content. Some articles may even vanish (different location, move behind a paywall, deleted, etc...). In this case those articles would become inaccessible when the OS deletes the cache.
It seems that the argument that only the list of metadata is user-generated can apply to any type of media (music, movies, etc...). Technically speaking all of the music on my phone could be downloaded on demand. Of course this requires an always connected, fat, and cheap network connection. Which is pretty much the opposite of what most folks have.
This also breaks the user's expectations. I was annoyed/surprised when I upgraded to iOS 5 and Instapaper had to re-download everything.
The first time that one of my several hundred megabyte foreign language dictionary files isn't available and needs to be re-downloaded when I need it in say, a meeting will trigger a severe re-evaluation of my use of the phone.
I just emailed the company that makes the dictionary app in question (pleco chinese dictionary, if anyone cares) and they confirmed this issue will affect them.
I'm definitely at risk of running into this predicament as a user. I'm more worried about the hundreds of megabytes of podcasts I download when WiFi is available for use throughout the day when WiFi isn't available.
To avoid this issue and enjoy the benefits of iOS 5, I'm going to have to clear out a bunch of apps, music, and photos and ensure that I always have a sufficient amount of "buffer" space so the cleaning is never triggered. I cannot be alone in this, and the fact that Apple is making this kind of thinking necessary for end users is kind of ridiculous. I really can't see this behavior lasting for very long, and I'm sure Apple will address it soon; this is the antithesis to the traditional iPhone experience. The only scenario where I could see this being purposeful is if Apple is really trying to hurt offline apps to increase data usage and appease carriers (maybe for pissing them off with iMessage?).
It seems fair to say that Instapaper's version of an article can't be "redownloaded" for various reasons (offline, paywall, article removed, etc.) so it would be OK to put it in the Documents folder.
The argument against that is that you're now syncing that article with iCloud in addition to Instapaper.
But I wonder whether the correct answer is instead to eliminate Instapaper's sync feature, and just let iCloud do it. Once you have system-level cloud sync, don't you want to let Apple do the work? Sync is hard, and it isn't really the core value of Instapaper.
Edit: I was wondering about iCloud only for iOS 5/MacOS 10.7.2 devices with iCloud accounts. But that story does fall apart for people with mixed devices. So never mind.
I see it being a long while before everyone has iCloud up and running. We installed it last night on a relatively new MBP, to learn it doesn't work without Lion, so now we are upgrading to Lion. It will only work on certain phones, I believe 3GS and above.
Maybe Apple will release iCloud for Snow Leopard, but as it is now, instapaper works all the way down to the terrible version 2 iPhone I have had to resort to using in order to not sign into a new contract waiting on the release of the 4s.
Instapaper would hurt too many current users who are just fine with how things work without the cloud.
Isn't NSApplicationSupportDirectory for stuff that isn't "Documents" and yet needs to be managed as app state (not in 'Caches')? Why not just use that instead? That's what my app does and it seems to be OK with iOS 5.
I do the same thing. Apple is unclear about this, however. ApplicationSupport is at /Library/Application Support, and /Library definitely does get backed up to iTunes.
That said, I think we're in the clear with respect to iCloud backups:
"Devices with an active iCloud account have their app data backed up to iCloud at appropriate times. And for devices that are plugged into a computer, iTunes performs an incremental backup of the app’s data files."
It's ambiguous. But "app data" sounds like documents, whereas "app's data files" sounds like big data files in /Library to me.
Well, according to the documentation that folder is backed up to iTunes as well. Actually, it seems, the only folder that skips backups to iTunes in the Library folder is the Caches folder.
I wouldn't say that the articles' metadata (URL, title, date of download, maybe thumbnail of the 1st page) is downloaded content. It's clearly user-generated, and should be in the "home" of the app IMHO.
The articles themselves, yes, send them to the cache. If the user needs to reclaim the storage used up by the articles, let the OS delete them. Then, when the user needs to read the article again, it will take some time to download. But don't get into an "all articles gone" situation. Just my 2 cents.
I'd submit that this is, in some ways, worse: instead of all articles being gone they're now all grayed out, so the user knows they were here, can still see there, but if he's offline he can not access them because only the metadata is left, the data itself is gone.
What problem are you trying to fix? If I'm offline, what good is being able to see my article list if I can't read the articles? If I'm online, why do I feel any less irritated when I try to go back to an article I was just reading only to find that I need to wait for it to download again?
I thought Instapaper scraped the page and stored that, to accommodate sites that require login for access. So not trivial to just re-download the articles. Plus the whole point is to have them available when you're offline.
I'd say in this case, the content is user-generated and should be backed up.
For the record, I almost immediately filled up my allotted iCloud backup space because of rdio, because I have 4.6 gigs of music synced for offline use, and it wanted to back that up.
I develop Definition (http://definitionapp.com) and I store the database in the Caches directory as well, after receiving the email from Apple to move. This is bad news for me, the entire offline dictionary could be wiped out.
Maybe the solution is to make it a choice for the user? An option to "clean up documents when space is low on the device" in the instapaper options. If checked, stuff gets stored in cache. If not, in documents.
That way, the default behavior is that "download something = want to keep it on device", but users can do the other one too if they want. I don't think the option is particularly useful, but it might make the app reviewer happy?
I would bet Apple's never going to make this sort of a choice obvious to the user -- it goes against the "just working" (whether or not it works the way you want) they're currently pushing. It sure does worry me to read all these comments though; it seems like you'll never be able to count on an app having the data it needs to run, without re-downloading. As somebody who spends half his day in an airplane or a subway, plus plenty of time in expensive places (like overseas), that's terrifying.
This is what needs to happen. Either there needs to be a user confirmation before information can be "cleaned" from an app, or there needs to be a setting in the Settings/Appname for disallowing cleaning of that app's data.
[Edit] Upon further examination of the new Settings/iCloud in iOS 5, what will need to happen is that the user should be able to disable iCloud syncing for individual 3rd party apps. That's the most consistent solution for the current interface. [/Edit]
Is it possible that a user could use Instapaper to save a document that they wouldn't be able to download later, even if they had access to an internet connection? If that's possible, I think Instapaper would have every right to store things in the Documents folder, since you're asking it to create something more akin to an archive than a temporary cache of data.
That seems to be the only way for Instapaper to give their users a reasonable experience, but it's still a bad solution. It doesn't make much sense for the Instapaper app to backup its users' cached articles to iCloud, when the whole point of the app is to backup the articles on the iPhone. If you have access to iCloud, you have access to the original article, so it's a waste of bandwidth. Not to mention that Instapaper risks getting hounded by Apple for the iCloud usage.
[+] [-] lukeredpath|14 years ago|reply
There is a category of data that is aimed at offline use. Streaming apps like Spotify, that let you download playlists for offline use. GPS apps that download hundreds of MB of map data. You get the idea.
On one hand, this data is a form of cache. The data is always available elsewhere (on the content provider servers) and it can be restored if necessary in a worst case scenario. But the key word here is "offline". This is the kind of data that, by definition depends on being around if the user is offline and therefore cannot be easily restored on demand, when the user needs it.
Obviously, having all of this stuff backed up to iCloud and using up GBs of people's capacity is not feasible or even logical. So this kind of data does not belong anywhere that iCloud will back up. But it must be stored somewhere that is safe from being purged.
Yes, a users GPS maps can be restored eventually but that doesn't help them when they are stranded in the middle of nowhere with nothing but a weak GPRS signal and all of their maps gone.
Apple have made an almighty cockup in overlooking the "offline data" use case.
In Marco's case, I'd agree that the articles represent user data that should be stored somewhere like Application Support, which will be backed by iCloud but I think that's probably fine in this case.
[+] [-] jeromeparadis|14 years ago|reply
* the lack of persistent, non backable offline storage * the addition of a auto-cleaning procedure
The auto-cleanup without a storage alternative is a blunder. iOS 5 decides for the user what should be deleted to free up space. An alternative would be to prompt what application data you would like to remove instead of a blind decision. Otherwise, that's the kind of programming that will one day lead robots to cleanup the human race! ;)
Of course, having permanent local storage that's not backed up would be a more user-friendly solution. However, it would probably lead people to run out of space when developers begin abusing it like some did with temp or cache storage.
It seems like a trend at Apple these days. With Lion, the OS now decides when an application should quit when unused.
[+] [-] pohl|14 years ago|reply
With respect to point #1, and in the context of what Instapaper is for, the user is intending to read an article offline, and the local copy of the article was generated by the user's intent (and can, therefore, be considered user-generated even though the user is not the author of the article.)
With respect to point #2, the articles fail the "can be downloaded again" test in light of the app's purpose of making the articles available to the user when the network is not available. When the network is unavailable, the articles cannot be downloaded again. Edit: JeffDClark makes another excellent point here that the article may have originally been behind a paywall and therefore cannot be guaranteed to be re-downloadable. The same is true if the author removed the original article from their webserver.
Ergo, put them in the Documents folder. Whether this would satisfy the app reviewer is another question, but it's worth a shot to carefully explain to them how your app is not violating the letter of the law.
[+] [-] smokey_the_bear|14 years ago|reply
[+] [-] qjz|14 years ago|reply
That is the title of Apple's main iCloud page at http://www.apple.com/icloud/.
iCloud is seamlessly integrated into your apps, so you can access your content on all your devices.
This is Apple's definition for the iCloud service. It doesn't matter what the data is, it's your data and Apple is promising to sync it between your devices, to preserve your experience.
In the case of Instapaper, the solution is obvious: Put the files in Documents. That is Instapaper's content and part of the experience that users want synced between devices.
If Apple penalizes developers and undermines the promise it is making to users because it decides to be miserly about bandwidth, then it has to admit it launched iCloud before it was ready.
[+] [-] mbateman|14 years ago|reply
If you go to the bottom of http://www.apple.com/icloud/features/documents.html you can see the web interface you have to use.
I know this is a tangent to the OP but I was very annoyed to discover this last night, I feel like Apple was pretty misleading here. There are clearly technical difficulties that still need to be surmounted.
[+] [-] ChrisLTD|14 years ago|reply
That's the key right there. The size of the Instapaper database is determined by the user. It will only fill up a user's iCloud space if they decide to save a lot of documents. Plus, they always have the option to buy additional space!
Aside from making sure the documents are reasonably compressed, the size of the Instapaper database should not be Marco's concern.
[+] [-] wlesieutre|14 years ago|reply
1. Only documents and other data that is user-generated, or that cannot otherwise be recreated by your application, should be stored in the <Application_Home>/Documents directory and will be automatically backed up by iCloud.
2. Data that can be downloaded again or regenerated should be stored in the <Application_Home>/Library/Caches directory. Examples of files you should put in the Caches directory include database cache files and downloadable content, such as that used by magazine, newspaper, and map applications.
I don't know if this is something enforced in the approval process, but it seems like Apple's intention is for redownloadable content to all be stored in temporary, wipeable locations.
[+] [-] unknown|14 years ago|reply
[deleted]
[+] [-] ender7|14 years ago|reply
One can probably easily imagine an interface for showing the user how much data a particular app is using and allow them to nuke the temporary stuff. It might even look beautiful. It might even be fun to use. But it's going to introduce a lot of hand-wringing and micro-managing and lots of mental overhead that Apple really, really wants to avoid.
[+] [-] sudoman|14 years ago|reply
On the other hand, if users and developers in a community are free to view, modify and share the source code of the operating system and programs they are using, then that creates a dynamic where software cannot remain defective by design, something which cannot be said about Apple's hardware or software.
If we were talking about free (as in speech) software, users would have control of their data, and their choice of how to manage it; not some distant programmers working for a corporation's bottom line.
[+] [-] thehigherlife|14 years ago|reply
[+] [-] unknown|14 years ago|reply
[deleted]
[+] [-] zbowling|14 years ago|reply
edit: I'm shocked this thread is so long and no body mentioned this. It's been on the apple developer forums for months as a solution.
[+] [-] mpakes|14 years ago|reply
https://twitter.com/#!/marcoarment/status/124587437799374848
[+] [-] sjwright|14 years ago|reply
1. Temp: No backup, cleared regularly
2. Cache: No backup, cleared when space is tight
3. Local: Local backup only, never cleared
4. Documents: Local/cloud backup, never cleared
5. Cloud: Cloud backup, cleared when space is tight
The problem seems to be that #3 doesn't exist. Yet you'd think it would be a common requirement for stuff like in-app purchases of large and essential content packs, for example, turn-by-turn navigation maps.
I'd hate to be on holiday and have a 10 megabyte podcast download automatically trigger the erasure of 1000 megabytes of navigation data.
[+] [-] mw1234|14 years ago|reply
[+] [-] smackfu|14 years ago|reply
I would guess the idea is to help enable these 500 MB per issue magazine downloads. You download a new issue, you nuke some old issue, no one cares. As long as that wasn't an issue you cared about.
[+] [-] voxmatt|14 years ago|reply
[+] [-] JeffDClark|14 years ago|reply
It seems that the argument that only the list of metadata is user-generated can apply to any type of media (music, movies, etc...). Technically speaking all of the music on my phone could be downloaded on demand. Of course this requires an always connected, fat, and cheap network connection. Which is pretty much the opposite of what most folks have.
This also breaks the user's expectations. I was annoyed/surprised when I upgraded to iOS 5 and Instapaper had to re-download everything.
[+] [-] jackvalentine|14 years ago|reply
[+] [-] jackvalentine|14 years ago|reply
The ball is in Apple's court.
[+] [-] rubergly|14 years ago|reply
To avoid this issue and enjoy the benefits of iOS 5, I'm going to have to clear out a bunch of apps, music, and photos and ensure that I always have a sufficient amount of "buffer" space so the cleaning is never triggered. I cannot be alone in this, and the fact that Apple is making this kind of thinking necessary for end users is kind of ridiculous. I really can't see this behavior lasting for very long, and I'm sure Apple will address it soon; this is the antithesis to the traditional iPhone experience. The only scenario where I could see this being purposeful is if Apple is really trying to hurt offline apps to increase data usage and appease carriers (maybe for pissing them off with iMessage?).
[+] [-] phillmv|14 years ago|reply
Alternatively, just turn off iCloud back ups.
[+] [-] Terry_B|14 years ago|reply
[+] [-] euroclydon|14 years ago|reply
[+] [-] wrs|14 years ago|reply
The argument against that is that you're now syncing that article with iCloud in addition to Instapaper.
But I wonder whether the correct answer is instead to eliminate Instapaper's sync feature, and just let iCloud do it. Once you have system-level cloud sync, don't you want to let Apple do the work? Sync is hard, and it isn't really the core value of Instapaper.
Edit: I was wondering about iCloud only for iOS 5/MacOS 10.7.2 devices with iCloud accounts. But that story does fall apart for people with mixed devices. So never mind.
[+] [-] biturd|14 years ago|reply
Maybe Apple will release iCloud for Snow Leopard, but as it is now, instapaper works all the way down to the terrible version 2 iPhone I have had to resort to using in order to not sign into a new contract waiting on the release of the 4s.
Instapaper would hurt too many current users who are just fine with how things work without the cloud.
[+] [-] akmiller|14 years ago|reply
[+] [-] psychotik|14 years ago|reply
[+] [-] mrgrieves|14 years ago|reply
http://developer.apple.com/library/ios/#DOCUMENTATION/FileMa...
That said, I think we're in the clear with respect to iCloud backups:
"Devices with an active iCloud account have their app data backed up to iCloud at appropriate times. And for devices that are plugged into a computer, iTunes performs an incremental backup of the app’s data files."
It's ambiguous. But "app data" sounds like documents, whereas "app's data files" sounds like big data files in /Library to me.
[+] [-] ufuk|14 years ago|reply
The relevant part of the documentation is here: http://developer.apple.com/library/mac/#documentation/FileMa...
[+] [-] hernan7|14 years ago|reply
The articles themselves, yes, send them to the cache. If the user needs to reclaim the storage used up by the articles, let the OS delete them. Then, when the user needs to read the article again, it will take some time to download. But don't get into an "all articles gone" situation. Just my 2 cents.
[+] [-] masklinn|14 years ago|reply
[+] [-] luchak|14 years ago|reply
[+] [-] scdc|14 years ago|reply
I'd say in this case, the content is user-generated and should be backed up.
[+] [-] unknown|14 years ago|reply
[deleted]
[+] [-] theatrus2|14 years ago|reply
This is an interesting twist especially with the 16GB devices which tend to actually be above 50%.
[+] [-] j_baker|14 years ago|reply
[+] [-] jmcnevin|14 years ago|reply
Solution: I disabled backups of rdio data.
[+] [-] sidwyn|14 years ago|reply
[+] [-] Scorponok|14 years ago|reply
That way, the default behavior is that "download something = want to keep it on device", but users can do the other one too if they want. I don't think the option is particularly useful, but it might make the app reviewer happy?
[+] [-] high5ths|14 years ago|reply
[+] [-] WiseWeasel|14 years ago|reply
[Edit] Upon further examination of the new Settings/iCloud in iOS 5, what will need to happen is that the user should be able to disable iCloud syncing for individual 3rd party apps. That's the most consistent solution for the current interface. [/Edit]
[+] [-] jmcnevin|14 years ago|reply
[+] [-] baddox|14 years ago|reply
[+] [-] sp332|14 years ago|reply