Yes, PLEASE. We run a wedding photography business, and our numbers are something like this:
- 40-60gigs shot in a weekend, need to offsite the RAWs as soon as possible. In six or seven years we've never needed offsite retrieval. Fine for it to be slow.
- Only need around 100-200g of raws locally for jobs in progress
- Once processed into final jpgs files are approx 8-10x smaller. Offsite these as well, and need access to last 1-2 yrs of jpgs randomly/sparsely w/in 1-2 days of a print order coming in
I definitely think some well thought out service that uses maybe a hybrid of glacier and S3 could be really awesome. Especially if the pain is taken out of getting the bolus 40-50g of photos "uploaded" in the first place.. We'd happily pay for a service that just sends us an external drive and a postage-paid box.. we drop it in the mail after a job and get another drive sent to us right away.
[EDIT: Occurs to me you might wonder what this looks like today. Today we get home, and immediately back up onto an 8-bay Drobo Pro (http://bit.ly/1diZUHD). It's a fancy proprietary RAID-like system. I think ours has 7 1TB Drives in it currently, and it's at about 70% capacity. Then we back up the same jobs to an external hard drive which gets driven offsite (used to be my office, but now that I work at home it's the in-laws place) and copied to a hard drive for permanent offsite storage. Once that happens, the cards from the camera can be cleared. After processing, the final JPGs get copied to the drobo and make it into the sneakernet offsite process as well.]
EXTRA CREDIT: Anyone who considers tackling this and finds themselves interested in the niche of wedding/event photographer should look at combining this service with shopping-cart/gallery solutions that currently exist. The marketplace for those is fucking awful. Feel free to email me, but googling will get you the big players. I just don't wanna bash them by name in a public forum. All well meaning folks I'm sure, but every one of them are terrible flash-based, no-mobile software. Do backup and gallery/print-ordering management correctly and we're a customer at four figures per year.
> Occurs to me you might wonder what this looks like today
I finally hit a sweet spot of performance, archive accessibility, and offsite safety, by using internal SSD for latest import workspace, direct attached WD Velociraptor (Thunderbolt) for current projects (can reconnect between Mac Mini server and laptops as needed), and 2 direct attached LaCie 4big Quadras (daisy chained via FW800) for archived projects (up to 32 TB).
Interestingly, because these are all direct attached, BackBlaze will happily back them up offsite at ridiculously low cost if that's your thing. Initial backup of several terabytes will take several months, though.
For fast cheap offsite backups, partner with a colleague with similar storage needs who uses the same ISP via the same head end. Just backup to each other. You can enjoy near LAN latencies at maxed out link speeds. This trick will save you both time and headaches.
Finally consider a product like the CyberPower OR2220PFCRT2U (this one also blends nicely in an AV cabinet) to keep an Airport Extreme and Mac Mini along with all that storage running for about 80 minutes of power outage.
Most (serious) safes are capable of withstanding a fire or a flood.
I don't have near the volume of pictures that you have (around 1.5 TB) but here's my setup:
- one disk attached to the Lightroom computer, with all images on it; it's local, so it's fast
- one NAS (NAS1) with a 4-disk RAID that backs up this drive (3 TB capacity)
- another NAS (NAS2) with just one 3 TB disk that backs up the other NAS
On NAS2 I rotate disks; disk A is in the safe when disk B is in the NAS.
I think my setup is quite similar to yours, except driving is replaced by "going to the safe in the basement".
If a disaster happen I should have at least the disk in the safe intact, which holds the data since the last rotation; and since backups from NAS1 to NAS2 are quite fast I try to have both disks pretty much in sync (after an important shoot I back up to disk A AND disk B).
Of course I won't know if this setup is good until my house does burn down (I hope I'll never know!) but it does give me peace of mind, which is at least something ;-)
Your happens today is very similar to mine. I'd say that I need additional offload of the RAID array (drobo in your case), while little is needed to be changed on my processing laptop
We applied to YC with exactly this idea. We made it to the interview, but were turned down. I wrote about that experience: http://tghw.com/blog/pull-hard
We pulled our bootstraps hard and tried to get it going, but soon realized that there's a pretty big flaw with the idea: most photographers have no idea that they should backup their photos. They know hard drives crash, but they never expect it to happen to them.
This meant that we'd essentially have to become insurance salesmen, convincing people that bad things will happen to them.
> For this experience, we spent less than $1,000. Compared to an entrepreneurship class at any college, that's quite cheap!
My undergrad was in USC's entrepreneurship program. One of the points they always hyped on was whether or not you were "selling vitamins or vicodin", so to speak. As you learned through Snaposit, most people when making chit-chat will be supportive of your venture, but you won't know how they really feel until they vote with their wallets.
Most of the professors were older guys who didn't get online startup culture, but one of their favorite tricks was to put up a landing page detailing a concept they were interested in, with a buy button. They'd spend some AdWords to get people to the landing page and measure the conversion rate. If enough people tried to buy to justify the service, they'd go out and build it.
Professional photographers know they need to backup their photos.
It looks like your product targeted consumers, but professionals seem like they'd be a better market. And it should have been priced way, way higher; the more it cost, the more secure it looks. I'd have a hard time trusting my money-making business assets to a $9/month service.
Of course, I'm not a professional photographer nor do I have your experience in trying to build this business, so who knows what my opinion is worth. :-)
Yeah, I'd expect the market for this is more niche but could be high revenue-per customer. As a wedding photography studio we'd pay 100$/mo in a heartbeat if this was done well.
[Edit]
Also note that OP wasn't really just pitching "backup". He wants you to reduce the footprint his images take up locally. I think the messaging to casual and hobbyist users probably ought to play on this. Note Loom's taglines "more room to play". Backup is probably boring for people not under the pressure of liability like pros are.
Would you be willing to open source your code? I am running out of hard disk space as well for my photos and have been thinking of writing coding up something very similar to this idea _when I get time_.
But I would be willing to contribute to an open source project that does this for self-hosting my photos etc. Contact me if interested, email's in my profile.
I really wish we as a developer community would better differentiate between "backup" and "preservation". They are both related to storage, but are fundamentally different problems. The article touches on the desire to have photos safe for decades, but doesn't really get into the strategy necessary to make that happen.
If we care about decades, storing the raw NEF or CR2 file is almost certainly the wrong approach. Those files could easily be as difficult to open in fifteen years as a ClarisWorks document is today (just to choose one example - proprietary formats for professional software from the late 90's are all pretty hard to open today). Also, important metadata is kept in the lrcat file. Unless we expect to always be able to open our copies of Lightroom, we need to migrate that data too.
We used to think of "archiving" as putting something on a shelf and forgetting about it, sometimes taking environmental factors like humidity into account. With our digital data, we need a plan for periodically checking it, both for fixity and to make sure at risk file formats are migrated to current best practices. This will require greater awareness for the complexities involved (and the need for open file formats) from everyone from creatives to storage and service providers.
By the way, every creative industry is struggling with this right now. Recording studios have session files created in ProTools 4. Designers have really old Quark files, even though they've been working in InDesign for the past five years. Composers have Finale files. A lot of these file formats depend on plugins to be able to display properly. I don't mean to sound discouraging, I think there's plenty of opportunities for viable businesses here. We just need to start using the right terms.
In a word, VMs. If you make sure you can open that NEF or CR2 file in a virtual machine, and the virtual machine image is in a standard, open format, you're set. You just have to worry about the image format becoming obsolete and not 10 or 20 different proprietary formats.
That said, I used to be a pretty prolific hobbyist musician and I haven't pulled this off myself. I have tons of old songs I can't open because they rely on a fiddly collection of old freeware plugins for freeware music apps from 11 years ago. I also have songs I can't open because they only play in a music tracker I wrote, which is open-source, but only runs on OpenBSD.
It is a tricky problem and it seems hard to tackle because the market for professional software inherently favors closed-source, which has no incentive to adopt open, non-siloed formats.
...do we know, when you store something in Glacier, does Amazon keep multiple copies, or just ONE on some unspecified media? Do they take a hash fingerprint, and compare it to the copy for bit rot corruption?
Even without getting to format obsolescence, the strategy of just keeping the bitstream reliably and verifiably retrievable for 'decades to come' is worth discussing.
One of the only ways to do this, is having everything stored in a fully documented format. For photos, I don't think much exist except the most naive raw format (think bitmap files, just in same colour space as NEF or the like), accompanied with an XML/JSON file with the metadata and a document describing this. Once it is needed, you have to convert it. Sounds about right you could build a business on that idea - first step is to find someone willing to pay for it though.
I've been looking into Glacier, but the pricing in the case where I might want to retrieve a substantial portion of my data is quite complex. As far as I can tell, the headline $0.01/GB rate for overages they quote is misleading. They don't, as you might expect from that language, calculate ([GB retrieved] - [5% of GB stored]) x $0.01 and bill you that.
Rather, they charge you on a monthly basis, according to your highest hourly retrieval rate from any single hour applied to the whole month (standardized at 720 hours). So if your peak hourly retrieval in a month was 10 GB/hr above your pro-rated free quota for that hour, you're charged for 7200 GB of retrieval overage, or $72, even if you retrieved nowhere near 7 TB of data. For companies with relatively constant retrieval this doesn't matter, but for small users, someone who has a large overage one day will be charged as if they were incurring the same overage continually all month.
To put it concretely, if you store 200GB of photos and then do a bulk retrieval, you aren't charged $1.90 ((200-10) x 0.01). Rather, the xfer is counted as 4 hours, so you have a peak transfer of 50 GB/hr. Your free daily retrieval is allocated to the hours in the day you retrieved data, so in this case 10/4 = 2.5 GB/hr free. So your overage is 47.5 GB/hr. Multiply by 720 and $0.01, and the fee for retrieving 200GB of photos is $342. Plus $22.88 in outgoing AWS bandwidth.
"SmugVault is an added service to your SmugMug account that lets you store almost anything for next to nothing. Including files not normally supported by SmugMug."
Although Smugmug is awesome, the issue I'd have with them is the lack of control. If my objects are in my AWS account, that feels more permanent long-term. It's just a gut feeling.
My company serves thousands of professional photographers (wedding, commercial, fashion, etc.). What you're describing is definitely an issue for photographers but I think getting them to use a tool like you've described would be an uphill battle.
Right now, most of them use Drobos or similar devices. You'd be asking them to add steps to their workflow which is always tough to do.
Also, I think your estimate of 18 mins to upload 6+GB is an order of magnitude off. We have tools that allow people to upload JPEGs and most customers claim it takes longer than that just for 100s of MBs.
My (wild) guess is that Adobe is going to get in this space too. It only makes sense to have your Lightroom catalog in the cloud. Maybe they wont offer an affordable solution for terabytes of data, but I'm not sure photogs want their images in two different "clouds".
I think the key is to do customer development from full-time photographers who aren't hackers. I'd be happy to send out a survey to some clients if you want do more research.
My Lightroom catalog is backed up to a folder in Dropbox, and this works really well.
You can't really have the current catalog in Dropbox because then each action and edit become really slow.
But if you set up Lightroom to do a backup every time you quit the program, and that backup is in Dropbox, you get the best of both worlds (strong backups + snappy edits).
Lightroom backups are really "backups", as they are named by date+time, so in Dropbox I have a history of everything I ever did in Lightroom, not just the current (last) version.
It's also fast because I suspect Dropbox is able to do some rsync magic and only re-upload new packets; my Lightroom catalog is around 150 MB and the upload to Dropbox is almost instantaneous (on a pretty lousy upstream bandwidth).
Of course this doesn't address the problem of backing up RAWs, but as far as Lightroom is concerned I feel pretty safe.
As a photographer with a couple of terabytes of old raws (I back up to Time Machine locally, and offsite at a friend's house occasionally), I'm very curious as to what Adobe will offer in this regard with LR5.
This idea is great, but it ignores on very real problem. Namely that you still have not addressed the issue of bandwidth. Take for example your 4th of july example. If I were to do that upload the way you describe, I would already be over my monthly bandwidth allotment with my ISP as would many hobbyists.
The US bandwidth situation is the elephant in the room.
There really is no good solution to automated offsite backup when it takes several days of uploading to handle a single day's worth of data capture. To make any internet-backup service work, you'd first have to pitch the photog on switching from their residential broadband package to something more expensive -- if that's even available short of their leasing office space somewhere.
I guess you could try to build out a reverse-CDN sort of network, with local relay nodes scattered across the various ISP networks, that might achieve faster uploads from the user (and maybe not get counted against bandwidth caps) which then use a fat pipe to send that data to larger regional storage nodes.
We want to take backing up, sharing, and monetizing and make them very simple and enjoyable to do. We've started rolling out slowly already, our goal is to be open to the public within a month or so.
I hope your business model doesn't rely too heavily on the "monetizing" aspect. I know there are lots of photo hosting services with a "sell prints" option, but I get the impression not very many people actually use those in enough volume to cover the hosting costs. I'm imagining a non-pro track that's targeted at the enthusiast who just wants backup and sharing, but won't actually sell any prints. I'll be watching.
I'm in a similar situation as the OP...an amateur but avid photographer with terabytes of RAW image files.
Currently, I have several multi-terabyte drives that I erratically back up to...I just got a 4TB drive that I'm going to try to backup everything I've ever shot in the last 3 years...
But I do want to move it to online storage...and I think this requires triage. For nearly every photoset, I've quickly gone through them with Lightroom and starred the ones that I kind of like, and then for maybe 1% of them, have taken the time to fully process, label, and upload them (in JPG form) to Flickr.
So for online storage, I think what I'll do is write a batch script that dumps all the unstarred RAW files as JPGs, because they chances that I'll ever need these photos in RAW is very slim, and then upload them. For photos that I've given at least one star and/or taken the time to properly edit and categorize, I'll send up the RAW file along with the processed JPG.
If I have about 3TB of RAW files...and maybe 10% of those are RAW images that weren't disposable...that's 300GB right there. And then the remaining RAW "keep em just in case" files would end up being compressed from 24MB to about 2-3 MB (let's say 15% as an average)...so 400GB.
700GB is still quite a big footprint. However, the biggest problem will be...let's say I give up local storage all together and rely on the cloud...what's the best way to browse any photoset at any arbitrary time? When it's all local, it's trivial to pop open Lightroom and do some quick browsing and filtering. For online storage, I'll have to put a ton of work into proper folder-naming, at the very least
Your last few sentences are exactly why I wrote this post. I want everything stored in the cloud but ALSO have a decent way of browsing them locally. The idea is to just store tiny jpegs (you can set how big they are, but something like 500-1000px wide likely, just for you to get a sense of which photo this is and allow you to remotely delete, retreive, etc)
Here's my rant, take it for what it's worth :) I keep 4 types of photos:
1.) Personal: This doesn't mean I don't delete, but I generally tend to keep photos even though they might not be really good photos, especially if I only have a few of that day or person. I know I might enjoy looking at them decades later anyway, a blurry photo is better than nothing.
2.) Archival: like taking a photo of a flat before moving in. These aren't very many, so I tend to keep them no questions asked.
3.) Siblings of Keepers: You could call these "proof I really made the keeper photo" photos. Let's say I come across an animal I never photographed before, and like one of the photos enough to publish; then I'll still keep the others around. Or keeping a wider crop in case you change your mind later, etc.
4.) Keepers: here I try to be as harsh as I can. Photography is about painting with light for the pros, for me it's mostly about selection, that's the part of it I enjoy most. I walk around, and from the thousands of things I see each day, only a few make me take out the camera. Then I select keepers (partly before the images even leave the camera), and even years after I might revisit some keepers again, decide that my standards improved, and delete them.
I don't want to tell others how to pursue their hobby, but I think the web is full of galleries not even the owners look at, and that generally, a lot of people need to learn how to delete. The same goes for blogs and whatnot, the idea that everything has to be written on this indelible roll of toilet paper seems silly.
On one extreme, there is being OCD and deliberating endlessly what to keep and how to rank it, on the other there is just making more stuff to throw behind yourself on a pile you never investigate, because it's too awkward to wade around in. Productivity lies somewhere in the middle I think.
I have a serious problem with the proposal offered in TFA and most variants in this thread: it encourages bad backup policy. Critical data needs three (or more) backups. In my experience, one cloud provider can only ever be considered as being one backup no matter their technical architecture.
For example, you can't (and in the broad "you", aren't even qualified to) audit the provider for data-loss or SLA-impacting SPOFs. You just have to assume that they're there. You also take on non-technical failure modes: the provider can go out of business, get bought by an uninterested owner (think Delicious), change focus (Google Reader), or experience myriad other problems.
FWIW, the best success I've had with helping others switch to good backup policy happened once it became economically feasible to buy a new laptop and two bus-powered external drives as big as the laptop's internal storage. The externals become bootable backups, one of which can be preiodically rotated to an offsite location. This has prevented severe dataloss events for myself and others far more times than I care to count now.
FWIW, I'm more technical than a typical user might be but I'd be very intrigued by a provider that used multiple cloud service and gave me the keys to the underlying S3 buckets (or whatever) so I could independently verify. That's probly a more valuable idea to a company selling backup to nerds though, I guess.
I still prefer local storage and self-hosted storage, but what I'd like is a good plugin for Lightroom to manage cloud storage and multiple, versioned USB drives (or network locations) for images, from within the Lightroom interface.
(for local storage, once you go beyond individual USB drives, I'd go with a Synology 1813+ NAS ($999 chassis, 8-bay) or an Areca ARC-8050 thunderbolt raid ($1499, 8-bay).)
I believe Photry (www.photry.com, currently in beta, I'm the founder) could be a possible solution in the long run. Since we started, a lot has changed on online photo storage market.
With that in mind we've started to look more on how photographers with larger photo volumes could benefit from our service. From the feedback from few photographers we've already included some features (RAW photos with smaller JPG thumbnails, workspaces and personal sharing for example) that will make the day to day workflow a bit easier. Currently our goal is to build something that would fit better for professional photographers (public for clients, ordering prints and possibly integrated payment flows, win/mac client + plugins for your favorite photo software).
Pricing wise we are currently not the cheapest but we have thoughts on how to make this better for high volume photographers. If you're one of them then please send me an email (martin at photry.com) and I would love to talk a bit further what we plan to do and if we would fit your storage needs.
A service would be great ... but you might be able to "roll-your-own" with some basic scripts to handle the conversion (down-sizing) and backup sync - and something like Koken to handle the sharing with friends & family.
For back-ups - just buy a huge disk or set of disks, and keep 'em off-site wherever you have a trustable location (friend, family, work - if allowed and bandwidth isn't an issue) and use CrashPlan's free software to sync a copy to the secondary location. It's free after the initial expense of the drives and gives you encrypted, but physical access to the content in case of an emergency. You could find a friend with the same problem, and just agree to dedicate a certain amount of disk-space to each-other for this purpose.
I'm the SO of a professional newborn photographer who generated somewhere in the neighborhood of 20-30TB of RAW photos, backups, LR xmp sidecars, etc in the last year. JPEGs don't even register. High end DSLR RAWs are gigantic!
I'm the technical support for overall workflow, storage & backup engineer, etc, etc. I wear many hats.
Right now I've got a backup on import to a FreeNAS (BSD) filer running on a HP Micro Server. Lightroom backups go to the same filer and are synced as well. My local server rsyncs the delta every night to an identical machine at a friends' house.
This isn't perfect. The one place that isn't 100% backed up is current projects in post-production. That's an acceptable trade off for now.
The most important thing to note is that there are two kinds of photographers: Those who never lost data, and those who care about backups. As mentioned elsewhere in this thread, converting members of Group A into Group B is a tough sell. Video testimonials might be a good option.
A storage service for photographers MUST be invisible to their workflow. Make it as easy to use as an egg timer. At the MOST, an addition of a single screen with sensible defaults (just click next) on photo import.
Ignoring error messages that get in their way is to be expected. Care more about their backups than they do. Send me (tech support guy) an email to notify if there haven't been backups added in X days, along with weekly reports of data usage per day -- how many photos, how large, what root folders, usage trends, etc.
An ideal installation would be automatic detection of what program they're using, what places files are stored (hint: It's not just on a local drive). Detect new file locations and back those up too.
An inexpensive (<$25) option of "only backup current projects I'm working on and anything I shot in the last month", that actually works... I'd sign up in a heartbeat.
Backup is about having several copies. If you delete files after you uploaded them to the cloud, you don't have a backup - with all the problems that come with it (say, if you delete a file by accident there will probably be no way to restore it).
If your files are important to you - you need a back up. If not - just get rid of them in the name of simplicity :)
PS. Also, uploading many gigabytes over a typical ADSL connection is just painfully slow... even local NAS is too slow over WiFi for my taste, and if I have to have a wire - I can just as well connect an external drive.
I wanted the benefit of Glacier plus backup functionality so I went with Arq. It encrypts everything and supports Glacier. Works really, really nicely.
I only used Glacier in this example but I did mention how it could be tied to additional cloud services. The thing is that not many like cheap long-term Glacier alternatives exist yet. Rest assured when they do, this ideal app would have them in the settings. :)
I backup offsite via CrashPlan which keeps files even after I delete them. I sent them a hard drive for my initial upload. Now, I've got about 800M backed up. I haven't yet had to get it all back :)
I wish I could trust the cloud because it would be a lot more convenient to not have to worry about storing photos, which involves a lot of effort once you advance beyond a casual hobby and you get into storing Terrabytes of large RAW files and maintaining adequate backups.
But the idea of storing any files in the cloud is very distasteful to me ever since the NSA Snowden story broke. Even something as innocent as a photographer taking pictures of a skyline or some beautiful architecture could land him in hot water if the government is looking at metadata and figuring out that he took lots of photos of a government building or a bridge or a power plant and randomly throwing him on some watch lists or no-fly lists. People have been hassled by the feds for a lot less.
My preference would be for all of these cloud storage providers like Google, Facebook, Apple, Dropbox, Amazon, etc to unambiguously and fiercely challenge the government on this and somehow unambiguously prove to people that they're respecting peoples' privacy and not complying with this stuff. Their denials so far have been obvious non-denials that are very carefully worded around "no direct access" when that's enough wiggle room to do just about anything.
Using some kind of cloud photo storage is a bad idea IMO until this gets sorted out.
- Local daemon running that manages this process, on PC or Mac. It can't be done solely from the browser.
- The software manages the import from the SD/CF cards, you put them all in, it sorts everything out and wipes the SD cards on the way out.
- You then use Lightroom to do whatever you're going to do. The software would do well to have a Lightroom plugin that lets you select particular images to mark as "high priority" and "low priority."
- Now the daemon uploads all of your RAWs, Lightroom metadata, and corrected JPGs to cloud storage.
- Based on your settings, these RAWs get discarded after some set amount of time (maybe immediately, maybe never), and the highest-quality JPGs are also discarded after a longer amount of time for lower-quality JPGs just in case you need a emergency backup. After 6 months you lose the raws, 1 year the highest quality JPG, 2 years the medium quality JPG, etc. You are charged per gigabyte per month so you can decide how much you care about backup. Basically offload the cognitive effort of discarding old data that isn't worth its upkeep costs anymore.
- The files are available via an API that you can use to build your own gallery/shopping cart site, with the ability to call a certain quality, PROOF watermarking, etc. on-the-fly. But the service offers a default site right there for you to do the same, that's stylable and can be implemented on your own domain.
- You can get your entire catalog shipped to you on a hard drive at any time for a few hundred dollars.
"I open the folder where Lightroom imports are stored and drag these shots to the RAWbox (fake name for this ideal photo app) desktop application."
You shouldn't be touching the directory structure. This tool should be either a plugin for Lightroom/Aperture/Photoshop or an entirely separate application.
I would lean towards separate application. It could be a background agent that detects when new files are added, would work for any known editor or user-defined directory.
[+] [-] famousactress|12 years ago|reply
- 40-60gigs shot in a weekend, need to offsite the RAWs as soon as possible. In six or seven years we've never needed offsite retrieval. Fine for it to be slow.
- Only need around 100-200g of raws locally for jobs in progress
- Once processed into final jpgs files are approx 8-10x smaller. Offsite these as well, and need access to last 1-2 yrs of jpgs randomly/sparsely w/in 1-2 days of a print order coming in
I definitely think some well thought out service that uses maybe a hybrid of glacier and S3 could be really awesome. Especially if the pain is taken out of getting the bolus 40-50g of photos "uploaded" in the first place.. We'd happily pay for a service that just sends us an external drive and a postage-paid box.. we drop it in the mail after a job and get another drive sent to us right away.
[EDIT: Occurs to me you might wonder what this looks like today. Today we get home, and immediately back up onto an 8-bay Drobo Pro (http://bit.ly/1diZUHD). It's a fancy proprietary RAID-like system. I think ours has 7 1TB Drives in it currently, and it's at about 70% capacity. Then we back up the same jobs to an external hard drive which gets driven offsite (used to be my office, but now that I work at home it's the in-laws place) and copied to a hard drive for permanent offsite storage. Once that happens, the cards from the camera can be cleared. After processing, the final JPGs get copied to the drobo and make it into the sneakernet offsite process as well.]
[+] [-] PStamatiou|12 years ago|reply
[+] [-] famousactress|12 years ago|reply
EXTRA CREDIT: Anyone who considers tackling this and finds themselves interested in the niche of wedding/event photographer should look at combining this service with shopping-cart/gallery solutions that currently exist. The marketplace for those is fucking awful. Feel free to email me, but googling will get you the big players. I just don't wanna bash them by name in a public forum. All well meaning folks I'm sure, but every one of them are terrible flash-based, no-mobile software. Do backup and gallery/print-ordering management correctly and we're a customer at four figures per year.
[+] [-] VikingCoder|12 years ago|reply
Do you want a one-time fee, or some yearly fee?
[+] [-] Terretta|12 years ago|reply
I finally hit a sweet spot of performance, archive accessibility, and offsite safety, by using internal SSD for latest import workspace, direct attached WD Velociraptor (Thunderbolt) for current projects (can reconnect between Mac Mini server and laptops as needed), and 2 direct attached LaCie 4big Quadras (daisy chained via FW800) for archived projects (up to 32 TB).
Interestingly, because these are all direct attached, BackBlaze will happily back them up offsite at ridiculously low cost if that's your thing. Initial backup of several terabytes will take several months, though.
For fast cheap offsite backups, partner with a colleague with similar storage needs who uses the same ISP via the same head end. Just backup to each other. You can enjoy near LAN latencies at maxed out link speeds. This trick will save you both time and headaches.
Finally consider a product like the CyberPower OR2220PFCRT2U (this one also blends nicely in an AV cabinet) to keep an Airport Extreme and Mac Mini along with all that storage running for about 80 minutes of power outage.
[+] [-] bambax|12 years ago|reply
I don't have near the volume of pictures that you have (around 1.5 TB) but here's my setup: - one disk attached to the Lightroom computer, with all images on it; it's local, so it's fast - one NAS (NAS1) with a 4-disk RAID that backs up this drive (3 TB capacity) - another NAS (NAS2) with just one 3 TB disk that backs up the other NAS
On NAS2 I rotate disks; disk A is in the safe when disk B is in the NAS.
I think my setup is quite similar to yours, except driving is replaced by "going to the safe in the basement".
If a disaster happen I should have at least the disk in the safe intact, which holds the data since the last rotation; and since backups from NAS1 to NAS2 are quite fast I try to have both disks pretty much in sync (after an important shoot I back up to disk A AND disk B).
Of course I won't know if this setup is good until my house does burn down (I hope I'll never know!) but it does give me peace of mind, which is at least something ;-)
[+] [-] xxpor|12 years ago|reply
[+] [-] unknown|12 years ago|reply
[deleted]
[+] [-] alinspired|12 years ago|reply
[+] [-] tghw|12 years ago|reply
We pulled our bootstraps hard and tried to get it going, but soon realized that there's a pretty big flaw with the idea: most photographers have no idea that they should backup their photos. They know hard drives crash, but they never expect it to happen to them.
This meant that we'd essentially have to become insurance salesmen, convincing people that bad things will happen to them.
Eventually, we decided to shut it down, which I also wrote about: http://tghw.com/blog/well-that-sucks-what-else-you-got
The old homepage is still up at http://www.snaposit.com/.
[+] [-] bsimpson|12 years ago|reply
My undergrad was in USC's entrepreneurship program. One of the points they always hyped on was whether or not you were "selling vitamins or vicodin", so to speak. As you learned through Snaposit, most people when making chit-chat will be supportive of your venture, but you won't know how they really feel until they vote with their wallets.
Most of the professors were older guys who didn't get online startup culture, but one of their favorite tricks was to put up a landing page detailing a concept they were interested in, with a buy button. They'd spend some AdWords to get people to the landing page and measure the conversion rate. If enough people tried to buy to justify the service, they'd go out and build it.
[+] [-] mfenniak|12 years ago|reply
It looks like your product targeted consumers, but professionals seem like they'd be a better market. And it should have been priced way, way higher; the more it cost, the more secure it looks. I'd have a hard time trusting my money-making business assets to a $9/month service.
Of course, I'm not a professional photographer nor do I have your experience in trying to build this business, so who knows what my opinion is worth. :-)
[+] [-] famousactress|12 years ago|reply
[Edit]
Also note that OP wasn't really just pitching "backup". He wants you to reduce the footprint his images take up locally. I think the messaging to casual and hobbyist users probably ought to play on this. Note Loom's taglines "more room to play". Backup is probably boring for people not under the pressure of liability like pros are.
[+] [-] tsycho|12 years ago|reply
[+] [-] esonderegger|12 years ago|reply
If we care about decades, storing the raw NEF or CR2 file is almost certainly the wrong approach. Those files could easily be as difficult to open in fifteen years as a ClarisWorks document is today (just to choose one example - proprietary formats for professional software from the late 90's are all pretty hard to open today). Also, important metadata is kept in the lrcat file. Unless we expect to always be able to open our copies of Lightroom, we need to migrate that data too.
We used to think of "archiving" as putting something on a shelf and forgetting about it, sometimes taking environmental factors like humidity into account. With our digital data, we need a plan for periodically checking it, both for fixity and to make sure at risk file formats are migrated to current best practices. This will require greater awareness for the complexities involved (and the need for open file formats) from everyone from creatives to storage and service providers.
By the way, every creative industry is struggling with this right now. Recording studios have session files created in ProTools 4. Designers have really old Quark files, even though they've been working in InDesign for the past five years. Composers have Finale files. A lot of these file formats depend on plugins to be able to display properly. I don't mean to sound discouraging, I think there's plenty of opportunities for viable businesses here. We just need to start using the right terms.
[+] [-] graue|12 years ago|reply
That said, I used to be a pretty prolific hobbyist musician and I haven't pulled this off myself. I have tons of old songs I can't open because they rely on a fiddly collection of old freeware plugins for freeware music apps from 11 years ago. I also have songs I can't open because they only play in a music tracker I wrote, which is open-source, but only runs on OpenBSD.
It is a tricky problem and it seems hard to tackle because the market for professional software inherently favors closed-source, which has no incentive to adopt open, non-siloed formats.
[+] [-] jrochkind1|12 years ago|reply
...do we know, when you store something in Glacier, does Amazon keep multiple copies, or just ONE on some unspecified media? Do they take a hash fingerprint, and compare it to the copy for bit rot corruption?
Even without getting to format obsolescence, the strategy of just keeping the bitstream reliably and verifiably retrievable for 'decades to come' is worth discussing.
[+] [-] hvidgaard|12 years ago|reply
[+] [-] _delirium|12 years ago|reply
Rather, they charge you on a monthly basis, according to your highest hourly retrieval rate from any single hour applied to the whole month (standardized at 720 hours). So if your peak hourly retrieval in a month was 10 GB/hr above your pro-rated free quota for that hour, you're charged for 7200 GB of retrieval overage, or $72, even if you retrieved nowhere near 7 TB of data. For companies with relatively constant retrieval this doesn't matter, but for small users, someone who has a large overage one day will be charged as if they were incurring the same overage continually all month.
To put it concretely, if you store 200GB of photos and then do a bulk retrieval, you aren't charged $1.90 ((200-10) x 0.01). Rather, the xfer is counted as 4 hours, so you have a peak transfer of 50 GB/hr. Your free daily retrieval is allocated to the hours in the day you retrieved data, so in this case 10/4 = 2.5 GB/hr free. So your overage is 47.5 GB/hr. Multiply by 720 and $0.01, and the fee for retrieving 200GB of photos is $342. Plus $22.88 in outgoing AWS bandwidth.
Or at least, I think that's what their pricing page says. Maybe someone else can better interpret this explanation: http://aws.amazon.com/glacier/faqs/#How_will_I_be_charged_wh...
[+] [-] stevewilhelm|12 years ago|reply
"SmugVault is an added service to your SmugMug account that lets you store almost anything for next to nothing. Including files not normally supported by SmugMug."
http://help.smugmug.com/customer/portal/articles/84562-more-...
[+] [-] sreitshamer|12 years ago|reply
[+] [-] callmeed|12 years ago|reply
Right now, most of them use Drobos or similar devices. You'd be asking them to add steps to their workflow which is always tough to do.
Also, I think your estimate of 18 mins to upload 6+GB is an order of magnitude off. We have tools that allow people to upload JPEGs and most customers claim it takes longer than that just for 100s of MBs.
My (wild) guess is that Adobe is going to get in this space too. It only makes sense to have your Lightroom catalog in the cloud. Maybe they wont offer an affordable solution for terabytes of data, but I'm not sure photogs want their images in two different "clouds".
I think the key is to do customer development from full-time photographers who aren't hackers. I'd be happy to send out a survey to some clients if you want do more research.
[+] [-] bambax|12 years ago|reply
You can't really have the current catalog in Dropbox because then each action and edit become really slow.
But if you set up Lightroom to do a backup every time you quit the program, and that backup is in Dropbox, you get the best of both worlds (strong backups + snappy edits).
Lightroom backups are really "backups", as they are named by date+time, so in Dropbox I have a history of everything I ever did in Lightroom, not just the current (last) version.
It's also fast because I suspect Dropbox is able to do some rsync magic and only re-upload new packets; my Lightroom catalog is around 150 MB and the upload to Dropbox is almost instantaneous (on a pretty lousy upstream bandwidth).
Of course this doesn't address the problem of backing up RAWs, but as far as Lightroom is concerned I feel pretty safe.
[+] [-] jwarren|12 years ago|reply
I think you're totally right about Adobe too. Those Smart Previews they added for LR5 - those are not just for space-starved SSD drives...
[+] [-] beat|12 years ago|reply
[+] [-] short_circut|12 years ago|reply
[+] [-] roc|12 years ago|reply
There really is no good solution to automated offsite backup when it takes several days of uploading to handle a single day's worth of data capture. To make any internet-backup service work, you'd first have to pitch the photog on switching from their residential broadband package to something more expensive -- if that's even available short of their leasing office space somewhere.
I guess you could try to build out a reverse-CDN sort of network, with local relay nodes scattered across the various ISP networks, that might achieve faster uploads from the user (and maybe not get counted against bandwidth caps) which then use a fat pipe to send that data to larger regional storage nodes.
Neither seems particularly plausible at scale.
[+] [-] joebo|12 years ago|reply
[+] [-] Hovertruck|12 years ago|reply
https://plover.co/
We want to take backing up, sharing, and monetizing and make them very simple and enjoyable to do. We've started rolling out slowly already, our goal is to be open to the public within a month or so.
[+] [-] hamburglar|12 years ago|reply
[+] [-] danso|12 years ago|reply
Currently, I have several multi-terabyte drives that I erratically back up to...I just got a 4TB drive that I'm going to try to backup everything I've ever shot in the last 3 years...
But I do want to move it to online storage...and I think this requires triage. For nearly every photoset, I've quickly gone through them with Lightroom and starred the ones that I kind of like, and then for maybe 1% of them, have taken the time to fully process, label, and upload them (in JPG form) to Flickr.
So for online storage, I think what I'll do is write a batch script that dumps all the unstarred RAW files as JPGs, because they chances that I'll ever need these photos in RAW is very slim, and then upload them. For photos that I've given at least one star and/or taken the time to properly edit and categorize, I'll send up the RAW file along with the processed JPG.
If I have about 3TB of RAW files...and maybe 10% of those are RAW images that weren't disposable...that's 300GB right there. And then the remaining RAW "keep em just in case" files would end up being compressed from 24MB to about 2-3 MB (let's say 15% as an average)...so 400GB.
700GB is still quite a big footprint. However, the biggest problem will be...let's say I give up local storage all together and rely on the cloud...what's the best way to browse any photoset at any arbitrary time? When it's all local, it's trivial to pop open Lightroom and do some quick browsing and filtering. For online storage, I'll have to put a ton of work into proper folder-naming, at the very least
[+] [-] PStamatiou|12 years ago|reply
[+] [-] PavlovsCat|12 years ago|reply
1.) Personal: This doesn't mean I don't delete, but I generally tend to keep photos even though they might not be really good photos, especially if I only have a few of that day or person. I know I might enjoy looking at them decades later anyway, a blurry photo is better than nothing.
2.) Archival: like taking a photo of a flat before moving in. These aren't very many, so I tend to keep them no questions asked.
3.) Siblings of Keepers: You could call these "proof I really made the keeper photo" photos. Let's say I come across an animal I never photographed before, and like one of the photos enough to publish; then I'll still keep the others around. Or keeping a wider crop in case you change your mind later, etc.
4.) Keepers: here I try to be as harsh as I can. Photography is about painting with light for the pros, for me it's mostly about selection, that's the part of it I enjoy most. I walk around, and from the thousands of things I see each day, only a few make me take out the camera. Then I select keepers (partly before the images even leave the camera), and even years after I might revisit some keepers again, decide that my standards improved, and delete them.
I don't want to tell others how to pursue their hobby, but I think the web is full of galleries not even the owners look at, and that generally, a lot of people need to learn how to delete. The same goes for blogs and whatnot, the idea that everything has to be written on this indelible roll of toilet paper seems silly.
On one extreme, there is being OCD and deliberating endlessly what to keep and how to rank it, on the other there is just making more stuff to throw behind yourself on a pile you never investigate, because it's too awkward to wade around in. Productivity lies somewhere in the middle I think.
[+] [-] saidajigumi|12 years ago|reply
For example, you can't (and in the broad "you", aren't even qualified to) audit the provider for data-loss or SLA-impacting SPOFs. You just have to assume that they're there. You also take on non-technical failure modes: the provider can go out of business, get bought by an uninterested owner (think Delicious), change focus (Google Reader), or experience myriad other problems.
FWIW, the best success I've had with helping others switch to good backup policy happened once it became economically feasible to buy a new laptop and two bus-powered external drives as big as the laptop's internal storage. The externals become bootable backups, one of which can be preiodically rotated to an offsite location. This has prevented severe dataloss events for myself and others far more times than I care to count now.
[+] [-] famousactress|12 years ago|reply
[+] [-] rdl|12 years ago|reply
(for local storage, once you go beyond individual USB drives, I'd go with a Synology 1813+ NAS ($999 chassis, 8-bay) or an Areca ARC-8050 thunderbolt raid ($1499, 8-bay).)
[+] [-] martin_kivi|12 years ago|reply
With that in mind we've started to look more on how photographers with larger photo volumes could benefit from our service. From the feedback from few photographers we've already included some features (RAW photos with smaller JPG thumbnails, workspaces and personal sharing for example) that will make the day to day workflow a bit easier. Currently our goal is to build something that would fit better for professional photographers (public for clients, ordering prints and possibly integrated payment flows, win/mac client + plugins for your favorite photo software).
Pricing wise we are currently not the cheapest but we have thoughts on how to make this better for high volume photographers. If you're one of them then please send me an email (martin at photry.com) and I would love to talk a bit further what we plan to do and if we would fit your storage needs.
[+] [-] uptown|12 years ago|reply
http://koken.me/
For back-ups - just buy a huge disk or set of disks, and keep 'em off-site wherever you have a trustable location (friend, family, work - if allowed and bandwidth isn't an issue) and use CrashPlan's free software to sync a copy to the secondary location. It's free after the initial expense of the drives and gives you encrypted, but physical access to the content in case of an emergency. You could find a friend with the same problem, and just agree to dedicate a certain amount of disk-space to each-other for this purpose.
[+] [-] fsckin|12 years ago|reply
I'm the technical support for overall workflow, storage & backup engineer, etc, etc. I wear many hats.
Right now I've got a backup on import to a FreeNAS (BSD) filer running on a HP Micro Server. Lightroom backups go to the same filer and are synced as well. My local server rsyncs the delta every night to an identical machine at a friends' house.
This isn't perfect. The one place that isn't 100% backed up is current projects in post-production. That's an acceptable trade off for now.
The most important thing to note is that there are two kinds of photographers: Those who never lost data, and those who care about backups. As mentioned elsewhere in this thread, converting members of Group A into Group B is a tough sell. Video testimonials might be a good option.
A storage service for photographers MUST be invisible to their workflow. Make it as easy to use as an egg timer. At the MOST, an addition of a single screen with sensible defaults (just click next) on photo import.
Ignoring error messages that get in their way is to be expected. Care more about their backups than they do. Send me (tech support guy) an email to notify if there haven't been backups added in X days, along with weekly reports of data usage per day -- how many photos, how large, what root folders, usage trends, etc.
An ideal installation would be automatic detection of what program they're using, what places files are stored (hint: It's not just on a local drive). Detect new file locations and back those up too.
An inexpensive (<$25) option of "only backup current projects I'm working on and anything I shot in the last month", that actually works... I'd sign up in a heartbeat.
[+] [-] ukd1|12 years ago|reply
[+] [-] kevingibbon|12 years ago|reply
[+] [-] antr|12 years ago|reply
[+] [-] PStamatiou|12 years ago|reply
[+] [-] azov|12 years ago|reply
If your files are important to you - you need a back up. If not - just get rid of them in the name of simplicity :)
PS. Also, uploading many gigabytes over a typical ADSL connection is just painfully slow... even local NAS is too slow over WiFi for my taste, and if I have to have a wire - I can just as well connect an external drive.
[+] [-] apaprocki|12 years ago|reply
[+] [-] PStamatiou|12 years ago|reply
[+] [-] dtauzell|12 years ago|reply
[+] [-] throwaway420|12 years ago|reply
But the idea of storing any files in the cloud is very distasteful to me ever since the NSA Snowden story broke. Even something as innocent as a photographer taking pictures of a skyline or some beautiful architecture could land him in hot water if the government is looking at metadata and figuring out that he took lots of photos of a government building or a bridge or a power plant and randomly throwing him on some watch lists or no-fly lists. People have been hassled by the feds for a lot less.
My preference would be for all of these cloud storage providers like Google, Facebook, Apple, Dropbox, Amazon, etc to unambiguously and fiercely challenge the government on this and somehow unambiguously prove to people that they're respecting peoples' privacy and not complying with this stuff. Their denials so far have been obvious non-denials that are very carefully worded around "no direct access" when that's enough wiggle room to do just about anything.
Using some kind of cloud photo storage is a bad idea IMO until this gets sorted out.
[+] [-] qq66|12 years ago|reply
- Local daemon running that manages this process, on PC or Mac. It can't be done solely from the browser.
- The software manages the import from the SD/CF cards, you put them all in, it sorts everything out and wipes the SD cards on the way out.
- You then use Lightroom to do whatever you're going to do. The software would do well to have a Lightroom plugin that lets you select particular images to mark as "high priority" and "low priority."
- Now the daemon uploads all of your RAWs, Lightroom metadata, and corrected JPGs to cloud storage.
- Based on your settings, these RAWs get discarded after some set amount of time (maybe immediately, maybe never), and the highest-quality JPGs are also discarded after a longer amount of time for lower-quality JPGs just in case you need a emergency backup. After 6 months you lose the raws, 1 year the highest quality JPG, 2 years the medium quality JPG, etc. You are charged per gigabyte per month so you can decide how much you care about backup. Basically offload the cognitive effort of discarding old data that isn't worth its upkeep costs anymore.
- The files are available via an API that you can use to build your own gallery/shopping cart site, with the ability to call a certain quality, PROOF watermarking, etc. on-the-fly. But the service offers a default site right there for you to do the same, that's stylable and can be implemented on your own domain.
- You can get your entire catalog shipped to you on a hard drive at any time for a few hundred dollars.
[+] [-] geuis|12 years ago|reply
You shouldn't be touching the directory structure. This tool should be either a plugin for Lightroom/Aperture/Photoshop or an entirely separate application.
I would lean towards separate application. It could be a background agent that detects when new files are added, would work for any known editor or user-defined directory.