Very cool that Backblaze continues to post things like this. Few people have this experience. Of those who do, few (I assume) break it out to this level of detail, actionable for others. Of those who have good experience and records, most would consider it proprietary or just decide not to post. Kudos to Backblaze.
We're all on this crazy spinning ball together. Early on Backblaze made the decision to stay out of the hardware business (it sounded lucrative after we developed the v1 storage pod, but everyone was a software person, so we went that route), we like to pay our hardware findings forward. It's a good thing to do!
I suppose it has an indirect benefits to Backblaze too. For a start the publishing of this useful information brings them to our attention (or refreshed our existing memory of the existence of their services) in a positive manner, and publishing honest information drive reliability might make the manufacturers case that little bit more about the overall quality of their consumer grade hardware (which means we all, including Backblaze, get better products to use).
This is also excellent PR and attention acquisition. I've referred backblaze as a possible option to look into for friends looking for backup solutions on more than one occasion just due to being reminded about their existence from posts like this (and I've never used them personally).
> The drives that just don’t work in our environment are Western Digital Green 3TB drives and Seagate LP (low power) 2TB drives. Both of these drives start accumulating errors as soon as they are put into production. We think this is related to vibration. The drives do somewhat better in the new low-vibration Backblaze Storage Pod, but still not well enough.
Another reason to avoid the WD Green 3TB: these drives aggressively put themselves to sleep to save power. It's literally a matter of streaming a video from disk and if the OS caches enough of the file, the drive will see there haven't been any accesses in a few seconds and stop spinning.
The video will of course glitch when the cached data runs out and the drive needs to spin up. Great design.
You can disable the aggressive parking behavior ("Intellipark") with WD's own wdidle3.exe [1] or idle3-tools [2].
Disabling Intellipark was literally the first thing I did with each of the three WD Green drives I've owned since 2011 (got two but one failed early on and was replaced), so I can't really compare their performance with this setting on and off; however, I can say that I haven't noticed the drives being parked more aggressively than similar Samsung or Seagate drives. I used the official wdidle3.exe under FreeDOS for each drive.
Don't use the Green line for anything performance related, even if it's watching videos. The Green line is simply not intended for that. I think it's designed to be an occasionally-accessed backup drive, or something like that.
The Black line is the performance-conscious line, look at the corresponding equivalents in that line. IIRC the Blacks are 10-15% more expensive, but they actually respond.
I have streamed hundreds of hours of high bitrate video (raw Blu-rays) from WD Green 3TB drives over my home gigabit network, and have never encountered this behavior. I couldn't be more pleased with the drives.
This has not been my experience. We run a bunch of GP drives, 1TB, 2TB, and 4TB, and I have not experienced them spinning down ever unless explicitly commanded to. They park heads for sure, but don't spin down. (These are normal GP, not AV-GP, never tried one of those.)
is this the "head parking" thing? The Reds are supposedly the same drive, but don't have the head parking algorithm, with tweak able TLER to make them more RAID friendly, and their rates are pretty acceptable.
I've never had good luck with WD drives, from the 1990s through the present. At first, I had no luck with Seagates, then they seemed to do pretty well up until recently, where now I'm seeing some radically bad results. I wish IBM would get back in the HD business.
From the article and throughout the comments here it seems Backblaze prefers cheaper drives over a few percentage points of reliability. It would be interesting to see some data showing the tradeoff, but I suspect it reveals too much of their operation. At first glance it appears you can get a drive with .9% failure rate (HGST 7K3000) for $127[1], and yet BB really likes the WD Red, which has a higher failure rate (3.2%) and cost[2].
What might shed light without revealing too much is information about where they source drives today (their sourcing coverage during the shortage was very cool!). I suspect they're finding some nice bulk discounts somewhere.
> What might shed light without revealing too much is information about where they source drives today
Backblaze employee here -> we are willing to buy from anybody, we have no loyalty. Lowest price (for a particular drive model) always wins. Once per month we ask about 20 common suppliers for their "best price". We have bought from "B&H Photo Video", NewEgg, Amazon, etc among others. We're always willing to add more possible vendors, but I think we drop you from the list if the vendor bid prices don't even come close for 3 months - that means you don't understand anything and you're wasting our time.
If systems are designed with the expectation that hardware can and will fail often, better reliability drives aren't worth the cost as long as cheaper drives are relatively comparable. In addition to cost savings, your system has better robustness when it is decoupled from hardware reliability.
For example, Google's Map Reduce paper has a section on fault tolerance that goes into detail about how they handle the issue of failing workers:
The price we pay as consumers isn't going to be the same as that paid by Backblaze when they buy 100 drives at a time. The fact that they pick the WD Red implies that they're getting a better deal on those, otherwise they wouldn't settle for the higher failure rates.
EDIT: comments from BB employees below actually say they often buy off the shelf from NewEgg and Amazon.
Definitely wish I had seen this a couple months ago before I bought two 3TB seagates. Although to be fair I was already pretty sure that seagates sucked (its good to see data backing that up) but getting two for $85 each was too hard to pass up. I'm a sucker for a deal. I buy HDs in pairs now so I'm not too worried about losing anything.
I am intrigued by backblaze's service though. A part of me feels like there must be a catch somewhere. I have a good 10TB I'd be happy to pay $5/month to backup but somehow I feel like they'd pull a comcast and say their "unlimited" claim doesn't apply to the 1% of users (or in this case maybe the .001%).
As a user with almost six terabytes of data, I can vouch for their service. I have never received any messages asking me to limit the amount of data I upload. I do try to compensate, for the sake of my conscience, by recommending the service to people (with much less data) who previously lacked a backup-strategy.
The catch is that the initial upload can take a while if your internet connection has limited upstream – not something Backblaze can help but I'm sure it has made some people think twice before having their external drives mirrored in the cloud.
Do you buy the same drive model when buying in pairs? I've anecdotally had 2 identical drives die of the same type (manufacturing batch) at the same time.
Any stats on power consumption? Over 5 years the difference between a drive that uses 6 Watts and one that uses 7 is 44kWh or about $5. Double that to include cooling costs and saving a Watt should be worth something like $10 to you, so a more expensive more efficient drive could be worth it. Do these drives all use similar amounts of power?
I wonder if any of this actually applies to consumer-grade drives.
My wife's hard drive actually just died. It was 160 GB WD in a black 2006 MacBook. The drive itself was a replacement from 2007 since the original drive died just over a year into its life.
Stupidly, since her Time Machine backup was misbehaving, I reformatted it and set it to start over. I spent the weekend recovering her data—with a lot of success, so no big deal. At any rate, this machine is long past its expiration date. It's time for a MacBook Air with an SSD, once the tax refund comes in.
Yev from Backblaze here -> All of our drives are consumer-grade. We try to avoid buying enterprise drives at all costs. These are all off-the-shelf internals and in some cases...externals that were made internal! :)
Any laptop with a mechanical hard drive makes my hair stand on its end. In 2014, it's just an unnecessary liability.
When my girlfriend and I started getting serious a few years back, the first thing I did was to replace her aging MacBook Pro with a SSD MacBook Air. It was the only way the relationship could move forward.
"If the price were right, we would be buying nothing but Hitachi drives."
I don't understand why they don't. Are the Hitachi drives really that much more expensive so that it doesn't justify their vastly longer lifespan? Even if they can get "free" replacement disks during the warranty period, that has a cost for them. And they mentioned that some replacement disks die even faster.
I'm sure Backblaze has crunched all these numbers - would love to see them. BTW thanks for sharing this data!
Looking at PCPartPicker (no drive shucking) Hitachi drives are at least $.02 more per GB than similar (7200RPM >= 1TB) WD and Seagate drives. Which means a difference of ~$20 per TB of storage, which at 1 pod (180TB) every few weeks means they're saving something on the order of 3.5k every few weeks using the cheaper drives. How much the additional failure rate costs them would be wild speculation, between RMAs/warranties and labor there are lots of assumptions to make, so I'll stop there.
eg:
Hitachi 0F10311 Deskstar 7K2000 (7200RPM) : $0.059/GB
Western Digital WD30EFRX RED (5400RPM) : $.044/GB
Seagate ST3000DM001 (7200RPM) : $.034/GB
Backblaze employee here - it is honestly just a spreadsheet that kicks out the answer. Every month we ask 20 or so suppliers for the lowest price for each drive type. If Hitachi are 10 percent more expensive but fail 10 percent less often, that balances out and we buy Hitachi. But if it is 12 percent more costly then we get the other brand. There is a tiny bit of free preference leeway given to Hitachi because it means less hassle to our over worked datacenter team...
The cost you see on newegg probably isn't the cost Blackblaze sees. They're getting bulk pricing, which WD is probably more eager to sell than Hitachi.
> We are focusing on 4TB drives for new pods. For these, our current favorite is the Seagate Desktop HDD.15 (ST4000DM000). We’ll have to keep an eye on them, though. Historically, Seagate drives have performed well at first, and then had higher failure rates later.
I'm a little surprised that they actually did the analysis to determine the Seagates tend to fail more, yet they are still putting most (or at least, quite a bit) of their faith in those.
Based on their own data, I would likely avoid those, or at least start leaning more toward Hitachi and WD.
Or maybe the initial cost of those is so much better that it compensates for any long-term expense.
You also have to consider that when the drives eventually fail, they will be replaced with hard drives of the future -- which will presumably be cheaper than the HDD of today. I.e. they depreciate quickly.
Interesting. I had always avoided Hitachi Deskstars after having heard they were nicknamed "Deathstars" for a reason. Perhaps that was once true, but clearly it's not anymore.
A better presentation of this data would show a failure rate for each brand and month/year of purchase.
For extremely simple devices like resistors or incandescent light bulbs, failure rate is relatively constant over the lifetime of device -- the chance of a functioning resistor with 10 hours of use failing during the next hour is the same as the chance of a functioning resistor with 1000 hours of use.
For complex devices with lots of interdependent parts, some of which are mechanical, the failure rate changes over time. There's an "infant mortality" or "lemon" phenomenon, where relatively new devices have higher defect rates (because fabrication and shipping sometimes result in imperfections which quickly cause failures), followed by a steep dropoff in failure rates (because observing a device operate correctly for dozens of hours is strong evidence that it doesn't suffer from a failure mode which often results in infant mortality).
Then there may be an increase in failure rates later, especially with devices that are partially or wholly mechanical (wear or damage type problems which do not cause immediate failure, but make it easier for a failure to occur).
You need empirical data to be quantitative about this curve, and it sounds like Backblaze has it, but their presentation in this article doesn't show it.
As I recall one of the studies of a few years ago, the one based on supercomputers, not Google's, showed there was very little infant mortality, and wear clearly set in after roughly one year in service. The results were quite striking, and nothing like the bathtub curve many expected and that you sort of sketch out.
This is a pretty textbook perfect application of survival/time-to-event analysis. Any chance the data behind it could be made available for teaching purposes?
Yev from Backblaze -> Where/what do you teach? We're unsure about releasing any more information at this time, but we're not opposed to it. What would it be used for?
Interesting to see that kind of a difference between hitachi and western digital, given that WD owns HGST. Are hitachi drives marketed as higher reliability drives, or was the acquisition by WD simply too recent for the quality of the two brands to "equalize"?
Backblaze's data shows that the number of errors is largely related to the age of the drive. Because the older drives are from before Hitachi was acquired by WD, it is going to take a few more years for the brands to equalize if they do combine the manufacturing of both lines.
Great post, as you get bigger populations of drives you can get a lot more visibility into their overall reliability. If there was one thing I could add to the analysis would be to split out the drives by serial number and split them out by firmware. Sometimes you find that all of the 'problem' in a set of problem drives is a single range of serial numbers.
We've had similar experiences with replacement drives, they are, by and large, significantly less reliable than "new" drives.
And last bit, we've got Western Digital drives here (a mix of 2.0 and 3.0 TB ones) They have been pretty solid performers for us.
Aside from the incredible usefulness of the data herein -- thanks, backblaze! -- this is also the kind of marketing that I don't mind.
Backblaze got egg on its face yesterday on HN when someone's critical report (rightly) got upvoted. Today they make up for it by giving us an interesting and useful data chart.
I know it sounds weird, but for whatever reason I read this as demonstrating a high level of corporate responsibility and attunement to customers. They could have compensated by instead, say, dropping a few grand on buying journalists and 'reviewers', like Microsoft does. But they didn't. Instead, they were cool. To me that signals that they'll also take care of whatever problems they've had recently. (Note: I have no affiliation with Backblaze.)
I'm currently not shopping for an online backup service but if that ever changes, I now have a good feeling about Backblaze, and I hope that other services take a similar approach to repairing customer relations when they are fraught.
Shit happens, even to backup providers. It's how you respond that matters most.
I have one in my rig, and whenever I do any disk access, I always have to wait about 5-8 seconds for it to spin up every other half hour. It seems to aggressively turn off. I have my base system on an SSD, and my games and other things on this 3TB drive.
I guess such spin up times would be unacceptable in backblaze's environment.
[+] [-] brokentone|12 years ago|reply
[+] [-] atYevP|12 years ago|reply
[+] [-] dspillett|12 years ago|reply
[+] [-] adventured|12 years ago|reply
[+] [-] justin66|12 years ago|reply
Another reason to avoid the WD Green 3TB: these drives aggressively put themselves to sleep to save power. It's literally a matter of streaming a video from disk and if the OS caches enough of the file, the drive will see there haven't been any accesses in a few seconds and stop spinning.
The video will of course glitch when the cached data runs out and the drive needs to spin up. Great design.
[+] [-] networked|12 years ago|reply
Disabling Intellipark was literally the first thing I did with each of the three WD Green drives I've owned since 2011 (got two but one failed early on and was replaced), so I can't really compare their performance with this setting on and off; however, I can say that I haven't noticed the drives being parked more aggressively than similar Samsung or Seagate drives. I used the official wdidle3.exe under FreeDOS for each drive.
[1] http://support.wdc.com/product/download.asp?groupid=609&sid=...
[2] http://idle3-tools.sourceforge.net/
Edit: changed "aggressive spin-down behavior" to "aggressive parking behavior". It actually isn't quite clear how disabling Intellipark affects disk spin-down behavior.
Edit 2: There's also wdantiparkd (http://www.sagaforce.com/sound/wdantiparkd/). I have not used it but it might help you if tuning the drive itself doesn't work.
[+] [-] B-Con|12 years ago|reply
The Black line is the performance-conscious line, look at the corresponding equivalents in that line. IIRC the Blacks are 10-15% more expensive, but they actually respond.
[+] [-] baddox|12 years ago|reply
[+] [-] lutorm|12 years ago|reply
[+] [-] caycep|12 years ago|reply
[+] [-] fuckpig|12 years ago|reply
[+] [-] deltaqueue|12 years ago|reply
What might shed light without revealing too much is information about where they source drives today (their sourcing coverage during the shortage was very cool!). I suspect they're finding some nice bulk discounts somewhere.
[1] http://www.amazon.com/Hitachi-Deskstar-7K3000-HDS723030ALA64... [2] http://www.amazon.com/WD-Red-NAS-Hard-Drive/dp/B008JJLW4M/ (both seem to be market consumer prices)
[+] [-] brianwski|12 years ago|reply
Backblaze employee here -> we are willing to buy from anybody, we have no loyalty. Lowest price (for a particular drive model) always wins. Once per month we ask about 20 common suppliers for their "best price". We have bought from "B&H Photo Video", NewEgg, Amazon, etc among others. We're always willing to add more possible vendors, but I think we drop you from the list if the vendor bid prices don't even come close for 3 months - that means you don't understand anything and you're wasting our time.
[+] [-] wting|12 years ago|reply
For example, Google's Map Reduce paper has a section on fault tolerance that goes into detail about how they handle the issue of failing workers:
http://research.google.com/archive/mapreduce.html
[+] [-] thinkling|12 years ago|reply
EDIT: comments from BB employees below actually say they often buy off the shelf from NewEgg and Amazon.
[+] [-] tacticus|12 years ago|reply
[+] [-] hackinthebochs|12 years ago|reply
I am intrigued by backblaze's service though. A part of me feels like there must be a catch somewhere. I have a good 10TB I'd be happy to pay $5/month to backup but somehow I feel like they'd pull a comcast and say their "unlimited" claim doesn't apply to the 1% of users (or in this case maybe the .001%).
[+] [-] Zirro|12 years ago|reply
http://cl.ly/image/0I0S1M0V0F3t
[+] [-] stdbrouw|12 years ago|reply
[+] [-] rajivm|12 years ago|reply
[+] [-] cbr|12 years ago|reply
[+] [-] ars|12 years ago|reply
[+] [-] freshyill|12 years ago|reply
My wife's hard drive actually just died. It was 160 GB WD in a black 2006 MacBook. The drive itself was a replacement from 2007 since the original drive died just over a year into its life.
Stupidly, since her Time Machine backup was misbehaving, I reformatted it and set it to start over. I spent the weekend recovering her data—with a lot of success, so no big deal. At any rate, this machine is long past its expiration date. It's time for a MacBook Air with an SSD, once the tax refund comes in.
[+] [-] atYevP|12 years ago|reply
[+] [-] GuiA|12 years ago|reply
When my girlfriend and I started getting serious a few years back, the first thing I did was to replace her aging MacBook Pro with a SSD MacBook Air. It was the only way the relationship could move forward.
[+] [-] olov|12 years ago|reply
I don't understand why they don't. Are the Hitachi drives really that much more expensive so that it doesn't justify their vastly longer lifespan? Even if they can get "free" replacement disks during the warranty period, that has a cost for them. And they mentioned that some replacement disks die even faster.
I'm sure Backblaze has crunched all these numbers - would love to see them. BTW thanks for sharing this data!
[+] [-] rtkwe|12 years ago|reply
eg:
http://pcpartpicker.com/parts/internal-hard-drive/#m=19,34,3...[+] [-] brianwski|12 years ago|reply
[+] [-] joeframbach|12 years ago|reply
[+] [-] mustafab|12 years ago|reply
[+] [-] Nickoladze|12 years ago|reply
[+] [-] acd|12 years ago|reply
French hardware site, component failure rates. Google translate it to english http://www.hardware.fr/articles/911-6/disques-durs.html
So its also important to take hard drive models into account.
Then there is the Google study Failure trends in large hard drive population http://static.googleusercontent.com/media/research.google.co...
[+] [-] jader201|12 years ago|reply
I'm a little surprised that they actually did the analysis to determine the Seagates tend to fail more, yet they are still putting most (or at least, quite a bit) of their faith in those.
Based on their own data, I would likely avoid those, or at least start leaning more toward Hitachi and WD.
Or maybe the initial cost of those is so much better that it compensates for any long-term expense.
[+] [-] gatehouse|12 years ago|reply
[+] [-] dnissley|12 years ago|reply
[+] [-] michaelt|12 years ago|reply
I guess this is why companies sometimes decide to rebrand/rename products.
[1] https://en.wikipedia.org/wiki/HGST_Deskstar#IBM_Deskstar_GXP...
[+] [-] csense|12 years ago|reply
For extremely simple devices like resistors or incandescent light bulbs, failure rate is relatively constant over the lifetime of device -- the chance of a functioning resistor with 10 hours of use failing during the next hour is the same as the chance of a functioning resistor with 1000 hours of use.
For complex devices with lots of interdependent parts, some of which are mechanical, the failure rate changes over time. There's an "infant mortality" or "lemon" phenomenon, where relatively new devices have higher defect rates (because fabrication and shipping sometimes result in imperfections which quickly cause failures), followed by a steep dropoff in failure rates (because observing a device operate correctly for dozens of hours is strong evidence that it doesn't suffer from a failure mode which often results in infant mortality).
Then there may be an increase in failure rates later, especially with devices that are partially or wholly mechanical (wear or damage type problems which do not cause immediate failure, but make it easier for a failure to occur).
You need empirical data to be quantitative about this curve, and it sounds like Backblaze has it, but their presentation in this article doesn't show it.
[+] [-] hga|12 years ago|reply
[+] [-] Fomite|12 years ago|reply
[+] [-] atYevP|12 years ago|reply
[+] [-] rythie|12 years ago|reply
[+] [-] atYevP|12 years ago|reply
1. How much data do you have?
2. What would you use it for?
3. How important is it if the API isn't the same as the S3 API?
4. Any specific certification requirements?
5. Would you have any SLA requirements?
6. Specific performance metrics? Think Amazon S3 vs. Glacier.
7. Are redundant data centers important?
Hypothetically!
[+] [-] staticshock|12 years ago|reply
[+] [-] davis_m|12 years ago|reply
[+] [-] DanBC|12 years ago|reply
[+] [-] yen223|12 years ago|reply
[+] [-] ChuckMcM|12 years ago|reply
We've had similar experiences with replacement drives, they are, by and large, significantly less reliable than "new" drives.
And last bit, we've got Western Digital drives here (a mix of 2.0 and 3.0 TB ones) They have been pretty solid performers for us.
[+] [-] polskibus|12 years ago|reply
Moreover, it seems that Deskstars are no longer manufactured (or have been rebranded). http://www.hgst.com/hard-drives/product-brands
[+] [-] undoware|12 years ago|reply
Backblaze got egg on its face yesterday on HN when someone's critical report (rightly) got upvoted. Today they make up for it by giving us an interesting and useful data chart.
I know it sounds weird, but for whatever reason I read this as demonstrating a high level of corporate responsibility and attunement to customers. They could have compensated by instead, say, dropping a few grand on buying journalists and 'reviewers', like Microsoft does. But they didn't. Instead, they were cool. To me that signals that they'll also take care of whatever problems they've had recently. (Note: I have no affiliation with Backblaze.)
I'm currently not shopping for an online backup service but if that ever changes, I now have a good feeling about Backblaze, and I hope that other services take a similar approach to repairing customer relations when they are fraught.
Shit happens, even to backup providers. It's how you respond that matters most.
[+] [-] cordite|12 years ago|reply
I have one in my rig, and whenever I do any disk access, I always have to wait about 5-8 seconds for it to spin up every other half hour. It seems to aggressively turn off. I have my base system on an SSD, and my games and other things on this 3TB drive.
I guess such spin up times would be unacceptable in backblaze's environment.