They got rid of the PCI bottleneck by switching to PCIe, a bottleneck which surprised me when they designed version 1.0 of their pod. They could have gone PCIe at the time, I maintain http://blog.zorinaq.com/?e=10 and they were SATA controllers at the time that met their technical requirements (nr. of ports, Linux support, etc).
FYI: Protocase (http://www.protocase.com/), the company that will build you the Backblaze case, sent me an email yesterday announcing that they are now selling completed Backblase storage pods without the hard drives for $5395.00. I'm not sure if this is the new design or not but I've set up 3 of these and finding some of the parts (SATA back planes) took weeks of searching and shady dealings.
Running LVM over raid 6 volumes is the 'standard' approach of many enterprise storage deployments. The 'magic' in a good RAID6 implementation is what it does when things go wrong, and lots of things do go wrong. The checksumming is great too.
At some point you can dump the sheet metal on the 'pods', just build your own rack unit. If you look at the 'big iron' systems from NetApp, EMC, and others you will see they make one big enclosure then can install simpler systems within that enclosure. What this buys you is that you can distribute enclosure services into the big box and take it off the individual boxes. That gives you better cost efficiencies. And as you point out your system can run with fewer fans, so you can put a couple of largish three phase fans in the big enclosure (dead simple and very reliable) and use them to push the air through all the other boxes. (or pull it). Then add an android tablet for the 'door' of the enclosure that tells you drive status, etc and you're practically a player in the storage game :-)
Does anyone know if Backblaze will ever support Linux? I've wanted to use their service for a while now, but their lack of Linux support has being a big turn down, and I don't think they have made any change in their statements regarding this 'issue'.
We would like to, we just haven't had time to get it done yet. It runs internally, but is lacking an installer and a GUI, and we would need to prioritize and choose one or more Linux distributions to launch with. Ubuntu is an obvious choice (we focus more on desktop backup than on servers). But some people also ask for CentOS and a few others. It bums me out the Linux community has not solved binary compatibility anywhere NEAR the same level that Microsoft or Apple has, and few in the Linux community seem interested in solving this issue which massively, MASSIVELY hinders development and deployment, but that is a side tangent...
Explanation about the "GUI" comment above -> the Backblaze backup client was simultaneously written from the ground up compiling on Mac OS, Windows, and Linux. The same tree and the same source compiles on all three on EVERY SVN CHECKIN. There is one exception, which is the GUI is an extremely simple stand alone process entirely natively written to match the host OS. On Mac OS it is in Objective C in the beautiful Apple GUI layout editor, on Windows we use Visual Studio and C++ and Win32. The firm rule is these GUIs are ONLY allowed to edit one or two simple XML files, and all the real encryption, compression, transmission is done by other cross platform processes. On Linux we configure the XML file with "vi". :-) The X-Windows GUI has not even been started.
The diagram of the Cost of a Petabyte is very interesting if it is true. It demonstrates the profitability of some SaaS models for selling what is a straight commodity.
However it seems in contradiction to the AWS, Rackspace model, which is a race to the bottom in that there are many competitors and they are selling a commodity (independent of the other high value services they are selling). There is some threshold of volume that is key in order to make money in that space.
Backblaze should really consider selling their pods - it would basically free money for them because whatever inventory they don't sell they can use for their service, and I'm sure there's lots of businesses that would pay good money for a cheaper alternative to storage servers from Dell or HP.
It'd be a major distraction, though. Setting up a separate support/services organization for that is not trivial.
Also, most businesses buy from the Dells or the HPs because they don't have the in-house expertise to manage a more bare-bones box (or more likely just don't want to). The companies that have the need/capability to manage more raw storage could just take the plans and build them out anyways I would imagine.
The blog post provides some fascinating data, thanks Backblaze!
$2100 per month for an entire rack worth of Pods (space, power, connectivity)
$74,000 is the cost to build 10 Pods to fill that rack.
If the Pods are assumed to have a lifetime of 3 years (most will last longer, but lets depreciate at this rate), and if the cost of capital is 20%/year, this equates to a monthly "payment/amortization" cost of $2750. Thus leading to a total cost per rack of $4850 ~ lets say $5000.
1350TB of raw storage is provided by this rack, which can be scaled to 13/15 to account for RAID6 (as revealed by brianwski here) - thus leaving 1170TB available for use. FS overhead etc, lets take this to provide 1PB of storage.
So, essentially, storage costs backblaze about 0.5 cents ($0.005) /GB-month. There are other costs ofcourse, Sean (amongst others) needs to get paid, etc. To go by a common thumb rule, for a minimum sale price, one third of sale price should be profit, another third should be org expenses/marketing/everything else and the remaining third should be the actual cash cost of the building/providing the product/service.
So very roughly, Backblaze could provide storage at about 1.5 cents /GB-month. Factor in 3-way software-level redundancy of data, and you are now upto 5 cents/GB-months for a very high quality storage service.
Contrasting this with 15c/GB-month that Amazon charges (in addition to transfer charges), I do have to wonder why Backblaze wants to stick to the "unlimited desktop backups" business. Even Google storage charges 17c/GB-month, in addition to per-request charges.
Its quite possible I have some factors wrong here and If anyone can spot anything wrong, I'd like to know. Nevertheless, it seems from the numbers provided that backblaze could make a killing in this market. I know I would be interested in using a pure storage backend - equivalent to S3. I use tarsnap and if tarsnap could reduce its backend costs by two-thirds, I know I'd be very happy.
What am I missing?
edit: wrote PB instead of TB. Numbers remain correct, though
I switched from JungleDisk to Backblaze a year ago and haven't looked back. BB is amazingly fast and painless to use. On top of that the cost is insanely cheap in comparison to using AWS (which Jungle Disk uses).
I am too a very happy switcher from Carbonite to Backblaze. It really is fast, as Valien says. The cost? Well, it has gotten cheaper since the dollah is dying.
It's 135TB worth of drives, but with RAID don't you see a far less useable amount?
Also, considering saving money on hardware costs is a key factor in Backblaze staying competitive, they must be saving money elsewhere and/or have other competitive advantages. Otherwise releasing this information would be akin to publishing a restaurant's 'secret sauce'.
Anybody who builds their own pod is welcome to use RAID or not, and which RAID you choose will affect your final numbers. At Backblaze, we configure it as RAID 6 groups, each group is 15 drives which includes 2 parity drives. So you are down to 13/15 = 86.67 percent of the raw unformatted space BEFORE the overhead of ext4 which adds a little tiny bit extra. So after formatting in our datacenter we are left with about 116 TBytes of storage space for customer files. On the other hand, we use lossless compression on the customer's files before transmitting to the pod, so it can sometimes appear like we are fitting more than 116 TBytes of customer data on a pod, if that makes sense.
I'd say any competitor worth worrying about would be able to build an equivalent system from scratch pretty easily, so I don't think they're risking that much. Convincing users that you'll be trustworthy, secure and around in 5-10 years is a much greater hurdle.
On the other hand, I read this article and I read their last one, when it too shot to the top of HN. Geeks go nuts for hardware porn, and are a great audience to sell these kinds of services to. So the marketing benefits are probably quite substantial.
Yikes, what? We all run Lion here at Backblaze and it works flawlessly for us, and our internal stats show THOUSANDS of Lion customers happily backing up. If you can provide more details of your setup we would LOVE to fix that for you. Since Mac customers make up more than half our customer base, we're completely, 100 percent dedicated to Lion. Send us info or contact us at our company name at twitter or facebook or use the help links off the homepage.
As a side note -> Backblaze only does one thing -> HTTPS POST. In user space. We do NOT extend the kernel, we do not have drivers (at all), we simply read files (we don't even write to your files), compress and encrypt them in RAM, and push them through the completely common HTTPS. It is unusual for Backblaze to cause any problems except a sluggish network or a hiccup in a skype phone call.
At the lowest level there are three RAID groups in each pod. Each RAID group is made of 15 drives configured in software RAID 6 with 2 parity drives. This means you can lose 2 dives and the data is entirely safe and intact. If 3 or more drives completely fail simultaneously (not just pop out of the RAID group or power down, but where that drive is lost forever, like it will never power up again) you will lose at least some of the data on that RAID group. Layered on top of the 15 drive RAID group is LVM. The specifics of the LVM config are there is one PV (Physical Volume) spanning the 15 drives, then on top of that are one VG (Volume Group) spanning the same exact 15 drives. Then on top of that are as many LV (Logical Volumes) as it takes to keep each logical volume under the ext4 limit of 16 TB. With 3 TB Hitachi drives, there are 3 separate LV on top of the same exact 15 drives. Finally, there is one ext4 file system sitting on top of each of the LV (one to one with the LV). Disclaimer: I work at Backblaze, but datacenter and pods aren't my main area of focus.
The ECC RAM absolutely does find and corrects problems (we see them in the logs). However, just to be absolutely clear we would not need ECC RAM -> Backblaze checksums EVERYTHING on an end-to-end basis (mostly we use SHA-1). This is so important I cannot stress this highly enough, each and every file and portion of file we store has our own checksum on the end, and we use this all over the place. For example, we pass over the data every week or so reading it, recalculating the checksums, and if a single bit has been thrown we heal it up either from our own copies of the data or ask the client to re-transmit that file or part of that file.
At the large amount of data we store, our checksums catch errors at EVERY level - RAM, hard drive, network transmission, everywhere. I suppose consumers just do not notice when a single bit in one of their JPEG photos has been flipped -> one pixel gets every so slightly more red or something. Only one photo changes out of their collection of thousands. But at our crazy numbers of files stored we see it (and fix it) daily.
[+] [-] mrb|14 years ago|reply
They got rid of the PCI bottleneck by switching to PCIe, a bottleneck which surprised me when they designed version 1.0 of their pod. They could have gone PCIe at the time, I maintain http://blog.zorinaq.com/?e=10 and they were SATA controllers at the time that met their technical requirements (nr. of ports, Linux support, etc).
[+] [-] SoftwareMaven|14 years ago|reply
"In the first generation storage pod, we ran out of the faster PCIe slots and had to use one slower PCI slot, creating a bottleneck."
[+] [-] cpt_yesterday|14 years ago|reply
[+] [-] SoftwareMaven|14 years ago|reply
[+] [-] mother|14 years ago|reply
Feels similar to what the internet archive does.
[+] [-] aw3c2|14 years ago|reply
[+] [-] ChuckMcM|14 years ago|reply
Running LVM over raid 6 volumes is the 'standard' approach of many enterprise storage deployments. The 'magic' in a good RAID6 implementation is what it does when things go wrong, and lots of things do go wrong. The checksumming is great too.
At some point you can dump the sheet metal on the 'pods', just build your own rack unit. If you look at the 'big iron' systems from NetApp, EMC, and others you will see they make one big enclosure then can install simpler systems within that enclosure. What this buys you is that you can distribute enclosure services into the big box and take it off the individual boxes. That gives you better cost efficiencies. And as you point out your system can run with fewer fans, so you can put a couple of largish three phase fans in the big enclosure (dead simple and very reliable) and use them to push the air through all the other boxes. (or pull it). Then add an android tablet for the 'door' of the enclosure that tells you drive status, etc and you're practically a player in the storage game :-)
[+] [-] alfet|14 years ago|reply
[+] [-] brianwski|14 years ago|reply
Explanation about the "GUI" comment above -> the Backblaze backup client was simultaneously written from the ground up compiling on Mac OS, Windows, and Linux. The same tree and the same source compiles on all three on EVERY SVN CHECKIN. There is one exception, which is the GUI is an extremely simple stand alone process entirely natively written to match the host OS. On Mac OS it is in Objective C in the beautiful Apple GUI layout editor, on Windows we use Visual Studio and C++ and Win32. The firm rule is these GUIs are ONLY allowed to edit one or two simple XML files, and all the real encryption, compression, transmission is done by other cross platform processes. On Linux we configure the XML file with "vi". :-) The X-Windows GUI has not even been started.
[+] [-] ck2|14 years ago|reply
Recently was as low as $107 from amazon in retail package:
http://camelcamelcamel.com/Hitachi-Deskstar-CoolSpin-Interna...
[+] [-] neelm|14 years ago|reply
However it seems in contradiction to the AWS, Rackspace model, which is a race to the bottom in that there are many competitors and they are selling a commodity (independent of the other high value services they are selling). There is some threshold of volume that is key in order to make money in that space.
[+] [-] kylec|14 years ago|reply
[+] [-] SystemOut|14 years ago|reply
Also, most businesses buy from the Dells or the HPs because they don't have the in-house expertise to manage a more bare-bones box (or more likely just don't want to). The companies that have the need/capability to manage more raw storage could just take the plans and build them out anyways I would imagine.
[+] [-] ghoul2|14 years ago|reply
$2100 per month for an entire rack worth of Pods (space, power, connectivity) $74,000 is the cost to build 10 Pods to fill that rack. If the Pods are assumed to have a lifetime of 3 years (most will last longer, but lets depreciate at this rate), and if the cost of capital is 20%/year, this equates to a monthly "payment/amortization" cost of $2750. Thus leading to a total cost per rack of $4850 ~ lets say $5000.
1350TB of raw storage is provided by this rack, which can be scaled to 13/15 to account for RAID6 (as revealed by brianwski here) - thus leaving 1170TB available for use. FS overhead etc, lets take this to provide 1PB of storage.
So, essentially, storage costs backblaze about 0.5 cents ($0.005) /GB-month. There are other costs ofcourse, Sean (amongst others) needs to get paid, etc. To go by a common thumb rule, for a minimum sale price, one third of sale price should be profit, another third should be org expenses/marketing/everything else and the remaining third should be the actual cash cost of the building/providing the product/service.
So very roughly, Backblaze could provide storage at about 1.5 cents /GB-month. Factor in 3-way software-level redundancy of data, and you are now upto 5 cents/GB-months for a very high quality storage service.
Contrasting this with 15c/GB-month that Amazon charges (in addition to transfer charges), I do have to wonder why Backblaze wants to stick to the "unlimited desktop backups" business. Even Google storage charges 17c/GB-month, in addition to per-request charges.
Its quite possible I have some factors wrong here and If anyone can spot anything wrong, I'd like to know. Nevertheless, it seems from the numbers provided that backblaze could make a killing in this market. I know I would be interested in using a pure storage backend - equivalent to S3. I use tarsnap and if tarsnap could reduce its backend costs by two-thirds, I know I'd be very happy.
What am I missing?
edit: wrote PB instead of TB. Numbers remain correct, though
[+] [-] ghshephard|14 years ago|reply
http://phx.corporate-ir.net/phoenix.zhtml?c=176060&p=iro...
[+] [-] wmf|14 years ago|reply
[+] [-] Valien|14 years ago|reply
[+] [-] unkoman|14 years ago|reply
[+] [-] rdoherty|14 years ago|reply
Also, considering saving money on hardware costs is a key factor in Backblaze staying competitive, they must be saving money elsewhere and/or have other competitive advantages. Otherwise releasing this information would be akin to publishing a restaurant's 'secret sauce'.
[+] [-] brianwski|14 years ago|reply
[+] [-] reitzensteinm|14 years ago|reply
On the other hand, I read this article and I read their last one, when it too shot to the top of HN. Geeks go nuts for hardware porn, and are a great audience to sell these kinds of services to. So the marketing benefits are probably quite substantial.
[+] [-] ComputerGuru|14 years ago|reply
[+] [-] brianwski|14 years ago|reply
As a side note -> Backblaze only does one thing -> HTTPS POST. In user space. We do NOT extend the kernel, we do not have drivers (at all), we simply read files (we don't even write to your files), compress and encrypt them in RAM, and push them through the completely common HTTPS. It is unusual for Backblaze to cause any problems except a sluggish network or a hiccup in a skype phone call.
[+] [-] mrbill|14 years ago|reply
[+] [-] bgentry|14 years ago|reply
--edit--
I know it's RAID6 from the article. What I'm wondering is, how many drives in an array? How many arrays per box?
[+] [-] brianwski|14 years ago|reply
[+] [-] unknown|14 years ago|reply
[deleted]
[+] [-] jbooth|14 years ago|reply
[+] [-] brianwski|14 years ago|reply
The ECC RAM absolutely does find and corrects problems (we see them in the logs). However, just to be absolutely clear we would not need ECC RAM -> Backblaze checksums EVERYTHING on an end-to-end basis (mostly we use SHA-1). This is so important I cannot stress this highly enough, each and every file and portion of file we store has our own checksum on the end, and we use this all over the place. For example, we pass over the data every week or so reading it, recalculating the checksums, and if a single bit has been thrown we heal it up either from our own copies of the data or ask the client to re-transmit that file or part of that file.
At the large amount of data we store, our checksums catch errors at EVERY level - RAM, hard drive, network transmission, everywhere. I suppose consumers just do not notice when a single bit in one of their JPEG photos has been flipped -> one pixel gets every so slightly more red or something. Only one photo changes out of their collection of thousands. But at our crazy numbers of files stored we see it (and fix it) daily.