Introduction
Cloud backup provider Backblaze trotted out details of its third-generation DIY storage pod last week. The new version is mainly comprised of component upgrades, which are summarized in the comparison table below. Despite the upgraded components, at about $1900 for the chassis sans drives, V3 is about $40 cheaper to build than V2.
| Component | V3 | V2 [details] | V1 [details] |
|---|---|---|---|
| CPU | Intel Core i3 processor i3-2100 @ 3.1 GHz | Intel i3 540 @ 3.06 GHz CPU | Intel E8600 Wolfdale @ 3.33 GHz |
| Motherboard | Supermicro MBD-X9SCL-F (MicroATX) | SuperMicro MBD-X8SIL-F-B | Intel BOXDG43NB LGA 775 G43 ATX |
| SATA Cards | Syba PCI Express SATA II 4-Port RAID Controller Card SY-PEX40008 (Qty 3) | Syba PCI Express SATA II 4 x Ports RAID Controller Card SY-PEX40008 (Qty 3) | 2 port – Syba SD-SA2PEX-2IR PCI Express SATA II Controller Card (SiI3132) (Qty 3) 4 port – Addonics ADSA4R5 4-Port SATA II PCI Controller (SiI3124) (Qty 1) |
| RAM | 4GB Samsung BDDR3-1333 PC3-10600 (Certified by Supermicro) (Qty 2 = 8GB) | Crucial CT25672BA1339 2GB, DDR3 PC3-10600 (4x 2GB = 8GB total) | Kingston KVR800D2N6K2/4G 2x2GB 240-Pin SDRAM DDR2 800 (PC2 6400) (Qty 1) |
| Boot Drive | WD SCORPIO BLUE WD1600BPVT 160GB 2.5" Internal | Western Digital Caviar Blue WD1600AAJS 160GB 7200 RPM | Western Digital Caviar WD800BB 80GB 7200 RPM IDE Ultra ATA100 3.5" |
| Power Supply | Zippy PSM-5760V Power Supply (760W) (Qty 2) | Zippy PSM-5760 Power Supply (760W) (Qty 2) | Zippy PSM-5760 Power Supply (760W) (Qty 2) |
Backblaze’ blog post does a good job of summarizing the whys and wherefores of component selection, so I won’t repeat them here. But two things caught my eye in the blog post, so I exchanged a few follow-up emails with Backblaze Co-Founder and CEO Gleb Budman to get the info that follows.
Software
Backblaze’s use for the pod doesn’t rely on high read and write speed for any individual pod. Backblaze’ proprietary backup system balances the read/write load over many pods, so the Gigabit connections to each pod are nowhere near saturated.
Backblaze’s version of the pods continues to run Debian 5, with the drives formatted using ext4. But I asked Gleb if he could provide insight into what people using the Backblaze pod hardware design use for their software. He said many pod users "just use the Storage Pods as supersized NAS drives, storing corporate data on them as an inexpensive central repository" with drives in RAID arrays.
Gleb also said some companies use pods in large-scale deployments. These users "don’t RAID them and address the drives directly and then deal with redundancy in their own software applications". He also offered that most pod users run Linus, but some users have installed Windows "particularly those that choose to use them as simply a server for central corporate storage".
Budman also offered that they had heard of some installs of Hadoop on pods. Apparently, this is a work in progress, since they have heard that Hadoop doesn’t deal well with the large number of drives (45) in a pod and has trouble balancing data across drives.
The 45drives.com wiki OS page notes successful use of FreeNAS and Openfiler by folks who use the pods as big ol’ NASes.
Drives
Like any company whose business depends on lots of storage, Backblaze was hit hard by the Thailand flooding in fall 2011. So they have gotten very creative in buying drives at cheap prices, often depending on a volunteer army of "drive farmers".
But when I checked the current list of drives used, I thought maybe they had gotten a bit too creative. Note that all the five drives listed are plain ol’ desktop drives, except for the WD Red, which is designed specifically for NAS applications. There’s nary an "Enterprise" grade drive in the bunch.
4 TB drives
- Hitachi Deskstar 5K4000 (HDS5C4040ALE630)
3 TB drives
- Hitachi Deskstar 5K3000 (HDS5C3030ALA630)
- Hitachi Deskstar 7K3000 (HDS723030ALA640)
- WD Red (WD30EFRX-68AX9N)
- Seagate Barracuda 7200.14 (ST3000DM001-9YN166)
A hot topic in the SNB forums is always drive selection, with the standard advice tending to be to use Enterprise grade drives if you want higher reliability. So I asked Gleb why Backblaze is using only consumer desktop drives. He replied:
This has actually been a very significant discussion area for us over the years. The price of enterprise drives is approximately 2x higher than consumer grade drives, so there has to be a tremendous benefit from the drives. When we started, many folks said "consumer drives will fail sooner". However, after over 5 years of running consumer drives, we have not seen any huge failure rates. In fact, we tested enterprise drives in our Storage Pods a few years back and they performed WORSE than the consumer drives.
For the most part, it’s just math. We assume all drives will fail at some point and the software handles that. Thus, if a consumer drive fails over average in 2 years and an enterprise one in 4 years, it would be a breakeven decision. However, realistically, we haven’t seen failure rates anywhere near that level – the drives fail in the single-digit % per year range. Within a few months, we’re planning on actually publishing the stats on the failure rates of different drives.
There is some subtlety. Some drives simply do not work well in these environments (RAID, [lots of] vibration, etc.). For example, Western Digital Green drives have worked poorly, so while we would like to use them, we don’t. But the consumer-grade Hitachi and Seagate drives we listed work well.
Finally, the hard drive companies are starting to realize there is a market for companies like ours that have massive scale, but are focused on density at low cost. The Western Digital Red drive is targeted at that market and has worked great in our Storage Pods.
Most large cloud providers either do not, or will not, use "enterprise drives". This is the future.
I then followed up with a few more questions.
SNB: What do you think WD got right in designing the Red drive?
Backblaze: The WD Green drives were not designed to be in RAID arrays. So it wasn’t about how long they survive, it was about not working in RAID systems. Technically, the WD Red drives are designed for arrays up to 5 drives. However, they’ve worked great in our systems. We believe [WD has] improved the drives’ ability to handle vibrations, but do not have explicit knowledge of that.
SNB: If large cloud providers aren’t using "enterprise drives", what is the future for these drives?
Backblaze: There are a real variety of needs and people will pay for higher requirements. Some of those will (continue) to be packaged as "enterprise" drives. For example, performance is a real requirement for some applications. Having a 15k RPM drive is important for some companies and they will pay more for that.
Having drives that are more long-term reliable will continue to be a market as there are there are certain companies who do not have control over the software stack and thus want the most reliable drive they can get so they minimize issues they have to deal with or downtime.
But for cloud companies, reliability will be primarily a cost tradeoff…and the difference in value between a drive that fails in 4 years vs. 5 years is 20%, not 2x.