Mid-range DIY Storage Server Buyer’s Guide, December 2010
Having recently published the high-end home/ small business December 2010 buyer’s guide, I received a lot of feedback requesting items for the mid and low-end guides. The mid-range I define as a minimum of six drives in the system with a maximum of fourteen drives. Anything more than fourteen drives and getting a 4U storage chassis becomes cost effective. Furthermore with only one add-in or onboard 8-port controller to handle 7-14 drives this seemed like a strong cut-off point.A few points before starting:
- First, hot-swap drives, either tray-less or with trays/ caddies I consider mandatory on a storage build. No doubt, skipping the trays leads to significant cost savings, however maintaining drives without hot swap trays stinks. Quite a few studies peg drive AFR in the 5% range, and higher for the first year. There is a good chance that several drives will fail in a fourteen drive system in the first three years, and oftentimes at the worst possible time. Simply disengaging a locking mechanism is much easier than removing case panels and unscrewing.
- Second, at fourteen drives I would start to consider a redundant PSU, for home use. For businesses, I think redundant PSUs are mandatory (as well as hot swap drives.)
- Third, I am going to have a higher-end mid-range build as well as a lower-end build in this review, both capable of connecting at least 14 drives. The low-end build will focus on 2-6 drive systems.
- Fourth, I think a mid-range server should be, at minimum, able to run a few operating systems in virtual machines, not as much as the high-end servers, but at least one or two small test virtual machines.
- Finally, this guide will not include drive selection. Again, 2TB hard drive prices are now in the $60-100 range and a generational shift is occurring as 3TB drives ship in quantity starting in early 2011.
The Intel Xeon X3440 is currently my favorite mid-range CPU. With four cores plus hyper-threading, the X3440 is a very strong LGA 1156 CPU that sacrifices some clock speed for a significantly lower cost versus its higher clocked brethren. Other considerations for this selection were that the slightly less expensive Intel Core i3-530 or i3-540 CPUs work with unbuffered ECC DIMMs, but the Intel rep I contacted about supporting ECC functions said that the i3-530 does not, in fact, support ECC functions. There are conflicting reports on this for sure, and you can get UDIMMS to work with the i3-530/ i3-540, but without the ECC functions enabled, one may as well be using non-ECC DIMMs.
AMD makes a strong showing in this area but falls short solely because of there being very few, and harder to find socket AM3 motherboards with IPMI 2.0 and KVM-over-IP. One thing I would like to see is a socket AM3 platform with server features that takes advantage of AMD’s inclusion of ECC support on many consumer level CPUs.
Here I will bifurcate the recommendation for one major reason, FreeBSD compatibility. While the LSI SAS 2008 chipset has become a favorite, it is not compatible with the current, stable versions of FreeBSD (although newer versions do support it.) Common elements will be the requirement for IPMI 2.0 and KVM-over-IP. Fourteen drive systems are large enough to create significant vibration and noise to the point where they will oftentimes be locked in an equipment closet making remote management a great feature. For the higher end build I will recommend the Supermicro X8SI6-F. As the review explains, the X8SI6-F combines a LSI SAS 2008 controller, IPMI 2.0/ KVM-over-IP, dual Intel NICs making it a very well integrated server board.
For the lower end midrange build I will recommend the Supermicro X8SIL-F. It was a very close call between this and recommending an AMD-based system, but the lack of a good selection of socket AM3 motherboards with IPMI 2.0/ KVM-over-IP make that recommendation difficult. The X8SI6-F does have some distinct advantages mostly due to its larger size including an additional internal USB header, onboard SAS 6.0gbps, and two extra DIMM slots which can be populated when using RDIMMs and a compatible Xeon CPU.
RAID Controller/ HBA Selection
For the high-end build, with the Supermicro X8SI6-F’s onboard LSI SAS 2008 controller, does not require an additional controller to reach 14 drives of total storage connectivity. The lower-end mid-range build does require a controller and here I would recommend a LSI 1068e based controller.
The main reason for this recommendation is because the LSI 1068e works well with many types of systems, including FreeBSD/ FreeNAS, which is an attractive platform in the mid-range space. If FreeBSD/ FreeNAS are not operating systems that will be run on a system, the SAS 2008 based cards like the LSI 9211-8i are probably the way to go.
In general, and especially with systems containing more than six drives, I highly recommend using ECC memory. The ECC price premium for unregistered ECC DIMMs is relatively low compared to total component cost. I have been using Kingston KVR1333D3E9SK2/4G (4GB ECC UDIMM) kits for awhile now and they work well. Both configurations should work well with 8GB of RAM so I will use two kits in the spec. One can always opt for higher capacity kits. On the X8SI6-F, one can also use six 2GB RDIMMs for 12GB which provides extra flexibility when paired with the X3440 and the larger ATX form factor.
The 14 drive system space was very difficult to recommend a chassis for. For the higher end mid-range system I had a difficult time choosing between the Supermicro CSE-836TQ-R800B which is a 16 drive 3U rackmount chassis and a SC933T-R760B which is a 15 drive 3U rackmount using a 760w triple redundant PSU. When purchasing a system with a redundant PSU, it is often cost-effective to purchase the chassis with the PSU. The 836TQ-R800B is a bit over sized for this build since it holds 16 drives (two more than the onboard controllers can handle) and is a rackmount enclosure. The Chenbro RM41416T2-B-650R is a 4U (slightly larger) sixteen hot swap drive case with a 650w redundant power supply that is about $80 less expensive than the Supermicro CSE-836TQ-R800B but well over $150 more than the SC933T-R760B. In the end, the Supermicro SC933T-R760B did get the final selection because it was the cheapest solution that met the requirements (and exceeded them with the triple redundant PSU.)
For the non-redundant PSU version, I was admittedly a bit lost. My first inclination was to use a large full-tower case and then add hot swap 4-in-3 enclosures which would utilize twelve 5.25″ external bays. The cost ended up being in the $300-350 range. A lot of users go this route on similar systems, and it is attractive option for building a system over time. An eight drive system may expand both in terms of drives and racks over time, spreading purchasing costs. On the other hand, at this point the Norco cases really provide a lot of value. I ended up recommending the RPC-4220 over the RPC-4020 because after owning both for a long time, I will say the RPC-4220 has much easier hot swap mechanisms.
Power Supply Selection
For the higher end mid range build, the redundant PSU is included with the chassis. In the high-end buyer’s guide I noted that I have a current non-redundant preference for the Corsair AX series Gold certified PSUs. Here I think the AX750 is a good choice. It should be noted that a quality Seasonic single-rail PSU is a strong alternative here. Also it should be noted that the 750w version is fine with 14-drives and a X3440 so I will not recommend any other versions. Both the Corsair and Seasonic X-750 feature fully modular connectors which is important in non-redundant servers. A fully modular system can be unplugged at the PSU, unit replaced, and cables plugged into the new unit much faster than in non-modular PSUs.
A consideration for a lot of people will be noise, and the Corsair PSU is much quieter than a triple redundant Supermicro PSU which sounds like a relative of a jet engine. Combined with the Norco RPC-4220 and an optional 120mm fan bracket for that case, a decently quiet server can be built with the non-redundant PSU build. I have a version of this (one can see it in the Intel SASUC8I picture above) and I will say that the decreased volume is welcome in a home setting.
Final Configuration 1 (non-redundant PSU)
- CPU: Intel Xeon X3440
- Motherboard: Supermicro X8SIL-F
- Chassis: Norco RPC-4220
- Controllers/ SAS Expanders: 1x Intel SASUC8I
- Memory: 2x KVR1333D3E9SK2/4G
- Power Supply: Corsair Professional Series Gold Certified 750w CMPSU-750AX
Approximate Final Cost (without drives): $1180
Final Configuration 2 (redundant PSUs)
- CPU: Intel Xeon X3440
- Motherboard: Supermicro X8SI6-F
- Chassis/ Power Supply: Supermicro SC933T-R760B
- Memory: 2x KVR1333D3E9SK2/4G
Approximate Final Cost (without drives): $1400
Of the three guides (high-end, mid-range, and low-end), I actually found this one to be the most difficult to write by a large margin. No manufacturer caters to the fourteen (max) drive market most likely because SAS controllers come in multiples of four ports. Furthermore, an eight drive system will likely move very close to the realm of a lower-end system while a fourteen drive system may be overkill for many users. As with any of these buyer’s guides, feel free to modify the build to best suit your needs. Making the design decision to build a bare bones mid-range server can save a lot of money here. One could use a cheap Phenom II X2 or X4 CPU, a consumer level motherboard, supporting ECC functions but not IPMI 2.0, and non-hot swap drives and save four hundred or more dollars over the non-redundant PSU build. Doing so sacrifices serviceability though, which becomes important when a user has 20TB+ of data stored in a system. The bottom line is that in the mid-range there is a lot of flexibility when it comes to components.