The Big WHS: February 2011 Update
It has been a long time since I have posted about The Big WHS. At the last update in May 2010, the machine occupied 8U using two 4U enclosures, and was topping 60TB of raw storage capacity. Since then there have been quite a few developments that I thought I would write about.
The current hardware list is as follows:
- CPU: Intel Xeon W3550
- Motherboard: Supermicro X8ST3-F
- Memory: 24GB ECC unbuffered DDR3 DIMMs (6x 4GB)
- Case (1): Norco RPC-4020
- Case (2): Norco RPC-4220
- Drives: 8x Seagate 7200rpm 1.5TB, 4x Hitachi 5K3000 2TB, 20x Hitachi 7200rpm 2TB, 8x Western Digital Green 1.5TB EADS, 2x Western Digital Green 2TB EARS.
- SSD: 2x Intel X25-V 40GB
- Controller: Areca ARC-1680LP (second one)
- SAS Expanders: 2x HP SAS Expander (one in each enclosure)
- NIC (additional): 1x Intel Pro/1000 PT Quad , 1x Intel Intel Pro/1000 GT (PCI), Intel EXPX9501AT 10GigE
- Host OS: Windows Server 2008 R2 with Hyper-V installed
- Fan Partitions updated with 120mm fans
- PCMIG board to power the HP SAS Expander in the Norco RPC-4220
- Main switch – Dell PowerConnect 2724
The Big WHS Changes
First off, the CPU was changed to a proper Xeon W3550 CPU adding both ECC support as well as slightly higher clock speeds. It was a simple remove and replace for the old Core i7-920 so this was an easy update. Frankly, in any system this large, not using ECC memory is a sub-optimal idea and this helped a lot towards that end. I could see a day though where a DP motherboard is added into the equation which would require another component swap. Memory has been changed to 24GB of ECC DRAM accordingly. I did find through trial and error that not all memory suitable for Intel DP motherboards are suitable for the X58 system. Below see a picture of what 24GB (and hundreds of dollars) worth of incompatible RAM looks like. Bottom line, always check the hardware compatibility lists.
The Areca 1680LP started acting up which caused me to re-evaluate the SAS controller situation. Next steps are moving to a simple HBA architecture. WHS is unable to use all drives, and due to new project load, my storage consumption has slowed significantly. I will likely keep the Hyper-V VM using Drive Extender Storage and getting more elegant with the Windows Home Server 2011 environment in the future. One important note for HP SAS expander owners is that I did see PCB warp due to all of the SFF-8087 connections in the card. The new curvature of the HP SAS Expander can be seen in the below photograph.
I started to use Mellanox Infiniband dual-port connections between the Big WHS and other machines (as can be seen in the photo above. The short story is, on Windows the things work great, on other operating systems they are not as easy to use. I have started testing Intel EXPX9501AT 10GigE adapters and I do like the architecture except that 10GigE switches are too expensive at the moment. For those that read the forums, there is a 10GigE adapter thread with some additional information.
Additional hard drives have been added with some of the older Seagate 7200.11 1.5TB drives removed and the Hitachi 7k2000 1TB drives and Seagate 7200.11 1TB drives were completely removed leaving the minimum capacity of 1.5TB for any installed drives. Total raw capacity is 76TB of rotating disks at the moment.
On the cooling front, I had a fan controller become non-functional so I did swap out the fan partitions for Norco’s 120mm fan partitions (see the test fittings above). As a positive note, this leaves one with enough fan headers on the Supermicro X8ST3-F to power all fans in the primary case off of the motherboard and therefore get remote IPMI 2.0 fan readings. One can see more experiences in the Norco RPC-4020 and RPC-4220 120mm fan partition guide.
From a software side, the big additions have been Microsoft Windows Server 2008 R2 SP1 and now new Windows Home Server 2011 and Windows Small Business Server 2011 Essentials virtual machines.
The maximum capacity I will see with The Big WHS is 80TB, only about 4TB greater than today. Beyond that point, I will be getting much more elegant with the solutions, incorporating tiered storage and a host of other enhancements. 2011 is shaping up to be a very exciting year indeed. One thing that I am glad about is that the Big WHS is no longer a common testbed for this site’s reviews. Multiple test systems have allowed the Big WHS to deliver two consecutive 90 day up time periods with the shutdowns being for a dual hardware installation and Windows Update related installations. Frankly, the stability has been awesome.