Oftentimes those building their own small business and home storage servers are looking to save considerable amounts of money. In the process, users oftentimes look to the lowest cost motherboard they can find, with the wisdom that add-in cards can get the features that they want. On the other end of the spectrum are server motherboards like the Supermicro X8SI6-F that have everything or almost everything users need onboard. The Supermicro X8SI6-F costs in the $300 range, which would be a turn-off for many users, but when one looks at the features, $300 looks like a relative bargain.
Test Configuration (Base):
- CPU(s): Intel Core i3-530 and Intel Xeon X3440
- Motherboard: Supermicro X8SI6-F
- Memory: 2x 4GB kits of Kingston ECC 1333MHz DDR3 KVR1333D3E9SK2/4G (Unbuffered)
- Case: Norco RPC-4220
- Drives: 8x Hitachi 7,200rpm 2TB, 2x Western Digital Green 2TB, 6x Seagate 1.5TB 7,200rpm
- OS(es): Windows Server 2008 R2 with Hyper-V installed, Windows Home Server (in Hyper-V), Windows Home Server V2 Codename Vail (in Hyper-V), NexentaCore 3.0.1, OpenFiler (v2.3), FreeNAS (0.7.1), VMWare ESXi 4.1
The LGA1156 platform has been a strong platform for enthusiasts and is quickly catching on with HTPC users for its solid performance and low power consumption. Motherboard manufacturers like Asus, Gigabyte, MSI, and others all have enthusiast level boards with the LGA1156 chipset with multi-stage power (for overclocking), NF200 switched PCIe lanes, USB 3, SATA 3, onboard HD audio, HDMI outputs and etc. Some LGA1156 boards also include dual Realtek gigabit NICs although a single NIC is standard. Frankly, most if not all of these features are of low value in storage servers.
In a storage server, one typically looks for a few features:
1. Enough CPU power to perform any parity calculations required
2. Connectivity for multiple hard drives
3. Management features
4. Intel network ports
5. Compatible devices for a range of server operating systems
6. Enough RAM for cache purposes (especially important for ZFS caching)
7. Solid customer support and reliability
With the first item above being almost a given assuming a user properly sizes their CPU (few storage servers built on this platform will need a six core CPU), the remaining items will be the focus of this review.
Hard Drive Connectivity and the X8SI6-F… Awesome!
If there is one area that the X8SI6-F excels in, it is clearly in connecting hard drives. The X8SI6-F has an onboard Intel ICH controller as part of the 3420 chipset that provides six SATA 2.0 ports. This is a fairly standard feature on server motherboards. One will not find an eSATA port on the backplane because typically rack mount enclosures are large and internal storage is much preferred. Although the Intel ICH is a great controller, and does does support RAID functions, operating systems can have issues with the ICH in RAID mode. While Windows operating systems have no issues, operating systems such as OpenSolaris/ NexentaCore do not work well with “fake RAID” implementations found on chipset RAID solutions. Setting the controller to a non-RAID mode brings compatibility in line, and with the excellent latency characteristics, makes both hard drive performance and SSD performance (very important for cache/ L2ARC drive performance).
Going beyond the simple ICH, the X8SI6-F provides eight SAS 2.0 (6.0gpbs) ports onboard courtesy of a LSI SAS 2008 chip onboard. Connectivity is provided through two SFF-8087 ports which make cabling in chassis like the popular Norco RPC-4220 and RPC-4224 very clean. Windows Server 2008 R2, Windows Home Server, various Linux flavors, OpenSolaris/ OpenIndiana/ NexentaCore and VMware ESX and ESXi all work really well with the LSI SAS 2008 controller.
For all intents and purposes, the Supermicro X8SI6-F’s onboard LSI SAS 2008 controller is equivalent to the LSI 9211-8i add-in card. Just for reference, the LSI 9211-8i retails for around $260 US. The important thing here is that even if the onboard controller did fail, one could purchase a compatible add-in card and recover the array. Just to test this, using Windows Server 2008 R2 I was able to unplug drives (all idling) from the onboard controller and into a LSI 9211-8i controller without the OS and was able to access the drives seconds later.
Aside from the obvious cost savings, both implementations support the very popular HP SAS Expander. Connecting one of the Supermicro X8SI6-F’s ports to the HP SAS Expander yields 32 SAS/SATA ports (through the SAS Expander), 4 additional SAS/ SATA ports (through the second SFF-8087 connector which can also be used with a second expander), and then 6 SATA ports via the Intel ICH. Adding one SFF-8087 to SFF-8087 cable and a HP SAS Expander gives connectivity for 42 drives! One could also either daisy chain the SAS Expanders or connect a SAS Expander to the second of the Supermicro X8SI6-F’s SAS ports and have connectivity for 70 drives.
One typical question I see on forums is how does the SAS 2008 chip perform with mechanical drives and a HP SAS Expander. Here is a preview ATTO Benchmark showing over 900MB/s to 1GB/s in read speeds which is more than adequate for many file servers. As a spoiler for an upcoming feature, the performance numbers are virtually identical to the discrete LSI 9211-8i’s performance.
For users looking to build systems with less than eight drives, this is not an issue. For users looking for a solution that can scale inexpensively, the X8SI6-F provides a clear value proposition.
One of the things that I consistently offer advice on is management features on any platform intended for server use. Ideally, a storage server’s only maintenance should be with its hard drives. Mechanical drives fail, and sometimes capacity needs to be expanded. Unfortunately, sometimes things go awry. A CPU heat sink clip may dislodge causing a CPU overheat situation. A kernel may panic causing the inability to login to a machine. An error prior to OS boot may cause a machine to become totally unresponsive. For all of these situations, server grade motherboards have features to help an administrator recover from a potentially negative situation.
I have discussed Supermicro’s IPMI implementation with KVM-over-IP at great length lately, including the highly recommended utility IPMIview. Like the other Supermicro “-F” boards reviewed here, the X8SI6-F’s IMPI 2.0 implementation utilizing the Nuvoton WPCM450 Baseboard Management Controller (BMC) includes ability to mount CD’s and DVD’s over the onboard dedicated 10/100 Realtek controller. The KVM-over-IP pulls a bit more bandwidth (<2mbps from what I have seen) than RDC, so being able to offload that traffic is not essential from a performance point of view, but it is welcome. Furthermore, the additional NIC allows the management interface to be placed on a separate management network which is a best practice considering that the BMC allows for remote rebooting and local console access over a network interface.
Bottom line, the IPMI 2.0 and KVM-over-IP functions on the Supermicro X8SI6-F and similar motherboards are simply must-haves for anyone except those running a server next to their desktop, and even then it is a nice to have. Recently I had a Hyper-V VM bring down a Windows Server 2008 R2 server and I was able to reboot the machine from my car en-route to dinner. It was the difference between having to miss dinner due to an unscheduled stop to the server (which was a 35 minute drive away each way) and a 90 second stop in the parking lot of the restaurant. Those situations are where building servers off of consumer grade hardware clearly do not pay off.
Intel Network Ports
Supermicro’s X8SI6-F, like the X8SIL-F it has two Intel 82574L based gigabit NICs onboard. Anyone reading my site will note that I am biased heavily in favor of Intel based gigabit NICs as they are the most consistently high-performing NICs that I have used. Furthermore, the compatibility of the 82577L with Windows, Linux, FreeBSD, VMware ESX and ESXi, and OpenSolaris is a given. The OS and Hypervisor support is generally very good for these Intel gigabit NICs. Second, I have seen much better throughput on Intel NIC to Intel NIC gigabit networks versus networks with Realtek and Marvell network controllers. After months of using Realtek NICs in my servers, the cost savings were just not worth it.
The Intel 82574L supports features like TCP/UDP CPU offloading and TSO for lower CPU utilization, PXE/ iSCSI boot, Jumbo frames, two transmit and receive queues per port, a 40KB packet buffer, and etc. For reference, the Intel 82574L is the same controller used in Intel’s Gigabit CT (EXPI9301CTBLK) PCIe x1 network adapter. In essence, the Supermicro X8SI6-F and similar motherboards save one $60-70 and a PCIe slot (if one were to get a dual port Intel NIC) or two versus a lower-cost board where NICs would be required separately. Another important note is that the onboard NICs do not require a raised PCB expansion card for network connectivity which helps airflow and also in 1U and 2U enclosures where expansion ports are limited, oftentimes requiring riser cards.
After installing, seven different operating systems/ hypervisors over the past ten days, the device compatibility is great. By that I mean that every operating system did not require searching for drivers in order to get all onboard components to work. The only issue was with FreeBSD/ FreeNAS installations which do not have default LSI SAS 2.0 drivers except for FreeBSD 9-Current.
With how finicky both OpenSolaris variants and VMware ESXi can be with drivers, having something that simply works out of the box is great. The Asus P6T7 WS Supercomputer, for example, was the exact opposite where ESXi 4.0 gave multiple compatibility errors during installation necessitating the use of add-in cards.
The Supermicro X8SI6-F is based on the Intel 3420 chipset. Paired with an Intel Xeon X34xx series CPU, the X8SI6-F supports up to 16GB of unbuffered ECC DIMMs in four slots. With registered ECC DIMMs, the X8SI6-F supports up to 32GB of memory. For users that are willing to risk their data to non-ECC DIMMs, the 16GB limit also applies. It should be noted that like other motherboards based on the 3400 series chipset, it is strongly recommended to use memory that are on the hardware compatibility list.
Internal USB Headers
A really nice feature of most server motherboards is the presence of an internal USB header. The Supermicro X8SI6-F has two such headers, much like the Supermicro X8ST3-F. This allows one to add two USB thumb drives to the interior of the enclosure housing the Supermicro X8SI6-F. For home and small business storage servers, this is an enormous benefit. FreeNAS, Openfiler, unRaid, untangle and many other operating systems can install directly to a relatively low-cost flash drive saving an onboard SATA port. Installed in the server’s enclosure, the USB stick will the USB drives are unlikely to become dislodged for any reason. With two such ports I have made it practice on my X8ST3-F based servers to have a secondary 2GB drive with a recovery Linux distribution in the event something goes severely awry. Having another boot option for the $20 or less cost of a USB drive has come in handy, especially when combined with the KVM-over-IP features of the X8SI6-F.
Expansion slot wise, the X8SI6-F has a two PCIe slots, one a PCIe 2.0 x8 slot and another PCIe x4 electrical in an x8 physical. For legacy cards, there is a PCI slot onboard also. One will note that this is one slot less than the X8SIL-F, which is due to the LSI SAS 2008 controller occupying those PCIe lanes. Unlike a LGA1366 platform, any LGA 1156 platform is going to have less available PCIe lanes (unless a PCIe switch is implemented and those are usually not recommended in server environments.)
Suffice to say, one is sacrificing a PCIe slot for an onboard SAS 2.0 RAID controller, but for anyone looking to purchase the X8SI6-F, this is a known and welcome tradeoff.
ATX Form Factor
In contrast to the mATX Supermicro X8SIL-F, the X8SI6-F is a full sized ATX motherboard. This has advantages and disadvantages. One advantage of this form factor is that it allows the SAS ports to line up very well with many enclosures that have front facing disk headers in the front left (looking at the drive bays) of the enclosure. Using 1ft cables I was easily able to connect the X8SI6-F’s SAS ports to both the Norco RPC-4220 drive backplanes as well as a HP SAS Expander sitting in the bottom PCIe slot (x4 electrical). The main disadvantage is that this will not fit if someone is looking to make a DIY JBOD DAS-SAS Expander enclosure as I provided a guide for earlier. I would, however, offer that this board is more appropriate for a user looking to make a head unit instead of a DAS disk shelf.
As one can see above the ATX form factor of the X8SI6-F is significantly larger than that of the mATX X8SIL-F.
Supermicro provides a nice package of six SATA cables and four SFF-8087 breakout cables. This means that for users installing the X8SI6-F in conventional tower cases or other chassis with individual backplane ports, the cables are there to support fourteen drives immediately.
The motherboard manual does a good job of explaining a lot of the jumper configurations for things like disabling onboard LAN, video and SAS. One will note that the IO back plane is actually the same as the X8SIL-F which is a big plus if you swap motherboards ever.
Supermicro Quality and Support
Supermicro competes in the enterprise space and therefore has great support for technically inclined users. For example, after receiving a X8SIL-F PCB Revision v1.01 which did not work with the Intel Core i3-530 I had installed, I contacted Supermicro support about this issue. It was a known incompatibility and Supermicro was willing to immediately exchange my board. Supermicro is one of those companies that I have found over the past decade that if you do your due diligence before contacting support (double-check everything and do a quick Google search), they do a great job helping you. Overall, I have installed approximately 50 Supermicro based systems in the past 18 months and have yet to experience a failure. A small sample size for sure, but it is the only measure of reliability that I have to go by.
When looking at a $300 motherboard, many users may automatically dismiss the hardware as overly expensive. On the other hand, the Supermicro X8SI6-F has an onboard equivelant of a $260 LSI 9211-8i and dual Intel Gigabit CT (EXPI9301CTBLK) network adapters ($35 each). Just those three components alone would cost more than $330 if purchased separately. The caveat, of course, is that one wants to use Intel network adapters (highly likely) and a LSI SAS 2008 based SAS controller. If this is the case, then the price of the motherboard is, in fact, less than the cost of the discrete components. Compared to adding these components to a consumer level motherboard, this solution presents a clear cost savings. With the exception of those users running FreeBSD versions prior to release 9-Current, the value of having all of the KVM-over-IP and IPMI 2.0 features is priceless.
Supermicro offers a compelling platform for anyone using an operating system to manage storage pools and redundancy the X8SI6-F is a tremendous value. While some may consider a $300 motherboard to be “expensive,” the integrated components effectively cost more separate than the entire motherboard. What the X8SI6-F does deliver on it the ability to attach a very large number of drives at a very low cost. Unless one is looking for a hardware RAID 6 solution, the X8SI6-F is perhaps a category killing product for LGA 1156 based storage servers.
i was looking for a review of this thanks for giving a break from gtx 580 reviews today
you should get sponsorship from Supermicro. 😉
joking aside, great review
one thing i would be interested in….
what is the BASE power consumption of a SM X8SIL-F + LSI SAS 2008 add-in card *VS* a SM X8SI6-F
Also interested along with David in getting an idea of power consumption….my decision time is very quickly arriving and since I’m going to be paying my electric bill I suddenly have an interest in what it will be pulling down 😉
My only concern with this board is lack of PCIe slots….I suppose though something like the HP Expander would work with the onboard sas ports?
David and No1451: I will post the power consumption number of both setups next week. I also have a few other pieces that I will be writing this weekend including an Asus setup.
No1451: The ATTO benchmark above was taken with one of the onboard SAS (SFF-8087) ports connected to the HP SAS Expander which was sitting in the PCIe 1.0 x4 slot leaving the other PCIe 2.0 x8 slot open. At that point you have 42 drives worth of connectivity dual NICs, IPMI, dual USB headers, and an open PCI and PCIe x8 slot which is plenty for most users.
On the electric bill, I totally understand. Just to give you an idea though, I think of 1w = $1/ year difference.
Heh, duh, completely missed that part, that’s what I get for skimming over it while trying to listen to a lecture 🙂
Thanks for the review, it’s really refreshing to get commentary rather than just raw graphs, usability ranks just as high up there as specs/performance in my opinion.
Was wondering where you can buy the motherboard you reviewed.. Want to do a similar setup to the one you did for the review but can’t seem to find a retailer.
Also is this board a new board??
They are fairly hard to find at the moment. Newegg has been OOS for a few weeks. eBay/ Amazon usually have sellers, and the other option is to purchase from somewhere that carries the entire Supermicro line.
I have brought this concern that the retail availability is not up to par at this time to Supermicro. Hopefully they take action. I was considering trying to buy a 10 board bulk pack and just selling them myself to help with availability as they are good boards.
Something that I am having a little bit of trouble coming up with a number for, how many arrays can something like this controller handle? Trying to figure out a solution to speed up my WHS, thought maybe a number of small raid arrays running under the surface might do the trick.
So “speeding” up your WHS is a big project. A lot of times it comes down to configuration options as single drives can usually saturate 60-120% of a GigE port’s full speed depending on where the read/ write is happening.
What you ideally want is 3 drive RAID 5 arrays or larger RAID 6 arrays. The LSI SAS 2008 controllers are really RAID 0, 1, 10 controllers and IT mode for making them HBA’s (albeit SAS Expander aware). You can get the iButton for RAID 5, but it would be better just to get a hardware RAID controller with onboard write cache and battery back up.
For awhile now 2TB drives have been great. Remember, it does cost money to connect every drive since you need a controller port, cabling, space in a case/ a larger enclosure and etc. Also, more drives mean more power consumption which can be a few watts at idle or a few dollars per year.
Yeah, unfortunately hardware raid isn’t in the cards, I’m simply not willing to spend the money. JBOD it will continue to be by the looks of it
Thanks for this great review. I am looking for hardware running a home storageserver. My main focus is quality components, low power consumption and at least 9 x SATA. This board would be ideal for that matter, but I am unsure about the power consumption. Could you elaborate on this issue?
At first I was thinking about RAID5, but the costs to make it just right and the fact that you need enterprise discs to make it a solid array mad me think of JBOD. BUT, I couldn’t figure out what happens to my JBOD array when one disc fails. Do I loose all my data in the array of just the data on the failed disc? If I only use the data on the failed disc, how will I know what data I lost. Are files equally distributed over the JBOD array? Is there a way to choose on what fysical disc the files will be saved? In my case I would like to use JBOD to make a giant drive (8TB) with HD-movies (files of 8 to 20GB each). That’s much easier than four seperate drives. If one drive fails I loose 25% of my movies. I am fine with that, just takes some work to collect the data again. Of course I want to avoid loosing all my movies when one drive fails. Will JBOD suit me or not?
Does anyone have any recommendations for compatible memory for 16gb? I don’t think I need FBDIMMs.
I’m putting together an ESXi lab for my home with a X3440 and this board is perfect. I don’t want to do without iLO type features anymore and IPMI is a perfect solution!
Love your Supermicro MB reviews! There aren’t enough thorough reviews of these things around.
Oliver: Power consumption for this type of setup is relatively low, but will, of course be more than an Atom D510 based system for example. The X8SIL-F review has power consumption figures which are a few watts lower due to the exclusion of the SAS 2008 controller and mATX form factor of the X8SIL-F.
Using disks separately will be fine for you. Large RAID 5 volumes are scary tough given the size of modern drives.
Brenden: Thanks for the kind words. As for 16GB of UDIMMs, you could go with four sticks of Kingston KVR1333D3E9S/4G which would give you 16GB of UDIMMs on the Intel 3420 chipset. I have had fairly good results with Kingston modules in the past year.
It doesn’t work under FreeNAS ?
The current stable FreeNAS release does not support the LSI SAS2008 controller out of the box. See: http://www.freebsd.org/releases/7.0R/hardware.html#DISK
One suggestion is to try the new ZFSguru which is based off of a newer FreeBSD kernel and includes drivers for the newer cards.
Hi Patrick, do you thing i would have any problemds running an I5-661 1156 on this board, i have looked arounfd but only see I3 mentioned
Thank you for you love to this quality product. I bought it 1 month ago and love it very much.
“Unless one is looking for a hardware RAID 6 solution…”
Why not? I use it with Adaptec 5805Z Hardware raid with 512MB DRAM cache and 5 two TB WD Red Raid Edition in Raid6 and have very attractive file server in home environment.
Onboard LSI and SAS3 mode gives me good decision for future upgrade to high speed SSD storage.
It is very cheap – I pay 140$ for this MB with 2.66 Xeon and 24 GB Buffered ECC Samsung RAM, 80$ for Adaptec with Zero module and 300$ for 5 2TB WD Red HDD.
This server is present perfect for KyKyGLE. You are Welcome…. 🙂