My main server, the Big WHS now houses over 60TB of storage, runs multiple VM’s, and has over 10 Gigabit NICs. At the heart of this server, is a Supermicro X8ST3-F. It was not the first motherboard I tried in the server, as I originally tried using an ASUS P6T7 WS Supercomputer in the Big WHS, but it has been running solidly since its first installation. Aside from its stability, it also comes with many PCIe slots, an onboard LSI 1068e based 8 port SATA/ SAS controller, dual Intel Gigabit NICs, onboard video, and IPMI 2.0 with KVM over IP.
For this review, I am using the Big WHS. This is not my standard test configuration, however if one is building a server off of the X8ST3-F instead of a lower-end motherboard, the server is most likely going to have many disks, lots of RAM, and many ethernet ports.
- CPU: Intel Core i7 920
- Motherboard: Supermicro X8ST3-F
- Memory: Patriot Viper 12GB DDR3 1600
- Case (1): Norco RPC-4020
- Case (2): Norco RPC-4220
- Drives: 12x Seagate 7200rpm 1.5TB, 2x 7200.11 1TB, 12x Hitachi 7200rpm 2TB and 2x 1TB, 8x Western Digital Green 1.5TB EADS, 2x Western Digital Green 2TB EARS.
- SSD: 2x Intel X25-V 40GB
- Controller: Areca ARC-1680LP
- SAS Expanders: 2x HP SAS Expander (one in each enclosure)
- NIC (additional): 2x Intel Pro/1000 PT Quad , Intel Pro/1000 GT (PCI)
- Host OS: Windows Server 2008 R2 with Hyper-V installed
- Fan Controllers: Various
- PCMIG board to power the HP SAS Expander in the Norco RPC-4220
- Main switch – Dell PowerConnect 2724
One of the key differentiators for the X8ST3-F is the number of expansion slots available. The X58 based X8ST3-F has three PCIe x8 electrical slots and one PCIe x4 electrical slot housed in one x16 physical slot and three x8 physical slots. Beyond this, there are two legacy PCI slots. The four PCIe slots can easily handle quad port NICs, raid cards, HP SAS Expanders, and etc. Beyond these the two PCI slots allow for additional Intel Gigabit NICs. While this may seem like a fairly expandable system, one needs to remember that there are two onboard Intel NICs, a management Realtek 10/100 NIC, and an onboard LSI 1068e controller. Practically speaking the X8ST3-F gives the user the option of configuring plenty of network bandwidth alongside enough hard drive expansion options to handle any enclosure. Case and point, the Big WHS usually has 12 Intel Gigabit NICs (two onboard, eight from Quad port cards, and another two in PCI slots) and port capacity for 46 internal drives.
Another strong possibility for the X8ST3-F is as a NVIDIA CUDA machine using either single or dual slot cards. While consumer boards can handle more GPUs using switched PCIe slots, the X8ST3-F has many onboard features (LSI RAID controller, Intel NICs, IPMI 2.0, and etc) that the consumer boards lack.
Aside from the expansion slots, it is worth mentioning that the Intel LGA1366 platform also offers six DDR3 slots. This is an important difference between the Intel LGA1156 platform and AMD AM2+/ AM3 platforms because one can either utilize more RAM or utilize lower density DDR3 modules.
Integrated LSI SAS1068E RAID Controller
The Supermicro X8ST3-F differentiates itself among many of Supermicro’s other offerings by providing an integrated 8-port LSI SAS1068E controller. The only major downside to this controller is the fact that it only supports raid 0, 1, and 10 in stock form. For a lot of users, this is enough as raid 1 provides redundancy and raid 10 provides redundancy and speed without the overhead of performing parity calculations. One of the cool features is that using a reverse Mini-SAS to SFF-8087 cable (I used a 3ware/ AMCC CBL-SFF8087OCR-06M) one can connect a HP SAS Expander and use the onboard controller for 36+ disks using one expander (a second can be added easily). This is a solid solution for those using OpenSolaris, FreeBSD, or Linux software raid based solutions. The LSI SAS1068E is not the fastest controller by any means, but it is supported by Microsoft Windows Server products as well as open source operating systems alike.
It should be mentioned that one can purchase and use a AOC-IButton68 to enable RAID 5 support on the X8ST3-F’s onboardLSI SAS1068E controller. This iButton based device has a street price of about $75. Frankly, I would advise against this unless there is a need to run RAID 5 on this controller. Dedicated controllers, while costing considerably more, do have faster chips for parity calculations, can run RAID 6, have dedicated cache, and can use battery back up units. Then again, if one really wants basic RAID 5 this is not a bad option.
Dual Integrated Intel Gigabit LAN
Like the Supermicro X8SIL-F, the Supermicro’s X8ST3-F has two Intel 82574L based gigabit NICs onboard. For servers, Intel Gigabit NICs are superior to other offerings. This is for two reasons. First, operating system and Hypervisor (VMWare ESX / ESXi, Microsoft Hyper-V, and etc) support is basically universal for Intel NICs. Second, I have seen much better throughput on Intel NIC to Intel NIC gigabit networks versus networks with Realtek and Marvell network controllers.
Each Intel 82574L supports features such as like TCP/UDP CPU offloading and TSO for lower CPU utilization, PXE/ iSCSI boot, Jumbo frames, two transmit and receive queues per port, a 40KB packet buffer, and etc. The Intel 82574L is the same controller used in Intel’s Gigabit CT (EXPI9301CTBLK) PCIe x1 network adapter. In essence, the Supermicro X8ST3-F saves one $60-70 and a PCIe slot (if one were to get a dual port Intel NIC) or two PCIe slots versus a lower-cost board where Intel NICs would be required separately.
Also of note again, with the X8ST3-F, the Supermicro IMPI 2.0 feature includes the ability to mount CD’s and DVD’s over the onboard 10/100 Realtek controller. Furthermore, the keyboard and video inputs and outputs can also go over the IPMI 2.0/ KVM over IP interface offloading this data from the dual Intel gigabit NICs. The net result is that the Intel NICs will have more bandwidth available for sending and receiving content on the X8SIL-F because it has the third NIC (Realtek) for management functions.
IPMI 2.0 and KVM over IP
One of the best features of Supermicro boards is the IPMI 2.0 and KVM over IP built in to many of the boards. Aside from remote system monitoring of things such as CPU voltages, temperatures, fan speends, and other parameters, Supermicro’s implementation has great remote control features through the web interface. My three favorite IPMI 2.0 and KVM over IP features are depicted below.
One will notice a few things in the above screenshot. First, this interface is slightly different than the X8SIL-F, yet seems to provide the same functionality. The browser window depicted on the left shows remote power control of the server that allows one to remotely reboot and power up or down the server among other things. On the right side of the above screenshot (“Redirection Viewer” window), one can see the Windows Server 2008 R2 login screen as seen through the Java based KVM over IP application. If one were to plug in a monitor to the X8ST3-F’s VGA output, this is the same output they would see. Unlike RDC or VNC which can be used to have remote monitor output in an operating system, Supermicro’s KVM over IP works at the system board level so one can use it for virtually any system maintenence task, including BIOS configuration.
The keyboard and mouse functionalities compliment the VGA over IP so remote administration is greatly added. A final feature of the X8ST3-F’s IPMI 2.0 is the ability to mount CD or DVD ISO files as well as floppy images over the network. To the X8ST3-F’s operating system (and to some extent the BIOS) these network mounted images look like they are local drives meaning that one can easily use this interface to boot and install operating systems or other software even without physical CD/ DVD drives attached. In fact, one could use this interface in a diskless system to run network stored LiveCD operating systems on the server. Overall, Supermicro’s IPMI 2.0 and KVM over IP are great features that are very hard to do without once one becomes accustomed to the convenience. It works well enough that over the past few months I have yet to connect a CD or DVD ROM drive, monitor, keyboard, or mouse to the system.
Dual Internal USB headers
The X8ST3-F has dual internal USB headers. This allows one to add a USB thumb drives to the interior of the enclosure housing the Supermicro X8SIL-F. Small business storage servers, this is a great feature since one can have two USB drives either for dual OSes or one OS USB drive and a secondary firmware update USB drive. FreeNAS, Openfiler, unRaid, untangle and many other operating systems can install directly to a relatively low-cost flash drive saving onboard SAS/SATA port (the X8ST3-F has 8x SAS/SATA ports on the onboard LSI 1068e and six onboard SATA II ports form the ICH10R). Since the USB sticks can be installed inside the enclosure, it offers more protection from becoming wrongfully uninstalled than an externally installed drive. Finally, one can mirror the USB thumb drives in the event that one drive has flash wear out from usage.
For video, the Supermicro X8ST3-F provides a Matrox G200eW based GPU that outputs only to analog VGA on the backplane. The saving grace of this solution is the integration with the Winbond WPCM450 BMC which provides IPMI 2.0 features and KVM over IP. As a side note, if you are running a quad core, LGA1366 based Intel CPU, you have the same 8MB of cache as the Matrox G200eW has in DDR2 memory. As a comparison, high-end dedicated GPUs currently have 2GB or more of onboard GDDR5 memory. Of course, the saving grace is that with KVM over IP, most users will not need more than what this Matrox G200eW has to offer.
An important note here is that the onboard GPU does save the need to use a PCIe slot for graphics while at the same time not providing the airflow restrictions of an add-in GPU.
The Supermicro X8ST3-F differentiates itself with its onboard components. First, IPMI 2.0 along with the Matrox G200eW allow for remote server administration. Supermicro’s IPMI 2.0 implementation means that a user does not need to install a cheap video card nor a CD/DVD ROM in the server. Second, the LSI SAS1068E is a solid SAS controller that is well supported in various operating systems and does not require an expansion slot. Furthermore, the LSI controller supports HP SAS Expanders making the X8ST3-F a good choice for storage servers based upon software raid such as ZFS/ Raid-Z or Raid-Z2. Third, the dual onboard Intel 82574L gigabit NICs save a user from having to purchase a dual port Intel Pro/1000 PCIe card and again save an expansion slot.
It is hard to think of a $340 motherboard as a value board, yet the X8ST3-F is if one uses the onboard components. All tallied, $20 for a generic DVD ROM drive and $30 for a cheap VGA card (neither of which will work as well for network administration as Supermicro’s IPMI 2.0 implementation) save a user $50. The LSI SAS1068E is the same ASIC used in the Intel SASUC8I RAID Controller which is a $160 8-port controller. Finally, the two onboard Intel Gigabit NICs are roughly equivalent to an Intel Pro/1000 PT Dual port card which runs approximately $160 also. In essence, the Supermicro X8ST3-F provides lots of expandability plus about $370 worth of onboard hardware for only $340. That is a fairly strong value proposition.