The Supermicro X10DRG-Q is a motherboard with a purpose: provide a platform for many PCIe accelerator cards. Supermicro has been making similar boards for several platform generations. A typical purchaser of these motherboards are those who need Tesla, Xeon Phi or GPU compute platforms. As such, it is a motherboard and platform we have wanted to test for years. Today we get that chance with the dual Intel Xeon E5-2600 V3 offering.
As we will see, the Supermicro X10DRG-Q has a few features which warrant a few test configuration tweaks compared to our normal setup.
- CPU: 2x Intel Xeon E5-2690 V3
- Motherboard: Supermicro X10DRG-Q
- Memory: 8x SK. Hynix 16GB DDR4 2133MHz ECC RDIMMs
- PCIe cards: 2x Xeon Phi 31S1P, 2x NVIDIA K4000, Mellanox ConnectX-3 FDR VPI Infiniband/ 40GbE
- SSD: 3x SanDisk CloudSpeed 1000E 800GB SATA, 3x Intel S3700 100GB
- Operating Systems: Ubuntu 14.04 LTS, Windows Server 2012 R2, VMware ESXi 5.5, CentOS 6.5
Overall, we needed to change our configuration slightly for this motherboard and utilized a Supermicro SYS-7048GR-TR as our test bed.
The Supermicro X10DRG-Q
Upon unboxing the Supermicro X10DRG-Q, one can tell it is something different. The motherboard measures 15.2″ x 13.2″ in order to give the motherboard enough space for add-in slots. The top portion of the motherboard near the dual LGA2011-3 sockets looks much like a standard dual socket motherboard. The expansion slot area shows why the motherboard is special. We are using a Supermicro SC747TQ-R1620B chassis to house the X10DRG-Q due to its size, and highly recommend that purchasers of the motherboard do the same.
As one can see each socket has eight DDR4 RDIMM slots which can be filled with up to 1TB of memory between the 16 slots. The CPU sockets are offset which helps cool hotter processors. As with any PCIe compute platform, great cooling is essential. One should note that the heatsink mounts are a narrow ILM design. See our guide on narrow versus square ILMs to understand the differences. Since both layouts have been available for years, this is something to keep in mind when configuring a solution.
Getting to the big selling point of the motherboard, the PCIe expansion slot layout is nothing short of impressive. One can see that there are five PCIe x16 physical slots and two PCIe x8 physical slots. Each PCIe x16 slot is two slots away from the next which is important for double width add-in cards. Four of the slots are PCIe 3.0 x16 electrical which provides the data connectivity for larger cards. One of the PCIe x16 physical slots and one of the x8 physical slots are PCIe 3.0 x8 electrical. The final PCIe x8 slot is a PCIe 2.0 x4 electrical slot. The bottom line here is that there are 64x PCIe 3.0 lanes dedicated for the four bottom PCIe x16 slots. This configuration yields 4-way NVIDIA SLI support. The top two slots are PCIe 3.0 x8 electrical. This is one of the few or only platforms that expose a full 80 PCIe 3.0 lanes to users for expansion card customization. One, of course, requires two Intel Xeon E5 V3 series processors to support all 80 lanes (40 per processor.)
The PCIe 2.0 x4 (in a x8) slot has another feature: Thunderbolt support. Supermicro has an add-in card (AOC) dedicated to supporting Thunderbolt support. The small five pin header above the PCIe 2.0 slot is a header for the Thunderbolt ports.
The motherboard’s BIOS has built-in support for Thunderbolt as well.
We took a pair of Intel Xeon Phi 31S1P cards to show off how the spacing practically works. One can see that the cards fit perfectly and do not cover another PCIe slot as they would in a standard motherboard. The cards slot below the DDR4 DIMM slots and leave the three top slots open for PCIe based storage or networking.
One differentiation point for the motherboard is the bottom PCIe x16 slot. There is a double width slot on the X10DRG-Q. The PCB below this slot is relatively bare, so having a double slot PCIe card in the bottom position will not cover up many critical headers like it would on other motherboards.
Moving on, Supermicro utilizes the onboard Intel C612 PCH SATA III ports. There are 10x SATA 6.0gbps ports. Two of the ports have connectors for SATADOM modules. Along this edge there is also a USB 3.0 front panel header and an internal Type-A USB 3.0 header. That provides the USB connectivity one would need in most situations. These headers are located relatively far from the PCIe slots which is a nod to the intended application of the motherboard.
Moving to the rear panel, we see a very standard Supermicro rear I/O layout. This does help if one ever needs to replace the I/O shield. There is a legacy serial and VGA port flanking the other ports. The two blue colored USB ports are USB 3.0. Atop the USB 3.0 ports there is a Realtek controlled IPMI LAN management port. Additionally, there are two USB 2.0 ports for peripherals or USB storage. The two gigabit network ports are controlled by an Intel i350 Ethernet controller. The i350 is a higher quality/ cost part than the Intel i210 NICs that lower cost solutions have onboard.
Overall the Supermicro X10DRG-Q is built for one purpose: housing many PCIe expansion cards. Whether that is four fast NVIDIA GPUs for 4-way SLI or four Tesla/ Intel Xeon Phi cards, the Supermicro X10DRG-Q is equipped to handle high powered systems.
To capture these images we utilize our FLIR Ex series professional thermal imaging camera and turn on FLIR’s MSX enhancements so we can see components outlined clearly. We put the system under 100% CPU load for a period of 24 hours to let “heat soak” set in prior to taking the images.
Even after a full day of fully utilizing the processors, the highest recorded temperature was 44.8C, which is very cool. One can see that there are very few components sitting in front of the four PCIe 3.0 x16 slots. As a result, the Supermciro X10DRG-Q has a well optimized cooling layout for installing four large GPUs or MICs.
Supermicro’s IPMI and KVM-over-IP as described a few times on this site, allows for a lot of deployment flexibility. Things such as fan speeds, chassis intrusion sensors, thermal sensors, and etc. can be monitored remotely. Alerts can be setup to notify the admins of issues. Beyond this, the functionality also allows for remote power control. You can do remote power up, power down, and reset of the server in the event that it becomes unresponsive. In fact, the test system has never had a keyboard, mouse, CD/DVD ROM, or monitor hooked up to it, even after multiple BIOS tweaks and operating system/ hypervisor installations.
Another important feature is the ability to remotely mount CD images and floppy images to the machine over the dedicated management Ethernet controller. This keeps maintenance traffic off of the primary Intel NICs. At the same time it removes the need for an optical disk to be connected to the Supermicro motherboard.
The latest revision of Supermicro’s BIOS has a great feature: the BMC IP address shows up on the post screen! This is a super feature because if one has a KVM cart hooked up to the system, it gives an indicator of which machine one is connected to during post.
Supermicro does include KVM-over-IP functionality with the motherboard. We have been testing servers from HP, Dell, Lenovo and Intel that all required an additional add-in key to get this functionality. It is an absolute time and money saver in the datacenter and other vendors should follow Supermicro’s lead in this space.
We have been wanting to take one of Supermicro’s GPU compute platforms out for a spin and the Supermicro X10DRG-Q certainly exceeded our expectations. The platform booted up the first time in every configuration we tried. Mellanox FDR Infiniband cards, 40GbE Intel Fortville cards, RAID controllers, Xeon Phi’s, NVIDIA Quadro cards we tested all booted without issue. For those seeking a quad Xeon Phi or quad Tesla/ Quadro solution, the Supermicro X10DRG-Q should be atop lists.