Our review of the Gigabyte R181-NA0 is going to cover one of our favorite types of platforms. With the Intel Xeon Scalable generation, we saw more than a simple processor transition. At the same time as the new platforms rolled out, we saw a shift to NVMe SSDs in a significant way. While previous generations utilized SATA SSDs in 10x 2.5″ arrays, with the Intel Xeon Scalable generation, the market transitioned to NVMe-first.
Why 10x NVMe 1U Systems Matter
Before delving into this review, we wanted to discuss why the 1U 10x NVMe form factor is unmistakably the form factor we like right now. Case and point, we have had an entire cluster of 1U 10x NVMe systems running as part of our DemoEval service for a software-defined storage client for over a year.
Utilizing 10x NVMe drives means a system will need to use 40x PCIe 3.0 lanes. Technically this was possible from a single socket Intel Xeon E5-2600 series CPU (V1-V4) however, it is not practical. The Intel Xeon E5 series was limited to 40x PCIe lanes per CPU so that left other connectivity, such as NICS, connected to the PCH for very limited performance. That has essentially made dual socket Intel platforms, like the Gigabyte R181-NA0, the go-to NVMe architecture. The AMD EPYC “P” series parts change this equation. However, when utilizing 100GbE, the four NUMA node design can still present some tuning challenges. With the Intel Xeon Scalable generation, we now have 48x PCIe 3.0 lanes per CPU which have made 10x NVMe 1U systems even more exciting.
Beyond the processor architecture changes, Intel Xeon Scalable launched at the long-predicted changeover from SATA SSDs to NVMe SSDs. In the past few quarters, and by most accounts what will continue, is the gradual decline of SATA SSDs in servers. Here is a forecast slide we highlighted from Toshiba, a maker of SSDs that shows this trend.
SATA has no roadmap past SATA III 6.0gbps yet today’s controllers, and NAND can push a PCIe 3.0 x4 bus to its limit around five or six times faster.
That brings us to why the 1U 10x NVMe server is so important. One can utilize 10x NVMe SSDs, a large amount of RAM, and 25/40/50/100GbE networking to build out high throughput software-defined storage and hyper-converged clusters. 2U 24-bay Intel designs use PCIe switches. As we saw with the How Intel Xeon Changes Impacted Single Root Deep Learning Servers piece, the new Skylake-SP CPUs (and AMD EPYC for that matter) can only handle a PCIe 3.0 x16 backhaul from PCIe switch to CPU. In a 2U 24 bay chassis that means one has either 16 or 32 PCIe lanes to the drives. In effect, one adds the latency and power consumption of PCIe switches and loses bandwidth. The 1U 10x NVMe solutions do not need PCIe switches and thus provide full bandwidth to the drives.
With potential storage bandwidth hitting 30GB/s these are simply awesome hyper-converged and software-defined storage building blocks. The 1U form factor also yields higher memory and compute density than the 2U designs.
In this review, we are going to focus on how the Gigabyte R181-NA0 can fulfill these roles.
Gigabyte R181-NA0 Test Configuration Overview
Gigabyte sent us the base server for review, and we outfitted it with a few different configurations.
- Server: Gigabyte R281-G30 2U server
- CPUs: 2x Intel Xeon Platinum 8180, 2x Intel Xeon Gold 6152, 2x Intel Xeon Gold 6138, 2x Intel Xeon Gold 5119T, 2x Intel Xeon Silver 4116, 2x Intel Xeon Bronze 3106
- RAM: 192GB in 12x 16GB DDR4-2666MHz RDIMMs
- SATA III Storage: 2x 128GB SATA DOMs
- 25GbE NIC: Broadcom 25GbE OCP Mezzanine
- 40GbE NIC: Mellanox ConnectX-3 EN Pro
- 100GBE NIC: Mellanox ConnectX-4
- NVMe SSDs: 10x Intel DC P3320 2TB, 10x Intel DC P3520 2TB, 10x Intel P3700 400GB
Given the focus on the review, we wanted to show NVMe performance. One realization we had was that we have many different types of SSDs. We did plug and operational testing also using: Intel Optane 900p 280GB/ 480GB, Samsung XS1715 400GB, XM1725 800GB, and Intel DC P4510 2TB and 8TB drives as well. We only had three sets of 10x drives available in the lab but they range from extreme value to high endurance SSDs.
As we go through the server hardware, we are going to show how networking and expansion options abound in the platform. The server has 1GbE networking built-in but we see the vast majority of servers based on this platform being deployed with 25/40/50/100GbE so we wanted to be sure to work with that. Unlike the Gigabyte R281-G30 we reviewed, the Gigabyte R181-NA0 only has one accessible OCP slot because the other OCP mezzanine slot is used for NVMe connectivity. We also had to swap out the 100GbE and 40GbE NICs for our tests.
On the RAM side, the system supports up to 24x RDIMMs. We used a single DIMM per channel configuration but there is room for significant expansion here. We also used a variety of CPUs, however, we think that if you are buying this server you will move up the Xeon CPU stack well beyond the Xeon Bronze series. We still wanted to show a broad set of performance scenarios so you can get a sense of different options.
Next, we are going to delve into the hardware details. We will then look at the management interface and a block diagram of the platform. We are then going to look at performance, power consumption, and then give our final thoughts on the platform.