Supermicro SYS-821GE-TNHR 8x NVIDIA H200 GPU Air Cooled AI Server

1

Supermicro SYS-821GE-TNHR CPU and PCIe Tray Overview

The front of the system has the NVIDIA HGX H200 8-GPU on top, but we wanted to start on the bottom with the CPU tray.

Supermicro SYS 821GE TNHR Front Angle
Supermicro SYS 821GE TNHR Front Angle

The entire CPU tray can be removed from the server, again, without removing the chassis. These systems are so big and heavy and have so many network connections, that keeping the chassis in the rack saves a lot of time with service. What is more, if there is an issue with a DIMM or something like that, you can just swap in a new CPU tray and turn the system on again while taking the problematic CPU tray away to replace a component.

Supermicro SYS 821GE TNHR Front CPU Tray Angle
Supermicro SYS 821GE TNHR Front CPU Tray Angle

The CPUs in this system are Intel Xeon, specifically 4th and 5th Gen Intel Xeon Scalable CPUs. For codenames, if you want to search STH archives, these are Sapphire Rapids and Emerald Rapids. While NVIDIA’s reference DGX platform uses Intel Xeon CPUs, some folks prefer AMD EPYC, and Supermicro has this same server in EPYC as well.

Supermicro SYS 821GE TNHR Front CPU Left Intel Xeon
Supermicro SYS 821GE TNHR Front CPU Left Intel Xeon

The CPUs are a bit different here since they have fanned heatsinks. They will still get airflow if the active fan fails, but these fan modules are fairly easy to replace since they are held in place by a single screw.

Supermicro SYS 821GE TNHR Front CPU Right Intel Xeon
Supermicro SYS 821GE TNHR Front CPU Right Intel Xeon

One reason that, even with 12 channel CPUs on the market from both AMD and Intel, folks like 8-channel memory CPUs is that with the smaller width sockets one can get 2 DIMMs per channel. That means 16 DIMMs per CPU or 32 DIMMs total.

Supermicro SYS 821GE TNHR Front CPU Tray DDR5 Memory 1
Supermicro SYS 821GE TNHR Front CPU Tray DDR5 Memory 1

Given the GPUs have 1.1T8GB of HBM3e memory, having 32 DIMM slots makes it easier and less costly to get well over 1TB of DDR5 on the CPUs. We often hear folks who want a ratio of over 1:1 for CPU DDR5 capacity to GPU HBM capacity.

Supermicro SYS 821GE TNHR Front CPU Tray DDR5 Memory 3
Supermicro SYS 821GE TNHR Front CPU Tray DDR5 Memory 3

Flipping the tray around, we can see that behind the CPUs we get the PCIe switch architecture, and below we get an extension of the motherboard.

Supermicro SYS 821GE TNHR Front CPU Tray Connectors 1
Supermicro SYS 821GE TNHR Front CPU Tray Connectors 1

Aside from the PCH, perhaps the most notable feature below is the dual M.2 SSD slots. These are not the easiest to get to, but at least they are tool-less in their installation.

Supermicro SYS 821GE TNHR Front CPU Tray Dual M.2 2
Supermicro SYS 821GE TNHR Front CPU Tray Dual M.2 2

Above the motherboard, and behind the CPUs is the PCIe switch tray. In the center section, there are four Broadcom PCIe switches.

Supermicro SYS 821GE TNHR Front CPU Tray GPU PCIe Switch New Flat MCIO Cables
Supermicro SYS 821GE TNHR Front CPU Tray GPU PCIe Switch New Flat MCIO Cables

This is a server that normally sits in Supermicro’s lab, as a result, we were able to show something neat here. The older style rounded MCIO cables have been common for a few years now. Supermicro is switching to new flat cables to improve airflow. This system has both installed to show the difference.

Supermicro SYS 821GE TNHR Front CPU Tray GPU PCIe Switch Board Old And New Cables
Supermicro SYS 821GE TNHR Front CPU Tray GPU PCIe Switch Board Old And New Cables

On both sides of this system, we have another PCIe switch board. This one connects the CPU to four NVMe SSDs on the front, and a DPU and another NIC on the rear NIC tray.

Supermicro SYS 821GE TNHR Front CPU Tray NIC PCIe Switch Board 1
Supermicro SYS 821GE TNHR Front CPU Tray NIC PCIe Switch Board 1

In these systems, it is fairly common to have either one or two DPUs for the CPU complex, as well as having SSDs for the CPU side.

Supermicro SYS 821GE TNHR Front CPU Tray NIC PCIe Switch Board 2
Supermicro SYS 821GE TNHR Front CPU Tray NIC PCIe Switch Board 2

With six PCIe switches, we have a massive array of high-density connectors on the board.

Supermicro SYS 821GE TNHR Front CPU Tray Connectors 1
Supermicro SYS 821GE TNHR Front CPU Tray Connectors 1

The center connectors are focused on the main four PCIe switches for the GPUs and their NICs.

Supermicro SYS 821GE TNHR Front CPU Tray Connectors 2
Supermicro SYS 821GE TNHR Front CPU Tray Connectors 2

The side connectors for the PCIe switches focused more on the CPUs to DPU or NIC path.

Supermicro SYS 821GE TNHR Front CPU Tray Connectors 4
Supermicro SYS 821GE TNHR Front CPU Tray Connectors 4

We will have more in the topology section, but next, let us get to the NVIDIA HGX H200 8-GPU tray.

1 COMMENT

  1. Did SuperMicro say anything about how they ensure networking reliability, with the optics on the hot-aisle side? Optics are notoriously unreliable in GPU work (look up “network flaps”) with the hot aisle heat and increased dust both likely to be problems.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.