NVIDIA ConnectX-8 C8240 is certainly a SuperNIC. With two QSFP112 ports, a PCIe Gen6 switch, and the ability to even run at 800Gbps bidirectional speeds in PCIe Gen5 servers, this NIC has a lot to offer. This is a review we have been working on for quite some time at STH, since pushing 800Gbps unidirectional and 1.6Tbps combined bidirectional bandwidth through servers with PCIe Gen5 is far from trivial. Luckily, we can now get data on high-end devices like this.
Just as a quick thank you to NVIDIA for sending the cards and to our great STH community for helping with getting the second Gen5 x16 links working. This is a review that has taken months to do simply because we wanted to do something really cool.
NVIDIA ConnectX-8 C8240 Dual 400GbE Hardware Overview
The card itself is a low-profile card that uses either full height or low profile brackets.
Something else that is very different in this generation is that the NIC looks a lot more like NVIDIA’s data center GPUs. Instead of a green or black Mellanox-esque PCB with plenty of components, the components are now housed under a gold and black NVIDIA shroud.
The model we are looking at is the C8240. That model number roughly translates to ConnectX-8 2-port 400G. As a result, we have dual QSFP112 ports. The card is capable of using these ports at 400GbE speeds, or using them for InfiniBand. Many years ago we went into Change Mellanox ConnectX-3 VPI Cards between Infiniband and Ethernet in Windows and later Changing Mellanox ConnectX VPI Ports to Ethernet or InfiniBand in Linux. That is one of NVIDIA’s (formerly Mellanox’s) key differentiators. You can deploy one model across Ethernet or InfiniBand networks.
Perhaps more commonly seen on STH is the C8180 model. That is ConnectX-8 1-port 800G. That uses an OSFP flat-top 800G port that you can split into two 400GbE ports. One day, we might get to review that model as well.
The other end is extremely important. First off, this is where the airflow from a server will have to go through to get to the heatsink on the device. As you can see, the opening here is quite narrow. Also, 400G optics can use quite a bit of power, and the cooling for the optics also goes through this airflow channel. You need a good amount of airflow going through the ConnectX-8’s to cool them
The more impactful depiction here, for our review. On our units, we have the x16 connector. For many generations, Mellanox and now NVIDIA NICs have had multi-host capabilities. You can connect multiple CPUs to a ConnectX series NIC that supports it, and they can share a single NIC. That use case, and the related use case of driving two 400GbE ports from a PCIe Gen5 server requires additional PCIe lanes. Off the onboard 48-lane PCIe Gen6 switch, we can use this connector to deliver an additional PCIe Gen5 x16 link to the NIC, beyond the CEM edge connector.
NVIDIA has a kit to allow this. We also found a Lenovo custom kit that adds additional flexibility, but it is for Lenovo servers. As part of this effort over the last few months, we had custom cables made so that we could more densely configure our Supermicro Hyper server with more high-speed network interfaces. More on that in the next section of this review. Just getting that sorted was a massive effort.
Key to understanding why we had to do this is the PCIe x16 connector. While this is a PCIe Gen6 x16 connector, today’s servers are PCIe Gen5. To drive both 400G ports at 400G speeds simultaneously, we needed twice the number of PCIe lanes that the edge connector offers.
Let us get to the options for doing that next.
