ASUS XA NB3I-E12 Power Consumption
We did not have this in our test lab, but it seemed useful to discuss power consumption, given that there are 10x 3.2kW power supplies. First off, those ten power supplies are for 5+5 redundancy, not because this is a 32kW server.
Something that is worth noting, however, is that the idle power consumption is significant. Each GPU idles at around 180W, plus there are the CPUs, DDR5, NICs, switches, and more in the systems that must be cooled via fans. Here is a look at the idle power consumption:
These servers now use well over 3kW at idle. We got over 10kW, but pushing components differently, and adding different configurations can get this to over 12kW, which is quite a bit at around 1.3kW/U. Putting four of these in 36U will require 48-50kW, and five in 45U will put you right on the edge at 60kW/ rack if the systems are all running at top speed all the time. Do not overlook the power of these systems.
STH Server Spider: ASUS XA NB3I-E12
In the second half of 2018, we introduced the STH Server Spider as a quick reference to where a server system’s aptitude lies. For this system, networking and form factor are major parts of the story.
Overall, this server is not designed for storage, nor for the maximum CPU compute density. Instead, it is designed to house a lot of GPU compute and networking. Many will ask why we did not max out the GPU compute and networking density. This is still an air-cooled server, so if you really wanted to maximize the GPU density, you would want a liquid-cooled solution. Here, the server is just below 1 GPU/U of rack space, but we have seen 4 GPUs/U in liquid-cooled racks. Still, for an air-cooled server, this is very dense.
Final Words
We have been reviewing 8x SXM GPU servers for around a decade. It is amazing to see just how far they have come. We now have well over 2TB of HBM3e memory, new number formats and tensor processing, and even 8x 800Gbps for 6.4Tbps of GPU networking in a single machine. Having seen the NVIDIA HGX 8-GPU over the years, this still makes a lot of sense in data centers where NVL72 racks simply cannot be used due to their liquid-cooling and power-density requirements. That is perhaps what ASUS did with the XA NB3I-E12, taking high-end GPUs and allowing them to be air-cooled to fit into more existing data center footprints. ASUS uses its modular server design, which makes servicing easier. While this may just seem like a “neat” feature at first, it is also very practical. Being able to monitor the system and quickly take it apart for service helps keep these high-dollar AI systems working. As systems have become more complex, there are many more service items, which is why this design direction is so important.
Let us face it, the NVIDIA Blackwell Ultra GPUs with the NVIDIA ConnectX-8 networking make for a really cool combination. We learned a lot just by taking this system apart and seeing how it worked. Hopefully, you learned something from this teardown as well.
