ASUS ESC8000A-E11 8x GPU AMD EPYC Server Review

5

ASUS ESC8000A-E11 Internal Hardware Overview

We are generally going to go through the system from front to back, and show some key items along the way, and we are going to pause around the fans since that is a big story here.

ASUS ESC8000A E11 Chassis Internal Overview
ASUS ESC8000A E11 Chassis Internal Overview

Before we get too far, we did want to mention a few items since there are many configuration options here. For example, this is a stock photo of the machine without a PIKE SAS card in front, but a dual M.2 slot option instead. Also, on the GPU side of this chassis, we have the eight PCIe Gen4 x16 slots and can see their cables with fans for the bottom expansion cards underneath. What is different here is that one can see that ASUS has covers between the top and bottom sets of fans to blank off slots when a card is not installed. This helps ensure there is airflow if one has fewer than eight double-width GPUs. We did not have these blanks but wanted to show them.

ASUS ESC8000A E11 Internal Overview Top Down
ASUS ESC8000A E11 Internal Overview Top Down

In terms of CPUs, there are two AMD EPYC CPUs along with a full set of DIMM slots between the front section and the fan partition. These sit on the bottom U of the chassis while the GPUs occupy the top 3U of the 4U chassis.

ASUS ESC8000A E11 Dual AMD EPYC And 32 DIMM Slots 1
ASUS ESC8000A E11 Dual AMD EPYC And 32 DIMM Slots 1

Along with the two AMD EPYC CPUs, we get 32x DIMM slots. We have 16x 32GB DIMMs here, but one can fit up to 32x 256GB of memory for 8TB total.

ASUS ESC8000A E11 Dual AMD EPYC And 32 DIMM Slots 2
ASUS ESC8000A E11 Dual AMD EPYC And 32 DIMM Slots 2

The system supports dual AMD EPYC 7003 CPUs. If you recall, on the front bottom of the system, there were two fans on each side. These fans have ducting to direct airflow over each CPU and are secured with a thumbscrew retention gasket. This design looked strange at first, but it is actually a very unique design that we ended up liking.

ASUS ESC8000A E11 CPU Fan Shroud 1
ASUS ESC8000A E11 CPU Fan Shroud 1

The impact of this is that the system can cool not just eight GPUs, but also two 280W TDP AMD EPYC 7003 Milan CPUs. Some systems have been limited to 225W or 205W TDP CPUs in these dense GPU servers, but this cooling design helps keep these CPUs cool with a 1U heatsink. That is also important because it allows for more airflow to the GPU area by keeping the CPU heatsink shorter and cooled only via the dual fan inlet portion of the front of the chassis.

ASUS ESC8000A E11 CPU Fan Shroud 2
ASUS ESC8000A E11 CPU Fan Shroud 2

In terms of features, you may not expect to see, there is a quite hard-to-access MicroSD card slot on the motherboard behind the DIMMs.

ASUS ESC8000A E11 MicroSD Card Slot
ASUS ESC8000A E11 MicroSD Card Slot

All of the PCIe slots, both for the GPUs and also for NICs/ storage are handled via cabling. So behind the CPUs and memory, we have a large number of PCIe cable headers.

ASUS ESC8000A E11 PCIe Cables
ASUS ESC8000A E11 PCIe Cables

ASUS actually partitions off the chassis into a bottom 1U and a top 3U just behind the CPUs and memory. If one recalls from the overview picture above, the rear PCIe slots are cooled via fans that sit in this bottom 1U of space after some cool air is mixed with the warmed air from the CPUs. That is a very interesting design.

ASUS ESC8000A E11 Area Under Fans
ASUS ESC8000A E11 Area Under Fans

It also leaves four fans to cool the GPUs and a fifth fan primarily cooling the power supplies, but also adding some GPU cooling.

ASUS ESC8000A E11 Fans 1
ASUS ESC8000A E11 Fans 1

These fans are hot-swappable. It may seem like a small design point, but cooling in these systems is a big deal. Using this 1U on the bottom for non GPUs, and the top 3U mostly for GPUs allows ASUS to use a single 3U row of fans to cool the GPUs. We often see fan assemblies in this class of system that have two fans stacked atop each other extending through the 4U of the chassis. This is a truly interesting design.

ASUS ESC8000A E11 Hot Swap Fan
ASUS ESC8000A E11 Hot Swap Fan

These fans push air through GPUs. Each of the GPUs has a retention plate that both secures the GPU with the third point of contact as well as ducts air from the fans through passive GPU heatsinks.

ASUS ESC8000A E11 8x NVIDIA A40 Top Shot
ASUS ESC8000A E11 8x NVIDIA A40 Top Shot

Power is provided by a pair of power cables. One can change these cables depending on the GPUs that one uses. A nice feature is that by being on this bar at the top of the chassis, these are very easy to swap out.

ASUS ESC8000A E11 GPU Power
ASUS ESC8000A E11 GPU Power

Here we have eight GPUs installed. One will notice there is very little room between them.

ASUS ESC8000A E11 8x NVIDIA A40 1
ASUS ESC8000A E11 8x NVIDIA A40 1

These are NVIDIA A40 GPUs. As 300W 48GB Ampere GPUs, they would have been part of the Quadro line in years past. Now, they are simply marketed as the NVIDIA A40, but that is why these are passively cooled GPUs with display outputs.

ASUS ESC8000A E11 8x NVIDIA A40 4
ASUS ESC8000A E11 8x NVIDIA A40 4

We did not have NVLink bridges, but one could, in theory, get bridges to connect GPUs via NVLink here instead of just using PCIe.

Next, let us get to management.

1
2
3
4
REVIEW OVERVIEW
Design & Aesthetics
9.2
Performance
9.5
Feature Set
9.1
Value
9.0
Previous articleSupermicro SYS-220GQ-TNAR+ a NVIDIA Redstone 2U Server
Next articleTyan Thunder HX FT83A-B7129 10x GPU System Review
Patrick has been running STH since 2009 and covers a wide variety of SME, SMB, and SOHO IT topics. Patrick is a consultant in the technology industry and has worked with numerous large hardware and storage vendors in the Silicon Valley. The goal of STH is simply to help users find some information about server, storage and networking, building blocks. If you have any helpful information please feel free to post on the forums.

5 COMMENTS

  1. Very interesting system Patrick. Thanks for making this article about it. It is rather impressive to see this amount of GPU and CPU power, along with full DIMM capacity. It is not clear from the ASUS website or the article, but it appears that this system also repurposes one xGMI link, like the R7525, to reach 160 PCIe lanes.

    The screenshots from the ASPEED AST2600 iKVM look exactly like the Lenovo SR655.

  2. The choicest web hosting as a replacement for small and good province | Hostupp
    If you want to start a web business and use cobweb hosting championing your website, then earliest of all, you have to be au courant of the net hosting services. You should be acquainted with what cobweb hosting is before choosing it as your preferred maintenance provider. If you are looking in behalf of the most suitable trap hosting service in UK that can help you step on it an online self-assurance in prospective, Hostupp is the conservative preference for you. With our web server optimization tools and premium network applications and plugins, we deliver a one and only spider’s web suffer and bear higher conversions with our perspicacious features at affordable prices. We believe in sacrifice scalable solutions without any compromise on quality or deposit through providing 24/7 person support. If you are looking to convert your story into reality, you contain happen to the sound place.

    The Best Hosting Serving in 2021. That is not our words; this is powwow of mouth. We had the most satisfied customers from round 30 countries in the domain who are enjoying our network play and drug demonstrative customer strengthen services since starting operations. Start your stoop proceed promoting success with Hostupp now!

    Super solidly speed, uncontrolled bandwidth, importance apps and plugins

    While comparing to other hosting companies terminated the web, you will find them claiming their server speeds as highest practical ones but when it comes to physical results; they ease up behind a extensive way. With our ultra fast data processing features, dedicated hardware organization and SSD servers we provide a blazingly fast messenger worry hurriedness on all your pleasure ranging from static websites to online stores built with wordpress or magento ecommerce stage, hosting wordpress blogs and wordpress website visitors.

    We are Hostupp – The later of snare hosting

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.