AMD EPYC 7002 Platform and Architectural Updates
The AMD EPYC 7002 is socket compatible with previous generation SP3 platforms. One can install an AMD EPYC 7502P, for example, in a first-generation AMD EPYC server. At the same time, one will likely not get full functionality because the platform has changed. AMD has said the Rome socket is the Milan socket. That tells us that the next-generation Milan AMD EPYC 7003 series is a PCIe Gen4 part. It also tells us that investment in this generation will be socket-compatible with the next generation parts. We will have more on the look-ahead at the end of this article.
AMD EPYC 7002 Platform Updates
We are going to start this section looking at the platform because that is how most of our readers will experience the chips. We already previewed the big revel today in Why AMD EPYC Rome 2P Will Have 128-160 PCIe Gen4 Lanes and a Bonus. We are going to discuss this more about how the second generation AMD EPYC 7002 series is creating a more scalable platform than the previous generation EPYC and current Intel Xeon Scalable.
Here is the overview slide of the AMD EPYC 7002 series. Headlining this release are up to 64 cores/ 128 threads per socket, large caches (up to 256MB per socket), on a quarter of the NUMA domains, reasonable power consumption in the 120-225W range, more floating point performance, PCIe Gen4 support, again at 128 lanes per CPU, and faster memory. That was an enormous sentence. You can see the SKU list in the previous section for full specs.
What AMD is doing in this generation is moving DDR4 memory controllers, PCIe Gen4 I/O, cross-socket interconnects, and even the SATA lanes to a centralized 14nm I/O die. Beyond that, there can be up to 8x 7nm chiplets with the x86 compute that are connected directly to the I/O die.
Key to this is that AMD is presenting a more modern platform architecture. AMD is designing the I/O die as the center of the server. The x86 compute cores hang off of this die like PCIe Gen4 devices, and system memory. In contrast, Intel looks as the x86 die as the center of a computer. We know the future is going to lead to disaggregation with features like CXL interconnects where direct memory access models will work across a system. AMD is already shipping an architecture that follows that future design principle, even if it is not supporting CXL in this generation. AMD’s design is between modern and forward-looking. Intel’s current Xeon lineup is what we will look back on in five years and tell stories that sound like “back in the old days we used to have monolithic die processors.”
AMD is being very aggressive because by being an early mover to the modern system architecture, it has a massive advantage in scaling up in a socket and system over Intel.
We wanted to note here, Intel is moving to chiplet. You can read about Intel Foveros as the direction Intel is moving. AMD is just there in this segment quarters ahead of Intel.
Moving to the I/O die changes the NUMA design of the server. The AMD EPYC 7002 series is presented, by default, as a single NUMA node per socket, down from four in the previous generation. We are going to show tuning this in our later topology section. That change means that there is only 104ns for the same socket latency and 201ns socket-to-socket. This is a 15-19% improvement over the previous generation.
Perhaps the biggest change is the I/O subsystem. The diagram will look similar to what we saw in the AMD EPYC 7001 series. In this generation, AMD has PCIe Gen4 giving a 2x per lane improvement over Intel Xeon. AMD also has significantly more lanes. AMD gets up to 128 lanes + an additional lane for the BMC in single socket, and 160 lanes + 2 lanes for BMCs in dual-socket platforms. You can read more about the 160 + 2 configuration in Why AMD EPYC Rome 2P Will Have 128-160 PCIe Gen4 Lanes and a Bonus.
There is a big catch here. AMD EPYC 7002 platforms need to be a newer spec to take advantage of these features. For example, your first-generation EPYC servers cannot get access to the additional PCIe lane for the BMC by simply adding a second-generation AMD EPYC CPU. Instead, one needs an updated platform. AMD does not broadcast this, but the PCIe Gen4 and DDR4-3200 memory speed support almost always will require a new motherboard PCB to support the higher-speed signaling. We see most of the PCIe Gen4 systems to support the new 2nd Generation AMD EPYC 7002 features.
On the memory side, AMD also leapfrogs Intel Xeon. The 2nd generation Intel Xeon Scalable processors support up to DDR4-2933 when Intel Optane DCPMM is not used and only DDR4-2666 when Optane DCPMM is used. Intel also is based on a six channel design on its mainstream processors.
AMD now has DDR4-3200 memory speed support and eight channels. With 33% more memory channels, and higher speeds, the AMD EPYC 7002 generation has up to 45% more theoretical memory bandwidth. There is more. The AMD EPYC 7002 series also supports up to 4TB of memory per socket. Intel Xeon Scalable 2nd generation only supports up to 1TB on standard SKUs. Intel charges around $3000 more to address up to 2TB per socket (M SKUs) and $7000 more per CPU to go up to 4.5TB (L SKUs.)
At a platform level, this is what the AMD EPYC 7002 looks like compared to the newest Intel Xeon Scalable CPUs:
Here are the AMD EPYC advantages shown on the chart.
When we talk about AMD having a more expansive platform than Intel, and by a large margin, this is exactly why. These are not 10-15% generational improvements. These are more than double in many areas. This is why we are saying AMD has a disruptive platform.
AMD EPYC 7002 Security Platform Updates
AMD has been pushing its security story. Many are familiar with how the company has been less impacted by speculative execution than Intel (e.g. L1TF and Foreshadow.) Beyond that, AMD is building platforms specifically for security. Part of this is an Arm Cortex-A5 security processor within the AMD EPYC I/O die. This provides a hardware root of trust for AMD platforms. We have seen examples of using AMD’s security features already. This includes the HPE ProLiant DL385 Gen10 and DL325 Gen10 that the company says are its most secure servers when combined with iLO5 features. That is the world’s #2 server vendor saying its AMD EPYC platforms are more secure than its higher-volume Intel Xeon servers.
With the first generation of AMD EPYC, the company announced Secure Memory Encryption (SME) and Secure Encrypted Virtualization (SEV). In the 2nd generation AMD EPYC, the company is expanding those capabilities.
All memory can be encrypted by a single key which makes attacking memory a harder vector on AMD. In this generation, AMD also can reach to PCIe devices to access encrypted pages.
Another generational improvement example, the AMD EPYC 7002 generation has 509 encrypted memory keys which allow for significantly more guest VMs. In the first generation with AMD EPYC 7001, SEV could handle 15 keys for 15 guests. In this generation, the chip can handle 509 keys which is a much more usable number, especially on systems with up to 128 cores/ 256 threads.
In terms of OS Support, most major modern Linux OSes support the feature and we hear that support is coming from VMware and Microsoft.
Overall, AMD is pushing its security story because it says customers are asking for more secure platforms. If that is something your virtualization cluster can benefit from and is in your RFP, then AMD has an advantage.
Next, we are going to look at the microarchitecture before seeing what these platform aspects mean for system topology and memory bandwidth.