AMD EPYC Genoa Gaps Intel Xeon in Stunning Fashion

16

AMD EPYC Genoa Zen 4 Performance

There is something that we noticed in this generation that we had to address. When we design benchmark suites, and many workloads you see online ideally try to run one workload over an entire CPU. If all goes well, this would be the result, a 100% x 384 thread system:

AMD EPYC 9654 2P 384 Thread HTOP 100 Percent
AMD EPYC 9654 2P 384 Thread HTOP 100 Percent

In real-world workloads, if you run a single workload on a large chip like this, sometimes there are single-threaded parts of a workload. That leads to some really poor performance on big chips because those workloads look something like this with 1 of 384 threads running at 100%.

AMD EPYC 9654 2P 384 Thread HTOP 1 Core
AMD EPYC 9654 2P 384 Thread HTOP 1 Core

The above is a bigger issue today than it was in the past. On a dual 4-core / 8-thread server, a single thread is over 6% of the total thread count. On a dual 96-core/192-thread server, a single thread is just over 0.26%.

Depending on the workload, these periods where a workload is held up by a single thread became a substantial issue. As a result, we had to do a lot more profiling than we normally do.

The other fun case we ran into was the below view:

AMD EPYC 9654 2P 256 Thread HTOP Challenge
AMD EPYC 9654 2P 256 Thread HTOP Challenge

We found a number of workloads that have scaled for years but were limited to 256 threads. That means 1/3rd of the threads are not being used.

That brings up the philosophical question: “Are these CPUs, especially 96-core CPUs designed to run single workloads?” John and I spoke about this for some time. The world of benchmarking has almost always been running a single workload across the entire CPU. That is rendering workloads, HPC workloads, and so forth that will use entire chips. Still, most of the chips are really being used for containerized or virtualized workloads. The cloud segment is a prime example of this. The key point is that going forward, we are going to be increasingly using bare metal containers and then virtualized workloads to scale. This is similar to what VMware VMmark does, but KVM is the bigger hypervisor with its cloud adoption and VMware puts restrictions on VMmark. Still, looking at both will be important in the future since there is an argument that hitting a single-threaded part of a workload on a 384-thread system is terrible for overall performance.

Python Linux 4.4.2 Kernel Compile Benchmark

This is one of the most requested benchmarks for STH over the past few years. The task was simple, we have a standard configuration file, the Linux 4.4.2 kernel from kernel.org, and make the standard auto-generated configuration utilizing every thread in the system. We are expressing results in terms of compiles per hour to make the results easier to read.

AMD EPYC 9004 Genoa Linux Kernel Compile Benchmark
AMD EPYC 9004 Genoa Linux Kernel Compile Benchmark

This is one of the workloads where we had to look at the scaling performance. In the future, we are going to express more benchmarks in terms of runs/ time because that metric will eventually allow for comparisons where the system is running parallel runs as well.

c-ray 1.1 Performance

We have been using c-ray for our performance testing for years now. It is a ray tracing benchmark that is extremely popular to show differences in processors under multi-threaded workloads. Here are the 8K results:

AMD EPYC 9004 Genoa C Ray 8K Rendering Performance
AMD EPYC 9004 Genoa C Ray 8K Rendering Performance

This looks perhaps much closer than it is. AMD is far ahead because this is not being expressed in runs per hour. A fun note, we started collecting data on this rendering-style benchmark back when 8K renders would stress four-socket servers for many minutes. Now, the new generation completes the run in 13 seconds.

7-zip Compression Performance

7-zip is a widely used compression/ decompression program that works cross-platform. We started using the program during our early days with Windows testing. It is now part of Linux-Bench. We are using our legacy runs here to show scaling even without hitting accelerators.

AMD EPYC 9004 Genoa 7zip Compression
AMD EPYC 9004 Genoa 7zip Compression

Again, this is a stellar performance, albeit we see scaling challenges at higher core counts on the compression side. Compression, however, is a function that will be ubiquitous in the future but will also warrant offloading to accelerators.

Chess Benchmarking

Chess is an interesting use case since it has almost unlimited complexity. Over the years, we have received a number of requests to bring back chess benchmarking. We have been profiling systems and are ready to start sharing results:

AMD EPYC 9004 Genoa Chess Performance
AMD EPYC 9004 Genoa Chess Performance

A major challenge here was that our benchmark stopped scaling at 256 threads. We had to split the benchmark up to run in two 192-thread instances via containers to get the result above. Otherwise, a third of the chip was not being used.

SPEC CPU2017 Results

First, we are going to show the most commonly used enterprise and cloud benchmark, SPEC CPU2017’s integer rate performance:

AMD EPYC 9654 Estimated SPECrate2017_int_base At Launch
AMD EPYC 9654 Estimated SPECrate2017_int_base At Launch

Here, adding more cores, and more clock speeds and memory bandwidth yielded crushing results. We managed to get better than some of the initial estimates that were in the 1590 range. From what we have heard, OEMs doing their full platform tuning will be just shy of 1800 at 1790. That is higher than we got, but it is a crushing figure. AMD will have effectively 3x the top dual-socket Intel Xeon 8380 result in the same socket count. It also means AMD is achieving better performance per core, even when packing cores into 96-core parts.

On the floating point side, we see something similar:

AMD EPYC 9654 Estimated SPECrate2017_fp_base At Launch
AMD EPYC 9654 Estimated SPECrate2017_fp_base At Launch

Here, AMD is expected to get a massive performance jump because of AVX-512. The dual Intel Xeon Platinum 8380’s were much more competitive 40 cores v. 64 cores per socket, because of AVX-512. Now that core counts jumped AMD pops to a more than 2x jump over Intel’s top-end. Remember, Intel is only adding around 50% more cores at its maximum and getting some IPC benefits from the new generation, but it needs to get more than 2x the performance to reach AMD. We also saw some preview publication results from OEMs that were higher, so we added those results as well.

Next, we are going to get to some more complex workloads so see how Genoa performs.

16 COMMENTS

  1. $131 for the cheapest DDR5 DIMM (16GB) from Supermicro’s online store

    That’s $3,144 just for memory in a basic two-socket server with all DIMMs populated.

    Combined with the huge jump in pricing, I get the feeling that this generation is going to eat us alive if we’re not getting those sweet hyperscaler discounts.

  2. I like that the inter CPU PCIe5 links can be user configured, retargeted at peripherals instead. Takes flexibility to a new level.

  3. Hmm… Looks like Intel’s about to get forked again by the AMD monster. AMD’s been killing it ever since Zen 1. So cool to see the fierce competitive dynamic between these two companies. So Intel, YOU have a choice to make. Better choose wisely. I’m betting they already have their decisions made. 🙂

  4. Do we know whether Sienna will effectively eliminate the niche for threadripper parts; or are they sufficiently distinct in some ways as to remain as separate lines?

    In a similar vein, has there been any talk(whether from AMD or system vendors) about doing ryzen designs with ECC that’s actually a feature rather than just not-explicitly-disabled to answer some of the smaller xeons and server-flavored atom derivatives?

    This generation of epyc looks properly mean; but not exactly ready to chase xeon-d or the atom-derivatives down to their respective size and price.

  5. I look at the 360W TDP and think “TDPs are up so much.” Then I realize that divided over 96 cores that’s only 3.75W per core. And then my mind is blown when I think that servers of the mid 2000s had single core processors that used 130-150W for that single core.

  6. Why is the “Sienna” product stack even designed for 2P configurations?

    It seems like the lower-end market would be better served by “Sienna” being 1P only, and anything that would have been served by a 2P “Sienna” system instead use a 1P “Genoa” system.

  7. Dunno, AMD has the tech, why not support single and dual sockets? With single and dual socket Sienna you should be able to be price *AND* price/perf compared to the Intel 8 channel memory boards for uses that aren’t memory bandwidth intensive. For those looking for max performance and bandwidth/core AMD will beat Intel with the 12 channel (actually 24 channel x 32 bit) Epyc. So basically Intel will be sandwiched by the cheaper 6 channel from below and the more expensive 12 channel from above.

  8. With PCIe 5 support apparently being so expensive on the board level, wouldn’t it be possible to only support PCIe 4 (or even 3) on some boards to save costs?

  9. All other benchmarks is amazing but I see molecular dynamics test in other website and Huston we have a problem! Why?

  10. Looks great for anyone that can use all that capacity, but for those of us with more modest infrastructure needs there seems to be a bit of a gap developing where you are paying a large proportion of the cost of a server platform to support all those PCIE 5 lanes and DDR5 chips that you simply don’t need.

    Flip side to this is that Ryzen platforms don’t give enough PCIE capacity (and questions about the ECC support), and Intel W680 platforms seem almost impossible to actually get hold of.

    Hopefully Milan systems will be around for a good while yet.

  11. You are jumping around WAY too much.

    How about stating how many levels there are in CPUS. But keep it at 5 or less “levels” of CPU and then compare them side by side without jumping around all over the place. It’s like you’ve had five cups of coffee too many.

    You obviously know what you are talking about. But I want to focus on specific types of chips because I’m not interesting in all of them. So if you broke it down in levels and I could skip to the level I’m interested in with how AMD is vs Intel then things would be a lot more interesting.

    You could have sections where you say that they are the same no matter what or how they are different. But be consistent from section to section where you start off with the lowest level of CPUs and go up from there to the top.

  12. There may have been a hint on pages 3-4 but I’m missing what those 2000 extra pins do, 50% more memory channels, CXL, PCIe lanes (already 160 on previous generation), and …

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.