AMD EPYC Genoa Power Consumption
On the power consumption side, Genoa is formidable. We generally saw between 1kW-1.2kW testing dual socket AMD EPYC 9654 systems. That may sound like a lot, but we will note that previous generation top-end Intel and AMD top-end systems were often in the 800W-1kW range.
We also tried bumping TDP from 360W to 400W. Performance was a single-digit percentage better, but we saw 100-120W more at the wall using that mode in dual-socket configurations. With modern air-cooled systems, adding more CPU TDP often comes with a 15-20% “fan tax” for additional power consumption.
On a performance-per-watt basis, this is phenomenal. AMD increasing performance often by 2-3x while increasing power consumption by 20-40% is a trade-off most organizations will make any day.
We are just going to quickly note here that this is not an area where looking at PSU/ PDU power consumption is very important. Increasing power density also increases air cooling requirements in a non-linear fashion. Further, a lot of AMD’s performance comes from the 12-channel DDR5. If we take just package power, we would look at the CPU’s increase in power for performance, but not the additional power being consumed by the memory to achieve that level of performance. In a 24x DDR5 server, it is not uncommon to see memory using well over 100W, or ~10% of system power. The difference between Genoa and Sapphire Rapids will also come at a system power cost of those additional memory channels.
Still, performance per watt has increased greatly with Genoa.
AMD EPYC Genoa Dual Channel DDR5 and More
One point that we learned is that AMD will not be supporting DDR5-4800 2DPC configurations in servers like ours at launch.
This is a symptom of the new DDR5 as well as having more memory channels. It is also why we received many OEM employees sending us notes so intrigued about our Microsoft OCP show Genoa pictures.
AMD’s platform maturity, generally, is not at the same level as Intel’s platform which is launching in 2023. That feels strange to say, but Intel’s Sapphire Rapids has been so delayed, and the Sapphire Rapids ES/QS chips being so prevalent, that vendors have systems that are ready, and are just waiting on chips.
So AMD has something that is more like a minimum viable product in terms of a platform, while Intel has something very mature, but it needs to produce more chips. AMD’s minimum viable product is still good enough (or much better than that) for the majority of the market. Also, fitting 48x DIMMs into a dual-socket server is not an easy feat itself, so there are practical physical constraints that will keep many servers at 1DPC only, like the QCT system we tested.
Market Impact 2023: Big Versus Small Servers
One of the more interesting impacts of Genoa is not at the high end. Instead, at the lower end of the market, Genoa is more challenging. The new EPYC CPUs require PCIe Gen5, with better materials for motherboards. Also, AMD’s new chips have 12 memory channels. For 96-core machines, 12 memory channels feel like the right choice. For a 16-core machine, it feels imbalanced.
Many servers reside outside of hyper-scale data centers. Many customers are not adopting DPUs in 2023. Others are not even adopting PCIe Gen5 NVMe SSDs or other devices throughout fleets. As a result, the new platforms are going to simply be too expensive.
Just for some sense taking a server with 64GB DIMMs and a 32 core CPU. In the AMD EPYC 7003 generation, that is 64GB DDR4 x 8. In the AMD EPYC 9004 generation, that is 64GB DDR5 x 12 to fill memory channels. Current spot pricing for DDR5 is down to around a 50% premium over DDR4. Adding 50% more modules at 50% higher prices is a reason we are seeing things like Non-Binary DDR5 capacities.
AMD’s bet is likely twofold. First, many customers are going to stay on AMD EPYC 7003 Milan in 2023. If an organization has under 512GB of memory per socket demands, does not need DDR5 bandwidth, and is using only 100GbE NICs and a few SSDs, then Milan is going to be fine and perhaps the better option. The second is really the upcoming AMD EPYC Siena line. Many applications in the world transition slower than technology advances. For many single socket 16-32 core servers, as an example, Genoa is too much and Siena will target a class of those use cases.
Whereas we have seen AMD transition for years to new platforms, with Milan, we saw lower-end Rome occupy lower-cost market segments for some time. Now with Genoa getting larger, AMD needs something to address the lower end.
![Pinterest](https://pinterest.com/pin/create/button/?url=https://www.servethehome.com/amd-epyc-genoa-gaps-intel-xeon-in-stunning-fashion/&media=https://www.servethehome.com/wp-content/uploads/2022/11/AMD-EPYC-9654-Genoa-CPU-1.jpg&description=AMD EPYC 9004 "Genoa" gaps Intel Xeon in stunning fashion with 96 cores, PCIe Gen5, and DDR5 and massive performance gains){kind=link}
Any chance of letting us know what the idle power consumption is?
$131 for the cheapest DDR5 DIMM (16GB) from Supermicro’s online store
That’s $3,144 just for memory in a basic two-socket server with all DIMMs populated.
Combined with the huge jump in pricing, I get the feeling that this generation is going to eat us alive if we’re not getting those sweet hyperscaler discounts.
I like that the inter CPU PCIe5 links can be user configured, retargeted at peripherals instead. Takes flexibility to a new level.
Hmm… Looks like Intel’s about to get forked again by the AMD monster. AMD’s been killing it ever since Zen 1. So cool to see the fierce competitive dynamic between these two companies. So Intel, YOU have a choice to make. Better choose wisely. I’m betting they already have their decisions made. 🙂
2 hrs later I’ve finished. These look amazing. Great work explaining STH
Do we know whether Sienna will effectively eliminate the niche for threadripper parts; or are they sufficiently distinct in some ways as to remain as separate lines?
In a similar vein, has there been any talk(whether from AMD or system vendors) about doing ryzen designs with ECC that’s actually a feature rather than just not-explicitly-disabled to answer some of the smaller xeons and server-flavored atom derivatives?
This generation of epyc looks properly mean; but not exactly ready to chase xeon-d or the atom-derivatives down to their respective size and price.
I look at the 360W TDP and think “TDPs are up so much.” Then I realize that divided over 96 cores that’s only 3.75W per core. And then my mind is blown when I think that servers of the mid 2000s had single core processors that used 130-150W for that single core.
Why is the “Sienna” product stack even designed for 2P configurations?
It seems like the lower-end market would be better served by “Sienna” being 1P only, and anything that would have been served by a 2P “Sienna” system instead use a 1P “Genoa” system.
Dunno, AMD has the tech, why not support single and dual sockets? With single and dual socket Sienna you should be able to be price *AND* price/perf compared to the Intel 8 channel memory boards for uses that aren’t memory bandwidth intensive. For those looking for max performance and bandwidth/core AMD will beat Intel with the 12 channel (actually 24 channel x 32 bit) Epyc. So basically Intel will be sandwiched by the cheaper 6 channel from below and the more expensive 12 channel from above.
With PCIe 5 support apparently being so expensive on the board level, wouldn’t it be possible to only support PCIe 4 (or even 3) on some boards to save costs?
All other benchmarks is amazing but I see molecular dynamics test in other website and Huston we have a problem! Why?
Olaf Nov 11 I think that’s why they’ll just keep selling Milan
@Chris S
Siena is a 1p only platform.
Looks great for anyone that can use all that capacity, but for those of us with more modest infrastructure needs there seems to be a bit of a gap developing where you are paying a large proportion of the cost of a server platform to support all those PCIE 5 lanes and DDR5 chips that you simply don’t need.
Flip side to this is that Ryzen platforms don’t give enough PCIE capacity (and questions about the ECC support), and Intel W680 platforms seem almost impossible to actually get hold of.
Hopefully Milan systems will be around for a good while yet.
You are jumping around WAY too much.
How about stating how many levels there are in CPUS. But keep it at 5 or less “levels” of CPU and then compare them side by side without jumping around all over the place. It’s like you’ve had five cups of coffee too many.
You obviously know what you are talking about. But I want to focus on specific types of chips because I’m not interesting in all of them. So if you broke it down in levels and I could skip to the level I’m interested in with how AMD is vs Intel then things would be a lot more interesting.
You could have sections where you say that they are the same no matter what or how they are different. But be consistent from section to section where you start off with the lowest level of CPUs and go up from there to the top.
There may have been a hint on pages 3-4 but I’m missing what those 2000 extra pins do, 50% more memory channels, CXL, PCIe lanes (already 160 on previous generation), and …
Does anyone know of any benchmarking for the 9174F?
On your EPYC 9004 series SKU comparison the 24 cores 9224 is listed with 64MB of L3.
As a chiplet has a maximum of 8 cores one need a minimum of 3 chiplets to get 24 cores.
So unless AMD disable part of the L3 cache of those chiplets a minimum of 96 MB of L3 should be shown.
I will venture the 9224 is a 4 chiplets sku with 6 cores per chiplet which should give a total of 128MB of L3.
EricT – I just looked up the spec, it says 64MB https://www.amd.com/en/products/cpu/amd-epyc-9224
Patrick, I know, but it must be a clerical error, or they have decided to reduce the 4 chiplets L3 to 16MB which I very much doubt.
3 chiplets are not an option either as 64 is not divisible by 3 😉
Maybe you can ask AMD what the real spec is, because 64MB seems weird?
@EricT I got to use one of these machines (9224) and it is indeed 4 chiplets, with 64MB L3 cache total. Evidently a result of parts binning and with a small bonus of some power saving.