Intel Xeon 6900P Memory Subsystem is FAST
The Intel Xeon 6900P has an absolutely screaming memory subsystem. First off, let us talk about DDR5-6400. This is what a DDR5-6400 64GB RDIMM looks like on the front.
Here is the back side. Both sides will look familiar to folks that have been using DDR5 in servers for a few years now.
If you want to learn more about DDR5, see our Why DDR5 is Absolutely Necessary in Modern Servers and the associated video:
New for this generation is the MRDIMM. Here is what one looks like on the front:
Here is the back. On both sides, you will notice the buffer chips at the bottom that make the MCR DIMM/ MRDIMM work.
To understand what is going on here, the DIMMs are able to read from two ranks at the same time instead of one. As a result, the MRDIMMs are able to achieve 8800MT/s transfer rates up from 6400MT/s.
The result is faster memory speeds running at 8800MT/s.
Of course, we should take a second and just note that there is a big caveat here, and it is somewhat strange. JEDEC will have a MRDIMM spec, and the MRDIMMs for Xeon 6 with P cores is actually not that spec, instead it is the MCR DIMM spec. When we use the MRDIMM with Xeon 6, we really mean MCR DIMM. The concepts are largely similar, but it is unlikely when AMD supports MRDIMMs that you will take a Xeon 6 MRDIMM (really a MCR DIMM) and be able to use it in an AMD system.
We covered this for our Substack subscribers, but the current MRDIMM/ MCR DIMM for Xeon 6 is the same thing. We also noted that during some of the standards meetings, MRDIMM was pronounced “Mr. DIMM.” If you do not believe this, here is Micron’s FAQ for its current generation of MRDIMMs for Intel platforms.
On the one hand, we wish Intel kept MCR DIMM. It is a bit confusing to call the MCR DIMM a MRDIMM, when the MRDIMMs are going to be a different standard in the future. With that said, Intel offers MRDIMM/ MCR DIMM, and that is going to take its parts to a new level. Notable here is that AMD is absent. So Micron’s MRDIMMs, albeit MCR DIMMs, are for Intel platforms, but not AMD. AMD will say this is a one generation iteration of a multiplexed DIMM and that it is holding the ground of only adopting once MRDIMMs are mainstream. Intel will counter with the fact that if you need memory bandwidth, it has a high-speed solution in this generation so you do not have to wait for a future technology and generation of chips.
Just to give you some sense of how massive the platform bandwidth is, here is a STREAM run using DDR5.
That is good, but not necessarily superior. Here is the 128-core version.
Those are some big numbers. Scaling to two sockets was around twice that amount as we would expect.
Just to give you some sense, NVIDIA markets its Grace CPU as having huge memory bandwidth. This is the Grace side of a GH200 system with 480GB of memory:
Doubling that STREAM Triad number would give us around 660GB/s for a full Grace Superchip with 144 cores and 960GB of memory. Intel is now closing in on a 960GB Grace Superchip. The benefit of Intel’s solution is that you can customize your DIMM configuration. There is also room to go up from where we are.
P-core, E-Core, Granite Rapids, Sierra Forest, let us get to that next.
Wow can’t even hide Patrick’s love affair with Intel anymore can we ? Intel has not even properly launched this but yet it’s 128c Intel vs 96c Genoa, but AMD will have same 128c in 2 weeks time……just be honest finally and call it servingintel.com ;-)
Yawn… Still low on PCIe lanes for a server footprint when GPUs and NVME storage is coming fast and furious. Intel needs to be sold so someone can finally innovate.
Whether love or not, the numbers are looking good. For many an important question will be yield rates and pricing.
I wonder why Epyc is missing from the povray speed comparison.
One thing I’d like to see is a 4-core VM running Geekbench 6 while everything else is idle. After that Geekbench for an 8-core VM, 16-core, 32-core and so forth under similar circumstances. This sort of scaling analysis would help determine how well balanced the MCRDIMM memory subsystem is to the high-core-count processors–just the kind of investigative journalism needed right now.
As an asside, I had to work over eight captchas for this post.
The keyword would be availability. I checked just now, and these newer parts don’t have 1k Tray Pricing published yet. So not sure when would they be available. It felt painful to restrict the On-Premise Server procurement specification at 64 cores to get competitive bidding across vendors. Hope for the best.
It is hard not to get excited about competition, Intel has finally done it, they launched something faster than AMDs previous generation… Intel’s focus on AMX accelerations seems to have paid off, I guess we shall see when Turin launches in a few weeks.
@Patrick how do you manage to call ~53GB/S “closing in on” ~330GB/S? Even dual GR is slower by a factor of three.
Well there’s 90 minutes of my life well spent. I’d like to thank Patrick and the STH krew on this one.
Rodigas I didn’t get that sense at all. Intel’s the first to launch a 500W TDP part on a modern process and they’ve got cores and memory bandwidth so they’re winning for now. In Q4 when Turin is out we’ll say he loves AMD. It’s shaping up like Intel will at least show it’s competitive with Turin. That’s great for everyone.
Eric O – I’d like to see GB5 not 6. You’ve hit it before, GB6 is useless even for workstation class multicore.
Ram is right, these aren’t really available yet. Paper launch, or HyperScale only launch.
Emerth do you mean that 543391MB/S is much more than 330GB/S? The screenshots in the memory section show Intel’s far ahead. With MCRDIMMs adding 38% more bandwidth they’re getting close to 750GB/s on one CPU. So maybe they meant to say the Grace dual chip is almost up to a GR MCRDIMM single chip?
Intel’s doing a fine job. I can’t wait for 18A chips.
@RamShanker & francis:
– ASUS has a webpage up; search for ASUS “RS920Q-E12”, not quite for sale yet, but there’s a PDF.
– NextPlatform has published a guesstimate of 6980P U$24,980 and 6979P U$24,590; with lower prices for trays. Prices are fairly meaningless ATM with the competitor’s launch imminent.
> Intel is differentiating on things like features depending on the core type and SKU
Glad for the competition, but really wish they’d simplify the stack of SKUs. Is Intel On Demand gone?
Did Intel say why all the black ink on that Clearwater Forest chip?
That seems to be a very risky chip … gaa/bspd/hybrid bonding,18A all being introduced in the same chip. Did they actually demo one?
Never thought I’d see a vulture capitalist group (Apollo Global Management) investing in Intel. I thought Gelsinger was supposed to be Intel’s savior?
As others have pointed out, these seems a bit bias on the Intel side of things.
Yes, we’re all glad to see them finally getting their house in order and competing, but do better on containing your fanfare.
Wow. So many haters claiming bias. Go back and re-read the linked epic Rome review from 2019.
When I compare that to this one and all I see is that good products get good reviews (this one) and great products get great reviews (Rome). I also noticed how thankful Patrick is to have intel be competitive in the top of the line, which it is with this latest launch and how awesome it was back in 2019 to have AMD jump out and surpass intel just a few years after they were nearly bankrupt.
For detailed benchmarks I refer all to Phoronix – but a very nice piece by ServeTheHome.
many thanks, L
So, different kinds of “leadership” …
According to Micheal at Phoronix this year’s Intel 6980P is 12% faster than last year’s AMD 9684X.
But, the 6980P has 700 TDP and the 9684X has 400 TDP (while remembering that comparisons of their TDPs isn’t exactly equal) and AMD costs U$10K less. So, 75% more TDP and 5x more $ (unfairly comparing guestimated MSRP vs discount pricing). With the new Turin (coming RSN) offering moar Coors and a big bump over AMDs last generation; in the same socket.
Making a tortoise and hare comparison would be confusing as to who is who and who is ahead at a particular point in time.
We appreciate the effort it takes to put together these articles and enjoy reading them; except for the shameful core latency mushy pea soup image, while other sites has tack sharp puny numbers and a reasonable sized image file nonetheless.
I need Intel to go up so I can give Grandma her retirement back…
We all know Turin is coming. At least AMD now needs to push really hard instead of just coasting because Intel’s been so far behind. Let Intel have its weeks at the top.
On the plus Epyc now has some competition coming. The one big pain point will be software licensing where it’s licensed per Core.
What is up with that lscpu output for the SNC3 configuration? It reports:
Node 0: 43 threads
Node 1: 43 threads
Node 2: 73 threads
Node 3: 86 threads
Node 4: 86 threads
Node 5: 84 threads
And then threads 256-352 are completely unaccounted.
@emerth: I see 0.5TB/s in stream on 128 cores while NVLD seems to go to 0.6 TB/s — so I’d agree with “closing” here.
Intel fanboys forgot AMD Turin with 192 cores? That is always the case, Intel concentrated for quarter year profits instead of keeping R&D on good shape. Now better to concentrate selling factories to someones that need “old school” stuff. Game over. There could be some ligth if they could boost soon out 256 c, which is very unlikely. AMD will do it soon anyway, most likely minor change to just add 20 % more cores. But fanboys are fanboys and always forgotting the truth.