Intel has been hinting at its next-generation Xeon Scalable part for some time. One of the most anticipated features, by a large margin, is the support Intel Optane Persistent Memory. We recently covered the technology in our piece Intel Optane DC Persistent Memory Sampling Today Revenue Delivery 2018. With the introduction of Intel Optane Persistent Memory, one spec must change, RAM capacity. We asked Intel about the capacity at the event and Intel punted. Luckily, we now have some data on what these configurations are going to look like.
Intel Xeon Scalable Skylake-SP Refrefresher
Current Intel Xeon Scalable Skylake-SP CPUs are limited to 768GB of memory with 1.5TB of RAM being addressable by “M” series SKUs. These “M” SKUs are variants of standard chips which can support 12x 128GB per CPU for a total of 1.5TB of RAM. We covered in our Intel Xeon Scalable Processor Family SKUs and Value Analysis that Intel charges a $3000 premium for these high-memory parts.
The new Intel Xeon Scalable platform released in 2017 is called Purley. The Purley platform, like other previous generation platforms from Intel, is designed to support at least two generations of CPUs. Skylake is the first and current generation. Cascade Lake will be the next generation. The Purley platform features six-channel memory with up to two DIMMs per channel.
Originally, the Purley platform with the Skylake generation was slated to support Intel Optane Persistent Memory DIMMs in the higher-end SKUs (e.g. Xeon Platinum) but that support was punted to the Cascade Lake generation.
Intel Xeon Scalable Cascade Lake with Optane Persistent Memory
The Cascade Lake generation we expect Intel to launch in 2018, whether that is a general availability launch or a launch to select partners (hyperscalers) is going to be a good question, but Intel has a mature process in 14nm. We also expect Intel Xeon Cascade Lake generation to have 28 cores and feature a higher TDP which will allow for higher clock speeds. We do not expect that Intel will release PCIe 4.0 support with Cascade Lake, but we may be wrong on that one. Intel has said that the Cascade Lake generation is going to have hardware mitigations for Spectre and Meltdown.
Memory support is the more interesting story. Intel is at a severe disadvantage to other architectures such as AMD EPYC 7000 series, Cavium ThudnerX2, and IBM Power9 in terms of memory capacity and bandwidth. The six-channel memory controller was a 50% improvement over the Xeon E5 generations, but Intel’s competitors went much bigger in their designs.
At the recent event, we asked Intel the obvious question: If Intel Optane Persistent Memory DIMMs (codenamed Apache Pass) will come in 128GB, 256GB, and 512GB capacity points, does that not necessarily mean that the RAM capacity needs to grow beyond 768GB (standard) / 1.5TB (“M” parts)? We were given a fairly generic response, but we can now come up with a good picture of what this looks like.
Getting to 3.84TB Memory in a Socket
Although Intel says they have not shared next-generation memory support specs, they basically have. While we do not have a support matrix saying, for example, which SKU levels (e.g. Platinum, Gold) will support Optane Persistent Memory and how much, we can at least infer the top-end.
Intel showed a picture of a system running 3D XPoint as early as last year. We covered this in Intel Xeon Scalable Processor Family shown off with 3D XPoint. Here is the image Intel shared at that time:
In that picture, Intel disclosed what we expect to be the most popular configuration, six standard DIMMs and six Optane Persistent Memory DIMMs per socket. This six and six configuration makes sense as we do not expect systems to run with only Optane DIMMs for write endurance purposes. Now that we have the Optane Persistent Memory DIMM capacities of up to 512GB, it is easy to extrapolate what this means for the high-end popular configuration.
With 6x 512GB DIMMs per socket, that means that systems will have up to 3TB of Optane per socket, or up to 6TB available in a standard dual socket server. With six sticks of 128GB RAM per socket, that is 768GB. Combined, we get 512GB * 6 + 128 * 6 = 3840GB of memory in DIMM slots per CPU or 7680GB in a dual socket server. That is a 2-4x capacity increase over current generation servers. This is still assuming 128GB DIMMs. We have heard of 256GB DDR4 DIMM development, but those are not generally available at this point.
That has a major impact on systems. We will see 30TB of memory in 2U 4-node systems with Cascade Lake:
What is really interesting from this view is the implication of the QCT QuantaMesh T42S-2U. The other three systems are all following 6 RAM DIMMs + 6 Optane DIMMs per socket. The QCT T42S-2U is using 2x 64GB DIMMs plus 6x 48GB DIMMs per node. Here is the stock photo we had of the motherboard which confirms the memory slot configuration.
128GB DDR4 DIMMs currently sell for around $3500 each street price so a server. Add in 512GB Optane Persistent Memory DIMMs and this configuration will dwarf the CPUs costs, even as those will likely rise in the Cascade Lake generation. We can see how Intel will try to extract more value from its CPUs by charging a premium for “M” series parts in the Platinum and perhaps Gold ranges to support these high-end configurations.
Final Words
Intel has been around for over 50 years. It should be no surprise that the company is building new generations of processors. Each new generations of processors are expected to have more capacity and new features. Cascade Lake will be no different. The 3D XPoint integration with the Optane Persistent Memory is going to be a big deal in the server market as there are an emerging set of use cases that absolutely demand and need the technology.
I thought this was speculation but between that Intel photo and the Quanta slide that says Cascade and Optane and 7.68TB on current models so we know 3647 that’s right.
Whether Optane is memory is debatable but I’ll give y’all a pass
Funny how the EPYC competition work. AMD really kicked Intel into ass. Good for customers!
I cant wait to see some test Web App Postgres test with 1TB Optane Memory + 128GB DRAM. Assuming it is any good all of a sudden all those Database scaling problem are gone or postponed.