3rd Generation Xeon Scalable Big UPI Link Update
The UPI change is perhaps the other big feature with this generation. One gets 6x UPI links. These links still operate at 10.4GT/s which makes sense if one thinks of this as a Cascade Lake refinement. With Ice Lake and PCIe Gen4 companies will have to do a lot of signaling work in and out of sockets and enhance motherboard PCB quality so that is a more logical time to move to 11.2GT/s UPI.
To understand why six UPI links are important, we can go back to the original Skylake multi-socket topology diagram from our 2017 Intel Xeon Scalable Processor Family Platform Level Overview.
As you can see, in a 4-socket 3x UPI server, each CPU uses one of its three UPI links to communicate with each of the other three CPUs in a server. Conversely, in a 2-socket configuration, either two or three UPI links are available between sockets. The Cascade Lake “Refresh” SKUs had only two UPI links, as did the original Xeon Gold 5100/ 5200 series for example. With a 6x UPI configuration, one can have three pairs of UPI links meaning each CPU can use two UPI links to communicate with every other CPU. That is the magic of Cooper Lake 4-socket. Whether you are using a Cascade Refresh CPU or a 4-socket Cooper Lake CPU, you get two socket-to-socket UPI links. Of course, one can use non-refresh SKUs in 3x UPI dual-socket configurations and get more bandwidth, but that is more of a niche case these days.
8-socket configurations are available, with Cooper, but many fewer vendors will support those. For example, Lenovo is not announcing an 8-socket platform even though it has an 8-socket Cascade Lake platform. That is partly due to customer refresh cycles.
3rd Generation Xeon Scalable Memory Update
Perhaps the third big prong of the Cooper Lake change is the memory update. There are both speed changes as well as updated DCPMM support. It will also mark the end of an era on the memory side with a bridge to the future.
The first big change on the memory side is that we get some DDR4-3200 support. That DDR4-2933 support is not on all SKUs, just the Platinum parts. A cynic could say that it is only reserved for the high-end SKUs on the highest end parts so that Intel could claim DIMM speed parity with AMD and a slew of Arm vendors. In 2020, the fact is that DDR4-3200 is already a fairly mature technology, but Intel was planning for DDR4-3200 in Whitley which is being released later than anticipated. From what we understand, just getting DDR4-3200 support at all is something that the Intel engineering teams had to push to be able to get out of the current memory controllers. For standard RDIMMs it is 1DPC (DIMM per channel) support but if using LRDIMMs or 3DS memory we expect this to work in 2DPC mode. Remember, Intel Xeon memory controllers have an additional task, they need to support Optane DCPMM as well so Intel engineers have additional complexity.
On the Optane DCPMM support, we get Intel Optane DC Persistent Memory 200 Modules in 128GB, 256GB, and 512GB capacities again. These are now called the “Data Center Intel Optane Persistent Memory 200 series”, “Intel Optane PMem 200 series.” Given that, it seems like the Optane DCPMM first-generation will be the Optane PMem 100 series. The new modules were codenamed “Barlow Pass.”
A few notes. First, Optane PMem uses a new controller. The PMem 100 “Apache Pass” modules utilized a controller codenamed “Elk Valley.” With the new PMem 200 “Barlow Pass” generation we get the “Barlow Valley” controller. The key here is that it enables more bandwidth per channel and better performance at lower power consumption than the first generation part. Intel is saying 38% performance increase per channel and 25% more memory bandwidth in aggregate than previous generations. A big portion of that gain is the move to DDR4-3200 which we will discuss later. The other portion is the faster silicon.
Intel, says Barlow Pass or Optane PMem 200 supports 3200MT/s speeds. Intel also says it supports Memory Mode. Those are both true statements, however, not for Cooper Lake which is being launched alongside the new modules. Breaking these down:
- Optane PMem 200 is limited to 2666MT/s support, as it is in the Cascade Lake Xeon generation (2nd Generation Intel Xeon Scalable.)
- Optane PMem 200 does not support Memory Mode on Cooper Lake, nor obviously, any hybrid/ mixed mode. On the 3rd Generation Intel Xeon Scalable designs, PMem 200 is limited to App Direct mode. One will notice that the benchmarks and case studies Intel offered with PMem 200 are for applications such as SAP HANA. SAP HANA is App Direct only.
Of course, let us be 100% clear here. Memory Mode in Cascade Lake where the DCPMM or now Optane PMem 100 is used in a non-persistent manner to simply extend a server’s memory footprint is very popular. Intel has logic to push cold memory pages to Optane PMem 100 modules while keeping hot pages in DRAM. For many, including some of STH’s future hosting servers (we just got PMem 100 modules the day before this will go live for this purpose) this is a great feature that greatly reduces memory costs. Rumor has it, a large percentage of buyers use Memory Mode. That is not available in Cooper Lake.
Since Memory Mode is listed in PMem 200 specs, we thus know it will make a comeback in the mainstream Whitley/ Ice Lake Xeon generation.
With the same 2666MT/s speed and no Memory Mode, one may logically wonder if first-generation Optane PMem 100 modules can be used in Cooper Lake. Officially, this is not a supported configuration. Interestingly enough, one also cannot utilize PMem 200 in 8-socket configurations either. If you buy a 512GB PMem 200 part with Cooper, it is actually less flexible than a 512GB PMem 100 part with Cascade Lake with respect to Memory Mode.
While Intel is technically correct marketing that the Optane PMem 200 modules are capable of 3200MT/s and Memory Mode, and that they can be used with Cooper Lake, there is a fine line here. All of that cannot happen together. There are likely a huge number of journalists and analysts, even technical ones, who will miss this. We confirmed the above details with Intel prior to the announcement. Intel confirmed all of the above except Ice Lake Xeon supporting Memory Mode, but since Memory Mode is in the product brief, we know when that is coming back since it is the only logical place for that to re-appear.
Next, let us focus on PCIe support.