3rd Generation Xeon Scalable PCIe Support
Another major external interface point for the CPU is PCIe lanes. These lanes connect all of the NICs, NVMe SSDs, storage controllers (PCIe), and accelerators and other parts of the system.
Since the 3rd Generation Intel Xeon Scalable Cooper Lake parts are 14nm parts largely related to Cascade Lake, we get 48x PCIe lanes. These are still PCIe Gen3 lanes, not the Gen4 lanes that virtually every major AMD and Arm server processor are using in this generation. That 48x PCIe Gen3 lane spec mirrors the first and second-generation Intel Xeon Scalable lines. Intel is likely re-using the existing PCIe Gen3 controller design here.
Over the next 18 months, we expect a massive change for Intel in the PCIe space. Since Intel is re-confirming the vision for Sapphire Rapids Xeon platforms in 2021, a mere 18 months away, we have some sense of what is next. Sapphire Rapids is set to support CXL and CXL is based on PCIe Gen5. As a result, Intel’s mid-2020 Cooper Lake part supports PCIe Gen3. Ice Lake will hit in 2020/ 2021 with PCIe Gen4, but PCIe Gen4 will be supplanted within a year by PCIe Gen5 and Sapphire Rapids.
That has a few very interesting industry impacts. The first is that since AMD effectively has a 5-6 quarter lead over Intel on PCIe Gen4 support, most of the hardware ecosystem outside of Intel is using AMD EPYC 7002 “Rome” as the reference platform for PCIe Gen4. Whereas PCIe Gen3 was built atop many years of Intel Xeon, PCIe Gen4 is different. PCIe Gen4 will see 60% of the time between its x86 introduction to the next-gen introduction lifecycle as an AMD-only standard. That is why the NVIDIA DGX A100 is using AMD EPYC 7002 CPUs. Intel was too late shipping PCIe Gen4 to make the NVIDIA Ampere generation of accelerators. As a result, Ampere servers are going to be heavily AMD EPYC in the near future.
For Intel’s portfolio approach, things are awkward as well. Intel’s solid-state drive division, as an example now has a quandary as well.
These are codenamed “Arbordale Plus” SSDs which means that they are PCIe Gen4 capable parts. Intel is releasing them with Cooper Lake, but there is a good chance that they would be faster on AMD EPYC 7002 Rome CPUs with twice the PCIe bandwidth. While Intel had a huge lead with original PCIe Gen3 NVMe SSDs where Xeon was the de-facto standard, that is not the case anymore. Other vendors are using AMD EPYC 7002 platforms for PCIe Gen4 NVMe SSD validation, not Xeon. Intel either needs to go EPYC 7002 validation first for the next six months or it essentially is way behinds its SSD competition.
As a quick aside here, Intel’s next-gen VROC capabilities that will increase the performance of the solution are not in Cooper Lake. Again, Cooper Lake is built largely upon the Skylake/ Cascade Lake base Those will likely be introduced in Ice Lake since Intel needs an update.
That quandary is not just on the NAND SSD side. The Intel Optane DC P4800X, for example, has been around for years as a PCIe Gen3 device. That was codenamed “Cold Stream” for those wondering. Since we have the new generation of Intel Optane PMem 200, we know Intel has a new controller. We would expect new technology to be introduced to the PCIe SSDs as well, especially given Optane’s popularity in the PCIe array space. The next stream is Alder Stream which Intel has discussed a bit about. Since Alder Stream was designed to go hand-in-hand with Ice Lake Xeons, it is likely a PCIe Gen4 device. Again, Intel has the drives, they will work on Cooper Lake Xeons, but they were designed for PCIe Gen4 which right now is mostly going to be AMD EPYC 7002. Indeed, if Intel waits to release Alder Stram until it has PCIe Gen4, it effectively loses 33% of the remaining time when PCIe Gen4 is the top-end connectivity in the market.
Moving beyond storage, we saw the Intel Ethernet 800 Series 100GbE NIC Launch in 2019. We expect the Intel 800 series (E810) to become available in 2020. Even though it was “launched” in 2019, pushing out Ice Lake Xeon with PCIe Gen4, and with Cooper Lake Xeon being a PCIe Gen3 platform, a dual 100GbE network controller cannot run at full bandwidth on a PCIe Gen3 x16 link, but it can on a Gen4 x16 link. Mellanox (now NVIDIA) has now been in the market with generations of PCIe Gen4 NICs and is now well beyond 100GbE networking. Columbiaville, if you recall will bring features Intel networking desperately needs such as RoCEv2 support along with iWARP for RDMA which is important at these speeds. Beyond the 100GbE adapters with “Rapids” codenames (not to be confused with Sapphire Rapids), we also expect E810 refreshes for 25GbE and 50GbE networking. Based on the E810, these will also be PCIe Gen4 capable in preparation for Ice Lake Xeons and Whitley. With Cooper Lake, these are still stuck at PCIe Gen3.
Looking to accelerators, Intel Agilex was designed for PCIe Gen4 support as are the newly acquired Habana Labs AI accelerators.
The common theme is that Intel’s entire PCIe Gen4 portfolio either will have to go PCIe Gen4 led by AMD EPYC 7002 “Rome” platforms, or they will have their periods of relevance shortened by the fast-following Sapphire Rapids Xeons and PCIe Gen5.
Next, we are going to talk Lewisburg-Refresh briefly, then discuss our final words.
Why doesn’t anyone else have info on the PMem 200 in memory mode that you’re talking about on page 3. STH is the only place reporting that.
Fred we confirmed this with Intel before the piece went live. To be fair, it was very hard to figure out based on what Intel released in their materials.
This is the most in-depth coverage of Cooper I’ve seen. There’s crazy detail here. excellent work Patrick and STH
Does anyone know what the “MOQ” refers to in the Optane specs, the values (4 and 50) align with the existing Optane DIMMs but could never find what it referred to.
Thanks
“a stopgap step”
From the processor king that has been pumping $Bs Q after Q?
AMD for pumping out products, but Intel for pumping in $$$…
@binkyto… MGTFY.com. OH what help a google could be. MOQ typically refers to Minimum Order Quantity. however…if you had actually googled “Optane” + “MOQ”, without the quotes of course, you’d have come to this link:
https://ark.intel.com/content/www/us/en/ark/products/190348/intel-optane-persistent-memory-128gb-module.html
wherein it says:
Intel® Optane™ Persistent Memory 128GB Module (1.0) 4 Pack
Ordering Code
NMA1XXD128GPSU4
Recommended Customer Price
$1499.00
Intel® Optane™ Persistent Memory 128GB Module (1.0) 50 Pack
Ordering Code
NMA1XXD128GPSUF
Recommended Customer Price
$1499.00
4 pack and 50 packs. they have different sku’s. you’r welcome.
Isn’t MOQ minimum order quantity BinkyTo?
Nice writeup STH.
That Optane PMem 200 I’ve read Anandtech, Toms, and nextplatform and none of them mentioned it in their articles.
Fire.
Keep cutting through the marketing BS at these companies.
Can’t wait to see how it stacks up with the exiting parts and Epyc 7xx2
The use of place names to distinguish these products creates confusion. Reading articles about Intel CPUs requires one to have a decoder ring handy.
Intel should stop being obscurantist.
Is the TDP 15W or 18W? If you go to their site it still mentions 18W.
Also some of their datasheet still mentions the specs for the 100 series. Like here: https://www.tomshardware.com/news/intel-announces-xeon-scalable-cooper-lake-cpus-optane-persistent-memory-200-series
Will the rated 8.1GB/3.15GB R/W bandwidth improve with the Icelake platform? Why does the datasheet say it supports up to 2666MT/s?
Intel got hurt bad by their 10nm process. The delays that caused allowed AMD to leapfrog. The question now is who will be first to market with PCIe Gen 5. That will be an enormous advantage in the server space.
I love competition. I really do.
I’m going to have to dig into this further tomorrow, but what has really caught my eye this week is working Sapphire Rapids chips in Intel labs. Sapphire Rapids is supposed to have both DDR5 and PCIe 5, and they are targeting a 2021 release. The reporting on AMD’s roadmap that I have seen has Genoa and the SP5 socket (with DDR5 and PCIe 5) coming out in 2022. I wonder how many months Intel powered servers will have those features before AMD’s get them? DDR5 looks like a huge upgrade. I thought AMD’s technical dominance in servers would last a couple more years, I might have been wrong.
@Wayne Borean: From a business point of view, Amd has done nothing to really stress Intel as regards profits/revenue over the last 3 years since Epyc has been on the market. Most of the marketshare that Epyc has gained over the last 3 years is miniscule when you look at the revenu/profit that Amd earns from Epyc.
If Amd cannot compete on a volume basis with Intel when Zen 3 come out, Intel will continue to dominate in the only ONE true area that counts for any business: increasing revenue/profits over your competitors
Amd has amazing performance compared to Intel but at what cost??
Once Ice Lake server comes out in volume then and only then will we see a head to head competition and based on the last 3 years when Amd clearly had the performance lead, it does not look too good for Amd in the server space!!!