3rd Generation Xeon Scalable PCIe Support
Another major external interface point for the CPU is PCIe lanes. These lanes connect all of the NICs, NVMe SSDs, storage controllers (PCIe), and accelerators and other parts of the system.
Since the 3rd Generation Intel Xeon Scalable Cooper Lake parts are 14nm parts largely related to Cascade Lake, we get 48x PCIe lanes. These are still PCIe Gen3 lanes, not the Gen4 lanes that virtually every major AMD and Arm server processor are using in this generation. That 48x PCIe Gen3 lane spec mirrors the first and second-generation Intel Xeon Scalable lines. Intel is likely re-using the existing PCIe Gen3 controller design here.
Over the next 18 months, we expect a massive change for Intel in the PCIe space. Since Intel is re-confirming the vision for Sapphire Rapids Xeon platforms in 2021, a mere 18 months away, we have some sense of what is next. Sapphire Rapids is set to support CXL and CXL is based on PCIe Gen5. As a result, Intel’s mid-2020 Cooper Lake part supports PCIe Gen3. Ice Lake will hit in 2020/ 2021 with PCIe Gen4, but PCIe Gen4 will be supplanted within a year by PCIe Gen5 and Sapphire Rapids.
That has a few very interesting industry impacts. The first is that since AMD effectively has a 5-6 quarter lead over Intel on PCIe Gen4 support, most of the hardware ecosystem outside of Intel is using AMD EPYC 7002 “Rome” as the reference platform for PCIe Gen4. Whereas PCIe Gen3 was built atop many years of Intel Xeon, PCIe Gen4 is different. PCIe Gen4 will see 60% of the time between its x86 introduction to the next-gen introduction lifecycle as an AMD-only standard. That is why the NVIDIA DGX A100 is using AMD EPYC 7002 CPUs. Intel was too late shipping PCIe Gen4 to make the NVIDIA Ampere generation of accelerators. As a result, Ampere servers are going to be heavily AMD EPYC in the near future.
For Intel’s portfolio approach, things are awkward as well. Intel’s solid-state drive division, as an example now has a quandary as well.
These are codenamed “Arbordale Plus” SSDs which means that they are PCIe Gen4 capable parts. Intel is releasing them with Cooper Lake, but there is a good chance that they would be faster on AMD EPYC 7002 Rome CPUs with twice the PCIe bandwidth. While Intel had a huge lead with original PCIe Gen3 NVMe SSDs where Xeon was the de-facto standard, that is not the case anymore. Other vendors are using AMD EPYC 7002 platforms for PCIe Gen4 NVMe SSD validation, not Xeon. Intel either needs to go EPYC 7002 validation first for the next six months or it essentially is way behinds its SSD competition.
As a quick aside here, Intel’s next-gen VROC capabilities that will increase the performance of the solution are not in Cooper Lake. Again, Cooper Lake is built largely upon the Skylake/ Cascade Lake base Those will likely be introduced in Ice Lake since Intel needs an update.
That quandary is not just on the NAND SSD side. The Intel Optane DC P4800X, for example, has been around for years as a PCIe Gen3 device. That was codenamed “Cold Stream” for those wondering. Since we have the new generation of Intel Optane PMem 200, we know Intel has a new controller. We would expect new technology to be introduced to the PCIe SSDs as well, especially given Optane’s popularity in the PCIe array space. The next stream is Alder Stream which Intel has discussed a bit about. Since Alder Stream was designed to go hand-in-hand with Ice Lake Xeons, it is likely a PCIe Gen4 device. Again, Intel has the drives, they will work on Cooper Lake Xeons, but they were designed for PCIe Gen4 which right now is mostly going to be AMD EPYC 7002. Indeed, if Intel waits to release Alder Stram until it has PCIe Gen4, it effectively loses 33% of the remaining time when PCIe Gen4 is the top-end connectivity in the market.
Moving beyond storage, we saw the Intel Ethernet 800 Series 100GbE NIC Launch in 2019. We expect the Intel 800 series (E810) to become available in 2020. Even though it was “launched” in 2019, pushing out Ice Lake Xeon with PCIe Gen4, and with Cooper Lake Xeon being a PCIe Gen3 platform, a dual 100GbE network controller cannot run at full bandwidth on a PCIe Gen3 x16 link, but it can on a Gen4 x16 link. Mellanox (now NVIDIA) has now been in the market with generations of PCIe Gen4 NICs and is now well beyond 100GbE networking. Columbiaville, if you recall will bring features Intel networking desperately needs such as RoCEv2 support along with iWARP for RDMA which is important at these speeds. Beyond the 100GbE adapters with “Rapids” codenames (not to be confused with Sapphire Rapids), we also expect E810 refreshes for 25GbE and 50GbE networking. Based on the E810, these will also be PCIe Gen4 capable in preparation for Ice Lake Xeons and Whitley. With Cooper Lake, these are still stuck at PCIe Gen3.
The common theme is that Intel’s entire PCIe Gen4 portfolio either will have to go PCIe Gen4 led by AMD EPYC 7002 “Rome” platforms, or they will have their periods of relevance shortened by the fast-following Sapphire Rapids Xeons and PCIe Gen5.
Next, we are going to talk Lewisburg-Refresh briefly, then discuss our final words.