Recently, Facebook introduced us to its next-gen Cooper Lake Xeon Platform. For those who are unaware Facebook is expected to be a major Cooper Lake customer (if not the largest) and driving force behind the platform. A few months ago, Intel confirmed first to STH that one of the two Cooper Lake platforms based on its next-gen Whitley platform is going away. Whitley was to be shared with the next-generation 10nm Ice Lake Xeons that will formally succeed the 2nd generation Intel Xeon Scalable processors we have today. Still, the Cedar Island platform is going ahead at Facebook and we have details of how Facebook will utilize the platforms.
Facebook Cooper Lake Generation Server Platforms Video
As with many of our articles this year, we have a video overview of this piece that you can find on the STH YouTube channel:
Feel free to listen to the video if you prefer to learn about these new servers that way.
Why Cooper Lake Matters
Cooper Lake is important for Intel as it is planned to publicly launch in the next month and a half. This is still a PCIe Gen3 14nm chip, but it has at least one killer feature: bfloat16.
This bfloat16 support is perhaps the killer feature of the new Cooper Lake platforms and is something that we expect to see more in future CPUs. Indeed, even the NVIDIA A100 just launched supports bfloat16. The new numerical format uses a sign bit plus a shorter 7-bit mantissa but the same 8-bit exponent as FP32 to provide similar accuracy for AI workloads while minimizing the bits that are moved around the system and computed. That gives both memory utilization benefits as well as about twice the performance as standard FP32.
Beyond this, the fact that Facebook is using the CPU is very important. Indeed, it is in some ways showing us why we are not seeing a Cascade Lake-D part in this generation when we look at the system design. It is also showing us what Facebook is using for mainstream and scale-up applications.
Mainstream 2-Socket Cooper Lake Platform
We have seen the mainstream 2-socket Cooper Lake Xeon Platform for some time. Our first public glimpse was from Facebook/ Intel at its OCP Summit 2019 Keynote.
That 2 socket platform provides a building block for larger systems we now have more information on. You can read more about some of the larger-scale systems in last year’s Facebook Zion Accelerator Platform for OAM.
This the internal view of Facebook’s next-generation 2-socket platform called “Sonora Pass”:
One of the strange bits from the OCP Summit 2020 is that it looks like Intel was trying to censor the fact that Cooper Lake Xeons in this generation have 6 memory channels. Looking underneath the air shroud, we can see that the two sockets each have 12 DIMMs.
Counting DIMM slots, we get a total of 12. Cooper Lake will support the new Barlow Pass Intel Optane DCPMMs which are the next-generation of DCPMMs after the current “Apache Pass” modules we have today. More on that when we discuss the Xeon D replacement use case.
When we zoom into the platform we can see:
- Two OCP NIC 3.0 slots
- Two PCIe FHHL slots
- One 3.5” SATA HDD slot
- Two USB 3.0 ports and OCP USB debug port
- Up to eight 40x40mm fans
- our E1.S 25mm for expansion
In the FHHL slots we can see M.2 accelerator modules. There is even a hard drive slot between the two PCIe slots. Boot options can include a SATA hard drive (via PCH) or M.2 SATA or NVMe boot devices. For storage, Facebook is adopting the E1.S 25mm so it is moving beyond just M.2 for storage here. Facebook’s decision to adopt E1 is also a likely reason we see E1.S announcements such as the Samsung PM9A3 this week.
A very interesting feature is that the platform is designed to be integrated into either 21″ or 19″ racks. You can see from the pictures that the motherboard does not reach across the entire chassis which helps enable that flexibility. A major step in OCP thinking is how to leverage designs across 21″ and traditional 19″ form factors. Higher volumes mean lower costs and increased quality. That is a driving force that will help OCP adoption.
Looking at the block diagram, we can see the SKT-P+ CPX Cooper Lake sockets tied via 10.4GT/s UPI links. That is the same UPI speed as current Cascade Lake Xeons which use Socket P.
We also see that we have PCIe Gen3 support even for the OCP 3.0 NIC slots. Counting lanes, it looks like these are 48x PCIe Gen3 designs as well. We also still have a PCH even as AMD and Arm competitors have moved to PCH-less designs in this market. The BMC is a standard AST2500/ AST2520.
Sonora Pass is a flexible platform so it seems as though we have a very good sense of the upcoming Cooper Lake platform’s key features.
While dual-socket servers are interesting, Facebook is using Cooper Lake somewhere else that may impact Intel Xeon D sales. We are going to explore that on the next page.