Traditional 2.5″ form factors that encompass hard drives and SSDs are on notice. As the market transitions to new generations of PCIe connectivity, EDSFF is going to start taking share from the 2.5″ SSD market. While one could say this is a “future” technology since it will be more prevalent in a few years, that is not necessarily the case. We have the 1U half-petabyte Supermicro EDSFF server in this hands-on look. While it is around a half-petabyte using today’s 15-16TB EDSFF drives, the 30TB+ EDSFF drives are on the horizon which will make these 1PB/ U platforms. In this article, we are going to take a look at the system along with the 15TB Intel SSDs that are being used in the systems today. While this may be more prevalent in the future, you can buy these servers and drives today.
This is part of our visit to the Supermicro HQ post-lockdown series. We previously looked at the Supermicro 3rd Gen Xeon Scalable Server which we showcased alongside the Cooper Lake launch. We also looked at the Simply Double platform and the company’s new 60-Bay Top Loading Storage Server. As a part of that series, we have an accompanying video:
We are going to have more detail in this article, but want to provide the option to listen. As a quick note, Supermicro filmed the video bits at their HQ, provided the systems in their demo room, and their product managers that were able to make it. We did a whole series but are tagging this as sponsored since we relied upon their facilities instead of our own. I was able to pick the products we would look at and have editorial control of the pieces (nobody is reviewing these pieces outside of STH before they go live either.) In full transparency, this was the only way to get something like this done, including looking at a number of products in one shot, without going to a trade show during shelter-in-place. Look for more in this series coming to STH over the coming weeks.
What Makes EDSFF Different and Important
I wanted to take a moment to talk about why EDSFF is different. Your typical 2.5” drive bay was designed for SATA and SAS rotating hard drives. The 2.5” form factor is a size that the industry thought could service both enterprise storage and the notebook segment.
Things have changed. We rarely see notebooks with 2.5” drives these days. Buying SAS hard drives for performance enterprise storage means you are buying drives with lower performance, endurance, capacity, and reliability, with higher power consumption.
EDSFF is designed to be a PCIe standard for SSDs. Primarily, these are NVMe SSDs. The longer and thinner form factor better aligns with the fact that instead of spinning disks, we now have NAND chips on PCB. EDSFF allows for better placement of these NAND packages for optimal cooling. Cooling is important because it can dictate how much power a device can use and the thermals it can dissipate. The better the cooling, the more power a device can use and therefore more performance it can achieve.
Beyond the typical PCIe x4 NVMe SSD, EDSFF is designed for other storage as well. Intel has discussed Optane in EDSFF as another storage type. Beyond that, Intel has also shown off FPGAs in ruler form factors. EDSFF is for storage today, but it is ready for accelerators tomorrow with optional connectivity beyond 4-lane PCIe. Other vendors are starting to get into the EDSFF arena, but Intel has been shipping the ruler form factors to major customers for some time.
1U Half-Petabyte Supermicro EDSFF Server
The server we are looking at is the Supermicro SSG-1029P-NEL32R. This is a 1U server that has 32x EDSFF E1.Long slots grouped into sets of 8 drives. One can see that there are extra spaces in the chassis for airflow.
The drives we saw were Intel 15.36TB units. Intel has been saying that they would have 30TB+ drives for years (see: The Intel Ruler SSD: Already Moving Markets) but as we understand it you can order the system with 15.36TB drives today, but 30.72TB drives are not yet available. When they are, this effectively becomes a 1PB in 1U server.
Looking inside the system, we see the PCIe Gen3 expanders to the middle of the chassis (left in the image below) that are cabled to the main motherboard. There is then a fan partition and two Intel Xeon Scalable (1st and 2nd Gen) processors. Each CPU gets a full 12 DIMM slots which means we get up to 24 DIMMs and can use Intel Optane DCPMM / Optane PMem 100 with the system as well.
We also get dual redundant power supplies and each CPU also gets access to a PCIe Gen3 x16 slot. Part of the concept here is that one can get PCIe lanes to the PCIe Expander and drives then PCIe lanes to a 100Gbps NIC all on the same Xeon CPU avoiding inter-socket NUMA traversal.
We wanted to take a quick note here and just mention that there are other options as well. For example, Supermicro has solutions such as the SSG-136R-NEL32JBF are built in a similar manner except without the server component to make a JBOF configuration. Below you can see an example of the JBOF configuration:
A key item to note here is the modularity. While these show the difference of the rear of the system being customized for server or JBOF applications, the cabling to the PCIe switch boards in the middle of the system is important as well. This allows Supermicro to use the same switch PCBs with different configurations. It is also what can make integration with new PCIe and CPU generations easier as we move to PCIe Gen4.
The Supermicro SSG-1029P-NEL32R utilizes a PCIe Gen3 x4 connector for each drive. Again, this is a PCIe connector so one can, in theory, use other PCIe EDSFF devices in this system so long as they fit in the thermal/ power envelopes. Today the application is SSDs, but tomorrow, the industry is looking to a more expansive ecosystem.
Something we wanted to take a quick moment to show is the latch. Here is a picture of the latch closed and the 15.36TB Intel SSD atop the system.
Here is the latch extended atop the system.
While a latch may seem strange to highlight, it turns out that it is actually a big deal. Where we had drive trays with 2.5″ and 3.5″ drives, with EDSFF we do not. As a result, the latching mechanisms are a big deal among server vendors and they are screwed onto the front of these drives. Being able to provide smooth hot-swap functionality on a device this long requires a mechanically sound latching mechanism.
The ecosystem is looking at EDSFF to provide more than just storage even to accelerators and perhaps even memory in the CXL PCIe Gen5 era. 2.5″ storage form factors will be limited in the not-too-distant future as the PCIe interface is transitioning from PCIe Gen 3 to Gen 4 in 2019/2020 and then the PCIe Gen5 transition will likely start in 2021. Handling what will effectively be a 4x interface bandwidth jump will require devices to use more power and that is where EDSFF will become extremely important.
While that is a look at the future, the impact is here today. Supermicro says it has some large government/ defense sector as well as some service provider customers for EDSFF today. We also know Facebook is adopting EDSFF in its next-gen platform and Microsoft is as well. At some point, being able to fit 32x 15.36TB drives in 1U of rack space is great for density.
While we looked specifically at the new Supermicro 1U EDSFF storage server, the company also has JBOFs and even a 2U4N BigTwin E1.S EDSFF solution.