Dell EMC PowerEdge R7415 Topology
Although the Dell EMC PowerEdge R7415 is a single socket solution, for those accustomed to Intel Xeon designs it is fundamentally different. Any thorough review of AMD EPYC 7001 series servers needs to address this difference in detail since it is perhaps the biggest difference between AMD EPYC and Intel Xeon systems today.
Each AMD EPYC 7001 series CPU uses a four NUMA node implementation. You can read more about why in our AMD EPYC 7000 Series Architecture Overview for Non-CE or EE Majors article or learn about it in this video:
That helps AMD deliver a tremendous value since it can stitch together smaller pieces of silicon and make bigger chips than Intel can, at a lower price point. The impact of this shows itself in a key area: NUMA topology. As a thorough review of an AMD EPYC 7001 series server, we felt the need to show the topology. Here is the layout of the Dell EMC PowerEdge R7415:
As you can see, each NUMA node in our test system is comprised of 8 cores and 16 threads. There are four DIMM slots per NUMA node. In our case two are populated for each with 32GB RDIMMs which means each NUMA node shows 64GB of RAM.
The larger implication is the PCIe root complex. From this diagram, you can see the NVMe drives attached to NUMANode P#2 and NUMANode P#3. The 10GbE and 1GbE NICs plus the Dell PERC H740p are connected to NUMANode P#2.
The implication of this is that when workloads or PCIe devices attached to one die need to access RAM or a NIC on one of the other three NUMA nodes, data must traverse the AMD Infinity Fabric. AMD Infinity Fabric is the magic AMD uses to make this setup work. For those coming from an Intel Xeon world, this effectively looks like how data flows around a 4-socket Xeon system. Here is an example from our piece AMD EPYC Infinity Fabric Latency DDR4 2400 v 2666: A Snapshot.
In many cases, users will notice little to no difference between Intel and AMD, but there are a few cases where this matters just like it does in Intel 4-socket systems.
The other impact, aside from latency, is that RAM accessed Infinity Fabric is not as fast as local RAM bandwidth. Here is the 4×4 matrix of memory bandwidth between cores. Local access will be around 32.8GB/s on each NUMA node. Remote access will be around 19GB/s in this architecture.
This has an implication for minimum RAM population. With single socket AMD EPYC 7001 series systems, like the Dell EMC PowerEdge R7415, we suggest filling all eight memory channels with DDR4-2666 memory. Here is why:
- AMD Infinity Fabric between each die on the EPYC CPU is driven by memory speed. You want this fabric to run as fast as possible. Use DDR4-2666 with the AMD EPYC 7001 generation of CPUs.
- AMD EPYC will function with a single DIMM per socket, but performance will have a severely negative impact due to three of the four sets of cores having to access RAM over the Infinity Fabric.
- Intel Xeon functions well with a single DIMM because it is currently a single die design. With AMD EPYC, you will want at least four DIMMs per socket to ensure that each of the four dies has RAM directly attached to their memory controllers.
For those looking to purchase a Dell EMC PowerEdge R7415 here is the short version: order 8 DIMMs in the PowerEdge R7415. If you need to order less RAM for cost reasons, get at least 4 DIMMs. Many of the AMD EPYC HPE Proliant baseline configurations do not follow this advice and perform poorly in our lab as a result.
We are going to have a dedicated piece on why in the near future. Stay tuned to STH for that. In the meantime, that is the key item you need to know as an output of our forthcoming analysis.
Next, we are going to take a look at the Dell EMC R7415 management. We are then going to look at the performance, power consumption, and our closing thoughts.