We wanted to discuss our test configuration before moving on to our performance numbers.
AMD EPYC 3251 Test Configuration
Our test configuration was the AMD Wallaby platform. Since this is an embedded SoC, the CPU is affixed to the motherboard.
- Motherboard: AMD Wallaby
- CPU: AMD EPYC 3251
- RAM: 2x 16GB DDR4-2400 RDIMMs (Samsung), 2x 16GB DDR4-2666 RDIMMs (Samsung), 2x 32GB DDR4-2400 RDIMMs (Micron), 2x 32GB DDR4-2400 RDIMMs (SK.Hynix)
- GPU: AMD FirePro W2100
- SSD: Intel DC S3710 400GB
- OS SSD: Samsung 850 Pro 256GB SATA 2.5″
There are a few major caveats that we wanted to point out using a platform like this.
First off, the vast majority of embedded platforms that we test have IPMI interfaces. That usually means an Arm-based ASPEED AST2500 series BMC these days (AST2400 series previously.) That chip is accompanied by a single DRAM package. It also is usually wired to an Ethernet LAN port and provides some graphics capability. All of this adds additional power consumption to the tune of several watts. That is a big deal in the embedded space when power is usually under 100W at the wall. Our system had an AMD FirePro W2100 GPU in the PCIe x16 slot. That GPU sits mostly idle during our testing, but it adds to the power figures and is rated for <26W power consumption which is huge in an embedded platform.
Generally, we use SATA DOMs for OS drives as that is incredibly common in the embedded market. The Wallaby platform did not have SATA DOM power that we could find so we ended up using a Samsung 2.5″ SSD. This is a departure from our standard testing procedure.
We tried several DACs with the Wallaby platform but ended up using short-range optics instead. Typically, our embedded reviews use DACs connected to SFP+ ports. In this test platform, we needed to use recommended optics. With development platforms, one often needs to use specific I/O hardware. We saw this in our early Intel Xeon D-1500 work as was noted in Intel Xeon D-1540 and SFP+ 10GbE Microserver X552 NIC where the Intel development platform even required a specific powered USB hub. We had to make a concession on our standard setup which was necessary.
This is a decent system with 32GB of RAM. We tried 64GB of RAM in 2x 32GB RDIMM configurations successfully. We would have tried 128GB in 4x 32GB but we were limited by the physical platform.
Finally, we performed testing with both a more production firmware as well as a “debug” firmware. Getting reliable data meant that we had to calibrate the two sets of results since we had more control over the debug firmware. We saw deltas in the 2-3% range. That meant that the differences were easy to spot, but they were there. As a result of what we are seeing with firmware updates, we think that this may be a case like the EPYC 7000 series where future firmware updates will get additional performance from the platform.
Deciding to Proceed
With all of these caveats, we still decided to proceed with publishing our review for a simple reason. The embedded server CPU/ control plane market is still largely an Intel domain with the Intel Xeon D-1500, D-2100 and Atom C3000 series. Arm-based servers are starting to show more performance. If you have x86 code, and want to run it on non-Intel hardware to give your supply chain diversification, then the market needs AMD EPYC 3000. Our role in the market for years has been to stay on the bleeding edge of these embedded CPUs so we wanted to get our readers information.
Our goal with this piece is to help our readers understand a relative performance ranking between CPUs. That way if you see two options from a sales rep quote, you have some idea of what you are getting with each option. The market needs this data if there is going to be an x86 competitor to the Intel Xeon D monopoly in the space. We wanted to get data out into the community in an attempt to get more designs built and options for customers.
AMD EPYC 3251 Wallaby Platform Topology
Unlike the larger AMD EPYC 7000 series, the AMD EPYC 3251 has a relatively simple topology. There is a single Zen die so the architecture looks largely like the mainstream AMD Ryzen platform actually.
You can see the various cache sizes here, as well as the PCIe root that goes into the die. We took this snapshot with just the OS SSD installed, not with the full test configuration. Still, one can see the basic setup of the core structure. For some perspective, we are essentially seeing this Zen die shot logically presented and flipped 90 degrees.
You can see the CCX and cache structures in that die shot. The topology diagram also matches with the Zen CCX complex overview slide.
One commonly voiced concern on the AMD EPYC 7000 series is the four NUMA nodes. The 8 core, and below, AMD EPYC 3000 series parts like the AMD EPYC 3251 are single NUMA node designs. The larger AMD EPYC 3000 series parts are made from two dies and should have two NUMA nodes. Intel in comparison is still a single NUMA node on all of its current Xeon D designs.
Next, we are going to look at the AMD EPYC 3251 benchmarks. After that, we are going to discuss power consumption as well as market positioning before concluding with our final thoughts.
Do ya’ll have the 16 core one?
I’m going to ask our PM team if their fixin’ to make up our appliance with embedded 3000 EPYC.
I’ll read the rest later, but you’re right on the need for an Intel alternative. These side channel attacks and the embedded clock bugs show a need in the market for a second x86 supplier.
Can someone make a damn mini ITX motherboard with these, 4 DIMM slits and some NVMe headers?
Are the single die parts way cheaper? The higher core counts looked like huge cost savings but I’m not seeing any on AMD’s site. Only 4 SKUs there https://www.amd.com/en/products/specifications/embedded/8161
Model: AMD EPYC™ Embedded 3251
Product Type: SOC
Family: AMD EPYC™ Embedded Processors
Line: EPYC Embedded SOC
OPN: PE3251BGR88AF
TDP: 55W
CPU Type: Zen
CPU Base Freq.: 2.5GHz
CPU Max Freq.: 3.1GHz
# of CPU Cores: 8
# of Threads: 16
GPU Support: No
Security Processor: Yes
Total L2 Cache: 4MB
Total L3 Cache: 16MB
System Memory Type: DDR4
DDR4 Rate (Max): 2666 MHz
Memory Controller: Dual Channel
ECC: Yes
USB 2.0: 0
USB 3.0: 4
USB 3.1 Gen1: 0
USB 3.1 Gen2: 0
SATA: 8
Low Speed Interfaces: EMMC, eSPI, GPIO, I2C, LPC, SMBus, SPI, UART
# of PCI controllers: 8
Gen3: 32
Gen2/3: 0
Gen2: 0
Ambient Temp Range: 0-105°C
Enhanced Temp Support: Extended Temp (0-105°C)
Infrastructure: SP4
Last Time Buy: 2028
Recommended for new designs: Yes
Unless I missed something, but who makes the NIC? The lspci output showed the NIC as an AMD network card, but who did AMD license the NIC from? What network driver is it the chip using? That could provide some clues.
RC it’s on the first page showing as an AMD network device so that’s AMD NIC IP.
I’m just surprised AMD building their own NIC IP. I suspect they licensed it as many others do. E.g. Asmedia makes the chipsets.
Great review.
RC it’s a SoC.
Hi Misha, I understand it is an SoC. Typically what you do as part of SoC designs is license IP (in chip design IP means a blob of VHDL / Verilog code), so let’s say I design an ARM SoC, I get an ARM core from ARM, I may get a video core IP from Imigation, a flash controller IP from someone else. Yes, you can design pieces in-house too. Something like networking is so complicated and I don’t recall AMD having any experience in there, I doubt they started from scratch and just licensed IP from a Marvell or some other company. I’m curious what IP it is unless it is truly their own design.
I found the answer to the AMD NIC question. I noticed the screenshot showed ‘amd-xgbe’ driver, so I looked at the Linux kernel source. Based on the code AMD licensed “Synopsys DWC ETHER XGMAC”, see https://www.synopsys.com/dw/ipdir.php?ds=dwc_ether_xgmac. I don’t know how much this IP does as it provides a MAC layer, so AMD likely had to build some parts themselves, but I’m not sure.
Finally, Patrick, thanks.
I would assume that’s $315 USD in bulk lots of 1000 for the chip? Wholesaler price?
So could we assume at least $550 / $600 US motherboards using this?
I’d still strongly consider it, it certainly seems to perform. I’d prefer perhaps something 15w lighter and $100 lighter, but we’ll have to wait for more boards.
As I posted on the forums, incredibly hugely disappointing that AsRock, SuperMicro, Tyan all either told me “nope, no interest in this!” or flat out didn’t respond!
These would make a monster FreeNAS machine, especially the 3201 model with 8c/8t at only 30Watts!
Patrick, do you know if this problem impacts the Epyc 7xxx or 3xxx series?
https://bugzilla.kernel.org/show_bug.cgi?id=196683
https://forums.freenas.org/index.php?threads/ryzen-stability-on-11-0-u4.59017/
$315 is the tray pricing. Anyone buying these embedded parts is likely buying quantities for motherboard runs.
I believe that kernel bug is for Ryzen users and has not been confirmed on EPYC.
what do you think, is Intel any part of delaying the go-to-market for these chips?
@kpin:
I hadn’t thought of that, it seems unlikely but I wouldn’t put it past them to try that.
I did contact SuperMicro, Tyan and AsRock about this CPU multiple times since Feb and have either had no response or an outright “no” from them, which was disapointing.
However, SuperMicro already do some AMD Epyc stuff, so they aren’t ‘frightened’ of Intel, to my knowledge.
We need a competitive board to the likes of SMCI’s x11sdv-*c-tlnf8 boards. (I hope I got the model name right from memory.)
Would be great if we could get the same treats the Intel line currently has.
Anyone know of a board coming out with the dual 10 Gb option? the supermicro boards which just came out are all 1 Gb.
@Nathan
Asrock Rack EPYC3251D4I-2T supports dual 10 GbE NIC speed (and M.2 slot supports 22110 which is unique to my understanding).
See https://www.asrockrack.com/general/productdetail.asp?Model=EPYC3251D4I-2T#Specifications
cpu socket ???
norbi31 – no CPU socket. These are embedded parts.
Any sign of the higher level SKU 3451, 3401, 3351,3301? that is not COM Express Type 7? And they seem vaporware as well.
https://www.amd.com/system/files/documents/3000-family-product-brief.pdf
Steve! Yes. Stay tuned in a few weeks to STH.
Any updates. This platform looks very cool, and I would like to use it for routers and small appliances, but I can’t find any consumer motherboard or system that actually exposes the built in 10Gbps NICs as SFP+. I did found some boards from Supermicro and ASRock Rack, but they all use Intel NICs for their 10Gbps and 1Gbps ports. Weird. Would be great to see native 4x SFP+ and 2x SFP (via external controller), and this SoC can do it quite well and cost competitively, yet I can’t find products doing that. Weird.
Some information about idle power use to someone like me who uses the platform in residential setting where the system is running 24/7, workload is bursty and idle power use is very important.
With the following configuration I’m getting 22.3W at idle.
M11SDV-8C-LN4F with EPYC 3251.
Stock fan replaced with Revoltec RL036 (mostly due to noise reduction, but likely it helps power use too).
64GB DDR4-2400 RAM (2x32GB HMA84GR7MFR4N-UH)
WD SN750 1TB NVME SSD
Only single gigabit network cable is connected. BMC is left unconnected.
PSU is 120W 12V power brick (Chieftec CDP-120ITX).
VGA is disabled via jumper (saves 1W)
“powertop –auto-tune” has been run (saves 0.2W if VGA is enabled, 1.1W if VGA is disabled)
The measurement has been made on console login screen of stock Debian 11 that had powertop installed. The power meter is not calibrated.