ASRock Rack TURIN2D16-2T Block Diagram
This block diagram might be the most unique in the industry, and by a long-shot.
First off, we can see that the PCIe Gen5 x16 slots both come from CPU2. That is the CPU directly above the PCIe slots. If you only had one CPU, of course in CPU1, then there would be no Gen5 slots available, only MCIO connections. Placement there likely means that the trace lengths from CPU1 to the PCIe x16 slots were too long to handle Gen5. We have been talking about this for years, but this is it in product form.
Normally on a motherboard the PCIe connectivity is on the CPU sides without DDR5 DIMM slots, but here, there is no room for MCIO connectors or slots. That makes PCB traces even longer in the motherboard.
Another interesting thing is that this is a 3x Infinity Fabric link design, not a 4x. There are two x16 ports open, so it feels like if this was a larger motherboard, this number of PCIe expansion slots could be handled with a 4x IF link design.
While the motherboard itself might look like ones that have come before it, there is so much going on here. This block diagram is perhaps the best one we have seen illustrating how hard it is for motherboard designers to handle the newer larger CPUs in traditional motherboard form factors.
ASRock Rack TURIN2D16-2T Management
In terms of management, we showed that this motherboard uses the ASPEED AST2600 BMC.
Instead of going into this in-depth, since we just did that in the ASRock Rack AM5D4ID-2T/BCM review, we are just going to show the key features from that. Logging in, we can see an ASRock Rack skinned management interface. This is an industry-standard IPMI interface.
Included are features like HTML5 iKVM with remote media. Companies like Dell, HPE, and Lenovo charge a lot for iKVM functionality. Now, companies like Supermicro charge for remote media mountable via the HTML5 iKVM. This is a small feature, but one that is handy for many users and it is great that ASRock includes this with the board.
Another new feature with this generation is that the default password is admin/ admin, but then it immediately prompts for a change with some validation rules (e.g. you cannot just make “admin” the new password.) This is done to comply with local regulations.
Next, let us get to the performance.
ASRock Rack TURIN2D16-2T Performance
At this point, we had to swap to the AMD EPYC 9575F SKUs since we still have a fairly limited selection of Turin CPUs in the lab below the top-bin parts.
The 8-channel platform certainly hurt performance a bit versus a larger 12-channel platform, but that is the cost of using newer bigger CPUs on a small motherboard. Something also to keep in mind is that we are using less hardware since we are using 2/3rds of the DDR5 DIMMs. For some folks, that means lower memory performance and capacity. For others, it means less platform cost and slightly less performance. At the same time, there are a number of workloads that are not terribly memory bandwidth constrained and where you just need CPU cores where this works well. We used a Turin CPU, but if you had two AMD EPYC 9754’s with 128 cores each, this motherboard can fit 256 threads and 512 cores which is great for a motherboard this size.
Next, let us get to the CPU support list.
PCB tracing also responsible for MCIO and triple power connectors effectively swapping places?
Similar looking to the Gigabyte MZ73-LM2, except that’s 500W CPUs, 4 PCIe 5, 2 x 10Gb/s LAN, and 24 DIMMs, etc.
I have a Asrock GENOAD8X-2TBCM with an 84c Epyc, which recently died withing warrant time (doesn’t post anymore), but I had a supplier which tries to screw me over, and at the moment I have contacted my lawyer to get my money back. I have the feeling there was always something wrong with the GENOAD8X-2TBCM, I never could enter anything via remote management when the OS was booted, also I tried to install XCP-NG but not one of the NVMe were detected during install, so I installed proxmox on it which ran, but there were always random crashes.
This board looks like a great update, unfortunately it is still nowhere to be found in Europe, disadvantage would be that I need a second Epyc 84c, does anyone know if running an assymetric configuration is possible 32cEpyc and 84c Epyc, probably not right ? I’m a bit afraid if I buy this board that I also can not install XCP-NG, which is my favorite Hypervisor, since I have it running on a Ryzen 5950x and it runs for months without problems, if something crashed it is a VM but the hypervisor is running without hickups.
On the other hand I”m contenmplating a Supermicro H13SSL-NT, more memoryslots but less pcie 5.0 slots, don’t like the smaller ATX formfactor and with supermicro I had a bad experience of networkcards getting too hot and dying because of that, I’m running these systems in a FractalXL case with a lot of fans, but its not servergrade chassis with the accompanying air jetstream which Supermicro might be expecting for their boards.
@Rupi
Both CPUs need to be identical.
XCP-ng is running a relatively old kernel (4.19) so it might have problems with newer hardware. For NVMe there’s an option in BIOS to override the device firmware with BIOS firmware. It’s in Advanced -> NVMe Configuration. It might make the drives visible in older kernels.
@Kyle, ok, thanks for the info, I could run in with 1 CPU for a while although I can’t use my 100GB optical card until I buy another CPU, I expect them to drop with the release of the Turin CPU’s.
I even had that same problem installing XCP-ng 8.3, weird they would use an older kernel with their latest release. Regarding the NVME firmware, never overrode the default setting hopefully that was the problem. Of course I can’t try it on my broken board, but definitely will try it on this board if I decide to buy next week, will have to order it from the US, never did that probably some customs tariffs will be imposed, but here in Europe mainboard prices are rediculously high, so I guess that evens it out.
Hi
I’m not sure if folks noticed,
but it only has 3 QPI lanes between processors.
It does not support the higher Turins, 450w
it may have 5 PCIE slots but only 3 are at PCI Gen 5
the on board m.2 is PCI gen 4
and all in all with few exceptions on an extra port for 2 more 4x cxl or nvme express;
it seems a downgrade to the mZ73-LM0 Rev 2 or Rev 3 from gigabyte
Did I miss something?
Jay
Jay they’ve got an entire section devoted to that. That GB board’s got more DDR5 but 24 fewer PCIe lanes. Gen 4 or gen 5, you often need slots and lanes. If you’re buying a 32 core epyc then you might be connecting network and ssds but your not gonna fill 12ch 9 of 10 times.
Now THIS is what I referred to in my comments on the 48-DIMM motherboards STH covered a while back! THIS is what I talking about! Put one CPU and memory stack IN FRONT the other and then 48 DIMMs (24 per processor) won’t be such a problem to physically place and route. In fact, adapt THIS board a little by eliminating the PCIe 4.0 slots, re-arrange the other components (BMC, glue logic, I/O chips, VRMs, MCIO connections etc.) a bit and this board could even take 24 DIMMs at 12 per CPU! Re-interpret this board into a more typical board for a rackmount and the footprint will allow a full 64 DDR4 or DDR5 DIMMs, WITHOUT increasing the width OR needing to create a special proprietary board just for that purpose! You might even could get away with this even on dual LGA-7529 CPUs (which I think are a good bit larger than AMD’s socket SP5/LGA-6096)!! Not sure about the upcoming LGA-9324 for Diamond Rapids, though, as THAT CPU might be a bit too big for even this to work.
EDIT: Insert the word “was” between the “I” and “talking” into the line “THIS is what I talking about!”. I can’t proofread apparently.
@Harold, if you look at the Phoronix articles “DDR5 Memory Channel Scaling Performance With AMD EPYC 9004 Series” and “8 vs. 12 Channel DDR5-6000 Memory Performance With AMD 5th Gen EPYC” you’ll have your answer, filling the slots (and more of them) is like free cores; it can add significant speed ups, and allow you to use smaller DIMMs at a lower cost.
The 2x MCIO 8i is what’s replacing PCIe slots that are too far away (without retimers).
With that MB you still get 2 slots with one CPU, unlike the ASRock which requires 2 CPUs to be usable. I think we need to hear “why” the ASRock is “better”, not simply a declaration that it is; when available information suggests otherwise.