Building Something Fun and Unique
This motherboard has been through a ton of different configurations. We have loaded NVIDIA RTX 3090‘s, A100’s, and other GPUs in the system. The passively cooled A100’s were a bit tough just due to cooling and airflow in this chassis. Still, we think most users will utilize GPUs in this platform. It was not unique enough though.
We also worked through a number of SSD configurations including using the Hyper M.2 card. One can absolutely add a high-end GPU as well as GPUs in the system. One of the great features is that with onboard 10Gbase-T, one can fill the PCIe slots without being stuck with 1GbE networking.
In terms of memory, we started with an 8x 32GB configuration but then switched to an 8x 64GB configuration. The 8x 32GB was a good fit for the 3955WX, but somewhere between switching to the 3975WX and 3995WX it became apparent that getting more memory was important. This will add to the cost of the platform since one of the big features and reasons to go WEPYC is the additional memory bandwidth and capacity.
This is a picture of perhaps the coolest configuration we did with the system. Here one can see 7x NVIDIA BlueField-2 DPUs. Each has 16GB of memory, an 8 core Arm CPU, 64GB of storage, an out-of-band management port, and dual 100GbE/ Infiniband ports.
Just for some sense of scale, this added 56 arm cores, 112GB of memory, and 1.4Tbps of network ports (plus seven management ports) to the system. More importantly, it meant that we were running seven additional systems from this one machine.
Not only did we have Ubuntu installed on the main system, but also each of the BlueField-2 DPUs. This is actually the system we used for the A Quick Look at Logging Into a Mellanox NVIDIA BlueField-2 DPU piece.
This is a very uncommon configuration, but it is important. This is clearly not going to be a tested configuration on the platform. In the past, running an extra card or two would be an issue on this type of board, especially when it is not a validated board. Here, we have a very high-end solution that worked out of the box which was frankly bordering on shocking. If you saw our Why you want Plenum Fiber Optic Cable Video, you will know that we are installing around 1700 fibers and 100km of fiber to do more networking. This type of system is why. We have enough fiber with it to run all 14x 100GbE links without a switch in the same room as the workstation. Still, this was immensely cool to get to build on a system like this.
Final Words
Overall, the biggest challenge with this platform comes down to price. The motherboard is not cheap and CPUs are not cheap. Utilizing 8 channels of memory is also costly. Finally adding storage, accelerators, networking, and GPUs to the platform to take advantage of all of the I/O offered is costly.
With that said, there are also awesome capabilities available. Having a platform that one can use as either a workstation or a server is great. Being able to scale the core counts and memory capacity can have huge benefits for some users. Likewise, being able to handle basically the limit of PCIe expansion for today’s systems to build a unique workstation platform is also great. Frankly, this feels like a step or two beyond the motherboard in the Lenovo ThinkStation P620 we reviewed in terms of capabilities and even aesthetics.
At some point, if you are in need of a platform for the AMD Ryzen Threadripper Pro series, the ASUS Pro WS WRX80E-SAGE SE WiFi is truly awesome.
Notes:
1. Cool setup.
2. You went the GPU/Mellanox(NIC) setup. A Storage/Mellanox(NIC) setup is similarly interesting. WEPYC is also destined to outclass most professional storage solutions in network and storage bandwidth.
3. In addition that platform is also ideal for high-end KVM/Hyper-V/Container platforms. In fact this is the first time, where a terminal server can outclass any workstation in CAD/CAM, GPU, gaming, parallel processing. Tags: GPU-P in Hyper-V, PCIe cut-through,…
4. The article sounds “slightly” one-sided. From our testing the Supermicro M12SWA-TF is slightly faster.
Nice read.
Curious how you got fan control on your motherboard. I’ve had one for about a week, have a support ticket in with Asus, but no response. Cpu is 100% all the time.
I have issues with GPUs, I run 4x 3090.
My power supply is 1000W for 2x 3090 and 1600W For System and 2 GPUs.
First of all, GPUs are plug-in slots 1-2-3-4 not as recommended in slots 1-3-5-7 (when I use the recommended, the system won’t boot)
When I use slots 1-2-3-4 I can’t shut down the system at all. Some GPUs kick out (red lamps on ROG STRIX) and the powerplay won’t stop at all.
Can anyone help.
Fan controls are over BMC not BIOS on this motherboard, read the manual.
I bought a new Motherboard a week ago along with the 3955wx… Inserted RAM chips 1 by 1 starting DIMM_A, B, C, D….H.
The motherboard screamed at me with 1 long and 2 short beeps when I inserted sticks in the A and D slots (one by one as well as together); in all the other slots, it POSTed just fine with 00 Code.
I felt bad to torture the RAM slots till I was convinced that the RAM slots were either dead or had some issues. I even removed and inserted the CPU, checked all the pins of the CPU socket…. it was all clean.
Now the board is under RMA… I will surely update once I receive the new one.
Cheers guys.