Selecting Ultra EPYC Components
As much as we love Sun, 4GB of RAM is not going to cut it. The 530W PSU is not going to help when we have 200W+ CPUs, 250W+ GPUs, and many other components. It was time to pick the components to make something truly, Ultra… EPYC.
Ultra EPYC CPU: AMD EPYC 7371
Current generation AMD EPYC 7000 “Naples” generation CPUs top out at 32 cores. While we have plenty of AMD EPYC 7551P and EPYC 7601 chips in the lab, they suffer from lower clock speeds. As a workstation, one really wants higher clock speeds if possible. We could have gone with an AMD Ryzen Threadripper 2990WX 32-core part, but that would have limited our memory capacity. Plus, this is a Sun Ultra that we were turning into an Opteron-worthy successor.
Enter the AMD EPYC 7371. You can read all about the chip in our AMD EPYC 7371 Review. Price wise, the AMD EPYC 7371 is fairly reasonable at $1550. It is capable of dual-socket operation, however, we are constrained with an ATX case. ATX and even EATX struggle to fit two modern server CPU sockets.
With that said, we are using a 205W TDP CPU here. The Intel Core 2 Duo we are replacing is a 65W part.
Ultra EPYC Motherboard: Gigabyte MZ01-CE0
We are going to have our Gigabyte MZ01-CE0 review coming soon. For now, let us just say it is an AMD EPYC platform that is designed to work well even in an ATX form factor.
The MZ01-CE0 has spaces for four PCIe x16 cards, even with double-width coolers. The solution even has 10GbE built-in which is exciting for a workstation. To fit the ATX form factor, the Gigabyte MZ01-CE0 cannot fit a full 16x DIMM slots so one gets 8x DIMM slots, one to fill each of AMD EPYC’s memory channels. The Sun Ultra 24 had gigabit Ethernet which was a leading feature in 2007.
Ultra EPYC Power Supply: EVGA SuperNOVA 1000W G3
When EVGA got wind of this project, they offered to send an EVGA 1000W G3 power supply. The modular cabling will help since cable management in 2007 was not as robust as today.
I also use an EVGA SuperNOVA 1000W G3 model in my personal Threadripper workstation, so I was happy to use this one. One of the nice things that these PSUs have is the ability to shut off the fan if the load is too low. Being able to have a cache of the same types of cables is also nice.
Ultra EPYC GPU: NVIDIA GeForce RTX 2080 Ti
This one caused a bit of a conundrum. What GPU to use. The NVIDIA Quadro series has gone up in price quite a bit since 2007, so a Quadro RTX was out of the budget. Since we started this build before the new Titan RTX was launched, we are using the NVIDIA GeForce RTX 2080 Ti.
We came close to using the AMD Radeon Vega Frontier Edition in this build, but since we are replacing NVIDIA’s first generation CUDA card, it seemed reasonable to stay NVIDIA.
As a quick note, the outgoing NVIDIA Quadro FX 1700 was a 42W TDP part, we are replacing that with a 250W RTX 2080 Ti. That is an enormous leap.
Ultra EPYC CPU Cooling: Noctua NH-U9 TR4-SP3
Since the Sun Ultra 24 does not have radiator mounts, we needed an air cooler. Further, this is not the largest case, so we needed to find a 4U air cooler for a 205W TDP AMD EPYC.
I re-read William’s Noctua NH-U9 TR4-SP3 AMD EPYC 4U Cooler Review and immediately ordered one from Amazon.
This was probably the biggest configuration miss that we had, but frankly, there are not a lot of great options to remedy in this case. We are going to show why during the build.
Ultra EPYC NVMe Storage: Intel Optane 905P 380GB M.2 SSD
The system has four hot-swap bays. We even ordered an extra four 3.5″ trays just to match. Still, for basic storage, we wanted the best. You can get fast and big storage, but Intel Optane 905P 380GB M.2 NVMe SSD review shows why it is an amazing part.
The M.2 SSD fits directly onto the motherboard port. We also are using an EK cooler that came bundled with the drive. In a server, there is enough airflow. In this workstation, we were worried.
Ultra EPYC SAS3 Storage: Broadcom HBA and Micron 3.2TB SAS3 SSDs
The donor Ultra 24 had a SAS/ SATA backplane. We decided to use it for this build, even though we would recommend against it. In fact, this is the part of the build that we would strongly urge our readers to not follow. We used a Broadcom SAS 9340-8i HBA which is our standard IT/IR-mode HBA. We are using the Micron S630DC 3.2TB SAS3 SSDs for 12.8TB of SAS storage.
This is a part of the build that can be left out for the better. The Micron S630DC 3.2TB drives run too hot. One can use WD coolers to attach them but we had issues there as well. The older SAS/ SATA backplane negotiates at 6gbps SAS2 speeds, but when we add a cooler to keep the SSDs cool while in operation, the links negotiate down to SAS1 3.0gbps speeds. That is absolutely silly.
We could have run large hard drives, but this is 2019, hard drives belong in network storage. They are loud and too slow to sit next to you in a workstation.
Ultra EPYC RAM: 256GB in 32GBx8
The RAM side was easy. 64GB DIMMs are out of our budget. We have 8x DIMM slots. That means 8x 32GB DDR4-2666 RDIMMs.
With AMD Threadripper, 128GB is more of a realistic number using 8x 16GB unbuffered DIMMs. Since this is a high-end workstation, we are using 32GB RDIMMs. That is a great upgrade over the 4GB circa 2007. We also had an 8x 16GB 128GB ECC RDIMM configuration we tried which works well in cases, for example, when you need to put together 48x 32GB RDIMMs for a single node in the lab (review on that one coming soon) and that is all you can spare.
Ultra EPYC Networking: Mellanox 100GbE and EDR
We tend to use a lot of Mellanox ConnectX-3 Pro EN 40GbE cards. They more or less have invaded our labs with only a few Intel and Chelsio 40GbE adapters in the mix. Since this is the Ultra EPYC, why not utilize a network standard that is 100x faster than the 1GbE the system came standard with 12 years ago.
We are using Mellanox network cards almost everywhere these days for our 100GbE networking. The Mellanox VPI cards can handle either 100GbE or 100Gbps EDR Infiniband duties, making this more flexible than one may have assumed. This is a challenge to cool. Using a 40GbE NIC would be much easier to keep cool in this type of case. We wanted to use the extra PCIe x16 lanes that the motherboard has, so that made a lot of sense.
Ultra EPYC: Beauty Mark Update
We could have left the “beauty mark” of the CD/ DVD drive alone and hooked it up via SATA. Instead, we decided to update this to a more modern USB 3.0 hub.
Of course, we found a black faceplate to keep the aesthetically displeasing and defining feature of the Sun Ultra 24 intact. This Kingwin KW525-7U3C looked like it was a worthy upgrade to the optical drive since it allows for an array of USB 3.0 thumb drives. Also, the Gigabyte MZ01-CE0 does not have audio, USB 2.0 and firewire headers galore so one has to make due with a single USB 3.0 front panel header.
Now, let us take a look at the final Ultra EPYC.