Cavium ThunderX2 Power Consumption
As a reminder from the original piece, the chips we are using are not the final binned parts. Cavium says that the shipping parts will be binned for power and should run more than 10W lower per chip (edit: Cavium told us 15-20W). With that said, we wanted to show the dramatic improvement that lowering the new firmware had on power consumption:
Note these results were taken using a 208V Schneider Electric / APC PDU at 17.8C for the original results and 17.6C for the updated results and 71% RH. Our testing window shown here had a +/- 0.3C and +/- 2% RH variance so this is reasonable and in-line with our standard environments.
The bigger question we wanted to explore today is whether the Cavium ThunderX2 is competitive with its x86 competition in terms of power consumption. We used the Gigabyte R281-G30 chassis as our Intel Xeon Skylake-SP platform and swapped power supply assemblies to ensure we were using the same PSUs in both platforms. On the AMD EPYC side, we are using a 2U Supermicro AS-2023US-TR4 “Ultra” platform as it was the closest dual socket AMD platform we had in the lab. Here is what the power consumption looks like:
As you can see, the significantly improved power consumption puts it in-line with where Intel Xeon Skylake-SP and AMD EPYC systems are. Also, we wanted to note that the AMD EPYC and Intel Xeon Scalable figures are a bit higher than we put in our reviews. We fully configured the systems, meaning, we needed to match add-in Mellanox 100GbE NICs, Broadcom 9305-24i SAS HBAs, 25GbE NICs, Intel Optane 900p AIC 280GB drives, SATA SSDs. We also used 256GB of RAM on all three platforms. Since Intel Xeon Scalable (Skylake-SP) CPUs only have six memory channels we used 8 DIMMs per CPU which means one channel was in 2 DPC mode. We manually set DDR4 frequencies on the Intel system to DDR4-2666 to ensure we had as close to apples-to-apples as possible. It is fair to stay that the systems are not identical. That is a tough task given these utilize three different architectures. At the same time, this is perhaps about as close as one can configure the three platforms and it took us a tremendous amount of lab labor and hardware on hand to get this close.
Validating the Firmware Update
One area that we were aware of is that it is possible for chip companies to update firmware and drastically decrease power consumption to the detriment of performance.
We re-ran benchmarks overnight from our original article. As we were going through the data, figures were within +/- 0.8% of our original tests and not always lower so after 12 hours of workloads we made the call that the firmware update did not have an appreciable impact on performance.
Assuming the production chips are binned for power and Cavium sees 30-40W lower power consumption in a dual socket server from that exercise, we arrive at a fairly simple conclusion. Cavium ThunderX2 is competitive in terms of performance and power with Intel and AMD. If you were taking “Cavium is Arm, thus it must have significantly lower power consumption” that would be an incorrect assumption. Cavium ThunderX2 is designed to match Intel and AMD offerings in a broad range of applications and therefore uses a similar amount of power. At the same time, we are seeing the ThunderX2 make waves since it bests AMD and Intel by utilizing a single die with 8 channel DDR4 memory. That is a claim neither AMD nor Intel can match.
We are going to update our original article with a link to this piece to preserve continuity.