On the Supercomputing 2018 floor, we saw a hunk of metal that seemed inert at first. It looked like so many other liquid coolers with a series of tubes and copper bits. Instead, this hunk of metal allows Cray to achieve something almost unimaginable by 2019. Using liquid cooling by CoolIT Systems, Cray is able to build systems with up to 1024 threads per U in 2019 with room to increase that number as processors evolve. Cray already has a Shasta design win in Perlmutter for AMD EPYC “Milan” generation that the company highlighted as a successor to 2019’s “Rome.” You can read more about AMD’s recent roadmap disclosure in our AMD Next Horizon Event Live Coverage.
Cray Shasta Liquid Cooling by CoolIT
With eight CPUs and cooling for up to 64 DIMMs, the liquid cooling solution allows Cray to handle twice the CPU density of typical 2U 4-node systems while reducing power consumption by 20-30% over air cooling. The design has another implication as we are told it is able to easily handle significantly higher wattage parts.
You will immediately notice that the tops of each cold plate are smooth. This is for good reason. We are told that the Cray Shasta cooling includes both top and bottom mounted components.
The PCBs atop the AMD EPYC sockets are the fabric uplink cards. You can see that they are cooled by the same CoolIT cold plates.
The Cray Shasta AMD EPYC platform is designed for 8 DIMMs per CPU which means 64 DIMMs in 1DPC mode per system.
This is where the industry is heading. With next-generation NVIDIA data center GPUs likely to exceed 350W and CPUs also looking to move from the 180-205W range today to the 250-350W range in a few months, there is a looming question on how all of this will be cooled. What we do know is that we are moving to an era with more than 1000 threads per U with on roadmap products due in the not-too-distant future.