Alibaba Immersion Cooling at Scale Research

0
3M Immersion Cooling
3M Immersion Cooling

In a standing room only talk just before the end of the OCP Summit 2018, Alibaba gave a talk about its project around immersion cooling at scale. The talk centered around the Alibaba immersion cooling hyperscale project for next-generation data centers. Alibaba has conducted hundreds of experiments and is looking to go into production with 100kW+ racks as soon as 2019.

Apologies in advance for the photos/ angles. It was a standing room only affair and we had to fight to get a spot to snap photos. 

The Alibaba Immersion Cooling Project

Alibaba is working on a major immersion cooling project for next-generation high-density data centers. For those not familiar with immersion cooling, the concept is that you put electronics in non-conductive fluids and take advantage of the better thermal conductivity versus air. Alibaba is using 3M Fluids for their project and 3M has been pushing this idea for many years.

Alibaba What Is Immersion Cooling
Alibaba What Is Immersion Cooling

Here is the slide on air cooling versus cold plate (used in water cooling servers such as Baidu’s GPU liquid cooling) and Alibaba immersion cooling. The short summary is that immersion cooling is seen as the best option.

Alibaba Immersion Cooling Versus Cold Plate And Air
Alibaba Immersion Cooling Versus Cold Plate And Air

The reasons for Alibaba’s immersion cooling project were to increase efficiency. Cooling is a major data center cost so increasing efficiency has a direct impact on TCO.

Alibaba Why Immersion Cooling TCO
Alibaba Why Immersion Cooling TCO

In hotter regions, where free cooling is not practical, Alibaba is seeing significantly lower PUE numbers than with air or traditional liquid cooling setups with cold plates.

Alibaba said that while they are still in the first phase of the project, they see this as a key enabler towards 100kW and higher racks and Alibaba expects to achieve 3x or more density increases over air cooling.

Alibaba Why Immersion Cooling Density
Alibaba Why Immersion Cooling Density

Another area that Alibaba is researching is reliability impacts due to not having moving fans that also cause vibrations and move air over the assembly.

Alibaba Why Immersion Cooling
Alibaba Why Immersion Cooling

Some of the challenges with immersion cooling of existing systems were fairly interesting. An example of this is that the thermal interface materials between hot components like CPUs and GPUs is thermal paste in air-cooled scenarios. In immersion cooling, different strategies must be used to increase surface areas of heat sources. Alibaba said that they needed to keep the high-speed signaling in the middle PCB layers to stay away from the 3M fluid.

Alibaba Immersion Cooling Basic Level Challenges
Alibaba Immersion Cooling Basic Level Challenges

Another intriguing feature was that conventional optics and other components need to be sealed properly in order to function.

Alibaba Immersion Cooling Component Level Challenges
Alibaba Immersion Cooling Component Level Challenges

Some of the issues noted in tank design were novel as well. For example, the first tank design had issues with tiny cracks leaking fluid due to shipping damage. Another example was that in southern China when the ambient temperatures became too high condensation occurred on the tank walls.

Alibaba Immersion Cooling System Level Challenges
Alibaba Immersion Cooling System Level Challenges

Currently, Alibaba is in its development and testing phase of the project. It expects next-generation production deployment to happen in 2019.

Alibaba Immersion Cooling Progress And Plan
Alibaba Immersion Cooling Progress And Plan

The presentation at OCP Summit 2018 was interesting, but the last points made were important. Alibaba is the first, and probably only, cloud service provider to share their immersion cooling project at this point. Alibaba also stated the need for ecosystem partners to help with the heavy lifting of design and building the tooling necessary to turn immersion cooling into an industry standard rather than a leading edge science project.

Alibaba Immersion Cooling Key Takeaways
Alibaba Immersion Cooling Key Takeaways

Alibaba Immersion Cooling Project Final Words

There are many small-scale immersion cooling examples that have cropped up over the years, but this is intriguing. Alibaba is specifically looking into the 100kW+ rack space for density will still maintaining a low PUE of 1.05-1.07. There are major challenges though and the company has been investing in research to bring the technology to the next phase of deployment. In the talk, the company also discussed how this was for next-generation data centers. From the discussion, it seemed like the project required significant data center differences. A great example is how a typical rack today can be serviced by pulling gear out horizontally from vertical racks while with immersion cooling you generally pull gear out vertically from horizontal racks. If you think the two biggest barriers to adoption are the physical facilities to deploy at scale and the design research to ensure entire data centers can be maintained for useful lifetimes, then it takes a company of Alibaba’s scale to undertake such an ambitious project.

SHARE
Previous articleDirectly Download an ISO to Proxmox VE for VM Creation
Next articleZFS on Linux 0.7.7 Disappearing File Bug
Patrick has been running STH since 2009 and covers a wide variety of SME, SMB, and SOHO IT topics. Patrick is a consultant in the technology industry and has worked with numerous large hardware and storage vendors in the Silicon Valley. The goal of STH is simply to help users find some information about server, storage and networking, building blocks. If you have any helpful information please feel free to post on the forums.

LEAVE A REPLY

Please enter your comment!
Please enter your name here