When your iPhone overheats, it shuts off or even becomes permanently damaged. A supercomputer—which is approximately 200,000 times more powerful and 192,000 times bigger than an iPhone—is no different, except that the consequences are greater because a supercomputer is expensive (millions of dollars) and the data it’s crunching may have national security implications. Therefore, it’s imperative that supercomputers are cooled properly.
Crossroads, the new supercomputer coming to Los Alamos National Laboratory in 2022, will be direct-liquid cooled.
Crossroads, the new supercomputer coming to Los Alamos National Laboratory in 2022, will be direct-liquid cooled—heat from this supercomputer will be removed by a liquid coolant that’s a mix of treated wastewater, detergent, and Nalco 460 CCL 100. The detergent helps prevent corrosion of the cooling pipes, and Nalco is a chemical that prevents algae growth.
Liquid cooling was used in the 1980s before air cooling became popular in the ’90s. Now the pendulum is swinging back to liquid because recent advances in liquid cooling have made it more efficient than air cooling. Liquid carries away more heat from the computer’s core than air, and liquid directly removes heat, which is more effective than blowing cool air over a supercomputer.
A direct-liquid cooling system circulates 75-degree liquid coolant through a closed system that directly contacts the areas in the supercomputer where the most heat is generated. Unlike the 60-degree coolants used in previous generations of cooling systems, the warmer, 75-degree coolant in the current system uses less energy to do its job. The coolant carries the heat away from the computer, allowing the machine to operate at a viable 110 degrees.
The coolant, now warmer because of its contact with the computer, flows through a series of loops inside large towers to cool off. The water in the towers is evaporated to expel heat into the atmosphere. The number of times the coolant runs through these loops is called the cycles of concentration (CoC).
With each CoC, some of the water evaporates, so after several CoCs, new water is needed. Any remaining old water is tested for environmental safety and released into a nearby canyon. New treated wastewater from the Laboratory is used to replace the spent tower water.
The higher the CoC, the more times water circulates before it needs to be replaced. Higher CoCs use less water for the same job. One way to increase the CoCs is to improve the quality of water being used. Well water, for example, has a CoC of 2.5—meaning it can circulate in the towers 2.5 times before new water is needed.
The water cooling the Laboratory’s Strategic Computing Complex’s supercomputers is repurposed water from the Lab’s Sanitary Effluent Reclamation Facility (SERF). In 2019, SERF improved its water purification process, which gave the Lab’s treated wastewater a CoC of six and greatly reduced the total amount of water necessary to cool the Lab’s current supercomputer, Trinity. Instead of using 88,000 gallons daily, Trinity now uses 50,000 gallons. Crossroads, which will be even more of a workhorse than Trinity, is estimated to use 90,000 gallons—far less than it would have without the sustainability project.
Less environmental impact plus a “large jump in efficiency,” makes direct-liquid cooling a big win, according to High Performance Computing Division Leader Gary Grider. But those aren’t the only perks. “The machine is super quiet since there are no fans and just pumps,” says Grider, who was at the Lab in the ’80s and can remember the original liquid-cooling technology. “It’s been a long, long time since the systems were cooled with just water.”
