...manufacturers increasingly turning to liquid cooling as a practical solution. The majority of manufacturers have turned to liquid-cooled enclosed racks or rear-door heat exchangers, in which chilled water is delivered to the racks. Some manufacturers are now looking to cold plate cooling solutions that take the heat directly off problem components, such as the CPUs, to get it directly out of the facility.

This paper describes work done at the Pacific Northwest National Labs (PNNL) under a Department of Energy-funded program titled "Energy Smart Data Center." An 8.2 kW (27,980 Btu/h) rack of HP rx2600 2U servers has been converted from air-cooling to liquid spray cooling (CPUs only). The rack has been integrated into PNNL's main cluster and subjected to a suite of acceptance tests. Under the testing, the spray-cooled CPUs ran an average of 10[degrees]C (18[degrees]F) cooler than the air-cooled CPUs. Other peripheral devices, such as the memory DIMMs, ran an average of 8[degrees]C (14.4[degrees]F) cooler, and the power pod board was measured at 15[degrees]C (27[degrees]F) cooler. Since installation in July 2005, the rack has been undergoing a one-year uptime and reliability investigation. As part of the investigation, the rack has been subjected to monthly robustness testing and ongoing performance evaluation while running applications such as High Performance Linpack, parts of the NASA NPB-2 Benchmark Suite, and NW Chem. The rack has undergone three months' worth of robustness testing with no major events. Including the robustness testing, the rack uptime is at 95.54% over 299 days. While undergoing application testing, no computational performance differences have been observed between the liquid-cooled and standard air-cooled racks. A small-scale (8-10 racks) spray-cooled Energy Smart Data Center is now being designed as a final step to demonstrate the feasibility of scaling liquid cooling at the single rack up to an entire facility.

INTRODUCTION

Data center heat flux has been of concern for a number of years, and there appears to be little relief in sight for the near future. ASHRAE has published an update to the classic Uptime power trend chart, which shows equipment heat loads continuing to increase (ASHRAE 2005, in particular Figure 3.10). Equipment heat flux is calculated by dividing the rack power by the rack footprint (i.e., width x depth). Schmidt (2005) shows actual equipment heat loads superimposed on the updated ASHRAE chart. The actual loads provide an increased level of confidence in the original Uptime chart, which points to the extreme heat flux densities that are starting to appear in data centers today.

It is projected that individual server power dissipation will reach 800 W (2,730 Btu/h) in a 1U vertical space around 2007 and that this will last through the end of the product lifespan in approximately 2013 (Scientific Computing 2006). For a 42U rack containing 36 x 1U servers, the rack heat load will reach 28.8 kW (98,270 Btu/h). Rasmussen (2005) shows the typical rack heat load at 1.7 kW (5,801 Btu/h) (2003 data), with the maximum 1U server rack heat load at approximately 16 kW (54,590 Btu/h) and the maximum blade server heat load at approximately 20 kW (68,240 Btu/h). Scaling Rasmussen's data to a 30 kW (102,400 Btu/h) rack of blade servers indicates that approximately 3000 cfm (1.4 [m.sup.3]/s) of airflow are needed to air cool the rack. Using Rasmussen's data to scale up to a 50 kW (170,600 Btu/h) rack shows that approximately 5000 cfm (2.4 [m.sup.3]/s) of airflow is needed to air-cool such a rack. This trend is unsustainable, and datacom equipment manufacturers are aware of this. Alternative cooling solutions are now being heavily investigated, and the vast majority of them involve liquid cooling.

Several vendors use chilled facility water in their racks to cool air delivered to the rack with an air-to-liquid heat exchanger. One vendor places an air-to-liquid heat exchanger on the side of a rack of servers (Vendor #1 2006). The rack's hot exhaust air is directed into the heat exchanger, is cooled, and then blown out the front, ready for reuse by the servers. Product literature states that a maximum capacity of 30 kW (102,400 Btu/h) can be provided when 7.2[degrees]C (44.9[degrees]F) water is delivered at 68.1 L/min (18.1 gpm). This vendor touts a key advantage as being the ability of this solution to be used with and without raised floors.

A second vendor delivers chilled refrigerant to evaporator units mounted to the tops of racks and in the data center's ceilings (see Vendor #2 [2005]). This solution is used in conjunction with raised floors and supplements the chilled-air supply from the floor tiles. The product literature states that this solution can provide a maximum additional cooling capacity of 16 kW (54,590 Btu/h) per ceiling-mounted unit and 8 kW (27,300 Btu/h) per rack-mounted unit. The number of units installed and their placement depend on the heat load of the racks.

A third vendor takes air conditioning one step further by placing the servers in a totally enclosed rack and delivering chilled air to the front of the servers. The product literature states that a maximum capacity of 30 kW (102,400 Btu/h) can be provided when 7[degrees]C (44.6[degrees]F) water is delivered at 75.7 L/min (20 gpm).

A fourth vendor uses an air-to-liquid heat exchanger mounted to the rear (exhaust side) of a rack of servers. In one implementation, a central distribution unit (CDU) is used to condition facility water, via a mixing valve, to above a facility's dew point, which it then distributes to multiple rear-door heat exchangers. The heat exchanger does not use additional fans and relies on the server fans to push the exhaust air through the heat exchanger. The manufacturer states that, on average, a typical heat exchanger mounted in the rear door can provide a cooling capacity of 15 kW (51,180 Btu/h) when 18[degrees]C (64.4[degrees]F) water is provided at 37.8 L/min (10 gpm).

The foregoing provides a brief description of several vendor solutions that do not integrate their cooling solutions into the servers themselves. The authors are particularly interested in liquid-cooling solutions that do, in fact, integrate into the servers, as this is the focus of the paper. There are numerous published studies that focus on the performance of cold plate solutions. Patterson et al. (2004) conducted a numerical study of heat transfer in stacked microchannels (application to high-end electronics cooling). A number of different internal (water) flow arrangements were studied with the objective of minimizing wall...

NOTE: All illustrations and photos have been removed from this article.



More articles from ASHRAE Transactions
Comparison between underfloor supply and overhead supply ventilation d..., January 01, 2007
Capture index: an airflow-based rack cooling performance metric., January 01, 2007
Characterization of a high-density data center., January 01, 2007
Advanced solar irradiance model and procedure for spectral solar heat ..., January 01, 2007
Two-dimensional CFD and conduction simulations of heat transfer in hor..., January 01, 2007

Looking for additional articles?
Search our database of over 3 million articles.

Looking for more in-depth information on this industry?
Search our complete database of Industry & Market reports by text, subject, publication name or publication date.

About Goliath
Whether you're looking for sales prospects, competitive information, company analysis or best practices in managing your organization, Goliath can help you meet your business needs.

Our extensive business information databases empower business professionals with both the breadth and depth of credible, authoritative information they need to support their business goals. Whether it be strategic planning, sales prospecting, company research or defining management best practices - Goliath is your leading source for accurate information.