|
Article Excerpt
INTRODUCTION
The last couple of years have seen significant processor micro-architecture changes driven mainly by power consumption and heat dissipation issues. Diminishing performance returns with increasing power budgets have forced manufacturers to accelerate the development of multi-core architectures and sophisticated power management strategies. Barring the immediate negative consequences of an emerging programmability crisis, this push towards multi-core architectures has managed to slow the steady Thermal Design Power (TDP) increase for new processor generations for the near future.
The processor power roadmaps notwithstanding, total rack power dissipations have been following a different road-map. Schmidt (2005) shows actual rack heat load data superimposed over ASHRAE predicted curves. In 2005, actual data shows a single rack of IBM BladeCenters dissipating 95,540 Btu/hr (28 kW) of heat. Using the same ASHRAE data as Schmidt (2005) shows rack heat loads growing at a compounded annual growth rate (CAGR) of between 16.5 and 30.6% prior to 2005, and slowing to between 3.8 and 4.6% thereafter (range covers traditional 1U servers as well as blade servers). The increasing rack heat loads are making data center thermal management increasingly difficult. As a result, increasing numbers of data center operators are starting to consider alternative cooling solutions such as in-chassis liquid cooling solutions.
There are numerous studies in the literature that cover alternative cooling solutions. This paper will focus on liquid-based cooling solutions - both rack level and device-level (in - chassis).
Sorell and Rodgers (2006), ASHRAE (2006), and Ellsworth et al. (2007) provide good overviews of a variety of liquid-based cooling solutions. Sorell and Rodgers (2006) evaluated eight different cooling scenarios, including the currently entrenched computer room air handler-based solutions (CRAHs), as well as upcoming in-chassis cooling solutions that use liquid-cooled cold plates on hot components. They found that the greater the number of media the waste heat had to be transferred through, the more energy inefficient the solution. To this end, Sorell and Rodgers conclude that the in-chassis liquid cooling solutions provide the most energy efficient cooling solution for data centers. ASHRAE (2007) deals with the topic of liquid-cooled data centers in great depth in the book entitled "Liquid Cooling Guidelines for Datacom Equipment Centers". The book covers facility piping design (Chapter 3), liquid cooling implementation for datacom equipment (Chapter 4), and liquid cooling infrastructure requirements for chilled-water systems (Chapter 5). Ellsworth et al. (2007) provide an overview of a number of liquid cooling architectures for (high availability) data centers. The study looks at various architectures that provide varying levels of redundancy and availability, and briefly discusses the suitability of each architecture type to overall rack heat loads. The overall message of the paper is that liquid cooling is almost becoming a necessity as rack heat loads increase.
Schmidt et al. (2005a), Villa (2007), and Regimbal (2007) discuss both rack level and device-level cooling solutions. Schmidt et al. (2005a) have designed a fanless air-to-liquid heat exchanger that mounts to the rear (exhaust side) of a rack of servers (relies on the server fans to push the air through the heat exchanger). In one implementation, a central distribution unit (CDU) is used to condition facility water, via a mixing valve, to a temperature above the facility's dew point - the conditioned water is then distributed to a number of rack-mounted heat exchangers. Schmidt et al. report that, on average, a typical heat exchanger can provide a cooling capacity of 51,180 Btu/hr (15 kW) when 64.4[degrees]F (18[degrees]C) water is provided at 37.8 liters/min (10 gal/min). Villa (2007) discusses the advent of liquid cooled data centers, and in particular, how to prepare for extreme heat density racks (102,400+ Btu/hr (30+ kW) per rack). Data is presented on the projected timing for: (1) the deployment of liquid cooled enclosed racks, (2) delivery of liquid directly to the processors, and (3) deployment of a combination of (1) and (2). Regimbal (2007) conducted an in-depth analysis of the cooling requirements for Pacific Northwest National Lab's (PNNL) high performance computing center. He compared product offerings that included rack/ ceiling mounted supplemental cooling solutions, enclosed rack solutions, and a solution that uses direct liquid (spray) cooling of the processors. His basic conclusion is that some form of advanced cooling is necessary for PNNL, and of the three solutions evaluated, the device-level cooling solution provided the best approach.
Hannemann and Chu (2007) studied alternative cooling solutions, including device-level cooling, while Cader et al. (2006a) and Cader et al. (2007a) compare air handler-based facilities to device-level cooling for high performance microprocessors. Hannemann and Chu's (2007) study encompassed traditional raised floor air-cooling, rear door heat exchanger cooling (water and R-134A), and device-level cooling of the processors (water and R-134A). The primary objective of the paper was to determine the optimal solution to cool data centers that would initially house 51,182 Btu/hr (15 kW) racks, and then 102,400 Btu/hr (30 kW) racks. The paper concludes that the refrigerant-based liquid-cooled cold plate cooling solution provides the best alternative. Cader et al. (2006b) and Cader et al. (2007b) provide details on a Department of Energy (DOE) energy efficient data center study currently underway at the Pacific Northwest National Labs (PNNL). The main focus of this study is to quantify the ability of liquid cooling to significantly impact the energy efficiency of a traditional air-cooled data center. In collaboration with a liquid (spray) cooling vendor, PNNL has evaluated the vendor's cooling solution for a number of items including required facility preparation, solution installation, solution management, thermal performance, application performance (running production software), maintainability, reliability, availability, and total cost of ownership. Over the past three years of the study, the liquid-cooled solution has performed well enough for the DOE and PNNL to build a small-scale liquid-cooled supercomputer. Results from the liquid-cooled supercomputer at PNNL will be discussed later in the paper.
The present paper will focus on several aspects of relevance to liquid-cooled data centers. In particular, the paper will cover plumbing and facility water delivery for retrofit and greenfield data centers, implementation of liquid-cooling in a data center (encompassing IT equipment and facility preparation, and equipment installation), and will also provide some brief results from a liquid-cooled data center installed at PNNL. Finally, the paper...
|
