IMPROVED CFD MODELING OF A SMALL DATA CENTER TEST CELL

Waleed A. Abdelmaksoud, H. Ezzat Khalifa and Thong Q. Dang, Syracuse University, 263 Link Hall, Syracuse, NY, USA and Roger R. Schmidt and Madhusudan Iyengar IBM Corporation, 2455 South Rd., Poughkeepsie, NY, USA Phone: (315) 447-8644 Email: wamarouf@syr.edu

ABSTRACT An improved CFD model is presented to predict the thermal field in a small data center test cell. CFD analysis reported in an earlier paper [19] tended to exaggerate hot and cold spots in a small data center test cell as a result of the inability of the CFD model to predict the correct level of mixing of the cold air emanating from the perforated floor tiles and the warmer room air. The same can be said about the mixing of the hot air stream exiting the rack and the surrounding room air. The overall RMS error in the earlier study was 4C between numerical results and test data. Another indication of low mixing in the CFD results is illustrated in the comparison of the temperature profile along a vertical line in front of the rack between CFD results and test data. While the temperature of the cold flow from the under floor plenum entering the data center was measured to be 12.8C, temperature measurement at the same location via a temperature mapping tool showed that it is 15C at 6" above the floor and rising quickly to 17C at a height of 4 ft. The present investigations yield several improvements to the CFD model, including modeling of the perforated tile flow and rack exhaust flow to conserve both mass and momentum, inclusion of the effects of buoyancy, and an improved thermal boundary condition for the floor. With these improvements, the overall RMS error was reduced to 2C between numerical results and the same test data reported in [19]. KEY WORDS: tile modeling, rack modeling, leakage, mixing, buoyancy NOMENCLATURE Archimedes Number, [-] volumetric thermal expansion coefficient, 1/K gravitational acceleration, m/s2 length scale, m power, kW tile porosity, [-] temperature, C, or K characteristic velocity, m/s

convection air cooling remains the most common thermal management method [3]. With high power density and thermal loads, rack and tile airflows had to be increased, and the resulting velocity and temperature fields became much more complex. These in turn increased the potential for highly nonuniform temperature distributions and the presence of undesirable hot spots [e.g., 4, 5], necessitating the use of design and optimization tools of increasing sophistication. Foremost among these tools are the use of numerical simulations, especially computational fluid dynamics (CFD). Previous CFD investigations of flow and temperature distributions in data centers [6-21] have employed relatively coarse grids and the widely-accepted k- turbulence model with standard wall functions, and much of the published work deals with low-power racks. In all of these references buoyancy was ignored, presumably on the basis of an assumption that given the relatively high airflows per unit floor area compared with office and residential buildings and the moderate air temperature differentials in the data center, cooling is dominated by forced convection. Some of these papers also reported comparison with test data [e.g., 8, 15, 19, 20 and 21]. Of specific relevance to the work presented in this paper is the work of Iyengar et al [19], who reported the results of a detailed experimental study of the temperature distribution in a small data center test cell, along with comparisons of the experimental data with CFD predictions. These papers reported temperature prediction errors in excess of 5C and root mean square (RMS) error of ~4C. The relatively large temperature errors could be attributed to one or more of the following factors: (1) coarseness of the meshes employed; (2) inadequacy of the turbulence models, (3) inaccurate flow/thermal boundary conditions on the floor and ceiling to account for tile perimeter or ceiling leakage flows; (4) incorrect modeling of the tile and rack exit flows; and (5) neglecting of buoyancy (Herrlin and Belady [22] presented a numerical study in which the effect of gravity was taken into account and showed that buoyancy effects can be significant in data centers). To obtain improved predictions, it is important that higher-fidelity CFD models for high-density data centers be developed and validated against a carefully assembled body of experimental data.

Ar

g L P

T V

INTRODUCTION The escalating power dissipation of modern high-density data centers [1] has made the cooling of such data centers much more challenging. While water cooling of servers and racks is gaining popularity in high-density data centers [2], forced

IBM DATA CENTER BENCHMARK TEST CASE The IBM data set reported in Ref. [19] is chosen as the benchmark test case because it deals with a relatively highpower rack (23 kW rack at 2,400 cfm, corresponding to a temperature rise across the rack of ~20C), and provides a relatively dense 3D temperature measurement map. The layout of the IBM data center experimental facility designed for CFD model validation is shown in Fig. 1. The test cell contains a single 23 kW simulated air-cooled computer rack and a single computer room air conditioner (CRAC). Detailed description of this experimental facility, the test conditions, and the measured temperature data are reported in Ref. [19]. The test domain measured 83.61 m2 (900 ft2 covered with 25 x 9 tiles, each measuring 2 2), and had a floor -to-ceiling x height of 3.05 m (10 ft). Three perforated tiles with a 27% porosity ( = 0.27) were located in front of the rack (number 1 is the tile closest to the rack, and number 3 is the tile farthest from the rack), and five open tiles were located on the other side of the CRAC away from the simulated server rack (the numbering start from left to right looking at the plan view) to bypass enough supply air to prevent over-pressuring of the underfloor plenum.

for many of the components in a data center to limit the grid size of a large data center to several millions computational cells. Note that in the IBM benchmark test case in Ref. [19], the smallest length scale is even smaller, i.e., the gap around the floor and ceiling tiles which accounts for the leakage flow, which is as much as 50% of the CRAC flow. Table 1. Test conditions Parameter Rack power Rack flow rate CRAC flow Rack tile 1 flowrate & temperature Rack tile 2 flowrate & temperature Rack tile 3 flowrate & temperature Bypass tile 1 flowrate & temperature Bypass tile 2 flowrate & temperature Bypass tile 3 flowrate & temperature Bypass tile 4 flowrate & temperature Bypass tile 5 flowrate & temperature Tile perimeter leakage flow Underfloor leakage flow Value 23 kW 2,400 cfm 10,200 cfm 800 cfm, 12.8C 780 cfm, 12.6C 810 cfm, 12.5C 700 cfm, 12.6C 730 cfm, 12.9C 690 cfm, 13.7C 240 cfm, 13.9C 680 cfm, 13.5C 3,240 cfm 1,530 cfm Comment Setting on rack Setting on rack Manufacturers Measured Measured Measured Measured Measured Measured Measured Measured Estimated [13] Calculated

CRAC

5.49m (18)

15.24m (50)

Fig. 1: Test cell plan view (2x2' tiles; 10 height). The CRAC had a manufacturers rated cooling capacity of 70 kWth and assumed to deliver 4.81 m3/s (10,200 cfm) of conditioned cold air to the test cell through a 0.43-m-high (17) underfloor plenum. The simulated rack, which measured 0.61 m (width), 1.22 m (depth) and 2.0 m (height), was designed to flow 1.133 m3/s (2400 cfm) of air at a design power output of 23 kW. A summary of the test conditions and other relevant information is given in Table 1. The air flow rate through each perforated tile and server rack was measured by the Alnor velometer flow hood. The temperature measurements were taken by thermocouples mounted on a movable cart, and was reported at 9 different heights (0.5 ft to 8.5 ft) over a 7x12 horizontal grid, with grid points at the tile centers [19]. The temperature measurements were estimated to be accurate to better than 1C. The CFD model employed in the Ref. [19] was constructed using the commercial CFD software Flotherm. The solver employed the standard k- turbulence model. With length scales in the problem varying from the size of the room (10s of feet) to the smallest size of less than 1" corresponding to the tile openings (pores), it is clear that models must be developed

Rack

The grid size employed in Ref. [19] comprised 130,000 cells, corresponding to an average cell size of 5" (smallest grid size is on the order of 2"). As a result, many simplifications were made in the CFD model. These include: The available flow area for each 2 x 2 tile was assumed to be 4 ft2. The reasoning was that since the size of the small openings (pores) is very small (often less than 1"), the potential cores of these jets would be a few inches, and these small jets would quickly merge into a single large jet. Note that the actual open area of the perforated tile used in the IBM benchmark test case was 27%, or 1.08 ft2, and the pattern of the perforated tile is nonsymmetric with openings having length scale as small as ". As a result, the flow rate of the air can be matched with measured data, but not the momentum. Same as above, the available face area at the inlet and exit of the rack was assumed to be fully open. Because the simulated servers are placed in a standard computer rack with perforated doors estimated to be on the order of 35% open, the flow rate through the rack is conserved, but the momentum flux of the exhaust air is not conserved. The leakage flow was modeled as flow sources entering the data center through the floor tiles and the ceiling at a constant temperature. Note that the uncertainties are high

on these estimates, both in terms of leakage flow distribution and temperature. Buoyancy was neglected in the CFD simulation. The under floor plenum was not modeled and hence, the tile flow entering the data center was treated as an inflow boundary (into the computational domain) with uniform condition over each tile.

effect. Another possibility is due to the thermal boundary condition imposed in the CFD model, i.e., the ceiling temperature, which was set at 20C. A follow-up study on the IBM benchmark test case was carried out by Zhang et al. (Ref. [18]) to improve the results. Some of the issues related to possible model inaccuracies described earlier were investigated. These include effects of detailed modeling of the racks, along with grid size, turbulence models, and handling of the leakage flow. Much of that work was concentrated on the detailed modeling of the rack, which included three different levels of details, varying from a simple black box to a model where some details of the server simulators (i.e., fans and column shaped heaters) were included. Grid size was varied from 80,000 cells to 600,000 cells. Another attempted improvement to the CFD model was to investigate different ways of modeling the leakage flow, primarily through the variation of the leakage flow temperature. The conclusion from this study was that the model improvements investigated in the paper fail to improve the agreement between the CFD predictions and the test data. In particular, the RMS errors at different heights were still hovering around 4C. BUOYANCY AND RADIATION EFFECTS Buoyancy is a dominant driving force in many indoor environmental flows, especially in the thermal plumes of heated or cooled objects in contact with cold or warm ventilation air jets. The Archimedes number (Ar) is the preferred parameter to use in estimating the relative importance of natural and forced convection 1 [23]. Archimedes number is a measure of the relative magnitude of buoyancy and inertia forces in the flow:

Figure 2 presents a sample comparison from Ref. [19] of the measured and computed temperatures in a horizontal plane 8.5 ft above the floor (1.5 ft below the ceiling). The general conclusion from the study of Iyengar et al. [19] was that the CFD model tends to exaggerate hot and cold spots as a result of the low mixing level predicted by the CFD model between the cold tile flow entering the data center and the warm room air, and between the hot air stream exiting the rack and the surrounding room air. The error was found to be the highest in the area around the racks inlet/exhaust air path. Another indication of low mixing in the CFD results was the error in the predicted temperature profile along a vertical line in front of the rack (actually, along the center line of the first perforated tile in front of the rack). While the temperature of the cold flow from the under floor plenum entering the data center was measured to be 12.8C (Table 1), temperature measurement at the same location via the temperature mapping tool showed that it is 15C at 6" above the floor and rising quickly to 17C at a height of 4 ft. If the cold jet coming out of the tile is a single jet (as assumed in the CFD model), then it is expected that its potential core would extend 3-4 jet diameters (or 6-to-8 ft in this case), and the temperature would remain close to the underfloor plenum air temperature at 12.8C. Since the measured temperatures are 2.2-to-4.2C higher, it is speculated that warmer room air must have been entrained into the tile exit flow very quickly, a few inches above the floor.

Ar =

gLT
V2

(1)

Fig. 2: Comparison of temperature field between CFD results (left) and test data (right) at 8.5 ft height (near ceiling) taken from Ref. [19]. Another interesting observation from the results obtained by Iyengar et al. [19] relates to the potential importance of buoyancy. Note that the CFD results did not include buoyancy effect. Inspection of the temperature data of Fig. 2 in the area from behind the rack to the CRAC inlet and toward the opposite wall reveals that the measured temperature is significantly higher than the CFD results (~3C), indicating that the test data show the rise of the hot exhaust gas due to buoyancy while the CFD results of Ref. [19] did not show this

in which is the volumetric thermal expansion coefficient of the fluid ( = 1/T for a perfect gas), g is the gravitational acceleration, L is a vertical length scale, T is the temperature difference between the hot and cold features (wall-room air or jet-room air), and V is a characteristic velocity (the velocity of the air jet, say). An Archimedes number of order 1, Ar ~O(1), indicates that the buoyancy and inertia forces are of the same order of magnitude and that both of them must be taken into account in numerical simulations. A very small Archimedes number indicates that the buoyancy effect is negligible, while the opposite is true for very large Archimedes numbers. In an air-cooled data center, the length scale of interest is the height of the rack (typically ~2 m). There are many characteristic temperature differences that can be used in a data center ranging from local ones, e.g., difference between inlet and exit of a server, to global ones, e.g., from cold aisle

The Richardson number, Ri, is an alternative expression of the relative magnitude of buoyancy and inertia forces.

tile exit to CRAC inlet. For whole data center simulations, the latter is more appropriate as a T scaling parameter. For a data center with power dissipation P [kW] and Q [m3/s] of cooling air supply, T can be estimated from:

radiative heat transfer from the human body often exceeds convective heat transfer [24], radiative heat transfer in highdensity data centers accounts only for ~1% of the heat generated in the racks and may be safely neglected in whole data center CFD analysis. PERFORATED TILE MODEL Another important factor that is commonly ignored in data center CFD analysis is the effect of momentum conservation on entrainment and mixing of the air emerging from a perforated surface (tile or rack back door). A detailed, poreby-pore definition of the velocity boundary conditions at the tile or rack exit in the CFD is a challenging task and is impracticable in whole data center simulations. A simplified model of the tile that yields a reasonably accurate prediction of the flow and temperature fields in front of the rack must be developed. The simplest and most widely used approach is to assume a uniform velocity boundary condition (BC) over the entire face area of the tile. While mass conservation is assured in this case, the momentum of the air will not be conserved, and the mixing and entrainment of surrounding warm air will be under-estimated. In the other extreme, if the net open area of the tile (aggregate area of the pores) is concentrated in a single central opening, a high-velocity jet with a long potential core will result. While this reduced-area jet will have both the correct flow rate and momentum and may produce the correct far-field entrainment characteristics, its concentrated geometry would lead to erroneous predictions of velocity and temperature in front of the rack in the jets potential core and beyond (possibly extending as far as the top of the rack). A simple, yet reasonably accurate method for modeling the tile exit flow is the momentum source method described in a companion paper by Abdelmaksoud et al. [25], Schwarz [26] and Chen [27]. The method proposed by Abdelmaksoud et al. [25] was validated against experimental data obtained in an isothermal test chamber. One of the important observations from the collected data is that the core velocity for a 25% perforated tile discharging 1000 cfm into the chamber is ~2 m/s (Fig. 3). This is significantly greater than the 1.27 m/s value based on a fully open tile (Fig. 3), but it is also significantly lower than the velocity based on the aggregate pore area (25%), which would have been 5.07 m/s. The large velocity deficit is due to entrainment and momentum dissipation in small recirculation regions surrounding the jets issuing from pores before they expand and merge together in the first few inches above the tile. The momentum source method corrects for the momentum deficit in the jet issuing from the entire face area of the tile by adding a body-force field in the computational volume immediately above the tile face [25]. An abbreviated description of this method is given here. Consider the case where the perforated tile employed in the CFD model is 100% open. Mass conservation is assured by simply setting the normal velocity through the tile as (Q/Atile), where Atile is the physical face area of the tile (i.e., 4 ft2 in this case). Clearly, the correct jet momentum is obtained in this case only if the perforation of the tile is 100% ( = 1). If the tile is not fully

T =

P , c p Q

(2)

where and cp are the density [kg/m3] and specific heat [kJ/kg-K] of air, respectively. The characteristic velocity for a data center is taken to be the average velocity of the air issuing from the floor tiles, viz.,

V=

Q , At

(3)

with At representing the aggregate area of the open tiles. Substituting Eqs. (2) and (3) in the definition of Ar in Eq. (1), we obtain the following expression for an Archimedes number appropriate for a data center 2:

Ar =

gLP c p QV 2

(4)

For a tile flow rate of 0.377 m3/s (~800 cfm), the exit velocity will be ~1.0 m/s (~3.33 ft/s), based on three 2 x 2 fully open perforated tiles supplying ~1.133 m3/s (~2,400 cfm) per 23 kW rack, T would be ~17C, and the Archimedes number for this data center would be 1.1, indicating that natural convection will be nearly as important as forced convection under these conditions. Note that T in typical rack flows in data centers is ~10C. This discussion shows that, contrary to common practice, the effect of buoyancy can be important in data center CFD analysis as was highlighted in Ref. [22]. Not only will buoyancy affect the distribution of the cold air discharged from the floor tiles, but also it will influence the trajectory and mixing of the hot air jets emerging from the back of the servers. To verify the importance of buoyancy, we decided to include buoyancy in our calculations, and compare the results with and without buoyancy to the experimental results reported in [19]. With respect to computational time, the inclusion of buoyancy effect involves solving the energy equation during the iterative process of the flow equations. As a result, the additional computational time is relatively small, ~30% more. For this reason, we recommend that buoyancy be included in CFD simulations of data centers, unless it is obvious that it is negligible (i.e. isothermal case). As for radiation from the hot surfaces of the racks to the walls, ceiling and floor of the data center, a simple analysis shows that, unlike buildings designed for human occupancy where

This is by no means the only way of expressing Ar for a data center, but it is adequate for relative estimation.

open, then the correct velocity of a jet issuing from a pore is ~(Q/Atile), corresponding to the tile total momentum flux of Q(Q/Atile). In this case, a body force term is added in the momentum equation in the vertical direction to correct for the difference in momentum. The simplest form of the body force field is [25],

Figure 4 depicts the three tile modeling concepts used in this paper: a) fully open face (no momentum conservation); b) momentum source; c) two slots. Both models (b) and (c) account for momentum conservation, albeit in different ways. A similar treatment of momentum was applied to the perforated rack rear door. NUMERICAL MODEL In this section, we summarize the CFD model of the IBM benchmark test case using the different tile models discussed above. All CFD simulations were performed using a commercial CFD software package (FLUENT version 6.3). The flow was assumed incompressible, and the ReynoldsAveraged Navier-Stokes (RANS) equations are solved with the SIMPLEC algorithm. All convective transport terms are discretized using a 2nd order-accurate upwind scheme, while the diffusion terms are discretized using a 2nd order-accurate central-difference scheme. Pressure interpolation was achieved with a 2nd order-accurate scheme. Computational grids were generated using the commercial grid-generation software Gambit. The standard k- turbulence model was employed. Buoyancy was modeled using the incompressible ideal gas method. Table 2 summarizes the inputs used in the CFD simulations. Table 2. CFD input data for IBM benchmark test cases Parameter Rack CRAC flow Rack tile 1 flowrate Rack tile 2 flowrate Rack tile 3 flowrate Bypass tile 1 flowrate Bypass tile 2 flowrate Bypass tile 3 flowrate Bypass tile 4 flowrate Bypass tile 5 flowrate Floor leakage Ceiling leakage Side Walls Buoyancy Turbulence Value 23 kW, 2,400 cfm 10,200 cfm 800 cfm, 12.8C 780 cfm, 12.6C 810 cfm, 12.5C 700 cfm, 12.6C 730 cfm, 12.9C 690 cfm, 13.7C 240 cfm, 13.9C 680 cfm, 13.5C 3,240 cfm 1,530 cfm, 20C Adiabatic Incompressible ideal gas Standard k-

Fy =

Q Q 1 Q A A tile tile

(5)

In the above expression, Fy is the body force per unit volume in the vertical direction, and is the volume of a finite region above the tile where this body force field is to be distributed. A volume of 24 x 24 x 4 above the 2 x 2 perforated tile was found to produce good agreement with the experimental results [25]. Figure 3 shows a sample comparison of the velocity profile between body-force CFD model and test data at a height of 1 ft above a 25% perforated tile. The results show very good prediction of the velocity magnitude in the potential core.

Fig. 3: Comparison of velocity profile between test data and models for symmetrical perforated tile with = 25%. Another relatively simple approach is to model the perforated tile as two or more rectangular openings (slots) separated by blank lanes or lands [25]. This allows for the surrounding air to penetrate between the slots and be more effectively entrained by the jets issuing from these openings, while also accounting for mass and momentum conservation. The disadvantage of this method over the body-force method is the higher grid requirement, which depends on the number of openings per tile used in the model [25].

& & & & & & & &

temp. temp. temp. temp. temp. temp. temp. temp.

(a)

(b)

(c)

Fig. 4: Perforated tile models; (a) fully open model, (b) momentum source model, (c) two-slot model.

The rack was modeled as a box with openings in the front, and either a large opening or several open slots in the back. We employed the recirculation model of FLUENT, in which the airflow inside the rack is not resolved. In this model, the air enters the rack and exits with a temperature differential computed from the rack flow rate and heat added. The advantage of this recirculation model is that there is no need to model the rack interior, obviating the need to grid the internal volume of the rack and the associated computational overhead.

In the present study, the numerical computations were carried out with a fine structured grid of 5,000,000 cells (see Fig. 5). The computational time is 10 hours (Case 1) to 20 hours (Case 3) using 28 processors on a Beowulf system equipped with AMD 1.6 GHz Opteron processors. The smallest cell cubes (1 x 1 x 1 in) were located around the rack and the rack tiles, and the cells grow to (1 x 1 x 4 in) away from the rack.

more accurate definition of the floor thermal boundary conditions be obtained from a CFD model incorporating the underfloor plenum, with a suitable thermal resistance interposed across the floor tiles. However, because no data were available near the ceiling, we set the ceiling temperature to be constant at 20C.

Fig. 5: Model grid distribution (top view). CASE STUDIES Four CFD cases are presented in this paper. These four cases were selected to reveal the importance of accounting correctly for tile exit momentum, buoyancy, and floor thermal boundary conditions in CFD simulations. Table 3 lists the prescribed floor temperature boundary conditions, tile and rack exit momentum treatment, and buoyancy treatment in the four cases analyzed. All cases use the input data listed in Table 2. Table 3. CFD Case studies for IBM benchmark data center Fig. 6: Prescribed floor temperature distribution in C. RESULTS AND DISCUSSION Figure 7 shows temperature profiles at the front and back of the rack. The circle markers represent the measured data, while the solid lines represent the different CFD simulation cases listed in Table 3. The measured data were taken along a vertical line extending from the center of the tile closest to the rack (i.e., 1 ft away from the rack), except for Case 3, where the CFD results are averages of two vertical lines each of which is centered on one of the slots. This was necessary to avoid the blocked land between the two slots.
3.0 2.5 2.0
Height (m)

Case No. B.C. Uniform floor temperature B.C. Tile & rack fully open model Tile & rack momentum source model Tile & rack slots model Buoyancy

1 --x

2 x - -

3 x --

4 x - -x

Front of Rack

Back of Rack

1.5 1.0 0.5 0.0 10.0

Uniform floor temperature B.C. (): 13oC floor and floor leakage temperature. Non-uniform floor temperature B.C. (X): floor and floor leakage temperature are based on the test data at 0.5 ft. A blank cell marked as -- means that the model is not applicable.

15.0

20.0

25.0 Temp.(C)

30.0

35.0

40.0

Experimental data

Case1

Case2

Case3

Case4

Case 1 is similar to the previous work of Iyengar et al. [19], in which the temperatures on the floor and the ceiling were set to constant values. Cases 2, 3 and 4 account for tile flow exit momentum. Cases 2 and 4 (with and without buoyancy) are designed to assess the effect of buoyancy in CFD simulations. Figure 6 presents the contours of the measured temperatures 6 above the floor. It is evident that a constant floor (and floor leakage) temperature assumption is not accurate. Since the floor temperature was not measured in Ref. [19], we took the floor and floor leakage temperature boundary conditions in Cases 2, 3 and 4 to be equal to the experimental air temperatures 6 above the floor. In future work, it is advisable that, absent measurements of the actual floor temperature, a

Fig. 7: Temperature profile at front and back of the rack. It can be seen in Fig. 7 that insofar as the temperature in front of the rack is concerned, it is not possible to state categorically that the improved models yield appreciably better results than the simpler (Case 1) model. Nevertheless, it is worth noting that the temperature errors in front of the rack for Case 3 (slot model with buoyancy and non-uniform floor temperature) are generally within the experimental uncertainty (1C). Note that, in Case 3, the fluctuations in the temperature profile at the exit of the rack come from the modeling of the rack exit door with a discrete number of openings separated by blank lanes or lands (see Fig. 4c). This can also be seen in the

temperature contour plot discussed later (Fig. 11). It is worth pointing out that in Cases 2 and 3 where buoyancy is turned on, the density at the rack exit is lower than in Case 1 (by about 5%), and hence the T across the rack for Cases 2 and 3 is ~1oC higher than Case 1 for the same heat load of 23 kW (the rack flow rate is 2,400 cfm in all these cases). This explains why the rack exit temperature of Case 2 is higher than Case 1. On the other hand, the use of the multiple-slot model in Case 3 results in the lowest rack exit temperature, even though both the rack inlet temperature and the T across the rack in Case 3 are higher than Case 1. This is due to the intense mixing between the hot air leaving the rack and the surrounding air through the gaps between the slots in Case 3. Figures 8, 9 and 10 show temperature contour plots at heights of 0.5 ft, 4.5 ft and 8.5 ft from the floor, respectively, for the experimental measurements of Ref. [19] and for Cases 1, 2, 3 and 4 of the present work. The partial domain shown in these contour plots extends only from the right side of the CRAC (see Fig. 1 and Fig. 5), covering only the area where experimental data were taken. Note that by imposing the floor temperature boundary condition shown in Fig. 6 in Cases 2-4, the temperature contours at the 0.5 ft height match the test data better than the constant-temperature boundary condition imposed in Case 1 [19]. Overall, Cases 2 and 3, which account for tile flow momentum, buoyancy and the non-uniform floor temperature, exhibit more smearing of the temperature contours, making them qualitatively closer to the contours of the measured data. This is particularly true for Case 3 and can be attributed to the enhanced mixing due to the more accurate accounting for momentum and entrainment. In all four CFD cases presented here, the region where the CFD results deviate most from the test data is the lower rightcorner region. Here, all the CFD results predict higher temperature level than the test data, with Case 4 predicting the highest temperature level, while Case 3 predicting the lowest temperature level (see Figs. 9 and 10). Case 4, which is Case 2 without buoyancy, shows the disadvantage of neglecting buoyancy even with better accounting for momentum. By including buoyancy, the rack exhaust warm air flows upward toward the ceiling, as the experimental data indicate. Without buoyancy (Case 4), the warm exhaust from the rack lingers at the lower levels (see Fig. 9 lower right corner). Fig. 10: Temperature contours at 4.5 ft height from the floor. Another observation from Figs. 9 and 10 is that the cold air jet discharged vertically from the tiles tends to show a pronounced deflection to the left (toward the CARC side) in the two cases without buoyancy (Cases 1 and 4). It is surmised that the absence of buoyancy in Case 4 causes the warm rack exhaust to recirculate at lower elevations between the rack and the right wall, streaming to the front of the rack, and causing this deflection. With buoyancy, the warm rack exhaust rises to higher elevations, and less of it is likely to be recirculated around the rack side and interfere with the cold air jet emanating from the tile. The combined effects of momentum conservation and buoyancy are also presented in Fig. 11, which depicts temperature contour plots in a vertical plane bisecting the rack (front-to-back). The effects of enhanced mixing/entrainment with the two-slot model, and of buoyancy are clearly evident. We note that the shorter vertical throw of the cold air jet in Cases 1 and 4 (without buoyancy) compared with Case 2 (with buoyancy) is an artifact of the fact that the contours are in a plane cutting through a left-deflected jet (a separate analysis of a symmetrical case with no deflection shows that the cold air jet has a shorter throw in the presence of buoyancy). Fig. 8: Temperature contours at 0.5 ft height from the floor.

Rack

Measured

Case1

Rack

Case2

CRAC

Rack

Case3

Case4

Rack

Rack

Fig. 9: Temperature contours at 4.5 ft height from the floor.

Case1

Case2

Measured Rack Rack Rack

Case3

Case4

Rack

Rack

Fig. 11: Temperature contours in a vertical plane bisecting the rack (front-to-back). SUMMARY AND CONCLUSIONS The main purpose of the present study was twofold: (1) to assess the importance of conserving both mass and momentum at tile exit and rack rear doors in data center CFD simulation, and (2) to highlight the importance of including buoyancy for Archimedes number of order ~1 or higher. It is shown that the inclusion of these effects can have a significant influence on CFD simulations of the temperature fields in the vicinity of the racks. By accounting for these effects in the CFD model, we were able to obtain better agreement between the CFD simulation results and the measured data. The overall RMS temperature error was reduced from the ~4C obtained from CFD simulations that do not account for these effects, to ~2C when these effects were accounted for.

ACKNOWLEDGEMENT The authors express their thanks to Dr. Basman Elhadidi for his helpful suggestions and insightful discussions. REFERENCES [1] ASHRAE, Datacom Equipment Power Trends and Cooling Applications, ASHRAE, Atlanta, GA, USA, 2005. [2] ASHRAE, Liquid Cooling Guidelines for Datacom Equipment Centers, ASHRAE, Atlanta, GA, USA, 2006. [3] ASHRAE, High Density Data Centers: Case Studies and Best Practices, ASHRAE, Atlanta, GA, USA, 2008. [4] Schmidt, R., Iyengar, M., Beaty, D. and Shrivastava, S., Thermal Profile of a High Density Data Center Hot Spot Heat Fluxes of 512 Watts/ft2, ASHRAE Trans., Vol. 111(2), 2005. [5] Schmidt, R., Iyengar, M., and Mayhugh, S., Thermal Profile of World's 3rd Fastest Supercomputer - IBM's ASCI Purple Cluster, ASHRAE Trans., Vol. 112(2), 2006. [6] Hamann, H., J. Lacey, M. O'Boyle, Schmidt, R., and Iyengar, M., Rapid 3-Dimensional Thermal Characterization of large Scale Computing Facilities, IEEE Transactions of Components and Packaging Technologies, Vol. 31, No. 2, June, pp. 444-448, 2008. [7] Schmidt, R., Effect of Data Center Characteristics on Data Processing Equipment Inlet Temperatures, Proceedings of InterPACK Conference, Kauai, HI, USA, July 8th 13th, 2001.

[8] Noh, H., K., Song, and S. K. Chun, The Cooling Characteristic on the Air Supply and Return Flow System in the Telecommunication Cabinet Room, Proceedings of International Telecommunications Energy Conference (INTELEC), 33-2, pp. 777-784, 1998. [9] Patel, C., C. Bash, and C. Belady, Computational Fluid Dynamics Modeling of High Compute Density Data Centers to Assure System Inlet Air Specifications, Proceedings of the Pacific Rim /ASME International Electronics Packaging Technical Conference and Exhibition (InterPACK), Paper no. IPACK2001-15622, Kauai, HI, USA, 2001. [10] Schmidt, R., Karki, K., and Patankar, S., Raised Floor Data Center: Perforated Tile Flow Rates for Various Tile Layouts, Proceedings of the Intersociety Conference on Thermal Phenomena (ITherm), pp. 571-575, 2004. [11] Iyengar, M., Schmidt, R., Sharma, A., McVicker, G., Shrivastava, S., Sri-Jayantha, S., Anemiya, Y., Dang, H., Chainer, T., and Sammakia, B., Thermal Characterization of Non-Raised Floor air Cooled Data Centers using Numerical Modeling, Proceedings of the InterPACK Conference, San Francisco, CA, USA, July 17th-22nd, 2005. [12] VanGilder, J., and Schmidt, R., Airflow Uniformity Through Perforated Tiles in a Raised Floor Data Center, Proceedings of the InterPACK Conference, Paper no. IPACK2005-73375, San Francisco, CA, USA, 2005. [13] Radmehr, A., Schmidt, R., Karki, K., and Patankar, S., Distributed Leakage Flow in Raised Floor Data Centers, Proceedings of the InterPACK Conference, Paper no. IPACK2005-73273, San Francisco, CA, USA, 2005. [14] Shrivastava, S., Sammakia, B., Schmidt, R., and Iyengar, M., Comparative Analysis of Different Data Center Airflow Management Configurations, Proceedings of the InterPACK Conference, Paper no. IPACK2005-73234, San Francisco, CA, USA, 2005. [15] Shrivastava, S., Iyengar, M., Sammakia, B., Schmidt, R., and VanGilder, J., Experimental-Numerical Comparison for a High-Density Data Center: Hot Spot Heat Fluxes in Excess of 500 W/ft2, Transactions of the IEEE Components Packaging and Manufacturing Technologies IEEE Trans. CPMT, Vol. 32, No. 1, pp. 166-172, 2009. [16] Bhopte, S., Schmidt, R., Agonafer, D. and Sammakia, B., Optimization of Data Center Room Layout to Minimize Rack inlet air Temperature, Proceedings of the InterPACK Conference, San Francisco, CA, USA, July 17th-22nd, 2005. [17] Sorell, V., Escalante, S., and Yang, J., Comparison of Overhead and Underfloor Air Delivery Systems in a data Center Environment using CFD Modeling, ASHRAE Trans., Vol. 111(2), pp. 756-764, 2005. [18] Zhang, X., VanGilder, J., Iyengar, M., and Schmidt, R., Effect of Rack Modeling Detail on the Numerical Results of a Data Center Test Cell, Proceeding of the 11th IEEE ITherm Conference, Orlando, FL, USA, May 27th-31st,2008. [19] Iyengar, M., Schmidt, R., Hamann, H. and VanGilder, J., Comparison between Numerical and Experimental Temperature Distributions in a Small Data Center Test Cell, Proceedings of IPACK2007, ASME InterPACK '07, Vancouver, BC, Canada, July 8-12, 2007. [20] Amemiya, Y., Iyengar, M., H.F. Herman, M. OBoyle, M. Schappert, J. Shen, and T. van Kessel, Comparison of

Experimental Temperature Results with Numerical Modeling Predictions of a Real-World Compact Data Center Facility, Proceedings of IPACK2007, ASME InterPACK '07, Vancouver, BC, Canada, July 8-12, 2007. [21] Schmidt, R., Iyengar, M., and Caricari, J., Data Center housing the Worlds 3rd Fastest Supercomputer - above floor thermal measurements compared to CFD analysis, Proceedings of the Pacific Rim/ASME International Electronic Packaging Technical Conference (InterPACK), Paper number IPACK2005-33507, Vancouver, BC, Canada, July 8th-12th, 2007. [22] Herrlin, M., and Belady, C., Gravity Assisted Air Mixing in Data Centers and How it Affects the Rack Cooling Effectiveness, Proceedings of the Intersociety Conference on Thermal Phenomena (ITherm), San Diego, CA, USA, May 30-June 2, 2006. [23] Fluent User Manual, Fluent Inc, 2005. [24] H. B. Awbi, Ventilation of Buildings, Taylor & Francis Group: Spon Press, London, UK, 2003. [25] Abdelmaksoud, W., Dang, T. Q., Khalifa, H.E., Elhadidi, B., Schmidt, R., and Iyengar, M., Experimental and Computational Study of Perforated Floor Tile in Data Centers, Proceedings of the Intersociety Conference on Thermal Phenomena (ITherm), Las Vegas, USA, 2010. [26] Schwarz, W., Integrating ASHRAE-Funded Research into Airflow Modeling Software, Seminar 25, ASHRAE Annual Meeting, June 25, 2002. [27] Chen, Q., Simplified Diffuser Boundary Conditions for Numerical Room Airflow Models, ASHRAE RP-1009 Final Report, August 2000.