1 Taylor Engineering, LLC

HVAC System Design

Mark Hydeman, P.E., FASHRAE
Taylor Engineering, LLC
mhydeman@taylor-engineering.com
2 Taylor Engineering, LLC
it takes 2,000 to 3,000 times the
volume of air to cool what you can
with water!
How do you effectively fight a fire?
With air, or with water?
3 Taylor Engineering, LLC
State of the present: with air
4 Taylor Engineering, LLC
Air system design overview
Data center layout
Airflow configurations
Distribution: overhead or underfloor
Control: constant or variable volume
Airflow issues
Economizers
Humidity control issues


5 Taylor Engineering, LLC
Server airflow front to
back or front to back and
top are recommended
2004, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org). Reprinted by permission from
ASHRAE Thermal Guidelines for Data Processing Environments. This material may not be copied nor distributed in either paper or digital form
without ASHRAEs permission.
Data center layout
Cold Aisle
Hot Aisle
6 Taylor Engineering, LLC
Data center layout
2004, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org). Reprinted by permission from
ASHRAE Thermal Guidelines for Data Processing Environments. This material may not be copied nor distributed in either paper or digital form
without ASHRAEs permission.
Underfloor Supply
Cold Aisle
Hot Aisle
Only 1 pressure
zone for UF!
7 Taylor Engineering, LLC
Data center layout
2004, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org). Reprinted by permission from
ASHRAE Thermal Guidelines for Data Processing Environments. This material may not be copied nor distributed in either paper or digital form
without ASHRAEs permission.
Overhead Supply
Cold Aisle
Hot Aisle
You can
incorporate VAV
on each branch
8 Taylor Engineering, LLC
Elevation at a cold aisle looking at racks
Typical temperature profile with UF supply
Too hot Too hot
Just right
Too cold
There are numerous references in ASHRAE. See for example V. Sorell et al; Comparison of
Overhead and Underfloor Air Delivery Systems in a Data Center Environment Using CFD
Modeling; ASHRAE Symposium Paper DE-05-11-5; 2005
9 Taylor Engineering, LLC
Elevation at a cold aisle looking at racks
Too warm Too warm
Just right
Typical temperature profile with OH supply
10 Taylor Engineering, LLC
Aisle capping
2004, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org). Reprinted by permission from
ASHRAE Thermal Guidelines for Data Processing Environments. This material may not be copied nor distributed in either paper or digital form
without ASHRAEs permission.
Cold Aisle Caps
End cap
Hot aisle lid
APC reprinted with permission
Cold Aisle
Hot Aisle
11 Taylor Engineering, LLC
Aisle capping
2004, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org). Reprinted by permission from
ASHRAE Thermal Guidelines for Data Processing Environments. This material may not be copied nor distributed in either paper or digital form
without ASHRAEs permission.
Cold Aisle Caps
LBNL has recently performed
research on aisle capping
Cold Aisle
Hot Aisle
12 Taylor Engineering, LLC
Overhead (OH) vs. Underfloor (UF)
Issue Overhead (OH) Supply Underfloor (UF) Supply
Capacity Limited by space and aisle velocity. Limited by free area of floor tiles.
Balancing Continuous on both outlet and branch. Usually limited to incremental changes by
diffuser type. Some tiles have balancing
dampers. Also underfloor velocities can
starve floor grilles!
Control Up to one pressure zone by branch. Only one pressure zone per floor, can
provide multiple temperature zones.
Temperature
Control
Most uniform. Commonly cold at bottom and hot at top.
First Cost Best (if you eliminate the floor). Generally worse.
Energy Cost Best. Worst.
Aisle Capping Hot or cold aisle possible. Hot or cold aisle possible.
13 Taylor Engineering, LLC
Airflow design disjoint
IT departments select servers and racks
Engineers size the fans and cooling
capacity
Whats missing
in this picture?


14 Taylor Engineering, LLC
Airflow with constant volume systems
Hot spots
Higher hot aisle
temperature
Possible equipment
failure or degradation

Servers Supply HVAC

V V
_
15 Taylor Engineering, LLC
Least hot spots
Higher air velocities
Higher fan energy
Reduced economizer
effectiveness (due to
lower return
temperatures)

Servers Supply HVAC

V V
_
Airflow with constant volume systems
16 Taylor Engineering, LLC

Note most of these observations apply to
overhead and underfloor distribution
With constant volume fans on the servers
you can only be right at one condition of
server loading!
The solution is to employ variable speed
server and distribution fans
Airflow with constant volume systems
17 Taylor Engineering, LLC
Partial flow condition
Best energy performance
but difficult to control

Servers Supply HVAC

V V
_
Airflow with variable volume systems
18 Taylor Engineering, LLC
How Do You Balance Airflow?
Spreadsheet
CFD
Monitoring/Site
Measurements

Image from TileFlow
http://www.inres.com/Products/TileFlow/tileflow.html,
Used with permission from Innovative Research, Inc.
19 Taylor Engineering, LLC
Thermal report
From ASHRAEs Thermal Guidelines for Data Processing Environments
20 Taylor Engineering, LLC
Whats the server airflow?
SUN SUN DELL DELL
V490 V240 2850 6850
num fans 9 3 n/a n/a
total CFM (max) 150 55.65 42 185
total CFM (min) 27 126
fan speed single speed variable 2 speed 2 speed
fan control n/a inlet temp. 77F inlet 77F inlet
Form Factor (in U's) 5 2 2 4
heat min config (btuh) 798 454
heat max config (btuh) 5,459 1,639 2,222 4,236
heat max (watts) 1,599 480 651 1,241
dT min config - 13 - 3
dT max config 33 27 48 21
servers per rack 8 21 21 10
CFM/rack (hi inlet temp) 1,200 1,169 882 1,850
CFM/rack (low inlet temp) 1,200 567 1,260
max load / rack (kW) 13 10 14 12
21 Taylor Engineering, LLC
Best air delivery practices
Arrange racks in hot aisle/cold aisle configuration
Try to match or exceed server airflow by aisle
Get thermal report data from IT if possible
Plan for worst case
Get variable speed or two speed fans on servers if possible
Provide variable airflow fans for AC unit supply
Also consider using air handlers rather than CRACs for improved
performance (to be elaborated on later)
Use overhead supply where possible
Provide aisle capping (preferably cold aisles, refer to LBNL
presentation for more details)
Plug floor leaks and provide blank off plates in racks
Draw return from as high as possible
Use CFD to inform design and operation

22 Taylor Engineering, LLC
Air-side economizer
23 Taylor Engineering, LLC
Air-Side Economizer issues
Hygroscopic dust
LBNL is doing some research on this
Design humidity conditions
See following slides
24 Taylor Engineering, LLC
Design conditions at the zone
2005, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org). Reprinted by permission from
ASHRAE Design Considerations for Data and Communications Equipment Centers. This material may not be copied nor distributed i n either
paper or digital form without ASHRAEs permission.
25 Taylor Engineering, LLC
San Francisco
BAROMETRIC PRESSURE: 29.904 in. HG
PSYCHROMETRIC
CHART
Normal Temperature
I-P Units
16 FEET
Weather Hours
360 to 321
320 to 281
280 to 241
240 to 201
200 to 161
160 to 121
120 to 81
80 to 41
40 to 1
30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
DRY BULB TEMPERATURE - F
.001
.002
.003
.004
.005
.006
.007
.008
.009
.010
.011
.012
.013
.014
.015
.016
.017
.018
.019
.020
1
0
1
5
2
0
2
5
3
0
3
5
4
0
Chart by: HANDS DOWN SOFTWARE, www.handsdownsoftware.com
15%
2
5
%
10% RELATIVE HUMIDITY
20%
30%
40%
50%
6
0
%
7
0
%
8
0
%
9
0
%
30
3
5
35
4
0
40
4
5
45
5
0
50
5
5
55
6
0
60
6
5
65
7
0
70
7
5
W
E
T
B
U
L
B
T
E
M
P
E
R
A
T
U
R
E
- F
75
8
0
1
3
.
0
1
4
.
0
V
O
L
U
M
E
-

C
U
.
F
T
.
P
E
R

L
B
.

D
R
Y
A
I
R
H
U
M
I
D
I
T
Y

R
A
T
I
O

-

P
O
U
N
D
S

M
O
I
S
T
U
R
E

P
E
R

P
O
U
N
D

D
R
Y

A
I
R
Class1;Allow
NEBS;Recommend
Class1;Recommend
San Francisco Climate Data Bins
with Data Center Guideline Zones
BAROMETRIC PRESSURE: 29.904 in. HG
PSYCHROMETRIC
CHART
Normal Temperature
I-P Units
16 FEET
Chart by: HANDS DOWN SOFTWARE, www.handsdownsoftware.com
X
Design Target
Upper Allowed Humidity Limit
Lower Allowed Humidity Limit (20%RH)
Negligible time of possible
concern for humidification
26 Taylor Engineering, LLC
Los Angeles
BAROMETRIC PRESSURE: 29.808 in. HG
PSYCHROMETRIC
CHART
Normal Temperature
I-P Units
105 FEET
Weather Hours
396 to 353
352 to 309
308 to 265
264 to 221
220 to 177
176 to 133
132 to 89
88 to 45
44 to 1
30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
DRY BULB TEMPERATURE - F
.001
.002
.003
.004
.005
.006
.007
.008
.009
.010
.011
.012
.013
.014
.015
.016
.017
.018
.019
.020
1
0
1
5
2
0
2
5
3
0
3
5
4
0
Chart by: HANDS DOWN SOFTWARE, www.handsdownsoftware.com
15%
2
5
%
10% RELATIVE HUMIDITY
20%
30%
40%
50%
6
0
%
7
0
%
8
0
%
9
0
%
30
3
5
35
4
0
40
4
5
45
5
0
50
5
5
55
6
0
60
6
5
65 7
0
70
7
5
W
E
T
B
U
L
B
T
E
M
P
E
R
A
T
U
R
E
- F
75
8
0
1
3
.
0
1
4
.
0
V
O
L
U
M
E
-

C
U
.
F
T
.
P
E
R

L
B
.

D
R
Y
A
I
R
H
U
M
I
D
I
T
Y

R
A
T
I
O

-

P
O
U
N
D
S

M
O
I
S
T
U
R
E

P
E
R

P
O
U
N
D

D
R
Y

A
I
R
Class1;Allow
NEBS;Recommend
Class1;Recommend
Los Angeles Climate Data Bins
with Data Center Guideline Zones
BAROMETRIC PRESSURE: 29.808 in. HG
PSYCHROMETRIC
CHART
Normal Temperature
I-P Units
105 FEET
Chart by: HANDS DOWN SOFTWARE, www.handsdownsoftware.com
X
Design Target
Upper Allowed Humidity Limit
Lower Allowed Humidity Limit (20%RH)
Only a few hours of possible
concern for humidification
27 Taylor Engineering, LLC
BAROMETRIC PRESSURE: 29.893 in. HG
PSYCHROMETRIC
CHART
Normal Temperature
I-P Units
26 FEET
Weather Hours
270 to 241
240 to 211
210 to 181
180 to 151
150 to 121
120 to 91
90 to 61
60 to 31
30 to 1
30 35 40 45 50 55 60 65 70 75 80 85 90 95 100
DRY BULB TEMPERATURE - F
.001
.002
.003
.004
.005
.006
.007
.008
.009
.010
.011
.012
.013
.014
.015
.016
.017
.018
.019
.020
1
0
1
5
2
0
2
5
3
0
3
5
4
0
Chart by: HANDS DOWN SOFTWARE, www.handsdownsoftware.com
15%
2
5
%
10% RELATIVE HUMIDITY
20%
30%
40%
50%
6
0
%
7
0
%
8
0
%
9
0
%
3
0
3
5
35
4
0
40
4
5
45
5
0
50
5
5
55
6
0
60
6
5
65 7
0
70
7
5
W
E
T
B
U
L
B
T
E
M
P
E
R
A
T
U
R
E
- F
75
8
0
1
3
.
0
1
4
.
0
V
O
L
U
M
E
-

C
U
.
F
T
.
P
E
R

L
B
.

D
R
Y
A
I
R
H
U
M
I
D
I
T
Y

R
A
T
I
O

-

P
O
U
N
D
S

M
O
I
S
T
U
R
E

P
E
R

P
O
U
N
D

D
R
Y

A
I
R
Class1;Allow
NEBS;Recommend
Class1;Recommend
Sacramento Climate Data Bins
with Data Center Guideline Zones
BAROMETRIC PRESSURE: 29.893 in. HG
PSYCHROMETRIC
CHART
Normal Temperature
I-P Units
26 FEET
Chart by: HANDS DOWN SOFTWARE, www.handsdownsoftware.com
Sacramento
X
Design Target
Upper Allowed Humidity Limit
Lower Allowed Humidity Limit (20%RH)
Negligible time of possible
concern for humidification
28 Taylor Engineering, LLC
Lower humidity limit
Mitigate electrostatic discharge (ESD)
Recommended procedures
Personnel grounding
Cable grounding
Recommended equipment
Grounding wrist straps on racks
Grounded plate for cables
Grounded flooring
Servers rated for ESD resistance
Industry practices
Telecom industry has no lower limit
The Electrostatic Discharge Association has removed humidity control as a primary
ESD control measure in their ESD/ANSI S20.20 standard
Humidity controls are a point of failure and are hard to maintain
Many data centers operate without humidification
This needs more research
And for some physical media (tape storage, printing and bursting)
Old technology not found in most data centers
It is best to segregate these items rather than humidify the entire data center

29 Taylor Engineering, LLC
ESD control: floor grounding
Image from Panduit, reprinted with permission
Water-Side Economizer
Integrated
Heat
Exchanger in
series with
chillers on
CHW side
31 Taylor Engineering, LLC
Economizer Summary
Air-Side Economizers
Provides free cooling when
dry-bulb temperatures are
below 78F-80F.
May increase particulates
(LBNL research indicates this
is of little concern).
Should be integrated to be
most effective.
Improves plant redundancy!
Can work in conjunction with
water-side economizers on
data centers!
Need to incorporate relief.
Water-Side Economizers
Provides low energy cooling
when wet-bulb temperatures
are below 55F-60F.
Avoids increased particulates
(and low humidity if that
concerns you).
Should be integrated to be
most effective (see previous
slide).
Improves plant redundancy!
Can work in conjunction with
air-side economizers on data
centers!


Both are proven technologies on data centers!
32 Taylor Engineering, LLC
A case study of two designs
Collocation facility in the
Bay Area
Side by side designs in
same facility over two
phases
Motivation for the second
design was to reduce
cost
Case study was
developed by Lawrence
Berkeley National
Laboratory (LBNL)
Data Centers 8.1 and 8.2
Both sections at ~30%
build-out during
monitoring

33 Taylor Engineering, LLC
A tale of two designs: overview
Phase 1 Data Center (8.1)
26,200 ft2
27 W/ft2 design
Traditional under-floor design
with CRAC units
Air-cooled DX
Humidity controls (45%-55%)
Phase 2 Data Center (8.2)
73,000 ft2
50 W/ft2 design
Under-floor supply from
central AHUs with CHW coils
Water-cooled plant
Air-side economizers
No humidity controls
34 Taylor Engineering, LLC
A tale of two designs: a closer look
Normalized efficiency metric:
servers
systems cooling
cooling
kW
kW
_

Data normalized to computer loads
-
0.20
0.40
0.60
0.80
1.00
1.20
1.40
1.60
N
o
r
m
a
l
i
z
e
d

e
n
e
r
g
y
Computer Loads UPS Losses HVAC Lighting
~1/4 of the
normalized energy
Phase 1 Data Center (8.1)
Phase 2 Data Center (8.2)
35 Taylor Engineering, LLC
A tale of two designs: results
Phase 1 Data Center (8.1)
Around 2x the HVAC
installed cost ($/ft2)
Around 4x the energy bills
(when normalized to server
load)
Acoustical problems
Higher maintenance costs
Lost floor space in data
center due to CRACs
Phase 2 Data Center (8.2)
Preferred by the facility
operators and data center
personnel
36 Taylor Engineering, LLC
Two data centers: summary
What made the difference?
Airside economizers
No humidity controls
Water-cooled chilled water system
AHUs instead of CRAC units
37 Taylor Engineering, LLC
Custom CRAH Unit (Large)
38 Taylor Engineering, LLC
Example CRAH Unit Comparison
Option 1
Model Std CRAC Custom Model 1 Custom Model 2
Budget Cost 16,235 $ 23,000 $ 41,000 $
Number of units 21 13 4
net total cooling (btuh) 434,900 410,000 841,000
net sensible (btuh) 397,400 399,000 818,000
sensible (tons) 33.1 33.3 68.2
CFM 16,500 25,000 50,000
SAT 49.90 59.30 59.00
airside dT 25.10 15.70 16.00
Internal SP 2 0.8 0.8
1.8 1.8
no. fans 3 3 2
fan type Centrifugal Plenum Plenum
no. motors 1 3 2
HP/motor 15 5 15
total HP 15 15 30
BHP/motor 15 4.7 11.5
Unit BHP 15 14.1 23
unit width 122 122 122
depth 35 36 72
height 76 156 168
filter type ASHRAE 20% MERV 13 MERV 13
Water PD (ft) 13.5 ft 11.1 11.1
CHW dT 14F 20 20
GPM 66.80 44.00 88.00
Total GPM 1,403 924 66%
Total BHP 315 275 87%
Option 2
39 Taylor Engineering, LLC
34% less water flow
13% less fan energy
More if you consider the supply air temperature and airflow issues
Excess fan capacity on new units
36% higher cost for units, but
Fewer piping connections
Fewer electrical connections
Fewer control panels
No need for control gateway
Can use the existing distribution piping and pumps (case study)
Can use high quality sensors and place them where they make sense
Possibly less turbulence at discharge?


Example CRAH Unit Comparison
40 Taylor Engineering, LLC
Air cooling issues
Limitations on the data densities served (~200w/sf)
Air delivery limitations
Real estate
Working conditions
Hot aisles are approaching OSHA limits
Costly infrastructure
High energy costs
Management over time
Reliability
Loss of power recovery
Particulates
41 Taylor Engineering, LLC
Take Aways
Use air- or water-side economizers where possible
Consider personal grounding in lieu of humidification
Consider AHUs as an alternative to CRACs
Consider VSDs on fans, pumps, chillers and towers
Refer to ASHRAE, LBNL and Uptime Institute for more
recommendations

42 Taylor Engineering, LLC
State of the future: cooling with liquid