Beruflich Dokumente
Kultur Dokumente
Guy Wagner
gwagner@ecooling.com
Electronic Cooling Solutions, Inc.
2915 Copper Road
Santa Clara, California, 95051
USA
(970) 481-8872
www.ecooling.com
1
Objective of the Study
This study compares three different liquid cooling technologies to determine which
of the three methods is able to cool the highest power density processor chips.
The first method consists of pumping a liquid through a cold plate provided by
CoolIT Systems Inc mounted over a 25.4 mm square heat source. The second
method is two-phase immersion cooling of the 25.4 mm square heat source in a
bath of 3M Novec 649 liquid with a boiling point of 49C. The third method of
cooling consists of single-phase immersion cooling of the heat source using mineral
oil as the coolant under both natural convection and forced convection cooling
conditions.
Based on the results from testing the constructed setups, the CoolIT Systems cold
plate solution was able to achieve the highest power dissipation and had the lowest
thermal resistance. The 2-phase immersion cooling using Novec 649 performed
second best and the single-phase mineral oil immersion solution had the lowest
performance.
This study will present experimental results of the three liquid cooling methods
investigated along with the design details of the heat source and the experimental
setup used to determine the cooling limits of each of the configurations.
2
Executive Summary
Three different solutions are evaluated based on their cooling performance for a high power
heat source: 1- CoolIT cold plate, 2- Two-Phase immersion with 3M-Novec 649, 3- Single-phase mineral
oil immersion
An experimental setup, consisting of a heater module with attachment to the cooling solution has been
designed and built. It was determined which power dissipation can be supported in a 2.54 cm x 2.54 cm
area for each of the three cooling solutions. Note that uniform power dissipation is assumed over the die.
The measured temperature is assumed to be equivalent to the temperature of a die with the same surface
area. The maximum die temperature is set to 100 C.
The maximum dissipation achieved and optimal thermal resistances for the three solutions are shown below:
- CoolIT cold plate 1,480 watts Thermal resistance: 0.048 K/W
- 2-Phase immersion 740 watts Thermal resistance: 0.056 K/W
- Single-phase mineral oil immersion 450 watts Thermal resistance: 0.140 K/W
Based on the results from testing the constructed setups, it can be seen that the CoolIT cold plate solution
was able to achieve the highest power dissipation and had the lowest thermal resistance. The 2-phase
immersion solution performed second best and the single-phase oil immersion solution had the lowest
ranking performance.
Each cooling technology has advantages and disadvantages. In some instances power for pumps was
required to circulate fluid. Each of the three setups used facility water to remove heat from the system.
3
Heater Module
This surface represents the surface of the die.
The three cooling solutions evaluated were attached to this surface
4
Common Heater Base
Fill material not shown
(filled with EasyFlo 60 resin)
Copper block
representing the die
Ceramic Heater
5
Ceramic Heater Limitations
From a conversation with Component General the following limitations for the ceramic heaters are
set:
1. The formula P = V2/R indicates the heater can produce power that far exceeds the desired
testing range of 1500 W (V = volts, P = power in watts, and R = electrical resistance in ohms).
For a 6 Ohm heater, the voltage needed to reach 1,500 Watts is 95 Volts, well below the maximum
voltage capacity of the heater (750 V). So the primary limitation of the heater for this work is
temperature.
2. The maximum heater temperature is 150 C. The heater temperature should be kept below
this maximum rating. If the temperature of the heater exceeds 150 C the power is de-rated
and the heater can potentially fail.
6
Thermal Resistances Heater Module
At 1,500 watts, the temperature drop over the heater module (show below):
- 3 C: 0.5 mm copper between the thermocouple and the TIM
- 17 C: TC-HM03 TIM
Temperature difference between the thermal solution and thermocouple location: 20 C
TIM: TC-HM03*
Thermocouple Copper Block
Solder
Heater
7
Electrical Circuit Diagram for 6-Ohm Heater
6-Ohm
Heater
Power Supply 3
0.05 Ohm Shunt Voltage
V_max = 80 V
Resistor Difference i_max = 13 A
8
Common Heater Base - Cross Section
Spring and washer
5 mm Copper Block with TC assembly applies 30 PSI
hole drilled from side to the surface of the
Heat Sink/heat spreader
copper block
Top Plate
Thermocouple
Ceramic Heater
9
CoolIT Cold Plate
10
CoolIT Systems Liquid Cooling
The CoolIT solution consists of a high-performance hardware configuration that
includes a micro-channel cold plate and pump
Liquid
Liquid/Liquid Pump Rotameter Flow
heat exchanger Meter
Thermocouple Coolant_in
Insulated heat source
Thermocouple Heater_Block
Thermocouple Coolant_out
11
CoolIT Cold Plate Test Set up
Copper Block
and Heater
Pump Supply
Flow Meter
Liquid/Liquid Heat Exchanger
12
Measured Temperatures vs Power
Measured Temperatures
100
90 Copper Block
80 Water In In
Coolant 1,440 watts at 97.5 C
Water Out
Coolant Out or 1,480 watts at 100
Temperature (C)
C by extrapolation
70
60
Coolant flow: 1.39 LPM
50
40
30
20
0 200 400 600 800 1000 1200 1400 1600
Power (W)
The heater temperature as well as of the coolant temperature are linear over the 0 to 1,440 watt power
range.
- The coolant inlet temperature slightly increases at increased power dissipation
The next slide presents measured temperatures relative to the coolant inlet temperature.
13
Relative Measured Temperatures vs Power
Measured Temperatures Relative to Coolant Inlet Temperature
100
90
80
70
Copper Block Relative
Temperature (C)
60 Water Out
Coolant Relative
Out Relative
50
40
Coolant flow: 1.39 LPM
30
20
10
0
0 200 400 600 800 1000 1200 1400 1600
Power (W)
The coolant temperature at the outlet increased by 15 C for a 1,400 watt power dissipation.
The dashed line represents the maximum temperature increase for the copper block, assuming
that the coolant inlet temperature is 25 C.
For a 75 C temperature rise of the copper block 1,500 watts can be dissipated.
Note that the coolant temperature at the outlet side increases to 17 C.
14
CoolIT - Thermal Resistance
The graph below shows the thermal resistance as a function of the coolant flow rate at different power
levels. Thermal resistance for the 2.54 cm x 2.54 cm copper block is calculated by:
[T_copper block - T_water in]/Power
0.060
0.050
Resistance (K/W)
0.040
0.030 300 Watts Sept 25, 3:00pm - liquid-liquid Exchanger (graphite TIM)
800 Watts Sept 25, 4:00pm - liquid-liquid Exchanger (graphite TIM)
0.020
0.010
0.000
0.00 0.20 0.40 0.60 0.80 1.00 1.20 1.40 1.60
Flow Rate (LPM)
The thermal resistances are 0.048 and 0.050 K/W, with a slight dependence on the actual power dissipation.
Fortunately, lower thermal resistance is found at higher power dissipations.
15
CoolIT - Thermal Resistance
The optimal thermal resistance (as shown in the previous slide) of the CoolIT micro-channel cold plate
solution occurs at higher flow rates. At 1.39 LPM the resistance was:
The best case resistance measured using the CoolIT solution was 0.048 K/W
(T_coolant in is the temperature of the coolant flowing into the cold plate)
16
Two-Phase Immersion Cooling
Pin-Fin Heat Spreader with Boiling Enhancement Coating
17
Two-Phase Immersion Cooling
The solution consists of the heater module plus a heat sink with boiling enhancement coating
(3M L-20227). The heater module + heat sink are submerged in Novec 649. A vapor
condenser made of a copper tube and flushed with cold water is positioned 3 above the
liquid level.
Condenser
Coil
Fluid Tank
18
Two-Phase Immersion Fluid Selection
The fluid selection for the 2-phase immersion cooling was based on a comparison of the normal
boiling point (at atmospheric pressure), the latent heat, the Global Warming Potential and the di-
electric constant.
Fluorinert Perfluorocarbon
3M Tradename(PFC)
and Chemistry Fluorinert
NovecPerfluorocarbon
Hydrofluoroether(PFC)
(HFE) Novec
Novec Fluoroketone
Hydrofluoroether
(FK)(HFE)
84 Product
FC-72 Name FC-84 Novec
FC-3284 7000 Novec
FC-72 7100 FC-84
Novec 7200 Novec 7000 Novec 649 Novec 7100 Novec 774 Novec 7200
O Molecular
C6F14 Formula C7F16 CC3F57FOCH
11NO 3 C
C64FF14
9OCH3 C7FC 164F9OC2H5 C3F7C OCH
2F5C(O)CF(CF
3 3)2C4F9COCH
3F7C(O)CF(CF
3 3)2C4F9OC2H5
Molecular
338 Weight [g/mol]
388 200
299 338250 388 264 200 316 250 366 264
several
Number Isomers several 11 several 2 several 2 1 1 2 2 2
Normal
56 Boiling Point [C]
80 34
49 56 61 80 76 34 49 61 74 76
Critical
176 Temperature 202 [C] 165.0
161 176195.2 202 209.8 165.0 169 195.2 195 209.8
Critical
1.83 Pressure [MPa] 1.74 2.48
1.87 1.83 2.23 1.74 2.01 2.48 1.87 2.23 1.71 2.01
Vapor
30 Pressure at 25C 11 [kPa] 65
35 30 27 11 16 65 40 27 16 16
Surface
10.0 Tension [dynes/cm]
12 12.4
13.0 10.0 13.6 12 13.6 12.4 11.4 13.6 12.3 13.6
3
Liquid
1680 Density [kg/m1730] 1401
1610 16801510 1730 1420 1401 1610 1510 1.67 1420
Specific
1100 Heat [J/kg-K] 1100 1300
1100 11001180 1100 1220 1300 1103 1180 1130 1220
2 Thermal
0.057 Conductivity0.06[W/m-K] 0.075
0.062 0.057
0.069 0.06 0.068 0.075 0.059 0.069 0.06 0.068
Latent
88 Heat [kJ/kg] 90 142
105 88112 90 119 142 88 112 90 119
Solubility
10 H2O in Fluid11[ppmw] 60
14 10 95 11 92 60 20 95 1.8 92
Dielectric
1.8 Constant at1.81kHz 7.4
1.9 1.87.4 1.8 7.3 7.4 1.8 7.4 1.8 7.3
0 Global >9000 1
>9000 Warming Potential >9000
370 >9000 280 >9000 55 370 1 280 1 55
8-Hour
NA Exposure Limit NA[ppmv] 75
NA NA750 NA 200 75 150 750 ND (~150) 200
Acute
NA Lethal Conc. [ppmv]
NA >30,000
NA >100,000
NA NA >92,000 >30,000>100,000 >100,000<100,000 >92,000
Cardiac
N Sensitizer N NN N N N N N N N N N
rtness, low tox,
Advantage
compat with alumina high
extreme
latent heat,
inertness,
low GWP
low tox, compat with alumina high latent
lowest
heat,GWP,
low GWP
PFC-like dielectric
low latent heat
Disadvantage higher
highhydrocarbon
GWP, low latent solvency,
heat reduced dielectric props higher hydrocarbon
incompatible solvency,
with liquid
reduced
water dielectric props
After visiting Phil Tuma at 3M in St. Paul and inspecting the setup with Novec 649 at 3Ms facility,
it was determined that Novec 649 would be used in the 2-phase immersion experiments.
19
Two-Phase Immersion - Operation
During operation, the Novec 649 fluid boils at 49 C and the vapor bubbles rise to the
liquid/air surface.
The Novec 649 vapor will settle on top of the liquid because the Novec 649 vapor has a
density that is higher than air. The Novec 649 vapor condenses at the cold surface of the
condenser. There the heat is removed by the cooling water.
Shown below is a view of the vapor bubbles coming off of the heat sink surface.
20
Two-Phase Immersion - Operation
Shown below is a view of the solution operation.
The whirlpool with bubbles disturbs the fluid surface and creates movement of the entire fluid
volume.
The vapor condenses at the condenser surface and drips down into the tank, where the cycle is
repeated.
21
Two-Phase Immersion Cooling
Shown below is a view of the violent movement of the fluid volume due to the creation of
vapor bubbles. These vapor bubbles escape from the fluid volume and condense on the
copper coil filled with cold running water.
22
Two-Phase Immersion - Temperature vs Power
2-Phase Immersion Cooling
110
Copper Block Temperature (C)
100
60
50
100 200 300 400 500 600 700 800
Power (W)
The response of the copper block temperature as a function of power shows:
- A near linear response over 0 - 750 watt range
- The vertical orientation results in slightly higher temperatures than the horizontal orientation, but the
difference is within 5 C. It is suggested that the supply of liquid to the boiling enhanced surfaces of the
heat sink is better for the horizontal orientation.
23
Two-Phase Immersion Vertical Operation
Shown below is a view of the setup used to investigate the performance of the 2-phase
immersion cooling solution with the heat sink in a vertical orientation.
24
Two-Phase Immersion Vertical Operation
Shown below is a view of the bubbles created on the heat sink surfaces. The bubbles travel
vertically, escaping through the liquid-air interface. In the background the condensed liquid
drips down from the condenser.
25
Two-Phase Immersion Temperature Response
The response of the copper block temperature to a change in the power dissipation is shown below.
88
Overshoot
550 Watts
Copper Block Temperature (C)
86
84
82
Vertical
80
78 Horizontal
450 Watts
76
74
72
70
0 20 40 60 80 100 120 140 160 180
Time (s)
It is known from literature that a temperature overshoot may occur with 2-phase cooling during start-
up/power changes. The results show that there is no temperature overshoot in the horizontal orientation.
However, a small overshoot (< 2 C) is observed for the vertical orientation.
The temperature overshoot is reduced by the boiling enhancement coating.
26
Two-Phase Immersion Resistance vs Power
The graph shows the thermal resistance as a function of power in both vertical and horizontal orientations.
Thermal resistance for the 2.54 cm x 2.54 cm copper block is calculated by:
[ T_copper block T_fluid bath] / Power
0.070
0.060
Resistance (K/W)
0.050
0.020
0.010
0.000
0 100 200 300 400 500 600 700 800 900
Power (W)
The optimal thermal resistances for this solution are 0.056 and 0.067 K/W for horizontal and vertical
orientations, respectively. The form of the curve indicates that as boiling speeds up resistance declines.
Then, as surface contact between the fluid and the heat sink begins to decrease due to the increase in vapor
pockets, the thermal resistance increases.
27
Two-Phase Immersion Thermal Resistance
The optimal thermal resistance (as shown on the previous slide) for the 2-Phase
immersion solution occurred at a power dissipation of 300 watts. At this power level,
the thermal resistance was:
0.056 K/W in the horizontal orientation and 0.067 K/W in the vertical orientation
The best case resistance for the 2-Phase solution was 0.056 K/W
28
Single-Phase Oil Immersion Cooling
A sketch of the setup used to investigate the performance of the single-phase oil
immersion solution is shown below:
Tank
Flow
Liquid-Liquid heat exchanger
Oil bath
Temperature
Mineral Oil_in
Heater Module
The coolant pump delivers the mineral oil to a nozzle manifold, which creates impingement
jets on a pin fin heat sink.
29
Single-Phase Oil Immersion Cooling
Impingement
Oil Supply Line connected
Temperature to the top of the manifold
Mineral Oil_in
Nozzle Manifold
30
Single-Phase Oil Immersion, Bottom View
Impingement
Nozzle Manifold
31
Single-Phase Oil Immersion Cooling
Impingement
32
Single-Phase Oil Immersion Cooling
Natural Convection - Horizontal Setup
The setup with the heat sink positioned horizontally for the oil immersion test is shown below.
The experiment shows a stratification in the oil. The top layer has a higher temperature than
the lower bulk.
Fluid from
liquid liquid
exchanger
Fluid
Fluid to level
liquid liquid
exchanger
33
Single-Phase Oil Immersion Cooling
Natural Convection - Vertical Setup
The setup with the heat sink positioned vertically for the oil immersion test is shown below.
During the experiment a stratification of the oil was observed. At a power dissipation of
160 watts the temperature at the top was 45 C higher than the temperature at the
bottom.
Fluid from
liquid liquid Fluid to Fluid
exchanger liquid liquid level
exchanger
Visible
stratification
34
Single-Phase Oil Immersion Cooling
Impingement Setup
The setup with the heat sink and the nozzle manifold installed on the heat sink is shown below.
Nozzle
manifold
fluid line
35
Single-Phase Oil Immersion Results
Natural Convection
Test results show that the mineral oil temperature increases with power dissipation. The typical
oil bath temperature was 40 C. Allowing a 60 C temperature rise of the copper block, the
maximum power dissipation is 140 watts for the horizontal orientation and 160 watts for the
vertical orientation. Relative temperatures for horizontal and vertical orientations are plotted
below.
Single Phase Immersion with 40 oC Mineral Oil
Relative Temperatures
80.0
T_copper block - T_fluid bath (C)
60.0
50.0
Horizontal Test
40.0
160 watts
10.0
0.0
0 50 100 150
Power (W)
36
Single-Phase Oil Immersion Results
Forced Convection
Forced convection measurement results show that power dissipations up to 400 watts can
be supported with a 10.6 mm distance between the nozzle manifold and the heat sink and
power dissipations up to 450 watts can be supported with a 2.6 mm distance.
80 Horizontal with
70 Impingement (10.6 mm
60 gap)
50
40 Horizontal with
30 Impingement (2.6 mm gap)
20
0 100 200 300 400 500 600
Power (W)
37
Single-Phase Oil Immersion Resistance vs Power
The graph shows the thermal resistance as a function of power in four configurations: horizontal and vertical
with natural convection and horizontal with impingement at 10.6 and 2.6 mm distances. The thermal
resistance for the 2.54 cm x 2.54 cm copper block is calculated as follows:
[ T_copper block T_oil] / Power
0.60
Horizontal Test
Resistance (K/W)
0.50
Vertical Test
0.40 Horizontal with Impingement (10.6 mm gap)
Horizontal with Impingement (2.6 mm gap)
0.30
0.20
0.10
0.00
0 100 200 300 400 500 600
Power (W)
Optimal thermal resistances are 0.14 K/W and 0.16 K/W for setups with impingement and 0.37 K/W and
0.41 K/W for setups with natural convection. Lower resistances are found at higher power dissipations.
Note: For tests with impingement T_oil is the impinging fluid temperature, and for natural convection tests
T_oil is the fluid bath temperature.
38
Single-Phase Oil Immersion Thermal Resistance
For the natural convection case, the optimal thermal resistance of the single phase oil
immersion solution occurred at higher power dissipations. The thermal resistance of this
solution was:
0.37 K/W at 160 Watts in the vertical orientation and
0.41 K/W at 160 Watts in the horizontal orientation
The optimal thermal resistance of the natural convection case was 0.37 K/W
Thermal resistance for this case was calculated as
[T_copper block - T_fluid bath]/Power
For the forced convection case, the optimal thermal resistance of the single phase oil
immersion solution occurred again at higher power dissipations. The thermal resistance of
this case was:
0.14 K/W at 550 Watts with an impingement nozzle 2.6 mm away from the heat sink and
0.16 K/W at 430 Watts with an impingement nozzle 10.6 mm away from the heat sink
The optimal thermal resistance of the forced convection case was 0.14 K/W
Thermal resistance for this case was calculated as:
[T_copper block - T_impinging fluid]/Power
39
Summary
An experimental setup, consisting of a heater module attached to three cooling solutions was
designed and built. The power that can be dissipated from a 2.54 cm x 2.54 cm area heat
source for each of the solutions was determined. It was assumed that uniform power was
dissipated over the area of the heat source. The heat source mimics a die with the same
surface area that operates at a maximum temperature of 100 C.
Tests performed on the three setups revealed that:
- The CoolIT Cold plate performed best. A 1,480 watt component can be cooled.
- The 2-phase immersion using 3M Novec 649 and single phase impingement cooling using
mineral oil result in lower power dissipation capabilities: 740 watts and 450 watts for the 2-
phase Novec and the oil immersion solutions, respectively.
It should be noted that there are costs associated with the operation of each of these cooling
solutions. Below are some considerations:
- CoolIT Cold Plate: Power for the circulating pump
- Novec 649 2-Phase
- Sealing of tank for Novec 649
- Maintenance of fluid level
- Cost of Novec 649: $275/Gallon
- Mineral oil impingement cooling requires:
- 2 pumps (one for impingement and another one to remove heat from the bulk fluid).
- Requires a complete infrastructure to deal with the mineral oil to remove the oil from all
parts. In short, mineral oil is messy.
- All three cooling solutions used facility water to remove heat from the system
40
Summary Resistance Comparison
The chart below shows a comparison of the best case performance for the three technologies
41
Conclusions
A comparison of the three cooling solutions below shows that the cold plate design from CoolIT has better
cooling performance than the two other investigated cooling solutions, oil immersion single phase and
Novec 649 passive 2-phase cooling. The cooling solutions were attached to a 2.54 cm x 2.54 cm heater
block.
The maximum dissipation achieved for the three cooling solutions is shown below:
In short, the tests conducted show that the CoolIT cold plate solution had the capacity to handle higher
power densities than the other two cooling solution setups that were evaluated.
For the cold plate solution, power was required for a pump to circulate the liquid. The single phase
immersion solution required power for two pumps. The passive 2-phase solution required no pumping
power. All three solutions used facility water to remove heat from the system.
2-Phase cooling with Novec 649 can provide good performance without a pump. Negatives are the costs of
the liquid and the requirement to properly seal the tank to prevent the Novec 649 from either evaporating or
leaking out of the tank.
The single-phase mineral oil immersion solution should only be considered for power dissipation
applications below 450 watts.
42
9- Acknowledgements
Many thanks to
- Justin Dixon for building the measurement setups, performing the actual
thermal tests and making a major contribution to this write up.
- David Copeland for his advise and critical view, providing the liquid-liquid heat
exchangers, and providing the 6-ohm heaters used for the experiments.
- CoolIT for the hardware they provided to build up the measurement set ups.
- Phil Tuma from 3M for his advise on use of the Novec 649, donation of a heat
sink with the boiling enhanced coating, solder preforms and design
considerations.
- Tomoko Hara from Hitachi Chemical who supplied the high performance graphite
TIM samples, TC-HM03.
43
Appendix A. Hitachi TIM Data Sheet
44
Appendix B. Ceramic Heater
Mechanical Specifications
Electrical Specifications
45
Appendix C. Pump Power
Pump Power Consumption
35
30
25
Power (W)
20
15
10
0
0.00 0.20 0.40 0.60 0.80 1.00 1.20 1.40 1.60
Flow Rate (LPM)
The pump power as a function of the flow rate is presented in the graph above.
Note that for high power, the pump will deliver the max. flow rate => max pump power is 30
watts. The flow rate can be adjusted according to the heat load. The cold plate performance
is improved at higher flow rate.
46