http://www.datacenterknowledge.

com/archives/2008/02/25/globat-switch-failurecaused-outage/

DataCenterKnowledge _ How to
Extend Battery Life
DOWNTIME

Outage Highlights Battery Maintenance

BY RICH MILLER ON AUGUST 19, 2009

http://www.datacenterknowledge.com/archives/2009/08/19/how-to-extend-batterylife/

On August 5 some customers at an Internap Network Services (INAP) data center

in Boston lost connectivity for 1 hour, 20 minutes due to a battery failure in a DC power
plant.
The DC plant was responsible for powering network equipment for both Internap and
other carriers, Internap said in a statement. In some cases this network equipment was
used by our customers for IP connectivity. The failure was caused by a utility company
power interruption that cause the DC plant to fail over to battery backup. DC plants do
not afford the ability to simulate load via a load bank as can be done on a UPS or
generator. However, in this instance, when the batteries were called upon, they did not
carry the load. The batteries had not shown any prior signs of pending failure during
normal testing and are still being investigated at this time. Since the incident, the source
of the power fed to the networking equipment was changed to UPS-backed AC power to
avoid any potential repeat of this event.
The incident underscores the importance of battery maintenance and monitoring. Mark
Fontecchio atTech Target has a story that looks at four ways that data center managers
can extend the life of their UPS batteries.

About the Author

Rich Miller (7929 Posts)

Rich Miller is the founder and editor at large of Data Center Knowledge, and has been
reporting on the data center sector since 2000. He has tracked the growing impact of
high-density computing on the power and cooling of data centers, and the resulting push
for improved energy efficiency in these facilities.

Four ways to extend data center UPS battery life

http://searchdatacenter.techtarget.com/news/1364748/Four-ways-to-extend-datacenter-UPS-battery-life?track=sy185#
by:
Mark Fontecchio
News Director
Published: 14 Aug 2009

Proper purchase, maintenance, monitoring and use of flywheels are four ways for
data center managers can keep UPS batteries juiced a bit longer.

Data center backup power is a crucial element to uptime and availability.

During an outage, Uninterruptible power supplies (UPSes) enable the switch
from utility power to backup generator power. But there is a relatively
unreliable and expensive variable to most UPSes: batteries.

For more on data center

backup power:
Flywheel UPS, not
batteries, for data center "Simply stated, batteries represent three evils: They
company's backup power are big, they are expensive, and they are
expendable," said Pete Sacco, the president of
Data center power backup Franklin Lakes, N.J.-based engineering firm PTS
isn't just about batteries
anymore

Data Center Solutions. "Anything you can do to prolong the life of your
batteries will indeed extend the availability of your data center."
So here are four ways that data center facility managers can extend the life of
their UPS batteries.
Buy the right UPS battery for your data center
In most data centers, UPSes are designed to fully discharge the battery
source within 20 minutes. This is called high-rate discharge. High-rate
batteries have thinner plates and more plates than lower-rate batteries, but as
a result high-rate batteries can lead to a shorter lifespan.
There are many other factors in buying UPS batteries, such as the life
expectancy, voltage, front or top access, flag post or inserted post, and other
considerations. For example, a small format valve regulated lead acid (VRLA)
battery has a relatively low first cost but lower reliability. Meanwhile, a nickel
cadmium (NiCd) battery has the best reliability and longest service life, but the
first cost can be four times as much as other batteries.

Data center managers must also be aware of potential manufacturing

problems such as faulty post seal design and internal connection problems.
Install, operate and maintain batteries correctly
Incorrect installation can lead to shortened battery life, as can improper
maintenance. Good maintenance means giving the batteries well ventilated air
that is as close as possible to 77 degrees Fahrenheit, and making sure that all
batteries in the string are getting air within 3 degrees Fahrenheit to ensure that
some of the batteries don't age more quickly than others.
Why? Mixing batteries of different ages or internal resistances can prompt
batteries to die more quickly. Regular inspections of the batteries can unveil

problems such as loose intercell connections and damaged post seals, which
can lead to corrosion and even fires.
Data center managers should also be aware of the battery's discharge status
as well. A fully or nearly fully discharged battery that isn't recharged within 48
hours can lead to damage. Overdischarging a battery can cause recharge
problems, and excessive discharging -- for example, if a data center does
weekly testing of its backup systems that includes battery cycling -- can
reduce battery life.
Monitor the UPS batteries' capacity
When the internal resistance of a battery rises due to normal aging factors
such as corrosion, and when it has increased by as little as 30%, it's time to
replace them. This is most often measured through capacity testing, as most
manufacturers say that when a battery drops to 80% of its original capacity, it's
cooked.
Users should perform capacity testing on their batteries based on IEEE
standards; in particular, IEEE 1188 or IEEE 450.
Consider using flywheel UPS in conjunction with battery UPS
When flywheel UPSes are installed alongside battery UPSes, the flywheels
can often handle the shorter power outages that happen more often and can
prevent the batteries from cycling.
Art Smith, the chief engineer at KEYE-TV, a CBS affiliate in Austin, said his
Active Power flywheels have helped extend battery life by two years.
"I wanted the flywheel to filter out the 'dirty power,' those little power blips and
spikes," he said. "The batteries would filter those spikes out -- I understand
that -- but if you hit the batteries with those spikes, it's bad for them. It will kill
the battery life."

Let us know what you think about the story; email Mark Fontecchio, News
Writer. Also, check out our blogs: Data Center Facilities Pro, Mainframe
Propeller Head, and Server Farming.

Have You Checked Your Data Center

Batteries?
BY RICH MILLER ON JULY 22, 2008

http://www.datacenterknowledge.com/archives/2008/07/22/have-you-checked-yourdata-center-batteries/

Batteries have emerged as significant contributors in several recent data center

outages, underscoring the need to monitor and maintain battery banks. Heres a recap:
Battery failure was cited as the root cause of a May 23 outage at
HostDime and Surpass Hosting. The diagnosis conducted revealed
there was a fault within a battery string which is connected to the UPS.
It is this fault that disabled the UPS from being able to fully sustain
continuous power to the load. While the age of the battery supply
being employed was well within the manufacturers life span
expectancy, the entire battery supply was replaced with a new set.

Battery issues were also noted in a June 21 downtime event at AdHost

in Seattle. Very early indications suggest that the underlying nature of
this failure was a significant drop in amperage from the battery strings
which might have caused, for an as yet unknown reason, the (UPS) unit
to go into an uncontrolled over-voltage situation to compensate. The
end result was significant heat and smoke damage to the UPS unit.
Advocates of flywheel (rotary UPS) technology say they are preferable to batteries,
contending that repeated brief bumps to grid power can affect the life expectancy of
the battery.

Maintenance Key to Maximizing Battery Life

BY RICH MILLER ON JULY 22, 2008

http://www.datacenterknowledge.com/archives/2008/07/22/maintenance-key-tomaximizing-battery-life/
Maintenance and monitoring are important steps in ensuring that your data center batteries
operate at peak reliability. Charles ODonnell, vice president of Monitoring and Professional
Services for the Liebert Services business of Emerson Network Power, discusses the
importance of battery maintenance and the companys remote monitoring services. This video
runs approximately 2 minutes, 45 seconds.

UPS Failure Triggered Friendster Outage

BY RICH MILLER ON NOVEMBER 17, 2008
7 COMMENTS

http://www.datacenterknowledge.com/archives/2008/11/17/ups-failure-triggeredfriendster-outage/

A catastrophic UPS failure caused a power outage Thursday at a Santa Clara data
center operated by Quality Technology Services, triggering days of performance
problems for the social network Friendster. Quality Tech said the outage occurred
during planned maintenance when the facility was switched from utility power to backup
diesel generators.
Regrettably, the maintenance did not go as planned and we suffered a catastrophic
UPS failure at 8:22 am Pacific Standard Time, said Mark Waddington, President of
Quality Technology Services, in an incident report for customers. The UPS failed to
stand in and smoothly transfer power from the utility to the temporary generators due to
a voltage regulator problem with the temporary generators. The failure resulted in the
triggering of the FM200 (fire suppression) system in the enclosed battery room and the
subsequent EPO as part of our life safety system.

FM200 is a popular fire suppressionsystem that uses a chemical clean agent rather
than water. TheEPO (Emergency Power Off) button instantly cuts power in the data
center when a situation presents a risk to worker safety or equipment.
The Santa Clara facility was back on generator power within two hours, but Friendster
remained offline for more than 23 hours over three days. While it has been eclipsed
in the U.S. by MySpace and Facebook, Friendster has seen strong growth in
international markets (particularly the Philippines) and says it has 85 million users.
When Friendster came back online, many of its users found large chunks of their friend
lists missing, triggering rumors that the site had been hacked. Friendster addressed the
issue Saturday in its customer forum.
Were aware of the problem that some users are having with missing friends, a
Friendster rep posted. We experienced a major power outage the other day, that were
still recovering from. We are actively working on resolving the problem with missing
friends. Rest assured that no friends have actually been lost even though it may
appear so on the website! The problem should be fixed within the next 24 hours.
The generator test was part of Quality Techs deployment of 22,000 square feet of new
data center space at the Santa Clara data center. On Friday the company replaced
strings of batteries while engineers thoroughly checked the UPS systems. The facility
switched back to utility power Sunday night at 7:55 pm Pacific time.
Waddington said Quality Tech had spent five months preparing for the maintenance,
and promised additional information about the incident. In the coming hours, we will
complete an exhaustive study of what failed and why and issue a formal after action
report with detailed root cause analysis, Waddington wrote.
\==================