The Fundamentals of Forensic Investigation Procedures Guidebook

Technical Report

The Fundamentals of Forensic Investigation Procedures Guidebook

1001890

Final Report, March 2003

EPRI Project Manager H. Ng

EPRI 3412 Hillview Avenue, Palo Alto, California 94304 PO Box 10412, Palo Alto, California 94303 USA 800.313.3774 650.855.2121 askepri@epri.com www.epri.com

DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITIES

THIS DOCUMENT WAS PREPARED BY THE ORGANIZATION(S) NAMED BELOW AS AN ACCOUNT OF WORK SPONSORED OR COSPONSORED BY THE ELECTRIC POWER RESEARCH INSTITUTE, INC. (EPRI). NEITHER EPRI, ANY MEMBER OF EPRI, ANY COSPONSOR, THE ORGANIZATION(S) BELOW, NOR ANY PERSON ACTING ON BEHALF OF ANY OF THEM: (A) MAKES ANY WARRANTY OR REPRESENTATION WHATSOEVER, EXPRESS OR IMPLIED, (I) WITH RESPECT TO THE USE OF ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEM DISCLOSED IN THIS DOCUMENT, INCLUDING MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, OR (II) THAT SUCH USE DOES NOT INFRINGE ON OR INTERFERE WITH PRIVATELY OWNED RIGHTS, INCLUDING ANY PARTY'S INTELLECTUAL PROPERTY, OR (III) THAT THIS DOCUMENT IS SUITABLE TO ANY PARTICULAR USER'S CIRCUMSTANCE; OR (B) ASSUMES RESPONSIBILITY FOR ANY DAMAGES OR OTHER LIABILITY WHATSOEVER (INCLUDING ANY CONSEQUENTIAL DAMAGES, EVEN IF EPRI OR ANY EPRI REPRESENTATIVE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES) RESULTING FROM YOUR SELECTION OR USE OF THIS DOCUMENT OR ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEM DISCLOSED IN THIS DOCUMENT. ORGANIZATION(S) THAT PREPARED THIS DOCUMENT FTI Consulting

ORDERING INFORMATION
Requests for copies of this report should be directed to EPRI Orders and Conferences, 1355 Willow Way, Suite 278, Concord, CA 94520, (800) 313-3774, press 2 or internally x5379, (925) 609-9169, (925) 609-1310 (fax). Electric Power Research Institute and EPRI are registered service marks of the Electric Power Research Institute, Inc. EPRI. ELECTRIFY THE WORLD is a service mark of the Electric Power Research Institute, Inc. Copyright 2003 Electric Power Research Institute, Inc. All rights reserved.

CITATIONS
This report was prepared by FTI Consulting, Inc. 909 Commerce Road Annapolis, Maryland 21401 Principal Investigators J. Reynolds H. Manger R. Samm Technical Contributors A. Giuliante S. Rittenhouse W. Thue This report describes research sponsored by EPRI. The report is a corporate document that should be cited in the literature in the following manner: The Fundamentals of Forensic Investigation Procedures Guidebook, EPRI, Palo Alto, CA: 2003. 1001890.

iii

REPORT SUMMARY

No reference or guideline for the practice of or procedure for analyzing equipment failures and accidents involving electric utility distribution systems has been available until publication of The Fundamentals of Forensic Investigation Procedures Guidebook. Background The vast majority of equipment failures and accidents involving power utility distribution systems does not result in litigation. However, the forensic engineering investigation approach provides both a systematic means to improve reliability from all incidents and a means to provide utilities the most beneficial position to handle those incidents that could result in litigation. Objectives To provide basic forensic investigation procedures targeted for electric utilities. To suggest an approach for improving incident first-response results. To provide guidelines for consideration in evaluating, documenting, and preserving the incident site and equipment involved. To enhance the root cause analysis procedure. To improve utility distribution system equipment reliability. To be prepared for litigation in some incidents. To provide instructional materials.

Approach The project team has introduced the fundamental techniques for conducting a forensic investigation. The team has emphasized a formal approach and a forensic procedure for evaluating incidents, determining when an outside expert might be required, documenting the incident, collecting evidence, and interviewing witnesses. The team also has emphasized faulttree analysis methods to enhance root cause analysis by expanding the knowledge gained from all incident investigations and to assist company personnel prepare for possible litigation. An EPRI advisory committee and experienced utility forensic engineers held a series of reviews to ensure the materials technical accuracy and usefulness.

Results The forensic guidebook provides background, procedures, and fundamental tools for conducting investigations of distribution equipment accidents and failures (or any other utility equipment failure). This guidebook can help new as well as experienced engineers conduct a credible investigation. Utilities who incorporate these procedures will benefit from improvements in reliability to more successfully preparing and dealing with claims and litigation. EPRI Perspective As utilities continue to reduce headcount due to financial pressure, the knowledge to identify the cause of a failure or accident also is being lost. Normally, it takes an individual many years to learn how to conduct an accident investigation. In todays utility environment, there is very little time or money to get new engineers up to speed. The forensic guidebook was prepared to provide members with the information necessary so that any individual who follows the process defined will conduct a defendable investigation. This guide is intended for investigations that are not likely to end up in court. But the information uncovered in the investigation can withstand legal challenge if conducted properly. Keywords Distribution system reliability Forensic investigation procedures Root cause analysis Forensic engineering Incident first response Distribution equipment failures analysis

vi

ABSTRACT

Electric power distribution systems are well integrated into the public domain of their service areas. Accidents and equipment failures are not unexpected and may or may not involve nonutility property and/or personnel. The need has been recognized to provide utilities with the resources to conduct an enhanced root cause analysis of incidents by implementing forensic engineering investigation procedures. Utility personnel will be better prepared to respond to all distribution system incidents more effectively and to investigate them more thoroughly. The results of these investigations can be used to improve reliability and safety, and, in some cases, to prepare for litigation. The Fundamentals of Forensic Investigation Procedures Guidebook provides a comprehensive resource to assist the electric distribution utility in adopting forensic methods into their existing incident response and investigation procedures and in training personnel to use them effectively. Suggested forensic investigation procedures, forms, and checklists are provided. These materials will guide utilities in conducting an enhanced root cause analysis of an incident from the first response through site documentation and evidence collection, analysis, root cause determination, recommendations, reports, and presentations.

vii

ACKNOWLEDGMENTS
FTI and EPRI would like to acknowledge the important contributions of the following individuals for their participation and active guidance for the development of this guidebook: Bill Cox, Salt River Project, offered us the original Table of Contents, which was then modified to develop this guidebook. Frank Doherty, Consolidated Edison of New York Brian Shell, MidAmerican Energy Jim Fisher, Kansas City Power and Light

ix

CONTENTS

1 INTRODUCTION ....................................................................................................................1-1 2 BACKGROUND OF THE FORENSIC PROCESS .................................................................2-1 3 THE FORENSIC ANALYSIS PROCESS APPLIED TO ELECTRIC POWER DISTRIBUTION SYSTEMS .......................................................................................................3-1 4 PREPARATION FOR SITE INVESTIGATION .......................................................................4-1 5 INCIDENT INVESTIGATION FIRST RESPONSE..................................................................5-1 6 SITE INVESTIGATION PROCEDURES.................................................................................6-1 6.1 6.2 6.3 6.4 6.5 6.6 6.7 6.8 6.9 Response and Task Prioritization ................................................................................6-1 Restoration Coordination .............................................................................................6-1 Initial Facts Assembly ..................................................................................................6-2 Visual Inspection..........................................................................................................6-2 Documentation.............................................................................................................6-2 Facility and Pertinent Equipment Diagrams.................................................................6-4 Similar Equipment Comparison ...................................................................................6-4 Initial Physical Reconstruction of Damaged Equipment ..............................................6-5 Selected Testing at Incident Site .................................................................................6-5

6.10 Evidence and Sample Collection..................................................................................6-5 7 DATA COLLECTION..............................................................................................................7-1 7.1 7.2 7.3 7.4 7.5 7.6 System Electrical Maps and Data ................................................................................7-1 Original Equipment Manufacturer (OEM) Data ............................................................7-1 Equipment History as a Class......................................................................................7-1 Relevant Equipment Operation, Maintenance, and Commissioning Records .............7-2 System and Equipment Electrical Conditions ..............................................................7-2 Site Physical Setting ....................................................................................................7-2

xi

7.7 7.8 7.9

Site and System Environmental Conditions............................................................... 7-3 Relevant Standards, Codes, and Practices ............................................................... 7-3 External Factors Beyond Utility Company Control ..................................................... 7-3

7.10 Fault Recordings ...................................................................................................... 7-4 7.11 Relay and Protective Device Operation Data............................................................ 7-4 7.12 Official Reports and Media Materials ........................................................................ 7-5 8 WITNESS INTERVIEWS...................................................................................................... 8-1 8.1 8.2 8.3 Early Identification of Individuals ............................................................................... 8-1 Standard Procedures for Conducting a Forensic Interview........................................ 8-1 Utilization of Information............................................................................................ 8-2

9 FORENSIC-ROOT CAUSE ANALYSIS............................................................................... 9-1 9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8 Fault Tree Analysis (FTA) ......................................................................................... 9-1 Evaluation and Decision Procedure........................................................................... 9-4 Witness Evidence...................................................................................................... 9-4 Physical Evidence ..................................................................................................... 9-5 Electrical Evidence.................................................................................................... 9-5 Mechanical Evidence ................................................................................................ 9-5 Other Evidence ......................................................................................................... 9-6 Post-Event Testing.................................................................................................... 9-6 On-Site Testing................................................................................................. 9-7 Laboratory Testing............................................................................................ 9-7

9.8.1 9.8.2 9.9

Timeline of Events..................................................................................................... 9-7

9.10 Final Review............................................................................................................. 9-8 9.11 Conclusions.............................................................................................................. 9-8 9.12 Recommendations as Appropriate............................................................................ 9-9 10 REPORTS AND PRESENTATIONS .................................................................................10-1 10.1 10.2 Reports................................................................................................................10-1 Presentations ......................................................................................................10-2

11 CASE STUDIES ...............................................................................................................11-1 11.1 11.2 11.3 Distribution Transformer ......................................................................................11-1 Load-Break Disconnect Switch ............................................................................11-1 Department Store Feeder Failure ........................................................................11-2

xii

11.4 11.5 11.6

Cable Failure (URD) ............................................................................................11-4 Secondary Arcing in a Trench with a Co-Located Gas Line .................................11-5 Improper Storage of Submarine Cable Samples..................................................11-6

12 EXERCISE........................................................................................................................12-1 12.1 Distribution Customer Switchgear Fault Resulting in Two Substation Breaker Operations.........................................................................................................................12-1 A. B. C. D. Summary.............................................................................................................12-1 Investigation ........................................................................................................12-2 Conclusions.........................................................................................................12-3 Report .................................................................................................................12-4

Exercise Questions (Note - Select all answers that apply): ...........................................12-4 13 CHECKLISTS ...................................................................................................................13-1 13.1 13.2 13.3 13.4 13.5 13.6 13.7 13.8 13.9 13.10 13.11 13.12 13.13 13.14 13.15 Incident First Response Report (IFRR)................................................................13-2 Forensic Investigators Field Kit ...........................................................................13-5 Acceptable Camera Kit ........................................................................................13-5 Minimum Camera Kit ...........................................................................................13-5 Optimum Camera Kit ...........................................................................................13-5 Unacceptable Reliability Report (URR)................................................................13-7 Forensic Photography Tips..................................................................................13-9 Incident Site Photography....................................................................................13-9 Evidence Identification, Removal, and Retention .................................................13-9 Power System Conditions Data .........................................................................13-10 Relevant Standards, Codes, and Procedures ....................................................13-10 Event Records, Targets, Alarms, and Other Recorded Data..............................13-10 Digital Fault Recorder (DFR) Data.....................................................................13-11 Media Material...................................................................................................13-11 Official Reports..................................................................................................13-11

14 SUMMARY .......................................................................................................................14-1 15 GLOSSARY OF SPECIALIZED TERMS ..........................................................................15-1 16 BIBLIOGRAPHY...............................................................................................................16-1

xiii

LIST OF FIGURES
Figure 4-1 Utility Response Plan Sequence ............................................................................ 4-2 Figure 6-1 Sample Evidence Tag............................................................................................ 6-7 Figure 8-1 A Substation Site Plan ........................................................................................... 8-3 Figure 9-1 The Most Common FTA Symbols .......................................................................... 9-2 Figure 9-2 FTA Example of a Circuit Breaker Failure to Open................................................. 9-3 Figure 9-3 Timeline of Events for a Substation Sequence of Failures ..................................... 9-8

xv

1
INTRODUCTION
The Fundamentals of Forensic Investigation Procedures Guidebook (Forensic Guidebook) provides a comprehensive resource to assist electric utilities in conducting an enhanced root cause analysis of incidents, by implementing forensic engineering investigation procedures. The Forensic Guidebook is intended for use in either a classroom setting, when instructed by an experienced investigator, or as a self-study document, and as a source of forms and checklists for field use. It will enable utility personnel to better prepare and respond more effectively to distribution system incidents involving accidents and equipment failures. The Forensic Guidebook introduces the methodical forensic investigation approach to determine the root cause of distribution system incidents. Results can be used to improve reliability and safety, and, in some cases, to prepare for litigation. Fundamental techniques are provided for coordination with utility repair and restoration personnel, and the utilization of external forensic investigation specialists, when appropriate. Case studies are provided to demonstrate the application of forensic investigative techniques in response to selected accidents and failures. The words reliability and liability have very different meanings, although only differentiated by two initial letters. However, experience has shown that forensic investigations involving electric power distribution systems can enhance reliability, and liability can be thoroughly evaluated so that it can be properly dealt with and effectively resolved. It is interesting to note that reliability is provided to the customer by the utility, and liability is often claimed by the customer against the utility. Thus, an appropriate application of the forensic investigative process to electric power distribution system incidents can provide definition and direction regarding these two words. The vast majority of equipment failures and accidents on the distribution system do not result in litigation. A forensic approach to all incident investigations may appear unnecessary since it is not always clear if litigation will result. However, the opportunity to effectively investigate an incident and preserve important evidence vanishes rapidly, long before a lawsuit is filed. Accurate results of an incident investigation are always beneficial to understanding the contributing factors that enable steps to enhance reliability. The cost of one unfavorable judgment from a serious incident can be a major financial impact to a utility. The writers of the guidebook suggest that the utility investigate each serious incident thoroughly as if it were going to result in litigation, and each minor incident to the extent necessary to improve reliability. The forensic approach can benefit both. The Forensic Guidebook is intended to assist troubleshooters and others who could find themselves responding to a serious incident, a major outage, large equipment failures, or harmful contingencies. The information presented in this guidebook will enable the responding utility personnel to immediately initiate a forensic investigation to best determine and capture the true and relevant facts related to the incident. Emphasis will be directed to creating a utility 1-1

Introduction

investigation team to determine the cause of accidents, failures, and unplanned outages involving electric power distribution equipment and systems. A Word of Caution is appropriate here. As with the application of any guidebook, the reader must first conduct their actions in conformance with company policies and operating procedures. Specific individual circumstances of the event related to the prevailing laws and regulations in the jurisdiction of the power distribution system involved must also be considered.

1-2

2
BACKGROUND OF THE FORENSIC PROCESS
What is the forensic process?
The forensic process is an investigative methodology or approach that is structured to provide results that will be acceptable under the rules of evidence in a court of law. Although the term forensic conjures up visions of great detectives and medical examiners solving crimes, the relatively recent expansion of its use by the courts now includes nearly all professions. All practices have one thing in common; they must adhere to the rules of evidence from the beginning to the end of their investigation. The forensic investigative approach can be defined as the application of engineering and science within the jurisprudence system. This is useful in any investigative exercise to assure that no stones are left unturned.

Where did the forensic process come from?

The word forensic is derived from the Latin word, forensis, meaning belonging to, used in, or suitable to courts of law or to speak and debate in a public forum in legal matters, including trials, that can be traced back in time more than 2,000 years. What is more interesting is that recent research has revealed that forensic investigators were probably engaged by King Hammurabi of Babylonia, nearly 4,000 years ago, to determine the underlying or root cause, in order to resolve all manner of calamities in an ancient civilization, including fires and structural failures!

What is the forensic process used for today?

Any public presentation of an opinion on a controversial issue is subject to challenge and debate. Whether the forum is a town meeting, a state regulatory commission, a press conference, or a court of law, the presenter must be prepared to defend their position. A defendable position can only be achieved by a systematic and thorough investigation procedure while applying constant objectivity. The forensic process is such a procedure.

Of what benefit is the forensic process to electric power utilities?

Those utilities that practice investigation procedures of a forensic nature in the analysis of routine failures of equipment and minor accidents benefit from highly accurate and useful results to reduce accidents and improve reliability. In addition, given the experience that electric power 2-1

Background of the Forensic Process

utilities have already had with the courts, the forensic process has proven its value in litigation. The forensic process can be considered to be a specialized form of root cause failure analysis. It can also improve overall reliability of the distribution system to satisfy regulatory requirements, to reduce danger to employees and the public, to comply with workers compensation rules, and to provide accurate answers for public relations. The forensic process follows a very logical order in support of the scientific method. The principal phases of a forensic engineering investigation will usually include, but are not necessarily limited to the following: Obtain Initial Information and Facts; Prepare Investigation Response; Coordinate with Restoration; Document Site Conditions and Observations; Collect Information and Evidence; Conduct Tests and Research; Analyze the Evidence and Facts; Reconstruct the Incident; Develop Conclusions; Provide Recommendations as Advised; and Prepare Report or Presentation as Instructed.

The forensic investigation provides the benefits of a comprehensive root cause analysis. In addition, it preserves the option for the investigation conclusions and supporting evidence to be used in court, should the issue become a legal concern.

2-2

3
THE FORENSIC ANALYSIS PROCESS APPLIED TO ELECTRIC POWER DISTRIBUTION SYSTEMS
An electric power distribution system is the delivery system from electric power sources to retail customers. It is a complex network of equipment and connecting circuits. The majority of power reliability and quality problems originate in power distribution systems. While any equipment will fail given enough time, the historical data on interruptions suggest that the most frequent causes of power distribution system problems involve the following: Busbar Bushings Capacitors Circuit Breakers Connectors Elbows Insulators Load Tap Changers Overhead Lines Poles and Guy Wires Reclosers Splices Surge Arrestors Switches Switchgear Terminations Transformers Underground Cables

In minor, as well as in serious incidents, the first actions and observations are often critical to capture for the success of any investigation that follows. The first utility representative on the scene may be establishing the initial contact for the utility with emergency officials, property owners, and witnesses. When serious accidents occur, a forensic investigation that may soon 3-1

The Forensic Analysis Process Applied to Electric Power Distribution Systems

follow will benefit from these initial efforts to establish good cooperation and access to information. When an equipment failure causes a major loss or catastrophe, such as the loss of service to large areas of customers and/or special industrial consumers, an investigation will likely follow. This is especially true if the failure appears to be related to a piece of equipment that is critical to reliable future operations and very similar additional equipment exists on the system. In addition, chronic equipment failures that have not been resolved and have the potential to eventually cause a serious incident would benefit from a forensic process analysis. Upon arrival at a serious incident scene, the utility representative can anticipate the need for a formal forensic investigation if one or more of the following are noted: A death has occurred; or Injuries requiring emergency medical services for any person are involved; or The incident involves an unusually long period of loss of service that impacts a relatively large number of people in a customer service area; or Significant damages have resulted from a utility incident to non-utility owned property; or It is evident that a negative public relations issue could be raised in the news media.

When a failure occurs, the primary focus is to restore electric service to customers by the most expedient and safest means. For the crew sent to get the lights on, determining the cause of the failure is usually secondary to repairing the problems to restore service. The cause of the failure is left to be addressed by investigators after the service is restored. During the repair activities, any evidence that could reveal the cause may be substantially altered or discarded. It is important that first responders are observant as they restore service, and that they try to preserve potential evidence. In one instance, a forensic engineer was asked for help to determine the cause of a pad mount transformer explosion, which was involved in serious fire damages to two apartment buildings. Unfortunately, the damaged transformer parts had not been saved. This was particularly disappointing in this incident where both a fire and electrical arcing had occurred. Testing the beads of metal from the arc may have revealed if the fire came first, or if it was caused by the arc. Saving these beads and all other damaged parts as evidence can be very important in determining causation. Forensic investigation procedures utilize many of the techniques found in root cause analysis, but expands the search for facts to fully reconstruct the incident. Not only to determine what happened and how, but also why and what or who is responsible. Thus, the forensic investigative process utilizes the scientific method, current analytical techniques and liberal doses of common sense. This Forensic Guidebook is based on the same classic methods tuned for application by utility engineers and technical field specialists. The forensic analysis process provides the utility with a confidential full disclosure and understanding of both the good news and the bad news. That is, what are the defensible positions to establish? and what are the vulnerabilities to be concerned about? This part of the forensic analysis is achieved by asking, in addition to the question what went wrong? asking the questions why? and what could have been done to prevent it? Were any codes, standards, recommendations, guidelines, company procedures, or industry practices not in compliance? What are the vulnerable issues? or bad 3-2

The Forensic Analysis Process Applied to Electric Power Distribution Systems

news? In the wake of a serious accident or failure, there are a number of critical decisions to make that require complete and straightforward information in order to reach them in a timely and most beneficial manner. The complexity of making these decisions is compounded due to the fact that most utilities are rapidly losing their most experienced personnel who have the knowledge to determine the root cause of many types of distribution systems failures. The Forensic Guidebook will provide the fundamentals to develop the techniques and procedures to answer these very important questions. As you continue through this Forensic Guidebook, you will learn to appreciate these key points: 1. Being prepared to investigate with a team approach; 2. Having forensic investigation kits available; 3. Engaging an outside forensic expert when needed; 4. Not jumping to conclusions early with the obvious, it may be wrong; 5. Taking lots of pictures - film is cheap; 6. Learning how to interview witnesses; and 7. Saving all the evidence. While utilities operating electric power distribution systems may have developed procedures for determining the cause of interruptions using physical root cause analysis, the Forensic Guidebook provides a comprehensive analysis methodology for all incidents, and helps ensure that the utility will be properly prepared for any challenge that may come.

3-3

4
PREPARATION FOR SITE INVESTIGATION
In most cases, the first utility representative to reach the scene of an incident will usually be a troubleshooting crew in a light line repair truck. Distribution dispatch may have little more than a general description of the incident, the address and perhaps the circuit feeder involved, and an indication of what the caller is requesting of the utility. The first responder must be prepared to conduct the requested emergency services first, and then be able to initiate the gathering of information to understand what happened. For instance, the request may be to shut off the power to a burning commercial building, or in another case to restore service to a residential community after a vehicle has struck down a pole. The depth of factual information required must be sufficient to explain the situation to those who need to quickly make decisions as to what efforts are needed on behalf of the utility. The response plan should be designed to immediately trigger the addition of other company personnel onto the investigation team as soon as certain criteria are established. Questions need to be addressed such as who in the utility should be notified? and is an investigation by utility technical staff needed? Additional technical assistance from the engineering and operations departments, or assistance from the law and claims departments, is usually involved in major incidents. In some cases, forensic experts will be hired to conduct the analysis. The best time to prepare for an investigation is long before the major accident or failure incident happens. An organized response plan should be developed with specific objectives for the utility team of pre-assigned company personnel. The utility team provides an organized core of technical resources that would be available to also assist as needed on smaller routine incidents. Each team member must clearly understand his or her objectives and responsibilities. The utility response team should be made up of related equipment and operations specialists, who can immediately bring the best technical talent and any requested data or information to the investigation. The type of faulted equipment involved will obviously provide direction in selecting the technical talents needed for a specific response team. Experience with past major incidents has shown that one person must be identified to lead the investigation. The lead investigator will be responsible for minimizing the impact on the restoration of normal service and juggling all of the forensic aspects. If not, the results can be disappointing. Thus, the identified lead investigator must be the point person and responsible for all aspects of the investigation. Ideally, this lead investigator should have both a working knowledge of the investigation process, as well as knowledge of the equipment involved. The lead investigator will direct the investigations sequential tasks, coordinate with restoration and operations, as well as work with a team of utility staff to analyze data and information as it is collected. If the incident attracts the press and becomes a media event, the response plan should appoint an individual to be the contact and spokesperson for the utility. Usually, this is not the lead 4-1

Preparation for Site Investigation

investigator. The utility contact person should be an experienced public relations officer or lawyer from the utility corporate staff. This appointment should be part of pre-planning for the response, and made well before the first encounter with the press. For this reason, communications from the field to the spokesperson must be accurate and provided on a regular basis. Proper pre-planning among all team members can assure that this will occur. Pre-planning and preparation for the site investigation should also provide for properly handling the flow and storage of materials and documents produced during the investigation. If no litigation action results from the incident, then there is less concern for disclosure outside the utility. However, if litigation should occur, all materials and documents in files held by anyone, except the attorney of record, either a utility law department staff member or an outside law firm member representing the utility, are subject to discovery and access by the plaintiff. This information often includes some documents that may need to be placed in a confidential file as well. Eventually, almost all materials that are relevant must be disclosed. A useful utility response plan sequence would function as follows:

Figure 4-1 Utility Response Plan Sequence

4-2

5
INCIDENT INVESTIGATION FIRST RESPONSE
This section is intended for use by the first utility responder crew or troubleshooters sent to the incident site as they respond to the initial call. The troubleshooters main function is to make the area safe and restore service or to assess the need for others to accomplish this. The first utility responder also has the opportunity to provide very important assistance in the early documentation of the scene and the preservation of identified evidence. All investigation activities must be coordinated with the recovery and restoration team and avoid conflicts to achieve multiple sets of objectives. Although it is important to begin the investigation as quickly as possible, no investigation is so important that it should jeopardize safety, operation of the distribution system, or interfere with medical attention services or protection of property. Utility operations personnel must always follow their own local procedures, including notifying the legal and claims department immediately of major electric distribution system disturbances or major equipment failures, and/or that non-utility property was damaged and deaths or injuries were involved. When a serious incident involving utility equipment is reported, quick response by the investigation utility team to the site of the incident is essential. It will be their assignment to observe and evaluate the incident site conditions before they are altered, thoroughly document the distribution equipment involved, and take control of all pertinent evidence owned by the utility. A sample Incident First Response Report (IFRR) has been provided in Chapter 13, CHECKLISTS, as Section 13.1. This form is intended to enable the first utility responder to accumulate the initial information required in a consistent and orderly manner. Computer entry of this data is anticipated for use in an equipment reliability program. Each utility is encouraged to develop their own forms and checklists to be most suitable to their circumstances and procedures. The IFRR is a logical and systematic checklist for the first responding utility troubleshooter to use upon arrival at the incident scene. It is important to remember that this form, along with any other file materials, may be provided to those with claims against the utility at some future date. Therefore, it is essential to write notes clearly with only the facts and data collected. Notes should never include any personal comments, criticisms, remarks or opinions. The first item on the IFRR is to Size-up the Incident and report back to a supervisor. (Take overall incident scene photographs here.)

5-1

Incident Investigation First Response

The report contains four descriptions of Incident Conditions Found, including: I. Outage Only; II. Non-Utility Damages; III. Injuries or Death, Outage with High Impact, Significant Non-utility Damages, or Negative Media Coverage; and IV. Other, unusual or compound conditions. The first three Conditions Found may be used to determine the level of investigation to be applied as follows: Level of Incident I. Outage Only Level of Investigation to be Applied Troubleshooter Responds: First Responder Field Kit, (See Forensic Investigators Field Kit, Section 13.2), including: Checklists as Appropriate Acceptable or Minimum Camera Kit to obtain Quick Low Resolution Photos (See Acceptable Camera Kit, Section 13.3, and Minimum Camera Kit, Section 13.4) Incident First Response Report Forms Evidence Tags and Log Permanent Markers Unacceptable Reliability Report Forms Notebook with Cover Utility Team Responds: Utility Investigative Team Field Kit, (See Forensic Investigators Field Kit, Section 13.2) including: EPRI Forensic Guidebook Optimum High Resolution Camera Kit using SLR, Film, or 4 to 5 Mega Pixel Digital Camera (See Optimum Camera Kit, Section 13.5) Notebook with Cover Graph Paper Tape Recorder, Pocket-size Evidence Tags and Log Permanent Markers Tape Measure (100 feet and 6-inch scale) Flashlights, Small Pocket and Large Lantern Spare Batteries Zip-locking Plastic Bags (heavy duty/various) Note: It is assumed that the Utility Team will include, at a minimum, the first responder troubleshooter, a foreman or supervisor, and ideally a classroom-trained photographer.

II. Non-Utility Damages

5-2

Incident Investigation First Response

Level of Incident III. Injuries or Death, Outage with High Impact, Significant Non-Utility Damages, or Negative Media IV. Other, Unusual or Compound Conditions

Level of Investigation to be Applied Forensic Team Responds: Including all in Levels I and II plus a forensic expert and their additional equipment, as required. To Be Determined on a Case-by-Case Basis.

Continue to use the IFRR by checking each box for which data or information is available. Begin with IFRR Section 1, Incident Data, by recording the date and time of the event from time-based recordings, if available, or eyewitness accounts. Provide an accurate street address and the name of the owner of property involved, if not owned by the utility. In instances where the incident is not related to a street address, provide at least two reference landmarks or the geographical coordinates so that the site can be located on scaled maps. Further with maps, obtain the relevant local road and topographical map, a, detailed street map, the utility One Lines marked with the equipment configuration at the time of the incident. If maps are not available, check the appropriate box. Check the box and record the Operating Voltage and Feeder or Service Line involved. Next, move to IFRR Section 2, Description of Incident, by selecting the most appropriate item or items to generally indicate what happened. If more than one item was involved, place a number in the box to indicate the sequence, if possible. For instance, if a dig-in caused an electric shock and a power outage, indicate the sequence numerically in the individual item boxes. (Take enough photographs to indicate the apparent/estimated sequence of incident events.) IFRR Section 3, Damaged Utility Equipment and Resolution, requires a careful inspection of the equipment involved in the incident to describe those units that were damaged. Notes are encouraged to record the full identification of each piece of equipment involved along with details of specific damages observed. Although it may be too early to know with confidence what course of action or anticipated resolution will be taken, indicate the most likely as either Repair or Replace in the appropriate box. Again, details as to this course of action can be added to the written notes. (Take detailed and overall photographs of each equipment item checked.) Once it is determined, IFRR Section 4, Description of Restoration and Times Recorded, can be checked for the appropriate action selected. As in the Description of the Incident, restoration may occur in a sequence of steps and can be indicated numerically in the selected box. Be sure to indicate any additional details in the factual notes including the date and time of each Power Restoration action taken. It is not intended that IFRR Section 5, Initial Potential Causes and Contributing Factors, be a conclusion, and certainly not an opinion. However, there may be significant clues or early facts that will enable an Initial Classification to be selected. If the available clues and facts are 5-3

Incident Investigation First Response

not sufficient at this time, select 5.4.4, Unknown Cause. Facts and data used to make this selection should be included in the notebook while carefully indicating the source of the information. (Take detailed photographs of each item checked as described.) Personal observations are important to record on film related to transient or changing conditions, such as a fire in progress or the weather. As in other lists previously reviewed, the Initial Classification can involve more than one item. If a sequence is apparent, use the numeric indication in the appropriate box, otherwise check as many as may have played a role in the cause of the incident. Remember that the purpose of this section is not to state an opinion, but to suggest the initial indications that will enable others to continue to investigate the root cause, and/or to provide useful performance data input to equipment reliability programs, such as an RCM program. The Initial Classification may need to be modified at a later date when more facts become available. Internal and inherent causes include those that are internal to the equipment and that are initially deemed to be the cause of the incident. The word defect has strong legal meaning and its use can trigger an application of the federal product liability laws. Since the utility purchases products or equipment from manufacturers, a product liability claim places the utility on the plaintiff side of the courtroom - a condition that neither party desires. Therefore, most, if not all, of the product liability cases between utilities and their suppliers are resolved by arbitration or settlement to avoid the courts. Deterioration is the end result of a number of potential factors, including: environment, loading or usage, number of operations, fault experience, maintenance, and handling. External and non-inherent causes include failures that are external to the material that may be initially suspected to be the cause of the incident. Extreme or extended unusual weather conditions can have a significant impact on the failure potential of most power distribution system electrical equipment. Weather conditions such as high winds, lightning, hail, ice storms, prolonged heat and humidity storms, and floods are common external causes of equipment failures on electrical distribution systems. Evidence must be documented that correlate to the weather event as supported through official government weather data. Distribution system loading data will usually be available from the system load dispatch control for main feeders in the suspect circuit where overload and overheating may have occurred at the time of failure, and for the period prior to failure. Customer demand records can be collected for the same periods and used to determine if proper notice was provided to the utility for any customer changes that had been made to cause a significant increase in consumption. Mechanical causes include improper assembly and installation, such as bolts not tight enough or too tight, and cracked or broken parts. Underground cable can be cut and damaged by sharp rocks left in the backfill. Accidents including cars striking power poles, animals shorting across insulators, and other external factors that contact energized parts of the distribution facilities. Refer to Chapter 16, BIBLIOGRAPHY, Animal Caused Outages, 94-5, RER Manual 945, for animal outage mitigation procedures. Other known external causes include workmanship that may be related to the original installation or more recent repairs. Look for incorrect sizing of parts and/or improper installation. Improperly crimped connectors or the improper choice of connector type or size should be considered. 5-4

Incident Investigation First Response

In some power companies, there is an informal maintenance practice among the various construction departments to keep a scrap bin of parts or equipment removed from the system judged by the workmen to be satisfactory for reinstallation. Any equipment or device removed from the electric system for unacceptable reliability should be destroyed or rendered obviously unfit for reinstallation. As an illustration: a control wireman working during a planned outage of a major power circuit inadvertently damaged an important auxiliary relay controlling the tripping of several circuit breakers. Finding an identical relay in the scrap bin which appeared to be operational, he interchanged them, leaving no paper trail and with no authorization. Several months later the relay failed to operate due to the same defect for which it had been removed from service two years earlier. Major damage resulted. The implementation of an identification and control process to prevent the continued use of equipment in an unsatisfactory condition could reduce problems like the scrap bin incident. In Chapter 13, CHECKLISTS, a procedural form, the Unacceptable Reliability Report (URR) created for the Forensic Guidebook, is offered as a model to provide for documenting and tagging equipment that exhibits unacceptable reliability and has failed or been removed from service. (See URR Form, Section 13.6). In addition, data accumulated by the URR process will provide historical performance information for the specific class of equipment. This information is valuable to both improving reliability of performance, and to conducting root cause analysis investigations of failures. Thus, when investigating an equipment failure, it is always wise to check its historical performance as well as the most recent maintenance work required for the class of equipment. An RCM program, as found in EPRI documents, should also be considered, as it would benefit directly from the URR data collected and any RCA findings. Switching operations and any load management conducted by the utility, just prior to the incident, are essential inputs to the early evaluation process. Look for any other local or remote influence that may have contributed to the accident or failure. Harmonic vibration or galloping wires, observed as excessive movement in and between conductors, should be considered in cases where overhead lines touch each other or break during wind conditions. Lightning damage by a follow-on surge should also be considered. Cause Not Determined and/or Not Examined are frequently the case these days! Often this is due to the lack of even a basic root cause investigation. In some cases, only limited physical materials have been saved to be examined, but no obvious reason for the failure can be found, leading to an unknown cause result. Following the guidelines presented here will reduce the number of undetermined causes. Finally, IFRR Report Section 6, Retention of Equipment Involved, provides appropriate steps to take later at the incident scene. It is very important to follow the steps to tag, document, and log evidence, as described in Chapter 6, SITE INVESTIGATION PROCEDURES, Section 6.10, Evidence and Sample Collection. (Photograph each item retained from all sides.) A logbook must contain the same information as the tag for each piece of equipment or material retained as evidence to record the collection, access by others, any change of custody or alteration, and its final disposition. 5-5

Incident Investigation First Response

Evidence should be secured in a locked area and protected from deterioration by being placed indoors or outdoors under a protective cover, if too large to store inside. Cable should be sealed to prevent moisture from entering, even though there might be a hole through the sample. An index of each item placed in the initial storage location for evidence should be compiled and added to the case file of documents.

5-6

6
SITE INVESTIGATION PROCEDURES
The first actions taken by the utility upon learning from a troubleshooters report that a serious accident or failure has occurred on its system are unique to each incident. Although preparation is critical for success, experience and knowledge must be available to apply in the more complex and costly incidents. Site investigation procedures are based on a very logical process of collecting data and information to determine or reconstruct what happened and how it occurred.

6.1

Response and Task Prioritization

The seriousness of a particular incident must be considered in establishing the response. However, it may not be immediately apparent. For example, although the electric distribution utility is called to respond to structural fires to disconnect service, the cause of a fire may not be determined by others until later, when a claim is filed by an insurance company. There have been numerous situations where the possible injury or damages to a member of the public has not been known until a lawsuit was filed against the utility months, and perhaps as long as two years, after the incident. These situations may make it difficult to decide what evidence to save, and how extensive the inspection and documentation should be. When a serious incident occurs, the best policy is to assume a lawsuit will result, and conduct as comprehensive an investigation as practical. It is better to err on the side of conducting a thorough investigation, as adjustments can be made later; however, you can never go back to the same incident scene to recover what you missed.

6.2

Restoration Coordination

Make sure that the objectives and requirements of the investigation team are complimentary to and coordinate with those of the rescue and restoration crews. The lead investigator must communicate with the supervisor of the restoration work from the outset. Suddenly appearing on the scene three days after restoration has been underway will leave the investigator with little to say about the preservation of evidence and alterations to the incident scene. A situation like this can be the result of either delayed communications or the lack of a pre-plan for response. This can be avoided by having a trained and equipped investigation team of utility personnel and a contract specialist support staff on call. It is also important that the first utility responders and restoration crews be trained and knowledgeable of the forensic investigative process in order to capture the critical evidence and document the initial conditions as found. Together, these utility field staff members have the opportunity to make sure that critical damaged equipment parts and remains are not altered or thrown in the dumpster, but are protected until the investigation team arrives.

6-1

Site Investigation Procedures

6.3

Initial Facts Assembly

At the outset of the investigation, information will be coming from many sources. Some of it may be reliable and some may not. Collect all that is available and sort it out as soon as possible. Assemble the facts in a logical and chronological sequence to begin connecting the pieces of the puzzle. The initial facts should include the most recent prior activities involving the personnel injured or the failed equipment and the details of the local system in which it operates.

6.4

Visual Inspection

As with many forensic procedures, the site inspection produces the best results if it follows a thorough routine. It is recommended that the investigation begin with a visual inspection, starting with the periphery, at the outer boundary of the accident or equipment failure where no damages or injuries occurred. From there, the visual inspection should proceed toward the immediate and involved areas of the accident or equipment connected to the failed unit, noting any damages or unusual conditions. The observer should note damage patterns in surrounding areas and structures from heat, smoke, insulating liquids, and solids, buckling and bending, and fire suppression activities. As the area of origin or primary failure is reached, the attention to detail should increase and closer scrutiny will require brighter lights, magnifiers, and patience. This exercise should be repeated until the visual story is understood, and before anything is disturbed, moved, or discarded. Detailed documentation begins with the visual inspection.

6.5

Documentation

Documentation, or the collection of supporting materials to provide decisive evidence or information, comes in many forms. Forensic investigators have applied all manner of documentation techniques from hand sketches, to scale drawings, to 3D CAD drawings, to photography and videography, and to laser mapping. By far, the most important and versatile of all documentation methods is photography. Twenty-five centuries ago a Chinese philosopher, Confucius, declared, A picture is worth a st thousand words. If it were true in ancient China, it certainly is true today in the 21 century with all of the means to capture and display very high-resolution pictures. Today, forensic photography involves the production of undistorted images from investigative events, objects, and scenes. Documentation should be done in precise detail by a knowledgeable and competent photographer in a manner suitable for use in a court of law or any public forum. Many catastrophic accidents and failures have been solved through the use of photographic evidence. (See Chapter 13, CHECKLISTS, Forensic Photography Tips, Section 13.7). It is best to go through the visual inspection procedure, first, with camera in hand only to document very changeable conditions such as fire, smoke, arcing, unusual weather, or emergency activities that exist for a short time. When the emergency is under control, plan a photographic documentation sequence of the incident site and select key subjects on your first visual walk-around inspection. While inspecting the incident site, imagine describing the basic story of what happened in detail as the photographs are taken. This procedure provides two benefits: it encourages the investigator to think visually about the cause of the accident or failure 6-2

Site Investigation Procedures

and gather evidence to support it, and it provides a logical approach to make sure that the most important images are captured in a convincing sequence. There is a well-known statement that reminds forensic investigators of the importance of photography, and to encourage its extensive use Film is cheap. Small details of the accident or equipment failure can be captured on film, such as arc marks, fractures in solid insulation, and the color and contaminants in liquid insulation, and color patterns of overheating, to name just a few. The camera can also demonstrate the point of view of witnesses, record the position of tap changers, circuit breakers, switches and relay targets, and capture the trace of fault recorder charts. Begin to photograph from the outer limits of the incident scene and work towards the center of the damage or area of the injury. Be sure to document those objects that will disappear quickly and/or may be removed for safety reasons or to restore service. As an item is identified as evidence to be saved, photograph it from several angles to document its condition upon collection, as a precaution if any alterations are discovered later, or, in the worst case, the item is lost. A reliable camera and photo shop should be used to develop the exposed film as soon as possible. Most film processors now offer to burn a CD of the film images at the same time prints are made for a small extra fee. Take advantage of this as well. If a digital camera was used, be sure to download all images to a CD. Before storing the prints, negatives and CDs from the incident with the file in a safe place, make a caption list of all the photographs with the image or frame number, subject, time, and date. (See Incident Site Photography, Section 13.8.) Although digital photography is improving rapidly in quality and gradual acceptance by the courts, the well-known chemical film camera remains the equipment of choice by forensic investigators. Its traceable negative and quality of image, as well as a wide array of lens and lighting accessories, meet any documentation needs. Therefore, it is critical to carefully protect and store all film negatives in order to enable enlargements to be made and to satisfy any challenges of authenticity. Digital cameras are becoming very popular and can play an important role in the investigation by providing rapid photos/images for company-internal information. However, under current technology, digital cameras produce images that are computer files, which can be altered, leaving no trace of the changes. Although digital photographs are not generally accepted by the courts, they can be used by the early responders to provide rapid and important visual information to management from the remote incident site, via the Internet. The distribution utility investigation team should have access to good photographic equipment and experienced photographers. They must be able to respond quickly to any location on the system to document serious accident and major equipment failure scenes. The utility may establish a quick-response services contract with a local independent forensic photographer who can be relied on to respond to a scene within two hours. As a lower cost alternative, a small fully automatic 35mm point and shoot camera can produce very useful photographs, and an inexpensive digital camera can provide quick transmission of incident scenes to a supervisors office. These types of cameras are ideal for the utility trouble truck or emergency response crew to have available for use when documenting an accident scene or an equipment failure. The troubleshooters line truck should contain a kit with one of these cameras at all times. 6-3

Site Investigation Procedures

These small inexpensive cameras can take very good pictures of average scenes and good lighting, but are not able to take close-ups. However, the most important photos will be those of the scene from various positions and the location of key items. The acceptable camera kit should suffice until the optimum camera kit and experienced photographer are able to arrive. As a backup to the optimum and acceptable camera kits, keep a few one-shot or single-use cameras with flash handy to capture the important early photos if no other camera is available. Although the results may not be as good, at least you have a reasonable photograph, which is certainly far better than no photos at all.

6.6

Facility and Pertinent Equipment Diagrams

Diagrams of the facility and equipment involved in an accident or failure are required to produce an accurate composite drawing of the scene of the accident or equipment failure scenario to tie all of the factors together and reconstruct the event through visual means. This process is important to connect the puzzle pieces together, and test the various theories as to what could have happened. In some cases, investigators have found that their theory as to the sequence of events sounded good, but when played out on a scale model or diagram, the pieces of the theory did not fit together. Diagrams of the scene and equipment are also used to add the observations of damages from fire heat, smoke, and structural collapse that may occur during the incident to either or both, the site or equipment. This allows the sequence of events and cause and effect scenarios to be evaluated against the time data. Working with visuals improves the outcome and quality of findings in any investigation. Physical measurements of significant items in photographs can be valuable. While a survey crew may do this at a later date, the early measurements can be crucial. There is usually a question about the height of the power lines at the time of the accident, and Did the crew alter the height (or depth, in the case of underground) of the facilities? Accurate measurements taken as soon as possible after the incident are very important. A tape measure extended to a known length or a person of known height can provide a scaled reference next to an object or at a known distance in a photograph.

6.7

Similar Equipment Comparison

In systems that contain multiples of the same equipment that failed, the investigator has access to potentially identical equipment that may demonstrate the early stages deterioration. Also, undamaged identical equipment becomes a comparison example of the failed equipment to determine the before condition and its construction and operation. It is wise to identify a comparison example as close to the failed equipment for these reasons, and to retain it for testing in exploring potential failure modes. Equipment inspection and maintenance records for similar equipment that failed can provide important information as to historical problems that could result in failure. If other failures have occurred, the reports of their investigations will be very important to review for any similar conditions that may point to a repeating problem that can be identified and caught before any additional failures occur. An example that illustrates this point is when a new item may be 6-4

Site Investigation Procedures

defective or improperly installed and fails upon start-up. Records of ALL problems must be kept for this to appear in the database, and to help determine when to inspect new products.

6.8

Initial Physical Reconstruction of Damaged Equipment

While observing the scene of the accident or failure, attempt to visually reconstruct what the damaged equipment involved originally looked like before anything is moved. In many cases, the damage is more severe in certain areas than others, and the original parts or materials are not recognizable or present. This is when the use of comparison equipment or detailed Original Equipment Manufacturer (OEM) drawings, or both, is required. Remaining parts of equipment may be identified and oriented as originally installed or configured in the failed equipment. Photography and videography are very useful to document the attempt to physically reconstruct the equipment, to learn where the major damages occurred and what positions and settings may have existed at the time of failure.

6.9

Selected Testing at Incident Site

Selected operational testing of equipment at the incident site can be very useful when conducted properly. However, all precautions must be taken to prevent additional failures or outages. Reconstruction at a test facility or laboratory should be the first consideration. In some cases, complete isolation of the equipment can be assured in the field setting at the incident site. This is necessary where removal and transport to a laboratory could alter the electrical conditions causing the subsequent tests to be unreliable, or where field-grounding conditions cannot be replicated in the lab. This is particularly applicable to relay settings, resistance of connections, and ground resistance values as they affect the operation of protection systems. Field tests need to be carefully planned in order to be safe and obtain meaningful results. These tests may be conducted long after the actual incident and complete records must be kept of the set-up and results.

6.10 Evidence and Sample Collection

A forensic investigator must be prepared to support opinion statements. Opinions are likely to be challenged by other investigators, and therefore must be supported by sufficient evidence that will be court acceptable. Since evidence includes all the physical remains of an incident, among other things, it is important to know what to save and how it should be saved. Official agency and/or emergency response personnel have the authority under the law and thus the control of any public-related accident scene until they have completed their work. They will release the scene and property involved to the owners upon completion of their investigation. It is therefore the responsibility of the responding utility crew to assist the authorities as requested, and to coordinate any safety-related issues regarding the power distribution system. In some cases, the authorities may select and remove evidence for their investigation. The utility representative only has the right to locate and remove any utility-owned equipment or materials 6-5

Site Investigation Procedures

from the scene without the permission of the authorities. In selecting evidence, it is always beneficial to collect more than may appear to be needed. For example, if a burn mark on a cable is the size of a quarter, collect a sufficient length of the cable to capture two clear repeated identification stampings on the cables outer jacket, at a minimum. Each incident presents a different collection of evidence for selection. Accidental electrocutions may provide evidence such as: an exposed conductor burn mark on an overhead line; a long metal ladder with various burn marks at both ends; and burn marks on the ground where the person involved was standing. A commercial property fire may include evidence of electrical events such as: blown primary fuses on a distribution transformer; faulted cables in a CT cabinet; a heat damaged power meter; tripped circuit breakers in customer panels; and a clock stopped at a displayed time, for example, 09:30 hours. In these cases, the utility can remove the ladder from the scene as evidence, if abandoned by the owner, to preserve for access by all. Evidence in the fire case that can be removed by the utility would be everything listed, except the customer circuit breaker panels and clock. Always check with the law department when there are any questions about evidence collection. Evidence collection is the most sensitive activity performed during the investigation of a serious incident. If not done properly, the evidence may be worthless; and if collected and then altered or lost, can be very costly. This outcome is known as spoliation. Spoliation of evidence has been a problem in some cases where alteration or loss of evidence has heavily impacted the outcome of litigation. A large lawsuit occurred where samples of paper-insulated cables were saved, but not sealed to prevent moisture from getting into the tapes. This resulted in the dismissal of millions of dollars from the claim, based on the fact that the samples removed could not be adequately tested as moisture had penetrated the paper. Special care must be taken when storing evidence that could be altered by humidity or other environmental conditions. Although the most common problem is thought to be corrosion, the example above points out that moisture can have a damaging and irreversible effect on the dielectric strength of insulating materials. Regardless of the future needs of the investigation, it is always a good policy to protect sensitive evidence from any alteration due to the environment, as well as handling. The collection of evidence requires careful record keeping, as well as proper handling and storage. Each separate piece of physical evidence must be provided with a durable and wellattached tag that, at a minimum, contains the following information: 1. The date of collection; 2. A file or reference number for the incident; 3. The location where it was found; 4. The name of the person who recovered the object; 5. A short description including any markings found on the object; 6. An item number assigned to the object; 7. The initial storage location; and 8. The party taking custody and their contact information (normally recorded on reverse).

6-6

Site Investigation Procedures

A sample of an evidence tag is shown below in Figure 6-1. In addition to the tag, an evidence log must be maintained as a central record of all the evidence collected at an accident scene or failed equipment inspection. The log must contain, at a minimum, all of the information on the evidence tag and information as to the original storage location and any handling of the evidence for investigative purposes. The most important record in the log is that of any change in custody or destruction of the evidence. Those who are transferring custody must sign a separate document, and any destruction of evidence must have approval in writing from the owner or their attorney. These documents also become part of the evidence log file to be protected as it is also considered to be evidence. In the case of utility-owned evidence, the law department will have the final word on evidence.

Figure 6-1 Sample Evidence Tag

A log of all persons visually examining stored samples (name, company, date) should also be maintained, as no modifications of the samples are allowed without all sides being present. (See Evidence Identification, Removal, and Retention, Section 13.9.)

6-7

7
DATA COLLECTION
7.1 System Electrical Maps and Data

The most accurate and available distribution system records and drawings are essential to properly perform the investigation of any incident. Lack of this data or outdated drawings can impact the response result, and prevent a forensic investigation from reaching a root cause determination. It may also limit the application of improved reliability recommendations that may result. Thus, every effort should be made to maintain accurate records as well as providing for an accessible storage system. Incidents involving faults on lines fed directly from a substation usually require an assessment of the feeder electrical protection equipment and its inspection. In order to understand the logic and system configuration for the protection system, the connected substation design information must also be obtained. A search should be made for all substation drawings. Substation drawings may consist of the following: one-line drawings; three-line drawings; dc elementary drawings; tripping and logic drawings; panel layout drawings; and overall equipment placement diagrams.

7.2

Original Equipment Manufacturer (OEM) Data

Standard equipment data and operating specifications can be found in the OEM operation, maintenance and parts manuals. Although highly generic, these manuals are the best source of information, short of cornering a knowledgeable OEM representative field engineer. Data related to: circuit breakers; switches; disconnects; fuses; relays; instrument transformers; event and fault recorders; bus and cable systems; and other switchgear parts are first and perhaps only available from their respective OEM documents on hand. Transformer data will be available in the same manner covering a wide range of types and sizes from low-voltage instrument, to highvoltage substation transmission step-down transformers. Typical OEM data required to assist in evaluating the failure includes: model or type; serial number; voltage and/or current ratings; date of manufacture; insulation system and ratings; and circuit diagram.

7.3

Equipment History as a Class

With respect to electrical distribution equipment as a class, historical records and data can be important in establishing long-term effects and trends that may lead to a failure at a time when a sufficient triggering event occurs. Some electrical distribution equipment have experienced more problems than average for the class, and therefore become suspect when involved in an incident. Electric power industry organizations, such as: EPRI; Associated Edison Illuminating Companies (AEIC); National Rural Electric Cooperative Association (NRECA); and others, 7-1

Data Collection

provide information on equipment issues to its members through meetings and Internet bulletin boards. The OEM and other suppliers may provide advisories and after-market repair kits for recognized problems involving such items as lubricants, corona shields, and bushing gaskets that, if not selected properly or installed correctly, will lead to uncertain reliability and failure.

7.4

Relevant Equipment Operation, Maintenance, and Commissioning Records

Utility records should provide all of the installation and commissioning information from the early data, and maintenance and repair data that has followed. Reviewing the maintenance data is especially important to look for any trends of escalating failure-potential problems on a single device or among a group of similar devices. Operations data as to equipment loading, cycling, number of trips and total operating time can often provide clues as to the location and type of potential insulation degradation to be expected. Maintenance records should be reviewed against the operations data to note any anomalies that do not appear to be related to what would be expected from its service life. The forensic investigator is always looking for inconsistencies and abnormal conditions to help explain why only one device in many similar devices has failed.

7.5

System and Equipment Electrical Conditions

There is a need to define the event as to what happened and at what time. The power system conditions need to be identified as to the overall network, the lines in service just prior to the time of the incident, and the position of relevant bus tie circuit breakers. In addition, the loading on the system and the individual feeders or lines must be determined for a reasonable period prior to the incident, and any unusual variations must be analyzed. The available fault current related to the incident also needs to be determined to evaluate the protection system response from original design calculations and/or distribution system computer model studies that may have been conducted more recently. Finally, the number of customers interrupted and the duration(s) of the outage must be determined. (See Chapter 13, CHECKLISTS, Power System Conditions Data, Section 13.10.)

7.6

Site Physical Setting

The location of the incident site is often related directly to known local factors that can initiate or contribute to a failure or accident. Some locations have natural conditions, such as ocean shore areas that are prone to salt spray contamination of insulators. Other incident site locations are situated in highly congested urban areas where overhead lines are within code, but in proximity to residential buildings above the roofline. Some equipment is located in hazardous locations as well, such as those switchgear serving industrial chemical and petroleum plants. Industrial power distribution may also involve a high power demand from a metal alloy mill operating electric furnaces. Large cities, with nearly all facilities underground, offer a unique set of site physical setting issues in addition to their extremely demanding customer requirements. Thus, it is very important to describe and understand the incident location or site physical setting, and consider all of its effects on the power distribution system operation and possible failure modes.

7-2

Data Collection

7.7

Site and System Environmental Conditions

Depending on the location of an incident, environmental and site conditions may play a significant role in the determination of root cause. Equipment in a controlled environment, such as switchgear in a substation building, are less likely to be experiencing any stress due to environment, unless it is allowed to be altered. For instance, if proper ventilation is lost causing overheating, there may be an acceleration of a failure in process, and if moisture is allowed to enter an energized cubicle and create tracking paths, sudden and serious faults may quickly result. Thus, building maintenance records may contain clues as to contributing factors or may provide the reason for electrical equipment failures within the controlled environment. Outdoor environments not only have the weather to contend with, but airborne particle contaminants, gases, vibrations, wildlife, and physical impacts as well. Data as to the outdoor environment are best obtained by an inspection of the site on a clear day for best visibility and documentation. Weather data is obviously important to consider in outdoor equipment failure incidents and accidents involving flooding, high constant heat, high winds, and lightning. Although the recollection of the weather conditions by witnesses is important to collect, the official recognized weather data source is the U.S. Weather Data Service found on the Internet at www.ncdc.noaa.gov. Lightning strike data for North America is recorded by the National Lightning Detection Network. This data can be obtained from www.lightningstorm.com and may be useful for determining if the root cause is related to lightning.

7.8

Relevant Standards, Codes, and Practices

Regulated electric power distribution companies operate under tariffs that are licenses granted to them by the state, local jurisdiction, or municipality to supply power and charge certain rates, while meeting standards of quality and reliability. The tariff can be cancelled if a utility fails to meet power quality standards, measured by specific voltage variation limits and strict limits on variations from the standard 60-Hertz frequency. In addition, if the supply of quality power is not continuous to certain standards, the utility is also at risk of losing their license. In addition, there are safety operations codes for utilities, where the National Electric Safety Code (NESC), American National Standards Institute (ANSI C2) applies to electric supply and communications systems, and numerous Institute of Electrical and Electronics Engineers (IEEE) standards apply to specific equipment, design and installation, operations, and maintenance. Industry organizations, such as EPRI and others, provide guidelines and recommended practices. In addition there may be local regulations imposed by certain cities and government agencies as well as individual utility company practices and procedures. Whatever the incident, all standards, codes and regulations that may apply must be considered in the forensic investigation process, as the opposing view (plaintiff) may attempt to discover an issue of alleged noncompliance to develop a lawsuit against the utility. If that approach is not successful there may be an attempt to find an industry guideline or practice that was allegedly not followed. (See Relevant Standards, Codes, and Procedures, Section 13.11.)

7.9

External Factors Beyond Utility Company Control

There are other influencing factors leading up to and during the incident beyond the utility company control. Look for nearby construction activity, preceding property damage accidents, 7-3

Data Collection

customer electrical system modifications, and even sunspot activity that may be relevant to the incident. By collecting all of the available data concerning the conditions external to the utility both inside and outside the incident location on the distribution system, the triggering event or condition may be found that suggests the root cause. This could include a possible defect or weakness in the equipment. An external triggering event will be revealed by conducting a thorough search for all possible factors.

7.10 Fault Recordings

Aside from the distribution system records and drawings, the remaining available data should be prioritized for collection, based on the risk of its disappearance. For example, computer-stored fault recordings, relay targets, enunciator panel indications, and video surveillance tapes should be given priority in their collection since they could be erased, reset, or recycled if not quickly secured. (See Event Records, Targets, Alarms, and other Recorded Data, Section 13.12.) It is well known that computer data files can be easily altered and deleted. Therefore, when computer data is located for collection as evidence it must be collected or downloaded without alteration or editing if it is to be used in litigation. It is also important to use a new disc to make the unaltered master copy for storage, and then create a backup disc for access to the data. Always work with copies of recorded analog and digital data, including videotapes. When examining the digital fault recorder (DFR), identify all channels recorded as to input source and location, and any scaling factors that may be used for the data. Note the sampling frequencies and number of cycles monitored both pre- and post-fault. In order to minimize any misinterpretations, determine the anti-aliasing filter setting. It seems that there is always that one person in the utility that specializes in interpreting DFR data. Find him or her! (See Digital Fault Recorder (DFR) Data, Section 13.13.)

7.11 Relay and Protective Device Operation Data

Relays and the protective device system equipment may provide the only incident-related data to help establish the location of the fault and its electrical characteristics. When fortunate enough to have it installed, the data will be found in digital fault recorders. This data may be recorded local to the incident or at a remote site. Always retrieve the unfiltered record for all digital relays and all alarms, pre-event, event, and post-event. There may also be a sequence-of-events recorder, both local and remote. Obtain all related recordings with time and date stamp information. Also, do not forget to obtain copies of the station and dispatch logs. From this data, establish a timeline of events as to the sequence based on clock time, to the degree possible at this time. For relays, any special protection or remedial action schemes should also be noted. The goal will be to verify correct relay performance, and to determine the calculated clearing time as compared to the actual clearing time of the fault in an incident. If enough information is collected, the event could be reconstructed and played back to the relay system to verify its response.

7-4

Data Collection

CTs act as the sensors for the relays and must provide accurate and sufficient signals to activate the protective system. Also, the type of CT control cable (size, length and shielding) should be noted especially for remotely mounted relays located on panels inside a control house. Backup protective devices are essential to prevent initial faults from developing into a major sustained fault. Thus, the design of the backup needs to be determined, and then a description provided of what actually occurred.

7.12 Official Reports and Media Materials

Official reports are generated by the responding emergency services and police departments, and are usually available within a few weeks after a routine incident. The information provided is typically brief and factual, as they are required to file a report on each call to an emergency situation. Remember that if a crime has not been committed, there is no further action required on their part, and the case is closed. It is important to collect these reports to complete the timeline of events response, and to evaluate the first sightings of emergency response crews. It is also likely that the utility first response representative has seen, or will see, the same emergency crews and police officers in the future. The best approach is to work as closely as possible with all of the fellow responders to accumulate information as early as possible through the cooperation of others. For instance, the fire marshal and/or police will most likely identify some eyewitnesses before the utility responder arrives. The official emergency responders will also have firsthand knowledge of the conditions observed upon their arrival, before the call was made to the utility for assistance. The media listen to emergency dispatch calls and run to those that would be of interest to their audiences. They can be a nuisance or a benefit. If the story is that the utility equipment was involved, the utility responder can expect to see a microphone placed in front of them upon arrival, and numerous probing questions. There is no obligation for the utility responder to answer any questions and may politely inform the media no comment, and that a designated company representative will answer their questions. It is interesting to note that the press is very anxious to collect some of the same information as the root cause/forensic investigator in the utility truck. They are also armed with high-quality photographic cameras and broadcast quality video cameras. In addition, they may have arrived before the utility emergency response crews! This is a potential benefit to the investigation that follows. Be sure to make a note of the TV station or newspaper the media represent, and observe what they are documenting. It may include video scenes of the accident or failed equipment and interviews with the emergency response officers and witnesses. If the matter is a serious incident that has the potential to develop into a lawsuit, advise the law department of your observations and suggest that a copy of their material be obtained. In some cases, the media will provide at least a viewing. The next edition of the paper or the evening newscast will be worth viewing and recording as well. Thus, a good deal of information can be obtained from the incident scene by being a good observer and a good listener. That attitude is the hallmark of a good forensic investigator. (See Media Material, Section 13.14.) In addition to the emergency and police reports, there may be an Occupational Safety and Health Administration (OSHA) report if a worker was involved. This document is available through the state agency that administers the federal regulations in the jurisdiction where the accident 7-5

Data Collection

occurred. The purpose of their investigation and report is to determine if the employer caused or contributed to the accident and violated any OSHA regulations. They are required by OSHA to conduct an investigation whenever a worker is injured on the job causing a loss-of-time injury. Therefore, the OSHA investigation is specifically interested in the safety issues surrounding the accident, as they relate to safety training, personal protective gear, safety procedures and equipment available to the worker. A comprehensive review of the accident as it relates to these issues will be described in the final OSHA report. If violations to OSHA regulations are found, these will be noted. Usually, these reports are informative and may provide additional facts about the workers activity at the time of the accident, and the resulting injuries. If injuries or deaths have occurred as a result of the accident or failure, medical records will be generated that may be of importance in linking the incident to the injuries. Medical data becomes very important in electrocution accidents where the contact points and degree of injury provide information to reconstruct the electrical circuit created with the victims body, and to determine the relative contact fault current present. (See Official Reports, Section 13.15.)

7-6

8
WITNESS INTERVIEWS
Eyewitnesses are very important to the forensic investigation process as they have the ability to provide information about the sequence and timing of the incident scenario. In addition, eyewitnesses can observe the movement and actions of people or dynamic objects, and the sounds, light and odors given off prior to and during an incident. However, the good news ends there, as human beings also utilize their vast experience and knowledge to analyze and filter everything they see, hear, or smell. In addition, no two humans have the same tuning of their senses, and seldom have the same impression of an incident. This is why eyewitnesses are not entirely reliable sources of information, and further explains the common experience of forensic investigators, that you can expect as many versions of an accident as the number of eyewitnesses you interview. An eyewitness may have been a participant in the accident, or because of what they saw may be in shock and suffering trauma from their injuries. They are expecting to be interviewed, but are likely to be defensive about their role in the accident, and may become emotional. Unless the injured participant eyewitness is a utility employee, access for an interview may be controlled by either their attending physician or their attorney. It is, therefore, very important to notify the law department when there are injuries so that an interview of the non-employee eyewitness can be obtained as quickly as possible. Time allows memory to fade and to be influenced by others. This phenomenon is often demonstrated by the striking inconsistency between a statement taken within hours of the accident and a statement or deposition taken from the same witness months later, after a lawsuit has been filed against the utility.

8.1

Early Identification of Individuals

How do you find witnesses? A good place to start is with the emergency responders, particularly the fire and EMT crews, and the law enforcement officers. Eyewitnesses may still be on the site if an investigator arrives soon enough. If the media are involved and arrived before the scene was cleared, they may also be of assistance in locating witnesses. One witness can lead to another. Be sure to ask them who else may have seen the accident. It is not uncommon for witnesses to congregate at the scene and discuss their observations and ideas as to what happened with each other. Check police reports with witness statements and media accounts later to obtain the names and statements of eyewitnesses not identified earlier.

8.2

Standard Procedures for Conducting a Forensic Interview

An interview of a witness can be a very casual procedure where a conversation takes place over lunch between at least two people, where one is seeking information from the other as to their recollections of, and involvement in, an incident. No notes are taken during this social interview, 8-1

Witness Interviews

but may be reconstructed later from memory by the interviewer. Although a pleasant experience for both, the amount of information collected will be very limited and highly subject to inaccuracies. Asking questions and simultaneously taking down the answer in writing is more difficult and no more accurate. In some cases, the witness may be willing to write a brief one or two page statement in response to a few specific questions, and then sign and date each page. Although this is an acceptable informal statement for the courts, it is the most tedious and least informative procedure to use for interviewing a witness, and may limit the recall of any additional information from memory. By far, the most convenient and productive interviewing procedure is to use a tape recorder. Some witnesses will work well with maps or scale plan drawings and volunteer more information than you would expect them to know. Others will be intimidated and not be able to orient their witness position and hesitate to make a mark on the paper. The interview process will reveal these traits that can assist in the analysis of witness evidence. Figure 8-1 shown below is an example of an incident site plan drawing used to interview witnesses. Witnesses are requested to mark the drawing in response to a series of set questions. They are then asked to sign and date the drawing with their notes that becomes a part of the interview evidence file. Although it is best to leave the interview of witnesses to trained specialists, it is very acceptable for the utility First Responder to ask a bystander witness, What happened? and What did you see?

8.3

Utilization of Information

Eyewitness information can be utilized in many ways to help complete the reconstruction of an accident. In fact, interviewing witnesses could be the most important part of a forensic investigation. In most cases eyewitness can provide answers to the questions: What happened? What did you see? Where were you? What did you do? What did others do? A successful series of eyewitness interviews can provide much of the accident dynamics information and unrecorded transient effects that are otherwise not available from any other source. These answers provide the action links and relative timing information for the Incident Timeline and provide data to consider in constructing analytical diagrams, such as used in Fault Tree Analysis (FTA). A final word of advice is worth repeating; verify eyewitness information against other facts and evidence. In the end, there can be no significant inconsistencies in the eyewitness evidence supporting the root cause scenario of an incident. If there are any significant inconsistencies, the forensic investigation is not complete.

8-2

Witness Interviews

Figure 8-1 A Substation Site Plan

8-3

9
FORENSIC-ROOT CAUSE ANALYSIS
When the investigation reaches the analysis stage of the incident, the direction changes from collecting data and establishing facts to determining how all of the pieces of the puzzle fit together. If the evidence-gathering process has been carefully followed, the analysis will be less of a hassle since all materials are well organized and properly documented. The investigation has produced a mixture of facts, some with high reliability and others that remain questionable. Inevitably, there will be some facts that do not exactly support conclusions, as the final picture puzzle is not always a perfect fit for each piece. This means that the conclusions as to the root cause may need to consider other possible alternatives, and that an opinion of absolute certainty as to root cause may not be possible. Sorting out the facts in this manner is the evaluation of information about causal factors. When they are brought together in a logical sequence, the causal factors will accurately describe how an incident occurred. This is often referred to as a root cause analysis (RCA). There are a number of different analytical techniques used by forensic and accident investigators. Analytical techniques have been developed to provide a variety of responses, not all useful to every investigation being conducted. Whatever the technique, if at the end of the analysis the conclusions cannot be visualized and described in detail consistent with the facts through the incident sequence, then the analysis is not complete. This is the reality test to be applied throughout the analysis process with vigor and objectivity to challenge the final proposed conclusions and probable root cause. RCA is one such investigative technique that is applied to reveal potential initiating events that could ultimately lead to a final incident. RCA depends on the investigator to propose as many origins of an incident, within reason, and to then evaluate each against the available evidence to determine the single most likely root cause, if possible. In general, the best way to find a longterm solution to solve any problem is to first identify the causes, or roots of the problem, and then to completely remove them to protect against their return.

9.1

Fault Tree Analysis (FTA)

FTA is a logic-based technique using deductive reasoning to force the consideration of all conceivable events in a failure sequence and their cause and effect relationships. In addition, FTA also identifies the root causes that have initiated an incident sequence. The FTA process begins with the incident event, or a portion of it, and works in reverse through other known or reasoned events to the root causes. FTA uses a tree-shaped flow chart diagram incorporating computer logic symbols. A disadvantage of FTA is that the diagrams can become very large and spread over several sheets. 9-1

Forensic-Root Cause Analysis

The most common FTA symbols are shown below in Figure 9-1. They include a rectangle to represent an event that can be analyzed further, and is always used to indicate the single accident event located at the top of the tree. The tree continues downward and back in time through layers of precipitating events. As the number of events increase, the FTA diagram begins to look like a tree with the root causes placed at the bottom or at the lower end of each tree branch. The diamond shape is used to represent an improbable condition, or an event that either cannot be analyzed because of a lack of information, or cannot be used because discovery of specific evidence during the investigation eliminates it. The circle is used to represent a probable condition or an event that cannot be analyzed further, and is, therefore, the root cause of that branch of the tree. The AND gate logic symbol is placed between levels of the tree when all the events immediately beneath the gate must happen first in order for the event above it to occur. The OR gate logic symbol is used where only a single event must occur to make the next higher event happen.

Figure 9-1 The Most Common FTA Symbols

9-2

Forensic-Root Cause Analysis

Figure 9-2 FTA Example of a Circuit Breaker Failure to Open

An example of a fault tree constructed to analyze the failure of a circuit breaker system to open is shown above in Figure 9-2. A substation circuit breaker failed to open under fault conditions on a dedicated feeder resulting in serious damages to the customers switchgear and a substantial interruption of power. The FTA resulted in construction of the fault tree that depicts the top event as the circuit breaker failure to open due to a failure of the circuit breaker mechanism in the closed position, voltage remaining across the circuit breaker trip coil, or faulty control wiring thereby holding the circuit breaker closed. Control relays A and B are in series with the circuit breaker trip coil, and, therefore, must both have their contacts closed to hold the circuit breaker closed. Two conditions can cause the A and B control relay contacts to be held closed. First, the respective control circuits may have failed in the closed state, or second, their respective relay mechanisms failed in a closed position. Thus, the probable conditions of failure have been determined to be a failure of the circuit breaker mechanism, faulty control wiring, or failure of both relays A and B in the closed position. Further analysis of each probable condition should lead to a determination of the most probable initial event or the root cause of the failure. From the FTA diagram, the causal links and overall sequence of effects can be visualized which also helps to provide a thorough understanding of the incident. However, the value of this technique depends heavily on the quality and validity of the data inputs as well as the experience and skill of the user. FTA is a good analytical tool to graphically order the causes and effects of 9-3

Forensic-Root Cause Analysis

a mishap, and provide direction for collecting supporting information. (See Bibliography in Chapter 16 for further instruction.)

9.2

Evaluation and Decision Procedure

A forensic analysis must consider all of the evidence available from the investigation, including: Pre-incident data and information; Incident data and witness observations; Incident site documentation and evidence; Equipment tests and operations records; Reconstruction of the sequence of events; and Logic of the incident scenario, and the influence of all external factors.

The evaluation and decision process utilizes the scientific method to test theories and concepts as to cause against the available evidence and the laws of science. Methods such as RCA and FTA, or similar techniques, are used to aid in organizing the thought process to establish the most probable cause for any mishap, accident, or equipment failure. In the end, the facts and evidence must be consistent. They must be able to support a conclusion as to the probable cause of the incident under investigation. If legal action results, each conclusion will become an opinion that must be properly supported by admissible facts and evidence.

9.3

Witness Evidence

The human brain is the most sensitive living event recorder and storage device, but it comes with a data assessment and interpretation mechanism on the output of an unreliable storage device. However, there is little other evidence as important as good eyewitness accounts of an incident. Witness evidence is primarily obtained from interviews or reports containing firsthand information as to the incident details. It is very important to consider all of the witness statements with an open mind. Select the most frequently consistent answers to the basic list of questions asked of each witness. Develop a ranking of the most to the least consistent or lowconfidence answers, and consider any reasons for the low-ranking answers. When all witness statements and marked drawings are complete, the information provided should be placed in a time sequence based on standard clock time in coordination with known timed events. In most cases more than one time scale is required for the timeline of events. It is common practice to use at least one long period span for an historical timeline and one short period span for the incident reconstruction timeline. Always remember the inherent weaknesses of human recollection, and avoid depending on one witness statement to support a conclusion. Above all, incident witnesses, utility response personnel, and forensic investigators are all humans who possess an extra sensecommon sense. Use it liberally when considering the accuracy of witness statements.

9-4

Forensic-Root Cause Analysis

9.4

Physical Evidence

Physical evidence, such as enclosures, tools, and construction materials, should be collected by the investigators and inspected thoroughly without alteration. Selected physical evidence may be tested and carefully dissected under a memorandum of understanding (MOU) in conjunction with a joint inspection protocol presented to all interested parties. The collection of evidence includes identical or very similar items for comparison purposes. Comparison of physical evidence can be particularly useful when attempting to reconstruct the failed object to its original configuration and understand the sequence of damages received or produced by electrical, mechanical, or thermal energy. In some cases, it is important to request assistance from materials engineers or metallurgists when evaluating damages to physical evidence in order to understand the particular conditions required to cause the damages. The results of the physical evidence analysis will provide more pieces of the puzzle. They must be placed in the master timeline of cause and effect events to be tested for consistency with all other related facts.

9.5

Electrical Evidence

Electrical evidence consists of equipment and installation information, maintenance records and test data, along with records and data recordings from operations. Review and analysis of electrical evidence requires the leadership of an experienced investigative engineer. He or she can evaluate the historical evidence of the system or equipment, as well as reconstruct the electrical parameters of the incident. This is the core expertise required in all electric utility failure analyses. Frequently, electrical engineering specialists are required to join the lead investigating engineer in utility matters that involve particular facilities, equipment types, protection systems, and various operations. Be certain that all the required electrical technology expertise has been tapped for the analysis. Always include representatives from the operating and maintenance field crews where special knowledge is available based on firsthand experience that can be essential in relating to the real world. A forensic electrical engineer supported by a dedicated team of active distribution engineering and operations personnel are likely to solve what caused most accidents or failures, and at the same time, prepare to successfully handle any litigation that may result. After all, no one knows the system better than those who operate and maintain it 24/7, and no one knows the forensic investigative process of utility accidents better than those who have established themselves as experts in the profession.

9.6

Mechanical Evidence

Selected parts and operating mechanisms are included in the mechanical evidence as well as the damages to the electro-mechanical devices found in controls, switchgear, motors, pumps, and blowers. A mechanical engineer will usually be required to evaluate mechanical systems for electric power equipment installations. They will be called upon to evaluate substation ventilation systems, pumps for equipment cooling, and tap changer mechanisms in transformers that may have been involved in a failure. Other small but significant items, such as failed trip lever mechanisms, linkages, and containment seals, may need to be analyzed to determine their role as a potential root cause or contributing factor.

9-5

Forensic-Root Cause Analysis

9.7

Other Evidence

Other evidence includes industry codes and standards that play a very important part in the assessment of responsibility. Failure to meet or comply with such standards, as in clearances for overhead lines found in the NESC or related company guides, can provide a cause for damages compensation and even costly punitive damage awards. Thus, the analysis must include a comprehensive review of all applicable codes and standards related to the incident, all applicable conditions found at the accident site, and the installed protection equipment. If there is an issue of code non-conformance, it must be fully addressed as soon as possible. On the issue of accidents involving contact with electrical lines and clearances, the lines involved, whether overhead or underground, must be carefully and accurately measured before any adjustment is made to determine their compliance with the applicable codes. At a minimum, an overhead line must be measured from its lowest point on the span to the ground directly below, from the point of accidental contact to any adjacent structures and to the ground, from the pole attachment points at each end to the ground, and over the span distance between the two pole attachment points. The air temperature at the time measurements are taken must also be recorded with an accurate thermometer since they may not be the same as when the accident occurred. Determine the electrical load on the circuit, wind speed and direction to the line, and/or the temperature of the conductor. Each of these values has an influence on the sag of the line. Although these line measurements can be taken with a surveyors tape, it is more accurate to have the accident site officially surveyed with a laser transit that can easily include ground contours under the line and nearby structures that are also difficult to measure by hand. An independent survey of this kind may be worth the modest expense. External influences are often key factors in the accident determination, often having a direct effect on the equipment reliability through gradual degradation, and in some cases causing sudden failures and interruption of service. This is the area of causation that involves customer equipment, the environment, wildlife, both employees and contractors of the utility, and members of the general public. A wide range of issues need to be analyzed regarding climatic data, storm damage, natural and industry contamination, customer accidents and equipment failures, animal contact, human contact, vehicle impact, voltage surges, brown outs, and black outs.

9.8

Post-Event Testing

Testing of any evidence should be conducted with the utmost care so as not to alter the evidence in any way. Also, bear in mind that the equipment to be tested may have experienced damaging electrical and/or environmental conditions before its failure as well as at the time of the incident. All testing must be conducted after establishing a written understanding between all involved parties with an agreed upon joint test protocol for any contemplated destructive testing or alteration activity, beyond a hands-off visual inspection. When at all possible, the test protocol should be consistent with, or based on, a published test standard and conducted by trained personnel with calibrated test equipment. If a test cannot be conducted properly on the first attempt, it may be better to wait until it can be.

9-6

Forensic-Root Cause Analysis

9.8.1 On-Site Testing On-site testing must be carefully planned and conducted to avoid any additional damages or injuries. Any proposed testing of the operating system for investigative purposes must be reviewed and approved in advance by all responsible utility department managers involved, and conducted by authorized utility personnel. In some cases, on-site testing is required to evaluate the electrical condition of installed equipment. On-site testing may be applicable for relay protection and circuit breaker operations, TDR cable fault locators, infrared thermal imaging, ground connection and earth resistance measurements, and perhaps dielectric withstand testing, to name a few. Again, any site testing must be conducted safely and according to standard methods with calibrated equipment. The forensic process requires that the activity of testing be documented and the resulting data be considered evidence. Consideration should be given to notification of all parties if testing involves equipment from the incident. 9.8.2 Laboratory Testing Evidence that has been secured from the distribution system equipment related to the accident may hold the answers to important questions. Laboratory testing can address many variable issues such as: the quality of insulation material in a transformer winding; or the cause of the fracture of a step contact shaft in a tap changer; or the cause of repeated failures of lead-sheathed cable splices; or the identification of debris locked in a molten mass of aluminum that filled a cable splicing pit; or porcelain bushing failures. These issues, and numerous other questions, have been addressed by laboratory testing of evidence samples collected from failed distribution system equipment, and by testing for foreign materials that have found their way into the system. An experienced and certified laboratory can provide access to many technologies, including quality assurance electrical test and materials analysis equipment. With strict laboratory protocols and systems in place, the results can be relied upon and certified, if required. The same specific protocols apply here as mentioned previously using an MOU, where all parties must be notified and allowed to participate in any alteration or testing of the evidence of interest.

9.9

Timeline of Events

In addition to the FTA method of cause and effect analysis presented to link the events and conditions related to the sequence of events, the event evidence must also pass the test of time; not for durability, but for consistency of sequence with time. The sequence of events must fit the known timed events and the overall duration from the first root cause event to the final failure or moment of the incident. As with tests of other causation theories, any significant time discrepancies regarding events or actions must be explained or the investigation is not complete. A listing of clock times and data inputs can then be inserted in chronological order or by a more visual means to display the data that has been developed, known as a Timeline of Events. Figure 9-3 shown below is an example of an actual Timeline of Events taken from a forensic investigative report presented for public distribution.

9-7

Forensic-Root Cause Analysis

Figure 9-3 Timeline of Events for a Substation Sequence of Failures

9.10 Final Review

When the analysis is complete, and the answers to the key questions have been determined, the lead forensic investigator should submit their work for peer review before formal documentation begins. This assures that the work is complete, accurate and consistent within the bounds of good science and engineering, and that the results of the analysis support the conclusions being proposed. Regardless of the level of importance involved in the root cause analysis being conducted, it is good practice to step back and take the devils advocate role, and vigorously challenge the findings and conclusions reached before presenting the results or submitting the report.

9.11 Conclusions
Developing the conclusions to a forensic investigation requires that they be fully supported by evidence that can be used in a court to defend each conclusion. Review each conclusion and list the supporting evidence as if responding to a challenge. Because most readers of forensic investigation reports go right to the end to find out who did it, the conclusion section should be presented following a short statement of authenticity and degree of certainty. A logical sequence of statements prepared for a non-technical audience, but very understandable, should be listed to ensure a clear relationship to the evidence presented in the preceding sections of the forensic 9-8

Forensic-Root Cause Analysis

report. While a root cause analysis report may simply present the identified originating events as the conclusion, the conclusions presented in a forensic investigation report may be the opinions of an expert witness to testify on issues of liability. Both of these reports present the opportunity to take the lessons learned and feed the knowledge gained back into the distribution system reliability equation program as recommendations.

9.12 Recommendations as Appropriate

The most valuable benefit from an incident investigation conducted under the Forensic Guidelines presented here is to improve safety and reliability based on having a clear understanding of what actually happened and how it all began. Opportunities for improvement come from the lessons learned and should be as important to look for as the root cause. Recommendations to make these improvements are welcome at any time, the earlier the better. Utility staff and investigation teams are encouraged to notify management of these opportunities when confirmed. In some instances, improvements can be implemented before the investigation is completed. Once the root cause is known, steps can be taken to reduce or eliminate the opportunity for similar accidents or equipment failures. The benefits to improved safety and reliability resulting from a forensic-root cause analysis can be substantial. When a request is made to provide recommendations, a verbal response is best initially. Meet at the incident site and discuss the specifics while referring to the equipment or system involved. This is suggested since experience has shown that written recommendations, when discovered in litigation, can be misrepresented by others as an admission of responsibility on the part of the utility. Therefore, it is best to first discuss the proposed recommendations before presenting them in writing, especially where the incident has or could become the subject of a lawsuit. In addition, recommendations must be appropriate and practical for the utility to implement. Hearings involving public utility commissions and regulators may review high-impact power outages and serious incidents when considering issues of service reliability and the cost to customers. Therefore, written recommendations from reports on these incidents should be of the type that can be implemented in a timely fashion, and be cost effective.

9-9

10
REPORTS AND PRESENTATIONS
10.1 Reports
It is important to create a record of all the investigation work accomplished in resolving each incident. This becomes the permanent file for all documents including, but not limited to, data from various sources, including test results, photographs and videos, notes, response and equipment reports, witness statements, official reports, press materials, internal memos and correspondence, and logbooks of evidence. Other than the required paperwork to document activities, a report may be requested. Reports can take the form of a verbal presentation and/or a detailed written document. An incident report form may exist that fulfills this request. Regardless of the report form, it is very important to be accurate as to all references to facts and data in the permanent file. The report must correctly reflect the work conducted during the investigation and all conclusions must be supported by reliable and reproducible information. A factual report for the permanent file provides closure to the investigation and offers valuable feedback to a related reliability improvement program. Until requested specifically by a corporate representative, no written reports should be produced containing personal opinions or comments as to direct responsibility for the cause of the incident or its consequences. Always confirm the request for a report prior to beginning the first draft. As the final draft version of any written report is produced, eliminate or delete all previous drafts in any form and in all locations. Have a second person of similar background as the author review the final draft report for accuracy and completeness to provide quality control and avoid serious challenges later by others. At the end of this stage, there should be only one final report dated and signed by the author. Distribution of the report is controlled by company policies and in some cases the legal process. All reports should be considered company confidential and in some instances may become evidence in litigation. A logical format should be used to present the results of a forensic-root cause investigation in a written report. One format that has been used successfully is: 1.0 2.0 Introduction The accident or failure is introduced and the questions to be addressed are stated. Background A description of the facts concerning the pre-accident conditions, the location, summary of known accident events, and the results of the mishap are presented. Investigation All activities and research conducted during the investigation are discussed in 10-1

3.0

Reports and Presentations

order of performance along with a complete listing of all documents reviewed from the discovery process are listed. 4.0 Analysis A discussion of all test results and analytical techniques are discussed with the resulting findings from the application of the scientific method and common sense. Conclusions The answers to the questions, presented in section 1.0 Introduction, are presented in a logical sequence as opinions of the author(s). Recommendations This section, as appropriate, extends the knowledge of causation from the conclusions to practical steps to improve reliability and/or reduce accidents.

5.0

6.0

10.2 Presentations
Presentations of root cause analysis or forensic investigations could incorporate visuals and outlined information to be used in a computer package, such as PowerPoint, and in some matters animated reconstruction computer simulation. Use photographs and video segments to orient the audience, and bring the actual evidence and exemplars to enable the audience to examine the items for themselves and to hold the actual evidence recovered during the investigation. The investigator/presenter will be demonstrating their knowledge and thoroughness while gaining the audiences respect for helping them to understand a complex technical matter allowing the utility management, commissioners or the jury to render a proper decision, ruling, or verdict. And that is what it is all about; the forensic investigative process is designed to enable decisions of resolution to be made as accurately as possible in the face of challenging positions.

10-2

11
CASE STUDIES
The following case studies have been selected from areas of overhead and underground failures, fires, and electric shock incidents or Media Grabber attention level cases. A step-by-step outline of the investigation process has been applied to each case study as follows: A. B. C. D. Data Collection by First Responder; Data Collected by Site Investigation; Root Cause Analysis; and Report and Recommendations.

11.1 Distribution Transformer

This particular case involved frequent failures of submersible single-phase distribution transformers on a system where the vaults were subject to tidal flooding on a daily basis. It seemed obvious that the problem was water ingress into the transformers and that the probable entry was through a vertical rotating shaft that controlled a switch in series with an internal fuse. A. System maps were available; however, there appeared no correlation between the equipment configuration and the failures. The failed transformers were tagged and saved. Site investigation only confirmed that the transformers were subjected to tidal flooding. Four of the failed transformers were sent to an independent laboratory for analysis. In this case the utility was prudent to have saved the evidence because while three of the four failed because of internal moisture the point of ingress was different for each and in no case was the vertical shaft involved. The fourth transformer failed due to an overheated compression lug inside the transformer. At the conclusion of the analysis two key recommendations were made for two different manufacturers to improve future performance. The lessons learned were: dont jump to conclusions; save the evidence; the failed part; and when necessary, hire expert help, the laboratory and its engineers in this case.

B. C.

D.

11.2 Load-Break Disconnect Switch

A. In this case, a sudden rash of failures verified by Unacceptable Reliability Reports lead to calling in the manufacturer of a Load-Break Disconnect Switch for an analysis. The proper function depended on the operation of a parallel arc contact, which broke apart after the main contacts had separated. The arc was extinguished by drawing it through 11-1

Case Studies

two plates of ablative material, which created pressure to blow the arc out. One or two isolated failures would probably have gone unnoticed if a record of failures had not been kept. B. C. The switches were tagged and returned to the manufacturer. The manufacturer had the ablative material analyzed and found that the material supplier had changed the formulation so the molded parts would be easier to remove from the mold. No notice was given to the manufacturer. The recommendation was to return to the original ablative material formulation. The lesson learned was that the Unacceptable Reliability Report is a valuable tool in maintaining quality and performance.

D.

11.3 Department Store Feeder Failure

A. A 208 V cable failure occurred with two separate feeds. Limiters were not installed. One feed failed, and an explosion occurred. On a pleasant weekday afternoon, the street facing display windows of a multi-story ladies clothing store exploded outward and upward, showering pedestrians with glass fragments and tumbling many to the ground. Mannequins went rolling onto the sidewalk. Almost simultaneously, two electric cable manhole covers blew high in the air, one coming down edge-wise through the roof of a parked car. No one was seriously hurt, but from across the street, it looked like a disaster. Police, fire and medical emergency teams poured into the scene. It was a media event! There was a strong smell of gaseous fumes in the area, especially in the clothing store, which was without electric power. Adjoining stores and businesses reported flickering lights and gaseous smells for several minutes prior to the explosion. All of the businesses in the block were served with natural gas for heating from a municipal gas company, a division of the city government, at the rear, and electricity at the front from a 120/208volt network operated by an investor-owned public utility company. Much contention developed between these two likely suspects, and little investigation was done. The city fire marshal undertook authority, cordoned off the store, sidewalk, and street, and called for an investigation by an independent forensic engineering firm. B. The team of one mechanical engineer and one electrical engineer arrived on site the next morning and found nothing changed, except that the odor of gaseous fumes had mostly dissipated. It soon developed that the natural gas system was intact and not involved. Investigation in the front of the clothing store basement was difficult because of blast damage to non-structural partitions and merchandise. Power was received through a metering current transformer cabinet mounted on the front wall just below sidewalk level and at the center of the building between the support structures for the two display windows. Secondary distribution wiring showed no damage, all circuit breakers were closed and all fuses intact.

11-2

Case Studies

The current transformer cabinet itself was almost destroyed, the cabinet bulged, doors blown off, and the upper part showing evidence of extreme heat. The current transformers were not structurally damaged; the upper one of the three showed heat damage. There were two, 3-phase supplies to the cabinet, entering at the top from apparently divergent sources. The two sources for each phase were joined together by a single-bolt connector to the top bar of each current transformer. Each of the two source cables fastened to the top transformer had about 18-inch lengths of bare conductor exposed in the cabinet. All insulation was burned off, and the copper conductors showed coloration of extreme heat. In one conductor, there was a melted gap of 4 inches of missing conductor. Copper remnants were found spattered and pooled at the bottom of the cabinet. There was no evidence of a fault having initiated within the cabinet. C. Inquiry of the power company representative confirmed that there were indeed two separate 208-volt feeds without limiters, one from each of the manholes involved. Working with electrical inspectors furnished by the fire marshals office, the forensic engineers were able to partially reconstruct the damaged partitions enough to show that explosive gases emitted by the cable insulation materials overheated to near the melting point of copper would have traveled readily to, and gathered in, the spaces below the flooring of the two sidewalk-facing display cases. The melting open and resulting arcing of the cable would have triggered the explosion. The evidence led the forensic engineer to believe that an electrical fault existed between the manhole and the store in one of the two supply cables, and that because of the active sources on both sides of the fault, it did not burn open as is relied upon widely in the operation of low-voltage network systems. Heat generated in the cables carrying high current in the duct is controlled by the close proximity of the other 2-phase conductors through phase flux cancellation, and by the intimate contact with earth. However, once a cable is separated from its phase companions, and immersed in still air of the cabinet, the fault-current-carrying conductor would reach much higher temperatures and cause its insulation to melt. For this reason, economy, and possibly other reasons, such dual feeds are generally avoided in the supply to low-voltage network customers. Neither conduit could be cleared for new supply cables; a complete new service would be required. The fire marshal ordered that the two damaged supply cables within their conduits be exhumed in six 10-foot sections, and taken to the fire marshals evidence storage yard for examination and safekeeping. Examination of the twelve conduit ends exposed during exhumation confirmed the forensic engineers opinion as to the probable cause of the explosion; a double-fed 120-volt, single-phase fault which could not selfextinguish, led to drastic overheating of the conductors and their insulation materials. The gases driven off from the insulating materials included (progressively with temperature) methane; ethane; ethylene; acetylene; and various aromatic products of decomposition of the various binders and fillers used in the formulation of the insulating materials. The arcing of the open conductor in the metering cabinet set off the initial blast in the store, and the resulting doubling of the current to the fault, through the other supply circuit set off the manhole explosion which had also filled with gas. D. The power company representative at the scene had apparently a pre-disposition to believe that this was just another case of that leaky, old, municipal gas system filling up a 11-3

Case Studies

basement with natural gas which could be ignited by any chance electrical spark. The significance of the dual feed did not occur to him, and he had made no examination of the metering cabinet. There was some justification for this feeling in that there was a noticeable presence of the distinctive natural gas odor in several locations in the rear alley above the route of the gas supply lines. The forensic team, qualified to investigate the two possibilities, solved the problem by starting at the remote ends and working toward the source. The building electrical circuits were clear, the meter was clear, the metering current transformers were clear, the supply circuits were in ruins, with one-phase connection melted apart and apparently a dual supply. A source of the explosive gases was found in the ruined cable insulation. This same source could explain this manhole explosion. An accessible gas storage volume and means of ignition had been identified. There was strong mistrust by the building manager and his insurance agent of both the electric and gas supply companies; they agreed readily to the suggestion to have the investigation done by an independent forensic engineer. By his failure to note the possible effects of an irregularity in the supply arrangement and his bias toward blaming the gas supply company for any gas-fed explosions; the power company representative caused a significant delay in the restoration of electric service, embarrassment to his employer, and enlargement of the overall costs of the incident. The additional lesson learned was that severely overheated insulation produces explosive gases. This is a point that seems common knowledge now; however, only in the recent years.

11.4 Cable Failure (URD)

A backhoe trenching for the installation of a new sewer line dug into a section of the URD cable at the far end of a 4kV residential and small commercial distribution feeder. Some seven hours later and a half-mile away, a man returning home crossed a busy street where the traffic signal was inoperative and was seriously injured by an automobile. Was there a responsible connection? Attorneys for the injured man brought suit against the backhoe operating company, the underground cable marking company, and of course, the power company which was charged with taking too long to restore service to the traffic signal, thus causing the mans injuries. The suits threatened large-scale damages for injury, pain, and loss of future income. The electric power company found it expedient to engage a forensic engineer to analyze the companys actions and decisions regarding restoration of service to the many customers served by this underground distribution feeder, and to give unbiased opinions as to their conformation with good practice. A. More than a year had passed since the occurrence of the incident, but the power company had maintained good records of dispatch logs, troubleshooters daily call sheets, and personnel assignments. The forensic engineer interviewed the personnel involved, invoking their memories of when which actions occurred, and how decisions were made. Operating procedures were reviewed and safety considerations reviewed. Anticipating a jury trial, an educational paper was prepared describing the underground distribution system and its manner of

B.

11-4

Case Studies

operation, and the safety precautions, which must be followed with precision to avoid personal injury. A map of the area served by the feeder was prepared and overlaid with information regarding location of sectionalizing devices connections to other feeders, grounding points, routing of cables, and kVA size of connected distribution transformers. A timeline was developed from the paper record giving down-to-the-minute description of the progress being made in sectionalizing, location, making safe for work, and restoration of service. At each location requiring entry of a man into the underground system, the practical hazards such as: setting up a safe work place in the middle of a busy street with heavy traffic; removal of large quantities of water from manholes; testing for breathable atmosphere; and the limited workspace available was analyzed. C. The analysis revealed that the limiting factor preventing the restoration of electric service to the cable section that served the traffic signal was the inability to remove the damage done from a collapsed section of duct line. Since the rebuilding of the duct line would require nearly a full days work, the decision was made in late afternoon to insulate and cap the cable at the final manhole, thus allowing upon completion, the return to service of those customers on the final leg of the feeder. This included the traffic signal at the intersection where the man was injured. The successful presentation of the above material was convincing of the power companys well-run service restoration program. Its preparation was entirely dependent upon the fortunate daily attention to filling in the blanks of the routine forms of dispatch and assignment. With detailed knowledge of work practices, safety procedures, individual workers skills, travel distances involved, and manner and time of acquisition of specific knowledge, the forensic engineer was able to forestall all damaging questions by the opposing attorney. The lesson learned was to make and keep good records.

D.

11.5 Secondary Arcing in a Trench with a Co-Located Gas Line

An insulated cable with a solid aluminum conductor is a common way to get secondary power (i.e. 110/220 volts) from the street into the home in a trench. It is also the practice of some utilities to co-locate other utilities in a common trench, including gas lines with specifications as to the proper separation. During installation of these cables, nicks and gouges can occur in the insulation. As the voltage is very low, the circuit rarely trips out due to a short circuit to ground. However, moisture can get into the cut and react with the aluminum conductor. Over time, the circuit fails due to an open circuit, as the aluminum conductor has corroded into aluminum oxides. It was one utilitys practice to hook up an autotransformer to provide temporary service, as a quick fix in the middle of the night. A. On returning the next day to make repairs, the crews would find the open circuit caused by the corrosion of the aluminum conductor. Occasionally the gas and electric lines would be too close (actually touching). What was not realized was that arcing which occurred either during the failure (going to open circuit) or after re-energizing had burned through the gas line. In two known cases there was not an explosion during the burn through but later.

11-5

Case Studies

B.

The reason for the explosion and the loss of two houses is not as important as the question of why was there sustained arcing at 110 volts? It was not thought possible. Fortunately, no one was seriously injured. The utility hired an independent laboratory to determine under what conditions a 110-volt arc could be sustained. It took a number of trials and some experimentation to determine that a carbon (burned insulation) and water slurry could indeed sustain an arc discharge without blowing the primary fuse. A corrective program was put in place. The independent laboratory was an invaluable part of the process.

C.

D.

11.6 Improper Storage of Submarine Cable Samples

An anchor was dragged across a group of paper-insulated, lead-covered cables in a submarine installation. This caused them to fail with the result of very high repair costs. The repair operation consisted of cutting the cable back until dry paper insulation was found. (This is standard practice in such situations.) Samples of the good cable about 10 feet long were saved from the ends where splicing was underway. Instead of having the splicers put lead wipes over the ends of these samples to keep moisture from entering, they were wrapped in sheets of plastic and stored outside. The utility brought suit against the ship owner for damages that were about $40,000,000. The owners argued that the cables were ready to fail and so they should not have to pay for the repairs. When the samples were tested by a cable consulting firm, they reported that the insulation was wet and that they could not determine the future performance of the cable that was in service. The owners then said that it was the utilitys fault for not keeping the samples in a proper storage condition. The judge ruled prior to trial that this improper storage (spoliation) effectively reduced the amount of damages to $4,000,000. The utility was therefore unable to obtain compensation for $36,000,000 in damages because of their spoliation of evidence through the improper storage of critical samples. Although this is not the best example of the simplified A-B-C-D forensic process, it is being included as a critical example of how to keep damaged equipment or in this case evidence.

11-6

12
EXERCISE
A real-life investigation exercise scenario is presented as an expanded case example. It is followed by a choice of actions or decisions in response to multiple-choice questions for the reader to test their forensic investigation skills. The most appropriate responses for each exercise and a discussion as to why they were selected are provided.

12.1 Distribution Customer Switchgear Fault Resulting in Two Substation Breaker Operations
A. Summary A 15kV distribution cable fault, within the customers main service entrance switchgear, occurred early one morning. The fault caused a trip-out of the utility electrical feed to a commercial building. After the initial trip-out, the utility tried to re-energize the auxiliary electrical feed to the customer and that caused a second fault to occur in the same switchgear. No injuries were involved. A forensic investigation found the problem to be in the customers metal-enclosed switchgear and also pointed out some operational shortcomings. Incident A security agency responded to a fire alarm at a downtown building. They could see no obvious cause for the alarm. However, while investigating the cause, they heard a loud explosion in the switchgear room and saw a bright flash and then smoke. They called the fire department and reported an electrical fault situation, but that no fire had resulted. The building had two primary feeders and could be fed from either one. The incident tripped the normal supply feeder at the utility substation, but since they had not received a report from the fire department or from the building owner they assumed it was a problem in one of their underground lines and proceeded to locate the trouble. Utility personnel were dispatched to restore service to all the interrupted customers. The troubleshooter responding to this building had been instructed by system dispatch to transfer power to the alternate feed and proceeded to do so. When he made the transfer, another explosion occurred causing a substation breaker to trip. The customer was isolated from the utility system and grounds were applied so that the customers equipment could be inspected for the fault.

12-1

Exercise

B.

Investigation Witness Interviews The troubleshooter, security agents, firemen, building maintenance personnel, and operations supervisor were interviewed. Troubleshooter The troubleshooter had 17 years of experience and was wearing all required safety equipment, even though the switching was done in a room away from the faulted switchgear. Although he inspected the customer switchgear externally, since the fault was contained within the switch enclosure, he was not able to see the damage and proceeded as instructed. Security Agents The two security agents who arrived at the building could not see an obvious cause for the alarm and proceeded to inspect the fire alarm area panel. Due to building construction, security had responded several times in the past when the fire doors issued false alarms for the same area. They proceeded to check the panel, and just as they were leaving through the double steel doors (in less than 10 seconds), they heard a loud explosion behind them and saw a bright flash that appeared to them as a fireball. They also observed a large smoke cloud coming through the doors, which were blown-open by the blast. They then called the fire department. Fire Department The firemen reported that they responded to this no-fire event for public safety reasons. They also informed the investigators that the utility has an automatic ring-down procedure at the fire department headquarters; a direct phone line from the fire department to the utility that is to be used for all building fires. In this case, when the fire department determined that there was not an ongoing fire, the utility was not advised. Building Personnel The building maintenance personnel related that a contractor had inspected, tested, and cleaned the switchgear 7 months prior, and that the same contractor was called immediately to help with this inspection and make necessary repairs. They related that there had been no communication between the utility and others on the scene before the second failure, and one of the contractors electricians was coming downstairs when the second short occurred. They stated that there was the possibility of someone in the switchgear room at the time of the second fault and possibly being seriously injured. In further discussion with building personnel, they implied that the HV transfer switches associated with the building supply circuits were owned by, and the responsibility of, the utility.

12-2

Exercise

Operations Supervisor The operations supervisor verified that both 3-phase substation circuit breakers associated with the customers feeders had tripped because of faults that morning. He also stated that there had been no notification from the fire department of the incident at the subject building, and that the troubleshooter was dispatched to restore service to customers involved with the first trip. This was done since all information indicated a utility cable problem. Inspection of Switchgear An operation specialist arrived four hours after the occurrence to investigate the incident. He verified that adequate grounds were in place and that the area was safe for inspection. After speaking with the contractor foreman and determining that there would be no interference with the repair process, the primary distribution switchgear and the main service entrance switchgear in the basement of the building were inspected. It was obvious that the fault occurred in the main service entrance switchgear that resulted in a flashover from the ends of all three main bus bars to the grounded case of the switchgear. The distance from one phase of an incoming 15 kV cable appeared to have insufficient clearance to the bare copper bus bars. There was evidence of a fault from the cable jacket of this cable to the A-Phase bus bar. Detailed notes were taken as well as numerous photographs. Since all of the materials were the property of the customer, all evidence remained on site. C. Conclusions First Fault: The initial installation of a cable, which was fed into the switchgear from above, slipped downward toward the switchgear main bus. The ground shield of one phase of the incoming cables faulted to one of the main switchgear bus bars. This tripped the normal feeder circuit breaker at the utility substation. The utility assumed that the reason for the first trip-out was at a location other than the building in question, and then tried to re-energize the switchgear from the other feeder causing the second fault. The fault between the cable ground shield and the bus was re-energized and flashed again. Contaminated arcing products in the switchgear, as a result of the first incident, caused additional flashover at the ends of all three main bus bars to the switchgear case. 12-3

Second Fault:

Exercise

Findings and Recommendations The damage to the customer switchgear as a result of the second fault was minimal. The utility was not responsible for the repairs to the customer switchgear. The fire department did not notify the utility of the problem at the building. The troubleshooter did not communicate adequately with personnel at the building before he attempted the transfer of feeders. The building owners were unaware that they were responsible for maintenance of the HV transfer switches.

D.

Report A detailed report was forwarded to the operations and distribution engineering departments. Benefits It was obvious and could be documented, that the utility was not liable for damages to the customer switchgear. At the request of the utility, modifications were made to the switchgear, which allowed for better placement of the incoming switchgear cables. These modifications will make isolation from the utility system easier in the case of future problems at this building. Changes will be made in the automatic ring-down procedure so that the utility will now be notified of every incident involving electric power problems, even when no fire is involved. All troubleshooters will be instructed as to their responsibility for the safety of customer personnel before performing switching operations.

Exercise Questions (Note - Select all answers that apply): 1. Even though the troubleshooter was originally sent by the dispatcher to only perform switching, after the second trip-out occurred and it was obvious the problem was in the customers switch, he should: A. B. C. D. Determine if anyone was injured. Obtain the names of contractors, security personnel and building representatives. Start an investigation. Just leave the scene and return to duty.

12-4

Exercise

2.

When the operations supervisor was notified that the trip-out was due to a problem in the customers switchgear, he should ask the troubleshooter: A. B. C. D. Was any utility equipment involved? Was anyone injured? Was the fire department called? What time did the incident occur?

3.

Even though the employee investigator specialist arrived four hours after the occurrence and work was ongoing to make repairs, he should: A. Verify that adequate grounds are in place and that the area is safe for inspection. B. Contact the contractor foreman, identifying oneself, and assure that the investigation will not interfere with the repair process. C. Inspect the incident faulted area and take lots of pictures. D. Determine if there was any utility equipment involved.

4.

After the investigator specialist determined that utility was not responsible for the repairs to the customer switchgear, he should: A. Forget about the whole thing. B. Just mention it casually to his boss the next day. C. Write down something on a slip of paper and put it in the glove compartment of the line truck in case someone asks. D. Make some detailed notes in case there is further investigation, and for application later in a potential written report.

5.

The investigator specialist interviewed the contractor foreman on the day of the incident. Who else should he interview? A. The troubleshooter, as to reveal any additional information about initial conditions. B. The fire department, as to why the Utility was not notified of a possible high voltage problem. C. The operations supervisor, as to why there was no notification by the fire department D. His supervisor, as to what to do next.

6.

The investigation report should contain: A. B. C. D. Date, time, and place of the incident. A detailed reason for the fault. The fact that no utility equipment was involved. Recommendations for additional action by the Utility.

Answers to the above exercise questions are as follows: 1. A, B and C are correct: Incidents resulting in an injury are of paramount importance to the utility. Any possible company responsibility for the injury should be reported to supervision as soon as practicable. The name of persons involved should be obtained for future investigation and it is important to immediately get as much firsthand information as possible. 12-5

Exercise

2.

A and B are correct: A supervisor should always be interested in Company equipment, and any injuries involving Company equipment.

3.

A, B, C & D are correct: Look out for your personal safety first. Always identify yourself and explain what you intend to do. Try not to interfere with the restoration of service activities. You cannot take too many pictures. Once you have established that no Company equipment was involved, your accountability to the incident is eased.

4.

D is correct: Although you may believe there is no liability to the Company, others may feel there is reason for legal action against the utility. Detailed notes can help to direct any further analysis and may provide help in providing a written report that may be required later.

5.

A, B and C are correct: As can be seen from the exercise, interviewing the troubleshooter, the fire department, and the supervisor resulted in changes made in the Automatic Ringdown procedure and troubleshooters will have additional instructions as to their responsibility for the safety of customer personnel before performing switching operations.

6.

A, B, C and D are correct: A report should always contain the time, date, and place of the incident. Often it is impossible to determine the reason for the fault. If not, list and discuss all possibilities being considered. If possible, describe in detail the cause, and list and include appropriate photographs. Recommend possible remedies and any action(s) necessary by the Utility that could improve reliability. It is always important to identify the ownership and responsibility for the equipment that was involved in an incident.

12-6

13
CHECKLISTS
These Forensic Investigation Guidelines have generated a number of listings to aid in carrying out various functions and actions. They have been collected in this section with the intention of providing a handy reference guide to be consulted as needed for specific forensic investigation functions and actions. A field version can be produced by copying the following pages, or the utility can set up its own report form.

13-1

Checklists

13.1 Incident First Response Report (IFRR)

****

****

(Indicate Utility Company Name on the Line Above) INCIDENT FIRST REPONSE REPORT Date: Vehicle Identification Supervisor Name: Report Number: IFRR: (Suggested format: Seq. No. Month/Year ) Time of Arrival:

First Responder Name:

Size up the Incident - Determine What Happened and the Present Status. Call supervisor immediately to advise. Check ALL applicable boxes and add notes if clarification is needed. INCLUDE FACTS ONLY. DO NOT INCLUDE PERSONAL REMARKS, COMMENTS, CRITICISMS, OR OPINIONS. Level Conditions Found I. II. III. Outage Only Non-Utility Damages Injuries or Death Outage with High Impact Significant Non-Utility Damages Negative Media Coverage Other (Unusual or Compound Conditions)

IV.

Overall Incident Scene Photographs Taken By: Date:

13-2

Checklists

Section 1. 1.1 1.2 1.3 1.4 1.5 1.5.1 1.5.2 1.6 1.7 Section 2. 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14 2.15 2.16 Section 3. 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14

Incident Data Date & Time Street Address & City Property Owner, if not Utility Coordinates Map(s) # One Lines Available, and Marked with Equipment Configuration at Time of Incident One Lines Not Available Operating Voltage Feeder or Service Circuit Number Description of Incident (Be sure to check all that apply and number sequence, if possible.) Power Outage Fire/Explosion Electric Shock Line Contact Downed Lines Dig-In Equipment Failure Weather-related Overload Human Action Vehicle Accident Trees/Vegetation Mischief/Vandalism Wildlife Other Incident Sequence of Events Description Photographs Taken Damaged Utility Equipment And Resolution Arrestor Bushing Capacitors Circuit Breaker/Recloser Connector Elbows/Joints Fuse Insulator Load Tap Changers Overhead Line Pole/Crossarm/Guy Wire Splices, Overhead Splices, Underground Switches Anticipated Repair/Replace

13-3

Checklists

3.15 3.16 3.17 3.17.1 3.17.2 3.17.3 3.18 3.19 3.20

Switchgear Termination Transformer Overhead Underground Surface Underground Cable Other Overall and Detailed Photographs Taken of Each Equipment Item Checked By: Date: Description of Restoration and Times Recorded Re-routed Power Turned Power Off Restored Original Circuits Other Initial Potential Causes and Contributing Factors (Check All That May Apply) Internal/Inherent Causes Defective materials or manufacturing Deterioration/Service Experience External/Non-Inherent Causes Weather Related Overload/Overheating Mechanical Causes/Damage During Installation After Installation Other Known External Causes Workmanship Maintenance Operations Unknown Cause Cause Not Determined/Not Examined Other Detailed Photographs taken of Visible Item(s) Checked Retention of Equipment Involved (Check All That Apply)

Section 4. 4.1 4.2 4.3 4.4 Section 5. 5.1 5.1.1 5.1.2 5.2 5.2.1 5.2.2 5.3 5.3.1 5.3.2 5.4 5.4.1 5.4.2 5.4.3 5.4.4 5.5 5.6 5.7 Section 6. 6.1 6.2 6.3 6.3.1 6.3.2 6.3.3 6.3.4

Record Each Item and Tag Number in a Log Permanent Evidence Tag Attached Protected & Stored Indoors Outdoors Location All Sides of Each Retained Item Photographed By: Date: See Chapter 5, INCIDENT INVESTIGATION FIRST RESPONSE.

13-4

Checklists

13.2 Forensic Investigators Field Kit

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. EPRI Forensic Guidebook Camera Kit Notebook with Cover Graph Paper Tape Recorder, Pocketsize Evidence Tags and Log Markers, Permanent Tape Measure, 100 feet, and 6-inch Ruler Flashlights, Small Pocket and Large Lantern Spare Batteries Zip-Locking Plastic Bags, Various Sizes Field Kit Bag for all Items

13.3 Acceptable Camera Kit

1. 2. 3. 4. 5. 6. 6.1 6.2 6.3 Compact Automatic 35mm camera, with a 28 to 80mm zoom lens, built-in flash, and a time/date data back or equivalent digital camera with at least 2-mega-pixel resolution Spare batteries for the camera; carry two as they may not be found easily A lens cleaning kit Color print film of high quality, 200 ASA 36 exposure, minimum of six rolls per day A small camera bag to protect the items listed and keeps them together A few words of advice are in order regarding the use of the acceptable camera, as follows: Do not use the panorama frame option if the camera is so equipped, as it only eliminates the upper and lower edge of a perfectly good 35mm negative frame and you end up with essentially a smaller image. Hold the camera very steady as you press the shutter release button down, as the automatic system requires time to set the camera to take the picture and any movement may blur the photograph. Use the widest lens setting to obtain the sharpest photographs, and stay within 5 to 25 feet of the object(s) of interest.

13.4 Minimum Camera Kit

Six Single-Use (disposable) 35mm Cameras with Flash and Standard Size Image Frame

13.5 Optimum Camera Kit

1. 2. 3. 4. 5. 35mm SLR Camera Body Normal 55mm Macro Lens with 1:1 Capability, and UV Filter Wide-Angle Lens - 28mm, with UV Filter Ring Flash for Macro Lens Standard Flash with an Extension Cord 13-5

Checklists

6. 7. 8. 9. 10. 11.

Portable and Clamp-on Tripod Shutter Cable Release with Lock Spare Batteries for Flash and Camera Color Print Film of High Quality (Kodak or Fuji), 35mm 200 ASA 36 Exposure, Minimum of Six Rolls per Day Lens Cleaning Kit Sturdy, Well Padded, Camera Bag or Hard Case, to Hold All Above with Room to Spare for a Second Camera Body and another Lens.

13-6

Checklists

13.6 Unacceptable Reliability Report (URR)

**** ****

(Indicate Utility Company Name on the Line Above) UNACCEPTABLE RELIABILITY REPORT Report No.: URR Check ALL applicable boxes and add notes if clarification is needed. INCLUDE FACTS ONLY. DO NOT INCLUDE PERSONAL REMARKS, COMMENTS, CRITICISMS, OR OPINIONS. Section 1. 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10 1.11 1.12 1.13 1.14 1.15 1.16 1.17 1.17.1 1.17.2 1.17.3 1.18 1.19 Section 2. 2.1 2.2 2.3 2.4 2.5 2.6 2.7 Equipment Identification Arrestor Bushing Capacitors Circuit Breaker/Recloser Connector Elbows/Joints Fuse Insulator Load Tap Changers Overhead Line Pole/Crossarm/Guy Wire Splices, Overhead Splices, Underground Switches Switchgear Termination Transformer Overhead Underground Surface Underground Cable Other Equipment Data Original Equipment Manufacturer Date of Manufacturer Model or Type Serial Number Installation Location Installation Date Company Inventory Number 13-7

Checklists

2.8 2.8.1 2.8.2 2.8.3 2.8.4 2.8.5 3. 3.1 3.2 3.3 3.4 3.4.1 3.4.2 3.4.3 3.5 3.6 4. 4.1 4.1.1 4.1.2 4.1.3 4.1.4 4.1.5 4.1.6

Nameplate Ratings Voltage Current Duty Insulation Class Other Source of Unacceptable Rating Retirement of Unit Request from Copy Attached Unsatisfactory Performance in Service, Copy of Performance Report Attached Unsatisfactory Condition Test Results, Copy of Test Report Attached Incident Driven Date Location IFRR No. OEM Recommendation Other Action Taken Place URR Tag on Equipment Indicating the Following: Held as Evidence Removed from Service - Discarded Retired in Place Removed from Service -To be Rebuilt Removed from Service - Returned to OEM Other

Section 5. Certification This is to certify that the equipment referenced in this document has been declared and tagged unacceptable for continued use for the reasons noted, and is no longer in service. Processed by: Printed Name Signature Verified by: Printed Name Signature Date Date

13-8

Checklists

13.7 Forensic Photography Tips

1. Use flash except when it is so bright you would need to wear sunglasses. 2. Take advantage of sunlight. Photograph the east side in the morning and the west side in the afternoon. Move items into sunlight and rotate for multiple views. 3. Avoid using artificial lighting (incandescent and fluorescent bulbs and tubes). 4. Take multiple photographs of damaged equipment both overall and close-up views of arc marks, charring, abrasion, etc. 5. Take sufficient photographs of the incident scene as if they will be used to tell the story of what occurred. 6. Use 36 exposure film and shoot all frames available on each roll. 7. When shooting multiple rolls of film for one project, number each roll in the sequence it was taken. 8. Create a photo log describing each frame on the numbered rolls of film. 9. Store all photographic negatives and prints in a safe dry location along with any other file materials. 10. It is better to take too many photographs than not enough - film is cheap!

13.8 Incident Site Photography

1. 2. 3. 4. 5. Wide angle views of incident scene at least four photographs from points of the compass showing extent of damages. Normal views of the utility equipment and/or external factors involved in the sequence of events, at least two photographs for each piece of equipment or external factor. Close-up views of damaged utility equipment and external factors (non-utility owned property), at least two photographs for each piece of equipment and area of damage. Normal and close-up views of visible significant clues or early facts to support initial classification selection. At least two photographs for each visible clue or fact. Normal views of each item or equipment retained as evidence. At least one photograph for each side of the evidence before it is removed and when it is placed in storage.

13.9 Evidence Identification, Removal, and Retention

1. 2. 3. Identify and list all utility-owned equipment involved in the incident, both damaged and undamaged. Tag and log each item identified. Photograph each item tagged while in place prior to removal or disconnection. 13-9

Checklists

4. 5. 6. 7.

Document the phase and circuit connections for each tagged item to be removed or separated from a support or equipment to enable reconstruction for analysis. Remove and/or disconnect each tagged item with minimum alteration or disassembly. Protect removed items from alteration or degradation using appropriate packing and wrapping before storing. Place all items in climate-controlled secure retention space with access limited to the evidence custodian or their representative will be required.

13.10 Power System Conditions Data

1. 2. 3. 4. 5. 6. 7. 8. Network Lines In-Service Bus Tie Position Transformer Grounding Loading Available Fault Current Number of Customers Involved in Outage Time Duration of Outage

13.11 Relevant Standards, Codes, and Procedures

IEEE NEMA ANSI NFPA CIGRE Other

13.12 Event Records, Targets, Alarms, and Other Recorded Data

1. 2. 3. 4. 5. 6. 7. 8. .9 10. 11. Digital Fault Recorder (DFR) Records - Local and Remote Locations Relays - Local and Remote Locations Targets Unfiltered Record for Digital Relays Sequence of Event Records (SER) Local and Remote Locations Alarms - Retrieve all alarms; Pre Event, Event and Post Event Time Stamp information Device Settings, Data and Drawings Substation Recording Devices, Paper Charts and Tape Substation Logbook Communications Log

13-10

Checklists

13.13 Digital Fault Recorder (DFR) Data

1. 2. 3. 4. 5. Channel Names and Location Scaling (rms or peak) Sampling Frequencies and Number of Cycles Monitored Pre-fault, Fault and Post Fault Anti-Aliasing Filter Setting

13.14 Media Material

TV Newspaper Radio Internet Other

13.15 Official Reports

Police Fire Medical OSHA Other

13-11

14
SUMMARY
This Guidebook has described fundamental techniques emphasizing a formal forensic approach and a systematic procedure in evaluation of incidents; determining when an outside expert might be required; documentation; evidence collection; witness interviews; and fault tree analysis methods to enhance the root cause analysis by expanding the knowledge gained from all incident investigations, and to assist company personnel in preparation for possible litigation. The authors trust that the contents of the Forensic Guidebook will not only further the art, if not the science, in the root cause analysis investigation of electric power distribution accidents and failures that are inevitable, but produce the necessary information to create the most favorable outcome.

14-1

15
GLOSSARY OF SPECIALIZED TERMS
A glossary of technical and common descriptive terms routinely used in forensic investigations of electric power utility incidents is included here to provide generally accepted and user-friendly meanings. However, if a situation arises where the strict definition of a term is required, please refer to one of the IEEE standards or guides. ACCIDENT. An unplanned or unforeseen event due to carelessness that interrupts an activity and may cause injury and/or damage. An unexpected happening not due to any fault or misconduct of the injured party but for which legal relief may be pursued. An accident becomes serious when permanent injuries or death occur as a result and/or monetary claims exceed a certain level such as ten million. ANTI-ALIASING FILTER SETTING. Data reduction to improve operation and prevent overload-saturation. Dropped data points equal lower resolution. ARC. A discharge of electricity through a gas, normally air, producing a high temperature bright flash and sharp sound resulting in a partial vaporization of the arcing surfaces. ARC BEAD (on wire). A rounded globule of re-solidified metal at the end of the remains of an electrical conductor that was caused by arcing, and is characterized by a sharp line of demarcation between the melted and un-melted conductor surfaces. CIRCUIT BREAKER (FEEDER). An automatic remote-controlled, single or 3-phase switching device which, when used in a distribution system, is the starting point of a distribution feeder, and the device through which a feeder is connected to the substation bus. CURRENT LIMITING FUSE. A fuse that when its element melts from a specified current introduces a high impedance arc voltage to reduce the current magnitude and duration. CURRENT TRANSFORMER (CT). A bar-type instrument transformer that is intended to have its primary winding connected in series with the conductor carrying the primary current to be measured or controlled. The secondary winding is connected to the device that is used to measure or control the current. The current in the secondary winding is referred to as secondary current. The ratio of primary current to secondary current is the CT Ratio. DEPOSITION. Oral testimony given by a fact or expert witness while under oath outside of the court as part of the discovery process that is admissible in a trial as evidence.

15-1

Glossary of Specialized Terms

DIGITAL FAULT RECORDER (DFR). A multi-channel, continuously-monitoring device installed on principal distribution circuits to capture abnormal or fault waveforms in real time as high speed digital data recordings. DISCLOSURE. A process of exchange and recovery of data of all forms that may be relevant to a legal matter in response to requests by all parties to the other. DISCOVERY. A legally structured gathering of evidence over a limited time period in preparation for trial. DISTRIBUTION FEEDER OVER-CURRENT PROTECTION. Various devices designed to sense the presence of damaging levels of current caused by overloads or by insulation failures and to interrupt such currents quickly in order to limit or prevent damage to equipment. Such devices are fuses, circuit re-closers, and circuit breakers with protective relays. DISTRIBUTION SYSTEM. That portion of the overall electric power system, which supplies power from distribution feeders and transformers to customers at their standard utilization voltage. EVIDENCE. All manner of materials required to act as proof in a legal matter opinion described by the Rules of Evidence. EXPERT WITNESS. A person who is a specialist in a subject, often technical, and who may present his or her expert opinion in court without having been a witness to any occurrence relating to a lawsuit. FACT WITNESS. A person who testifies under oath as to firsthand evidence or knowledge concerning an occurrence to establish the facts in a lawsuit. FAILURE. The inability of a system or component to perform its required functions within specified design performance requirements. FAILURE ANALYSIS. A logical, systematic examination of an item, component, system organization, assembly, or structure, and its place and function within a system conducted in order to identify and analyze the probability, causes, and consequences of potential and real failures. FAULT TREE ANALYSIS (FTA). A cause resolution method utilizing tree analytical techniques as an exercise in deductive reasoning. Starting at the top of the tree with the accident logic flow symbols are used to enter other known or reasoned events leading to the root causes at the base. FORENSIC. A legal term for those things belonging to, used in, or suitable to courts of law or to public discussion and debate. FORENSIC ENGINEERING. The application of the art and science of engineering in the legal system for matters before courts of law, in hearings or arbitration proceedings. 15-2

Glossary of Specialized Terms

FUSE. A self-contained protection device with a specially constructed element, which carries the line current and melts disruptively upon sustained overload or sufficient inrush. GROUND FAULT. An undesired connection that allows a current to flow outside the normal circuit path, such as (a) through the equipment grounding conductor, (b) through conductive material other than the electrical system neutral, (c) through a person, or (d) through a combination of these ground return paths. HEARSAY. Verbal evidence based on information provided by another witness and not the personal knowledge of the source witness. Hearsay is not admissible in court, except when special conditions are met. HIGH RESISTANCE FAULT. An undesired connection characterized by a resistance in the path sufficient to significantly reduce the current flow through it. INCIDENT. A distinct occurrence or definite event such as an accident, failure, or mishap contingent upon or related to a critical outcome. INCIDENT FIRST RESPONSE REPORT. A strategic procedure form designed to direct the collection of facts and information that may be available to utility first responders. The organized live data collected will benefit company management in making early decisions as to further investigation requirements and/or providing improvements in reliability. JURISPRUDENCE. A system or body of law and the course of court decisions. LAWSUIT. A common term for a suit, action, or cause instituted between two parties in a court of law. A proceeding in a civil court under the process of law by one party to compel another for compensation or restitution. MANUFACTURERS SERVICE LETTER. Manufacturers issue letters describing problems that have been discovered with their equipment, and solutions to those problems. Engineers recognize that problems do occur, and a manufacturers willingness to disseminate such information quickly is a positive OBJECTIVE ANALYSIS. The open and equal consideration of all facts and information in determining the cause of an incident without the influence of individual thoughts and experience. The inverse of subjective analysis. OVERLOAD. Operation of equipment in excess of normal, full-load rating or of a conductor in excess of rated ampacity. If it persists for a sufficient length of time, it could cause overheating damage. POSSIBLE CAUSE. An insufficient reason due to the lack of evidence to believe that there is any specific cause and is unacceptable to present as an expert opinion. PROBABLE CAUSE. A sufficient reason to believe that, more likely than not, certain actions and/or materials are related to a specific cause that is acceptable to present as an expert opinion. 15-3

Glossary of Specialized Terms

PROXIMATE CAUSE. A single root cause that leads to the manifestation of an accident or failure. RELAYS (PROTECTIVE). The electrical switch, which controls an automatic operation of substation circuit breakers, reclosers, and other automatic devices. ROOT CAUSE ANALYSIS. A system reliability cause-and -effect model analysis method where the application of a physical root cause analysis identifies the actual cause of a component outage or equipment failure. SER. Sequence of Event Recorder is a device that monitors and records the state of on or off events such as the status of the circuit breaker - Open or Closed. The information can be used to establish the relative or absolute time-line of events. SCIENTIFIC METHOD. The systematic pursuit of knowledge involving the recognition and formulation of a problem, the collection of data through observation and experiment, and the formulation and testing of a hypothesis. SHORT CIRCUIT. An abnormal connection of low resistance between normal circuit conductors of different potential where the resistance is normally much greater. SPOLIATION OF EVIDENCE. A non-deliberate alteration to, or loss of, physical evidence related to an accident or failure subject to sanctions and damages against the custodian. SUBJECTIVE ANALYSIS. The consideration of all facts and information to determine the cause of an accident when influenced by individual experience and thoughts. The inverse of objective analysis. TIMELINE. A graphical representation of the events in the incident displayed in chronological order. TRANSIENT FAULT RECORDER. A device that monitors the instantaneous sampled value of voltage and /or current quantities at specified locations within a power system. The relative location sampling frequency and dynamic resolution determines the quality of the measured signal. If the transient fault recorder is a digital device, it is called a Digital Fault Recorder (DFR) and may also include additional binary changes of state for sequence of event information. UNACCEPTABLE RELIABILITY REPORT. A verified record of action taken, to discontinue use, in the event that a piece or model class of equipment is/are determined to no longer be reliable for service. WORK PRODUCT. The writings, notes, memoranda, reports, emails, or conversations provided by an investigator or expert witness to a lawyer representing a party in a lawsuit.

15-4

16
BIBLIOGRAPHY
The following bibliography includes public domain electric utilities forensic reports, forensic investigation and root cause determination technical papers, forensic engineering texts, and web sites of potential interest. A.C. Franklin, D.P. Franklin. J&P Transformer Book. 11th Edition. Butterworth & Co., Borough Green, England., 1983. [book] Animal Caused Outages, National Rural Electric Cooperative Association. RER Manual 94-5. 1996. [manual] Benjamin J.Cantor. Suggestions on being a Good Witness. http://www.macexperts.com/macnewsletters/spring 88. [website] Captain Kirk Greiner. Expert Responsibility. http://www.macexperts.com/macnewsletters/fall93/depo.htm. [website] Carl Meyer. Expert Witnessing - Explaining and Understanding Science. CRC Press. 1999. [book] Carper, Kenneth L. Forensic Engineering. Elsevier Science Publishing Co., Inc., New York, New York, 1989. [book] Charles T. Raymond. How to prepare for a Forensic Investigation. EC&M Magazine. Huntington, CT. September 1997. [magazine article] Electrical Transmission and Distribution Reference Book. ABB Power T&D Company Inc. Raleigh, North Carolina. 1997. IEEE Brown Book, IEEE Recommended Practice for Industrial and Commercial Power Systems Analysis. IEEE Std. 399-1997. [book] IEEE Draft Guide for Investigating and Analyzing Power Cable, Joint, and Termination Failures on Systems Rated 5 kV Through 46 kV. Publication No. P1511/D7.0. IEEE Guide for Failure Investigation Documentation and Analysis for Power Transformers and Shunt Reactors. IEEE Std. C57.125-1991. [book] IEEE Guide for Field Testing of Relaying Current Transformers. IEEE Std. C57.13.1-1981. 16-1

Bibliography

IEEE Guide for Protective Relay Applications to Power System Buses. IEEE Std. C37.97-1979. IEEE Guide for Protective Relay Applications to Power Transformers. IEEE Std. C37.91-2000. IEEE Guide for Protective Relaying of Utility-Consumer Interconnections. IEEE Std. C37.951989. IEEE Guide for the Application of Current Transformers Used for Protective Relaying Purposes. IEEE Std. C37.110-1996. IEEE Red Book, IEEE Recommended Practice for Electric Power Distribution for Industrial Plants. IEEE Std. 141-1993. [book] IEEE Standard for Relays and Relay Systems Associated with Electric Power Apparatus. IEEE Std. C37.90-1989. IEEE Standard Inverse-Time Characteristic Equations for Overcurrent Relays. IEEE Std. C37.112-1996. IEEE Standard Power Cable Ampacity Tables. IEEE Std. 835-1994. Dec. 30, 1994. ISBN 155937-478-0. IEEE Standard Surge Withstand Capability (SWC) Tests for Protective Relays and Relay Systems. PC37.90.1 Draft Standard Surge Withstand Capability (SWC) Tests for Relays and Relay Systems Associated with Electric Power Apparatus. IEEE Std. C37.90.1-1989. J.R. Reynolds, Jr. Crawford Generating Station and Columbus Park Substation Fires. Report of: Illinois Commerce Commission Staff. June 1991. [report] J.R. Reynolds, Jr. H.C. Manger. ABCs of Conducting an Electrical Failure Investigation Preparation, Investigation, Litigation. TechCon 99. (February 1999). [conference paper] J.R. Reynolds, Jr. Hinsdale Central Office Fire Final Report. Joint Report of Office of the State Fire Marshal and Illinois Commerce Commission Staff. March 1989. [report] J.R. Reynolds, Jr. Investigation of Electrical Power Equipment Failures - The Forensic Engineering Approach. Annual International Conference of Doble Clients. Boston, MA, (April 1999). [conference paper] J.R. Reynolds, P.E., MEWI. Reconstructing the Casualty: Digital Evidence. Eagle International Associates International Claims Handling Conference, Lloyds of London, October 8, 2001. [conference paper] Jack V. Matson. Effective Expert Witnessing. Third Edition. CRC Press. 1999. [book] Nicholas A.Miller. What Should an Attorney Expect from an Expert? http://www.macexperts.com/macnewsletters/spring 87. [website] 16-2

Bibliography

Patricia N. Blair. Witness at Deposition. http://www.mac-experts.com/macnewsletters. [website] R. Bartnikas, K. D. Srivastava. Power and Communication Cables, Theory and Applications. IEEE Press. 2000. IEEE Order Number PC5665. [book] R. Rogge Dunn. Persuasive Expert Testimony Part 2. http://www.macexperts.com/macnewsletters/spring 91. [website] Randall Noon. Introduction to Forensic Engineering. CRC Press, Inc., Boca Raton, Florida, 1992. [book] Robert G. Smith, Esq. Environmental Protection Magazine. Vol. 11. No. 4. p. 42. April 2000. [magazine article] Steven Lubet. Expert Testimony - A Guide for Expert Witnesses and the Attorneys Who Examine Them. NITA. 1998. [book] Ted S. Ferry. Modern Accident Investigation and Analysis. Second Edition. John Wiley & Sons, Inc., New York, 1988. [book] Terms Used by Power System Protection Engineers. Publication Number TP 130 0 031998 1 0. [book] The New IEEE Dictionary of Electrical and Electronics Terms. Fifth Edition. Newly Revised and Expanded. IEEE Std. 100-1992. 1993. ISBN 1-55937-240-0. Underground Distribution System Design and Installation Guide, National Rural Electric Cooperative Association. RER Manual 90-8. 1993. [manual] Underground Systems Reference Book. Edison Electric Institute. EEI No.55-16. 1957. [book] W. Fordham Cooper. Electrical Safety Engineering. Second Edition. Butterworths, Borough Green, England, 1989. [book]

16-3

Target: Distribution Systems

About EPRI EPRI creates science and technology solutions for the global energy and energy services industry. U.S. electric utilities established the Electric Power Research Institute in 1973 as a nonprofit research consortium for the benefit of utility members, their customers, and society. Now known simply as EPRI, the company provides a wide range of innovative products and services to more than 1000 energyrelated organizations in 40 countries. EPRIs multidisciplinary team of scientists and engineers draws on a worldwide network of technical and business expertise to help solve todays toughest energy and environmental problems. EPRI. Electrify the World

2003 Electric Power Research Institute (EPRI), Inc. All rights reserved. Electric Power Research Institute and EPRI are registered service marks of the Electric Power Research Institute, Inc. EPRI. ELECTRIFY THE WORLD is a service mark of the Electric Power Research Institute, Inc. Printed on recycled paper in the United States of America 1001890

EPRI 3412 Hillview Avenue, Palo Alto, California 94304 PO Box 10412, Palo Alto, California 94303 USA 800.313.3774 650.855.2121 askepri@epri.com www.epri.com