Beruflich Dokumente
Kultur Dokumente
Cause
Evaluation
Team
Discovery Bay, CA
Executive Summary
An ineffective corrective maintenance system at MDI led WSS operators to distrust the controls
systems and to accept a normalization of deviation.
Root Cause 3:
Current control room practices at Whisky Slough Station were not sufficient for a facility which has this
level of complexity and risk.
Corrective actions:
Root Cause 1:
CA1 Complete assessment and revise liquid management program as necessary to support
site specific requirements. Include analysis of moisture analyzer locations, database of
moisture/liquids locations, actions to be taken when liquids/moisture are found, as well as
predictions of locations that might have moisture/liquids intrusion into the system.
CA2 Complete a design Failure Modes and Effect Analysis (FMEA) for Design Standard H-14;
per the results update H-14 accordingly.
CA3 Review and revise as needed the Regulation Station Design Manual ensuring it addresses
the risk of excessive moisture/liquid in the system.
CA4 Utilize process safety methodology to evaluate the level of protection and control at WSS
based on the risk posed by operating the facility; per the results, update as necessary to address
the findings. Include an evaluation of V-35 (i.e. the need for out of position alarms and indications)
and other key routing of valves at this station.
CA5 Make temporary ERX for DFM a permanent installation with communication to SCADA
CA6 Make temporary moisture analyzer at WSS a permanent installation with communication to
SCADA
Root Cause 2:
CA7 Identify, prioritize and complete all open high risk corrective work for control systems at
Whisky Slough Station. Identify and complete other high risk issues that have not been entered in
our corrective maintenance system.
CA8 Implement a prioritization system for transmission corrective work (i.e. a work and
compliance matrix).
Root Cause 3:
CA9 Review, and update on-going operator training as necessary for station operators at MDI.
Include system, alarm response training and refresher training.
CA10 Determine the scope of implementing select Control Room Practices (CRM) principles at
manned facilities. Implement based on leadership guidance.
CA11 Revise and update operation procedures and develop addition tools (i.e. develop a
check list) for critical operations.
Additional information is included in the body of this report. This report was prepared by Kimberley
Corona and Katie Simone.
to as the boot, which forms a seal against the throttling plate. No manual regulator bypass line was
installed at this station.
Shown in Figure 2 is a cut-away view of the two stage pressure regulator set. The upstream regulator
(working monitor) is fitted with two pressure pilots: one for pressure regulation and the second for overpressure protection (the pressure monitor pilot). Upstream line pressure enters the working monitor
housing through the loading line highlighted in red. Pressure is reduced to the intermediate pressure
value of 645 psig (highlighted in yellow) as gas flow through the pressure reducing pilot is reduced to
the set point value determined by the spring tension applied to the pressure sensing diaphragm. Gas
pressure is reduced from the upstream side of the working monitor to the downstream side through a
throttling valve in the pressure pilot. When the downstream pressure reaches the adjusted set point
value the gas pressure causes the pressure pilot throttling valve to close. If no demand exists, the
working monitor will remain closed. When demand returns, the working monitor will open and feed
more gas into the downstream system. The same functionality is demonstrated in the regulator, but with
a lower set point value of 380 psig. When the downstream system pressure drops to 375 psig the
secondary regulation run will actively flow to maintain the pressure set point.
decision was made by operations personnel to take the district regulation station supplying DFM 302201 at
out of service to avoid a potential unsafe over pressure condition.
Interruption of service was deemed necessary as there was no option available to manually regulate
the gas flow through a bypass line or to safely regulate pressure through one regulator run while
repairing the other run since both the primary and secondary runs were affected by the abnormal
operating condition.
The gas technicians responding to the event found during disassembly of the pressure pilot, that
hydrates had formed in the pilot throttling valve assembly. Figure 6 shows the location of the hydrates
within the pressure pilot. Technicians also observed hydrates in the openings of the regulator throttling
plate. No analysis of the hydrate was conducted to determine composition of the hydrate.
10
Based on the timeline above, it can be concluded that the moisture came from L57B, ultimately from
McDonald Island indicated by the dramatically increased moisture levels seen when
was
fed from L57B at increased flowrates.
McDonald Island Background Information
McDonald Island is PG&Es largest gas storage field. It is comprised of three stations, Whisky Slough
Station (WSS), Turner Cut Station (TCS), and McDonald Island Compressor Station (MCS). WSS and
TCS process and dehydrate gas. MCS includes two electric compressors (K-1 and K-2) and three
rental gas-fired reciprocating compressors (K-7, K-8, and K-9). The three stations work in conjunction to
inject and withdraw gas into and out of the wells and storage field as demand requires.
In 2011, PG&E started a project to rebuild WSS. The scope of the project focused on the replacement
of obsolete equipment, not enhancements. A process hazard analysis (PHA) was completed on this
project, however the PHA did not identify all potential hazardous scenarios. For example, the
misoperation/failure of WSS V-35 was not analyzed to determine the potential hazardous scenarios
associated with this valve.
Typical Withdrawal Operations
To set-up McDonald Island (MDI) for withdrawal, the TCS and WSS operators respond to the request
by the GTCC by setting valves at WSS and/or TCS for withdrawal, setting the max well flow rates,
ensuring the glycol system is circulating and hot, and aligning the MCS valve lot for withdrawal. One
key valve that needs to be aligned is V-35, which is the bypass valve around the processing and
dehydration equipment. During withdrawal, V-35 needs to be closed to ensure that the gas is sent
through the processing and dehydration equipment. The normal withdrawal operation of WSS is to
supply gas to Line 57B/C through WSS Emergency Block Valve, V-56. The gas is transported,
processed, dehydrated, metered through WSS Master Meter, M-1, and then pressure controlled on the
platform via WSS regulating valves, V-42 and V-37. Figure 10 shows a simple flow diagram highlighted
with the normal withdrawal flow, for additional details see the highlighted operating schematic located in
the Exhibit C.
13
14
At 1200 on 12/26/15, another operator arrives on shift and takes over control of WSS, leaving the
other operator to maintain control of TCS. The WSS operator began rounds for the well reworks,
drained collected fluids from the rework separators, and then drained collected fluids from WSS
Filter Separators (C-11/12/13/14 and C-26). He did not drained the collected fluids from Separators
(C-34/35) at this time. Following his rounds, the WSS operator performed required inspections and
calculations required by the Spill Prevention, Control and Countermeasure regulations (SPCC)
which take about 2-3 hours to complete.
At 1700 on 12/26/15, shown on Figure 12, point 3, GTCC called to request additional withdrawal
capacity and wanted to identify the maximum capacity MDI could provide. Withdrawal rates were
increased to determine the maximum withdrawal rate. After conversations between GTCC and the
MDI operators, it was determined the higher rate was not required and withdrawal rates were
decreased.
At 1800 on 12/26/15, a shift change occurred, the TCS operator went off shift, the WSS operator
went to operate at TCS, and another operator began shift at WSS.
At 2345 on 12/26/15, GTCC called WSS and TCS to alert them of high moisture readings
downstream at
. The TCS and WSS operators checked the dehydration equipment,
checked valve positions at the station, dumped the separators, and took dew points samples. The
dew point samples results were 1 lbs/MMSCF at TCS and 2 lbs/MMSCF at WSS. The TCS and
WSS operators reported back to GTCC that everything was normal.
At 0300 on 12/27/15, shown on Figure 12, point 4, the Up Hole Safety Valves (UHSV) tripped on the
WSS wells. Around the same time, data indicates that V-35 closed. After reviewing data from the
processing equipment, the team concluded that V-35 was open when withdrawal started at 0700 on
12/26/15 and remained open until 0300 on 12/27/15. Figure 13 shows a simple flow diagram
highlighted with the withdrawal flow during this time frame, for additional details of how McDonald
Island was aligned on 12/26-27/15 see the highlighted operating schematic located in the Exhibit C.
16
The Controls Engineer confirmed that the indicators noted above are all independent and that a single
point of failure (i.e. PLC failure, power failure) would not result in the types of anomalies observed in the
data set.
When V-35 is open during withdrawal operations, moist gas from the wells would be sent through
L57B/C to
. The moisture levels observed at
coincides with the data supporting
the fact that V-35 was open during withdrawal.
Upon examination of the Operating Procedures for WSS, the Whisky Slough Station Operating
Procedure Section 3.0 Station Withdrawal (080661) does include a step to CHECK CLOSE Station
Block Valve, V-35. However, the Operating Procedure does not include further details on how to check
close the valve (i.e. check close using Cimplicity, the Plant Mimic Board, or field verification). There are
also no formal requirements to document the positions of key operating valves during operational
changes at WSS (i.e. a checklist). There is no formal shift turnover process. In addition, operators are
not required to perform a field walk down to verify key valve positions at the beginning of each shift.
Since the manned station at WSS is a remote facility from the GTCC, operators are not required to
practice Control Room Management (CRM) processes. This includes fatigue management, written shift
17
turnovers, and thorough documentation of valve manipulations in operating logs best practices. WSS
operators do not get the same rigorous training provided to operators at GTCC/GDCC. GTCC/GDCC
training includes system and alarm response management as well as simulator training. WSS operators
receive primarily on the job training.
Unfortunately due to lack of visibility at WSS, the team could not determine why V-35 was left open
during withdrawal and then closed at approximately 0300 on 12/27. For instance, the position of V-35 is
not stored in Historian and interviews with operators on shift during this event were inconclusive. In
addition, the operator logs for this time do not include details regarding the position of V-35 at the time
of this event. The operator logs only stated the following items regarding starting withdrawal at WSS:
In addition to the lack of visibility, the operators were required to operate the system with inadequate
controls and alarms. For example:
There is no out of position alarm or interlock to prevent V-35 from being open when withdrawal
operations are occurring.
At the time of the event there was no online moisture analyzer at MDI with visibility to GTCC.
Operators were required to take manual moisture readings once a shift, however on 12/26/15 no
moisture reading was taken from the time withdrawal started at 0700 on 12/26/15 until 2345 that
night after being notified of potential high moisture at
from GTCC.
There are no interlocks to prevent key valves from being operated in the local mode.
However it should be noted that during a site visit and interviews conducted by the root cause team on
1/20/16, MDI operators raised concerns with the controls and indication provided in the WSS control
room.For example, they provided the following information:
As of 1/20/16, 11 of 15 valve status indication lights on the WSS Plant Mimic Board were not
working. Specifically the status indication light for V-35 was not working.
The level transmitters seen in Cimplicity and the local level indicators show a different level for C34/35.
A week prior to this event, an operator had called for V-49 to open via Cimplicity, but upon field
verification, found this valve to still be closed and had to open the valve manually.
On 1/20/16, it was determined that the local/remote indication of V-35 on the Cimplicity HMI was
always indicating the valve to be in the local mode regardless of actual status in the field.
A corrective work order entered on 11/25/15 stated that C-14 level not reading correct, and that the
issue was fixed on 11/30/15. Approximately a month later the operator log identified another issue
with the C-14 level indication. However, no corrective action was entered on that day.
18
In conjunction with the concerns of the control system mentioned above, operators also identified
reservation that any issues submitted via corrective work orders would be addressed I a timely manner.
During interviews, operators spoke to a large backlog of maintenance requests that have been
neglected due to the need to complete compliance maintenance work. There is currently no formal
process in place to prioritize maintenance requests at this facility. Due to this distrust, it appears that
the operators have stopped entering items into Pipeline Maintenances (PLM) for corrective work. Of the
6 issues noted above during the interviews, 3 had no corrective work request inputted in PLM and the
other two were added into PLM after the interviews were conducted.
Root Cause/Causal Factors
Root Cause 1:
PG&Es asset and risk management practices did not adequately consider the risks associated with
liquid or moisture intrusion into the system.
Liquid or moisture can cause both regulators and monitors to fail unpredictably and most
stations have been designed without sufficient protection.
Existing pressure regulation design standards assume that gas is typically pipeline quality and
doesnt adequately consider liquid or moisture potential.
There are multiple potential sources of liquid or moisture which are not incorporated in the
design and maintenance practices:
o
Pipeline interconnects
The control system at McDonald Island does not adequately address the potential for liquid or
moisture intrusion.
Root Cause 2:
An ineffective corrective maintenance system at MDI led WSS operators to distrust the control systems
and an accept a normalization of deviation.
Control systems and indicators are not always accurate. Examples include:
o
11 of the 15 valve status indication lights on the WSS Status Board were not working (as
of 1/20/16).
Cimplicity (WSS HMI) was indicating V35 in local control regardless of actual remote or
local control in the field (as of 1/20/2016).
19
Corrective Actions
Root Cause 1
21
CA7 Identify, prioritize and complete all open high risk corrective work for control
systems at Whisky Slough Station. Identify and complete other high risk issues that
have not been entered in our corrective maintenance system.
Assigned: William Mojica
Due: June 30, 2016
CA8 Implement a prioritization system for transmission corrective work (i.e. a work
and compliance matrix).
Assigned: Terry White
Due: July 1, 2017
Root Cause 3
CA9 Determine the scope of implementing select Control Room Practices (CRM)
principles at manned facilities. Implement based on leadership guidance.
Assigned: Dan Menegus
Due: October 31, 2016
CA10 Review, and update on-going operator training as necessary for station
operators at MDI. Include system, alarm response training and refresher training.
Assigned: William Mojica
Due: August 30, 2016
CA11 Revise and update operation procedures and develop addition tools (i.e.
develop a check list) for critical operations.
Assigned: Terry White
Due: March 1, 2017
Extent Evaluation
Extent of Condition
22
This problem could exist when excess moisture or liquid is introduced or left in the system upstream of
regulation facilities which do not have designs resistant to liquid/moisture in the gas stream. Several
actions were taken early in the event. See Appendix C for a complete report. The following additional
actions have been recommended to further mitigate this risk:
1. Implement a program for routine pigging downstream of high risk potential sources or known
intrusion of liquid in the system.
Assigned: George Karkazis
Due: November 30, 2016
2. Identify locations for immediate installation of heaters, based in increased likelihood of liquids or
moisture.
Assigned: Terry White
Due: April 1, 2017
3. Using the results of CA1, install moisture monitoring strategically throughout the system and
consider analyzer redundancy, and perform manual moisture readings as a short-term barrier.
Assigned: Terry White
Due: March 1, 2021
4. Confirm that other moisture analyzers within the system are scaled and calibrated properly.
Assigned: Dan Menegus
Due: August 15, 2016
5. Complete a design review of moisture analyzer installations to ensure adequacy.
Assigned: Terry White
Due: May 1, 2017
Extent of Cause
Root Cause 1 applies to other design standards that may be overlooking other key failure modes and
their effects on the system. Actions 1, 2, and 3 below have been recommended to address this
potential gap.
Root Cause 2 applies to other areas within Gas Operations that may have an excess backlog of open
corrective work for control systems and other high risk issues. Action 4 below has been recommended
to address this potential gap.
23
Root Cause 3 applies to other manned stations where more rigorous control room management
practices will reduce risk. Corrective actions address this gap.
1. Consider completing a design FMEA for other design standards. (RC1)
Assigned: Austin Hastings
Due: June 12, 2017
2. Complete assessment of design standards to see if there are any other standards that provide
conflicting information. (RC1) Evaluate the effectiveness of the process to compare new/updated
procedures with existing procedures.
Assigned: Austin Hastings
Due: December 10, 2016
3. Evaluate existing stations based on the assessment results obtained from CA1. (RC1)
Assigned: Terry White
Due: March 1, 2018
4. Complete system-wide assessment for open corrective work for control systems at all other
stations. Identify and complete other high risk issues that have not been entered in our corrective
maintenance system for other manned stations. (RC2)
Assigned: William Mojica
Due: June 30, 2016
5. Complete assessment of level of concern at other manned facilities. Combine with CA4, use a
survey for both issues, undocumented issues, and distrust of controls.
Assigned: Terry White
Due: September 15, 2016
24
25
Exhibit B Timeline
2012-2013:
12/24/2015:
12/26/2015:
0700:
0730:
1030:
1430:
1800:
1830:
2300:
2345:
12/27/2015:
0050:
0130:
0300:
Process parameters indicated the V-35 was closed at WSS, gas sent from WSS is now
being dehydrated
0315:
Tech at
0500:
0700:
0800:
0900:
Moisture readings at
top out.
Dehydration System
bypassed (V-35
open)
OR
W/D from tubing
header and V-77
open
Valve open
(misoperation)
OR
Left open after
maintenance (valve
stroke)
AND
Operator didnt
realized it was open
Operators not
required to
document valve
manipulations
AND
Operators do not
trust electronic
indicators
Corrective
Maintenance System
ineffective
Date
12/26/2015
12/27/2015
M3
Temp
Time
M3 Flow Press (Assumed)
Zb
17:05
23.15
1027
50 0.9977
17:41
23.11
1015
50 0.9977
0:36
Zf
0.830883
0.830883
Uncorrected
Q
267,447
270,097
Cross
Section Velocity
Area (ft/sec)
2.29211 32.4115
2.29211 32.7327
Temp
50
17.8
1680 seconds
54,721.16 feet
10.36 miles
8553.3 4.3398609
2351.5 1.7063102
Wc
13.8
5.3
First Stage
Second Stage
J-T Cooling
Inlet
Outlet
Pressure Pressure
Outlet
(psia)
(psia) temp (F)
902.4
659.4
35.6
659.4
389.4
17.8
J-T Coefficient 1
0.05925
J-T Coefficient 2
0.06607
(assuming constant enthalpy)
Gas Composition
methane
ethane
propane
isobutane
butane
isopentane
pentane
hexane
CO2
nitrogen
Btu
Relative Density
94.1
4.4
1.4
0.12
0.011
0.004
0.0026
0.006
0.82
0.5
101.36
1058.8
0.6014
Normalized
92.834
4.3408
1.3812
0.1184
0.0109
0.0039
0.0026
0.0059
0.809
0.4933
100.00
The following equation [Kobayashi et al Correlation] was used to determine the hydrate formation temperature with the conditions at
.
Where:
Kobayashi Calculation
Formation Temperature
Relative
Density
P (psia)
P (bar)
900
62.05
890
61.36
880
60.67
870
59.98
860
59.29
850
58.61
840
57.92
830
57.23
820
56.54
810
55.85
800
55.16
790
54.47
780
53.78
770
53.09
760
52.40
750
51.71
740
51.02
730
50.33
720
49.64
710
48.95
700
48.26
690
47.57
680
46.88
670
46.19
660
45.51
650
44.82
640
44.13
630
43.44
620
42.75
610
42.06
600
41.37
Method
for
Hydrate
0.6014
T(C)
0.73
0.74
0.74
0.75
0.76
0.77
0.78
0.79
0.80
0.81
0.82
0.83
0.84
0.85
0.86
0.87
0.88
0.90
0.91
0.92
0.94
0.95
0.97
0.98
1.00
1.01
1.03
1.05
1.07
1.08
1.10
T(F)
33.31
33.32
33.34
33.35
33.37
33.39
33.40
33.42
33.44
33.45
33.47
33.49
33.51
33.53
33.55
33.57
33.59
33.61
33.64
33.66
33.69
33.71
33.74
33.77
33.79
33.82
33.85
33.88
33.92
33.95
33.99
590
580
570
560
550
540
530
520
510
500
490
480
470
460
450
440
430
420
410
400
390
380
370
360
350
340
330
40.68
39.99
39.30
38.61
37.92
37.23
36.54
35.85
35.16
34.47
33.78
33.09
32.41
31.72
31.03
30.34
29.65
28.96
28.27
27.58
26.89
26.20
25.51
24.82
24.13
23.44
22.75
1.12
1.14
1.17
1.19
1.21
1.24
1.26
1.29
1.32
1.35
1.38
1.42
1.45
1.49
1.53
1.57
1.61
1.66
1.71
1.76
1.81
1.87
1.94
2.00
2.08
2.15
2.24
34.02
34.06
34.10
34.14
34.18
34.23
34.28
34.33
34.38
34.43
34.49
34.55
34.61
34.68
34.75
34.82
34.90
34.98
35.07
35.16
35.26
35.37
35.48
35.61
35.74
35.88
36.03