Policy and Practice for Bypassing Interlocks

Click anywhere on the screen to continue

Why Do You Need This Information?

You were selected by your supervisor as someone who would most likely have cause to install an interlock-bypass.

Read the policy on the next few slides. It is followed by additional material that will explain why the policy was adopted, how the process flows, and offer some interlockbypass examples.

page 1

page 2

Why the Policy was Developed

On October 14, 2011, two YR magnets in the Northeast Spreader region caught fire and sustained heavy damage. Root cause was inadequate Work Control processes and practices. Two direct causes:
LCW valves for the dipoles were shut. A jumper (interlock-bypass) in the box supply prevented the thermal switches from tripping the power supply.

What you are about to see is damage caused by the overheating of magnet coils in the tunnel.
Keep in mind that this might have been avoided if the overheating protection (interlock) on the RSEP9A power supply had not been bypassed or if more people had known that it was bypassed.

YR9S04

YR7S03

Damaged Coils

YR7S03 Split

Recommendation* by Fire Investigation Team

* one of many

Recommendation #4 The use of interlock jumpers on equipment installed in the accelerator should be avoided. In circumstances where they are required, they should be clearly marked to aid in identification. All groups should have a systematic method of identifying the installation and removal of jumpers in addition to a method to identify where all jumpers are installed.

Interlock Bypass Team

These individuals were selected to develop a policy to address recommendation #4 and can answer any questions you may have concerning the new policy or its implementation:

Jonathan Creel (Cryo)

Randy Michaud (Operability) Anthony DiPette (Mechanical) Tom Oren (Documentation) Mike Drury (SRF) Paul Vasilauskis (MCC Ops) Ron Lauz (Electrical)

Accelerator Bypassed-Interlock Log

ABIL is electronic log that keeps track of bypassed interlocks in the accelerator
Follow the link (http://opsweb.acc.jlab.org/abil/pro/) to familiarize yourself with ABIL.
How to make or delete an entry in ABIL is straightforward and is not detailed here. You can Mouse Over various fields to view what input information is required. ABIL is set up to notify individuals, via e-mail, when interlocks are bypassed, bypasses are removed, and when their expected removal date is about to expire or has expired. ABIL notifies system owners when expirations dates have been exceeded. Individuals can customize ABIL to view all bypasses or only ones they installed, group by system or show ungrouped, sort on any column, move, resize, hide, unhide columns. Choices stay persistent between uses. Individuals can sign up to receive e-mails when bypasses are installed/removed within a given system. To do so go to the Setup My Email tab and select the systems on which you want to be notified.

Some features of ABIL:

Interlock Bypassed Tag

Tags located at each service building door and near every tunnel phone

Situational Examples
The following 9 example slides are meant to give you some idea of when and how to apply this policy. They do not cover every single instance you may come across. In most cases you will be able to determine if a tag and/or ABIL entry is required by simply using your best judgment.

Example 1
Crew Chief calls for support for a water flow fault on Chopper 1 in the Injector. The fault is impeding the scheduled program.
Technician determines that the flow is okay but the electronic interlock circuit has failed. Technician and Crew Chief discuss the situation and determine that its okay to bypass the interlock until the next maintenance day. Technician makes an entry in ABIL, gets a tag number from ABIL, fills out and installs a tag at the flow meter . Technician installs the bypass.

Example 2
Crew Chief calls for support for a magnet Box Power Supply that has faulted off and is impeding the scheduled program. In order for the Technician to troubleshoot the power supply he must bypass one of the door interlocks. Technician fills out a tag and installs it at a location that is obvious to anyone that the supply has an interlock bypassed. Technician bypasses the interlock. Technician replaces a faulty circuit board, determines the supply is operational, removes the bypass, removes the tag, and tells the Crew Chief the supply is fixed. Note: work is being performed during immediate maintenance and therefore, by policy, did not require an ABIL entry.

Example 3
Its a maintenance day and your ATLis to work on a water flow meter that malfunctioned and required bypassing the interlock has been approved. Technician removes the bypass. Technician replaces the flow meter and verifies operation. Technician removes the tag. Technician deletes the ABIL entry.

Example 4
Crew Chief calls for support for an RF cavity that has faulted off and is impeding the scheduled program. Technician determines that there is a water flow fault that will not clear. Technician and Crew Chief discuss the situation and determine that bypassing the RF Zone is the best course of action. However, the technicians want to continue making repairs in a non-intrusive way to operations. Zone is bypassed. Technician makes an entry in ABIL, gets a tag number from ABIL, fills out and installs a tag at a visible location. Technician bypasses the water flow interlock and continues working on the zone.
Note : This started as immediate maintenance but changed when the zone was bypassed. In that case an ABIL entry was not required for the water flow fault UNTIL the decision was made to bypass the zone with the interlock bypass in place.

Example 5
While running beam to Hall C it became necessary to ask permission to mask the Compton Ion Chamber FSD node. Operator/Crew Chief discuss this with the Hall C Run Coordinator and get permission to mask it. Operator makes an entry in ABIL. NO TAG REQUIRED. Operator masks the node. Note: An ABIL entry was required but a tag was not because the interlock-bypass was made via a software variable.

Example 6
Crew Chief calls for support for valves that will not open because of a vacuum fault that will not clear; this is impeding the scheduled program.
Technician determines that there is not a leak and adjusting the set point to a level below the allowable threshold will clear the problem. Technician and Crew Chief discuss the situation, call for approval (if necessary), and if approved, make the adjustment. No ABIL entry or tag are required. Note: An ABIL entry would be required if the set point required adjustment above the allowable threshold.

Example 7
Crew Chief calls for support for a valve that indicates being closed and will not respond.
Technician determines that the valve is actually open but the limit switch on the valve is faulty giving a false indication. Technician and Crew Chief discuss the situation and determine its okay to bypass the limit switch. Technician makes an entry in ABIL, gets a tag number from ABIL, fills out and installs a tag at a visible location. Technician installs a jumper on the switch readback connector so that the valve now indicates open.

Example 8
While running beam Crew Chief receives an accelerator ODH alarm. Crew Chief contacts the Safety System Group (SSG). SSG technician determines that the ODH head needs to be bypassed and gets permission from the Crew Chief to do so. Technician makes an ABIL entry. Technician disables the alarm in the PLC. No tag is required. Note: An ABIL entry was required but a tag was not because the interlock-bypass was made via a PLC set point.

Example 9
Technician is working on a piece of equipment in her shop. The job requires that she bypass an interlock in order to calibrate it. Technician installs a tag at a visible location. No ABIL entry is required. Technician installs the bypass. Tag remains in place as long as the interlock is bypassed. Lets say for this example that the bypass needs to remain in place because the bad part causing the problem is on order. A few days later the equipment is needed to repair the accelerator to restore the scheduled program. If the interlock bypass is still required AND it was determined that the equipment could be run with the bypass in place, an ABIL entry is required and the ABIL number placed on the existing tag.

Summary
You took this training because you were identified as someone who will likely be bypassing an interlock. This policy is important. We cannot afford another fire caused by an interlock being bypassed due to limited knowledge. You can direct any questions you may have to your supervisor or one of the team members that developed this policy. Typically both a tag and ABIL entry are required, but in some cases only one is. Tags are available both in the service buildings and in the tunnel.

Confirmation
If you have read and understand the policy, the reason for it, and the ten examples provided, click the box below to update your record.

CONFIRM