Incident Management Procedures Guide PDF

Incident Management & Communication
Incident Management
& Communications
Procedures
Guide
Version 1.91
Last Updated:
December 2, 2009
This page left intentionally blank
Table of Contents
Introduction............................................................................................................................................................. 4
Severity Level Definitions ...................................................................................................................................... 5
Communication Checklist....................................................................................................................................... 8
Manager On Call (MOC) ................................................................................................................................ 8
IT Center ......................................................................................................................................................... 9
Incident Manager On Call IMOC List ............................................................................................................... 10
Incident Manager On-Call (IMOC) .............................................................................................................. 11
University IT Technical Staff / Technicians on Call .................................................................................... 12
University IT Director - of affected unit(s)................................................................................................... 12
Information Security ..................................................................................................................................... 13
CIOs Office.................................................................................................................................................. 14
Communication Manager and/or Other Designated University IT Employees............................................ 15
Scribe ............................................................................................................................................................ 16
University IT Office Admins ........................................................................................................................ 16
University IT Staff Members ........................................................................................................................ 16
Provost .......................................................................................................................................................... 16
President........................................................................................................................................................ 16
Other University Executives ......................................................................................................................... 17
Students......................................................................................................................................................... 17
Faculty / Departments or Divisions .............................................................................................................. 17
University Staff............................................................................................................................................. 17
University Security ....................................................................................................................................... 17
University Facilities ...................................................................................................................................... 17
Rochester Management................................................................................................................................. 17
University Legal............................................................................................................................................ 17
University HR ............................................................................................................................................... 17
Communication Call Log...................................................................................................................................... 18
Security Level Definitions .................................................................................................................................... 20
Internal Communications Template...................................................................................................................... 21
External Communications Template..................................................................................................................... 23
University IT Technician Form ............................................................................................................................ 25
External Communication Matrix .......................................................................................................................... 26
Incident Command Center Wall Charts................................................................................................................ 31
IT Alert (G2Alert) Steps to Send a Severity 3 IT Alert:.................................................................................... 40
ISD Manager On-Call - University IT (Data Center Services) Alert Notification ............................................... 41
Appendix............................................................................................................................................................... 43
Roles & Responsibilities................................................................................................................................... 44
Incident Manager On-Call (IMOC) .............................................................................................................. 44
Manager On-Call (MOC).............................................................................................................................. 45
Communications Manager ............................................................................................................................ 46
Web Content Hack Immediate Actions ......................................................................................................... 51
Debrief Procedures............................................................................................................................................ 52
Debrief Agenda Template................................................................................................................................. 53
Updating Procedures......................................................................................................................................... 54
Change Control ................................................................................................................................................. 55
3
Introduction
Leaders in the University Information Technology organization acknowledged the need to develop a wider view
of incident management and communications. In the past, each University IT department utilized its own
incident escalation path. Consistency in delivering incident management and expected communication levels
were not meeting internal and external customer expectations, especially during high profile incidents.
This Incident Management & Communication Procedures manual contains Severity 3 incident response tools.
Severity 3 incidents are the highest level and most critical of events that occur within our organization.
Immediate action is required by multiple people to assist in recovering services affected by the incident. By
identifying scope and ownership of an incident early in the process, we can now triage to the appropriate teams,
who in turn establish their communication protocols and management roles within the context of the broader
incident management procedures.
Incident management and communication processes that had been used independently across the organization
are now merged into a single document and available across University IT. On-call escalation now has the
ability to mobilize an Incident Manager On-Call (IMOC) who coordinates the Incident Command Center and
communication methods to executives and customers. Each departments Manager On-Call (MOC) can now
concentrate on recovering services, without the need to communicate with multiple people. Technicians will
also benefit from these procedures by eliminating multiple communication paths and allowing them to
concentrate on technical issues. Each Severity 3 incident will have a Communication Manager, assigned to
assist with the creation of communication materials. A scribe will detail incident events. After the recovery
from an incident, a mandatory debrief meeting will be scheduled to complete the Sev 3. Documentation for the
debrief methodology has been finalized and is included in this manual.
A coordinated University IT response is essential to our business and services. Our customers demand it, our
internal resources need it, and the Information Technology Services Incident Management & Communication
Procedures Guide delivers it.
Severity Level Definitions
SeverityLevel
3.0
2.5
2.0
1.5
1.0
0.0
ServiceImpact
Enterprisewide
Enterprisewide
Limited
SingleorNone
SingleorNone
SingleorNone
Immediateneedfor
service
Noimmediateneed
forservice
Singledepartment
affectedbyservice
interruption
Singleuserservice
impact
Singleuserservice
impact
Singleuser
serviceimpact
Scopemaynotbedefined
Scopeisdefined
AgedGeneralUser
Ticket
NoServiceImpact
withcomplex
elevatedresolution
NoServiceImpact
withelevated
resolution
NoServiceImpact
Completeserviceoutage
VIPUserTicket
GeneralUserTicket
GeneralUser
Ticket
GeneralUser
Ticket
Triggersformal
communicationplan
Triggersformal
communicationplan
Resolutionby
TechLead/System
Lead
Resolutionby
SubjectMatter
Expert(SME)
Resolutionby
ITCenterstaff
Multipledepartments,
groups,andindividuals
SeverityLevel
Defined
DecisionMaker
Enterprisewideimpact
Universitywidesecurity
violation/compromise
IMOC/Director
Director
Manager
LevelIII:
TechLead,System
Lead
LevelII:
SubjectMatter
Expert
CallAgent
UniversityITDirector
UniversityITDirector
UniversityIT
Manager
LevelIIISupport
LevelIISupport
CallAgentOnly
UniversityITManager
UniversityITManager
LevelIIISupport
LevelIISupport
CallAgent
IMOC
IMOC
CallAgent
CIOOffice
AppropriateUniversity
ITPersonnel
SeniorManagement
ITCenter
AllUniversityIT
ITCenter
IMOCbyDirector
DirectorbyManager
ManagerbyLevel
III
LevelIIIbyLevelII
LevelIIbyCall
Agent
CallAgentby
User
CIOOfficebyIMOC
IMOCbyDirector
UserbyTicket
Assignee
UserbyTicket
Assignee
UserbyTicket
Assignee
UserbyTicket
Assignee
SeniorManagement
byCIOOffice
ITCenterbyDirector
Involvement
ISD
(ifDataCenteror
NetworkRelated)
WhoisNotifiedByWhom
Immediate
Notification
Additional
Notifications
ElevatedUser
Ticket
(Director,Manager)
Triggersinformal
communication
plan
ITCenterbyDirector
ISDbyHyperReach
AllUniversityIT
byHyperReachor
Email
Communication
PlanType
Formal*
Formal*
Informal
Informal
Informal
Informal
Communication
Methods
DirectContact
(phone,inperson)
DirectContact
(phone,inperson)
DirectContact
(phone,inperson)
ServiceTicket
ServiceTicket
HyperReach
HyperReach
AllUniversityIT
byHyperReachorEmail
Walkin,phone
call,email,web
form
Realtime
Communications
ITENS
ITENS
Communication Checklist
This document provides a high-level overview of the communication flow that needs to take place during a declared Severity 3 (Sev 3) incident.
University IT divisions will assess incidents as normal until a Sev 3 has been declared once elevated to a Sev 3, initiate this checklist.
Normal Business Hours (8:00am 5:00pm): Applies to weekdays and non-holidays.
Communication Flow
Manager On
Call (MOC)
Management Steps
Normal Business Hours (8:00am 5:00pm)
After Hours
Determines if University IT Security, University Security,

University Facilities, and/or Rochester Management need
to be engaged. If yes, engages each required units (6, 18,
19, 20).
Notify Unit Director
Declares Severity 3 Incident.
Notifies Customer Contact Centers: IT Center (5-2000)
and NC Ops Center (4-4357) and DC Ops (5-1205). If no
one is available to answer the call, the answering service
process will kick in.
Direct IT Center to maintain CHRON until scribe is
identified.
Notifies IMOC (3). Provides them with a brief of the
situation.
Assembles and leads technical teams/technicians that
must be on-site unless otherwise directed by IMOC.
Determines meeting location and initiates MOC Phone
Bridge if needed (1-866-603-2932 Access #6608484 Pin
#9058 (Host only))
Determines if University IT Security, University Security,

University Facilities, and/or Rochester Management need
to be engaged. If yes, engages each required unit (6, 18,
19, 20).
Notify Unit Director
Declares Severity 3 Incident.
Notifies Customer Contact Centers: IT Center (275-2000)
and NC Ops Center (274-4357) and DC Ops (275-1205).
If no one is available to answer the call, the answering
service process will kick in.
Begins and maintains CHRON until scribe is identified.
Identifies relevant vendors that may be needed.

MOC will determine if techs need to forward their phones
(internal calls only) allowing uninterrupted problem
solving.
Notifies IMOC (3). Provides IMOC with a situation brief

and determines on-site support needs. Determines
resources that need to be on-site. Potential use of IT Alert
(www.g2alert.com)
Assembles and leads technical teams/technicians that
must be on-site unless otherwise directed by IMOC.
Determines meeting location and initiates MOC Phone
Bridge if needed (1-866-603-2932 Access #6608484 Pin
#9058 (Host only))
If incident is over 12 hours, coordinates staffing schedule
Identifies relevant vendors that may be needed.
If the IT Center is not open, MOC for affected
department(s) is responsible for coordinating customer
communication.
8

Management Steps
Communication Flow
IT Center
If the IT Center is open, provide customers with
IMOC-supplied information
Ensures MOC(s)/MOC Designee of affected
department(s) was notified and is aware of the

situation.
Notifies both service center staff members.
If Towne House evacuation, contact NCS Manager
on Call to forward Operations phone numbers 59194 & 5-9195 to 5-2000
Notifies University IT-ORG. Communication should
provide a brief of the situation, what the solution is,
and if the event is still ongoing. Use IT Alert
(www.g2alert.com).
Provides guidelines for customer communication as
determined by the IMOC, MOC, and
Communications Manager or other key players as
needed based on incident type.
Triages calls and provides updates as requested by
MOC.
The Networking Operations Center serves as a hub
to coordinate the communication with customers and
University IT contacts. Both centers are effective at
handling this communication. Keeps the customer
list up-to-date and monitors the service impact by
customer base through direct customer contact.
Periodically checks in with customers to assess the
situation (Are fixes working? Are users still
experiencing problems?) be sure to include
University faculty, staff, and students in relevant
locations.
After Hours
If IT Center is open, provide customers with IMOC-
supplied information. If neither center is open, MOC

for affected department(s) is responsible for this
communication.
Ensures MOC(s) of affected department(s) was
notified and aware of the situation.

If Towne House evacuation, contact NCS Manager
on Call to forward Operations phone numbers 59194 & 5-9195 to 5-2000

Provides guidelines for customer communication as
determined by the IMOC, MOC, and
Communications Manager or other key players as
needed based on incident type.
Triages calls and provides updates as requested by
MOC.
The IT Center serves as a hub to coordinate the
communication with customers and University IT
contacts. Both centers are effective at handling this
communication. Keeps the customer list up-to-date
and monitors the service impact by customer base
through direct customer contact.
Periodically checks in with customers to assess the
situation (Are fixes working? Are users still

experiencing problems?) be sure to include
University faculty, staff, and students in relevant
locations.
Incident Manager On Call IMOC List

Contact Operations (275-9194) or (275-1205) for most current IMOC list
Group covers the following area(s):
The role of the Incident Manager On Call is to lead Severity 3 and Severity 2.5 incidents. The Incident
Manager On-Call is available 24x7.
Schedule
ROTATION
START DATE
2009
May
June
July
August
September
October
November
December
Crowley
Wirley
Barden
Myers
Fredericksen
Crowley
Wirley
Barden
Wirley
Barden
Myers
Fredericksen
Crowley
Wirley
Barden
Myers
Barden
Myers
Fredericksen
Crowley
Wirley
Barden
Myers
Fredericksen
2010
January
February
March
April
Myers
Fredericksen
Crowley
Wirley
Fredericksen
Crowley
Wirley
Barden
Crowley
Wirley
Barden
Myers
PRIMARY
SECONDARY
TERTIARY
OTHER
Personnel
NAME
Barden
Crowley
Fredericksen
Myers
Wirley
CALL FIRST
275.5458
275.8235
273.1714
273.1804
275.5615
CALL SECOND
cell - 317.3398
cell - 733.1365
cell - 313.4003
cell - 208.0939
cell - 638.2591
OTHER AVAIL.
home - 627.1602
pager - 220.3330
home - 586.5986
home - 349.7211
home - 671.9046
OTHER AVAIL.
cottage - 315.536.6634
home - 924.3273
10

Management Steps
Communication Flow
Incident
Manager On-Call (IMOC)
Evaluates the situation and gathers all the facts from

MOC.
Notifies CIO and Directors (5,7).
Initiates ISD Phone Bridge, if ISD systems are involved
1-866-945-2255 Access Code: 608965#

Initiates IMOC Phone Bridge, if necessary
585-273-3311
Access Code 144357 or
1-866-871-2663
Access Code 144357
Call Information Security MOC (DCS Operations [2751205] can provide contact number) to review situation and
determine if there has been a breach [SKIP this step if it is
clear that the event is NOT security related; see next page
for detail]. Information Security Office will make one of
three decisions (see item 6 for details):
1. Security Controlled
2. Security Related
3. No Security Impact
Engages Communications Manager and Scribe (8,9).
Contact DCS Production Control MOC to review impact of
incident with scheduled production jobs. Internal
communication should reflect potential impacts.
Contact SMS group to setup Service Monitoring (Uptime)
if necessary
Notifies University IT Computer Store/Sales if Blackboard,
Flex, or the Secure 1 server (front-end of CS online store)
is down (10). Otherwise, CSS can be notified as part of
University IT Org.
Provides regular updates to the CIO office.
Scheduled IMOC (not acting IMOC) schedules and leads
post-mortem/debrief session within one week of incident.
After Hours
Evaluates the situation and gathers all the facts from

MOC.
Notifies CIO and Directors for after hour incidents.

Initiates ISD Phone Bridge, if ISD systems are
involved
1-866-945-2255 Access Code: 608965#
Initiates IMOC Phone Bridge, if necessary

585-273-3311 Access Code 144357 or
1-866-871-2663 Access Code 144357
Calls in Information Security MOC to review situation
and determine if there has been a breach [SKIP this
step if it is clear that the event is NOT security
related; see next page for detail]. Information
Security Office will make one of three decisions (see
item 6 for details): Security Controlled, Security
Related, or No Security Impact
Coordinates CHRON and scribe duties. Calls in staff

for communications and scribe duties if needed.
Contact DCS Production Control MOC (DCS

Operations [275-1205] can provide contact number)
to review impact of incident with scheduled
production jobs. Internal communication should
reflect potential impacts.
Contact SMS group to setup Service Monitoring

(Uptime) if necessary
Communicates with key people & customers during

event.
Prepares a communication for release to University

IT-ORG and external groups in early AM next
business day. Communication should provide a brief
of the situation, what the solution is, and if the event
is still ongoing. Use IT Alert (www.g2alert.com).
Meets next morning with communications manager

to discuss future communications and follow-up (if
required).
Scheduled IMOC (not acting IMOC) schedules and

leads post-mortem/debrief session within one week
of incident.
11
Communication Flow
University IT
Technical Staff
/ Technicians
on Call
University IT
Director - of
affected unit(s)
Management Steps
After Hours
Technicians will be required to be on-site unless

otherwise directed by the IMOC or MOC.
If MOC determines, technicians can forward internal
calls for short periods of time.
Troubleshoots problem and begins working on
solutions.
Retrieve Technical Recovery Guides (TRGs) for
services affected.
Provides regular updates to MOC.
Participates in vendor calls as needed.
Periodically checks in with other University IT staff
members to assess the situation be sure to include
members in other locations.
Avoid incoming customer calls. These are
distractions to solving the issue at hand. If they are
calling your phone, route them to the Call Centers
(2).
Do not speak with internal (Currents/Campus Times)
or external (D&C/TV stations) media. Direct them to
University Communications.
Technicians will be required to be on-site

unless otherwise directed by the IMOC or
MOC.
Troubleshoots problem and begins working
on solutions.
Retrieve Technical Recovery Guides
(TRGs) for services affected.
Provides regular updates to MOC. If offsite, calls into MOC Phone Bridge if
needed (1-866-603-2932 Access
#6608484)
Participates in vendor calls as needed.
Periodically checks in with other University
IT staff members to assess the situation
be sure to include members in other
locations.
Avoid incoming customers calls. These
are distractions to solving the issue at
hand. If they are calling your phone, route
them to the Call Centers (2).
Do not speak with internal
(Currents/Campus Times) or external
(D&C/TV stations) media. Direct them to
University Communications.
Participates in discussions lead by MOC and IMOC.
May be onsite or working from home as

determined by MOC.
Participates in discussions lead by MOC.
Provides support to technical teams.
Provides any other support that may be
needed to help resolve the incident.
Provides support to technical teams.

Provides any other support that may be needed to
help resolve the incident.
12
Communication Flow
Information
Security
Security Controlled
Examples:
Missing person
Crimes (domestic and
international)
Major security breach
Security Related
Examples:
Worm outbreak
Virus problems
Management Steps
After Hours
Situation is critical and may involve highly sensitive
data.
Security Office takes control of incident
management and IMOC coordinates
communications.
Engages University Legal and/or University HR
(21,22).
Develops and distributes communications on a
limited basis. Some events will require Security
Office to keep all details confidential. Determines (if
critical security situation) what information can be
shared beyond the Security office.
If services are impacted, public communications will
be determined by Security Office. If servers are
down, notifies Operations Centers.
2. Security Related
Reviews situation and gathers facts from
technicians.
Participate in troubleshooting and helps to
implement solution.
Begins a parallel communication stream as may be
required by specific incidents.
Takes no action unless specifically asked to.
Incident is NOT security related in any way.
May be onsite or working from home as determined

by the type of security incident.
Situation is critical and may involve highly
sensitive data.
Security Office takes control of incident
management and IMOC coordinates
communications.
Engages University Legal and/or University
HR (21,22).
Develops and distributes communications
on a limited basis. Some events will require
Security Office to keep all details
confidential. Determines (if critical security
situation) what information can be shared
beyond the Security office.
If services are impacted, public
communications will be determined by
Security Office. If servers are down,
notifies Operations Centers.
2. Security Related
Reviews situation and gathers facts from
technicians.
Participate in troubleshooting and helps to
implement solution.
Begins a parallel communication stream as
may be required by specific incidents.
Takes no action unless specifically asked
to. Incident is NOT security related in any
way.
After incident debrief, IT Security will notify University Audit of major University IT incident. Notification will
include cc: to Julie Buehler for Audit communication retention.
13
Communication Flow
CIOs Office
Management Steps
After Hours
Receives details about incident from IMOC.
Receives details about incident from
Provides incident brief to Provost and President
IMOC.
Decides if the Provost and President
should be notified before the start of the
next business day.
Gathers with IMOC next business day
morning to review event and provides
business perspective (big picture) for the
incident.
(12,13).
Provides business perspective (big picture) for the
incident.
14
Communication Flow
Communication
Manager and/or
Other
Designated
University IT
Employees
(Set up where main
communication is taking
place)
Management Steps
Gathers details about incident.

Crafts messages for internal and external use.
Identifies appropriate communication channels.
Deploys communications according to incident
timeframe through identified channels/Working with

MOC and IMOC. [All Channels]
Provides guidelines for communications to the
Customer Service Centers and to the IT Admins so
they can handle calls appropriately and deliver the
same message (2,10).
Identifies channels for post-incident follow-up and
helps prepare messages for those channels.
Retain copy of all communications for debrief
session and for audit purposes.
After Hours
Picks up the next business day to continue ongoing communications (internal and external)
or to assist in closing out the incident.
If incident is closed:
Sends final communications when incident
is closed.
Identifies channels for post-incident followup and helps prepare messages for those
channels.
Retain copy of all communications for
debrief session and for audit purposes.
If incident is still open:
Gathers details about incident and reviews
CHRON.
Crafts messages for internal and external
use.
Identifies appropriate communication
channels.
Deploys communications according to
incident timeframe through identified
channels/Working with MOC and IMOC.
[All Channels]
Provides guidelines for communications to
the Customer Service Centers and to the
IT Admins so they can handle calls
appropriately and deliver the same
message.
Identifies channels for post-incident followup and helps prepare messages for those
channels.
Retain copy of all communications for
debrief session and for audit purposes.
15
Communication Flow
Scribe
(Set up where main
communication is taking
place)
10
Management Steps
Takes detailed notes during event to help complete
the CHRON and serve as a record of the event.

Types up info in CHRON template and distributes to
team at regular intervals during incident.
Prepares and send final CHRON at close of incident.
Provides this info for debrief meeting.
University IT
Office Admins
Uses guidelines for communications to customers
University IT
Staff Members
Uses guidelines for communications to customers
11
12
Provost
13
President
when responding to calls that may come in from

various areas.
when responding to calls that may come in from

various areas.
After Hours
Picks up in the AM of next business day.

Types up info in CHRON template and
distributes to team at regular intervals
during incident.
Prepares and send final CHRON at close
of incident. Provides this info for debrief
meeting.
Reviews CHRON already completed.
Continues CHRON and takes detailed
notes during the event.
Types up info in CHRON template and
distributes to team at regular intervals
during incident.
Prepares and send final CHRON at close
of incident. Provides this info for debrief
meeting.
In the AM of next business day:
Uses guidelines for communications to
customers when responding to calls that
may come in from various areas.
In the AM of next business day:
Uses guidelines for communications to
customers when responding to calls that
may come in from various areas.
Receives regular updates from CIO.

Disseminates info as needed to key staff members.
Receives regular updates from CIO.
Disseminates info as needed to key staff members.
16

Management Steps
Communication Flow
After Hours
14
Other
University
Executives
15
Students
16
Faculty /
Departments or
Divisions
17
University Staff
18
University
Security
19
University
Facilities
20
Rochester
Management
21
University
Legal
22
University HR
Participates as required by incident.
Participates as required by incident, specifically
Participates as required by incident,
when related to the Towne House building. 461-9440 or

467-2442 after hours
specifically when related to the Towne House

building. 461-9440 or 467-2442 after hours
when security related.

when security related.
specifically when security related.

specifically when security related.
17
Communication Call Log

Last revised On: 7/15/07
Who to contact
Notify?
Yes/No
Contacted By
1st
Contact
At:
2nd
Contact
At:
3rd
Contact
At:
4th
Contact
At:
IT Center: Provide key facts so centers can handle incoming calls consistently and triage accordingly.
IT Center @ 5-2000
Ops @ 5-9194
(TH Computer Room)
University IT Incident Management: Provide key facts and begin IM team mobilization and communications.
University IT Directors
(Sev 3 VIP list)
CIOs office @ 5-5240
Norm Acunis
(for Email Sevs)
Becky Kingcaid
(for Email Sevs or any Sev
affecting Executives in Wallis)
Information Security Office
(as needed)
Michelle Rogers
Bill Waterhouse
Main University IT Communication Channels: Provide high-level status of the event with updates as needed.
3-3999 Recording & Sev Page
Sent
University IT Notices Updated
(University IT website)
IT Center Plasma Screen
University IT Org
Phone Tree and/or G2 Alert
University IT Office Admins: Provide key facts so this team can handle incoming calls consistently and provide
departmental support as needed.
CIOs Office
Finance/Admin/Comm Office
AA Office
NC Office
DC Office
Security Office
External to University IT: Provide high-level status of the event with updates as needed.
Phonedown
Netdown
Presidents Office @ 5-8356
Nicholas Bigelow @ 5-8549
(President of Faculty Senate)
Provosts Office @ 5-5931
18
Who to contact
All Campus Admins.
(for email Sevs)
ISD @ 5-3200
Highland Hospital Comm Ctr
@ 473-2200
Michele Cairns @ 1-8463
Med Ctr Directors office
(Julie Choate, Roberta
Parker)
Comm Ctr @ 5-2222
(Voice Services including VM)
College Deans Office
@ 3-5000
University Security Office
Highland Hospital Security
University Facilities Office
University Human Resources
University Legal
Students
Faculty
University Staff Members
University IT Notices Post
(ITENS)
Campus Times
Currents Digest (Email Daily)
Currents (Print)
Notify?
Yes/No
Contacted By
1st
Contact
At:
2nd
Contact
At:
3rd
Contact
At:
4th
Contact
At:
Security Level Definitions

Department: Information Security Guiding Criteria
Security Controlled (Sec. 3)

Definition
Information has the potential of being disclosed or altered
that would:
1. Violate Laws, Regulations or Contractual Obligations
2. Significantly impact the reputation of the University
OR
Examples
Server has been compromised that has
Student Social Security Numbers.
Major worm outbreak is taking down
email, HRMS, etc.
Main University Web Page significantly
defaced.
A significant and growing number of SERVICES are rendered

unavailable without any operational remedy.
Security Related (Sec. 2)

Definition
Information has the potential of being disclosed or altered
that would:
1. Cause Significant Harm to the University
2. Alter or disclose information regarding an individual
or group in an unauthorized manner
3. Alter the results of Research or Business Processes in
an unauthorized manner.
Examples
Student Changes Grades.
Researcher changes research data
Worm outbreak is spreading rapidly
across ResNet.
OR
A significant and growing number of SYSTEMS are rendered
unavailable without any operational remedy.
Security Notified (Sec. 1)

Definition
Information that has been deemed non-critical has the
potential of being altered or disclosed, without adverse
impact to the University
OR
A number of information systems are rendered unavailable
without any operational remedy
Examples
Known information is taken from a
system without any impact.
Individual systems are hit with a
virus/worm. No trend across the
University is detected.
Internal Communications Template

(Internal University IT Staff Only)
Communications
Contact:
Release Date:
Incident:
Communication
Frequency:
University IT
Internal
Audience:
(check all that apply)
Who needs the

information?
1 Time Only
Initial Comm + Multiple Updates
University IT ALL
Employees
---------------CIO
Directors
University IT Managers
University IT Office
Admins
University IT Operations
Centers (IT Center/NCS
Ops/DCS Ops)
Executive Support Team

University IT Student
Workers (IT Center)
N&C
EC
A&A
S&P
Computer Sales/Store
University IT Finance &
Admin
Other
What information do they need?
Key Facts:
Item 1
Item 2
Item 3
Item 4
Item 5
Item 6
Channels:
Whats the best

way to reach
them?
Email
Web
Phone/Conf. Bridge
ITENS/University IT
Home Page
G2 Alert
ext. 3-3999
In Person/Meeting
Other
University IT Hotline
for follow-up/summary
Initial Communication Copy
Version 1:
Version 2:
Version 3:
Special
Instructions/Notes:
Communication
Channel
University IT Audience
Assigned To
Copy Version
Updates
Time
Date
Message
Page 22
Channels
Last Revised On: 4/17/06
External Communications Template

(External University Community and Press)
Communications
Contact:
Release Date:
Incident:
Communication
Frequency:
1 Time Only
Initial Communication + Additional Updates as Needed
Entire University
External
Community
Audience: - - - - - - - - - - - - - - - - - -
Who needs the

information?
All Faculty (All Schools)

Staff
All Students (All Schools)
Student Workers
(University IT)
Residential Assistants (RAs)
University Administration
Department Administrators
Deans (All Schools)
Provost
President
VP of Communications
Medical Center/ISD
Medical Center/Staff
Medical Center/
Communications Center
Highland Hospital
Communications Center
Memorial Art Gallery
Telephone Directory
Contacts
Key University IT Contacts
University Legal
University Security Office
University Facilities
University Human
Resources Dept.
Campus Times/Currents
Local Press/TV and Print
Other (Use this area for

communications to specific
Colleges)
Channels:
Whats the
best way to
reach them?
Email
Web
Phone
Currents Digest
ITENS - University IT
Home Page
G2 Alert
Fax
In Person Visit various
locations
IT Center Plasma Screen
Flyers post in relevant
areas
Other
Following Incident:
Currents Print
Campus Times
Flyer/Postcard
Follow-up Phone Call
What information do they need?
Key Facts:
Item 1
Item 2
Item 3
Item 4
Item 5
Communication Copy
Version 1:
Version 2:
Version 3:
Special
Instructions/Notes:
Communication
Channel
Audience
(External to University
IT)
Assigned To
Copy Version
Updates
Time
Date
Message
Channel(s)
University IT Technician Form: Incident Management & Communications

Time Alerted
Date
Time
Alerted by
Notified OPS
275-9194
275-9195
220-3283 pager
Message of initial alert:
Time OPS Notified
Vendor Case / Contact:
Systems affected
Services affected
MOC Conference Call Bridge & Pin

1-866-603-2932 pin 6608484#
IMOC Conference Call Bridge & Pin

1-866-871-2663 or 273-3311 pin 144357#
MOC
Time
IMOC
Event
SysAdmin (s)
MOC Notification
& Updates
External Communication Matrix
External To
University IT
Who To Contact
Who Can Contact

(from University IT)
Email
Web
Phone
College of Arts, Science, and Engineering

Deans
Vice Provost and Dean of the College Faculty
Peter Lennie
lennie@rochester.edu
3-5000
Dean of The College
Richard Feldman
richard.feldman@rochester.edu
3-5001
Dean of the School of Engineering

and Applied Sciences
Robert Clark
robert.clark@rochester.edu
5-4151
Vice Provost and Dean of Research

and Graduate Studies
Wendi Heinzelman
wendi.heinzelman@rochester.edu
5-4153
Dean of Sophomores
Vicki Roth
vicki.roth@rochester.edu
5-9049
Dean of Freshmen
Marcy Kraus
marcy.kraus@rochester.edu
5-2354
John Simonson
John Strong
Jim Prescott
Bob Lindholm
john.simonson@rochester.edu
5-3106
5-4873
5-8265
5-0870
Dean/Director
Doug Lowry
dlowry@esm.rochester.edu
263-2807
Computing Services
Jeremy Beyette
jbeyette@esm.rochester.edu
4-1160
David Guzick
david_guzick@urmc.rochester.edu
5-0017
School of Engineering Computing and

Networking Group (CNG)
Department Heads
All Faculty
All Students
Eastman School of Music
School of Medicine & Dentistry

Dean
26
External To
University IT
Who To Contact
Who Can Contact

Email
Web
Phone
School of Nursing
Dean
Kathy Parker
kathy_parker@urmc.rochester.edu
5-8902
William E. Simon Graduate School of Business Administration

Mark Zupan
Dean
mark.zupan@simon.rochester.edu
5-3316
Department of IT
joe.scacchetti@simon.rochester.edu
3-5215
raffaella.borasia@rochester.edu
5-8300
Joe Scacchetti
Margaret Warner Graduate School of Education and Human Development

Raffaella Borasi
Dean
Warner School Information Technology Service
Dave Garcia
River Campus Libraries

Dean
Susan Gibbons
sgibbons@library.rochester.edu
5-4461
Information Technologies
Mike Bell
michael.bell@rochester.edu
5-6875
Jerry Powell
jerry_powell@urmc.rochester.edu
784-6118
Medical Center/Strong Health/Highland

Information Systems Division (ISD)
Communications Center (Strong)
Communications Center (Highland)
Security (Strong)
Security (Highland)
Facilities (Highland)
27
External To
University IT
Who To Contact
Who Can Contact

Email
Web
Phone
University Administration
President
Deputy to the President
Joel Seligman
lamar.murphy@rochester.edu
6-3262
Melinda Smith
melinda.smith@rochester.edu
5-5931
Assistant Provost
Kathleen Moore
kathleen.moore@rochester.edu
5-2497
VP & General Secretary, Senior Advisor to the

President, and University Dean
Paul J. Burgett
pburgett@admin.rochester.edu
3-2284
VP of Communications
William Murphy
Provost
Provost Exec Assistant
Communications Administrator
Lamar Murphy
Ralph Kuncl
5-4124
Maureen Baisch
maureen.baisch@rochester.edu
5-4127
Ronald J. Paprocki
rpaprocki@admin.rochester.edu
5-2800
Helen W. Kostizak
hkostizak@admin.rochester.edu
5-2792
Douglas W. Phillips
dphillips@admin.rochester.edu
5-3311
Dianne Wittman
dianne.wittman@rochester.edu
5-8051
Sr. VP & Chief Advancement Officer
James D. Thompson
jim.thompson@rochester.edu
3-2158
Sr. VP & Vice Provost for Health Affairs and

Medical Center CEO
Brad Berk
bradford_berk@urmc.rochester.edu
5-3407
VP and General Counsel
Sue S. Stewart
sue.stewart@rochester.edu
3-5824
Sr. VP of Finance & Administration/CFO

Admin. Asst.
Sr. VP for Institutional Resources
Secretary
28
External To
University IT
Who To Contact
Who Can Contact

Email
Web
Phone
Memorial Art Gallery

The Mary W. and Donald R. Clark Director
Grant Holcomb
gholcomb@mag.rochester.edu
6-8902
Director
Robert McCrory
rmcc@lle.rochester.edu
5-4973
LLE Computer Support
Alex Rysken
arys@lle.rochester.edu
5-5333
Laboratory for Laser Energetics
Other University Departments

Security Office
Facilities
3-4567
Human Resources
ask-urhr@rochester.edu
5-8747
Office of Communications
Public Information Coordinator
Sharon Dickman
sdickman@admin.rochester.edu
5-4128
Publicist
Helene Snihur
hsnihur@admin.rochester.edu
5-7800
Editor, Currents
Jenny Leonard
jleonard@admin.rochester.edu
5-6076
Web Editor
Lori Packer
lori.packer@rochester.edu
5-5277
Other
Telephone Directory Contacts
Key University IT Contacts
Residential Assistants
University IT Student Workers
29
External To
University IT
Who To Contact
Who Can Contact

Email
Web
Phone
University Health Services (Director)

International Services Office (Director)
Cary Jensen
cary.jensen@rochester.edu
5-8928
Susan B. Anthony Center for Women's

Leadership
Nora Bredes
nora.bredes@rochester.edu
5-9283
University Intercessors
Gerald Gladstein
Frederick Jefferson
Ruth Lawrence
Kathy Sweetland
Office of Technology Transfer
Media (Internal to U of R and External)

Campus Times
Various
editor@campustimes.org
5-5342
Currents Digest
Jenny Leonard
jenny.leonard@rochester.edu
5-6076
Currents (Print)
Jenny Leonard
jenny.leonard@rochester.edu
5-6076
Local TV Stations
Sharon Dickman
sharon.dickman@rochester.edu
5-4128
Local Newspapers
Sharon Dickman
5-4128
Local Radio Stations
Sharon Dickman
5-4128
30
Incident Command Center Wall Charts
Respond
Time
Action
Are Employees Safe? x13
Injured:
Contact Security
(if necessary) x13
Contact Facilities
(if necessary) x3-4567
Security Contact:
Contact Rochester Management

(if necessary)
University IT Security Controlled
Event?
{ Contact Information}
Personnel On-Site:
Contact Information:
SECURITY CONTROLLED EVENT if either of the following exist:

3.
4.
Information has the potential of being disclosed or altered that would:

a. Violate Laws, Regulations or Contractual Obligations
b. Significantly Impact the Universitys Reputation
OR
A significant and growing number of SERVICES are rendered unavailable without any
operational remedy.
Contact University IT Security immediately

University IT SECURITY WILL COORDINATE RECOVERY
ACTIVITIES/COMMUNICATIONS
31
Time
Action
University IT Security Controlled
Event?
SECURITY RELATED EVENT if either of the following exist:

1.
{ Contact Information}
2.
Information has the potential of being disclosed or altered that would:

4. Cause Significant Harm to the University
5. Alter or disclose information regarding an individual or group in an unauthorized manner
6. Alter the results of Research or Business Processes in an unauthorized manner.
OR
A significant and growing number of SYSTEMS are rendered unavailable without any
operational remedy.
Contact University IT Security

IMOC will engage University IT Security to assist in recovery
SECURITY NOTIFICATION EVENT if either of the following exist:
1.
2.
Information that has been deemed non-critical has the potential of being altered or disclosed,
without adverse impact to the University
OR
A number of information systems are rendered unavailable without any operational remedy
CONTACT University IT SECURITY NOTIFICATION ONLY
Severity 3 Declared
Declared By:
Incident Command Center

Contact Information
Phone Numbers:
Fax Numbers:
Help Desk Notifications

CIO Notification
Who
IT Center
x5-2000
Contact Dave Lewis Must Make Verbal Contact; Cell 1st, Home Phone 2nd
32
Control
Time
Action
Technicians On-Site?
ISD Comm Bridge Setup (if necessary)
1-866-945-2255 Access Code: 608965#
IMOC Comm Bridge Setup
Notate Time Sent in Command Center Information
x33311 or 1-866-871-2663 144357#
MOC Comm Bridge Setup
Notate Time Sent in Command Center Information
1-866-609-2932 6608484
IT Alert Sent
Notate Time Sent in Communication Updates
www.g2alert.com
University IT-ORG Email Sent

University IT Notices Updated
x3-3999 NCS Notification
Customer Communications
33
Incident #1 Details
Brief Description of Problem
Services &
Servers
Affected
Customer(s)
Impacted
Resource
Assigned
Current
Status
Relief Person
& Next Shift
34
Incident #2 Details
Brief Description of Problem
Services &
Servers
Affected
Customer(s)
Impacted
Resource
Assigned
Current
Status
Relief Person
& Next Shift
35
Command Center Information

Address:
Fax #:
Location
Phone #
Access #/Pin Code
Phone #
Access #/Pin Code
Phone #
Access #/Pin Code
1-866-945-8855
608965
IMOC
IMOC Communication
Conference Bridge
Only
MOC
Conference Bridge
ISD
Conference Bridge
MOC/Technician
Communication Only
If necessary
36
Personnel
Name
Contact Info
Location
Relief Person &

Next Shift
IMOC
IMOC
Communication
Assistant
Scribe
Communication
Manager
MOC AA
MOC DC
MOC ISO
MOC NC
37
Communication Updates
Vehicle
Contact Info
IT Center
275-2000
Data Center Operators
275-9194
275-1205
IT Alert
https://g2alert.com
University IT
Organization Updated
1. University IT-ORG email list

(if avail)
2. IT Alert
3. Phone Tree
Performed By
Last Update
University IT Notices
Updated
Phone Update
273-3999
38
University
IT
Contact
Service/
Server
Vendor Contact Information

Company Contact Name
Phone #
Case #
39
IT Alert (G2Alert) Steps to Send a Severity 3 IT Alert:

1.
2.
3.
4.
Gather information concerning the incident: Incident details, Service(s) Application(s) and Server(s) affected
https://www.g2alert.net & login
Choose Messages, then choose Send A Message
Choose Create or Edit a Message or select an existing Message from the pulldown list
If Creating a New Message

Message Setup
Time of Day
Business Hours
After Hours
Choose Create A Message
Select Message
Choose Start with a Template, and choose appropriate timeframe template

Business Hours - Template
After Hours - Template
Create A Message
Remember: You must fill in EACH method below for the message to reach recipients via that method (Text, SMS and Voice)
***** Change the Message Name *****
Message Name
Voice Messages
Email Messages
Type as you would say it; you may
need spaces between letters
Change {service/application/server/event} to
reflect actual incident.
Change {service/application/server/event} to
reflect actual incident.
Press Send go to Sending Message below
Text Messages
Maximum 108 characters
Send
If Editing a Existing Message

Message Setup
Select Message
Business Hours
Time of Day
Choose Edit or Copy
After Hours
Remember: You must fill in EACH method below for the message to reach recipients via that method (Text, SMS and Voice)
Voice/Email/Text
Messages
Change message to reflect actual incident.
Change message to reflect actual incident.
Send
Sending Message
Message Setup
Time of Day
Business Hours
After Hours
List
(CL) ALERT: UnivIT Only DIRs/MGRs

(CL) ALERT: ISD Only MOCS & Bat Line
Request Confirmation
Of a Receipt
Always Choose Yes
Prompt Voice Message

Recipients to Join A
Conference Call
No
Yes
Device Preferences
Choose Send to ALL
Choose Send to Preferred Only
Send Alert
Always choose Now
5. Press Continue in the lower right hand corner.

6. After verifying the Send Message, choose Send. This will invoke the service to distribute the message.
Version 1.9
Confidential
40
ISD Manager On-Call - University IT (Data Center Services) Alert Notification

University Data Center Services uses IT Alert, automatic notification software, that contacts specified
individuals automatically, via cell phone, pager, home phone, e-mail, fax, or other, in the event of an
emergency.
The ISD Manager on Call will be contacted by IT Alert for any Severity 3 incident. At all hours, IT Alert will
contact the ISD Manager on Call listed below.
NOTE: The ISD Manager on Call will follow the ISD Incident Management Procedures to
activate and contact ISD Management as applicable.
Single system outages will be escalated through normal University IT escalation procedures. IT Alert
will not be activated.
The ISD Manager on Call will be contacted by each of their communication devices.
Contact will be made in the order shown below, pager, cell phone, work phone, home phone, and email.
The pecking order will continue until all of your devices have been reached.
The IT Alert Notification contacts all devices; it does not stop if it reaches you by one of your
contact devices, even if you have confirmed receipt.
ISD Manager On Call Schedule 2009 (see next page)
Version 1.9
Confidential
41
ISD Manager On Call Schedule 2009
Section 2. Contact and Communication Information

StartTime EndTime
MgrPerson
OnCall
Primary
Secondary
OnCall
Business
Phone
PagerOrCell
Phone
HomePhone
3130485
3145665
3145665
7348976
4159053
5866384
3155898776
3155898776
3155247430
6717570
3509588
7871639
3509588
5076791
5076791
7302299/755
5395cell
7302299/755
5395cell
5763651
5763651
4159053
7348976
4743569
4743569
5079270
5079270
2451884/880
1022
2451884/880
1022
5079270
5079270
4724184
4724184
8209274
7871639
3931229
3931229
6/16/09
6/16/09
7/7/09
7/7/09
7/21/09
7/6/09
7/6/09
7/20/09
7/20/09
8/3/09
RickHaverty
DaveLindsey
DaveLindsey
DianeKoretz
ChipNimick
Primary
Secondary
Primary
Secondary
Primary
7/21/09
8/3/09
GaryScialdone
Secondary
8/4/09
8/4/09
8/18/09
8/17/09
8/17/09
8/31/09
GaryScialdone
NancyBales
NancyBales
Primary
Secondary
Primary
7846126
7842949
7842949
3410403
7846115
7842480
/2751120
784
2480/275
1120
7848322
7848322
8/18/09
8/31/09
SueGraves
Secondary
7842435
9/1/09
9/1/09
9/15/09
9/15/09
10/6/09
10/6/09
10/20/09
10/20/09
11/3/09
9/14/09
9/14/09
10/5/09
10/5/09
10/19/09
10/19/09
11/2/09
11/2/09
11/16/09
SueGraves
TedVaczy
TedVaczy
ChipNimick
DianeKoretz
KathrinKenny
KathrinKenny
TinaDePalo
TinaDePalo
Primary
Secondary
Primary
Secondary
Primary
Secondary
Primary
Secondary
Primary
7842435
7846002
7846002
7846115
3410403
7846121
7846121
7848338
7848338
11/3/09
11/16/09 HalleMcNaney
Secondary
7844275
11/17/09
11/17/09
12/1/09
12/1/09
12/15/09
12/15/09
11/30/09 HalleMcNaney
11/30/09
TinaDePalo
12/14/09
TinaDePalo
12/14/09
MartyBush
1/3/10
MartyBush
1/3/10 DawnRobinson
Primary
Secondary
Primary
Secondary
Primary
Secondary
7848275
7848338
7848338
7848331
7848331
7846159
Version 1.9
Confidential
3353276
3353276
6242792
6242792
6717570
3155247430
3155244821
3155244821
5079270
5079270
2451884/8801022
2451884/8801022
5079270
5079270
4583519
4583519
3831213
42
Appendix
Version 1.9
Confidential
43
Roles & Responsibilities

Incident Manager On-Call (IMOC)
The Incident Manager On-Call is a Director-level role and is responsible for managing University IT-wide incidents. The
IMOC serves as a liaison to University executive offices and the University IT Managers On-Call during SEVERITY 3
incidents (defined below). They are on-call for one month, and are supported by a secondary and tertiary backup. The
IMOC is available 24x7 during their monthly assignment.
Definition of Severity 3:
The problem has a critical impact on key functions within the University or its reputation. Resolution takes highest
precedence.
IMOC responsibilities:
Evaluate the situation and gathers all the facts from all Managers On-Call.
Determine if the MOCs should be onsite during an incident that occurs outside normal business hours (8am-5pm
weekdays), also known as AFTER HOURS.
Oversee the Severity 3: Communication Checklist & Call Log process
Contact the CIO
Work directly with MOCs & technical teams as necessary
Notifies University IT Information Security Office to review incident and determine if a security breach has
occurred.
Serve as incident Communications Manager and oversee the gathering of information (CHRON) and customer
communications.
Determines the need/location of an Incident Command Center to manage the incident (also referred to as the
University IT War Room)
Designate an incident scribe. In direct contact with the incident scribe and oversees all notifications to University
IT ORG and if necessary, key University division contacts; Presidents Office, Provosts Office, Office of
Communications, College Deans Office, URMC (School of Nursing), Simon School, Warner School and Eastman
School of Music.
IMOC Schedule Changes

If an IMOC is unavailable (sick, vacation, etc.), the IMOC is responsible for the following:
1. Notifying the secondary or tertiary IMOC to serve in their place
2. Notify University IT Production Control of the change in schedule
a. Use the ITS Production Control distribution list in the GAL
b. Include start and stop dates and times for schedule modification
University IT Production Control will provide the IMOC update to the following:
1. SharePoint On-Call List https://sharepoint.its.rochester.edu/sites/DataCenter_OnCall/default.aspx
2. University IT Directors DL
IT Leadership in the GAL
3. University IT Managers
ITS Managers in the GAL
4. IT Centers
univithelp@rochester.edu and/or 5-2000
5. Ida Gatto
ida.gatto@rochester.edu and/or 5-9510
Version 1.9
Confidential
44
Manager On-Call (MOC)

The Manager On-Call is a Manager-level role and is responsible for managing business unit level incidents. The MOC
serves as a liaison for after hours notifications of the situations that are subject to off-hours resolution; receive calls from
the after hours dispatch service, provide severity level review, triage/filter and dispatch staff as required. They are
generally on-call for one week, and are supported by backup MOCs. The MOC is available 24x7 during their assignment.
For severe service outages referred to as SEVERITY 3, the MOC is required to contact the Incident Manager On-Call
(IMOC).
precedence.
MOC is responsible to:
Ensure that each call is reported [Chronology, HEAT or some other logging tool?] Only summary information
needs to be recorded for all of the single user problems. Severity 2 and Severity 3 problems require
communication as specified to ensure proper notification of service outages and also require logging basic
chronology of events to report significant progress in solving problems.
General Rule - State what you can do for the customer and not what you cant do by positive negotiations. Offer
your office phone number to the IT Center and the Operations Center number for inquiries by the customer on the
next business day.
Update the University IT MOC list, and individual unit on-call schedules should be used to determine the
appropriate triage and notification(s)
Certain service disruptions require contact with general dispatch points:
ISD Help Desk at x53200 can be your reference point for any ISD staff on call for desktop or Med Center
department network issues, such as with Omega. If x5-3200 is closed (after hours), you will be rolled
over to the Data Center (x5-9194 or x5-9195).
Energy Management at x34567, a.k.a. Customer Service Center, a.k.a. Energy Operations Center, is
your link to all trades-people in Facilities. Please note that if there are any issues concerning what
the dispatchers at x34567 ask you, you may ask them to patch you through to their Supervisor.
Communications Center at x52222 is your link to all Med Center On-Call people (with the exception of
ISD staff)
ResNet Help Desk at x35154. Laurel Contomanolis, and other ResLife Staff may be utilized to refer
issues to the Duty Dean, Resident Advisors, or to ResNet staff when the ResNet Help Desk is not open.
If none of these dispatch points work, consult with another Manager On-Call Rep or see if the
Directory's area listing ("Departments, Offices, and Services") offers contact information.
Disruptions of any voice related services in the Medical Center require communication to the
Administrator On-Call via the Communications Center.
Any safety issues must be immediately communicated to Security at 275-3333.
If a customer declares a situation to be an emergency, do not question that judgment. Consult
with Security, x13 or 275-3333, immediately.
Version 1.9
Confidential
45
Communications Manager
The Communications Manager is responsible for managing University IT-wide and University-wide communications for
University IT-wide incidents. [This is a role served during an incident and not a job title.] He/She serves as a
communications liaison to the IMOC during SEVERITY 3 incidents (defined below).
The IMOC may choose to serve in this role if another suitable employee is not identified. The Communications Manger
must review all communications with the IMOC before releasing them, unless otherwise stated by the IMOC. In some
cases, the CIO (or Assistant CIO, Other Directors) may require that all communications get reviewed by the CIOs Office
prior to deployment.
Communications Manager needs to compose and deploy updates during the course of the incident. The Communications
Manager should also send out a final message indicating the incident is closed and offering a status report to affected
users. [See sample text at the end of this document.] In some cases, the Communications Manager will need to provide
details and in other cases, it will be necessary to remain vague. IMOC and CIO will provide guidance on this.
Be sure to communicate with Becky Kingcaid/Alivin Ruiz if it is an issue that affects Wallis Hall. Becky will often re-tool
general messages based on the needs of users in Wallis hall. It is a good idea to send her copy before releasing to the
general public so she has a heads up.
Refer to the templates and checklists provided in the Incident Management Handbook for details on communication
channels, etc.
precedence.
Communications Manager Responsibilities:

Gathers details about incident.
Crafts messages for internal (University IT Only) and external (University-Wide) use.
Works with Office of Communications if communication outside of the University is required. Depending on the
situation, the Communications Manager may or may not be asked to speak to the press. But should never do so
unless given instructions to.
Deploys communications according to incident timeframe through identified channels/Working with MOC and
IMOC. [All Channels]
Provides guidelines for communications to the Customer Service Centers and to the IT Admins so they can
handle calls appropriately and deliver a consistent message.
Identifies channels for post-incident follow-up and helps prepare messages for those channels.
Retains copy of all communications for debrief session and for audit purposes.
If an incident occurs after normal business hours:
The Communications Manager picks up the next business day to continue ongoing communications (internal and
external) or to assist in closing out the incident.
Sends final communications i
Version 1.9
Confidential
46

Gathers details about incident and reviews CHRON.
Crafts messages for internal and external use.
Deploys communications according to incident timeframe through identified channels/Working with MOC and
IMOC. [All Channels]
Provides guidelines for communications to the Customer Service Centers and to the IT Admins so they can
handle calls appropriately and deliver the same message.
Certain service disruptions require contact with general dispatch points (this is usually done by the IMOC - but you
may be asked to continue to provide them with updates during the course of the incident):
ISD Help Desk at x53200 can be your reference point for any ISD staff on call for desktop or Med Center
department network issues, such as with Omega. If x5-3200 is closed (after hours), you will be rolled
over to the Data Center (x5-9194 or x5-9195).
Energy Management at x34567, a.k.a. Customer Service Center, a.k.a. Energy Operations Center, is
your link to all trades-people in Facilities. Please note that if there are any issues concerning what
the dispatchers at x34567 ask you, you may ask them to patch you through to their Supervisor.
Communications Center at x52222 is your link to all Med Center On-Call people (with the exception of
ISD staff)
ResNet Help Desk at x35154. Laurel Contomanolis, and other ResLife Staff may be utilized to refer
issues to the Duty Dean, Resident Advisors, or to ResNet staff when the ResNet Help Desk is not open.
If none of these dispatch points work, consult with another Manager On-Call Rep or see if the
Directory's area listing ("Departments, Offices, and Services") offers contact information.
Disruptions of any voice related services in the Medical Center require communication to the
Administrator On-Call via the Communications Center.
Any safety issues must be immediately communicated to Security at 275-3333.
If a customer declares a situation to be an emergency, do not question that judgment. Consult
with Security, x13 or 275-3333, immediately.
Version 1.9
Confidential
47
Sample Communication Copy

General Pointers:
Always include a heading/subject line even if email isnt used. Helps people get their bearings.
Be careful not to over promise on a solution or quick outcome.
Provide estimates when possible.
Indicate where people can go for additional information. Use Contact University IT in most cases with whatever
number is appropriate for the incident.
Dont provide too much technical information.
Speak in terms the average end user will understand.
Tell users what to expect.
Keep track of all communications in a Word document and add the time the communication was sent out.
Provide updates after major attempts to solve the problem, such as server reboots, hardware swaps, etc.
Sample INITIAL Messages:

Exchange Email Disruption
Between 8:30am and 10:00am today, some University faculty and staff experienced disruptions with email service. These
disruptions were confined to a subset of Exchange email users. University IT support teams have isolated and resolved
the issue.
We apologize for the inconvenience this may have caused you and we will continue to monitor the situation throughout
the day. If you have additional questions or begin to experience problems with your email, please contact University IT at
5-2000.
UNIX Email Disruption

University IT Support teams are currently investigating issues that may be resulting in delayed email delivery. We
apologize for the intermittent slowdowns you may be experiencing with email services. University IT teams are working
diligently to address the issue as quickly as possible.
As a precaution, our hardware vendors have been called in to assist with the investigation and we will be working with the
vendors to identify actions to minimize this type of disruption in the future. We expect to provide additional information by
5:00 this afternoon (1/19).
If you have additional questions, please contact University IT at 5-2000.
Sample SUBSEQUENT Messages:

UNIX Email Disruption 6:15PM Update
We are still experiencing intermittent email issues on the mail.rochester.edu mail server and we will be rebooting the
server at 6:20 p.m. this evening. Mail services will be unavailable for approximately 20 minutes. We appreciate your
patience as we continue to work on resolving this issue.
Please continue to check back for regular updates. You can also call the University IT at 5-2000 or 3-3999 (recorded
message).
Version 1.9
Confidential
48

We are still experiencing email issues on the mail.rochester.edu server following the reboot performed at 6:20 p.m. As we
work with our vendors to diagnose the problem, you may continue to experience intermittent availability of email.
Please continue to check back for regular updates. You can also call the University IT 5-2000 or 3-3999(recorded
message).
Faculty and students may still be experiencing intermittent disruptions with email service. Users experiencing these
problems are primarily within the College. We will continue to work with vendors to isolate the source of slow email
service. At this point, we are progressing through a detailed plan. We apologize for the inconvenience; we recognize the
impact that this has on you and are working to remedy the remaining issues.
UNIX Email Disruption 8:00AM Update

University faculty, students and staff who were experiencing disruptions with email on Thursday, January 19 can now log
into their email. You may experience a delay with your initial log in if you have a large quantity of unchecked messages in
your mailbox. University IT staff resolved some service disruptions and is maintaining a continuous effort to address the
issue.
University email services will be monitored throughout the day. Please contact Information Technology Services at 5-2000
if you need assistance.
Generic NetID template to be used when LDAP is disrupted.
We are experiencing a service disruption with the Universitys LDAP service. This means that applications requiring a
NetID for authentication are currently unavailable. IT support teams have identified what must be done to resolve the
issue OR IT support teams are working to identify the cause of this disruption. [if the reason is known and can be shared
in terms the users will understand, add a brief statement here]
We apologize for the inconvenience and we expect to have the problem resolved by [enter info here]. We will provide
additional updates as they are available [or enter a specific time(s)].
Please contact the IT Center at 275-2000 if you have additional questions.
Sometimes, we think we have fixed a problem and it comes back (or was never really fixed to begin with). Heres
an example of how to handle that.
First Message
We have received new information that some University faculty, students, and staff members are still experiencing
intermittent email issues.
We apologize for the slowdowns you have been experiencing the past few days. We recognize the importance of email
service and that this disruption has happened at an inopportune time. We are working diligently to restore full email
services.
Please contact University IT at 5-2000 if you need assistance. Status information is also posted on the IT Notices found at
www.rochester.edu/its/.
Second Message
Improvements to the email environment continue. We recognize the importance of email services and Information
Technology Services continues to work diligently to restore full email services. Please contact Information Technology
Services at 5-2000 if you need assistance. Status information will continue to be posted on the IT Notices found at
www.rochester.edu/its/.
Version 1.9
Confidential
49
Sample CLOSED Incident Message: [It is important to send out a final communication to let users know
that all has been restored and to offer an explanation of what to expect.]
Exchange Email Disruption
On Friday, May 5 at 8:25 a.m., University IT became aware of an issue with one of the Exchange 2003 email servers that
resulted in a brief email outage for a subset of Exchange email users. University IT support teams isolated and resolved
the issue and had email restored by 10:00 a.m.
During this time, emails were held in the queue and delivered when email services were brought back online. Please be
assured that no emails were lost during this event.
We apologize for the inconvenience this may have caused you and we will continue to monitor the situation throughout
the day. If you have additional questions please contact University IT at 5-2000.
In some cases, it may be necessary to provide information about an incident to people NOT directly
affected. An example communication is provided below.
Message for IT Key Contacts (Includes IT people outside of University IT was sent to help other IT support users who
were affected by the outage)
Dear IT Colleagues,
On Thursday, January 19, University faculty, students and staff started to experience intermittent disruptions with UNIX
email service on the mail.rochester.edu server. Users experiencing the problems were primarily within the College.
University IT worked with our vendors to isolate the source of slow email service.
Users who were experiencing disruptions with email on Thursday can now log into their email. They may experience a
delay with initial log in if their mailbox contains a large quantity of unchecked messages. Processing capacity was added
to help move mail through the various checkpoints (anti-spam/anti-virus). University IT staff resolved some service
disruptions and is maintaining a continuous effort to address the issue and University email services will be monitored
throughout the day.
Regular updates will be posted online at: www.rochester.edu/its/ - IT NOTICES. Please use this information to keep your
area up to date with this issue. University IT uses this area to communicate with the University community on a regular
basis and will be a source of information for you on this issue, regular updates on maintenance outages, and other
University IT services.
If you have additional questions, please contact University IT at 5-2000.
Version 1.9
Confidential
50
Web Content Hack Immediate Actions

On receipt of WebWatcher or other notification of a hack
1. Go to the page reported and see what has happened
2. If there does not appear to be anything different check with the owner of the file.
3. If confirmed hack begin notification of ? How should we start the escalation?
4. Do not delete or move any of the hacked files until the evidence is reviewed
5. Is this a OS hack or a content hack? If content hack continue
(We should have a procedure for assessing an OS hack)
6. Get the modified date and time of the hacked file
7. Using that time minus one hour find all files that have been modified
You are looking for hack tools and any additional hacked pages.
If nothing turns up use minus two hours etc.
8. Review these files for hack tools
9. Review log files for the hacked file access record and note the IP number
10. If more than one file is hacked find those in the log and capture the IP number
11. Preserve copies of the hacked files
12. Redeploy or restore the hacked file
13. Identify the ISP of the hacker and their entire IP range
14. Block that range at the router.
15. File an abuse report with the ISP of the hacker
16. File an incident report with Campus Safety
17. Evaluate the methods used & determine what actions can be taken to prevent a repeat.
Version 1.9
Confidential
51
Debrief Procedures
1. Debrief will be scheduled by the scheduled IMOC, not the acting IMOC. Meaning, if a scheduled IMOC is
unavailable to be IMOC and an acting IMOC is leading the incident, the scheduled IMOC will be responsible
for scheduling and leading the incident debrief. If circumstances prevent the scheduled IMOC from assigned
duties, the acting IMOC will be responsible.
2. Debrief should occur no later than one week after the incident, with one day after the incident preferred while
information and events are fresh in participants minds.
3. Debrief documentation is to be stored in the Incident Management and Communication SharePoint site,
located: https://sharepoint.its.rochester.edu/sites/ITS-IMC/Shared%20Documents/Forms/AllItems.aspx under
the Incident Reports and Debriefs.
4. After documentation is complete, send an email to Bill Waterhouse. He will in turn produce a UR Audit
update to be sent to the Universitys Audit department and Julie Buehler.
Version 1.9
Confidential
52
Debrief Agenda Template

Event Date:
Event Time:
Event Description:
Attendees:
Debrief Facilitator:
Debrief Date:
Item
1.
Notification
Was everyone notified in a timely manner?

What would have made it better?
2.
Turnout
Was everyone there who needed to be there?

What other personnel would have helped?
3.
Communications
4.
Personnel
Did we communicate to each other well?

Did we communicate to customers well?
How can we improve the process?
Did we have the correct personnel on-site throughout the incident?
Was the personnel rotation correct?
5.
Equipment
Were the room(s) equipped with the correct items to support the incident?
What other equipment would have helped?
6.
Intra-Departmental
Cooperation
Did the University IT business units work together in the best manner?
7.
Inter-Departmental
Cooperation
Did University IT work together with other University departments in the best manner?
8.
Initial Strategy
Did we use the best strategy to minimize incident timeframe?

What strategies would have improved turnaround?
9.
Execution
Did we execute the strategy in the best manner?

What could we have done better to improve turnaround?
10.
Clean Up
Was incident closed so everyone knew to step down from a Severity 3?

Was chronology published in a timely manner?
11.
Customer Impact
What feedback did we receive from customers?
12.
Follow Up Items
What open items still need attention?
13.
Lessons Learned /
Recommendations
What did we learn?

What would make incident response and communication better?
14.
Audit Notification
IT Security will provide incident notification to University Audit
Version 1.9
Confidential
53
Updating Procedures
The following procedures manual was initially developed by the Incident Management & Communications team between
February and June 2006. If you have any questions, concerns, or modifications to the following procedures, please
contact the IT Center (275-2000 itscenter@rochester.edu)
The following people had a major role in the creation of this document:
Project Sponsors
Kate Crowley
Network & Communications
Project Manager
Bill Waterhouse
Security & Policy
Project Participants
Norm Acunis
Mike Fitch
Karen McVige
Joe Pasquarelli
Jay Riley
Mercedes
Fredericksen
Jason Wagner

Data Center Production Control
Academic Technology
Applications & Architecture
Office of the CIO Communications
Academic Technology Emergency Preparedness
Others assisted with its creation, and Information Technology Services is thankful for the participation and guidance to
better serve our customers.
Version 1.9
Confidential
54
Change Control
Name Person
Section change
Description change
Date
Version
Number
Bill Waterhouse
IT Alert
Modified procedures to follow G2Alert alert

custom list modifications
12/08/2006
1.0
Add University IT Security to notify Audit of

major University IT incident
12/13/2006
1.1
Updated ISD On-Call schedule for 2007
1/3/2007
1.2
1/9/2007
1.3
7/15/07
1.4
12/10/07
1.6
02/01/08
1.7
5/29/09
1.8
6/17/09
1.9
Bill Waterhouse
Communication
Checklist Section 6
Debrief Document
Bill Waterhouse
Bill Waterhouse
B.J. Block
ISD On-Call Update
IM&C Quarterly
Update Q1 2007
IM&C Quarterly
Update Q3 2007
Appendix
Bill Waterhouse
IMOC Schedule 08
Bill Waterhouse
IM&C Quarterly
Update Q4 2007
Bill Waterhouse
IMOC Schedule
IT Alert (G2Alert)
ISD Manager On Call
Bill Waterhouse
Contact Information
Bridge Phone #
1. ISD Conference Call # in IMOC checklist

2. Service Monitoring query in IMOC
checklist
3. NCS MOC to forward Operations phone
numbers if TH evacuation
4. Debrief is required, and scheduled IMOC
will schedule (not acting IMOC)
5. Add Services Monitoring (Uptime) to IMOC
checklist
1. Changed name from ITS to University IT
2. Updated Information Security and Policy
Director to Bill Waterhouse
3. Updated contact information for Bill
Waterhouse
4. Updated IMOC schedule through
beginning of 2008
5. Changed debrief documentation to state
that the debrief should be sent to Bill
Waterhouse and he will send to audit
6. Updated email distribution lists to new
naming convention
7. General editing updates
Updated appendix to include Web Content
Hack Procedures
Updated 2008 IMOC schedule
1. Updated 2008 IMOC schedule
2. Updated 2008 ISD schedule
3. Recovered roles deleted from version 1.6
4. Updated MOC role to include University IT
MOC decision point
1. Updated University IT IMOC Schedule
2. ISD notified during any Severity 3 alert
3. Updated ISD IMOC information
1. Updated all internal & external contact
information
2. Added 3-3311 bridge # throughout doc
Version 1.9
Confidential
55

Incident Management Procedures Guide PDF

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Incident Management Procedures Guide PDF

Hochgeladen von

Copyright:

Verfügbare Formate

Incident Management & Communication

Incident Management & Communication

This page left intentionally blank

Incident Management & Communication

Incident Management & Communication

Incident Management & Communication

Severity Level Definitions

Incident Management & Communication

Incident Management & Communication

Incident Management & Communication

Determines if University IT Security, University Security,

Determines if University IT Security, University Security,

Identifies relevant vendors that may be needed.

Notifies IMOC (3). Provides IMOC with a situation brief

Incident Management & Communication

Normal Business Hours (8:00am 5:00pm)

If the IT Center is open, provide customers with

department(s) was notified and is aware of the

If IT Center is open, provide customers with IMOC-

supplied information. If neither center is open, MOC

notified and aware of the situation.

on Call to forward Operations phone numbers 59194 & 5-9195 to 5-2000

situation (Are fixes working? Are users still

Incident Management & Communication

Incident Manager On Call IMOC List

Incident Management & Communication

Normal Business Hours (8:00am 5:00pm)

Evaluates the situation and gathers all the facts from

1-866-945-2255 Access Code: 608965#

Evaluates the situation and gathers all the facts from

Notifies CIO and Directors for after hour incidents.

1-866-945-2255 Access Code: 608965#

Initiates IMOC Phone Bridge, if necessary

Coordinates CHRON and scribe duties. Calls in staff

Contact DCS Production Control MOC (DCS

Contact SMS group to setup Service Monitoring

Communicates with key people & customers during

Prepares a communication for release to University

Meets next morning with communications manager

Scheduled IMOC (not acting IMOC) schedules and

Incident Management & Communication

Technicians will be required to be on-site unless

Technicians will be required to be on-site

Participates in discussions lead by MOC and IMOC.

May be onsite or working from home as

Provides support to technical teams.

help resolve the incident.

Incident Management & Communication

May be onsite or working from home as determined

Incident Management & Communication

Receives details about incident from IMOC.

Receives details about incident from

Provides incident brief to Provost and President

Incident Management & Communication

Gathers details about incident.

timeframe through identified channels/Working with

Incident Management & Communication

Takes detailed notes during event to help complete

the CHRON and serve as a record of the event.

Uses guidelines for communications to customers

Uses guidelines for communications to customers

when responding to calls that may come in from

when responding to calls that may come in from