Module 4

A Business Model For Continuity Planning

4-1

Outline
The Business Continuity Management Program Components Of The Emergency Response Plan Components Of The Disaster Recovery Plan Components Of The Business Contingency Plan Components Of The Crisis Management Plan

4-2

The Business Continuity Management Program

The interruption of fundamental business processes for any extended period of time could have a debilitating affect on our basic infrastructure.and our way of life
E-Commerce Private and Business Online Trading Cash Advances At ATM Machines Personal and Commercial Online Banking Purchases By Credit Cards Just In Time Inventories

Communications
Student Services Grants and Endowments General Administration & Finance
4-3

The Business Continuity Management Program

ERP DRP BCP CMP

ERP Emergency Response Plan: Steps Taken To Immediately Respond To An

Event, Ensure Personnel Safety, Minimize Further Impact To Assets, And Make Proper Notifications.

DRP Disaster Recovery Plan: Steps Taken To Restore Specified Infrastructure

Requirements Such As Information Systems, Clinical Equipment Environments, Internal And External Network Connections, And Data Structures Utilizing Alternate Resources For Hardware, Software, Data, and Networks.

BCP Business Contingency Plan: Steps Taken To Restore Alternate Business

Processes In The Event That Automated Processes Or Business Infrastructures Are Unavailable, Employing Documented Workaround And/Or Manual Procedures And Alternate Resources.

CMP Crisis Management Plan: Steps Taken To Manage The Event To Ensure That
Order Is Maintained, Employee Assistance Is Being Provided, Proper Information Is Being Disseminated By Appropriate Representatives, Action Items Are Effectively Escalated, And Ongoing Internal And External Notifications Are Consistent.
4-4

The Business Continuity Management Program

ERP DRP BCP CMP

Working Components
Response - Notifications, assessments, escalations, declarations, etc. (established procedures) Recovery/Relocation - Mobilization, Quick-ship, Infrastructure, Network and Data recovery, etc.. Movement of staff, patients, and business units to alternate facilities (flexibility and adaptability)

Resumption - of Business Operations and I.T. functionality (business units must synch up processes and resume operations at an alternate site)
Re-assessment - of situation, strategies, planning, reactions (input from all involved parties) Restoration - Movement back to home site and/or normal operations (reconstituted at restored site by I.T. and/or Business Units

4-5

Components Of The Emergency Response Plan

First Response

Notification

Assessment and Status

Escalations

Declarations

Personnel Safety Damage Mitigation Local Authorities Evacuations

Initial Notifications Telephone Trees Command Center Assembly

Damage Assessment Initial Status Reporting Secondary Notifications

Organizational Committees Local Authorities Vendors Customers Media

Checklists Scripts Procedures Contact Lists Vendors Mobilization

4-6

Components Of The Emergency Response Plan

Assessing Impact
Cat 3 A major disruption in service affecting a subset of users or systems deemed to be non-critical for alternate site recovery. (Component outage / Local Recovery) Cat 2 Major disruption to one or more entities. Recovery of services at prime location is < 24 hours. Restoration at alternate site more lengthy than repairing at prime. (Multi-system and/ or user / Local Recovery <24 hours) Cat 1 Total system(s) outage affecting multiple entities, systems, and customers. Anticipated recovery at prime location impossible or expected to exceed 24 hours. Recovery at alternate site is more rapid than at prime location. (Multi-system and/or user / Local Recovery >24 hours ; Recover Remotely)
4-7

Components Of The Disaster Recovery Plan Disaster Recovery Planning

Steps taken to restore specified infrastructure requirements such as Information Systems, business equipment environments, internal and external network connections, and data structures utilizing alternate resources for hardware, software, data, and networks. What To Do When The Computer Goes Down

4-8

Components Of The Disaster Recovery Plan Disaster Recovery Is

The successful recovery of mission-critical I.T. services to the customer community in response to a crisis Flexible Response To A Crisis Place to Recover (Location/Equipment/Network) Defined Recovery Set (Critical Components) Reliable Backups Test Maintain Test Service Continuation

Disaster Recovery is NOT..

Recovery of full environment A business continuity plan A replacement for conventional service plans A trivial decision

4-9

Components Of The Disaster Recovery Plan

I.S. Infrastructure

Applications Analysis

Network Infrastructure

Opens Systems

Documentation

Hardware Systems

Databases
TSO/CICS Test Criteria/Objectives

Questionnaires Interviews Analysis Documented Profiles Test Criteria/Objectives Recovery Plans

Owned Equipment DR Vendor Equipment Connectivity Requirements Test Criteria/Objectives Remote Access Parameters Define rogue FTPs Identified Network Services

LDAP DNS Email Intranet/Internet Gateway Servers Test Criteria/Objectives

Checklists Scripts Procedures Contact Lists Test Criteria/Objectives

4 - 10

Components Of The Disaster Recovery Plan

Build The I.T. Infrastructure Recovery
Immediate Actions Personnel Safety Damage Mitigation Reporting Procedures Hardware Software Data Telecomm Reporting

Three Phased Approach

Documentation

Emergency Management
Coordinate Control Fund Approve

Alternate Capabilities Available Data

Recovery

Refurbish Replace Construct Return

4 - 11

Components Of The Disaster Recovery Plan

Applications Analysis
Issue Applications Surveys Complete the Surveys Analyze the data Identify key infrastructure components to establish computing environment. Incorporate all data into Application Recovery Plans and Application Recovery Timelines Develop a critical path timeline which will document: The order in which applications will be restored The dependencies among the applications (interfaces) The Recovery Time Objectives as influenced by application dependencies Test the Application Recovery Plans and Timelines Develop the individual application recovery plans (Profiles or Blueprints)

TEST

TEST

TEST

TEST

4 - 12

Components Of The Disaster Recovery Plan

I.T. Requirements
RECOVERY TIME OBJECTIVE: (RTO) The period of time in which systems, applications, or I.T. functions must be recovered after an outage. RTO's are often used as the basis for the development of recovery strategies, and as a determinant as to whether or not to implement the recovery strategies during a disaster situation. RECOVERY POINT OBJECTIVE: (RPO) The point in time to which systems and data must be restored after an outage. RPO's are often used as the basis for the development of backup strategies, and as a determinant of the amount of data that may need to be recreated after the systems or functions have been recovered.

4 - 13

Components Of The Disaster Recovery Plan

Are You HOT, COLD, or WARM ?

HOTSITE ~ An alternate facility that already has in place the

computer, telecommunications, and environmental infrastructure required to recover critical business functions or information systems.

WARM SITE ~ An alternate processing site which is equipped

with some hardware, and communications interfaces, electrical and environmental conditioning which is only capable of providing backup after additional provisioning, software or customization is performed

COLD SITE ~ An alternate site that contains physical space

and building infrastructure that must be provisioned at time of disaster to support recovery operations. SIMILAR TERMS: Shell Site; Backup Site; Recovery Site; Alternate Site
4 - 14

Components Of The Disaster Recovery Plan

Are You HOT, COLD, or WARM ?
Print Server Application Server Vendor Database CITRIX Server Database Server General Services Systems

Application Server Application Server Sybase Server

Assessor Database

General LedgerAccts Payable-Accts Rcvbls

Internal Solution

External Solution

Open Systems DR Model

-Data Replication To Local Storage -Failover and/or Quick Recovery -Local Connections For High Volume -Local AND Remote Recovery

Centralized DR Model
-Traditional Offsite Storage -Hotsite Location Approx 100 Miles From Primary Site -T1 Connection Between Hotsite and Local Internal Solution 4 - 15

Components Of The Disaster Recovery Plan

Are You HOT, COLD, or WARM ?
File Recovery data restore from local or offsite backups High Availability data and hardware restore to local standby equipment (i.e., test, QA, Dev) Disaster Tolerance Failover to available equipment internal to the organization, but remote to the primary site (i.e., test, QA, Dev other campus location) Disaster Recovery Traditional DR model with hotsite and offsite storage locations Note: Regardless of the strategy employed..backed up data should still be copied to offline media and rotated to offsite storage
4 - 16

Components Of The Emergency Response Plan

Assessing Impact
Ranking Highly Critical 1 Critical 2 Characteristic Adverse Impact To Business Operations Some Impact To Clinical Care and/or Patient Safety. Possible Adverse Impact to Critical Govt. Services Suspension of some business operations at either the Hospital but no direct impact on Clinical Care, Patient Safety, Critical Govt. Services Recovery Window Less Than 12 Hours Recovery Strategy High Availability Disaster Tolerance Failover. Backups via SAN Traditional Hotsite Strategy, possibly coupled with internal, open systems solution. Backups via SAN and tape Traditional Hotsite Strategy, possibly coupled with internal, open systems solution/. Backups via SAN and tape

1 To 24 Hours

Important 3

24-72 Hours

Deferrable 4

Minor inconveniences to business operations

> 72 Hours. May not be recovered until migration to the warm site, or the original site is repaired

Standard backups and restore procedures typical of component failures. Backups via tape.

4 - 17

Components Of The Disaster Recovery Plan

Centralized
I.T. Disaster Recovery Plan

Enterprise Services

Engineering Services

Applications

Network Services

Desktop Computing

Decentralized
I.T. Disaster Recovery Plans

Data Ctr 1

Data Ctr 2

Data Ctr 3

Telelcomm Cabinet

Network POP

Open System 1

Open System 2

Remote Site 1

Remote Site 2

Relays/Router s/Switches

Develop the plans by department execute the plans by location

4 - 18

Components Of The Disaster Recovery Plan

Data Collection Via Templates and Data Entry Screens

Employees Equipment Vendors Documents Tasks

Database

Plans Can Be Developed And Maintained By Department..But Will Be Executed By Locations

DRPs By Organization DRPs By Building/Flr DRPs By Department DRPs By Application

I.T. Disaster Recovery Plan Table of Contents I.T. Overview.1 Call List/Escalation Tree..8 Team Summaries.12 Response Phase Checklist..15 Recovery Phase Checklist..17 Resumption Phase Checklist..24 Restoration Phase Checklist..28 Equipment Inventory.38 Alternate Site Disaster Declaration Procedures..39 Mobilization Procedures43

The Impact Of An I.T. Outage Affects Not Only I.T. But All Departments and Business Functions Supported By I.T.

4 - 19

Components Of The Business Contingency Plan

DRP BCP

DRP Disaster Recovery Plan: Steps taken to restore specified infrastructure requirements such as Information Systems, business equipment environments, internal and external network connections, and data structures utilizing alternate resources for hardware, software, data, and networks.
- Hardware - Data and Data Structures - Networks - Production Support - System Software - Applications - Desktop Services

BCP Business Contingency Plan: Steps taken to restore alternate business processes in the event that automated processes or business infrastructures are unavailable, employing documented workaround and/or manual procedures and alternate resources.
- Relocation of Personnel - Availability of remote support services and network connections - Contingency office space
4 - 20

Components Of The Business Contingency Plan Business Contingency Planning

Steps taken to restore alternate business processes in the event that automated processes or business infrastructures are unavailable, employing documented workaround and/or manual procedures and alternate resources.

What To Do While The Computer Is Down

4 - 21

Components Of The Business Contingency Plan

Business Contingency Planning Is
The successful response to an interruption in normal operating procedures and thus services to the customer community Flexible Response To A Crisis Place to Initiate Contingency Operations (Systems/Network/Location/Personnel/Equipm ent) Documented Systems Workaround Procedures Alternate Resources

Business Continuity is NOT..

Disaster Recovery, Emergency Preparedness, or Crisis Management A Permanent Solution An I.T. Issue
4 - 22

Components Of The Business Contingency Plan

Alternate Mobilization Processes

Alternate Resources

Business Documentation Resumption

I.T. Workarounds Logistics Personnel & Skill Sets Manual Business Processes Location(s) Facilities Alternate Data Capture Transportation Vendors Personnel Hardware/Software Communications

Logistics Transition Back To I.T. Validation/Audit Normal Operations Business Cycles

Procedures Logistical Support Forms Contact Lists

4 - 23

Components Of The Business Contingency Plan

Business Continuity Planning Scenarios

Loss of I.T Services or Resources Loss of Functional Support Personnel Loss of Facility Loss of Network Connectivity Loss of Voice Communications Loss of 3rd Party Suppliers Loss of Business Partners
4 - 24

Components Of The Business Contingency Plan

Build Contingency Plans

Identify key functional components to establish the business environment Define the alternate process requirements for each component Ensure interdependent business processes are identified and can be synched up Define minimal processing requirements for each component

TEST

TEST

TEST

TEST
4 - 25

Components Of The Business Contingency Plan

Business Recovery Requirements
RECOVERY TIME OBJECTIVE: (RTO) When do I have to have an alternate process in place to address loss of primary functions (I.T. and otherwise) ? RECOVERY POINT OBJECTIVE: (RPO) How current does my information have to be when normal processes are resumed ?

4 - 26

Components Of The Business Contingency Plan

Organization-Wide Business Contingency Plans By Department

Department 1
Location 1 Location 2 Location 3 Location 4

Department 2
Location 1 Location 2 Location 3 Location 4

Department 3
Location 1 Location 2 Location 3 Location 4

Department 4
Location 1 Location 2 Location 3 Location 4

Department 5
Location 1 Location 2 Location 3 Location 4

Department 6
Location 1 Location 2 Location 3 Location 4

Department 7
Location 1 Location 2 Location 3 Location 4

Develop the Plans by department

4 - 27

Components Of The Business Contingency Plan

Organization-Wide Business Contingency Plans By Location

Location 1
1st Flr Unit #1 Unit #2 Unit #3 Unit #4 2nd Flr Unit #1 Unit #2 Unit #3 Unit #4 3rd Flr Unit #1 Unit #2 Unit #3 Unit #4 4th Flr Unit #1 Unit #2 Unit #3 Unit #4

Location 2
1st Flr Unit #1 Unit #2 Unit #3 Unit #4 2nd Flr Unit #1 Unit #2 Unit #3 Unit #4 3rd Flr Unit #1 Unit #2 Unit #3 Unit #4 4th Flr Unit #1 Unit #2 Unit #3 Unit #4

Location 3
1st Flr Unit #1 Unit #2 Unit #3 Unit #4 2nd Flr Unit #1 Unit #2 Unit #3 Unit #4 3rd Flr Unit #1 Unit #2 Unit #3 Unit #4 4th Flr Unit #1 Unit #2 Unit #3 Unit #4

Location 4
1st Flr Unit #1 Unit #2 Unit #3 Unit #4 2nd Flr Unit #1 Unit #2 Unit #3 Unit #4 3rd Flr Unit #1 Unit #2 Unit #3 Unit #4 4th Flr Unit #1 Unit #2 Unit #3 Unit #4

Location 5
1st Flr Unit #1 Unit #2 Unit #3 Unit #4 2nd Flr Unit #1 Unit #2 Unit #3 Unit #4 3rd Flr Unit #1 Unit #2 Unit #3 Unit #4 4th Flr Unit #1 Unit #2 Unit #3 Unit #4

Location 6
1st Flr Unit #1 Unit #2 Unit #3 Unit #4 2nd Flr Unit #1 Unit #2 Unit #3 Unit #4 3rd Flr Unit #1 Unit #2 Unit #3 Unit #4 4th Flr Unit #1 Unit #2 Unit #3 Unit #4

Location 7
1st Flr Unit #1 Unit #2 Unit #3 Unit #4 2nd Flr Unit #1 Unit #2 Unit #3 Unit #4 3rd Flr Unit #1 Unit #2 Unit #3 Unit #4 4th Flr Unit #1 Unit #2 Unit #3 Unit #4

Execute the Plans by Location

4 - 28

Components Of The Business Contingency Plan

Data Collection Via Templates and Data Entry Screens

People Processes Equipment Documents Tasks

Database

Plans Can Be Developed And Maintained By Department..But Will Be Executed By Locations

BCPs By Organization BCPs By Building/Flr BCPs By Department BCPs By System

I.T. Disaster Recovery Plan Table of Contents Business Unit Overview.1 Call List/Escalation Tree..8 Team Summaries.12 Response Phase Checklist..15 Mobilization Phase Checklist.17 Resumption Phase Checklist..24 Restoration Phase Checklist..28 Equipment Inventory.38 Alternate Site Disaster Declaration Procedures..39 Mobilization Procedures43

Events Dont Occur By Department.They Occur By Location

4 - 29

Components Of The Business Contingency Plan

Centralized Administration and Coordination Decentralized Development, Maintenance and Execution
Web-Enabled 24 x 7 x 365 access from anywhere with VPN connection Automated progress reporting during Plans development, maintenance, and execution Define relationship between BCPs and DRPs (RTO and RPO) Capable of expanding to include ERP and CMP Real-time updating to a single database, not multiple Plans Version Control on all Plans Concurrent Plan development Issue Templates

Import Templates
Develop BCPs Flexibility when producing BCPs..or executing BCPs Show me all Plans by Department. Show me all Plans by Building..

Show me all Plans by Building, by Floor..

Show me all Plans by Building, by Floor, by Department
4 - 30

Components Of The Business Contingency Plan

Negotiate The Service Level Agreement Between I.T. And Business Operations
Use Both The I.T. And Business RTO & RPO As The Basis
Disaster Recovery Plan Test Results Quantify Timelines Business Contingency Plan Exercises Qualify Impact

I.T. Capabilities Improve Timelines But At A Cost

Business Contingencies Reduce Impact - But Require I.T. Capabilities
Criticality Rankings

Systems Recovery Sequencing

Business Process Prioritization I.T. and Business Process Timelines Negotiated RTO and RPO
4 - 31

Components Of The Business Contingency Plan

Results
I.T. Better Understands The Customers Issues and Requirements
I.T. Obtains A Clearly Documented Set Of Customer Expectations For DRPs - Clarify and Justify Budget Forecasts - Establishes Specific Test Objectives

- Ensure Active Customer Involvement In Testing & Recovery Processes

Business Units Better Understand The Role Of I.T. In The Contingency Process Business Units Obtain A Set Of Parameters From Which To Develop their BCPs - Workaround Procedures During Downtime - Procedures For Capturing Lost Transactions From Downtime and During Recovery - Restoration Of Normal Environments
4 - 32

Components Of The Business Contingency Plan

Questions/Issues to consider:
Was the original disaster recovery initiative driven by I.T., business units, or Sr Management ? What are Sr. Managements expectations with respect to continuity of service ? Has a business impact analysis been done on some or all of the business units ? Quantified Impact Quantified Cost of DRP vs. Impact of Risk Acceptable Downtime Criteria (services, workstations, etc.) What discussions have taken place between I.T. and critical business units ? State of DRP State of BCP Quantified RTOs and RPOs Systems Development Life Cycles What are the business units expectation with respect to current I.T. RTOs and RTOs ? Are they driven by I.T. technologies or business requirements ? Are there current SLAs ? Service Center Problem/Change Control Network Outage Response Time Are regulatory compliance, industry certification, or audit issues creating more compelling reasons for addressing DRP and BCP ? 4 - 33

Components Of The Crisis Management Plan

Event Analysis

Reaction Planning

Communications

Documentation

Catastrophic Events Criminal Events Disease/Epidemics Technological or Safety Utility or Structural Weather Personal vs. Professional

Emotional Assistance Addressing Traumatic Stress Family Assistance Pgms Professional Assistance Provide Information & Counseling Post Incident Follow-up

Local Media Employees Local Authorities Openness Accuracy Balance Designate a point person Continuous Flow

Employee Checklists And Action Plans Press Release Data Employee Notification Mechanisms

4 - 34

Components Of The Crisis Management Plan 9/11/01 irrevocably changed the landscape of Crisis Management Planning forever
Planning: Single Event Institution & Regional Planning: All Hazards Plans

Limited Numbers Limited Type

Potentially Huge Numbers Unlimited Options

The new era of Terrorism..Weapons of mass destruction

Biological Chemical Radiological Nuclear Explosives and Incendiary devices
4 - 35

Components Of The Crisis Management Plan

Crisis Management Preparedness Key Elements
1. Identification of vulnerabilities 2. Performance of regional threat assessment 3. Assessment of system resources 4. Communications infrastructure 5. Standardization of plans 6. Dissemination of information

7. Analysis of system Surge Capacity

8. Collaboration with federal, state, local agencies

4 - 36

Components Of The Crisis Management Plan

Vulnerability Assessment

GENERAL
Communications Timely/Accurate Information Lack of coordination between entities and with external agencies Inadequate threat awareness

DISASTER-RELATED
Loss/Lack of communications Loss or degradation of physical plant Depletion of resources Loss of staff Training
4 - 37

Components Of The Crisis Management Plan

Crisis Management Preparedness Key Elements
Factors in determining regional risks: Landmarks / symbolic Ports Proximity to key cities

Large events
Focus on politics or finance Requires input from multiple sources

Law Enforcement, Fire, Military, Federal Agencies, Emergency Management Agencies

Use the threat assessment to direct planning efforts
4 - 38

Components Of The Crisis Management Plan

Vulnerability Number 1- Communications Infrastructure The number one problem identified in disaster drills was communications A robust communications infrastructure is vital during a crisis
Analyze existing communications infrastructure Perform a cost-benefit analysis of infrastructure augmentation Implement recommendations in order to create communications redundancy

Vulnerability Number 2- Crisis Plans Were Not Uniform

All of the components of the Enterprise had disaster plans, but they were not uniform All the plans had been developed in isolation of the remainder of the enterprise Needed to develop some type of Incident Command System/Emergency Command Center
Common language Predictable chain of management Creation of Alert Levels Translation of Alert Levels to Operational Levels with disaster preparedness standards
4 - 39

Components Of The Crisis Management Plan Regional Collaboration

Who does what?? Who calls whom??
Local

Federal

Fire/EMS/OES Law Enforcement Health Dept./Hazmat Hospitals State Health Dept. State OES/DHS Hospitals

Federal Emergency Mgmt Agency CDC Military Collaboration Individual Plans Supplement/Complement Broader Plans Clinical Care Response Public Health Response
4 - 40

Private Sector

State

The Business Continuity Management Program

When the issues surrounding both I.T. Disaster Recovery Plans and Business Unit Business Contingency Plans come together what is at stake becomes much clearer, and each can understand the others objectives and expectations. Only then can a total Business Continuation Program be effective.

And if the organization has an effective Business Continuation Program, not only can it assure that its goals and objectives will be met..but will also become a valued partner in the protection of the larger infrastructure..

4 - 41