http://www.rfgonline.

com/

Thursday, June 24, 2004

Network Policies, Procedures, and Processes (PPPs) Why They Matter!

RFG believes numerous enterprise networks continue to move unsuspectingly into a high-risk category because of network troubleshooting methodologies that rely primarily on the availability and expertise of network staff. In todays environment, qualified technical personnel are a challenge to find, expensive to hire, and difficult to retain. At the same time, network-related problems have become more difficult to diagnosis and correct as business application performance metrics become increasingly more stringent. Consequently, those knowledge bases that identify the approved processes and troubleshooting procedures required to comply with network service policy directive must be available, preferably online, on a 24x7x365 basis. IT executives should verify that those PPPs necessary to maintaining the required network availability and performance levels for critical business applications are available, complete, and up to date.

Business Imperatives:

PPPs are essential to verify that appropriate and timely actions are taken in the event of a major network problem or service disruption. Furthermore, PPPs should be reviewed and updated at least annually to reflect changes in architecture, network technology, or other enterprise business needs. IT executives should begin the PPP process by focusing on potential major failure areas that could affect the requisite levels of availability and connectivity mandated for vital applications. Fault and performance management represent two of the primary areas that PPPs should address because problems in either or both of these areas can escalate quickly. IT executives should focus on developing those PPPs that can address major network faults as well as performance degradations that must be quickly identified and resolved. Although network PPPs are available in many enterprises, their overall effectiveness and usefulness must be validated and tested periodically. While this may not seem important for many benign day-to-day network problems, the situation changes drastically in the event of a major failure or disaster. IT executives should verify that tested PPPs are available and are effective for those situations that could generate a major network failure or cause recurring service disruptions. These could negatively affect business application connectivity and user productivity.

In todays Byzantine networking environment, the availability of actionable information is essential to maintain the required performance levels users typically expect. Knowing exactly what actions to take and when to take them is of vital importance, especially in the event of a major network disruption. Conversely, if this knowledge base of information is not readily available, enterprises could incur sizeable losses in both revenue and productivity if strategic business applications are no longer accessible. Consequently, enterprises should capture the technical troubleshooting expertise available from their staff and utilize this information to develop online PPPs focused on maintaining and restoring the required levels of network availability and performance. The fact is that enterprises should avoid the risks involved with depending solely upon the availability and skill sets of certain technical personnel if a major network failure transpires. There have been numerous instances where a major network failure occurred and the key network engineers capable of troubleshooting the problem and restoring services were unavailable for many hours. All PPPs that must be referenced and followed in the event of a network failure must be readily available to all authorized personnel or they offer little value to anyone. They must also supply clear, actionable information that facilitates the restoration of network services for business critical applications. In addition, IT executives should carefully consider where PPPs are stored and how access to them is provided. The availability of, and access to, this important information must be guaranteed especially during a catastrophic event. If there are internal service level agreements (SLAs) in effect, PPPs should direct personnel to take appropriate actions to meet the guarantees stated in the relevant SLA documents. (See the RFG Research Note "Why Internal Service Level Agreements (SLAs) Are Good for Users but Maybe Not for IT.") Although network engineers may have the experience necessary to take the appropriate actions to troubleshoot and solve the majority of network problems, those troubleshooting action sequences required to maintain critical business application performance mandates should be documented and included in the network PPPs. In addition, all sanctioned troubleshooting techniques and processes should be made available for those personnel tasked with maintaining and restoring network services. (See the RFG Research Note "Are Network Technology Subject Matter Experts (SMEs) An Endangered Species?") Crucial PPPs that deal with network recovery and restoration should be available online and accessible from both corporate as well as remote locations. In addition, appropriate security measures should be implemented, such as a two-stage authentication process, to validate all users and to prevent unauthorized access or tampering. RFG believes a best practice methodology for getting the PPP process kick-started is to identify those corporate assets that must be protected in the event of any network related disaster. For example, if connectivity to remote business offices is deemed a high priority, the methodology required to restore service in the event of a major network failure or related disaster must be validated, documented, available, and tested. IT executives should first identify those network assets that require high levels of availability and reliability and establish and validate the PPPs necessary to meet the specific QoS objectives. (See the RFG Research Note "Quality of Service (QoS) The WAN Tune-Up Tool.") Moreover, most network failures and disruptions provide early warnings that can be detected by proactive fault and management tools responsible for assuring the continuous health of the network. However, these tools must be configured to detect those vital assets identified by current network management policies and procedures. (See the RFG Research Note "Why Network Management Systems Can Fail To Provide Enterprise Value.") IT executives should review the configuration of all network management tools to verify that they are capable of providing an early warning for a majority of network failures and disruptions. At the same time, if this evaluation uncovers strategic areas that are not fully protected by network management tools, appropriate policies and procedures should be developed to facilitate a quick and timely service restoration. There are a variety of catastrophic events that could totally disrupt an enterprise's entire communication network, or at least those network elements considered critical to business. IT executives should identify and address those potential single points of failure that could affect any vital communication links and prevent users from accessing important business applications. An example of a catastrophic event is a central office or a partial central office failure. Although central offices tend to be hardened against most failures, they do occur, leaving many enterprise data and telecommunication networks in a state of chaos until service can be restored. IT executives should identify and address all single points of failure that could cripple or prevent access to business applications. In addition, appropriate back-up solutions, such as using an alternate carrier or moving to satellite services, should be evaluated, tested, and on standby if certain business applications require reliable backup solutions. (See the RFG Research Note "An Update On The Importance Of Business Application Profiles.") Examples of the PPPs an enterprise could utilize to address a central office failure are illustrated below. These example PPPs also help to illustrate the complexity and dependencies that must be carefully addressed to adequately support and comply with enterprise network policy directives. Enterprise Network Policy Statement For A Central Office (CO) Failure

Business applications designated as critical assets that could be severely compromised or affected by a CO- related failure must be identified by the network organization. Furthermore, at least eighty percent of all applications deemed critical to business must be restored to an acceptable level of service and performance within thirty minutes of detection of a central office failure. All enterprise network policies must be available online to network staff personnel on a 24x7x365 basis with appropriate security measures implemented as dictated by security operations.

Procedures and Processes For Dealing With A CO Failure

All appropriate network management tools must be configured to track the performance and fault status of vital applications critical to business that traverse a single CO. Network management systems must provide accurate event correlation capable of quickly detecting a CO failure. Network engineering is responsible for working with service providers to verify and implement appropriate notification systems so that a CO failure is identified as quickly as possible. Network engineering is responsible for identifying all high-risk central office sites and working with the specific providers to implement backup strategies to protect connectivity to all critical business applications. Network engineering is responsible for stipulating and verifying the performance levels that can be achieved in a backup scenario as well as identifying their impact on both users and the application. The director of network engineering is responsible for proposing any new tools, and validating and updating current network management tools, to ensure hardware and software configurations comply with enterprise policy. In the event of a CO failure, affected remote offices must be notified by IT and directed to switch, or be automatically switched, either to dial-back up or DSL facility depending upon location and service availability. IT must deliver notification of a CO failure via e-mail on an unaffected communication circuit or via the voice network or wireless cellular service. All notifications must be delivered to affected locations within fifteen minutes after a CO failure is detected and validated. The director of IT is responsible for verifying that business critical application users are aware of their roles and responsibilities in the event that a CO failure occurs. IT is responsible for testing and verifying that remote office user notification systems and backup facilities are operational and available in the event of a CO failure. All procedures that deal with a CO failure must be reviewed and updated at least annually by the network engineering staff and approved by the director of network operations. Ancillary procedures and processes that could potentially affect full compliance with this enterprise policy must be identified, reviewed, and approved by the vice president of network services to verify they do not affect recovery mandates.

Finally, even though comprehensive network related PPPs might be available in many enterprises, the overall effectiveness and usefulness must still be validated and tested periodically. In fact, keeping track of network architecture changes and communication technologies, as well as vendor service level agreements (SLAs), are all essential to constructing and maintaining PPPs essential for maintaining reliable network connectivity. IT executives should focus on those business critical applications that must maintain levels of high-availability as their first priority. If the appropriate PPPs are constructed, tested, and validated to protect this category of applications, it is likely that the enterprise will survive the majority of network problems and failures that occur. IT executives should discuss appropriate PPPs with those key vendors and service providers that are involved in the end to end delivery of reliable access to critical business applications. It is important that this group of vendors and service providers are aware of the performance levels that critical business application must deliver and maintain in order to recommend any processes or procedures for possible improvements. Readily accessible PPPs should also be created and utilized in other critical IT areas where relying upon the availability and skill sets of certain personnel creates an unacceptable risk situation. RFG believes PPPs are essential for maintaining the requisite performance levels key business applications demand and for providing appropriate backup in the event of a major network failure or ancillary disruption. In addition, PPPs also allow IT executives to verify that best practices are being utilized to keep the network operational and to deliver the required levels of business application performance. IT executives should review existing PPPs, or construct new PPPs, to protect business critical applications from those network related failures and disruptions classified as enterprise crippling. RFG analyst John Stehman wrote this Research Note. Interested readers should contact RFG Client Services to arrange further discussion or an interview with Mr. Stehman.

RFG Research Notes provide concise, high-level analysis and recommendations on specific topics of interest to enterprise IT executives. The Notes also provide a framework for further detailed Inquiries by RFG clients, and for follow-up presentations and workshops by RFG research staff available to all interested IT decision-makers. For more information, contact Client Services by telephone at (US) +203/291-6900 or by email at clientservices@rfgonline.com.

Copyright 2004 Robert Frances Group, Inc. All rights reserved. Agenda products are published by Robert Frances Group, Inc., 22 Crescent Road, Westport, CT 06880. Telephone 203/291-6900. Facsimile 203/291-6906. http://www.rfgonline.com/. This publication and all Agenda publications may not be reproduced in any form or by any electronic or mechanical means without prior written permission. The information and materials presented herein represent to the best of our knowledge true and accurate information as of date of publication. It nevertheless is being provided on an "as is" basis. Reprints are available.