Beruflich Dokumente
Kultur Dokumente
www.nortel.com
NN10450-900 .
Carrier VoIP Release: (I)CVM12 Publication: NN10450-900 Document release date: 3 October 2008 Copyright 2008 Nortel Networks All Rights Reserved. Printed in Canada, the United States of America, and the United Kingdom LEGAL NOTICE While the information in this document is believed to be accurate and reliable, except as otherwise expressly agreed to in writing NORTEL PROVIDES THIS DOCUMENT "AS IS" WITHOUT WARRANTY OR CONDITION OF ANY KIND, EITHER EXPRESS OR IMPLIED. The information and/or products described in this document are subject to change without notice. Nortel, the Nortel logo, Business Made Simple and the Globemark are trademarks of Nortel. All other trademarks are the property of their respective owners.
Contents
New in this release
Features 9 Other changes 9
Introduction Disaster recovery fundamentals Disaster recovery configuration Preparation for disaster recovery
11 13 55 57
Recovering the SPFS system if updisk daemon dies 58 Installing the DDMS 59 Installing optional software on a Core and Billing Manager 64 Installing the OM Data Delivery application 68 Installing the Passport Log Streamer application 71 Installing the SBA and AFT software packages 73 Installing the GR740PT application server 74 Configuring RTB for a billing stream 76 SBRM core access 81 Configuring core access for SBRM through the Nortel Communication Server 2000 Core Manager 84 Creating the backup user ID on the core for SBRM 86 Applying software packages on a Core and Billing Manager using the CBMMTC interface 88 Cloning the image of the active node to the inactive node 91 Installing the Backup Restore Manager software 94 Configuring SSH between the backup restore manager servers 96 Configuring SBA backup volumes on the core 102 Starting or stopping the automated synchronous backup restore manager service 107 Invoking the network-level backup (SBRM-bkmgr) through telnet 110 Invoking the device-level backup (DBRM-bkmgr) through telnet 115 Backing up an XA-Core office image from disk to tape 119 Backing up Call Agent 126 Backing up the SST 129
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
4 Rescheduling backup time 132 Backing up security related files manually 134 Saving the Ethernet Routing Switch 8600 boot configuration file 136 Configuring the automated synchronous backup restore manager service 138 Configuring Automated INI file backup on a Media Server 2000 node 142 Changing SNMP community string passwords 145 Backing up INI files for all nodes 147 Displaying Media Server 2010 current node configuration 148 Backing up the APS-specific Oracle database and application files 150 Using the command line interface to perform backups 153 Performing a backup of Oracle/MySQL data on an SPFS-based server 156 Backing up UAS configuration files 160 Performing a manual backup of the Policy Controller database 162 Performing a backup of file systems on an SPFS-based server 164 Taking a full IEMS backup 169 Backing up an SPFS-based security client 171 Backing up a Multiservice Switch or Media Gateway device 172 Performing a backup operation 178 Creating system image backup tapes (S-tapes) manually 181 Displaying actions a user is authorized to perform 193 Logging in to the Nortel CS 2000 Core Manager 195 Preparation for SBA installation and configuration 197 Configuring SBA backup volumes on a billing stream 220 Accessing the MATE 222 Administration: Creating Disaster Recovery Floppy Disks 224 Power-On and boot a Policy Controller unit 225 Viewing the operational status of the NCGL platform 226 Viewing the operational status of the SIP Gateway application 228 Verifying synchronization status 232 Booting the XA-Core from a Reset Terminal 233 Resetting the Ethernet Routing Switch 8600 using a saved configuration file 236 Restoring a Call Agent 237 Restoring Session Server Trunks 240 Restoring a database to a Session Server unit 241 Restoring audio files to a UAS node 242 Restoring audio files to a Media Server 2000 Series node 243 Restoring the APS-specific Oracle database and application files 244 Restoring a Multiservice Switch or Media Gateway device 247 Performing a full system restore 253 Restoring a system configuration from a data snapshot 254 Restoring Core Element Manager data 260 Restoring the central security server 262 Performing a full restore of the software from S-tape 265 Performing a partial restore of the software from S-tape 270
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
5 Recovering backup files from lost backup volumes 275 Restoring the Oracle/MySQL data on an SPFS-based server 277 Clearing the MATE alarm 281 Stopping the SAM21 Manager server application 282 Stopping the SESM server application 284 Stopping the NPM server application 286 Stopping the APS server application 288 Starting the APS server application 290 Starting the SESM server application 293 Starting the SAM21 Manager server application 295 Starting the NPM server application 297 Performing a full system restore on an SPFS-based server 299 High-availability configuration (two servers) 302 Locking a GWC card 305 Locking the Call Agent 308 Installing the remote backup server 310 Scheduling automatic backups on the remote server 315 Viewing configuration information for remote server backups 318 Performing a manual backup of the target server 320 Viewing logs from a remote backup 322 Initiating a recovery back to the cluster 323 Initiating a switch over to the remote backup server 325 Disabling Ethernet Routing Switch 8600 WAN Edge routing 327 Disabling and re-enabling the router ports on the Ethernet Routing Switch 8600 329
331
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
6 Shutting down the IEMS server 359 Powering down the XA-Core 360 Performing a partial power down of the MG 9000 363 Performing a partial power down of the Media Gateway/Multiservice Switch 15000 364 Performing a partial power down of the USP 368 Performing a partial power down of the Ethernet Routing Switch 8600 369 Performing a partial power down of the Netra 240 server 370 Performing a partial power down of the XA-Core 371
Network power up
Powering up the XA-Core 380 Powering up the Message Switch 396 Powering up the Enhanced Network 399 Powering up an Ethernet Routing Switch 8600 401 Powering up the Link Peripheral Processors 402 Powering up the SDM 405 Powering up a Netra 240 server 406 Powering up the Border Control Point Manager 407 Powering up the SAMF frame 408 Powering up the Call Control Frame 410 Powering up a USP 412 Powering up a Media Gateway/Multiservice Switch 15000 414 Powering up a Media Gateway/Multiservice Switch 7400 415 Powering up the MDM workstation 417 Powering up an SPM device 418 Powering up the MCS servers 420 Powering up a Border Control Point 424 Powering up the IPmedia 2000 shelves 425 Powering up the SAM16 shelves 426 Powering up an MG 9000 device 427 Restoring certificates and Pre-shared Keys for IPSec 432 Restoring the MG 9000 manager system 436 Restoring files from a DVD-RW 437 Reloading or restarting a SAM21 Shelf Controller 440 Restarting or rebooting a GWC card 441 Performing a partial power up of the Netra 240 server 443
375
445
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
455
Reference
461
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Features
Other changes
There are no other changes in this document.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
10
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
11
Introduction
This document provides an overview of the Disaster Recovery process for Carrier VoIP solutions. A disaster is defined as one of the following events:
fire flood lightning storm explosion tornado earthquake hurricane terrorism any other incident causing damage beyond normal repair to telecommunications facilities
Use this document to restore communication capability within the affected central office or network.
Navigation
Disaster recovery fundamentals (page 13) Disaster recovery configuration (page 55) Preparation for disaster recovery (page 57) Network power down (page 331) Network power up (page 375) Disaster recovery maintenance (page 445) Disaster recovery power up (page 455) Reference (page 461)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
12 Introduction
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
13
Navigation
Power conservation (page 14) Backup and restore (page 15) Synchronized Backup Manager fundamentals (page 15) Sequence of events to run the Synchronized Backup Manager (page 17) Hardware and software requirements (page 18) Requirements and restrictions (page 19) CNM network element backup (page 21) Administration: OAMP workstation backup (page 22) Restoring Multiservice Switch or Media Gateway service data (page 23) Operational Measurement Delivery (page 24) Geographical survivability (page 30) Site failure in a geographic survivable configuration (page 32) Fresh pre-install of CBM 850 cold u0 (page 32) Maintaining sites in a geographic survivable configuration (page 33) Geographical survivability impacts to CS 2000-Compact (page 33) Protocols BGP and OSPF activation (page 41) USP mated pairs (page 42) SSL geographic survivability (page 47) Geographic survivability limitations and restrictions (page 49) Real Time Billing fundamentals (page 50)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Power conservation
You can partially power down several components to conserve power but keep the network in an operational state. Power conservation procedures typically involve powering down inactive or spare units, or keeping only a certain set of hardware or cards active. The following table lists the components and types of power procedures.
Table 1 Power procedures Component or network element XA-Core Ethernet Routing Switch 8600 SDM (CS 2000 Core Manager) Netra 240 Border Control Point Services Application Module Frame Call Control Frame USP Media Gateway/Multiservice Switch 15000 Media Gateway/Multiservice Switch 7480 MDM SPM MCS Border Control Point Manager Media Server 2000 Series UAS MG 9000 Full X X X X X X X X X X X X X X X X X X X X X X Partial X X
When you perform a partial power down for a component, you must follow the corresponding partial power down recovery procedure. Do not use a partial power down recovery procedure to recover from a full power down situation. Conversely, do not use a full power down recovery procedure to recover from a partial power down situation.
ATTENTION
Perform a recovery from a partial power down of the MG 9000 by using the same procedure as recovery from a full power down.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
15
Synchronous Backup Restore Manager (SBRM) is a network backup utility that is installed in the /opt/bkresmgr/nbm directory. The SBRM controls and synchronizes the activities related to backup across other devices that contain related backup software. The SBRM is invoked from the IEMS Backup Manager. Device Backup Restore Manger (DBRM) refers specifically to the device-level backup software that is installed in the /opt/bkresmgr/cbm directory. The DBRM is controlled by the SBRM and it is not recommended running the DBRM directly from a CLUI interface. There are few exceptions and they are specified explicitly in this document if it requires to be run at DBRM level. WARNING:Status commands executed at the DBRM cli may be idle even if a NBM (network backup) is in progress. To ensure that no backup is in progress when required to backup at DBRM level, please check the status of both NBM and DBRM.
Backup Restore Manager is a generic term encompassing both SBRM and DBRM functionality that refers to the software that is common to both.
The SBRM provides a centralized mechanism that you use to initiate a synchronous backup for the core components of the CVoIP Call Server. This capability reduces the possibility of error caused by manual backup procedures on the various component platforms spread across the Call Server solution.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
From Release (I)SN08, the Backup Manager provides synchronous centralized control of the backup functionality resident within the following Call Server components:
CS 2000 Management Tools (CSMT) server MG 9000 Manager (MG9K) server Supernode Data Manager (SDM)/Core Billing Manager (CBM) server. The software on the SDM/CBM only allows control of the XACore/3PC backup. Backup Restore Manager does not back up SDM/CBM data. Integrated Element Management System (IEMS) server
ATTENTION
The IEMS can reside with the CSMT (and other applications) on the CSMT server.
From the IEMS GUI, the user launches the network backup utility (SBRM) command line interface and initiates a backup for CVoIP Call Server components. The network backup utility (SBRM) receives the user request, and initiates a backup command sequence (using CLI commands over SSH) to execute a synchronous device-level backup of the CVoIP Call Server components. Once the backup is complete, the backup files are stored locally on each component. The following figure shows a high-level functional overview of the SBRM.
Figure 1 High-level functional overview of the SBRM
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
17
"Installing and configuring the Backup Restore Manager application on the CS 2000 Core Manager" in Nortel Carrier Voice over IP Upgrade and Patches ( (NN10440-450)) . SBRM core access (page 81) Configuring SSH between the backup restore manager servers (page 96) Configuring core access for SBRM through the Nortel Communication Server 2000 Core Manager (page 84) Creating the backup user ID on the core for SBRM (page 86) Configuring the automated synchronous backup restore manager service (page 138) Common procedures for all components (page 17)
"Installing and configuring the Backup Restore Manager application on the CS 2000 Core Manager" in Nortel Carrier Voice over IP Upgrade and Patches ( (NN10440-450)) . Creating the backup user ID on the core for SBRM (page 86) Configuring SSH between the backup restore manager servers (page 96) Configuring core access for SBRM through the Nortel Communication Server 2000 Core Manager (page 84) Common procedures for all components (page 17)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
To configure SSH between the backup manager restore servers (if not already completed), see the procedure, Configuring SSH between the backup restore manager servers (page 96). The following procedures provide additional information about the SBRM:
To configure automated execution of backups, see the procedure, Configuring the automated synchronous backup restore manager service (page 138). To control the automated execution of backups, see the procedure, Starting or stopping the automated synchronous backup restore manager service (page 107).
To configure the IEMS to support the SBRM, see the following documents:
To add the SBRM to the IEMS using the Java Web Start Client or the Web Client, see Nortel IEMS Configuration ( (NN10330-511)) . To launch the CLI for the SBRM using the IEMS Java Web Start Client, see Nortel IEMS Fundamentals ( (NN10329-111)) . You cannot launch the SBRM using the IEMS Web Client.
Invoke a network backup (where several components are backed up at once) from the IEMS Backup Manager. In rare instances, it can be necessary to run various portions of the backup process on individual components or servers. This capability is implemented as a CLUI interface.
Invoke the CLUI interface to perform a backup at the component-level by logging on to the appropriate component using a Telnet session. See the procedure, Invoking the network-level backup (SBRM-bkmgr) through telnet (page 110).
19
All platforms requiring Backup Manager software functionality must be Java compatible and must provide a resident Java runtime system. The SBRM software does not support restoration of generated synchronized backup images. You must follow existing manual restore procedures to utilize the backup images produced by the Backup Restore Manager. The Backup Restore Manager system does not perform automatic spooling of backup data or image files to secure or redundant servers. You must manage your secure server archiving capabilities. The Backup Restore Manager system publishes data or image files in predetermined and configurable directories on various servers:
SPSF Platform data/image file location: /data/bkresmgr/backup XACore/3PC data/image file location: Configurable
Backup of program store and associated patches is not part of the functionality associated with Synchronous Backup Restore Manager. If you need to completely restore a server or component, you must reinstall your applications and reapply the associated patches. The backup abort command can require manual cleanup or removal of some backup files. You receive notification at abort command completion. You cannot run the backup process while certain constructive or destructive system audits are in progress, for example, SESM CS 2000 Data Integrity Audit. In general, if you attempt a backup while these audits are in progress, the backup attempt is rejected. However, some constructive or destructive audits are postponed when a backup request is received, for example, MG9KEM Network Element Audit. The architecture of the system in question determines the appropriate behavior. In general, any system rejections of the backup request result in an appropriate message displayed to the Synchronous Backup Restore Manager client user. No system-wide coordination between Synchronous Backup Restore Manager and other applications, for example, Upgrade Manager, CS 2000 Audits, and CM dump scheduling, is delivered. You must manually manage all application activity interdependencies. For example, you must stop scheduled activities such as audits during system backup. Also, you must cancel automated CM image dumps because the Synchronous Backup Restore Manager drives the dump within a synchronized system-wide backup window.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
APS and NPM queries are not allowed during backup. This limitation is due to the application shut down during backup. Active clients are informed of the application shutdown. Nortel does not recommend that you backup the MG 9000 EM in isolation due to the CSMT platform data dependency. The MG 9000 EM database is hosted on the CSMT server. Synchronous Backup Restore Manager user authorization only controls the launch of the CLI. CLI command-specific authorization is not provided, therefore only users authorized to launch the Synchronous Backup Restore Manager CLI can perform query commands at the Synchronous Backup Restore Manager CLI. The software on the SDM/CBM only allows control of the XACore/3PC backup. Backup Restore Manager does not back up SDM/CBM data.
During the backup, use of the various GUI and command line provisioning interfaces is restricted from attempting the following actions which result in configuration and provisioning data changes:
Add, delete, or update GWCs at the Call Server Management Tools (CSMT) GUI or OSSGate interface. Add, delete, or update line service at the CSMT OSSGate interface. Manually invoke (or run by using the scheduler) the various audits at the CSMT GUI. Add, delete, or update objects in the IEMS topology by using the IEMS client. Add or modify collection jobs by using the IEMS client. Use the Runtime Administration interface at the IEMS client to perform OSS configuration. Use the Security Administration interface at the Integrated client to add or delete users, groups, or operations. Use the Security Administration interface at the IEMS client to change a user password. Add, delete, or update trunks or carriers by using the CSMT. Node provisioning or deprovisioning by using the SAM21Manager. ATM connection set provisioning or deprovisioning by using the SAM21 Manager.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
21
A single operational user interface for on-demand or scheduled backups. Automatic discovery of network elements. Consolidated view of backup configurations and the status of supported network elements. Policy-based backup for each type of network element. Multiple simultaneous backups. A framework to support additional network elements. The status of backup jobs can be monitored during any phase of the backup.
Some network element backup images are stored at the network element (native storage). Some network element backups are stored on the local network management server (central storage). Some network element backup images can be stored on either native storage or central storage. The Network Element Backup application launches and controls the backup process but some pre-backup and post-backup tasks must be performed manually (for example, inserting a tape in a tape drive before the backup and removing the tape from the tape drive after the backup). If the backup process requires tasks to be performed manually, an overview of these tasks is provided in the backup description for the network element and task details are provided in this document. Network Element Backup software is delivered with default backup parameters for each supported network element. You must update these backup parameters for any customizations applied to your network elements. Some backup parameters apply to all instances of that network element type, while other backup parameters apply only to selected instances of a network element. For more information, see CDMA Network Manager Administration -Network Element Backup (NN20000-344) .
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
ATTENTION
Nortel recommends that you use the automatic backups to ensure that the data that is stored on the tapes is up-to-date.
The OAMP workstation is equipped with a tape drive, software to support tape backup, and five blank data tapes. These tapes provide approximately one month of backups to be kept on-site.
ATTENTION
Nortel recommends that you label the tapes to indicate the OAMP workstation associated with the backups to ensure that any recovery operations are performed from the correct tape.
Backup schedule
Create a schedule for your automatic backups to ensure up-to-date storage of the system configuration of your OAMP workstation.
ATTENTION
Nortel recommends that you perform a full system backup once a week and modified (differential) backups once a day.
Manual backup
Nortel recommends that you perform a manual backup of your OAMP workstation after initial installation. To perform a manual backup of the system configuration for your OAMP workstation, see the documentation provided with your Colorado or Veritas software.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
23
ATTENTION
When you use automatic backup, you must leave your OAMP workstation turned on with Windows running and ensure that there is a tape in the tape drive.
The initial installation of your system configures the tape drive application to perform an automatic full-system backup once a week, every Saturday at 1:00 a.m. However, you can modify the settings in the tape drive application.
ATTENTION
If you do not change the default setting for automatic full system backup, you will need to change the tape every Friday.
ATTENTION
For automatic backups to work, the tape drive scheduler icon must be active in the notification box in the taskbar with the automated daily backups option enabled. Your OAMP workstation must remain on with Windows active and a tape must be in the tape drive.
A full restore restores all backed up service data to the selected devices. An incremental restore restores service data based on a specified date. Like the full restore, you can perform an incremental restore on either one or multiple devices. A selective restore restores specific service data that you select. Like the full restore, you can perform a selective restore on either one or multiple devices.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
ATTENTION
Nortel does not recommend that an active file (current view) be restored on a Multiservice Switch 7400/15000 or Media Gateway 7400/15000. When you restore the current view, you may overwrite the existing current view with different content. This action will cause an outage of the Backup and Restore tool.
Report elements
Report elements define the content of OM report files, and combine content of related OM groups for monitoring and analysis. A report element contains a user-defined report element name, a reporting interval for a report element (five minutes, or the office transfer period of 15 or 30 minutes), and names of the OM groups and registers.
Subtraction proles
The subtraction profile determines the change in the value of an OM group register between five-minute OM reports, as defined in a report element. The subtraction profile applies only after the reporting interval is set to five minutes. The following table lists the types of subtraction profiles.
Table 2 Subtraction profiles Type Single Double Non-subtracti on Description A single register represents a running total Two registers (base and extension) represent a running total Subtraction is not performed on selected registers
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
25
Weekly
Monthly
a user-defined file rotation schedule name a repetition rate for the rotation schedule based on either the number of report records collected or the number of hours to collect records a schedule that defines the time to rotate the report file
The data collection and file rotation schedules operate independently of each other. If a file rotation schedule event occurs during a scheduled data collection period, the file rotation schedule closes and rotates the OM report file, and a new OM report file with the same name is opened. The new file starts collecting immediately and continues until the end of the collection period. The open OM report file remains in the /omdata/open directory until the file rotation schedule closes it and rotates it to the /omdata/closedNotSent directory.
a user-defined file transfer destination name the valid IP address of a remote destination host (xxx.xxx.xxx.xxx)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
the FTP port address of the remote host (default: 21) the remote host logon ID and password
The core manager does not authenticate the IP and port addresses or the logon ID and password. An invalid destination causes the file transfer to fail. After a file fails to transfer, log entries are written to the customer log file at /var/adm/custlog. The file is not re-sent, and the report file must be transferred manually using either the OMFTP command, SFT or standard FTP.
user-defined file transfer schedule name repetition rate for the transfer schedule schedule defining when to transfer the report file (if using a repetition rate) remote file transfer destination host system (<16 destinations/schedule) destination storage directory for each defined transfer destination
The files are transferred downstream using FTP, and move from the /omdata/closedNotSent directory to the /omdata/closedSent directory. If a scheduled file transfer fails, a log is raised and the report file that could not be transferred moves to the /omdata/closedSent directory. The OMDD keeps track of the destination to which the report file could not be transferred. Then, at the next scheduled file transfer, the OMDD attempts to send the report file to the destination again. The OMDD will repeat this activity until one of the following situations occurs:
the file is transferred successfully the file exceeds the retention period for the closedNotSent directory the file gets deleted during an audit because the omdata filesystem usage has exceeded the allowable limit the file is deleted by the omdelete utility
Report registrations
A report registration links information from the report element and schedules for data collection, file rotation and file transfer to collect OM data. The user can create up to 32 report registrations. After a report registration has been created, it can be deleted but not modified. Each
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
27
report registration contains user-defined names for the report registration, report elements and each schedule type. The schedules become active immediately after the creation of the report registration. An OM report file opened by the data collection schedule in the /omdata/open directory uses the name of the report registration as part of the OM report file name. Linking a file transfer schedule into a report registration provides regular and automatic transfers of OM report files to remote downstream destinations. Unless you link a file transfer schedule to a report registration, you must manually transfer your OM report files downstream.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
28 Disaster recovery fundamentals Table 4 Formulas for calculating the limit for OMD data collection OMD data capacity transfer type 5- and 15-minute 5- and 30-minute Formula x+ y/3 = n <= 10,000 tuples (without loss of data) x+ z/6 = n <= 10,000 tuples (without loss of data)
where x y z n = = = < the number of OM tuples collected every 5 minutes the number of OM tuples collected every 15 minutes the number of OM tuples collected every 30 minutes 10,000 tuples
The following table lists the current OMD data collection capacity.
Table 5 OMD maximum data collection capacity Transfer type 5-minute 15-minute 30-minute Capacity (number of tuples) 6000 12,000 24,000
OM report le naming
Report files are named according to the report registration name, file creation date and time, name of the switch generating the OMs, and reporting interval. See the following example file name and explanation.
OM report le contents
Tuple information for an OM group can be viewed in CSV format from the OM report file on the core manager, and by entering the OMSHOW command on the MAP. The following table shows an OM report file.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Operational Measurement Delivery Table 6 Contents of an OM report file Re g1 Va Reg lu 2 Na e me 0 0 0 0 0 0 0 ANF ANF ANF ANF ANF ANF ANF Re Re g g2 31 Va Na lu m e e 0 0 0 0 0 0 0
29
Tim e 3:35 :00 3:35 :00 3:35 :00 3:35 :00 3:35 :00 3:35 :00 3:35 :00
Re g3 1 Va lu e
Audits
The OM report files are stored in the omdata filesystem. This filesystem is audited every 30 minutes for the amount of usage. To ensure that the omdata filesystem usage does not reach 100% at any time, the system performs the following actions:
After the filesystem usage reaches 60%, Major trouble log SDM338 is raised indicating that OM report files will be deleted at the time of the next audit, if usage exceeds 90%. Then, if usage exceeds 90% at the time of the next audit log SDM639 is raised indicating all report files in the closedSent directory will be deleted, and the report files are deleted. If omdata filesystem usage does not fall below 80% after deletion of all report files in the closedSent directory, report files from the closedNotSent directory will be deleted, starting from the oldest file, until the usage is at 80% or less. As each report file is deleted from the closedNotSent directory, log SDM631, which describes the action, is raised.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Geographical survivability
Geographic survivability is the distribution of components across a geographic area to ensure services continue in the event of a disaster. A disaster can include fire, flood, tropical storm, or act of terrorism. The following figure shows the topology for CS 2000 - Compact support for geographic survivability.
Figure 2 Geographic survivability topology
The CS 2000 - Compact supports geographic survivability through the failover and sparing functionality of individual nodes. The configuration for the CS 2000 Management Tools (CMT) and Integrated Element Management System (IEMS) servers and the Core and Billing Manager (CBM) 850 is as follows:
Site A is configured with the CMT and IEMS high-availability (HA) server pairs Site B is configured with the CBM 850 HA server pairs (referred to as CBM 850 HA u0 and u1) Site A is configured with an additional CBM in to be used in the event of disaster at Site B (referred to as CBM 850 standby u0) Site B is configured with an additional server for the CMT and IEMS in the event of disaster at Site A
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Geographical survivability
31
For a communication server, geographic survivability requires that redundant components (other than OAM components) reside at different sites (referred to as Site A and Site B). For OAM components, the Automatic Backup and Accelerated restore feature (known as remote backup) remotely backs up all data on the target unit. This provides a standby backup system ready to provide service should the primary system or cluster be unavailable for an extended period of time (for example, catastrophic site loss). The remote backup can assume the identity of the target system with data and files accurate to the last sync and will be located at a different site from the target system. Remote backup performs the backup with a TCP/IP connection, and stores an exact copy on the standby server which can be quickly and remotely activated. This remote backup copies all files in each file system marked for backup using the same behavior as a full system backup. A remote backup configuration tool is provided to set the necessary parameters for backup to automatically occur from one to four times a day. This tool also provides a facility for manually initiating a backup and monitoring its progress. The standby server has an identical copy of files from the last backup, so it can become the primary system by changing the boot pointer and rebooting. When the primary site becomes available, the remote backup feature can be reused to transfer current system configuration back to the primary site and system.
CAUTION
If configuration, provisioning, patching or other write-type operations occurred since the last remote backup, the remote backup system can be out of sync compared to the data in network elements and/or the primary OAM system. When initiating a switchover to a remote backup OAM server, do not execute configuration, provisioning, patching or other write-type operations through the remote backup OAM system until out-of-sync conditions are cleared. Take actions before initiating the switchover to a remote backup OAM server (that is, response to a geographic or other prolonged outage of the primary OAM system) to halt or prevent write-type operations by OSSs and operations personnel until an in-sync status is achieved. If a site outage is imminent (for example, threatening severe weather conditions), consider precautionary preparations to discontinue write-type operations. Make manual backups to the remote backup OAM servers to ensure data synchronization of the remote OAM server before performing a switchover.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Apply similar precautions when initiating a recovery back to the primary cluster. In this more controlled scenario, discontinue write-type operations before initiating this procedure
Site A failure
A loss of Site A would include a loss of the CMT and IEMS servers. This loss would result in the following losses of functionality:
all OAM&P functions of the CMT and IEMS, including GUI access, element management, alarms, and non-Core logs access to the CS 2000-Compact Core Manager on the CBM 850
The responsive action in this scenario is to initiate a switchover to the standby CMT and IEMS servers at Site B.
Site B failure
A loss of Site B would include a loss of the CBM 850 servers. This loss would result in the following losses of functionality:
transfer of billing records from the Core to the CBM 850 billing records resident in the CBM 850 that had not been offloaded to an OSS ability to receive scheduled Core OMs and logs (resulting in them being discarded) access to the CS 2000-Compact Core Manager backup bootp load repository
The responsive action in this scenario is to initiate a switchover to the standby CMT and IEMS servers at Site A.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
33
Installing the remote backup server (page 310) Scheduling automatic backups on the remote server (page 315) Viewing configuration information for remote server backups (page 318) Performing a manual backup of the target server (page 320) Initiating a recovery back to the cluster (page 323) Initiating a switch over to the remote backup server (page 325) Disabling Ethernet Routing Switch 8600 WAN Edge routing (page 327) Disabling and re-enabling the router ports on the Ethernet Routing Switch 8600 (page 329)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
34 Disaster recovery fundamentals Figure 3 Call Agent Card View Provisioning panel: Geographic Survivability enabled
Failure scenarios
The following table provides summaries of system responses during various failure scenarios when Geographical Survivability is enabled. The scenarios assume redundant configurations are located in two separate sites.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Geographical survivability impacts to CS 2000-Compact Table 7 Failure system response summaries Scenario Ethernet Routing Switch 8600 failure at one site Response At the site with the failed routing switch: All nodes lose mate connectivity by using Ethernet.
35
The Call Agent loses WAN backup connectivity. FC or GigE call data sync link remains up until the Call Agent resets. The Call Agent resets itself approximately 1 minute after losing network connectivity and tries to boot from the network. The Call Agent keeps trying to boot until the network recovers and the boot is successful. The Call Agent detects IST loss, but cannot disable Ethernet Routing Switch 8600 routing. USPc detects isolation and takes down SS7 links at the site with the Ethernet Routing Switch 8600 failure. USP mated pair configurations: Mated pairs connected to pair of gateway POP Ethernet Routing Switch 8600s: No impact to both USPs from a failure. Mated pairs split across a CS-LAN site (each connected to the Ethernet Routing Switch 8600 of local CS 2000 CS-LAN): For CS 2000 - Compacts that share the same CS-LAN, the USP detects isolation and takes down its SS7 links. An Ethernet Routing Switch 8600 failure does not impact USP mated pair behavior in other CS 2000 - Compact configurations. Mated pairs split across a CS-LAN site (each connected to one Ethernet Routing Switch 8600 connected to an edge site): No impact from a failure.
At the site with the in-service routing switch: The Call Agent remains active if it is already active. No outage occurs. If the Call Agent was not active, it takes activity within 2 seconds.
SOS goes through a warm or restart (approximately 20 seconds for the NTRX51HZ card with the MCPN905 processor and 30 seconds for the NTRX51GZ card with the MPCN765 processor). Other nodes go active and follow the Call Agent example after losing mate connectivity.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Scenario
Response
If Site B has the in-service router, the standby CMT server must be brought into service. (See the following ATTENTION statement.) If Site A has the in-service router, the standby CBM must be brought into service. (See the following ATTENTION statement.)
ATTENTION
Consider using this option based on the estimated time to recover the fault or failure of the CS LAN versus time and effort to bring up the cold standby system and recover the HA pair afterward. For example, if the time to recover the CS LAN is estimated to be 10 hours, consider if it is worth activating the cold standby CMT server or CBM. See procedure Initiating a switchover to the remote backup server. Optical frame failure at one site At the site with the failed optical frame: All nodes lose mate connectivity by using Ethernet.
Call Agent FC or GigE call data link connectivity is lost. WAN backup remains up. The active Call Agent remains active, but without sync. No outage occurs. The remainder of this scenario assumes the active Call Agent is at this site. If not already active, expect other nodes co-located with the active Call Agent to become active after losing mate connectivity. If not already inactive, expect other nodes co-located with the inactive Call Agent to become inactive after losing mate connectivity. (Done without mate connectivity.) USP mated pair configurations: Mated pairs connected to pair of gateway POP Ethernet Routing Switch 8600s: No impact to both USPs from a failure. Mated pairs split across a CS-LAN site (each connected to the Ethernet Routing Switch 8600 of local CS 2000 CS-LAN): For CS2K-1, only the USP at the active side provides service. For CS2K-2, both USPs provide service, but over a packet network. There are no impacts to other CS 2000s. Mated pairs split across a CS-LAN site (each connected to one Ethernet Routing Switch 8600 connected to an edge site): No impact from a failure.
At the site with the in-service optical frame: The inactive Call Agent detects IST loss. For an Ethernet Routing Switch 8600 CS LAN using OSPF, the Call Agent
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
37
Scenario
Response disables OSPF. For an Ethernet Routing Switch 8600 CS LAN using Border Gateway Protocol (BGP) or for a third-party CS LAN using either OSPF or BGP, execute manual action to disable routing at the site with the inactive Call Agent. See section Disabling Border Gateway Protocol (BGP) for instructions of disabling and enabling BGP. See section Enabling protocols BGP and OSPF.
Other nodes become inactive and follow the Call Agent example after losing mate connectivity. (Done without mate connectivity.) USPc detects isolation from the active Call Agent, and takes down SS7 links at the site with the inactive USP. If Site B has the failed optical frame, the standby CBM must be brought into service at Site A. (See the following ATTENTION statement.) If Site A has the failed optical frame, the standby CMT must be brought into service at Site B. (See the following ATTENTION statement.)
ATTENTION
Consider using this option based on the estimated time to recover the fault or failure of the CS LAN versus time and effort to bring up the cold standby system and recover the HA pair afterward. For example, if the time to recover the CS LAN is estimated to be 10 hours, consider if it is worth activating the cold standby CMT server or CBM. See procedure Initiating a switchover to the remote backup server. One site is destroyed in a catastrophic event At the site that is destroyed: There is no activity.
The SS7 network takes down the links to the destroyed building. USP mated pair configurations: Mated pair connected to pair of gateway POP Ethernet Routing Switch 8600s: No impact occurs to either USP as they are located in different sites from a failure. Mated pair split across a CS-LAN site (each connected to the Ethernet Routing Switch 8600 of local CS 2000 CS-LAN): The SS7 network takes down all links to the destroyed site. Mated pair split across a CS-LAN site (each connected to one Ethernet Routing Switch 8600 connected to an edge site): If the USPs are located at the same site where the failure occurred, the SS7 network takes down all links at the
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Scenario
Response destroyed site. Otherwise, no impact occurs to either USP from the failure.
At the site that is not destroyed: All nodes lose mate connectivity. The Call Agent loses all mate connectivity, including the backup WAN link, and drops sync. If the Call Agent is active, it remains active. No outage occurs. If the Call Agent is not active, it takes activity within 2 seconds. If the Call Agent is inactive, it takes activity within 2 seconds followed by a warm or SOS restart (approximately 20 seconds for the NTRX51HZ card with the MCPN905 processor, and 30 seconds for the NTRX51GZ card with the MCPN765 processor). Other nodes go active, and follow the Call Agent example after losing mate connectivity. If Site B was destroyed, initiate a switchover to the standby CBM at Site A. (See procedure Initiating a switch over to the remote backup server. If Site A was destroyed, initiate a switchover to the standby CMT at Site B. (See procedure Initiating a switch over to the remote backup server.
Active Call Agent card fails at one site
At the site with the failed Call Agent: All nodes at the site can communicate with their mates. At the site with the mate Call Agent: All nodes in the site can communicate with their mates.
The (inactive) Call Agent detects loss of connectivity with the mate, detects local and WAN connectivity, takes activity, and restarts the SOS. Other nodes experience disconnection from the SOS for approximately 20 seconds for the NTRX51HZ card with the MCPN905 processor, and 30 seconds for the NTRX51GZ card with the MCPN765 processor (normal restart behavior)
At the site with the failed inactive Call Agent: All nodes in the site can communicate with their mates. At the site with the mate Call Agent:
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
39
Scenario
Response
Recovery from isolation split brain (Active/Inactive)
All nodes in the site can communicate with their mates. The (active) Call Agent detects loss of connectivity with the mate, detects local and WAN connectivity, and stays active. No SWACT or restart is required.
At both sites: After the Call Agents can communicate with their mates, they recognize that both are active. The Call Agent that was inactive backs down, leaving the other Call Agent active. The fallout is to force Unit 0 active and Unit 1 inactive.
Other nodes can communicate with their mates, negotiate activity, and resume normal operations.
When negotiating activity, the Call Agent preference is always to remain on the same side, if that side supports activity. If failover of the Call Agent is necessary, the Call Agent switches activity in less than 2 seconds. When the Callp Application performs a restart, call processing is interrupted for approximately 20 seconds for the NTRX51HZ card with the MCPN905 processor, and 30 seconds for the NTRX51GZ card with the MCPN765 processor (normal restart behavior). Failovers of other Callp nodes follow the Call Agent failover within 2 seconds. In Enterprise-only configurations where the solution contains Message Controller (MC) cards connected to Message Switches (MS), ENET and TDM peripherals, the MC and MS cards are co-located in one of the main geographically redundant sites. During site isolation, when determining the appropriate master site, preference is given to the site that contains the MC cards. This assumes that the site is able to take activity. If necessary, activity is switched to this side during the activity negotiation.
Recovery scenarios
The following table provides a summary of system responses during recovery.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
40 Disaster recovery fundamentals Table 8 Recovery system response summary Scenario Recovery from isolation split brain (Active/Inactive) Response
When Call Agents can communicate with their mates, they recognize that both are active. The Call Agent that was inactive before the failure backs down, leaving the other Call Agent active. The fallout is to force Unit 0 active, and Unit 1 inactive. Other nodes can communicate with their mates, negotiate activity and resume normal operations.
All elements: continually monitor connections with their mates, and with other network elements with which they normally communicate. When connectivity is not present, they continue to monitor the connections for restored connectivity. (The elements continue monitoring regardless of their activity state.)
negotiate activity and services when connectivity recovers, and resume normal operations. USP mated pair configurations: Mated pair connected to pair of gateway POP Ethernet Routing Switch 8600s: No impact to USP mated pair. Mated pair split across a CS-LAN site (each connected to the Ethernet Routing Switch 8600 of local CS 2000 CS-LAN): USP mated pair does not negotiate activity during recovery. This is a result of USP mated pair operating in a load sharing mode; both are active during normal conditions. Mated pair split across a CS-LAN site (each connected to one Ethernet Routing Switch 8600 connected to an edge site): No impact from a failure.
When negotiating activity, the Call Agent preference is always to leave activity on the same side. When recovering to a full system configuration, activity remains on the same unit, without impact. During recovery from a split system (caused by incorrect message routing), node activity resolves in a few seconds. Call processing could require up to 15 minutes to recover completely. If an inactive call agent is not in service when the IST goes down, OSPF on the Ethernet Routing Switch 8600 is not disabled on the site of the inactive call agent, potentially resulting in failed calls due to misdirected messages. This condition can occur on site recovery after
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
41
the Ethernet Routing Switch 8600 returns to service and the optical ring is still recovering. In the event that this outage occurs, manually disable OSPF on the Ethernet Routing Switch 8600 or disconnect the links to the WAN on the site of the inactive call agent until it returns to its operational state.
In this example, each Ethernet Routing Switch 8600 includes two BGP neighbors. One BGP neighbor is the eBGP peer to the wide area network (WAN) Core router. The other BGP neighbor is the iBGP peer to its Ethernet Routing Switch 8600 mate.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
can be used with both MCPN 905 and MCPN 765 type Call Agent cards selection of either BGP or OSFP as the protocol during Geographic Survivability installation script setgeoconfig supports either BGP or OSPF input Call Control Agent (CCA) sends control commands (Disable/Enable BGP or OSPF) to the Ethernet Routing Switch 8600 based on protocol selection during installation Inactive CCA triggers command BGP disable command to the Ethernet Routing Switch 8600 when optical connectivity is lost between two sites. When recovering optical connectivity, the inactive CCA triggers command BGP enable to the Ethernet Routing Switch 8600. Existing routing protocol disable or enable event log is used with the corresponding OSPF or BGP protocol.
mated pair of USPs connect to a pair of gateway POP Ethernet Routing Switch 8600s, which is co-located with the CS-LAN or in a remote site from the CS 2000 or CS 2000 - Compact mated pair of USPs are split across a CS-LAN site, whereby each USP connects to an Ethernet Routing Switch 8600 in a local CS-LAN configuration mated pair of USPs are split across a CS-LAN site and each connects to a single Ethernet Routing Switch 8600, which also is connected to an edge site
For details on mated USP pairs, see NTP Nortel Universal Signaling Point Configuration ( (NN10093-511)) .
Mated pair connected to a pair of gateway POP Ethernet Routing Switch 8600s
While configuration also applies to a non-Geographic Survivability environment, in a Geographic Survivability environment, the Cores can share the same optical ring providing the distance is close. The mated USP pair is co-located and connect to two Ethernet Routing Switch 8600s, which also connect to a POP gateway.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
43
Two RTC cards cross-connect to two Ethernet Routing Switch 8600s on each USP. Dedicated pairs of IPS7 cards serving as inter-USP mated links cross-connect to two Ethernet Routing Switch 8600s on each USP. The mated link transfers SSNM messages. (The mated link transfers traffic during some failure conditions.) Other IPS7 card pairs dedicated to CS 2000 Cores connect to local Ethernet Routing Switch 8600 on each USP. All the CS 2000s share SS7 cards or have dedicated SS7 cards. Only ABS stand-alone servers can be used. Two ABS servers are co-located with two USP pairs and connected to two Ethernet Routing Switch 8600s. Both USP pairs are connected to only one IEMS server.
Mated pair of USPs split across a CS-LAN site (each connected to the Ethernet Routing Switch 8600 in a local CS 2000 CS-LAN)
In this configuration, a mated pair of USPs are split across a CS-LAN site, whereby each USP connects to an Ethernet Routing Switch 8600 in a local CS-LAN configuration.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Two RTC cards connect to different Ethernet switch boards on each mated USP pair. Dedicated pairs of IPS7 cards serving as inter-USP mated links cross-connect to two Ethernet Routing Switch 8600s on each USP. The mated link transfers SSNM messages. (The mated link transfers traffic during some failure conditions.) Other IPS7 cards dedicated to CS 2000 Cores connect to local Ethernet Routing Switch 8600 on each USP. All the CS 2000s share SS7 cards or have dedicated SS7 cards. The CBM and CMT can be used as an ABS server for mated USP pairs. An ABS server is co-located with one half of CS2Kc-1. Two UPS pairs connect to one IEMS server.
Mated pair of USPs split across a CS-LAN site (each connected to a single Ethernet Routing Switch 8600)
In this configuration, a mated pair of USPs are split across a CS-LAN site. Each pair connects to a single Ethernet Routing Switch 8600, which also could be connected to an edge site.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
45
The following figures show examples of this configuration. Both figures reflect connections for RTC and IPS7 cards.
Figure 7 Mated USP pair split across a CS-LAN site (each connected to a single Ethernet Routing Switch 8600)
The following figure shows the same configuration but with the Ethernet Routing Switch 8600 connected to an edge site.
Figure 8 Mated USP pair split across a CS-LAN site (each connected to a single Ethernet Routing Switch 8600, which connects to an edge site)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
VLAN assignment
All USP mated pair configurations support the following VLANs:
Both RTC cards of USP A are assigned to OAM_USP_A_VLAN. Both RTC cards of USP B are assigned to OAM_USP_B_VLAN. IPS7 cards serving as mated links on USP A are assigned to CallP_USP_A_Mate VLAN. IPS7 cards serving as mated links on USP B are assigned to CallP_USP_B_Mate VLAN. IPS7 cards dedicated to CS2Kc-1 on USP A are assigned to CallP_USP_A_CS2K1. IPS7 cards dedicated to CS2Kc-1 on USP B are assigned to CallP_USP_B_CS2K1. IPS7 cards dedicated to CS2Kc-2 on USP A are assigned to CallP_USP_A_CS2K2. IPS7 cards dedicated to CS2Kc-2 on USP B are assigned to CallP_USP_B_CS2K2.
Customers must guarantee a secure connection between the USP and the remote CS 2000 - Compact. For USPs to communicate with remote CS 2000 - Compacts over the Core network, the CCA, 3PC or XA-Core HIOPs must be advertised in the carriers network, although they have been configured in the call processing private VLAN. To reach the USPs, the VLAN must be advertised outside of the Ethernet Routing Switch 8600 CS-LAN. Other devices that have been provisioned in the same VLAN (for example, STORM-IA) can be exposed. To reduce the risk of DoS attack, provide new filtering rules and new traffic limitation rules on the GW-POP. For example, the new filtering rules on the GW-POP must accept traffic to the USP only if it emanated from the GWC. If IPsec is enabled, apply the following filtering rules to implement the GW-POP firewall.
Discard all traffic to the CCA except traffic from the GWC subnets. Discard all non-ESP or non-IKE traffic (UDP traffic to and from port
500).
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
47
Ensure that USP mated pair-specific VLANs are not disabled during a ring outage because the USPs are not split across the ring as in other geographic-configured VLANs. Nortel recommends that mated USPs installations not exceed 120 kms from the CS-LAN.
the Geo mode is enabled in the CS 2000 Call Agent window no active CS 2000 Compact Call Agent exists when an SSL System Manager or Session Manager starts
The SSL component should detect the isolation before it attempts activation. The component shuts itself down and waits until there is an active CS 2000 Compact Call Agent present before it restarts itself.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
configuration is based on the reachability of the active IP address of the CS 2000 Call Agent. Internet Control Message Protocol (ICMP) echo request (ping) messages are sent to the active IP address of the CS 2000 Compact Call Agent. SSL components that detect this IP address "unreachable" consider themselves to be in the non-surviving site (that is, network isolated). See NTP SSL Configuration (NN10023-511) for details on how to enable or disable pinging on the active IP address of the CS 2000 Call Agent.
the deployment of split session-server pairs is supported the SST maintains 99.999 % availability for Callp
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
49
the time for fail-over is equivalent, on the same order of magnitude, to the fail-over time for a non-geographically survivable configuration stable calls and billing are maintained during a fail-over
The physical distance between active and standby sites is limited to 120 KM/75 miles. Each site must have a single Ethernet Routing Switch 8600 with dual switch fabric and CPU blades or third-party equivalent. Synchronization of data between two CCA blades over GigE links applies to the CS 2100 market with reduced capabilities. This configuration applies just to Enterprise solutions. In Enterprise hybrid configurations (with TDM equipment homed at one site), the TDM equipment is not geographically redundant. In determining the master site, preference is given to the TDM side only when either side can support Callp. If necessary, perform a SWACT to the TDM side to allow Callp on that side. Because the Call Agent interacts with the Ethernet Routing Switch 8600, the feature requires that each site have only one routing switch and IST links configured between sites. Dual Ethernet Routing Switch 8600s at each site and SMLT links between sites are not supported. Interactions between the Call Agent and the CS LAN are supported to prevent split brain scenarios (by disabling OSPF) when the Ethernet Routing Switch 8600s are used for the CS LAN. Upgrades from previous releases in an Enterprise geographic survivable configuration (which have dual Ethernet Routing Switch 8600s at each site) require that the dual Ethernet Routing Switch 8600s be migrated to a single Ethernet Routing Switch 8600 site. When a total loss of communication between sites occurs (that is, all three master links are down), the two Call Agents cannot negotiate activity decision. The decision is based on connectivity check from each site the WAN network.
The WAN backup path mitigates the risk of optical ring failure. The
WAN connection check helps resolve activity when the backup path is down.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
CS 2000 - Compact supports only a single time zone setting. If the two physical sites are in different time zones, Nortel recommends that the time zone be set to either GMT or the time zone of one of the sites. Both Session Server units of a pair are located at the same site. For offices with Message Controllers, Nortel recommends that the Session Servers be located at the same site as the TDM components. For maximum redundancy, the WAN backup path must be configured separately from the optical network, as follows:
vlans route over the WAN network instead of over the optical ring vlans are not disabled with OSPF disable alarm generated for lack of connectivity
Gateways and services node components of CS 2000 - Compact are single units, and are not geographically redundant. Where the nodes are located and how they are connected to the network affects whether they survive a failure. Although the same nodes are supported in configurations with and without Geographic Survivability, there is no change in configuration or connection in the configuration with the Geographic Survivability configuration.
CAUTION
Line data integrity audit impacts the maintenance actions such as GWC status and GWC swact. At times, the bearer network cannot be retrieved from the core due to the bearer network timeout, as it is design intent.
ATTENTION
Outbound file transfer using FTP requires the downstream system to be running an FTP server that complies with RFC 959. In particular, for Real-Time Billing, the FTP server must support the "APPE" (Append) message.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
51
Terminology
To understand how the SBA processes and routes the billing records it receives for RTB, the following terminology must be understood:
Stream - A stream, or billing stream, can be conceptualized as a pipeline through which billing records received from the core pass. For each stream component that exists on the core, a corresponding stream component exists on the core manager. Billing records created by calls pass through the stream from their point of origination on the core to the core manager, where they are stored on disk. Sub-stream - A stream is further divided into Primary and Recovery sub-streams. The Primary sub-stream handles the current records being sent by the core. The Recovery sub-stream is only active after the SBA is unable to transfer records from the core to the core manager and temporarily stores the records on the core. When the core is once again able to re-establish the connection to the core manager, the stored records are sent to the core manager in a Recovery sub-stream while, concurrently, the current records are sent in the Primary sub-stream. Active file state - When records are written to a file that is open on the core manager, the file name on the core manager is prefixed with an "A", which means "active". When a billing files content is being written to a file on a downstream processor, the name of the file on the downstream processor is also prefixed with an "A". Unprocessed file state - After the file on the core manager receives all of its billing records, the file is closed and the name of the file is prefixed with a "U", which means "unprocessed". In the same manner, after the file content has been transferred to a downstream processor, the file receiving this content on the downstream processor is also prefixed with a "U". Processed file state - When a billing file on the core manager is closed and its content has been received by all designated downstream destinations, the file is then eligible for removal in order to free up disk space. The file name prefix then changes from "U" to "P", meaning "processed".
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
The tuple contains a field indicating whether it is active. Scheduled file transfers occur only when the tuple is active. An interval setting in the schedule tuple determines how often SBA checks to see whether there are unprocessed files waiting to be sent downstream. When this interval is exceeded, the files are transferred downstream.
53
Connection management
In normal operation, open files transferred by Real Time Billing are sent only to the Primary IP destination specified in the schedule tuple for each destination. If a problem occurs with that destination and open file transfer fails, the current file is closed. RTB will be tried again on the next open files based on the RTB MIB value RTBMaxConsecutiveFailures. After all file transfers allowed by the RTBMaxConsecutiveFailures RTB MIB value have been attempted, a critical alarm is raised, a log is issued, and RTB is moved to the SYSB state. In this state, open file transfer is not active. The retry behavior of RTB differs from that of a scheduled transfer. In the case of a scheduled transfer the primary address is tried first, and if it fails, attempts to re-transmit the file are repeated until the number of retries is exhausted. The retry attempts alternate between the primary and alternate destinations indicated in the schedule tuple. In the case of an RTB transfer, however, RTB will not attempt to re-transmit the file since that impacts the ability to send current records. Thus, RTB closes the file and retries transfer on the next file opened. In addition, unlike scheduled transfer, RTB only uses the primary destination. When RTB closes billing files, it cannot send the billing files downstream. The billing files are, however, automatically transferred from the core manager during the next scheduled transfer when the schedule tuple is active. The billing files can also be transferred manually. For the procedure used to transfer the billing files manually, see "Sending billing files from disk" in Nortel Core and Billing Manager 850 Accounting ( (NN10363-811)) . Manual intervention is required to restore RTB when it is in the SYSB state. The problem can often be attributed to a network connection that is no longer functioning properly. The RTB IPTest command can be used to "ping" the primary downstream address indicated in the schedule tuple to determine this. A SYSB state may also occur if the protocol has been changed in the schedule tuple to something other than the required protocol for RTB. When the problem that forced RTB into the SYSB state is resolved, the RTB Bsy and Rts commands can be used to bring RTB into service.
Alarms
For information about RTB alarms, refer to Nortel Core and Billing Manager 850 Fault Management ( (NN10351-911)) .
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
55
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Disaster recovery conguration task navigation Disaster recovery configuration (page 55) Preparation for disaster recovery (page 57) Network power down (page 331) Network power up (page 375) Disaster recovery power up (page 455) Device audits (page 473)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
57
Recovering the SPFS system if updisk daemon dies (page 58) Installing optional software on a Core and Billing Manager (page 64) Installing the Passport Log Streamer application (page 71) Installing the DDMS (page 59) Installing the OM Data Delivery application (page 68) Installing the SBA and AFT software packages (page 73) Installing the GR740PT application server (page 74)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
On the node displaying the updisk daemon or dataset failure
Step 1 2
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
59
Prerequisites
For a successful installation of DDMS, verify that the Log Delivery Service application is in service. If you make manual changes to the password lifetime value, or if you turn the Enhanced Password Control off or on, you must synchronize these changes with the DDMS software by performing a bsy or rts of the DDMS application. If you change any of the SDM01-SDM04 passwords manually, you must apply the same password changes in the DDMS configuration file.
Procedure steps
Step 1 Action At your workstation, at the CLI prompt on the core, configure the Enhanced Password Control for the SDM01 user ID:
permit sdm01 <sdm01_pswd> 4 10000 english all
ATTENTION
If Enhanced Password Control is in effect on the CM, the password must be at least six characters in length.
2 3 4
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
5 6
Create the first required logical volume for the DDMS application:
makelv /cbmdata/00/osscomms 16
Set the access privileges of the first logical volume for the DDMS application:
chmod 755 /cbmdata/00/osscomms
Set the ownership privileges of the first logical volume for the DDMS application:
chown maint:maint /cbmdata/00/osscomms
Create the second required logical volume for the DDMS application:
makelv /cbmdata/00/ossaps 112
Set the access privileges of the second logical volume for the DDMS application:
chmod 755 /cbmdata/00/ossaps
10
Set the ownership privileges of the second logical volume for the DDMS application:
chown maint:maint /cbmdata/00/ossaps
11
Create the third required logical volume for the DDMS application:
makelv /cbmdata/00/ossapslog 112
12
Set the access privileges of the third logical volume for the DDMS application:
chmod 755 /cbmdata/00/ossapslog
13
Set the ownership privileges of the third logical volume for the DDMS application:
chown maint:maint /cbmdata/00/ossapslog
14
Apply the two software application packages, OSS and Application Svcs and OSS Comms Svcs . Specify /cdrom/cdrom/applications/cbm/packages as the source directory when you perform that procedure. You are prompted automatically to configure the OSS Comms Svcs package.
15
ATTENTION
The OSS and Application Svcs package does not require configuration.
16
If you are prompted automatically to configure the OSS Comms Svcs package then go to 29 OR
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
61
If you are not prompted automatically to configure the OSS Comms Svcs continue with the next step. 17 18 Access the configuration level of the maintenance interface:
cbmmtc config
In the list of applications, locate the OSS Comms Svcs application and record its application number (located next to the names of the applications). Select the application:
select <application number>
In response to the command, the OSS Comms Svcs application is highlighted on the cbmmtc config screen. 19 20 Invoke the configuration of the OSS Comms Svcs application:
config
When prompted to enter the logroute tool, as shown in the following figure, press Enter:
Figure 10 DDMS logroute tool banner
The Logroute Main Menu appears. 21 Select the Device List menu
1
The Device List Menu screen appears. 22 23 Select 1 to display the Device List screen. If the list includes device /cbmdata/00/logs/ossaps/ossapslog then continue with the next step. OR If the list does not include device /cbmdata/00/logs/ossaps/ossap slog skip the next step. 24 25 26 Press the Enter key. . Begin to add a new device:
2
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Example Response:
Enter file name ==> /data/logs/
27
Routing for the DDMS is now set up. 28 29 30 31 32 33 When prompted, enter STD log format (from the range displayed). When prompted, set the ECORE option to ON. Select addrep:
a
Response:
Save completed -- press return to continue
34 35 36 37
Press the Enter key and return to the Add Device screen. Return to the Device List Menu screen:
5
Exit logroute:
6
The CM User Setup screen appears and the required CM users, SDM01-SDM04, for DDMS are added to the DDMS configuration file. The passwords for these users are the same as those entered in step 1.
ATTENTION
The user IDs and passwords are not case sensitive. You can change them after this installation is complete.
38
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
63
39 40
When prompted, enter the user name (for example, sdm01). When prompted enter the password.
ATTENTION
The first entry of a user name and password generates the following message: Error: file not valid. You can ignore this message.
41 42
If applicable, continue to add other user names and passwords at the prompt, otherwise continue with the next step. Exit the CM User Setup screen:
0
The DDMS Clients Configuration screen appears. 43 Add a new DDMS client.
1
ATTENTION
The DDMS clients are the CS 2000 Management Tools servers with the SESM load.
44
When prompted, enter the IP address for each of the CS 2000 Management Tools servers and press the Enter key, after each entry. If the CS 2000 Management Tools server is a cluster configuration, add the IP address of both the active and inactive units. After you have entered all the IP addresses, type
done
45 46
Variable denitions
Variable <sdm0n_pswd> <application number> Value The CM password for user (SDM01,SDM02,SD M03,SDM04) The number associated with the OSS Comms Svcs application, that you noted
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
ATTENTION
This procedure must be performed on a pre-cloned system. If the procedure is not performed on a pre-cloned system, clone the image of the active node to the inactive node of the cluster after the software package has been installed and configured, and after the active node has been made patch-current.
Procedure steps
At your workstation
Step 1
Action Open a connection to the active node of the CBM using SSH and log on as the root user: ssh -l root <ip_address> Enter the password for the root user. To install the DDMS application, perform steps 1 and 5 only in Installing the DDMS (page 59). Go to step 18. To install the OMDD application then perform step 1 only, in Installing the OM Data Delivery application (page 68). Go to step 18. To install the log delivery service application, perform Installing the Passport Log Streamer application (page 71). Go to step 24. To install the SBA or AFT applications, perform Installing the SBA and AFT software packages (page 73) . Go to step 24. To install the GR740PT application server, perform Installing the GR740PT application server (page 74). Go to step 24. To install the FTP proxy application, refer to Nortel Core and Billing Manager 850 Administration and Security ( (NN10358-611)) .
2 3 4 5 6 7 8 9 10 11 12 13
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
65
14 15
Go to step 17. To install the Backup Restore Manager software, perform Installing the Backup Restore Manager software (page 94), and then go to step 17. Go to step 24. To apply any other optional software application, perform Applying software packages on a Core and Billing Manager using the CBMMTC interface (page 88). If required, return to step 3 to continue applying new software applications. Ensure that you create all required logical volumes for the applications that you are install before you continue. If you created any logical volumes, reboot the CBM:
init 6
16 17
18 19 20 21
If required, complete installation of the DDMS application by performing the remaining steps of Installing the DDMS (page 59), starting with step 14. Go to step 24. If required, complete the installation of the OMDD application by performing the remaining steps of Installing the OM Data Delivery application (page 68), starting with step 2. Ensure that your CBMs are patch-current. For patching procedures, see Nortel ATM/IP Solution-level Administration and Security ( (NN10402-600)) . Clone the active node image to the inactive node by performing the procedure, Cloning the image of the active node to the inactive node (page 91). If applicable, return to the higher level task flow or procedure that directed you to this procedure. This procedure is complete.
--End--
22 23
24
25
26 27
Variable denitions
Variable <ip_address> Value IP address of the active node of the CBM cluster.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
The following table lists filesets (applications) that the CBM0090 load includes. The table shows which filesets are included with the CBM at the time of installation (Base) and which filesets are optional that you can install.
Table 9 Filesets available for the Core and Billing Manager Fileset SDM_BASE.version_20.81.0.0 CBM_SETUP NT_SIM.tools SDM_ACE SDM_AFT.DMS500 SDM_BASE.base SDM_BASE.comm SDM_BASE.gdd SDM_BASE.logs.client SDM_BASE.logs SDM_BASE.mtce SDM_BASE.omsl SDM_BASE.tasl SDM_BMI.bmi SDM_DDMS_ossaps SDM_DDMS_osscomms SDM_BASE.util SDM_DEBUG.tools SDM_DMA.dma SDM_FTP.proxy SDM_GR740PT.gr740pt SDM_LOGS.mdm SDM_OMDD.omdd SDM_REACHTHRU.rttl1 SDM_SBA.DMS500 Description Load lineup information CBM installation and upgrade tool; available only on CD Patching tools SDM ACE distribution SBA automatic file transfer Platform base Platform maintenance common Generic data delivery Log Delivery Service client Log Delivery Service Platform maintenance OM Access Service Table Access Service Base Maintenance Interface OSS and Application Svcs OSS Comms Svcs Platform Utilities SDM/CBM Debug Helper Tools DMS Maintenance Application FTP Proxy GR740 Pass Through Passport Log Streamer OM Delivery Reach Through SPM SDM Billing Application Type Base Base Base Optional Optional Base Base Base Optional Base Base Base Base Optional Optional Optional Base Base Optional Optional Optional Optional Optional Optional Optional
ATTENTION
Base = included with the CBM
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
67
Table 9 Filesets available for the Core and Billing Manager (contd.) Fileset SDM_SCFT.scft SDM_CLP.clp SDM_CLP.clp NTbkupmgr NTdtsv NTprxy NTsaf NTbulk Description SSH Core File Transfer CBM LI Proxy CBM LI Proxy Succession Provisioning Data Synch Manager CEM DMS Data Server CEM Telnet Ftp Handler CEM Store and Forward CEM Bulk Formatter Type Optional Optional Optional Optional Optional Optional Optional Base
ATTENTION
Base = included with the CBM
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Prerequisites
OM Access Service and Table Access Service application filesets must be installed and in service on the core manager. For the wireless market, the Nortel support group must increase the buffer size within the OM Access Service to 2.5 MB to accommodate the amount of data transferred by the front end for a transfer period of every 30 minutes.
Procedure steps
At your workstation
Step 1
Action Create the following logical volume (directory for a file system) required for the OMDD software package you are installing:
makelv /cbmdata/00/omdata 1008
Apply the SDM_OMDD.omdd software package located in the /cdrom/cdrom/applications/cbm/packages directory (refer to Applying software packages on a Core and Billing Manager using the CBMMTC interface for further information).
ATTENTION
You must specify the CD-ROM directory /cdrom/cdrom/application s/cbm/packages as the source directory path when performing the applying software packages procedure.
3 4
Enter cbmmtc config to access CBM maintenance interface config: Enter config <n> to configure OM Data Delivery The system indicates that the Tuple Number option is inactive and prompts you to determine whether you want to activate it.
ATTENTION
The Tuple Number option allows you to activate or disable a tuple number so that it can be included in a CSV file with other OM information.
Enter y to activate or n to disable the tuple number. The system prompts you to confirm whether the Multiservice Data Manager (MDM) and core manager are integrated.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
69
To indicate that the MDM is connected to the core manager for collecting Passport 15000 performance measurement data, type
y
7 8 9 10 11 12
When prompted, enter the IP address of the first MDM you want to connect. When prompted, enter the hostname of the first MDM. When prompted, enter the IP address of the second (alternate) MDM you want to connect. When prompted, enter the hostname of the second MDM. When prompted, enter the port for 5-minute performance PM (performance measurement) data. The default port is 1646. When prompted, enter the port for 30-minute PM data. The default port is 1647. You are prompted as to whether you want to use custom connection retry settings. In case of connection failure, OMDD will try connecting to the MDMs, alternatively.
13 14 15 16 17
Enter y to use custom retry settings. Go to step 16. Enter n to use the default retry settings. Go to step 19. To configure the custom retry settings, enter a numeric value (in seconds) for the first connection retry interval.
ATTENTION
Values higher than 300 seconds are not recommended as they can adversely affect recovery time.
18 19 20 21 22 23 24
Enter the number of retry attempts for the first retry interval. Enter a numeric value (in seconds) for the second connection retry interval. Enter the number of retry attempts for the second connection retry interval. Enter a numeric value (in seconds) for the third connection retry interval. To confirm (save) the configuration data, entery, otherwise entern to reenter all of the configuration data. The system indicates that the configuration is complete. Press the Enter key. The system indicates that the changes will take place after the OM Data Delivery application is restarted. Press the Enter key to restart the OM Data Delivery application.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
25 26
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
71
Log delivery service (base software) Log delivery service client (optional software) Generic Data Delivery (base software) Passport Log Streamer, if the core manager needs to communicate with the Multiservice Data Manager (MDM) for fault data. (optional software)
Prerequisites
Ensure that no disk faults exist on the core manager. Remove any restrictions for the configured MDM ports from all firewalls between the MDM and the CBM to ensure that the Passport Log Streamer can communicate with the configured MDMs and collect logs.
Procedure steps
At your workstation
Step 1
Action Perform the procedure to apply the SDM_LOGS.m dm_21.39.9.0.pkg software package located in the /cdrom/cdrom/applications/cbm/packages directory (refer to Applying software packages on a Core and Billing Manager using the CBMMTC interface for further information). Because you use the CD-ROM to install the application, specify /cdrom/cdrom/applications/cbm/packages as the directory path of the source directory when you perform that procedure. When prompted, enter the IP address for the first MDM node. When prompted, enter the IP address for the second MDM node. When prompted, enter the port number configured for the pserver application on the first MDM node. When prompted, enter the port number configured for the pserver application on the second MDM node.
2 3 4 5
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
6 7 8 9 10
When prompted, type n to indicate that you do not want to receive MDM logs. When prompted, type n to indicate that you do not want to specify Passport 15000 log filters. When prompted, type n to indicate that you do not want to specify Ethernet Routing Switch 8600 log filters. When prompted, enter y to confirm the configuration data. Place the log delivery service application and the Passport Log Streamer application into service by accessing the Application (Appl) level of the CBM maintenance interface:
appl
11 12
Busy the application filesets: bsy <fileset_number> <fileset_number> Return the application filesets to service: rts <fileset_number> <fileset_number> After the application filesets are in service, the system retrieves any current log records. To view or store the log records, see Nortel Core and Billing Manager 850 Fault Management ( (NN10351-911)) .
13 14
Variable denitions
Variable <fileset_number> Value The number next to each of the application filesets.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
73
Procedure steps
At your workstations
Step 1
Action Using the procedure Applying software packages on a Core and Billing Manager using the CBMMTC interface (page 88), apply the SBA and AFT software packages located in the /cdrom/cdrom/applications/cbm/packages directory. Create the necessary logical volumes (directories for file systems) required for the SBA using procedure "Adding a logical volume through the command line" in Nortel Core and Billing Manager 850 Accounting ( (NN10363-811)) . Configure the SBA for operation (refer to Nortel Core and Billing Manager 850 Accounting ( (NN10363-811)) for the appropriate procedures). Configure AFT for operation (refer to Nortel Core and Billing Manager 850 Accounting ( (NN10363-811)) . for the appropriate procedures). This procedure is complete.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Prerequisites
The settings for office parameters eadas_dc_interface and eadas_nm_interface in table OFCVAR, and the settings for the EADAS SOCs (OAM00005 and OAM00006) are correct for your configuration. OAM00004 for EADAS/DC is ON and that office parameters eadas_mpc_and_link and netminder_mpc_and_link are appropriately datafilled in table OFCVAR when BX25 connectivity is required.
Procedure steps
At your workstation
Step 1
Action Using the procedure Applying software packages on a Core and Billing Manager using the CBMMTC interface (page 88), apply the GR740PT software package located in the /cdrom/cdrom/applications/cbm/packages directory. You are prompted to enter the mode of security. If you are not configuring GR740PT in non-secure mode do select 1. Non-secure. If you are configuring GR740PT in local secure mode do select 2. Local (SSH) security. This procedure is complete.
--End--
2 3 4
Job aid
The following table lists the supported configurations for EADAS GR740PT application server:
Table 10 CM EADAS TCP/IP configurations Supported configuratio ns DC and NM over BX25 DC and NM over TCP/IP Setting for eadas _dc_interface X25 TCP_IP Setting for eadas_ nm_interface N/A N/A SOC OAM0 0005 ON ON SOC OAM00 006 IDLE IDLE
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
75
Table 10 CM EADAS TCP/IP configurations (contd.) Supported configuratio ns DC and Netminder over BX25 DC over BX25 and Netminder over TCP/IP DC over TCP/IP and Netminder over BX25 DC and Netminder over TCP/IP Setting for eadas _dc_interface X25 X25 TCP_IP TCP_IP Setting for eadas_ nm_interface X25 TCP_IP X25 TCP_IP SOC OAM0 0005 IDLE IDLE IDLE IDLE SOC OAM00 006 ON ON ON ON
The following table lists the channel assignments for EADAS. Note that DC EADAS channels 1, 2, and 3 support TR-740/746 compliant header and message. NM EADAS channels 1, 2, and 3 support SR3942 and TR746 to Netminder:
Table 11 EADAS channel assignments Description DC EADAS lc 1 DC EADAS lc 2 DC EADAS lc 3 NM EADAS lc 1 NM EADAS lc 2 NM EADAS lc 3 Service name DC_EADAS_LOG_CHAN1 DC_EADAS_LOG_CHAN2 DC_EADAS_LOG_CHAN3 NM_EADAS_LOG_CHAN1 NM_EADAS_LOG_CHAN2 NM_EADAS_LOG_CHAN3 TCP port 9550 9551 9552 9553 9554 9555 MTS offset 234 235 236 237 238 239
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Prerequisites
add RTB to a billing stream change the RTB configuration for a billing stream delete RTB from a billing stream
Configure the billing stream. Perform the procedure "Configuring a billing stream on the core manager" in Nortel Core and Billing Manager 850 Accounting ( (NN10363-811)) . RTB requires outbound file transfer and DIRP file format. Configure outbound file transfer for the stream. Perform the procedure Configuring the outbound file transfer schedule" in Nortel Core and Billing Manager 850 Accounting ( (NN10363-811)) . RTB supports Real-time File Transfer Protocol Wrapper (RFTPW), Real-time Key-based Secure File Transfer (RKSFT), and Real-time Secure File Transfer Protocol (RSFTP) protocols.
maximum number of retry attempts after RTB fails to transfer a billing file before RTB raises a critical alarm directory location on the data processing and management system (downstream OSS) of the RTB test file and partial file
Procedure steps
ATTENTION
Instructions for entering commands in the following procedure do not show the prompting symbol, such as #, >, or $, displayed by the system through a GUI or on a command line. Conguring RTB for a billing stream
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
77
Step 1
Action Log into the core manager as a user authorized to perform accounting-manage actions (refer to Nortel CS 2000 Core Manager Administration and Security ( (NN10170-611)) for information on how to log on to the core manager). Access the BILLMTC interface:
billmtc
2 3 4
Display and verify the schedule tuple: list <stream_name> Verify the following fields:
Before configuring RTB ensure that the fields contain the values shown in this list. For secure file transfer, you may choose SFTPW or RSFTP (password-based user authentication) or RKSFT (key-based user authentication). 5 6 7 8 9 10 11 Access the RTB level:
rtb
To add RTB to a billing stream, go to step 10. To change the RTB configuration for a billing stream, go to step 21. To delete RTB from a billing stream, go to step 33. Add RTB to a billing stream: add <stream_name> <file_format> <destination>
ATTENTION
If auto recovery is turned on, do not configure multiple RTB destinations with the same Test File Location or Partial File Location on the DPMS.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Enter the desired maximum retry attempts before RTB raises a critical alarm, and press the Enter key. The default value is 3. 12 Enter the directory on the DPMS where the RTB test file will reside and press the Enter key. The default directory is the Remote_Storage_Directory as configured in the Schedule tuple for this stream. 13 Enter the directory on the DPMS where the RTB remote partial file resides, and press the Enter key. The default directory is the Remote_Storage_Directory as configured in the Schedule tuple for this stream. Example response:
You entered: RTB Max Consecutive Failures: 5 RTB Remote Test File Location: /sba/autorec RTB Partial File Location: /sba/autorec Commit? [Save] {Save Edit Abort}:
14 15 16 17 18
If the displayed values are not correct, go to step 16. If the displayed values are correct, go to step 17. Edit and correct the displayed values:
edit
Activate the schedule tuple for the stream by performing "Configuring the outbound file transfer schedule" in Nortel Core and Billing Manager 850 Accounting ( (NN10363-811)) . To add RTB to another billing stream, go to step 10. If you do not want to add RTB to another billing stream, go to step 37. Before performing this step, ensure the RTB schedule tuple is in the OFFL state. Change the RTB configuration for a billing stream: change <stream_name> <file_format> <destination>
19 20 21
22
ATTENTION
If auto recovery is turned on, do not configure multiple RTB destinations with the same Test File Location or Partial File Location on the DPMS.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
79
Enter the desired maximum retry attempts before RTB raises a critical alarm, and press the Enter key. The default value is 3. 23 Enter the directory on the DPMS where the RTB test file resides, and press the Enter key. The default directory is the Remote_Storage_Directory as configured in the Schedule tuple for this stream. 24 Enter the directory on the DPMS where the RTB remote partial file resides, and press the Enter key. The default directory is the Remote_Storage_Directory as configured in the Schedule tuple for this stream. Example response:
You entered: RTB Max Consecutive Failures: 5 RTB Remote Test File Location: /sba/autorec RTB Partial File Location: /sba/autorec Commit? [Save] {Save Edit Abort}:
25 26 27 28 29
If the displayed values are not correct, go to step 27. If the displayed values are correct, go to step 28. Edit and correct the displayed values:
edit
Activate the schedule tuple for the stream by performing "Configuring the outbound file transfer schedule" in Nortel Core and Billing Manager 850 Accounting ( (NN10363-811)) . To change the RTB configuration on another billing stream, go to step 21. If you do not want to change the RTB configuration on another billing stream, go to step 37. Deactivate the schedule tuple for the stream by performing "Configuring the outbound file transfer schedule" in Nortel Core and Billing Manager 850 Accounting ( (NN10363-811)) . Before performing this step, ensure the RTB schedule tuple is in the OFFL state. Delete the RTB configuration from a billing stream: delete <stream_name> <file_format> <destination>
30 31 32
33
34
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
35 36
To delete RTB from another billing stream, go to step 33. Quit the BILLMTC interface:
quit all
--End--
Variable denitions
Variable <stream_name> <file_format> <destination> Value Specifies the name of the configured billing stream Specifies the file format of the configured billing stream Specifies the billing file transfer destination
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
81
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Configuring the bkmgrusr user ID and password to enable communication between the DBRM and SBRM (page 82) Configuring core access for SBRM through the Nortel Communication Server 2000 Core Manager (page 84)
Conguring the bkmgrusr user ID and password to enable communication between the DBRM and SBRM
Use this procedure to configure the bkmgrusr user ID and password in order for the Synchronous Backup Restore Manager (SBRM) to communicate with the Device Backup Restore Manager (DBRM) on the Communication Server 2000 Core Manager. This procedure applies only to the Communication Server 2000 Core Manager running on an AIX platform. The procedure does not apply to the Core and Billing Manager (CBM) running on an SPFS-based server. Prerequisites
You must be a user authorized to perform config-admin actions. For information on how to log on to the Communication Server 2000 Core Manager and how to display actions a user is authorized to perform, see Logging in to the CS 2000 Core Manager and Displaying actions a user is authorized to perform in Nortel CS 2000 Core Manager Administration and Security ( (NN10170-611)) .
Procedure steps Step 1 Action At your workstation, log onto the Communication Server 2000 Core Manager on which the DBRM is installed as a user authorized to perform config-admin actions. Create the bkmgrusr user, type mkuser bkmgrusr Create these groups: emsmtc, emsadm, and emsrw, type mkgroup emsmtc mkgroup emsadm mkgroup emsrw Add the bkmgrusr user to the primary group (maint) and to secondary groups, type (on one single line) chuser pgrp=maint groups=emsmtc,emsadm,emsrw
2 3
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Configuring the bkmgrusr user ID and password to enable communication between the DBRM and SBRM 83
home=/export/home/bkmgrusr admin=true shell=/bin/ksh bkmgrusr When you type this command, make sure that you typed a space between emsrw and home, and between ksh and bkmgrusr. 5 Confirm that the bkmgrusr user was added to the required groups, type groups bkmgrusr The system displays the groups to which bkmgrusr user belongs. 6 Set the password for the bkmgrusr user, type passwd bkmgrusr The bkmgrusr user is disabled until this step is performed. 7 8 Log off from the Communication Server 2000 Core Manager. Log on to Communication Server 2000 Core Manager as the bkmgrusr. When prompted, change the password for the bkmgrusr user. 9 Change to the home directory and create the .ssh directory, type cd /export/home/bkmgrusr mkdir .ssh chmod 700 .ssh
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Conguring core access for SBRM through the Nortel Communication Server 2000 Core Manager
Use this procedure to configure access to the core for the Synchronous Backup Restore Manager (SBRM). You must perform this procedure to enable the SBRM to automatically backup a core image. Perform this procedure every time that core user password expires or changes. This ensures that the password you set in this procedure matches that set for the user on the core.
Prerequisites
Perform Configuring the bkmgrusr user ID and password to enable communication between the DBRM and SBRM (page 82) before you perform this procedure for the first time. You must be a user authorized to perform config-admin actions. You must know the user name and password that were configured to enable core access in Configuring the bkmgrusr user ID and password to enable communication between the DBRM and SBRM (page 82). For information on how to log on to the Communication Server 2000 Core Manager and how to display actions a user is authorized to perform, see Logging in to the CS 2000 Core Manager and Displaying information about a user or role group in Nortel CS 2000 Core Manager Administration and Security ( (NN10170-611)) .
Procedure steps
Step 1 Action At the Nortel Communication Server 2000 Core Manager, log onto the core manager with the logon ID and password for a user authorized to perform config-admin actions. If you want to perform this procedure through the command line, at the command line prompt, change the directory to the directory containing appropriate configuration script, type cd /opt/nortel/bkresmgr/cbm/scriptssee If you want to perform this procedure through SDMMTC (SDM maintenance interface), run the configuration script, type ./bkmgr_config.sh then go to step 5. 3 Access the config level of the SDM maintenance interface, type # sdmmtc config
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
85
From the list of filesets, select the Succession Provisioning Data Sync Manager fileset (Backup Restore Manager fileset, SDM_BKM.bkm), and type config As the configuration script runs, you are prompted for the user name. Type the user name enabled to access the core in Configuring the bkmgrusr user ID and password to enable communication between the DBRM and SBRM (page 82).
You are prompted for the password. Type the password that was set up through the procedure, Configuring the bkmgrusr user ID and password to enable communication between the DBRM and SBRM (page 82). You are prompted for the logical volume where the backup is to be stored. This is the device on which the core image dump will be stored. make sure that this device has sufficient space for the backup file.
You are prompted for the core type, xa-Core or Compact. The software uses this information to determine if the core will have a message switch load.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
set dump_restore_in_progress field in ofcstd table start image dump ability to run itocci command set
Perform this procedure before you perform Configuring core access for SBRM through the Nortel Communication Server 2000 Core Manager (page 84).
Procedure steps
Step 1 Action At the CI prompt on the core, type permit <backupuser> <backupuser_pswd> 4 10000 english all If Enhanced Password Control is in effect on the CM and after the user is permitted on the switch, log onto the core manually with this user first. The core will prompt you to change the password at the first logon after the logon is permitted. Change the password and then perform the procedure, Configuring core access for SBRM through the Nortel Communication Server 2000 Core Manager (page 84) using the <backupuser> user you have created and the changed password. If Enhanced Password Control is in effect on the CM, the password must be at least six characters in length. The SBRM does not have the ability to manage passwords. Therefore, you must re-run the configuration script in Configuring core access for SBRM through the Nortel Communication Server 2000 Core Manager (page 84) to ensure the password for the <backupuser> user.
--End--
Variable denitions
Variable <all> Value The privilege setting.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
87
Variable <backupuser>
Value The user name that will be used by SBRM to log on to the core. Maximum length is 16 characters.
<backupuser_pswd>
The password for the <backupuser> user you are creating. Maximum length is 16 characters.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Applying software packages on a Core and Billing Manager using the CBMMTC interface
Use this procedure to install optional software packages on CBM cluster nodes.
Procedure steps
ATTENTION
Instructions for entering commands in this procedure do not show the prompting symbol, such as #, >, or $, displayed by the system through a GUI or on a command line. At your workstation
Step 1
Action In the command line prompt, enter cbmmtc apply to access the apply level of the cbm maintenance interface The system displays the apply level screen of the cbm maintenance interface, which shows a list of the packages (up to 12 software packages are displayed at a time), if any exist, in the default source directory.
2 3 4
To exit the CBM maintenance interface, type no and go to step 13. If the CBM software is on a CD-ROM, insert the CD-ROM into the CD drive if it is not already present in the drive. At the command line located at the bottom of the cbmmtc user interface screen, type:
source <source_directory_name>
Use /cdrom/cdrom/applications/cbm/packages as the <source_directory_name> if the software is on a CD-ROM. The system displays the apply level screen of the cbm maintenance interface, which shows a list of all packages in the source directory that you specified. 5 Locate the packages to be applied from the displayed list and take note of their numbers (located next to the names of the packages). Select the packages that you have decided to apply:
select <package number> ... <package number>
The packages you selected are highlighted on the cbmmtc apply screen. 6 To apply the selected packages:
apply
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
89
If there is a prerequisite package for the package(s) you have selected to apply on the system, the system SWIM tool automatically selects and applies the prerequisite package unless the package is currently selected to be applied. The system prompts if you want to continue with applying the selected packages.
8 9
If you do not want to continue the package application, type no and go to step 13. Type yes in response to the prompt. The status of each package application appears on the cbmmtc apply screen.
10
When the application is completed, the installed packages will appear in the list that displays when you enter the cbmmtc packages level. Verify that the status of the new packages indicates Applied under the Status column.
ATTENTION
It is important that packages installed on the system not be left with a Partial status. If any package installed application fails or otherwise shows a Partial status, contact your next level of support for assistance.
11 12 13 14
Go to step 13. Type no in response to the prompt. Exit from the cbm maintenance interface:
quit all
Variable denitions
Variable <source_directory_name> Value The full pathname of the directory containing the package that you want to apply. Because you are using a CD-ROM for the installation, specify /cdrom/cdrom/applications/cbm/packages as the source_directory_name The number associated with a package, that you noted. Each package number is separated by preceding and succeeding spaces.
<package number>
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
Example To select the Reach Through SPM application, which is number 6, and OM Delivery, which is number 11 in the sample screen display shown in the previous figure, enter select 6 11 To de-select any packages that you selected, re-enter the select command for the packages you want to de-select. The highlighting on the packages that you de-select will be removed.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
91
Procedure steps
ATTENTION
Instructions for entering commands in this procedure do not show the prompting symbol, such as #, >, or $, displayed by the system through a GUI or on a command line. At your workstation
Step 1
and press the Enter key. 2 3 4 If the system prompts you for the Ethernet address, go to step 4. If the system indicates it is using Ethernet address <EthernetAddress>, record the IP address and go to step 9. At the console connected to the inactive node, log on to the inactive node through the console using the root user ID and password. If the system is not already at the OK prompt, bring the system to the OK prompt:
init 0
Example response:
Sun Fire V240, No keyboard Copyright 1998-2002 Sun Microsystems, Inc. All rights reserved. OpenBoot 4.8.0.build_04, 2048 MB memory installed, Serial #52964131. Ethernet address 0:3:ba:28:2b:23, Host ID: 83282b23.
7 8
Record the Ethernet address that is displayed. At your workstation (session connected to Active node), enter the Ethernet address of the inactive node you recorded in step 6.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
9 10 11
If the system prompts you to enter the command: boot net image, go to step 11. If the system does not prompt you to enter the command: boot net - image, to step 12. At the console connected to the inactive node, boot the inactive node from the image of the active node by typing:
OK boot net - image
ATTENTION
There must be a space after the dash.
Example response:
SC Alert: Host System has Reset Sun Fire V240, No Keyboard Copyright 1998-2002 Sun Microsystems, Inc. All rights reserved. OpenBoot 4.8.0.build_04, 2048 MB memory installed, Serial #52964131. Ethernet address 0:3:ba:28:2b:23, Host ID: 83282b23. Rebooting with command: boot net - image . . . SC Alert: Host System has Reset
12
At your workstation (session connected to the Active node), monitor the progress of the cloning from the active node. Cloning the inactive node takes approximately one hour to complete. Example response:
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
93
Waiting for network response from unit1-priv0... received network response from unit1-priv0... Waiting for unit1-priv0 to clone data... waiting...1 waiting...2 waiting...3 unit1-priv0 is cloning: /export/d2 . . . Verifying cluster status of unit1-priv0 waiting for cluster filesystem status to become normal. Deleted snapshot 0. Deleted snapshot 1. Deleted snapshot 2. Deleted snapshot 3. d99: Soft Partition is cleared
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
ATTENTION
Instructions for entering commands in this procedure do not show the prompting symbol, such as #, >, or $, displayed by the system through a GUI or on a command line.
Step 1
ATTENTION
If Enhanced Password Control is in effect on the CM, the password must be at least six characters in length.
ATTENTION
If Enhanced Password Control is in effect on the CM and after the user is permitted on the switch, log in to the core manually with this user first. The core will prompt you to change the password at the first logon after the logon is permitted. Change the password and then perform step 1. The SBRM does not have the ability to manage passwords. Therefore, you must rerun the configuration script in step 4.
At your workstation, apply the software application package, NTbkupmgr . Specify /cdrom/cdrom/applications/cbm/packages as the source directory when you perform that procedure. When the installation is complete, exit from the cbmmtc. At the command line prompt, change directory to the directory containing configuration script:
cd /opt/nortel/bkresmgr/cbm/scripts
3 4
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
95
You are first prompted for the user name. The user name is that used to log in to the core to initiate an image dump. The script restricts the user name to a maximum of 16 characters. The user name entered must first be enabled on the core in step 1 You are prompted for the user name you entered in step 6. You are first prompted for the password. The configuration script restricts the password to a maximum of 16 characters. Use the password set up in step 1 You are prompted for the logical volume where the backup is to be stored. This is the device on which the core image dump will be stored.
7 8
ATTENTION
Verify that this device has enough space to store the backup.
10
You are prompted for the core type, either XA-core or Compact.
ATTENTION
This information is needed in order for the software to know whether the core will also have a Message Switch load.
--End--
Variable denitions
Variable <backupuser> Value The user name for the core, that is up to 16 characters in length, that will be used by SBRM for logon The password for the <backupuser> user you are creating, which can be up to 16 characters in length 4 is the priority
<backupuser_pswd>
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Prerequisites
You need the root user ID and password for the SBRM server. You need the password for the bkmgrusr on the DBRM servers to transfer the SSH key pair. The bkmgrusr is created on the DBRM server when SPFS is installed. You need the root password for the DBRM server when the DBRM server is a two-server configuration, to synchronize the SSH data between the two servers.
ATTENTION
Ensure provisioning activities must not be in progress or scheduled to take place during this procedure.
Procedure steps
At your workstation
Step 1
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
97
In a two-server configuration, log in to the Active side using its physical IP address. 2 3 4 5 When prompted, enter your user ID and password. Change to the root user
$ su -
When prompted, enter the root password. Change directory to the location of the backup restore manager configuration script
# cd /opt/bkresmgr/admin/bin
For an example, see Job aid (page 99). 7 Enter the number next to the Backup Restore Manager SSH setup option in the menu. For an example, see Job aid (page 99). 8 9 10 If you are configuring SSH for the first time, or changing the SSH key pair then, go to 10. If you are transferring an existing SSH key to Device Backup Restore Manager servers then, go to 27. Enter the number next to the Generate SSH key and transfer to Device Backup Restore Manager servers option in the menu. For an example, see Job aid (page 99). 11 When prompted, confirm you want to continue by typing:
y
For an example, see Job aid (page 99). 12 When prompted, press Enter to accept the default location and filename /export/home/bkmgrusr/.ssh/id_rsa for the key. If the default location and filename is not provided, enter /export/home/bkmgrusr/.ssh/id_rsa. If the location and filename /export/home/bkmgrusr/ .ssh/id_rsa already exists and you wish to overwrite, enter y. Otherwise, enter n. For an example, see Job aid (page 99). 13 When prompted, enter y to confirm you want to transfer the SSH key pair. For an example, see Job aid (page 99). 14 When prompted, enter the hostname or IP address of a DBRM server in your network.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
For an example, see Job aid (page 99). 15 When prompted, enter the number next to the platform type of the DBRM server. The SPFS platform type is for servers hosting the CS 2000 Management Tools, MG 9000 Manager and Core and Billing Manager (CBM). The SDM platform type is an AIX platform containing core management software. For an example, see Job aid (page 99). 16 When prompted, enter yes to confirm you want to continue connecting.
Warning: Permanently added comp5iems (45.123.456.98) (RSA) to the list of known hosts. Password:
17
When prompted, enter the password for the bkmgrusr on the DBRM server. For an example, see Job aid (page 99). When prompted, enter y if the DBRM server is a two-server configuration (HA), otherwise, enter n. If you have entered y (yes) then go to 21. If you have entered n (no) then go to 25. When prompted, enter y if you want to manually synchronize the SSH data, otherwise, enter n.
18 19 20 21
ATTENTION
The DBRM server root user password is required to complete this section.
22 23 24 25
If you have entered y (yes) then go to step 24. If you have entered n (no) then go to step 25. When prompted, enter the root password for the DBRM server. For an example, see Job aid (page 99). When prompted, indicate that you want to transfer the SSH key pair to the DBRM installation on the SBRM server in your network. The SBRM always makes a SSH connection to the DBRM residing on the local server so that the local IEMS application is notified of the synchronized backup by way of SBRM to DBRM messaging.
26
Enter the number next to the "Transfer existing SSH key to Device Backup Restore Manager servers" option in the menu. For example, see Job aid (page 99).
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
99
27
When prompted, confirm you want to transfer the SSH key pair by typing:
y
This returns you to step 14. For an example, see Job aid (page 99). 28 29 Enter x to exit the Backup Restore Manager SSH setup level by typing: Enter x to exit the Backup Restore Manager Configuration level.
--End--
Variable denitions
Variable <server> Value IP address or host name of the IEMS server where the SBRM software is installed.
Job aid
Table 12 Example responses to procedure steps Step 6 Example response Backup Restore Manager Configuration 1 - Backup Restore Manager SSH Setup 2 - Automated Synchronous Backup Restore Manager Service X - exit select - [X, 1-2] SSH Setup for Synchronous Backup Restore Manager 1 - Generate SSH key and transfer to Device Backup Restore Manager servers 2 - Transfer existing SSH key to Device Backup Restore Manager servers X - exit select - [X, 1-2] Generate SSH key for Synchronous Backup Restore Manager and transfer to Device Backup Restore Manager servers. WARNING: Running this script will generate a new SSH key for the bkmgrusr. This new SSH key must be transferred to all existing and new Device Backup Restore Manager servers. Do you want to continue? [y/n]:
10
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
100
Step 11
Example response Generating SSH key for bkmgrusr. Sun Microsystems Inc. SunOS 5.9 Generic May 2002 Generating public/private rsa key pair. Enter file in which to save the key (/export/home/bkmgrusr/.ssh/id_rsa): Your identification has been saved in /export/home/bkmgrusr/.ssh/i d_rsa. Your public key has been saved in /export/home/bkmgrusr/.ssh/id_rs a.pub. The key fingerprint is: 33:99:17:f0:ac:2a:3a:86:a9:d9:76:65:e6:70:4b:6:8 bkmgrusr@com p5iems Synchronizing files IF clustered node Transfer existing SSH key to Device Backup Restore Manager servers WARNING: You must have the password for the bkmgrusr on the Device Backup Restore Manager server to transfer the SSH key. Do you want to transfer the SSH key? [y/n]: Setting up to transfer SSH key to DBRM server. Enter hostname or IP address of DBRM server to transfer the SSH key to: Platform type of DBRM server to transfer SSH key to: 1 - SPFS 2 - SDM Enter platform type of DBRM server to transfer SSH key to [1-2]: Sun Microsystems Inc. SunOS 5.9 Generic May 2002 The authenticity of host comp5iems (45.123.456.98) cant be established. RSA key fingerprint is ff:eb:a8:87:14:d0:82:28:5c:43:70:5a:ab:af:e6:f d Are you sure you want to continue connecting (yes/no)?: Transfer of SSH key to <server> complete. Is DBRM <server> a High Availability (HA) server? [y/n]: WARNING: On HA DBRM servers, SSH setup is not complete unless one of the following steps is performed. a) The HA DBRM server <servername> is cloned. b) The SSH data on HA DBRM <servername> is manually synched. WARNING: Choosing this option requires the root password for <servername> Do you want to manually synchronize the SSH data on DBRM <servername> now [y/n] Synchronization of SSH data on DBRM <servername> complete.
12
13
14
15
17 22
25
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
101
Step 27
Example response Transfer existing SSH key to Device Backup Restore Manager servers WARNING: You must have the password for the bkmgrusr on the Device Backup Restore Manager server to transfer the SSH key. Do you want to transfer the SSH key? [y/n]: Setting up to transfer SSH key to DBRM server. Enter hostname or IP address of DBRM server to transfer the SSH key to:
28
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
102
Prerequisites
You must configure additional backup storage to prevent a temporary problem that forces the SBA into long-term backup mode. The billing stream is aware that the replaced volumes exist, and recovers files from both the swapped-out and swapped-in sets of volumes as part of the recovery process. The billing stream loses track of swapped-out volumes when a switch of activity (SwAct) or a restart is performed on the DMS or Communication Server 2000 prior to the completion of the recovery of the files.
There is a risk of losing some billing records when you reconfigure or swap-out backup volumes of a stream that is in backup mode during the transition process. During an emergency situation (outage), you must allow time for the recovery process to finish before you attempt to swap out an active backup volume. If you do not allow time for the recovery process to finish, the billing stream does not recognize the swapped-out volumes. If you are using or migrating to a XAC16 system, your backup volumes must be on IOP volumes. If your current backup volumes are on SLM or DDU volumes and you are running a previous release, you must migrate to IOP volumes before upgrading to this release.
ATTENTION
Ensure the size for backup volumes is sufficient. The absolute minimum size for backup volumes is 30MB.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
103
Procedure steps
Step 1 Action At your system, write down the dms_disk_space value recorded in Table 20 "Disk space requirements" (page 208) (answer 31), which shows the amount of disk space required for the backup volumes. Determine the amount of disk space of each disk type in your system to be used for storing the backup volumes. Divide the value recorded in step 1 by the maximum volume size supported for the appropriate disk types for your system, obtained from the table Table 14 "Metrics on various disk types " (page 106). Record these values. Ensure that the backup volumes can fit on the disks in your system. Compare the values recorded in step 2 with the maximum number of volumes supported for the disk types in your system, obtained from the table in the previous step. If the number of volumes obtained in the previous step is less than or equal to the maximum number allowed, go to step 4. OR If the number of volumes obtained in the previous step is greater than the maximum number allowed, contact the next level of support. 4 5 Post the billing stream by typing: mapci;mtc;appl;sdmbil;post <stream_name> Obtain information about the existing backup volumes for the billing stream : conf view <stream_name> SBA does not support the configuration of more than one billing stream at a time from multiple workstations. The last billing stream that is configured is the one that is saved. The system displays the name of each backup volume in the stream. Record each backup volume name for future reference. 6 7 Quit out of the MAPCI level :
quit all
Display and record the size of a volume and its number of free blocks: diskut;lv <volume name> Repeat the previous step for each volume name that you recorded in step 2.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
104
If you are configuring an IOP disk drive backup volume, create an alphanumeric name, consisting of a maximum of twelve characters, for each of the new backup volumes that you determined in Calculation of core manager Disk Space Requirements (page 209). Record each of these names for future reference. IOP volume names on the IOP disks can be up to twelve alphanumeric characters in length, with the first four characters reserved for the disk prefix. OR Create a twelve-character, alphanumeric name for each of the new backup volumes. Record each of these names for future reference. 3PC volume names on the 3PC disks can be up to twelve alphanumeric characters in length, with the first four characters reserved for the disk prefix.
ATTENTION
Logical volumes must be configured evenly across the disks.
10 11 12
Access the disk administration level: diskadm <disk prefix> Determine the free disk space:
dd
Note the following example, which is a response to the command performed in step 8, choosing the F02L disk name.
13
Determine the size of the largest free segment. If the size of the largest free segment is greater than or equal to the maximum allowable volume size for the disk type, go to the next step. OR If the size of the largest free segment is less than the maximum allowable volume size for the disk type contact your next level of support before proceeding with this procedure.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
105
14
Create a new logical volume : cv <volume> <size> ftfs For an example to create a new logical volume, see "Example" (page 106) .
15 16 17 18
Repeat this procedure until all new logical volumes have been created. Exit to the command prompt:
quit all
You must alert all operating company personnel who are associated with the system. Provide the names of the old and new backup volumes and the procedure you used to swap the volumes. They must be made aware of that any RESTARTs or SwActs that occur before the billing stream returns to normal mode can cause a loss of billing records. Also, it is imperative that the mode of the billing stream must be closely monitored to ensure that it returns to normal mode without an intervening RESTART or SwAct.
--End--
Variable denitions
Variable <disk prefix> <size> <stream_name> <volume> Value One of the prefixes assigned to the two disks; for example, F02L or F17D. The size of the volume. The name of the billing stream. The backup volume name.
Job aid
The following table lists the backup volumes for the various platforms.
Table 13 Disk drive backup volumes Platform BRISC Backup volume(s) DDU or SLM
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
106
Table 13 Disk drive backup volumes (contd.) Platform XA-core (for releases prior to SDM16 or CS2E03) XA-core (for SDM16 or CS2E0 and higher) Backup volume(s) DDU or IOP IOP
The following table provides metrics on various disk types that can be used in this procedure.
Table 14 Metrics on various disk types Disk type Maximum disks per core 2 2 10 2 Maximum volumes per device 32 32 32 32 Maximum volumes configurable for SBA 64 64 69 64 Maximum volume size
The following example shows how the user can create a new logical volume. Example cv AMA8 50 ftfs This entry prompts the system to create the logical volume F17LAMA8, consisting of 50 Mbyte (102400 512-byte blocks) of available disk space.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
107
Prerequisites
You need the root user ID and password for the SPFS-based server where the SBRM software is installed.
ATTENTION
Ensure provisioning activities must not be in progress or scheduled to take place during this procedure.
Procedure steps
Step 1 Action
ATTENTION
In a two-server configuration, log in to the Active side using its physical IP address.
2 3 4 5
When prompted, enter your user ID and password. Change to the root user
$ su - root
When prompted, enter the root password. Change directory to where the backup restore manager configuration script is located
# cd /opt/bkresmgr/admin/bin
For an example, see Job aid (page 108). 7 Enter the number next to the Automated Synchronous Backup Restore Manager Service option in the menu. For an example, see Job aid (page 108).
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
108
8 9 10
If you want to start the automated SBRM service then go to step 10. If you want to stop the automated SBRM service then go to step 12. To start the SBRM service enter the number next to the Start the Automated Synchronous Backup Restore Manager Service option in the menu. For example, see Job aid (page 108). Enter y to confirm you want to start the SBRM service.
y
11
For an example, see Job aid (page 108). Go to step 14 12 To stop the SBRM service enter the number next to the Stop the Automated Synchronous Backup Restore Manager Service option in the menu. For an example, see Job aid (page 108). 13 14 15 Enter y to confirm you want to stop the SBRM service. For an example, see Job aid (page 108). Enter x to exit the Automated Synchronous Backup Restore Manager Service level by typing Enter x to exit the Backup Restore Manager Configuration level by typing
--End--
Variable denitions
Variable server Value IP address or host name of the IEMS server where the SBRM software is installed.
Job aid
Step 6 Example response Backup Restore Manager Configuration 1 - Backup Restore Manager SSH Setup 2 - Automated Synchronous Backup Restore Manager Service X - exit select - [X, 1-2]
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
109
Automated Synchronous Backup Restore Manager Service 1 - Configure Automated Synchronous Backup Restore Manager Service 2 - Start the Automated Synchronous Backup Restore Manager Service 3 - Stop the Automated Synchronous Backup Restore Manager Service 4 - View the Automated Synchronous Backup Restore Manager Service Configuration X - exit select - [X, 1-4] Start the Automated Synchronous Backup Restore Manager Service WARNING: Running this script will start the Automated Synchronous Backup Restore Manager Service Starting the service does not initiate a backup until the day and time configured. Do you want to continue? - [y/n]: Registering with SERVMAN. Starting the Automated Synchronous Backup Restore Manager Service. Starting AUTO_BACKUP_MANAGER through servstart AUTO_BACKUP_MANAG ER Started Stop the Automated Synchronous Backup Restore Manager Service WARNING: Running this script will stop the Automated Synchronous Backup Restore Manager Service Stopping the service does not abort a backup in progress. Do you want to continue? - [y/n]: Stopping the Automated Synchronous Backup Restore Manager Service. Stopping group using servstop AUTO_BACKUP_MANAGER Stopped Deregistering with SERVMAN
10
11
13
14
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
110
Prerequisites
You need the root user ID and password for the IEMS server.
Procedure steps
Step 1 2 3 4 5 Action Log in to the IEMS server by entering the following command:
> telnet <server>
When prompted, enter your user ID and password. Change to the root user by entering:
$ su -
When prompted, enter the root password. Change the directory to where the network backup manager script (SBRM-bkmgr) is located by entering:
#cd /opt/nortel/bkresmgr/nbm
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
111
Variable denitions
Variable <server> <command> Value The IP address of the IEMS server. Any one of the following commands:
backup abort backup auto backup full query history query inventory query status quit or exit
or any of the following commands to perform a step by step backup for debugging:
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
112
Preparation for disaster recovery bkmgr> bkmgr> query status Current State: dumpdata Process Status: idle *** Component wcars2yx *** Step dumpdata for wcars2yx is failed. Current State: dumpdata Process Status: idle *** Component DB *** Step dumpdata for DB is failed. *** Component IS *** Step dumpdata for IS is complete. *** Component NPM *** Step dumpdata for NPM is complete. *** Component CEM *** Step dumpdata for CEM is complete. *** Component IEMS *** Step dumpdata for IEMS is complete. bkmgr> bkmgr> query history Last backup failed at Wed May 21 08:51:31 EDT 2008. *** Component wcars2yx *** *** Component DB *** Failed to create backup files bkmgr> bkmgr> backup abort Beginning step: rollback Completed step: rollback Cleaning up subcomponents... bkmgr> query status Current State: No backup in progress. Process Status: idle bkmgr> query history Last backup failed at Wed May 21 08:51:31 EDT 2008. *** Component wcars2yx *** *** Component DB *** Failed to create backup files bkmgr>
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure job aid Example of invoking the network-level backup (SBRM-bkmgr) through telnet bkmgr> backup auto Beginning step: precheck Completed step: precheck Beginning step: stopprov Completed step: stopprov Beginning step: dumpdata Error during step: dumpdata Auto backup failed. *** Component wcars2yx *** *** Component DB *** Failed to create backup files Restarting provisioning... Beginning step: rollback Completed step: rollback bkmgr> bkmgr> query status Current State: No backup in progress. Process Status: idle bkmgr>
113
precheck: performs a precheck of all devices/components. stopprov: locks the provisioning session of all devices/components. dumpdata: performs a backup of all devices/components. verify: verifies the backup data of all devices/components. startprov: unlocks the provisioning sessions of all devices/compone nts. activatenew: finalizes the backup data (move data to different directory/volume, ) for all devices/components.
To perform a complete backup, you can use one of the following 2 commands provided by the SBRM-bkmgr:
backup full: performs a complete backup of all devices/components. If a failure is encountered, the process stops at completion of the failed step, and you will need to abort or perform manual actions to recover from the failure. backup auto: performs a complete backup of all devices/components. If a failure is encountered, the process restores the provisioning session if it has been stopped.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
114
backup abort: aborts a backup after completion of the current step. The provisioning session will be unlocked if it has been locked by the backup process.
To perform a query, you can use one of the following three commands:
query status: displays the status of the backup at a high level. query history: displays information about the most recent backup. query inventory: displays the components/devices available for backup.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
115
Prerequisites
You need the root user ID and password for the SPFS-based server.
Procedure steps
Step 1 2 3 4 5 Action Log in to the SPFS-based server by entering:
> telnet <server>
When prompted, enter your user ID and password. Change to the root user by entering:
$ su -
When prompted, enter the root password. Change the directory to where device-level backup manager script (DBRM-bkmgr) is located by entering:
#cd /opt/nortel/bkresmgr/cbm
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
116
Variable denitions
Variable <server> <command> Value The IP address of the server. Any one of the following commands:
backup abort backup auto backup full query history query inventory query status quit or exit
or any of the following commands to perform a step by step backup for debugging:
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure job aid *** Component DB *** Step dumpdata for DB is failed. *** Component IS *** Step dumpdata for IS is complete. *** Component NPM *** Step dumpdata for NPM is complete. *** Component CEM *** Step dumpdata for CEM is complete. *** Component IEMS *** Step dumpdata for IEMS is complete. bkmgr> bkmgr> backup abort RC=success bkmgr> query status RC=success~status= Current State: No backup in progress. Process Status: idle bkmgr>
117
precheck: performs a precheck of all devices/components. stopprov: locks the provisioning session of all devices/components. dumpdata: performs a backup of all devices/components. verify: verifies the backup data of all devices/components. startprov: unlocks the provisioning sessions of all devices/compone nts. activatenew: finalizes the backup data (move data to different directory/volume, ) for all devices/components.
To perform a complete backup, you can use one of the following two commands provided by the DBRM-bkmgr:
backup full: performs a complete backup of all devices/components. If a failure is encountered, the process stops at completion of the failed step, and you will need to abort or perform manual actions to recover from the failure. backup auto: performs a complete backup of all devices/components. If a failure is encountered, the process restores the provisioning session if it has been stopped.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
118
backup abort: aborts a backup after completion of the current step. The provisioning session will be unlocked if it has been locked by the backup process.
To perform a query, you can use one of the following three commands:
query status: displays the status of the backup and shows the current state of the backup (that is, precheck, stopprov, dumpdata). query history: displays information about the most recent backup. query inventory: displays the components available for backup.
It is recommended that you use the query status and query history commands to verify the backup result. You should use the backup abort command to unlock the provisioning session if the backup failed.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
119
Procedure steps
Step 1 2 Action Access the MAP CI level display
>QUIT ALL
ATTENTION
The example of a MAP response identifies the autoload registered (ALR) image file by an asterisk (*) in the ALR column. Each image file has an index number at the beginning of the tuple line. The ALR image in the example of a MAP response has an index number of 0. The XA-Core selects the ALR image file first to boot the switch. If the ALR image file does not boot the switch then the XA-Core selects the next image file. The next image file is by sequence of the index number from the top of the table.
For an example, see Job aid (page 123). 4 To list the boot file for the message switch (MS) in ITOC, type
>LISTBOOTFILE MS
For an example, see Job aid (page 123). 5 Determine if the XA-Core and MS have image files that are autoload registered (ALR). The examples of a MAP response for steps 3 and 4 in the Job aid identify the ALR image files by an asterisk (*) in the ALR column. If the image files are ALR then, go to step 8.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
120
7 8
ATTENTION
In the example of a MAP response for step 3 in the Job aid, the name of the office image file for XA-Core is IMG0517CY_CM. Image file IMG0517CY_CM is ALR. Image file IMG0517CY_CM is in volume F02LIMAGE.
ATTENTION
In the example of a MAP response for step 4 in the Job aid, the name of the office image file for the MS is IMG0517CY_MS. Image file IMG0517CY_MS is ALR. Image file IMG0517CY_MS is in volume F02LIMAGE.
Record the names of the office image files for XA-Core and MS that are ALR. Also record the volume name that has these office image files. The ALR image file is the file that you copy to the XA-Core tape. 9 10 Quit the ITOCCI user interface
>QUIT
At the shelf;
WARNING
Static electricity damage
Wear a wrist strap connected to the wrist-strap grounding point of the frame supervisory panel (FSP) when you handle the tape and packlet. The use of the wrist strap protects the packlets against damage caused by electrostatic discharge (ESD).
Determine from office records or office personnel if the DAT tape drive is clean. See the XA-Core procedure How to clean the XA-Core tape drive in the Nortel XA-Core Fault and Performance Management ( (297-8991-510)) . 11 Get a tape cartridge that has the approval of Nortel. Determine the tape planned for a backup of an office image. Determine the tape to use from the office records or from office personnel. Make sure the tape write protection is at the position that permits recording (closed). The tape write protection is an entrance on one side of the tape that has a sliding door. The sliding door is open for write protection and closed to allow a write to the tape.
12
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
121
13
Insert the DAT tape cartridge into the XA-Core tape drive and close the drive door. The XA-Core tape drive is in the input/output processor (IOP) card of the XA-Core shelf.
14
CAUTION
File of tape lost when formatted
If the tape had files, the formatting of the tape loses the files. Make sure the tape has no files that your office needs.
. For an example, see Job aid (page 123). 15 Load the tape into the MAP disk utility >INSERTTAPE snnp TAPE WRITELABEL label_name For an example, see Job aid (page 123). 16 Confirm the command
>YES
ATTENTION
A volume can have more files listed by command LISTVOLS than by command LISTFL in the MAP disk utility. The difference in the number of files between the commands is because of directory files not displayed by command LISTFL.
To list the files in the volume that contains the office image, enter >LISTFL vol_name . For an example, see Job aid (page 123). 18 Begin the disk to tape backup process. To create a backup copy of the XA-Core image file, type: >BACKUP FILE file_name snnp TAPE . For an example, see Job aid (page 123). 19 20 If the command is successful, got step 21. If the command is not successful, go to step 29.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
122
21
Create a backup copy of the MS image fileX >BACKUP FILE file_name snnp TAPE For an example, see Job aid (page 123).
22 23 24
If the command is successful, go to step 24. If the command is not successful, go to step 29. Check the backup copies of the image files on the tapeX >LISTFL snnp TAPE For an example, see Job aid (page 123).
25
Eject the tape from the MAP disk utility after the backup procedure completes >EJECTTAPE snnp TAPE For an example, see Job aid (page 123).
26
Exit the MAP disk utility and return to the MAP CI level
>QUIT
For an example, see Job aid (page 123). 27 Remove the tape cartridge from the tape drive.
WARNING
Static electricity damage
Wear a wrist strap connected to the wrist-strap grounding point of the frame supervisory panel (FSP) when you handle the tape and packlet. The use of the wrist strap protects the packlets against damage caused by electrostatic discharge (ESD).
Set the tape write protection to the position that does not permit recording (open). The tape write protection is an entrance on one side of the tape that has a sliding door. The sliding door is open for write protection and closed to allow a write to the tape.
28 29
Store the tape cartridge according to office procedure. For additional help, contact the next level of support.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
123
Variable denitions
Variable <label_name> Value The alphanumeric name of the tape label that records the data. The name can be up to 32 characters long. If blank spaces are in the label name then enclose the label name with quotation marks. The number of the slot position on the XA-Core shelf for the IOP that has the tape device. The upper (U) or lower (L) packlet position of the IOP that has the tape device. The front (F) or rear (R) shelf position of the input output processor (IOP) that has the tape device. The name of the disk volume that contains the office image files.
<vol_name>
Job aid
Step 2 3 Example response ITOC User Interface is now active. ITOCCI: Image table Of Contents for XA : A Registered Generic Device File L Date Time Name R MM/DD/YYYY HH:MM:SS -- - -------------------- -------------- ----0 * 05/17/1999 19:26:29 F02LIMAGE IMG0517CY_CM Image Table of Contents for MS : A Registered Generic Device File L Date Time Name R MM/DD/YYYY HH:MM:SS -- - -------------------- -------------- ----0 * 05/17/1999 19:26:29 F02LIMAGE IMG0517CY_MS Disk utility is now active. DISKUT:
14
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
124
Step 15
Example response >INSERTTAPE F02UTAPE WRITELABEL IMAGE_1 Example of a MAP response ***** WARNING ***** Writing the label IMAGE_1 to tape volume F02UTAPE on node CM will destroy all files stored on this tape volume. Do you want to continue? Please confirm ("YES", "Y", "NO", or "N")
16
The INSERT operation may take up to 5 minutes to tension the tape. A tape is now available to user on unit 0, node CM. Name IMAGE_1 has been written to the tape label. >LISTFL F02LIMAGE Example of a MAP response File information for volume F02LIMAGE: {NOTE: 1 BLOCK = 512 BYTES } --------------------------------------FILE NAME O R I O O V FILE MAX NUM OF FILE LAST R E T P L L CODE REC RECORDS SIZE MODIFY G C O E D D LEN IN IN DATE C N FILE BLOCKS ---------------------------------------------IMG0517CY_MS I F Y 0 1020 7542 15360 990517 IMG0517CY_CM I F Y 0 1020 165180 329728 990517
17
18 21
FTFS file IMG0517CY_CM on disk volume F02LIMAGE on node CM backed up as file IMG0517CY_CM on tape device F02UTAPE on node CM. FTFS file IMG0517CY_MS on disk volume F02LIMAGE on node CM backup up as file IMG0517CY_MS on tape device F02UTAPE on node CM.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
125
Step 24
Example response >LISTFL F02UTAPE Example of a MAP response File information for tape volume F02UTAPE, node CM: {Note: 1 BLOCK = 512 BYTES} --------------------------------------------CREATE ORG FILE V FILE NUM OF REC FILE NAME DATE TYPE CODE L SIZE IN RECORDS LEN D BLOCKS IN FILE ---------------------------------------------990520 IMAG 0 329070 165180 1020 IMG0517CY_CM 990520 IMAG 0 15026 7542 1020 IMG0517CY_MS
25
The EJECT operation may take up to 5 minutes to position the tape to the beginning. Rewind of tape F02UTAPE, unit 0, on node CM is completed. This tape device is not available to the user now.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
126
Procedure steps
Step 1 Action At the map, list the available images. The image with the asterisk in the ALR column identifies the image that is set to Auto Load Record and is the image to backup.
At the Call Agent Manager, log on to the inactive Call Agent and change directory to the location of the image.
cd /3PC/sd0x/imagex
ATTENTION
Do not modify files at this level. Any modification to files must be completed through the MAP.
Open a file transfer protocol (FTP) session to the Communication Server 2000 Core Manager and transfer the file.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
127
4 5
ATTENTION
Use the rvf argument to append to the tape. After the backup completes, optionally verify the integrity of the backup by using tar tvf /dev/rmt0 to view the contents of the tape.
At the Communication Server 2000 Core Manager, verify that the size of the transferred file is the same as in step 2 and copy the file to tape.
Use the rm command to erase the oldest image so the volume does not fill up.
--End--
Variable denitions
Variable </3PC> Value Prefixed in all cases.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
128
Value Taken from the first four characters in the Generic Device name. Taken from the remaining characters in the Generic Device name.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
129
Prerequisites
Suspend all SIP Gateway application provisioning activities during the time of the database backup to ensure that an accurate and complete copy of the active unit database is created. Call processing is not affected and there is no enforcement in the database to prevent provisioning.
Procedure steps
Step 1 2 Action At the NCGL CLI or IEMS client, log on to the ACTIVE unit and change to the root user. Start the backup
/opt/apps/db_install/sd/bkup_ngss.sh
The following output is printed to the screen and the backup is recorded to /data/bckresmgr/backup.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
130
The backup filename is located in /data/bkresmgr/backup and is identified by hostname, date, and time as indicated in the following example: Example Response: unit0.backupfile.2005-04-12_17-10.tgz
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
131
For security purposes, ensure that a copy of the backup file is transferred to a secure location. Use the scp command to make a secure copy of the backup file to a secure, remote server on your network. This server should be continuously available for cases where a restoration of the unit become necessary, such as during an upgrade rollback.
4 5
If security related files are being backed up, refer to Backing up security related files manually (page 134). Repeat this procedure on the second unit.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
132
Procedure steps
At the NCGL CLI or IEMS client
Step 1 2 3
Action Log onto the unit and change to the root user. Remove the existing backup schedule from the crontab
/opt/apps/db_install/sd/uninstallBkres.sh
Modify the first five fields (indicated by "0 1 * * *" in the example). The fields are identified in order from left to right:
minute -- the minute of the hour that the command will run, values are 0 to 59 hour -- the hour of the day the command will run, values are 0 to 23 with 0 being midnight day -- the day of the month the command will run, values are 1 to 31, an asterisk indicates to run the command all days month -- the month of the year the command will run, values are 0 to 12, an asterisk indicates to run the command all months weekday -- the day of the week that the command will run, values are 0 to 6, with 0 being Sunday, an asterisk indicates to run the command all weekdays
Do not modify the sixth field (/opt/.../bkup_ngss.sh). When done, write and quit the file. 5 Install the new schedule in the crontab
/opt/apps/db_install/sd/installBkres.sh
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
133
After the scheduled time, verify that a backup file is created in the /data/bkresmgr/backup directory and that it is created at the new time, not the previously scheduled time. Repeat this procedure on the second unit.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
134
Procedure steps
Step 1 Action At the NCGL CLI or IEMS client, log on to either unit (usually the unit where the latest version of security certificates are stored), and change to root user. Navigate to the /opt/base/share/ssl directory:
cd /opt/base/share/ssl
2 3
Create a new directory to store backup copies of the certificate files: mkdir <SNxx_ddmmyyyy> Copy the certificates to the newly created backup directory:
cp * <SNxx_ddmmyyyy>
ATTENTION
Completing this step ensures that you have valid backup copies of the security certificates for restoring in case of an upgrade abort or rollback or for disaster recovery purposes.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
135
ATTENTION
If you want to make backup copies of the security certificates to a remote server then do ensure that the remote host that will store the security related files is secure, that the host is unavailable to general users, and offers restricted access to personnel with security related responsibilities.
Secure copy the files to the remote server: cd <SNxx_ddmmyyyy>scp *<user>@<remote_server>:</ path> The files are copied to the remote server.
--End--
Variable denitions
Variable <SNxx_ddmmyyyy> Value is the name of the new directory based on the currently installed release of the system software (for example SN08) and the current date in the format ddmmyyyy is a valid user ID on the remote server is the IP address of the remote server is where the file will be located on the remote server
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
136
ATTENTION
You must be directly connected to the switch to initiate a Boot Monitor session. You can connect using a Telnet connection only if the Boot Monitor CLI is already active.
Procedure steps
Step 1 Action
ATTENTION
To save a file to the standby CPU, you must enable TFTP on the standby CPU. To enable TFTP, enter flags tftpd true in the Boot Monitor CLI or config bootconfig flags tftpd true in the Run-Time CLI.
Variable denitions
Variable backup <value> Value Saves the specified file name and identifies the file as a backup file. Example save config file ralph.cfg backup 2 file <value> A filename in one of the following formats: [a.b.c.d]: <file>
/pcmcia/<file> /flash/<file>
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
137
Value Specifies what to save. Possible values for this parameter are config, bootconfig, log, and trace. Saves the specified file name to the standby CPU (currently not supported). Saves default and current configuration. If you omit this parameter, only parameters you have changed are saved.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
138
Prerequisites
You need the root user ID and password for the SPFS-based server where the SBRM software is installed.
Procedure steps
At your workstation
Step 1
In a two-server configuration, log in to the Active side using its physical IP address. 2 3 4 5 When prompted, enter your user ID and password. Change to the root user
$ su -
When prompted, enter the root password. Change to the directory where the backup restore manager configuration script is located
# cd /opt/bkresmgr/admin/bin
For an example, see Job aid (page 140). 7 Enter the number next to the Automated Synchronous Backup Restore Manager Service option in the menu. For an example, see Job aid (page 140). 8 Enter the number next to the View the Automated Synchronous Backup Restore Manager Service Configuration option in the menu to view the configuration settings..
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
139
For an example, see Job aid (page 140). 9 10 11 If you want to change the configuration settings of the SBRM service then go to step 11. If you do not want to change the configuration settings of the SBRM service then go to step 19. To change the configuration settings, enter the number next to the Configure Automated Synchronous Backup Restore Manager Service option in the menu. For an example, see Job aid (page 140). 12 13 14 15 When prompted, enter the number next to daily or weekly. For an example, see Job aid (page 140). If you select weekly then go to step 15. If you select daily the go to step 16. When prompted, enter the number next to the day of the week on which you want the backup to execute, or press the Enter key to accept the default value. When prompted, enter the time of day at which you want the backup to execute, or press the Enter key to accept the default. For an example, see Job aid (page 140). 17 If you want to start the automated SBRM service now, perform Starting or stopping the automated synchronous backup restore manager service (page 107). If you do not want to start the automated SBRM service now then go to step 11. Enter x to exit the Automated Synchronous Backup Restore Manager Service level. Enter x to exit the Backup Restore Manager Configuration level.
--End--
16
18 19 20
Variable denitions
Variable <server> Value IP address or host name of the IEMS server where the SBRM software is installed.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
140
Job aid
Step 6 Example response Backup Restore Manager Configuration 1 - Backup Restore Manager SSH Setup 2 - Automated Synchronous Backup Restore Manager Service X - exit select - [X, 1-2] Automated Synchronous Backup Restore Manager Service 1 - Configure Automated Synchronous Backup Restore Manager Service 2 - Start the Automated Synchronous Backup Restore Manager Service 3 - Stop the Automated Synchronous Backup Restore Manager Service 4 - View the Automated Synchronous Backup Restore Manager Service Configuration X - exit select - [X, 1-4] The Automated Synchronous Backup Restore Manager Service is not running Automated Synchronous Backup Restore Manager Service Configuration Settings Backup Enabled : N Backup Day : SUNDAY Backup Hour : 24 Automated Synchronous Backup Restore Manager Service Configuration Configuring the Automated Synchronous Backup Restore Manager Service does not START the service. Return to the previous menu and select Start the Automated Synchronous Backup Restore Manager Service to start the service. Do you want the backup to run 1.daily or 2.weekly [1/2] : Example response if daily is selected: Hour to run the Synchronous Backup Restore Manager based on 24 hour clock: Example 1 is 1 AM, 12 is noon, and 24 is midnight. This is the hour on the <DAY> the Synchronous Backup Restore Manager will execute a full backup. Enter hour to run the Synchronous Backup Restore Manager (default: 24) Example response if weekly is selected: Day of the week to run the Synchronous Backup Restore Manager: This is day of the week the Synchronous Backup Restore Manager will execute a full backup. 1 - MONDAY 2 - TUESDAY 3 - WEDNESDAY 4 - THURSDAY 5 - FRIDAY
11
12
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
141
6 - SATURDAY 7 - SUNDAY Enter day of week to run the Synchronous Backup Restore Manager (default: SUNDAY): 16 Automated Synchronous Backup Restore Manager Service Configuration complete. Configuring the Automated Synchronous Backup Restore Manager Service does not START the service. Return to the previous menu and select Start the Automated Synchronous Backup Restore Manager Service to start the service.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
142
Step 1
Action Select Applications -> IEMS -> Network Elements (refer to the figure below).
2 3
Select the Media Server 2000 series node. Right click the Media Server 2000 series node and select Configure MS2000 Automated INI Backup from the pop-up menu. The Automated INI File Backup Configuration pop-up menu displays.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
143
Click on the Enable Automated INI file backup box. The Middle portion of the pop-up window changes from gray to black. If the Weekly radio button is selected, the days of the week listed at the bottom of the screen are black as well.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
144
5 6 7 8
Enter a start time for the backup. Click the radio button for either daily or weekly automated backups. If the weekly radio button is selected, click the boxes of the day(s) of the week for which you want the backup to occur. Click on the OK button to complete the automatic INI backup configuration.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
145
Step 1
Action Select Applications -> IEMS -> Network Elements (refer to the figure below).
2 3
Select the Media Server 2000 series node icon. Right click the Media Server 2000 series node icon and select the Config and Maintenance Tool from the pop-up menu.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
146
4 5
Select the Password Management tab. Select the SNMP tab from the Password Management screen.
6 7
Enter the passwords for the read or write community strings. Click the Change Read String or Change Write String button to complete the password change.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
147
Prerequisites
Procedure steps
Step 1 2 3 4 5 Action At the Windows desktop interface Open a Telnet connection to the CS 2000 Management Tool. Log on with the appropriate user name. Invoke the MS 2000 Series CLUI /opt/nortel/NTsesm/bin/ms2000.sh When prompted, enter the appropriate user name and password. The Media Server 2000 Series CLUI Main Menu is displayed. Enter the number for the Back up the INI files for all nodes menu option. Sample response Backup of .ini for node 172.17.40.230 completed and copied to SDM Press <enter> to continue 6 7 8 Press Enter. The Media Server 2000 Series CLUI Main Menu appears. Enter x to exit the menu. Enterexit to end the Telnet session.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
148
Prerequisites
Procedure steps
Step 1 2 3 4 5 6 7 Action At the Windows desktop interface Open a Telnet connection to the CS 2000 Management Tool. Log in with the appropriate user name. Invoke the MS 2000 Series CLUI /opt/nortel/NTsesm/bin/ms2000.sh When prompted, enter the appropriate user name and password. The Media Server 2000 Series CLUI Main Menu is displayed. Enter the number to select the Node Maintenance and Configuration menu option. When prompted, enter the IP address of the Media Server node. The Main Menu is displayed. Enter the number to select the Configuration Menu, menu option. The Main Menu is displayed. 8 Enter the number to select theDisplay this nodes current configuration menu option. The current configuration data for the selected node is displayed. 9 10 11 Press Enter. The Main Configuration Menu is displayed. Follow the prompts to return to the Media Server 2000 Series CLUI Main Menu. Enter x to exit the Media Server 2000 Series CLUI Main Menu.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
149
12
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
150
To back up these files, use the following procedures: Performing a backup of Oracle/MySQL data on an SPFS-based server (page 156) to back up the Oracle database. This utility runs automatically each day at 1 a.m. and backs up the database to a 4mm DAT tape. The utility can also be run manually to back up the database to a disk file. Performing a backup of file systems on an SPFS-based server (page 164)to back up the UNIX file system, including all of the "Root" and "non-Root" database files.
Instructions for performing these procedures are found in NN10409-500. Two APS-specific utilities enable you to back up selected files, when only files required for APS operation must be restored. The "ips_export_db.sh" utility backs up the APS Oracle database. The "backup_appl_data.sh" utility backs up only the non-Root application files, "/audio_files," "PROV_data," "/user_audio_files," and Root application file, "/etc/inet/hosts." Both of these utilities can only be run manually. This procedure enables you to perform a complete manual backup of the APS-specific Oracle database files and application files. It is recommended that this procedure be performed once for each week.
Procedure steps
In a telnet connection to the APS server
Step 1 2
Action Open an xterm window, and log on using the "maint" logon and password. Become the "root" user by entering:
su - root
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
151
The system displays a log of the backup activity. 4 List the content of the tape on the terminal screen to verify the backup by entering the following commands:
cd /audio_files/aps_db_backup ls -l
A listing of the backed-up files displays. Look in this list for "dmp" files.
more README
A timestamp displays, which should show the time and date of this backup.
ATTENTION
At this point, you have completed the disk-only portion of the backup. Continue with the next step to complete the backup of the application files. The application files are backed up on tape.
Insert a write-enabled (white or grey tab is moved to the right where it can be seen) DAT tape into the 4mm DAT drive on the APS server. Rewind the tape
mt -f /dev/rmt/0c rewind
The system displays a log of the backup activity. 7 8 List the content of the tape on the terminal screen to verify that the backup was successful
tar tvf /dev/rmt/0c | more
Eject the backup tape, label it, and move the write-enable tab to the "read-only" position (white or grey tab is moved to the left where it cannot be seen), to prevent the data on the tape from being accidentally over-written. Store the tape for use later. Insert another write-enabled DAT tape into the drive to be used
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
152
for the automatic Oracle system back up that runs daily at 1:00 a.m.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
153
Prerequisites
Before you perform a backup, you need to start up the Backup Controller and Providers.
Procedure steps
Step 1 Action Enter the following command as one continuous command
/opt/MagellanNMS/bin/nsbck [-full <devtype> <devname> <devaddr> <id> <pw>] [-incr <devtype> <devname> <devaddr> <id> <pw>] [-view <viewname> <devtype> <devname> <devaddr> <id> <pw>][-cdir <backup_dir_path> [-f <backup_info_file>][-chost <controller_address>] [-nolog] | [-log [<logfile>]] [-nc <#_concurrent_connections>]
The service data backup files have the following naming convention:
./<devtype>/<devname>/<timestamp>.<dataset>/ \ <datafiles...>
ATTENTION
The period (.) represents the SNMP Devices Backup and Restore root directory.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
154
Variable denitions
Variable <backup_info_file> Value the name of the file containing backup information. Each line in the file has the same format as the -full, incr, or -view options. The backup directory used for this backup operation. the address of the Controller and has the format host[:port] The names of one or more files that are backed up. the dataset name. the IP address of the device and has the format n.n.n.n. the name of the device. The device type (PP4400, or PP4460) is the user ID for a Passport and the READ community string for a Passport 4400/4460. the name of the file to which output messages go. If not specified, the messages go to the file mbrbackup.log on the current directory (where the tool runs). the user password for a Passport and the WRITE community string for a Passport 4400/4460. has the format yyyymmddhhmmss indicates a selective backup. the name of the view file for a selective backup. the number of concurrent backups. indicates the remote Controller to be used in place of the Controller running on the local host. indicates that the backup information is obtained from a file. indicates a full backup. indicates an incremental backup. indicates that output messages go to stdout/stderr or to a log file. The default is stdout/stderr.
<pw>
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
155
Variable -nc
Value indicates the number of concurrent backups to be performed. By default, SNMP Devices Backup tries to back up 5 different devices concurrently. This parameter is useful when you are backing up a large number of devices. indicates that output messages are to be discarded.
-nolog
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
156
Multiservice Data Manager (MDM) Communication Server Management Tools Integrated Element Management System (IEMS) Audio Provisioning Server (APS) Media Gateway (MG) 9000 Manager Network Patch Manager (NPM) Core and Billing Manager (CBM)
An Oracle/MySQL data backup is not required when the SPFS-based server is hosting the CBM or the MG 9000 Manager. If the SPFS-based server is hosting the IEMS, it is highly recommended to purge the IEMS event and performance data prior to executing the data backup. This reduces the size of the space used by the IEMS, and therefore, reduces the backup time, and possibility of a backup failure. The purge capability is only available in (I)SN07 onward. For more information, see "Configuring dark office backups on an SPFS-based server" in Nortel ATM/IP Solution-level Configuration ( (NN10409-500)) .
ATTENTION
It is recommended that provisioning activities be put on hold during the time of the data backup.
Prerequisites
For a Sun Netra 240, you need one or more blank DVDs to store the data. The backup utility limits the storage to 4GB on a DVD-R or DVD-RW. If you are using a new DVD-RW or want to reuse a DVD-RW by erasing the contents, perform procedure Preparing a DVD-RW for use in Nortel ATM/IP Solution-level Administration and Security ( (NN10402-600))
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
157
ATTENTION
The database must be in sync with the Communication Server 2000 and the MG 9000 Manager (if present). Therefore, ensure you have the appropriate image of both before you proceed. Performing a restore from the Oracle/MySQL database alone can cause data mismatches at the Communication Server 2000 and the MG 9000 Manager (if present).
Procedure steps
ATTENTION
In a two-sever configuration, execute this procedure on the Active server.
Step 1
Action At the server, insert the blank DVD into the drive. In a two-server configuration, insert the blank DVD into the drive of the Active server. At your workstation, log on to the server by typing
> telnet <server>
and press the Enter key. 3 4 When prompted, enter your user ID and password. Change to the root user by typing
$ su -
and press the Enter key. 5 6 When prompted, enter the root password. If the server is hosting the IEMS, and you want to purge the event and performance data, go to step 7, otherwise, go to step 10
ATTENTION
This step stops the IEMS server. Ensure it is acceptable at this time to stop the IEMS server application.
8 9 10
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
158
Example response:
[12/30/06:17:35:03] cs2k0bd:/> /opt/sspfs/bks/bkdata Backup behavior is set to 1 ... *** Note: 8X or higher DVD-RW media may result in a lower success rate. *** *** DVD will be ejected after the backup is complete *** Do you wish to continue the backup (yes or no)? [yes] yes
11 12
Reinsert the backup DVD into the drive. List the content of the DVD
# tar tvf /cdrom/*bkdata*/*.tar
Example response:
-rw-rw-rw- 0/1 1874917 Mar 2 10:17 2005 opt/oracle.dmp .gz -rw-rw-rw- 0/1 1007616 Mar 2 10:17 2005 opt/critdata.c pio
When a DVD backup spans more than one disk, all the DVDs with the exception of the last one produce a file error during the verification process. This error message does not interfere with the backup process but can reappear several times as the backup spans multiple disks. 13 14 Go to the root directory
# cd /
If the DVD drive tray does not open after you have determined that the DVD drive is not busy and is not being read from or written to, enter the following commands in the order listed:
# /etc/init.d/volmgt stop
Then press the eject button located on the front of the DVD drive.
# /etc/init.d/volmgt start
15
Remove the DVD from the drive, label it, and store it in a safe place.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
159
16
To restore the data from DVD, refer to the procedure "Restoring the oracle data on an SPFS-based server" in Nortel ATM/IP Solution-level Administration and Security ( (NN10402-600)) .
--End--
Variable denitions
Variable <server> Value IP address or hostname of the SPFS-based server on which you want to perform the backup. In a two-server configuration, enter the physical IP address of the active server.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
160
Procedure steps
ATTENTION
At the time of installation, the UAS is configured to automatically back up configuration files each day at 2:00 am. If an APS node is configured in the network, all UAS nodes in the network can be backed up to the APS node. If an APS node is not configured in the network, the configuration files for UAS nodes in the network can be backed up to a remote UNIX server instead. This procedure enables you to set up automatic backup to a remote server. At the Windows desktop interface
Step 1 2 3
Action Open the command interface, select Start -> Run then type cmd in the displayed window. Open a telnet session to the remote UNIX server and log in as the Root user. Enter the following commands to create a new backup directory
cd /;mkdir /opt;chmod 777 opt cd /opt; mkdir uas;chmod 777 uas cd uas; mkdir uas_conf_backup;chmod 777 uas_conf_backup cd / cd /opt/uas/uas_conf_backup
ATTENTION
The commands from this point forward are specific to a Sun Solaris system only.
Configure NFS to share the "/opt/uas" filesystem and start the NFS server
echo "share -F nfs /opt/uas" >> /etc/dfs/dfstab
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
161
Create a user logon called "Administrator" that does not require a password:
/usr/sbin/useradd -d /export/home/Administrator -g 1 -s /bin/ksh -m -u 1002 Administrator 2> /dev/null passwd -d Administrator 2> /dev/null
At the local system console, enter the IP address of the remote server in the "Backup Storage IP" field of the Local Configuration Interface GUI screen, using the procedure "Modifying configuration parameters through the Local Configuration Interface GUI" in the document (NN10095-511), entitled "UAS Configuration Management."
--End--
Job aid
Back up UAS configuration files are stored in configuration files. The configuration files include:
uas.conf - containing configuration parameters that support the function of the UAS, including CG6000C card settings, Call Agent definition, APS hostname definition, network element settings, and conferencing service state definition ugw.conf - containing trunk configuration information for PRI Solutions snmpd.cnf - containing parameters that support the SNMP function, including management station address, SNMP user names, community names, and trap version hosts - containing parameters that support the function of the APS, including APS hostname and IP address atmhard.con - containing ATM bearer interface settings that link a local port ATM address to a particular ATM interface port atmconn.con - containing ATM bearer connections settings that provide the UAS with a remote gateways name and ATM address mainsa.conf - containing Main Subagent program settings specifying the kinds of error and log messages to be sent to the management station atmSvcProfile.con - containing data on Switched Virtual Channel (SVC) traffic parameters associated with AAL2 SVCs atmhardloop.con - containing information associated with the loopback of SVCs
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
162
ATTENTION
If you perform provisioning changes during this procedure, it can lead to database corruption and possible system outages. To ensure that you create an accurate and complete copy of the active unit database, verify that all provisioning changes are stopped before continuing this procedure.
Prerequisites
ATTENTION
The Policy Controller database and the CS 2000 database must always be in sync. CS 2000 core images, CS 2000 GWC Manager Oracle database backups, and the Policy Controller application backups must always be maintained in sync for emergency recovery. To ensure they are in sync, perform images and backups at the same time.
Procedure steps
Step 1 2 3 Action Log on to the active Policy Controller unit and change to the root user. Change directory to the database directory by typing
# cd /opt/apps/database/solid/dbfiles
ATTENTION
If other backup copies of the database exist with the same filename, you have the option of deleting those files or putting the backup copy into the backup directory under a new file name.
Put a copy of the database for the active unit in the backup directory by typing
# cp solid.db /opt/apps/database/solid/backup
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
163
Job aid
For security purposes, ensure that a backup copy of the database file is transferred to a secure location. Use the UNIX scp command to make a secure copy of the backup database file to a secure, remote server on your network. This server should be continuously available for cases where a restoration of the Policy Controller application database become necessary, such as during an upgrade rollback. A root password for the remote server may be required.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
164
Prerequisites
Communication Server 2000 Management Tools Integrated Element Management System (IEMS) Audio Provisioning Server (APS) Media Gateway (MG) 9000 Manager Network Patch Manager Core Billing Manager (CBM)
For Sun Netra 240, use one or more blank DVD-R or DVD-RW disks to store the data. The backup utility limits the storage to 4 GB on a DVD-R and DVD-RW. If you are using a new DVD-RW, or want to reuse a used DVD-RW and need to erase the contents, complete procedure "Preparing a DVD-RW for use" in Nortel ATM/IP Solution-level Administration and Security ( (NN10402-600)) .
For Nortel Core and Billing Manager 850, you must be a user in a role group authorized to perform config-admin actions. For Nortel Core and Billing Manager 850, you must obtain non-restricted shell access.
Procedure steps
ATTENTION
In a two-sever configuration, execute this procedure on the Active server.
Step 1 2
Action Insert the blank DVD into the drive. In a two-server configuration, insert the blank DVD into the Active server. Log on to the server by typing > telnet <server>
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
165
and pressing the Enter key. 3 4 When prompted, enter your user ID and password. Change to the root user by typing
$ su -
and pressing the Enter key. 5 When prompted, enter the root password In a two-server configuration, ensure you are on the active server by typing ubmstat. If ClusterIndicatorSTBY is displayed in the response, which indicates you are on the inactive server, log off of that server and log on to the other server through telnet using the physical IP address of the other unit. The response must display ClusterIndicatorACT, which indicates you are on the active server 6 Back up the file systems by typing
# /opt/nortel/sspfs/bks/bkfullsys
and pressing the Enter key. When using a DVD, the system will prompt you to insert another blank disk if more than one is needed. 7 When prompted, continue the backup by typing
yes
ATTENTION
If you need to abort an in-progress backup, press Ctrl+c and wait for confirmation that the backup has been aborted.
166
9 10
Insert the backup DVD into the drive. If the backup resides on multiple DVDs, insert the first backup DVD. Access the command line interface by typing
# cli
and pressing the Enter key. 11 Enter the number next to the Other option. Example response
Other 1 - Log Rotation 2 - copylogs (Logviewer - Capture Various Logs For Debugging Purposes) 3 - logextractor (Logviewer - Extracting compressed log files captured by c...) 4 - logbrowser (Logviewer - Browse logs on the system) 5 - logviewer utility cleanup (Logviewer - Clean up space use by Logviewer ...) 6 - capt_rtes (Capture run-time environment for debugging) 7 - free_disk_space (Free Some Disk Space) 8 - sun_explorer (Execute the Sun Explorer Data Gathering Tool) 9 - mount_image (Mount A Generic Iso Image To The SSPFS unit) 10 - umount_image (Un-Mount A Generic Iso Image From The SSPFS Unit) 11 - bklist (List DVD backup archive) 12 - validate_pam_conf (Validate and/or fix module pointers in /etc/pam.conf...) 13 - set_disk_quota (Set maximum disk space allowed for a user)
12
List the content of the DVD by entering the number next to the bklist option. Example response
=== Executing "bklist" cdrin0: no media 15:34:24 NOTE: Please insert DVD volume 1, hit enter to continue, X to exit
If the content resides on multiple DVDs, the system will prompt you to insert each DVD. Example response
-r-xr-xr-x 0/2 209596 2002-03-14 09:29 usr/dt/bin/dtc alc -r-xr-xr-x 0/2 738032 2002-03-14 09:49 usr/dt/bin/dtcm 17:49:30 NOTE: End of volume
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps 17:49:30 NOTE: Ejecting DVD.. 17:49:36 NOTE: Checking DVD cdrom0: no media Please insert the next volume disk and hit x <enter> to abort: 17:52:56 NOTE: Waiting for DVD to mount -r-xr-xr-x 0/2 103616 2002-03-14 09:49 usr/dt/bin/dtc m_delete -r-xr-xr-x 0/2 644920 2003-06-19 02:38 usr/dt/bin/dtf ile
167
13
When prompted, press the Enter key to execute the command. When the command has completed successfully, you will see a system response. Example response
-rw-r--r-- 0/1 600 2007-02-09 16:27 .disk_conf/fslist. out 17:59:11 NOTE: Backup Verification Passed 17:59:11 NOTE: gtar command exited with:0 17:59:11 NOTE: End of backup archive listing ==== "bklist" execution completed
14
and pressing the Enter key. If the disk drive tray will not open after you have determined that the disk drive is not busy and is not being read from or written to, enter the following commands:
# /etc/init.d/volmgt stop # /etc/init.d/volmgt start
Then, retry the eject cdrom command previously shown and press the Enter key. 16 Remove the DVD from the drive, label it, and store it in a safe place. The label should include the date of backup, disk number of backup, host name, and unit number. If the backup resides on multiple DVDs then insert the next backup DVD in the disk drive and go to step 15.
--End--
17
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
168
Variable denitions
Variable <server> Value The IP address or host name of the SPFS-based server on which you are performing the backup. In a two-server configuration, enter the physical IP address of the active server.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
169
A backup of the oracle database A backup of the servman critical system files A backup of the central security database and files
ATTENTION
Ensure that there is enough space in the /data partition for each of these backups before continuing.
Prerequisites
The user must have root user privileges to perform this procedure.
Procedure steps
Step 1 Action At your workstation establish a connection to the IEMS server through Telnet or SSH, using the host name or IP address of the server. In a two-server configuration, establish the connection to the active server using the physical IP address of the active server, and ensure you are on the active server using the ubmstat command. For detailed steps, refer to . 2 3 Log on using the root user ID and password. Select your next step.
If this server is hosting the CMT and IEMS this server is only hosting the IEMS Do the next step step 6
4 5 6
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
170
7 8
9 10 11
Contact your next level of support. Exit the backup manager utility by entering:
quit
The Security Services configuration settings and data are backed up to the following file: /data/bkresmgr/<date><time>backupSS2.0<host_name>.tar
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
171
Prerequisites
Procedure steps
Step 1 2 3 4 5 Action At your workstation, open A telnet session to the server by typing > telnet <server> When prompted, enter your user ID and password. Change to the root user by typing su - root When prompted, enter the root password. Enter the following command:
cat etc/critdata.conf
The system returns a list of all non-oracle data files that will be backed up from the client machine. 6 Use the Synchronous Backup Restore Manager (SBRM) to backup the central security client data. When SBRM is run, all of the SPFS data, including the security client data and oracle data, is backed up. For details, see .
--End--
Variable denitions
Variable <server> Value IP address or host name of the server where IEMS resides.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
172
A full backup copies all service data on the selected device or devices. An incremental backup copies only service data changed or created since the last backup. Like the full backup, you can perform an incremental backup on either one or multiple devices. A selective backup copies specific service data that you select. You can perform a selective backup on either one or multiple devices.
The following information applies to using the Service Data Backup and Restore tool to backup Multiservice Switch 7400/15000 or Media Gateway 7400/15000 nodes:
Adding Multiservice Switch 7400/15000 or Media Gateway 7400/15000 devices to the backup list (page 173) Removing Multiservice Switch 7400/15000 or Media Gateway 7400/15000 devices from the backup list (page 173) Viewing the Multiservice Switch 7400/15000 or Media Gateway 7400/15000 backup repository (page 174) Defining a specific userid and password for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 in the backup list (page 174) Changing the Default User Authentication (page 175) Performing a full backup for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 device (page 175) Performing a selective backup for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 device (page 176)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Removing Multiservice Switch 7400/15000 or Media Gateway 7400/15000 devices from the backup list 173
Adding Multiservice Switch 7400/15000 or Media Gateway 7400/15000 devices to the backup list
Use this procedure to add devices to the list of devices that you wish to backup. Procedure steps Step 1 Action Open the Backup and Restore window. From the MDM window, select Configuration->Passport Devices->Administration->P assport Service Data Backup/Restore. Select the Backup Configuration tab. Select Add to launch the Add Devices Dialog. From the left pane, select a group of devices or expand the group and use the Ctrl key to select multiple devices within a group. From the drop down list, in the right pane, select a backup mode (Incremental or Full). If a specific userid and password is required for the device, enter the values in the user ID and Password fields and uncheck the Use default checkbox. If you wish to use the default userid and password, click the Use defaultcheckbox. Click OK. The devices are displayed in the Device List. 9 This procedure is complete.
--End--
2 3 4
5 6
7 8
Removing Multiservice Switch 7400/15000 or Media Gateway 7400/15000 devices from the backup list
Use this procedure to remove devices from the list of devices that you wish to backup. Procedure steps
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
174
Step 1
Action Open the Backup and Restore window. From the MDM window, select Configuration->Passport Devices->Administration->P assport Service Data Backup/Restore. Select the Backup Configuration tab. In the Device List, select the devices you wish to remove. Click Remove. In the confirmation dialog, select Yes to confirm or No to cancel the removal. This procedure is complete.
--End--
2 3 4 5 6
Viewing the Multiservice Switch 7400/15000 or Media Gateway 7400/15000 backup repository
If the backup server is running locally, use the following procedure to view all the backup files in the backup repository. Procedure steps Step 1 2 3 4 Action Telnet into the remote workstation with the appropriate userid and password. Navigate to the repository directory. Use Unix directory commands to view the contents of the backup repository. This procedure is complete.
--End--
Dening a specic userid and password for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 in the backup list
A specific userid and password can be defined when you add the node to the node list or you can set it later using the following procedure. Procedure steps
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Performing a full backup for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 device 175
Step 1
Action Open the Backup and Restore window. From the MDM window, select Configuration->Passport Devices->Administration->P assport Service Data Backup/Restore. Select the Backup Configuration tab. Select a node in the Device List section. In the Device Details section, in the Authentication tab, enter a userid and password. Clear the Use defaultcheckbox. This procedure is complete.
--End--
2 3 4 5 6
2 3 4
Performing a full backup for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 device
Use this procedure to perform a complete system backup of a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 device.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
176
Procedure steps Step 1 Action Open the Backup and Restore window. From the MDM window, select Configuration->Passport Devices->Administration->P assport Service Data Backup/Restore. Select the Backup Configuration tab. Select Add to launch the Add Devices Dialog. From the left pane, select a group of devices or expand the group and use the Ctrl key to select multiple devices within a group. Click in the Mode title and selectFull from the drop-down list. Click Backup. The progress of the backup is displayed in the Messages area. If the backup is unsuccessful, an error dialog is displayed that specifies the device and the reason for the failure. Note: To cancel a backup in progress click Cancel. 7 This procedure is complete.
--End--
2 3 4
5 6
Performing a selective backup for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 device
Use the following procedure to backup only a single specified file to the repository. Procedure steps Step 1 Action Open the Backup and Restore window. From the MDM window, select Configuration->Passport Devices->Administration->P assport Service Data Backup/Restore. Select the Backup Configuration tab. Select Add to launch the Add Devices Dialog. From the left pane, select a group of devices or expand the group and use the Ctrl key to select multiple devices within a group.
2 3 4
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Performing a selective backup for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 device 177
5 6 7
Select a device from the Devices list. In the Mode title, select Selective from the drop-down list. The Configuration column in the table is enabled. Click in the Configuration cell to display a pull-down list that displays all the available views on the Multiservice Switch 7400/15000 or Media Gateway 7400/15000. Note: This step may take a few seconds to complete because the application must access the Multiservice Switch 7400/15000 or Media Gateway 7400/15000 and list all the views names.
Click Backup. The progress of the backup is displayed in the Status area. If the backup is unsuccessful, an error dialog is displayed that specifies the device and the reason for the failure. Note: To cancel a backup in progress click Cancel.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
178
Procedure steps
At the OAMP workstation
Step 1 2 3
Action Click Administration>backup. Enter a description of the data snapshot in the snapshot-descri ption box. You can enter up to 32 characters. Click the corresponding check box to activate the transfer snapshot to alternate boot server Nortel recommends that you enable this option. If you dont select this option, a warning dialog box appears. Click Yes to continue.
Click the corresponding checkbox to activate the set as active snapshot on alternate boot server Nortel recommends that you enable this option. If you dont select this option, a warning dialog box appears. Click Yes to continue.
5 6 7 8 9 10 11
Click the corresponding checkbox to activate the delete oldest snapshot on alternate boot server if maximum reached (16) Click the corresponding checkbox to activate the delete oldest snapshot on rtc if maximum reached (5) Click Create to create the data snapshot. This can take several minutes, depending on system activity. Click Close to close the Backup Active RTC window and return to the Administration window. If you selected transfer snapshot to alternate boot server on alt boot server, go to step 15. Select Administration>file-manager. The File Manager window appears. Select the active RTC system node from the Destination list in the right window pane.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
179
12
Select the new data snapshot from the Snapshot box in the Destination portion of the window. Make note of the timestamp of the data snapshot. Copy the snapshot to your alternate boot server by clicking <--. An hourglass appears while the data snapshot is copied. The boxes in the Source portion (left window) of the window are updated with the information for the copied data snapshot when the copy operation is complete. These files are large and can take several minutes to copy from an RTC system node to your alternate boot server.
13
14 15 16
When the file is transferred, click Close. Update the Alternate Boot Snapshot using the ABS Settings ABS window: Open the Alternate Boot Server (ABS) Configuration Manager window by clicking ABS Settings on the Administration>alternate-boot-server window. Select an alternate snapshot by clicking the arrow button. Click Modify to save your changes and close the window. Your backup is complete.
--End--
17 18 19
Job aid
The same backup procedure may be performed through the CLI.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
180
Preparation for disaster recovery Table 15 CLI return codes Rcode 0x40100001 0x40100002 0x40100003 0x40100004 0x40100005 0x80100001 0x80100002 0x80100003 0x80100004 0x80100005 0x80100006 0x80100007 0x80100008 0x80100009 0x8010000A Rcode text snapshot-description must be string, length (1 - 32), no additional character filter rtc-delete-oldest-option must be string, length (1 - 1), boolean (Y, y, N, n) abs-transfer-option must be string, length (1 - 1), boolean (Y, y, N, n) abs-active-option must be string, length (1 - 1), boolean (Y, y, N, n) abs-delete-oldest-option must be string, length (1 - 1), boolean (Y, y, N, n) missing snapshot-description insufficient disk space on rtc maximum number of snapshots reached on rtc maximum number of snapshots reached on abs unable to delete oldest snapshot on rtc unable to delete oldest snapshot on abs unable to transfer snapshot to abs unable to set snapshot as active on abs unable to connect to abs, run abs test for details invalid combination of command options
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Prerequisites
181
ATTENTION
If you want to schedule automatic system image backups, see SDM Security and Administration document.
Prerequisites
boot (startup) files AIX operating system system configuration data SDM CS 2000 Core Manager software
ATTENTION
This procedure must be performed ONLY from a local console by a trained AIX system administrator with root user privileges.
ATTENTION
All volume groups on the SDM CS 2000 Core Manager must be fully mirrored (Mirrored) before performing this procedure. If not, an error message is displayed.
ATTENTION
If your system includes the SuperNode Billing Application (SBA), you must use tape drive DAT0 to perform this procedure.
ATTENTION
The files under the /data file system are temporary files only, and are excluded from system image backup.
initial installation and commissioning of the SDM CS 2000 Core Manager changes to the configuration of disks or logical volumes
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
182
installation of a new version of SDM CS 2000 Core Manager platform software installation of new hardware changes or upgrades to existing hardware
A system image backup takes a minimum of 10 minutes to complete, depending on the size of your file systems.
Recommended tapes
To complete this procedure, use one of the digital audio tape (DAT) drive tapes approved by Nortel. The brands approved by Nortel are: Hewlett Packard (HP), Maxell, Verbatim, Imation. The tape lengths approved by Nortel are:
The 125M tape is approved for UMFIOs only, assuming that your system is equipped with DDS3-capable devices to read the content of the tape.
Procedure steps
The following flowchart provides an overview of the procedure. Use the instructions in the procedure that follows the flowchart to perform the procedure.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps Figure 13 Summary of creating system image backup tapes (S-tapes)
183
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
184
Creating system image backup tapes (S-tapes) At the local VT100 console
Step 1 2
Action Log on to the SDM CS 2000 Core Manager as the root user. Access the maintenance interface:
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
185
CAUTION
System mirroring must be MIRRORED
You cannot perform this procedure until disk mirroring of all volume groups is Mirrored. If necessary, contact the personnel responsible for your next level of support. When disk mirroring is Mirrored, continue this procedure.
Example response:
Volume Group Status Free(MB) rootvg Mirrored 608 Logical Volume Location Size(MB) %full/threshold 1 / rootvg 20 25/ 80 2 /usr rootvg 192 85/ 90 3 /var rootvg 64 11/ 80 4 /tmp rootvg 24 6/ 90 5 /home rootvg 300 4/ 70 6 /sdm rootvg 300 44/ 90 Logical volumes showing: 1 to 6 of 6
4 5 6 7
If the disks are "Mirrored" then go to step 6 If the disks are not "Mirrored" then contact next level of support Access the administration (Admin) menu level of the RMI:
> admin
Example response:
Currently there is a backup running on bnode73.Please execute yours later.Exiting . . .
ATTENTION
If another operator attempts to use the Backup and Restore utility when it is in use, an error message is displayed.
From the System Image Backup and Restore Menu, select Create a System Image on Tape (S-tape):
>2
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
186
After you select option 2, you are prompted to select the tape drive. Example response:
Select the tape drive you wish to use: Enter Enter Enter ( 0, 1 0 to return to previous menu 1 for tape drive DAT0 in Main Chassis-Slot 2 2 for tape drive DAT1 in Main Chassis-Slot 13 or 2 ) ==>
ATTENTION
Use tape drive DAT0 (option 1) if your system also includes SBA.
ATTENTION
If your system includes SBA, and you wish to use tape drive DAT1 (option 2), the following message is displayed:
Example response:
You have selected DAT 1. This is the default DAT drive for the Billing application, and may currently be in use for the emergency storage of billing records. If you continue to use DAT 1, make sure that the correct tape is in the drive, and that billing records will not be lost during the backup restore operation. Do you wish to continue with DAT 1? ( y | n )
If you wish to continue using DAT1 then do enter y press the Enter key If you do not wish to use DAT1 then do enter n press the Enter key The system prompts you to return to the System Image Backup and Restore Menu if you do not wish to use DAT1. After you select the tape drive, you are prompted to insert a tape in the drive you have selected. Example response:
Please insert a 4mm DAT tape into the tape drive DAT0. Caution: This action will overwrite the content on the inserted tape.Do you want to proceed? ( y | n ) ==> At the SDMCS 2000 Core Manager
10
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
187
CAUTION
System image backup tape
Creating a system image overwrites the contents of the inserted tape. Ensure that you are using the correct tape before starting the system image backup. If your system includes SBA and you are using DAT1, ensure that the tape drive does not contain an SBA tape.
Ensure that the appropriate SDM CS 2000 Core Manager tape drive contains a 4-mm digital audio tape (DAT) either 90 m or 120 m long. This tape will be designated as the system image backup tape (S-tape).
ATTENTION
For the complete list of approved tapes, see the Recommended tapes (page 182). At the local console
11 12
When you are certain you are using the correct tape, enter:
>y
Read the system message to determine if there is enough room on the temporary directory for the system image backup to proceed.
ATTENTION
If there is not enough room on the temporary directory, an error message appears.
Example response:
Rewinding the tape... The /tmp directory is not big enough. Trying to expand /tmp by 6600KB... Failed to expand the /tmp directory because there isnt enough free disk space left on the rootvg. Please erase some files under /tmp directory to create at least 6600KB for the full system image backup. Enter any key and return to exit ==>
13 14 15
If there is enough disk space then do step 14 If there is not enough disk space then do step 11 Erase enough files from the temporary directory to create the required amount of disk space specified in the error message:
> rm -rf /tmp/<filenames>
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
188
ATTENTION
If you have trouble erasing files from the temporary directory to free up disk space, contact the personnel responsible for your next level of support.
16
Execute the system image backup again. The system image backup begins. Example response:
Rewinding the tape... Starting the system image backup on bnode73. The backup takes a minimum of 10 minutes, depending on the size of your file systems. When the backup is complete, you will be asked to remove the tape from the tape drive. System image backup is in progress ...
ATTENTION
This backup process takes approximately 10 minutes to complete, depending on the amount of data stored on the disk.
17 18 19 20
Read the system message. If the backup is successfully completed then do step 16 If the backup fails then do contact your next level of support The system informs you if the backup is successful. When the backup is complete, the system prompts you to remove the tape and label it as an S-tape. Example response:
The tape backup started on Wed Oct 16 08:21:15 EDT 1997 completed successfully on Wed Oct 16 08:37:37 EDT 1997. A log file /tmp/s-tape.log has been created. Please remove the backup tape from the tape drive. Label the tape as shown below and store it in a safe place. System Image Tape (S-tape) The Machine Node Id: bnode73 Date: Wed Oct 16 08:37:37 EDT 1997 Eject the S-tape from the tape drive? ( y | n ) ==>
21
y to eject the tape, or n if you do not wish to eject the tape, and wish to list its contents.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
189
If you protect and label the tape then do step 16 If you eject the tape, the screen displays "Tape ejected." below the information displayed in step 14. You are then prompted to return to the System Image Backup and Restore Main Menu. Response:
Tape ejected. Would you like to return to the previous menu? ( y | n)
22 23
Place the write-protected tab of the S-tape in the open position, to prevent accidental erasure. When you are ready for the system to return to the System Image Backup and Restore Main Menu, enter
>y
24
Determine if the backup is successful. The system informs you if the system image backup is successful, as shown in the response in step 14. You may also wish to view the s-tape.log file or list the files on the S-tape.
25 26 27 28 29 30 31 32
If you wish to view the s-tape.log file then do step 19 If you wish to list the S-tape then do step 28 If the backup is successful then do step 38 If the backup fails then do contact your next level of support Exit the System Image Backup and Restore Main Menu:
>0
This screen informs you that the system image backup was completed successfully. Example response:
bosboot: Boot image is 5881 512 byte blocks. 0+1 records in. 1+0 records out. Backing up the system... ..................................................... ..... 0512 038 mksysb: Backup Completed Successfully.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
190
Preparation for disaster recovery The S-tape backup started on Wed Oct 16 09:24:07 EDT 1997 completed successfully on Wed Oct 16 09:36:03 EDT 1997
33 34 35 36 37 38 39 40
Determine if you wish to list the S-tape. If you wish to list the S-tape then do step 24 If you do not wish to list the S-tape then do step 40. Return to the login directory:
# cd
From the System Image Backup and Restore Menu, select List Contents of the System Image Tape (S-tape):
>3
41
After you select option 3, you are prompted to select the tape drive. Example response:
Select a tape drive you wish to use: Enter Enter Enter ( 0, 1 0 to return to previous menu 1 for tape drive DAT0 in Main Chassis-Slot 2 2 for tape drive DAT1 in Main Chassis-Slot 13 or 2 ) ==>
ATTENTION
Use tape drive DAT0 (option 1) if your system also includes SBA.
42
Example response:
You have selected DAT 1. This is the default DAT drive for the Billing application, and may currently be in use for the emergency storage of billing records. If you continue to use DAT 1, make sure that the correct tape is in the drive, and that billing records will not be lost during the backup restore operation. Do you wish to continue with DAT 1? ( y | n )
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
191
If you do not wish to use DAT1, the system prompts you to return to the System Image Backup and Restore Menu. 43 44 If you wish to continue using DAT1 then enter
y
ATTENTION
If your system includes SBA, and you still wish to use DAT1 (option 2), the following message is displayed:
45
After you select the tape drive, you are prompted to insert the S-tape into the tape drive that you selected in step 30. Example response:
Please insert your System Image Backup tape (S-tape) into the tape drive DAT0 and allow at least 5 minutes to complete the listing. A log file will be saved in /tmp/s-tape.toc. Are you ready to proceed? ( y | n )
46
ATTENTION
Wait until the tape drive stabilizes (yellow LED is off) before you proceed. At the local VT100 terminal
47 48
The contents of the S-tape are displayed. When the listing is complete, the system prompts you to return to the System Image Backup and Restore Menu. Example response:
Would you like to return to the previous menu? ( y | n )
49
50 51 52
If you have not already done so, remove the S-tape from the tape drive by pressing the eject button on the tape drive. Label the tape according to your office practices, and store it in a safe location. If you ejected an SBA tape, reinsert the tape.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
192
53
ATTENTION
If you wish to exit the RMI, enter QUIT ALL.
--End--
Variable denitions
Variable <n> Value is the option (1 or 2) for the tape drive you wish to use
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
193
Prerequisites
Procedure steps
Step 1 Action At the local or remote VT100 console Log into the core manager as a user authorized to perform security-view actions. 2 3 4 Access the maintenance interface, type sdmmtc Access the Admin level, type admin Access the User level, type user A list of users and authorization groups is displayed. 5 Display the actions a user in a particular role group is authorized to perform, type dispgrpr <group> act Sample response for the secadm group Authorized actions for secadm security-admin security-manage security-view 6 Exit the maintenance interface, type quit all
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
194
Variable denitions
Variable <group> Value Value can be: maint, secadm, or emsadm
Job aid
Role groups allow you to group individual users according to the task each user has to complete. The following table lists the standard actions for each role group in a standalone system.
Role group emsadm Standard action Authorized to perform these actions: fault-view, fault-manage, fault-admin, accounting-admin, config-view, config-manage, config-admin, accounting-view, accounting-manage, accounting-admin, performance-view, performance-manage, performance-ad min Gr740Oss Authorized to perform these actions: performance-manage maint Authorized to perform these actions: fault-view, fault-manage, config-view, config-manage, accounting-view, acc ounting-manage, performance-view, performance-manage passthru root (user) secadm No authorized actions. Authorized to perform all actions. Authorized to perform these actions: security-view, security-manage, security-admin
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
195
Prerequisites
You must be a user authorized to perform actions associated with the procedure. It is recommended that you log in to the CS 2000 Core Manger through SSH (secure shell) using a password.
Procedure steps
At the local or remote VT100 console
Step 1
Action Log in to the CS 2000 Core Manager using one of the following commands for SSH access:
ssh <userID>@<IPaddress | hostname>
2 3
Enter your password. Access a user interface, for example, access the maintenance interface:
sdmmtc
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
196
Variable denitions
Variable <userID> <IPaddress> <hostname> Value is your user ID is the IP address of the CS 2000 Core Manager is the host name for the CS 2000 Core Manager
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
197
The number of days for storing the billing files in the file system and within the file system capacity for the current new stream.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
198
Table 16 General stream information (contd.) # 4 Question What is the associated stream name? Explanation associated_stream This question applies only to filter streams. The associated stream name parameter specifies the name of the associated CM billing stream. Type: string Range: 1 to 4 characters Example: AMA, OCC (not case sensitive) 5 What is the name of the Filter Criteria file? filter_criteria_file This question applies only to filter streams. Enter the filter criteria file name that contains the expression to be applied for the filtered stream. Type: string Range: 1 to 255 characters (case sensitive) 6 What is the record format of this stream? record_format The stream record format on the SBA must match the record format of the DMS Switch stream. The only record formats supported by this product and release are Answer
BC (Bellcore AMA format) SMDR (Station Message Detail Recording) CDR300 CDR250
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
199
Table 16 General stream information (contd.) # Question Explanation Type: enumeration Range: BC, SMDR, CDR300, CDR250 (not case sensitive) 7 What is the file format of this stream? file_format This is the format of the billing files that SBA creates on the core manager. Type: enumeration Range: DNS, DIRP (not case sensitive) 8 What is the name of the logical volume on the core manager for storing the billing files for this stream? logical_volume_name The logical volume is the name of the directory where the billing files are stored for this stream. Type: string Range: 1 to 255 characters 9 Will file transfers for this stream be initiated by file_transfer_mode Billing files always move from SBA to the downstream destination, but the file transfers can be initiated by Answer
If Outbound is chosen, the SBA must be configured with additional file transfer information. The outbound file transfer questionnaires must be completed. If Inbound is chosen, the outbound file transfer questionnaires are not needed. Type: enumeration Range: Inbound, Outbound Default: Outbound (not case sensitive)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
200
Table 16 General stream information (contd.) # 10 Question What is the desired state for the stream? Explanation sba_stream_state The stream state controls where the records are sent. Answer
ON: records are sent only to the SBA OFF: records are sent only to an existing DIRP system BOTH: records are sent to both SBA and to an existing DIRP system
The BOTH state is intended for startup verification of SBA processing against DIRP processing. Extended use of the BOTH state can result in SBA performance problems. A system generating more than 175000 CDRs per hour does not support BOTH or OFF mode. Type: enumeration Range: On, Off, Both (not case sensitive) 11 Do you want the files renamed with close date? files_renamed_with_ close_date This question applies only if the file format is DIRP. Type: Boolean Range: Yes, No Default: No (not case sensitive) 12 Do you want to reset DIRP sequence number at midnight? This question appears only when you answer No to the question, "Do you want the files renamed with close date?" Reset_DIRP_sequence_number _ at_midnight This parameter enables resetting the DIRP sequence number to zero after midnight, before opening a new billing file. Type: Boolean Range: Yes, No Default: No (not case sensitive)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
201
Table 16 General stream information (contd.) # 13 Question Do you want the files closed for file transfer and writetape? Explanation files_closed_on_file_ transfer This question applies only if the file format is DIRP. Type: Boolean Range: Yes, No Default: No (not case sensitive) 14 Do you want DIRP blocks closed based on time (applicable only for DIRP file format)? This question appears only when file_type=DIRP and record_format=BAF or CDR250. DIRP_blocks_closed_ based_on_time This parameter specifies whether the DIRP blocks are to be closed after a defined elapsed time. SBA block flushing does not support customized DIRP file formats that do not allow hex AA padding at the end of a block. This type of DIRP file expects CDRs to be of equal size, and each block ends with a special event record. Therefore, GSP and MCI CDR DIRP files are not supported. It is recommended that block flushing be used with real-time transfer mechanisms such as Real-Time Billing (RTB). Type: Boolean Range: Yes, No Default: No (not case sensitive) 15 File DIRP block closure time limit (in seconds) This question appears only when you answer Yes to DIRP_ blocks_ closed_base d_on_time (question 13) DIRP_block_closure_ time_limit This parameter specifies the maximum amount of time in seconds that a DIRP block is kept open before it is closed. Type: Integer Range: 1 through 120 Default: 1 Answer
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
202
This section contains a series of questionnaires that you must complete before you configure the SuperNode Billing Application (SBA) on the core manager for the first time. The following table contains a list of questions concerning general stream information. Record your answers in the spaces provided.
Table 17 General stream information # 1 Question What is the name of this stream? Explanation stream_name The stream name on the SBA must match the stream name on the DMS Switch. This name must match a stream name in the CM table CRSFMT. Type: string Range: 1 to 4 characters. (not case sensitive) 2 What is the desired disk utilization? desired_disk_utilization The percentage of disk utilization that should be maintained under normal conditions. The billing application will attempt to keep the disk utilization for the volume containing this billing data. Type: integer Range: 1090 3 Will file transfers for this stream be initiated by file_transfer_mode Billing files always move from SBA to the downstream destination, but the file transfers can be initiated by Answer
SBA (this is called outbound) or the downstream destination (this is called inbound)
If Outbound is chosen, the SBA must be configured with additional file transfer information. The outbound file transfer questionnaires must be completed.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
203
Table 17 General stream information (contd.) # Question Explanation If Inbound is chosen, the outbound file transfer questionnaires are not needed. Type: enumeration Range: Inbound, Outbound Default: Outbound (not case sensitive) 4 What is the file format of this stream? file_format This is the format of the billing files that SBA creates on the core manager. Type: enumeration Range: DIRP (not case sensitive) 5 Do you want the file to be rotated on a regular schedule What do you want the primary file rotation schedule to be? Value: Yes or No Answer
file_rotation_schedule 1. Rotates files every 24 hours daily (that is, close files at midnight each day) 2. Rotates files every 12 hours daily (that is, close files at noon and midnight each day) 3. Rotates files every 6 hours daily (that is, close files at 6 am, noon, 6 pm and midnight each day) 4. Rotates files every hour on the hour daily (that is, close files at 1 am, 2 am, 3 am, daily etc.) Type: integer Range: 14
Do you want file sequence number to be reset at midnight? Do you want to close the file on record criteria?
Value: Yes or No
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
204
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
205
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
206
Table 19 File closure limits (contd.) # 24 Question Do you want Files closed at scheduled intervals from midnight? Explanation file_closed_at_scheduled_i ntervals_from_ midnight The response to this prompt determines if SBA closes billing files at scheduled intervals calculated relative to midnight. If the answer to this question is Yes, you are prompted to choose options, as shown in question #24. If the answer is No, skip question #24 and go on to question #25. Type: Boolean Range: Yes, No Default: No (not case sensitive) 25 What is the scheduled file closure time option for this Stream? scheduled_file_closure_tim e_option Choose a time in response to a prompt to determine when a billing file is closed, if you answer Yes to question #23. Ignore this question if the answer to question #23 is No. Options: 1) Close billing files every 2) Close billing files every 3) Close billing files every 4) Close billing files every 5) Close billing files every 6) Close billing files every 7) Close billing files every 8) Close billing files every 9) Close billing files every Type: number Range: 1 through 9 Default: 5 Answer
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
207
Table 19 File closure limits (contd.) # 26 Question What is the maximum number of records generated each day for this stream? Explanation records_per_day This is used to calculate the maximum number of Answer
Type: number Units: Records per day 27 What is the maximum size of a record? bytes_per_record This is used to calculate a value for the maximum number of bytes per file. Type: number Units: Bytes per record Range: none 28 What is the maximum number of records per billing file for this stream? records_per_file This controls the maximum number of records a billing file can contain before SBA automatically closes the file. The recommended value based on a target of 300 files a day will be calculated and provided as the default value, if the average number of records per day is one or more. Type: number Units: records per file Range: BC 10000 to 50000016384 00 SMDR 1000 to 5000001638400 29 What is the maximum number of bytes per billing file for this stream? bytes_per_file This controls the maximum size (in bytes) of a billing file before SBA automatically closes it. A recommended value is calculated with the following formula: Records per day * average record size / 300 = Bytes per file
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
208
Table 19 File closure limits (contd.) # Question Explanation Type: number Units: bytes per file Range: BC:1000000 to 104857600 SMDR: 100000 to 104857600 30 What is the average record size? (not applicable if the number of records per day is 0) average_record_size This parameter specifies the maximum size of a record. The default value is 80, but depends on the record type and the record size as defined on the CM. This prompt appears when the Number of records per day parameter is set to a value other than zero (0) Answer
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
209
Question
Explanation The formula for calculating SBA-required disk space on the core manager is described in Calculation of core manager Disk Space Requirements (page 209). Type: number Units: Mbytes Range: NA Default: none Space is allocated in 16 Mb increments.
Answer
32
How much disk space is needed for backup of billing records on the DMS Switch for this stream?
dms_disk_space If the DMS switch is unable to send the billing records to the core manager, they are backed up to the DMS disk space. The allocated DMS disk space must be capable of holding at least a one day accumulation of SBA billing records. The formula for calculating SBA-required disk space on the DMS switch is described in Calculation of Disk Space Requirements (page 210). Type: number Units: Mbytes Range: NA Default: none
BBHCA (Billable busy hour call attempts), multiplied by the ALCR ALCR (average length of a call record in bytes), multiplied by 10 hours, multiplied by CRRD (Call-record retention days), divided by 1048576 (the number of bytes in a megabyte), divided by the desired disk utilization.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
210
For this calculation, the desired disk utilization is a percentage that is expressed as a decimal from 0.1 and 0.9. This formula must be applied to each billing stream with the total of all streams representing the total megabytes of disk space required. The maximum number of files to hold billing records for a billing stream is 15,000. The maximum number of files in closedSent directory is 4,000. The calculation of 10 hours multiplied by BBHCA is an experience-based factor that can be used to estimate 24 hours of call traffic. If you know that the stream has a high BBHCA for more or less than 10 hours per day, increase or decrease the hours value. Calculation Example Assumptions:
BBHCA = 150000 Average length of call records = 85 bytes Call retention days = 10 Desired disk utilization = 0.6 (60%)
BBHCA (Billable busy hour call attempts) multiplied by ALCR (Average length of a call record in bytes), multiplied by 10 hours, multiplied by CRRD (Call-record retention days)
This formula must be applied to each billing stream with the total of all streams representing the total DMS Switch disk space required. The calculation of 10 hours multiplied by BBHCA is an experience-based factor that can be used to estimate 24 hours of call traffic. If you know that the stream has a high BBHCA for more or less than 10 hours per day, increase or decrease the hours value.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
BBHCA = 150000 Average length of call records = 85 bytes Call retention days = 2
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
212
Question
Explanation Type: numeric String Range: 1 to 15 characters Default: none Example: Eventure
Answer
34
protocol FTPW uses the File Transfer Protocol RFTPW (real time file transfer protocol wrapper) is used for the Real-Time Billing (RTB) application. RFTPW is supported only if the RTB application is configured. SFTPW (secure file transfer protocol wrapper) provides secure outbound file transfer using the OpenSSH SFTP client. SFTPW is supported only if OpenSSH is installed on the Core Manager. RSFTP (password-based user authentication) used for secure Real-Time Billing application when configured with OpenSSH SFTP client on the Core Manager. The initial host key acceptance of the downstream processor must be performed manually before the SFTP is used to transfer files. This must be performed for each downstream destination. KSFTP (key-based secure file transfer protocol wrapper) provides secure outbound file transfer with public key authentication, using the OpenSSH SFTP client. KSFTP is supported only if OpenSSH is installed on the Core Manager. RKSFT (key-based authentication) used for secure Real-Time Billing application when configured with OpenSSH SFTP client on the Core Manager.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Question
Explanation sendtape does not support SFTPW/RSFT P. Type: enumeration Range: FTPW, RFTPW, SFTPW, RSFTP, KSFTP , RKSFT, Default: FTPW (not case sensitive)
Answer
35
primary_destination The primary destination is the IP address that the SBA logs on to, and transfers the billing files. Type: IP Address Range: 0.0.0.0 to 255.255.255.25 5 Example: 47.202.35.189
36
primary_port The primary port number is associated with the primary IP address. Type: numeric Range: SFTPW, RSFTP, KSFTP, or RKSFT: 22, 1025 to 65535 Default: 22, for SFTPW, RSFTP, KSFTP, or RKSFTSFTPW or RSFTP Example: 22 FTPW or RFTPW: 21, 1025 to 65535 Default 21, for FTPW or RFTPW Example: 22
37
alternate_destination The alternate destination is the IP address that the SBA logs into and transfers the billing files if SBA encounters problems in connecting to the primary destination. If there is no alternate destination, make this entry identical to the primary IP address. Type: IP Address Range: 0.0.0.0 to 255.255.255.25 5 Example: 47.202.35.189
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
214
# 38
Explanation alternate_port The alternate port number is associated with the alternate IP address. Type: numeric Range: SFTPW or RSFTP: 22, 1025 to 65535 Default: 22 for SFTPW or RSFTP SFTPW, RSFTP , KSFTP, or RKSFT: 22, 1025, 65535 FTPW or RFTPW: 21, 1025 to 65535 Default: 21 for FTPW or RFTPW Default: 22 for SFTPW, RSFTP, KSFTP, or RKSFT Example: 22 FTPW or RFTPW: 21, 1025 to 65535 Default 21, for FTPW or RFTPW
Answer
39
What is the logon for the downstream destination for this stream?
remote_login The SBA user ID needed to log on to the downstream destination, and to transfer the billing files. Type: string Range: 1 to 20 alphanumeric characters Default: none Example: amadns (case sensitive)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
# 40
Question What is the password for the logon ID in question 24 for this stream?
Explanation remote_password This is the SBA password used to log on to the downstream destination to transfer the billing files. Type: string Range: 1 to 20 alphanumeric characters Default: none Example: abracadabra (case sensitive) The prompt for this entry is not generated if the RKSFT/KSFTPRSFTP (real-time/key-based file transfer authentication) protocol is selected)
Answer
41
What is the directory path on the downstream destination where the transferred billing files are to be stored?
remote_storage_ directory This is the full path to the directory on the downstream destination to which SBA transfers billing files. If this value is a period (.), the SBA FTP client does not issue a change working directory (CWD) command when a file transfer occurs. Type: string Range: 1 to 255 characters Example: /users/amadns/billing (case sensitive)
42
field_separator This is a single character that the SBA uses to separate the components of billing file names when they are transferred to the downstream destination. If the downstream destination is a UNIX system, the recommended field separator is a period (.); this results in a file name such as 020001.030002.00001.01.2. If the downstream destination is a system that does not allow more than one period (.) in the filename, the recommended field separator is an underscore (_); this results in a file name such as 020001_030002_00001_01_2.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
216
Question
Explanation Type: character Range: any printable character Default: . (period) (case sensitive)
Answer
43
file_extension The filename extension is a short character string that SBA appends to the billing file names when it transfers them to the downstream destination. The extension can be set as a part of the schedule add command by providing a value to the following prompt: File extension can have between 0 and 3 characters. Enter Blank for no file extension. Press Enter to accept Blank. Enter File_Extension: Nortel does not recommend the use of the .gz file extension when you configure the outbound file transfer schedule. Use of the .gz extension can make it difficult for you to distinguish between compressed and uncompressed files on the downstream system.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Outbound file transfer schedule Table 22 Outbound file transfer protocol # 44 Question What is the maximum number of times SBA attempts to complete a failed session with the downstream destination for this stream? After a session for this stream fails, what is the maximum time in seconds that SBA must wait before attempting re-connection to the downstream destination? Explanation protocol_max_retries Type: number Range: 0 to 10 Default: 3 Answer
217
45
Answer
218
Table 23 Outbound file transfer schedule (contd.) # 47 Question When should SBA start initiating file transfers to the downstream destination each day? Explanation schedule_start_time Determines the time of day when SBA starts file transfers to the downstream destination. See the examples following this table for more information. Type: Time of Day Units: hh:mm Range: 00:00 to 23:59 Default: none 48 When should SBA stop initiating file transfers to the downstream destination each day? schedule_stop_time Determines the time of day when SBA ends file transfers to the downstream destination. See the examples following this table for more information. Type: Time of Day Units: hh:mm Range: 00:00 to 23:59 Default: none 49 Within the daily time window defined in questions 46 and 47, how often should the SBA transfer files to the downstream destination? schedule_interval This specifies the interval, in minutes, at which SBA is to initiate billing file transfers to the downstream destination. This interval is only active during the window of time specified by the start time (question 46) and stop time (question 47). See the examples following this table for more information. Type: Number Units: Minutes Range: 5 to 1440 Default: 120 Answer
The following are some examples that show different answers to questions for the start time (question 46), stop time (question 47), and the interval (question 48) and the resulting SBA file transfer times. If your start time and stop time are identical, then SBA is setup for continuous outbound file transfer.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
219
Interval 240
SBA Actions The SBA transfers files every four hours, at the beginning of the hour, starting at midnight. The SBA transfers files every thirty minutes at 10 minutes and 40 minutes after the hour, between 10:10 pm and 2 am. The SBA transfers files every five hours at 15 minutes after the hour, starting at 3:15 am.
Resulting Transfers The SBA initiates file transfers at 12:00 midnight, 4:00 am, 8:00 am, 12:00 noon, 4:00 pm and 8:00 pm The SBA initiates file transfers at 10:10 pm, 10:40 pm, 11:10 pm, 11:40 pm, 12:10 am, 12:40 am, 1:10 am and 1:40 am SBA initiates file transfers at 3:15 am, 8:15 am, 1:15 pm, 6:15 pm and 11:15 pm.
22:10
2:00
30
3:15
3:15
300
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
220
Procedure steps
Step 1 Action At the MAP Access the billing level, type mapci;mtc;appl;sdmbil 2 To add volumes, type addvol <stream_name> <volume1> ... <volume5> Repeat this step to add as required. Example Command to add five volumes addvol AMA S00DAMA1 S01DAMA2 S00DAMA3 S01DAMA4 S00DAMA5 3 To remove volumes, type remvol <stream_name> <volume1> ... <volume5> Repeat this step as required. Example Command to remove five volumes remvol AMA S00DAMA1 S01DAMA2 S00DAMA3 S01DAMA4 S00DAMA5
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
221
Variable denitions
Variable <stream_name> Value The name of the billing stream to which to add the volume. The volume name. You can add, or remove, a maximum of five volumes at one time. Separate each entry with a space.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
222
Prerequisites
You must have the root user ID and password to log on to the server. You may log on to the server as root or emsadmin.
Procedure steps
Instructions for entering commands in the following procedure do not show the prompting symbol, such as #, >, or $, displayed by the system through a GUI or on a command line.
At the workstation UNIX prompt or VT-100 terminal prompt:
Step 1 2
Action Log onto the CBM. Get the current hostname by entering:
GetCurrentHostName
Example response: <CBM hostname>-<unit0 / unit1> 3 Access the Report Registration Menu: If the hostname returned in step 2 contains "unit0" then the MATE hostname is "unit1" If the hostname returned in step 2 contains "unit1" then the MATE hostname is "unit0" 4 Determine if the MATE is running by entering:
ping <mate hostname>
The <mate hostname> is the one determined in step 3. 5 6 If the MATE is not Active, you must access the MATE through a local VT100 terminal. Go to step 7. Access the MATE using SSH by typing: ssh root@ <mate hostname> You can log on to the MATE without a password. To exit the MATE, type > exit to return to the local system.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
223
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
224
Procedure steps
At the OAM&P workstation
Step 1 2
Action Label two formatted floppy disks as Disaster Recovery Disk 1 and Disaster Recovery Disk 2. Insert Disaster Recovery Disk 1 into the A: drive in your OAM&P workstation and follow the directions.
ATTENTION
The disaster recovery floppy disk creation process does not format the disks for you, but it does erase all of the data on the disks. Ensure that you have copied important data from the disks before proceeding.
ATTENTION
For more information about creating disaster recovery disks, see the tape drive documentation provided with the OAM&P workstation.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
225
Prerequisites
If the unit was a replacement unit recently installed, ensure that all power cabling connections have been properly installed and secured at the rear of the chassis and SAM-F frame.
Procedure steps
At the front panel of the Policy Controller unit
Step 1
Action If necessary, power on the Policy Controller using the main power switch located on the front panel.
If desired, at the Policy Controller console, monitor the boot progress of the unit.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
226
ATTENTION
This procedure does not describe how to view customer logs or alarms. For detailed instructions about viewing customer logs or alarms, see Nortel Session Server Trunks Fault Management ( (NN10332-911)) .
Procedure steps
At the CS 2000 NCGL Platform Manager or IEMS client
Step 1
Action At the Communication Server 2000 Session Server Launch point, click the Carrier VoIP Communication Server 2000 NCGL Platform Manager link.
The Platform Main Page menu appears.
ATTENTION
The platform (System) Information page does not update automatically. Click the System Information link to update it.
Review the System Information page. 3 4 5 Review the Node Maintenance page. The Node Maintenance page refreshes every 45 seconds. Review the System Status page. The Chassis Information section. Review the Network Connectivity page. The Network Connectivity page refreshes every 45 seconds.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps Figure 14 Crossover and LAN Ethernet cable connections for SST units
227
Review the Disk Services page. The Disk Services page does not update automatically. Click the Disk Services link to update it.
Review the Services page. The Services page does not update automatically. Click the Services link to update it.
Review the Administration page. The Administration page does not update automatically. Click the link to update it.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
228
Procedure steps
At the CS 2000 Session Server Manager or IEMS client
Step 1
Action At the Communication Server 2000 Session Server Launch point, click the Carrier VoIP Communication Server 2000 Session Server Manager. Click to open the Session Server, Maintenance, Application folder. Click the SIP Gateway link. Monitor the status of the SIP Gateway application.
2 3 4
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
229
The view refreshes according to the value shown in the box at the bottom of the status section. 5 6 To increase or decrease the refresh rate, select a different value and click Refresh Rate. Optionally, you can perform the following service affecting actions:
Lock the SIP Gateway application Unlock the SIP Gateway application Suspend the SIP Gateway application Unsuspend the SIP Gateway application Cold SwAct the SIP Gateway application
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
230
To view the number of active calls currently handled by the application, and the synchronization status of the units, click QueryInfo.
--End--
Job aid
The following table provides a description of the Node Status fields.
Table 24 Node status field descriptions Field Unit Connection Status Bar Unit Number Activity State Description Indicates which unit in the node the CS 2000 Session Server Manager is connected to. Identifies the two units in the node, labeled 0 and 1. Indicates which unit is active and which is inactive (standby). Acts as an indirect indicator of fault-tolerant status; when both units have an Operational status of enabled, the node is fault-tolerant. Indicates the service status of each unit, enabled or disabled.
Operational State
The following table provides a description of the SIP Gateway application Status fields.
Table 25 SIP Gateway application Status field descriptions Field Administrative State Operational State Procedural Status Control Status Indication Locked, Unlocked, or ShuttingDown Enabled or Disabled Terminating or Suspended or -
Use the following table to interpret the SIP Gateway areas CCITT X.731-style and related DMS-style status indicators.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid Table 26 SIP Gateway Maintenance field descriptions and interpretation of service states Administrative State Locked Locked Locked Operational State Disabled Enabled Enabled Procedural Status in-service in-service Terminating Control Status Suspended in-service in-service
231
DMS styleServic e States Offline (OFFL) Manual Busy(MA NB) Manual Busy Transitioning(M ANBP) In Service(INSV) System Busy(SY SB) Going out of service(INSVD)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
232
Procedure steps
Step 1 Action At the Communication Server 2000 Session Server Manager or IEMS client, select Communication Server 2000 Session Server Manager from the launch point menu. Select Session Server > Maintenance > Application > SIP Gateway from the left side menu: At the bottom of the SIP Gateway Maintenance panel, locate and click the QueryInfo button. The synchronization status of the units is displayed at the bottom of the query results panel. If the units are not in sync, check for alarm conditions.
2 3 4
If applicable, return to the higher level task flow or procedure that directed you to this procedure.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
233
ATTENTION
Booting the switch causes the switch to drop all calls.
ATTENTION
An image loads from a small computer systems interface (SCSI) device. The SCSI device can be in a disk or a digital audio tape (DAT).
CAUTION
Call your next level of support
Contact your next level of support before performing this procedure.
CAUTION
Extended service interruption
A longer recovery time occurs for a switch boot from tape than a switch boot from disk. To minimize recovery time, boot from disk.
CAUTION
Extended service interruption
Log on procedures can vary depending on your office configuration. If you need additional help, contact your next level of support.
Procedure steps
Step 1 Action See the office records and determine the name of the XA-Core recording device that contains the last office image file.
ATTENTION
The XA-Core recording device is a disk drive or a tape drive for a digital audio tape (DAT). Record the name of the XA-Core device.
At the XA-Core shelf make sure that the disk drive or the tape drive that you recorded in step 1 is in-service.
ATTENTION
The device is in-service when the green light-emitting diode (LED) is on (illuminated).
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
234
If the device is in service, at the XA-Core reset terminal override the XA-Core reset terminal by typing:
>\OVERRIDE
Boot the XA-Core by typing: >\BOOT <nn> <s> <p> Example Response:
>\BOOT 4 F L
Type Y to confirm the command. 6 Monitor the reset terminal display to determine if the switch has booted.
ATTENTION
The reset terminal displays a response to indicate a boot in progress. The response also displays diagnostic messages and alphanumeric addresses. When the switch has completely booted, a prompt appears on the display.
7 8 9
If the response is a prompt then go to step 9. At the MAP terminal If the response has no prompt after approximately 15 minutes then contact your next level of support. Press the <BREAK> key to determine if you have to log in.
ATTENTION
The log on message indicates that you have to manually log in. An automatic log in can occur if the office parameters have automatic log in.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
235
10 11 12
If log in is not automatic then go to step 11. if the log in is automatic then go to step 15. Login to the MAP terminal by typing:
>LOGIN
13
14
15 16 17
If the MAP response is not a successful completion, contact your next level of support.
--End--
Variable denitions
Variable <nn> <p> <password> <s> Value The slot number parameter value to indicate the number of the physical shelf slot - 0 to 18. The position parameter value that indicates the IOP bay - either upper (u) or lower (l). The name of the password for the account. The side parameter value to indicate the circuit pack or packlet location in the physical shelf front (f) or rear (r). The name of the user for the account.
<user_name>
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
236
ATTENTION
You must be directly connected to the switch to initiate a Boot Monitor session. You can only connect using a Telnet connection if the Boot Monitor CLI is already active.
Prerequisites
You must have access to the Boot Monitor CLI through a direct connection to the switch or a Telnet connection. For more information about accessing the Boot Monitor CLI, see "Managing the Ethernet Routing Switch 8000 Series Switch" using the Command Line Interface Release 3.2, 313194-*.
Procedure steps
Step 1 Action At the Boot Monitor CLI issue the boot command by typing
monitor# boot [<file>] [config <value>]
--End--
Variable denitions
Variable <file> Value software image device and file name in the format [a.b.c.d:]<file> | /pcmcia/<file> | /flash/<file>. The file name, including the directory structure, can be up to 1024 characters. software configuration device and file name in the format [a.b.c.d:]<file> | /pcmcia/<file> | /flash/<file>. The file name, including the directory structure, can be up to 1024 characters.
config <value>
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure step
237
CAUTION
Possible service interruption
Do not use this procedure when in an emergency situation with no stable call processing application image. If in a situation without a restartable image, contact Nortel Global Network Product Support (GNPS) immediate. Attempting to use this method without a valid call processing application image could fail due to constant resets on the Call Agent.
Procedure step
Step 1 2 3 Action Insert the DAT cassette with the image to restore. Restore the image from tape. This step requires root privilege. For example, see Job aid (page 238). At the Call Agent Manager log on to the inactive Call Agent and change directory to the location in which to restore the image. Verify that enough disk space exists to hold the image. For example, see Job aid (page 238). 4 Open a file transfer protocol (FTP) session to the Communication Server 2000 Core Manager, and transfer the image. It may be necessary to become the super user to transfer the file. For example, see Job aid (page 238). 5 At the MAP enter the DISKUT level and use the IMPORT command to make the image available to the call processing application. For example, see Job aid (page 238). If additional space is needed to import the image, the IMPORT command offers to expand the volume. 6 7 Set the image in the Image Table of Contents (ITOC). For example, see Job aid (page 238). The restored image is now available for booting.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
238
Job aid
Figure 15 DAT cassette
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
239
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
240
CAUTION
Possible service disruption
Performing a restore of the SIP Gateway application database to the active unit is a service affecting activity and can cause data mismatches at the Communication server 2000.
Prerequisites
You must have secure file transfer access, such as scp, to the unit from a remote host if the backup file is stored on a remote host. Backups are also stored locally on each unit in directory /data/bkresmgr/backup.
CAUTION
Both units must be free of any Session Server Manager patches before the database is restored. This is accomplished by reinstalling the current version of the SIP Gateway application software (performed in step 1 below). It is not necessary to remove NCGL patches.
Procedure steps
Step 1 Action For detailed steps for this procedure, see Nortel Carrier Voice over IP Network Upgrades and Patches (NN10440-450).
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
241
ATTENTION
Service affecting
Perform this procedure on the active unit only. Restoring the SIP Gateway application database to the active unit affects service and can cause data mismatches at the CS 2000 call server.
Prerequisites
Back up the Session Server Trunk before you perform this procedure.
Procedure steps
Step 1 Action At the Session Server CLI or IEMS client Log on to the active Session Server to which you will restore the database. 2 3 4 Change to the root user. Change directories, type $ cd /opt/apps/database/solid_install Run the database restore script, type $ ./restorebackup.sh
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
242
ATTENTION
For more information about re-installation of a UAS node, contact your Nortel service representative.
Procedure steps
Step 1 Action After the re-installation of the UAS node has been completed, determine whether you want to enable provisioning of the node occur during the next audio distribution cycle or immediately. If you want to enable provisioning of the node to occur during the next audio distribution cycle, go to step 4. If you want audio provisioning of the node to occur immediately, go to step 5. Perform "Enabling provisioning of a UAS node" in the document (NN10095-511), entitled "UAS Configuration Management," in your UAS document suite.
2 3 4
ATTENTION
Provisioning of the node will begin during the next audio distribution cycle. The distribution cycle occurs once for each hour.
Perform "Provisioning a UAS node" in the document (NN10095-511), entitled "UAS Configuration Management," in your UAS document suite.
ATTENTION
Provisioning of the node will begin immediately although as much as a five-minute delay may occur before actual provisioning activity begins.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
243
ATTENTION
For more information about re-installation of a Media Server 2000 Series node, contact your Nortel service representative.
Procedure steps
Step 1 Action After the re-installation of the Media Server 2000 Series node has been completed, determine whether you want to enable provisioning of the node occur during the next audio distribution cycle or immediately. If you want to enable provisioning of the node to occur during the next audio distribution cycle then go to step 4. If you want audio provisioning of the node to occur immediately then go to step 5. Perform the "Enabling provisioning of a Media Server 2000 Series node" (see the Media Server 2000 Series Configuration Management document).
2 3 4
ATTENTION
Provisioning of the node will begin during the next audio distribution cycle. The distribution cycle occurs once for each hour.
Perform the procedure "Provisioning a Media Server 2000 Series node" (see the Media Server 2000 Series Configuration Management document).
ATTENTION
Provisioning of the node will begin immediately although as much as a five-minute delay may occur before actual provisioning activity begins.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
244
Procedure steps
Step 1 2 3 4 5 6 7 Action Open an xterm window and log in using the "maint" logon and password. Become the "root" user by entering
su - root
If you want to restore Oracle database files from tape then go to step 5. If you want to restore Oracle database files from disk then go to step 10. Insert the appropriate "ORACLE" backup tape into the DDS-3 tape drive. Rewind the backup tape by performing the following command
mt -f /dev/rmt/0c rewind
Start the restoration of the APS Oracle database from the tape by entering the following command
ips_export_db.sh -t /dev/rmt/0c -restore
ATTENTION
This command is entered on a single line.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
245
Messages logging the progress of the restoration display on the screen. When you are prompted about continuing the restoration even if the current database will be destroyed, enter "y" (yes). 8 After the restoration of the Oracle database is complete, rewind the backup tape by performing the following command
mt -f /dev/rmt/0c rewind
9 10
Eject the backup tape and store it for possible use later. Go to step 11. Start the restoration of the APS Oracle database from the disk by entering the following command
ips_export_db.sh -diskonly -restore
ATTENTION
This command is entered on a single line.
Messages logging the progress of the restoration display on the screen. When you are prompted about continuing the restoration even if the current database will be destroyed, enter "y" (yes). 11 12 13 14 Insert the appropriate "application file" backup tape into the DDS-3 tape drive. Change directory to the root directory by typing
cd /
Restore the application files (/PROV_data, /audio_files, and /user_audio_files) by entering the following command
tar xvf /dev/rmt/0c
15
When the restoration of the application files completes, remove the tape from the tape drive and store for possible use later. Insert another write-enabled DAT tape into the drive to be used for the automatic Oracle system back up that runs daily at 1:00 a.m. Verify that the application files have been restored by entering the following commands
cd /PROV_data ls -l
16
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
246
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
247
A full restore restores all backed up service data to the selected device or devices. An incremental restore restores service data based on a specified date. Like the full restore, you can perform an incremental restore on either one or multiple devices. A selective restore restores specific service data that you select. Like the full restore, you can perform a selective restore on either one or multiple devices.
ATTENTION
It is not recommended that an active file (current view) be restored on a Multiservice Switch 7400/15000 or Media Gateway 7400/15000. When you restore the current view, you may overwrite the existing current view with different content. This action will cause an outage of the Backup and Restore tool.
The following information applies to using the Service Data Backup and Restore tool to perform a restore for Multiservice Switch 7400/15000 or Media Gateway 7400/15000 nodes:
Adding Multiservice Switch 7400/15000 or Media Gateway 7400/15000 devices to the restore list (page 248) Removing Multiservice Switch 7400/15000 or Media Gateway 7400/15000 devices from the restore list (page 248) Defining a specific userid and password for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 in the backup list (page 174) Performing a full restore for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 device (page 249) Performing an incremental restore for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 device (page 250)Performing a selective restore for a Multiservice Switch
Activating the restored Multiservice Switch or Media Gateway data (page 252)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
248
Adding Multiservice Switch 7400/15000 or Media Gateway 7400/15000 devices to the restore list
Use this procedure to add devices to the list of devices that you wish to restore. Procedure steps Step 1 Action Open the Backup and Restore window. From the MDM window, select Configuration->Passport Devices->Administration->P assport Service Data Backup/Restore. Select the Restore Configuration tab. Select Add to launch the Add Devices Dialog. From the left pane, select a group of devices or expand the group and use the Ctrl key to select multiple devices within a group. From the drop down list, in the right pane, select a restore mode (Incremental or Full). If a specific userid and password is required for the device, enter the values in the user ID and Password fields and uncheck the Use default checkbox. If you wish to use the default userid and password, click the Use defaultcheckbox. Click OK. This procedure is complete.
--End--
2 3 4
5 6
7 8 9
Removing Multiservice Switch 7400/15000 or Media Gateway 7400/15000 devices from the restore list
Use this procedure to remove devices from the list of devices that you wish to restore. Procedure steps Step 1 Action Open the Backup and Restore window. From the MDM window, select Configuration->Passport Devices->Administration->P assport Service Data Backup/Restore.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Performing a full restore for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 device 249
2 3 4 5 6
Select the Restore Configuration tab. In the Device List, select the devices you wish to remove. Click Remove. In the confirmation dialog, select Yes to confirm or No to cancel the removal. This procedure is complete.
--End--
Dening a specic userid and password for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 in the restore list
A specific userid and password can be defined when you add the device to the restore list or you can set it later using the following procedure. Procedure steps Step 1 Action Open the Backup and Restore window. From the MDM window, select Configuration->Passport Devices->Administration->P assport Service Data Backup/Restore. Select the Restore Configuration tab. Select a device in the Device List section. In the Device Details section, in the Authentication tab, enter a userid and password. Clear the Use default checkbox. This procedure is complete.
--End--
2 3 4 5 6
Performing a full restore for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 device
Use this procedure to perform a complete system restore of a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 device. Procedure steps
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
250
Step 1
Action Open the Backup and Restore window. From the MDM window, select Configuration->Passport Devices->Administration->P assport Service Data Backup/Restore. Select the Restore Configuration tab. Select Add to launch the Add Devices Dialog. From the left pane, select a group of devices or expand the group and use the Ctrl key to select multiple devices within a group. Click in the Mode title and selectFull from the drop-down list. Click Restore. Activate the restored data by executing the procedure Activating the restored Multiservice Switch or Media Gateway data (page 252). This procedure is complete.
--End--
2 3 4
5 6 7
Performing an incremental restore for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 device
Use the following procedure to copy files, based on a specific date, from the repository back to the Multiservice Switch 7400/15000 or Media Gateway 7400/15000. Procedure steps Step 1 Action Open the Backup and Restore window. From the MDM window, select Configuration->Passport Devices->Administration->P assport Service Data Backup/Restore. Select the Restore Configuration tab. Select Add to launch the Add Devices Dialog. From the left pane, select a group of devices or expand the group and use the Ctrl key to select multiple devices within a group. Select a device from the Devices list. In the Mode title, select Incremental from the drop-down list. Define a date in the Date column (For example July 4, 2008)
2 3 4
5 6 7
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Performing a selective restore for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 device 251
Click Restore. All files with a date and time earlier than the specified date are restored.
Activate the restored data by executing the procedure Activating the restored Multiservice Switch or Media Gateway data (page 252). This procedure is complete.
--End--
10
Performing a selective restore for a Multiservice Switch 7400/15000 or Media Gateway 7400/15000 device
Use the following procedure to copy a specific file from the repository back to the Multiservice Switch 7400/15000 or Media Gateway 7400/15000. Procedure steps Step 1 Action Open the Backup and Restore window. From the MDM window, select Configuration->Passport Devices->Administration->P assport Service Data Backup/Restore. Select the Restore Configuration tab. Select Add to launch the Add Devices Dialog. From the left pane, select a group of devices or expand the group and use the Ctrl key to select multiple devices within a group. Select a device from the Devices list. In the Mode title, select Selective from the drop-down list. The Configuration column in the table is enabled. 7 Click in the Configuration cell to display a pull-down list that displays all the available views on the Multiservice Switch 7400/15000 or Media Gateway 7400/15000. Note: This step may take a few seconds to complete because the application must access the Multiservice Switch 7400/15000 or Media Gateway 7400/15000 and list all the views names. 8 Click Restore.
2 3 4
5 6
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
252
Activate the restored data by executing the procedure Activating the restored Multiservice Switch or Media Gateway data (page 252). This procedure is complete.
--End--
10
3 4 5 6
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
253
An emergency situation requires restoration of system configuration from a stored data snapshot. You use the file manager function to ensure that there are copies of the data snapshot to be restored on both RTC system nodes. You make the data snapshot the boot snapshot for both RTC system nodes. You deactivate the linksets and change the path state of the application server process (ASP) paths to Down for your system, as appropriate. You perform a complete office recovery (COR) on your system, which means that you unseat the RTC system nodes, turn off both power switches on the shelf, reseat the RTC system nodes, and then restore power. Your stored data snapshot is now the running data snapshot.
The following procedures must be executed in the order given to restore data snapshots stored on your alternate boot server or on the RTC system nodes.
Restoring a system configuration from a data snapshot (page 254) Procedure 2 Performing a restore operation from a backup CD-RW (page 257)
CAUTION
Risk of service loss
Performing a COR on your system is service affecting. Nortel recommends that you do this during off-peak hours and that you ensure that a mated system is available.
CAUTION
Risk of service loss
Do not power down the shelf when the SCSI disk light on the RTC system node is on. To do so could cause a disk failure.
CAUTION
Risk of equipment damage
Wear wrist straps, and use standard antistatic precautions.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
254
Procedure steps
At the OAMP workstation
Step 1 2 3 4
Action Click Administration>file-manager. If the RTC system node is not locked, click Lock. If the RTC system node is not offline, click Offline. Make note of which RTC system node is the active RTC system node. This information is required later in the procedure. Ensure that the LED is off for the SCSI Disk Drive associated with this RTC system node before unseating the mission card. Unseating the RTC card stops the write and read access to the SCSI card to avoid SCSI failure.
Unseat the mission card on the inactive RTC by pressing outward on the top and bottom latches of the mission card to release it from the Communications Application Module (CAM) shelf. Grasp the top and bottom latches of the mission card and gently pull it to disconnect it from the associated TM. Do not remove the mission card from the CAM shelf. Repeat steps 2 and 3 for the RTC mission card for the active RTC system node. Turn off both the A and B power switches on the rear of the control CAM shelf of your system. Reseat the RTC mission card for an RTC system by. gently sliding the mission card back into place. Apply pressure to the faceplate until you feel resistance. Snap the top and bottom latches of the mission card inward, toward one another. Two audible clicks can be heard when the mission card is seated properly. Repeat steps 5 through 7 for the RTC mission card for the other RTC system node.
8 9 10
11
12
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
255
13
Return power to the shelf. To do this, turn on both the A and B power switches on the rear of the control CAM shelf of your system. System start up can take several minutes, depending on the configuration of your system.
14 15 16
Click Configuration>platform>node. Click the Graphical View tab. Confirm that all system nodes are enabled (green status light). Note: Nortel recommends that you do not attempt any provisioning changes until all system nodes recover.If the GUI fails to log in when it is reestablishing the connection, the boot snapshot may not contain the current users user-account. Provide another valid user-account that is configured in the snapshot.
17 18
Log into your system. Open the RTC system node administration tab for the RTC system node in slot 12. To do this, click Configuration>platform>node. Click the Graphical view tab, and select slot 12. If the table is opened before the GUI reconnects to the RTC, the old data is displayed. After the GUI reconnects to the RTC, reload the table.
19
20
If the data snapshot descriptions match, proceed to step 17. If the snapshot descriptions do not match, return to step 10 and attempt the procedure again. If the procedure has failed twice in row, contact your next level of support. For the current active snapshot the snapshot-date and snapshot-version fields are empty.
21
Update the system time settings in the Set Date/Time window, click Administration> Set Date/Time. Note: This step is not required if simple network time protocol (SNTP) is enabled for this system.
22
Use the following procedure to perform a restore backup from a backup tape
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
256
ATTENTION
This procedure can overwrite existing data files on your workstation. Make sure you have a current data backup before you begin this procedure. At the Windows-based OAMP workstation
Step 1 2
Action Insert the tape containing the data that you want to restore. Double-click the Backup Exec icon on the desktop if you are running a Veritas program or the Colorado Backup II icon if you are running a Colorado program. Select Restore files using the Restore Wizard. Click Ok. Click Next. Select from media in the device. Click Next. The system loads the information from the backup tape. Select the backup date and time for the data that you want to restore. Click Ok. The system loads the information from the backup tape. Click the + button on the tree control to expand the tree box next to the C: drive. Select any directories that you want to restore. Click Next. Click Next a second time. Select Always replace the file on my computer. Click Start. The system restores the backup data. Click Ok. The data restore procedure is complete.
--End--
3 4 5 6 7 8 9 10 11 12 13 14 15 16
ATTENTION
This procedure can overwrite existing USP data files on your workstation. Make sure you have a current data backup before you begin this procedure.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
257
Before performing this procedure, ensure that the ABS is installed in the workstation. The root privilege is required to perform this procedure.
Procedure 2 Performing a restore operation from a backup CD-RW At the Solaris-based OAMP workstation
Step 1
Action Remove the current USP data by entering the following command at the prompt:
rm -Rf <USP data directory>
Example rm -Rf /opt/usp/ntssgusp/abs/sites 2 3 Insert the CD-RW containing the USP data that you want to restore into the CD-ROM drive. Go to the partition where the ABS is installed by entering the following:
cd <partition>
Example cd / The ABS is installed in the root partition by default. 4 Restore the USP data by entering the following command:
ufsrestore xf /<cdrom directory>/<dumped filename>
where <cdrom directory> is the directory where the CD-ROM is mounted. Example ufsrestore xf /cdrom/cdrom0/usp_data_20051115.dmp The ABS is installed in the root partition by default. The following is displayed; enter what is in bold text when prompted: You have not read any volumes yet. Unless you know which volume your file(s) are on you should start with the last volume and work towards the first. 5 Enter "1" in response to the promp as shown:
Specify next volume #:
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
258
If the data files are large, it can take several minutes to restore the data. 6 Enter "n" in response to the promp as shown:
set owner mode for .?[yn] n
8 9
where <cdrom directory> is the directory where the CD-ROM is mounted. Example ufsrestore xf /cdrom/cdrom0/dhcp_20051115.dmp The following is displayed; enter what is in bold text when prompted: You have not read any volumes yet. Unless you know which volume your file(s) are on you should start with the last volume and work towards the first. 10 Enter "1" in response to the promp as shown:
Specify next volume #: 1
If the data files are large, it can take several minutes to restore the data. 11 Enter "n" in response to the promp as shown:
set owner mode for .?[yn] n
12
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps n
259
13
Example cp /cdrom/cdrom0/dfstab /etc/dfs/dfstab 14 Restart the NFS server by entering the following commands:
/etc/init.d/nfs.server stop /etc/init.d/nfs.server start
15 16
Remove the CD-RW from the CD-ROM drive and return it to its regular storage location. This procedure is complete.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
260
Prerequisites
The user must have root user privileges to perform this procedure.
Procedure steps
Step 1 2 3 4 5 Action At your workstation, open a Telnet session to the server, type > telnet <server> When prompted, enter your user ID and password. Change to the root user, type $ su When prompted, enter the root password. Make sure that the /opt/nortel/cem/data/coreEMS/configBackup directory contains the following files, type # ls /opt/nortel/d ata/coreEMS/configBackup
Example response CEM server STOPPED 7 If the CEM server is running, stop the CEM server, type
# servstop CEM
Example response
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
261
Variable denitions
Variable <server> Value The IP address or host name of the server where IEMS resides.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
262
ATTENTION
SSL uses certificates to ensure security. Certificates from one server cannot be used on another server. If you want to take a back up file from an SSL server and restore it to a different server, see , after you complete the restore. Run a script to set up IS authentication, session, and policy traffic to operate under SSL. The servers in the same high availability cluster can use the same SSL certificate. If the restored IS SSL certificate is expired, see after you complete the restore.
Prerequisites
You must have a backup tar file. You must have root user privileges to perform this procedure.
Procedure steps
Step 1 Action At your workstation, open A Telnet session to the server, type telnet <server> If you are upgrading the active server, Telnet to the active server. If you are upgrading the inactive server, Telnet to the inactive server. 2 3 4 5 6 7 8 When prompted, enter your user ID and password. Change to the root user, type su - root When prompted, enter the root password. Stop the security services, type # servstop RADSVR and press Enter. Type # servstop IS and press Enter. Type # servstop WEBSERVICES and press Enter. List the available security server backup files and determine the backup file to be restored by typing
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
263
# ls /data/bkresmgr and press Enter. Note the name of the security server backup file you intend to restore. You will need this information in a later step. 9 List the available security server backup files and determine the backup file to be restored by typing # ls /data/bkresmgr and press Enter. The format of the security server backup files are<data><time> backupSS2.0<host_name>.tar The name of the security server backup file you intend to restore. 10 Change directories, type
# cd /opt/nortel/applications/security/current_s1is ext/swmgmt/bin
11
Create a backup of the nsswitch.conf file, type # cp /etc/nsswitch.conf /etc/nsswitch.conf.<DATE> Record the name of the nsswitch.conf.<DATE> file. This file is required after the restore operation.
12 13
Disable SAML, type # cp /etc/nsswitch.conf.files /etc/nsswitch.conf Perform the restore operation, type
# ./brr_security.sh -restore /data/bkresmgr/<date><tim e>backupSS2.0<host_name>.tar
14 15 16 17 18
Restart the security services, type # servstart and press Enter. Type WEBSERVICES and press Enter. Type # servstart IS and press Enter. Type # servstart RADSVR Enable SAML by typing # cp /etc/nsswitch.conf.<DATE> /etc/nsswitch.conf and press Enter.
19
If you restored the back up image from a different server with a different certificate, or if the restored certificate is expired, see to replace the invalid or expired certificate on the IEMS server. Clone the active server to the inactive server. For details, see Clone the active server to the inactive server. For details, see .
20
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
264
21
Clone the active server to the inactive server. For details, see .
--End--
Variable denitions
Variable <DATE> <date><time> <host_name>.tar <server> Value The backup file created or should be restored. The backup file to be restored. The name of the security server backup file you intend to restore. The IP address or host name of the server where IEMS resides.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
265
Prerequisites
You must be a trained AIX system administrator who has root user privileges to the core manager. You must mirror all volume groups on the core manager before you perform this procedure. If you perform this procedure when disk mirroring is not at the Mirrored state, the system displays an error message. If your system includes the SuperNode Billing Application (SBA), you must use tape drive DAT0 to perform this procedure.
Procedure steps
Step 1 2 3 Action At the local VT100 console Log into the SDM CS 2000 Core Manager as a root user. Access the maintenance interface, type # sdmmtc Determine if the core manager maintenance interface is available. If core manager maintenance interface is available, go to the next step. If core manager maintenance interface is not available, go to step 10. 4 5 Access the administration level, type admin. Perform a full restore of the core manager, type restore Sample responseSelect the tape drive you want to restore from, or type Abort to abort: Enter 0 for the tape drive in the main chassis slot 2. Enter 1 for the tape drive in the main chassis slot 3. 6 Type the number for the tape drive to use.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
266
To use the tape drive in slot 2, type 0 To use the tape drive in slot 13, type 1 Sample responseInsert the backup-tape into the tape drive in the main chassis slot 2. When completed press [Enter] to start the restore. 7 When prompted, type y Sample response Insert the backup-tape into the tape drive in the main chassis slot 2. When completed press [Enter] to start the restore. 8 Insert the backup S-tape into the specified tape drive. Wait until the tape drive stabilizes (yellow LED is off) before you proceed. 9 10 Press Enter to start the restore process. Go to step 16. At the core manager The core manager maintenance interface is not available (step 2), make sure that one of the core manager tape drives (slot 2 or 13 in the main chassis) contains the system image backup tape (S-tape). Use tape drive DAT0 (option to perform a full restore from an S-tape) if your system also includes SBA. At the Modular Supervisory Panel 11 Reboot the core manager. If the prompt is available at a local VT100 console, reboot the core manager, type shutdown -Fr. If the prompt is not available, reboot the core manager by turning the power off, then on. Use the MSP breaker that supplies power to the core manager to turn the power off and on. At the local VT100 console 12 When the system COLD Start message appears, interrupt the boot process. Interrupt the process by press Esc twice. The system takes about 4 minutes to initialize. Proceed according to he system prompt. 13 If the prompt is FX-Bug and you are in a menu, type the number to select Go to System Debugger. Go to step 15. 14 If the prompt is FX-Diag, Switch the directory to FX-Bug, type > sd
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
267
15
View the input/output devices on the core manager to verify the address of the tape drive from the FX-Bug prompt, type Fx-Bug> ioi If the FX-Bug prompt appears (response from step 12), type Fx-Bug> pboot <address_for_Archive_Python> Valid choices are: pboot 1 50 if the tape drive is located in slot 2 pboot 6 50 if the tape drive is located in slot 13 Wait for the system to reboot. This takes approximately 4 minutes.
16
17
After the system reboots, you are prompted to define the console setting and the language setting. Define the console setting, type 1. The system does not echo your entry. In case of any failures, contact your next level of support.
18
To select the language setting, type 1. The Welcome to Base Operating System Installation and Maintenance menu appears.
19
Type the number to select the Change/Show Installation Settings and Install menu option. The System Backup Installation and Settings menu appears. Sample response System Backup Installation and Settings Either type 0 and press Enter to install with the current settings, or type the number of the setting you want to change and press Enter. Setting: Current Choice(s): 1 Disk(s) where you want to install hdisk0... Use Maps No 2 Shrink File System No >>> 0 Install with the settings listed above. The elipses (...) under Current Choice(s) indicates that more than one disk is currently in use.
20
The default disk hdisk0, which is located in slot 2 of the main chassis. If your core manager contains one disk drive in each domain of the main chassis, accept the default setting. If you have additional disk drives, you may wish to change the settings. If you want to accept the default setting, go to step 27. If you want to change the setting, type
21
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
268
>1 The Change Disk(s) Where You Want to Install menu appears, with a list of all the available disks on which you can install the system backup image. The >>> symbols identify the currently selected disks. 22 You must install the system backup on one disk in each domain to achieve fault-tolerant operation. Valid choices in response to the prompt in the Change Disk(s) Where You Want to Install menu are, are hdisk0 and hdisk5. The rootvg disks for installation must have location codes c1-f2-00-0 for domain 0 c1-f13-00-0 for domain 1 23 Type the number to select one or more disks, and press Enter. To undo a disk you selected, type the number again and press Enter. After you select the disk, the System Backup Installation and Settings menu appears. 24 25 At the prompt. type 0 and return to step 20. To accept the default setting (step 21), type 0 The restore process begins, and takes at least 30 minutes to complete. The progress monitor shows the approximate percentage of the tasks completed, and the elapsed time. If an error message appears at the end of the restore process, datavg did not import successfully. Contact the next level of support. If you are performing this procedure as part of removing an I/O expansion chassis (Upgrades, (NTRX50EC)) you must manually reboot the system. At the FX-bug> prompt, type pboot 1 0. At the core manager 26 Remove the S-tape from the tape drive and store it in a secure location. At the local or remote terminal 27 28 29 30 Log in to the core manager as a root user, press Enter when you see the TERM=(vt100) prompt. Start the core manager maintenance interface, type # sdmmtc Access the storage level, type > storage Determine the mirror status of the disks.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
269
ATTENTION
If the status of the disks is Integrating or Not Mirrored, stop this procedure. You cannot continue until the status is Mirrored. If necessary, contact your next level of support.
31 32
Access the LAN level, type > lan Check the state of DCE. Sample response DCE State: SysB
33
Access the application (APPL) level to check the state of any DCE-based applications, type > appl Sample response # Application State 1 Table Access Service . 2 Log Delivery Service . 3 OM Access Service . 4 Secure File Transfer Fail 5 Enhanced Terminal Access Fail 6 Exception Reporting Fail Applications showing 1 to 6 of 6
ATTENTION
DCE and DCE-based applications can fail if the key tab files restored from tape contain obsolete keys.
34
If the DCE state is SysB at the LAN menu level of the RMI (response from step 32), and the logs indicate an error with the security client service in DCE, restore the service. If the state of some DCE-based applications Fail (response from step 32), try to restore them by busying (BSY) and returning to service (RTS) the applications from the SDM APPL level. If that , restore them by performing the procedure to add the application server to the DCE cell in the Configuration Management document. You must create a new system image backup tape.
--End--
35
36
37
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
270
Prerequisites
CAUTION
Possible loss of data
Use this procedure at the discretion of the system administrator.
Perform a partial restore only if you are familiar with the files, and know exactly which files are to be restored. If you restore the wrong files, you may inadvertently corrupt core manager software.
ATTENTION
This procedure must be performed by a trained AIX system administrator authorized to perform fault-admin actions.
ATTENTION
All volume groups on the core manager must be fully mirrored (Mirrored) before performing this procedure. If you attempt to perform this procedure when disk mirroring is not Mirrored, an error message is displayed on the screen.
ATTENTION
If your system includes the SuperNode Billing Application (SBA), use tape drive DAT0 to perform this procedure.
ATTENTION
Instructions for entering commands in the following procedure do not show the prompting symbol, such as #, >, or $, displayed by the system through a GUI or on a command line.
Procedure steps
Step 1 Action At the core manager insert the S-tape into the tape drive you selected.
ATTENTION
Wait until the tape drive stabilizes (yellow LED is off) before you proceed.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
271
At the local or remote VT100 console when you are ready to continue this procedure, enter:
y
The contents of the S-tape are listed on the screen. When the listing is complete, the system prompts you to return to the System Image Backup and Restore Menu. Example response:
Would you like to return to the previous menu? (y|n)
3 4 5 6
If you are listing the contents of the tape to verify that the file has been restored then go to step 14. If you are listing the contents of the tape to verify the file name or directory that you wish to restore go to step 6. From the System Image Backup and Restore Menu, select "Restore Files from the System Image Tape (S-tape)":
4
After you select option 4, you are prompted to select the tape drive. Example response:
Select a tape drive you wish to use: Enter 0 to return to previous menu Enter 1 for tape drive DAT0 in Main Chassis-Slot 2 Enter 2 for tape drive DAT1 in Main Chassis-Slot 13 ( 0, 1 or 2 ) ==>
ATTENTION
Use tape drive DAT0 (option 1) if your system also includes SBA.
Enter the number for the tape drive you want to use (1 or 2).
ATTENTION
If your system includes SBA, and you still wish to use tape drive DAT1 (option 2), the following message is displayed:
Example response:
You have selected DAT 1. This is the default DAT drive for the Billing application, and may currently be in use for the emergency storage of billing records.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
272
Preparation for disaster recovery If you continue to use DAT 1, make sure that the correct tape is in the drive, and that billing records will not be lost during the backup/restore operation. Do you wish to continue with DAT 1? ( y | n )
if you wish to continue using DAT1, enter y I If you do not wish to use DAT1, enter n
The system prompts you to return to the System Image Backup and Restore Menu if you do not wish to use DAT1. 9 After you select the tape drive, you are prompted to insert the S-tape into the appropriate tape drive. A warning is displayed advising that this procedure must only be completed by qualified core manager system administrators. The warning also advises that files and directories must be entered exactly as they appear in the file listing. Insert the S-tape in the appropriate tape drive.
ATTENTION
Wait until the tape drive stabilizes (yellow LED is off) before you proceed.
Example response:
Are you ready to enter the name of the file or directory? (y|n)
10
Example response:
Enter the name of the directory or file that you wish to restore as ./<your-full-path>/<your-file-or-directory>. Note: Tape processing may take a few minutes to complete. A log file /tmp/s-tape.restore will be created. ==>
11
Enter the full path name of the directory or file that you wish to restore, exactly as shown in the file listing, including "/" at the beginning.
ATTENTION
A log file /tmp/s-tape.restore is created when the restore is completed.
ATTENTION
An error message is displayed if the restore is unsuccessful. If this occurs, go to step 25.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
273
12
During the restore process, the screen does not display any additional information. When the file restore is complete, the file you have restored is displayed. The system then prompts you to return to the System Image Backup and Restore Menu. Example response:
Would you like to return to the previous menu? ( y | n )
ATTENTION
If the restore has failed, an error message is displayed before the prompt, advising you to list the contents of the tape, and perform the procedure again.
13 14
Determine if the restore was successful. The system displays the file that you have restored. You may also wish to view the s-tape.restore file or list the files on the S-tape. If the restore is successful then go to step 25. If the restore failed then contact your next level of support If you wish to view the s-tape.restore file then go to step 18. Exit the System Image Backup and Restore Menu:
0
15 16 17 18 19 20 21 22 23 24 25
Continue pressing the Enter key until the files that you have restored, and the date of the restore, are displayed. If successful then go to step 26. If failed then go to contact your next level of support. Exit the System Image Backup and Restore Menu:
0
ATTENTION
If you then wish to exit the maintenance interface, type quit all and press the Enter key.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
274
26
At the core manager remove the S-tape and store it in a secure place.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
275
Prerequisites
You must have the names of the swapped out volumes. For more information see, "Configuring SBA backup volumes on the core" in the Accounting document.
Procedure steps
Step 1 2 3 Action Post the billing stream. > mapci;mtc;appl;sdmbil;post <stream_name> Quit back to the appl;sdmbil level.
> quit
Confirm that the names of the billing streams existing backup volumes are the swapped in volumes you created earlier. > conf view <stream_name>
ATTENTION
SBA does not support the configuration of more than one billing stream at a time from multiple workstations. The last billing stream that is configured is the one that is saved.
Read the notes you made when you performed the procedure, "Configuring SBA backup volumes on the core" in the Accounting document, to confirm that the backup volumes that you listed in step 3 are the backup volumes you created with that procedure. If the backup volume names are the backup volume names configured when you performed the procedure then continue with step 7. If the backup volume names are not the backup volume names configured when you performed the procedure then determine if someone else reconfigured the backup volumes before you continue with step 7.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
276
7 8
Configure the billing stream of the logical volumes you created when you performed the procedure, "Configuring SBA backup volumes on the core" in the Accounting document. > addvol <stream_name> <volume1> ... <volume5> Example Response: To add five volumes, the command is: addvol AMA S00DAMA S01DAMA S02DAMA S03DAMA S04DAMA Repeat this step until all of the volumes have been added to the stream.
ATTENTION
The non-empty backup volumes are automatically detected by the SBA audits. In addition, the SBA places the billing stream into recovery mode and the volumes from the original backup volumes are sent to the core manager.
ATTENTION
You must provide, to all operating company personnel who work on the core, the names of the old and new backup volumes and the procedure you used to swap the volumes. They must be made aware that any restarts or switch of activity (SwAct) that occur before the billing stream returns to normal mode can cause a serious loss of billing records. The mode of the billing stream must be closely monitored to ensure that it returns to normal mode without an intervening RESTART or SwAct.
--End--
Variable denitions
Variable <stream_name> <volume5> Value Name of the billing stream. Volume name. Up to five volumes (with each entry separated by a space) can be added at one time.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
277
Audio Provisioning Server (APS) Core and Billing Manager (CBM) Communication server 2000 Management Tools Communication server 2000 SAM21 Manager Integrated Element Management System (IEMS) Media Gateway (MG) 9000 Manager Multiservice Data Manager (MDM) Network Patch Manager
Although the MDM, MG 9000 Manager and CBM do not host a database, this restore procedure still applies to these components as it can be used to recover other critical files.
Prerequisites
You need the DVD on which you backed up the Oracle/MySQL data. If the data was backed up by the backup restore manager, you need the name of the file located in directory /data/bkresmgr/backup.
Procedure steps
Step 1 2 Action Insert the DVD on which you backed up the Oracle/MySQL data, into the drive. Log on to the server. > telnet <server> Press the Enter key. 3 When prompted, enter your user ID and password.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
278
and press the Enter key. 5 6 When prompted, enter the root password. Verify the permissions on the restore log directory (bkslog).
# ls -alrt /var/opt/nortel
7 8
If the permissions of bkslog are drwxrwxrwt then go to step 10 If the permissions of bkslog are not drwxrwxrwt, then change the permissions of bkslog.
# chmod 1777 /var/opt/nortel/bkslog
and press the Enter key. 9 Determine if server applications have been stopped.
# servquery -status all
and press the Enter key. 10 11 If not already done, stop the server applications that run on the server. For Communication server 2000 Management Tools server applications see Stopping the SESM server application, Stopping the SAM21 Manager server application, Stopping the APS server application, Nortel ATM/IP Solution-level Administration and Security ( (NN10402-600)) , . For MG 9000 Manager and mid-tier server application see Nortel MG 9000 Fault Management ( (NN10074-911)) , Nortel MG 9000 Administration and Security ( (NN10162-611)) . For IEMS server application see Stopping the NPM server application, Nortel IEMS Administration and Security ( (NN10336-611)) . Verify the permissions, owner and group of /data/oradata by typing:
#ls -l /data
12
13
14
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
279
15
16 17
The DVD should eject automatically. Remove the DVD from the drive and proceed to the next step. Restore the database from the backup file.
$ /opt/nortel/sspfs/bks/rsdata -f /data/bkresmgr/bac kup/<filename>
and press the enter key. 18 19 20 If an alarm appears, or the database did not restore properly, contact Nortel. Otherwise, continue with the next step. If no alarm appears, and you do not need to restore the certificate, continue with the next step. If no alarm appears, and you need to restore the certificate, determine if WebPKProxy is configured.
# ls /data/pkclient/certificates
and press the Enter key. 21 22 23 If the files CA1_Cert.pem, CA2_Cert.pem, and TRUSTED_Cert.p em do not exist, continue with the next step. If the three files listed in step 17 are associated with this machine, reconfigure the WebPKProxy as follows. From the CLI, select Configuration ---> Security Services, Configuration ---> PKManager Certificate, Installation ---> install_certs (Install Certificates supplied by PKManager). Wait for the script to complete. Verify that no errors are displayed.
24
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
280
25
From the CLI, select Configuration ---> Security Services, Configuration ---> WebPKProxy/PKClient, Configuration---> register_pkclient. Enter the host name of the Certificate Manager. The Certificate Manager box must be fully functional for this step to succeed. Wait for the script to complete. Verify that no errors are displayed.
26 27
28
Start the server applications that run on the server. If one or more applications do not start, contact Nortel for assistance. If the files /etc/passwd, /etc/group were backed-up previously, the process prompts the following warning message:
WARNING !! Restoring /etc/passwd, /etc/shadow or /etc/group files could cause problems with respect to passwords, files ownership. Do you want to restore these files (y/n):[y] Select: n - No, if you have changed some passwords, added/changed new users/groups since the last backup. y - Yes, otherwise.
29
For Communication server 2000 Management Tools server applications see Starting the SESM server application, Starting the SAM21 Manager server application, Starting the APS server application, Nortel ATM/IP Solution-level Administration and Security ( (NN10402-600)) For MG 9000 Manager and mid-tier server applications see Nortel MG 9000 Fault Management ( (NN10074-911)) , Nortel MG 9000 Administration and Security ( (NN10162-611)) . For IEMS server application see Starting the NPM server application (page 297) Nortel IEMS Administration and Security ( (NN10336-611)) : "Starting the IEMS server or changing the IEMS server mode to standby."
--End--
30
31
Variable denitions
Variable <filena me> Value The name of the backup file created by the backup restore manager.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
281
Procedure steps
At the workstation UNIX prompt or VT-100 terminal prompt:
Step 1 2
Action Log in to the CBM using the root or emsadmin user ID and password. Start the cbmmtc tool by typing:
# cbmmtc
Check the MATE column on the banner. If the state is not "." (dot), this indicates the presence of an alarm. 3 4 5 6 Access the MATE by performing the procedure Accessing the MATE (page 222). Clear the alarms by performing the procedure Clearing the MATE alarm (page 281). Check to see if all fault conditions have now cleared.
querycbm flt
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
282
Procedure steps
Step 1 Action At your workstation, establish a logon session to the server, using one of the following methods In a two-server configuration, log in to the active server using the physical IP address of the active server, and ensure you are on the active server using the ubmstat command. 2 Verify the status of the SAM21 Manager server application by typing:
# servman query -status -group SAM21EM
and pressing the Enter key. 3 If SAM21EM is running continue with the next step and if SAM21EM is not running then you have completed this procedure. Stop the SAM21EM server application by typing:
# servstop SAM21EM
and pressing the Enter key. 5 Verify the SAM21EM server application stopped by typing:
# servman query -status -group SAM21EM
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
283
If applicable, return to the high level task or procedure that directed you to this procedure.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
284
Prerequisites
You must be logged in to the active server as the root user using telnet (unsecure) or ssh (secure).
Procedure steps
ATTENTION
In a two-server configuration, you must perform this procedure on the active server.
Step 1
If the server response is ClusterIndicatorACTthen you are on the active server. If the server response is ClusterIndicatorSTBY then you are on the inactive server. You must be logged in to the active server. 2 3 Verify the status of the SESM server application by typing:
# servman query -status -group SESMService
If SESM service is running then continue with step 4 and if SESM service is not running then you have completed this procedure. Stop the SESM server application by typing:
# servstop SESMService
and pressing the Enter key. 5 6 Wait approximately 3 to 5 minutes before you proceed to the next step to allow the SESM server application to stop. To verify the SESM services stopped by typing:
# servman query -status -group SESMService
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
285
If applicable, return to the high level task or procedure that directed you to this procedure.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
286
Prerequisites
You need root user privileges to perform this procedure. It is recommended that all users exit the NPM CLUI and GUI before stopping the NPM server application.
Procedure steps
ATTENTION
In a two-server configuration, perform the steps that follow on the Active server.
Step 1
Action At your workstation, to log in to the server type > telnet <server> and press the Enter key. In a two-server configuration, enter the physical IP address of the Active server (unit 0 or unit 1).
2 3
When prompted, enter your user ID and password. Change to the root user by typing:
$ su - root
and pressing the Enter key. 4 When prompted, enter the root password. In a two-server configuration, ensure you are on the Active server by typing ubmstat. If ClusterIndicatorSTBY is displayed in the response, which indicates you are on the Inactive server, log out of that server and log in to the other server through telnet using the physical IP address of the other unit. The response must display ClusterIndicatorACT, which indicates you are on the Active server. 5 Verify the status of the NPM server application by typing:
# servman query -status -group NPM
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
287
If the NPM server application is running then continue step 7 and if the NPM server application is not running then you have completed this procedure. Stop the NPM server application by typing:
# servstop NPM
and pressing the Enter key. 8 Verify the NPM server application is no longer running by typing:
# servman query -status -group NPM
--End--
Variable denitions
Variable <server> Value IP address or host name of the SPFS-based server where the NPM server application resides.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
288
Prerequisites
Before performing this procedure, verify that the APS server application is running. Enter:
# servman query -status -group APS
Procedure steps
ATTENTION
In a two-server configuration, perform the steps that follow on the Active server.
Step 1
Action At your workstation to log in to the server type > telnet <server> and press the Enter key. In a two-server configuration, enter the physical IP address of the Active server (unit 0 or unit 1).
2 3
When prompted, enter your user ID and password. Change to the root user by typing:
$ su - root
and pressing the Enter key. 4 When prompted, enter the root password. In a two-server configuration, ensure you are on the Active server by typing ubmstat. If ClusterIndicatorSTBY is displayed in the response, which indicates you are on the Inactive server, log out of that server and log in to the other server through telnet using the physical IP address of the other unit. The response must display ClusterIndicatorACT, which indicates you are on the Active server. 5 Stop the APS server application by typing:
# servstop APS
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
289
and pressing the Enter key. 6 Verify the APS server application stopped by typing:
# servman query -status -group APS
--End--
Variable denitions
Variable <server> Value IP address or host name of the SPFS-based server where the APS resides.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
290
Procedure steps
ATTENTION
In a two-server configuration, perform the steps that follow on the Active server.
Step 1
Action At your workstation, log on to the server by typing: > telnet <server> and pressing the Enter key. In a two-server configuration, enter the physical IP address of the Active server (unit 0 or unit 1).
2 3
When prompted, enter your user ID and password. Change to the root user by typing:
$ su - root
and pressing the Enter key. 4 When prompted, enter the root password. In a two-server configuration, ensure you are on the Active server by typing ubmstat. If ClusterIndicatorSTBY is displayed in the response, which indicates you are on the Inactive server, log off of that server and log on to the other server through telnet using the physical IP address of the other unit. The response must display ClusterIndicatorACT, which indicates you are on the Active server. 5 Verify the status of the APS server application by typing:
# servman query -status -group APS
and pressing the Enter key. 6 7 If APS server is not running then continue the next step and if APS server is running then you have completed this procedure. Start the APS server application by typing:
# servstart APS
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
291
and pressing the Enter key. 8 Verify the APS server application started by typing
# servman query -status -group APS
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
292
--End--
Variable denitions
Variable <server> Value IP address or host name of the SPFS-based server where the APS resides.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
293
Procedure steps
Step 1 Action At your workstation, establish a connection to the server that is hosting the CS 2000 Management Tools through telnet or SSH, and log on using the root user ID and password. In a two-server configuration, log on to the active server using the physical IP address of the active server, and ensure you are on the active server using the ubmstat command. 2 Verify the status of the SESM server application by typing:
# servman query -status -group SESMService
and press the Enter key. 3 4 If the response indicates not running then go to step 4 and if the response indicates running then go to step 7. Start the SESM server application by typing:
# servstart SESMService
and press the Enter key. 5 6 Wait approximately 3 to 5 minutes before you proceed to the next step to allow the SESM server application to start. To Verify the SESM server application started verify the SESM services started by typing:
# servman query -status -group SESMService
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
294
Preparation for disaster recovery COMPONENT STATUS -------------Proxy Agent RUNNING RMI Regisry RUNNING Snmpfactory RUNNING MI2 Server RUNNING Current number of SESM processes running: 4 (of 4) SESM APPLICATION STATUS: All Applications Ready
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
295
Procedure steps
Step 1 Action At your workstation, establish a connection to the server that is hosting the Communication Server Management Tools through Telnet or SSH, and log on using the root user ID and password. In a two-server configuration, log on to the active server using the physical IP address of the active server, and ensure you are on the active server using the ubmstat command. 2 Verify the status of the SAM21 Manager server application by typing:
# servman query -status -group SAM21EM
and pressing the Enter key. 3 4 If SAM21EM is not running then continue the next step and if the SAM21EM is running then you have completed this procedure. Start the SAM21EM server application by typing:
# servstart SAM21EM
and pressing the Enter key. 5 Verify the SAM21EM server application started by typing:
# servman query -status -group SAM21EM
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
296
If applicable, return to the high level task or procedure that directed you to this procedure.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
297
Prerequisites
You need root user privileges to perform this procedure, and CORBA must be running in order for the NPM to come up.
Procedure steps
ATTENTION
In a two-server configuration, perform the steps that follow on the active server.
Step 1
Action At your workstation, log on to the server by typing: > telnet <server> and pressing the Enter key. In a two-server configuration, enter the physical IP address of the Active server (unit 0 or unit 1).
2 3
When prompted, enter your user ID and password. Change to the root user by typing:
$ su - root
and pressing the Enter key. 4 When prompted, enter the root password. In a two-server configuration, ensure you are on the Active server by typing ubmstat. If ClusterIndicatorSTBY is displayed in the response, which indicates you are on the Inactive server, log out of that server and log on to the other server through telnet using the physical IP address of the other unit. The response must display ClusterIndicatorACT, which indicates you are on the active server. 5 Verify the status of the NPM server application by typing:
# servman query -status -group NPM
and pressing the Enter key. 6 If the NPM server application is not running then continue the next step and if the NPM server application is running then you have completed this procedure.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
298
and pressing the Enter key. 8 Verify the NPM server application is running by typing:
# servman query -status -group NPM
--End--
Variable denitions
Variable <server> Value IP address or host name of the SPFS-based server where the NPM server application resides.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
299
ATTENTION
After performing a full system restore on IEMS you must also perform a subsequent central server restore. See "Restoring the Central Security Server" in Nortel IEMS Administration and Security ( (NN10336-611)) .
Integrated Element Management System (IEMS) Audio Provisioning Server (APS) Media Gateway (MG) 9000 Manager Network Patch Manager Core Billing Manager (CBM) Multiservice Data Manager (MDM)
Restoring the oracle data is not applicable to the CBM as it does not use an oracle database.
ATTENTION
System logs indicating application and database errors will be generated until the file systems and oracle data are restored on the system using this procedure and procedure Restoring the Oracle/MySQL data on an SPFS-based server. No database errors will be generated on the CBM as it does not use an oracle database.
ATTENTION
System logs indicating application and database errors will be generated until the file systems and oracle data are restored on the system using this procedure and procedure Restoring the Oracle/MySQL data on an SPFS-based server. No database errors will be generated on the CBM as it does not use an oracle database.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
300
Prerequisites
you need SPFS CD disk #1 or SPFS DVD for the release you are reverting to you need the DVD on which you backed up the file systems you need the DVD on which you backed up the system data
ATTENTION
The recovery must be performed on the system where the backup was created. Recovery cannot be performed on a different system.
Procedure steps
ATTENTION
For MDM and other components that use a simplex configuration with no oracle database, complete ONLY steps 1-8. If the additional steps are executed, errors will occur. Simplex conguration
Step 1 2 3 4 5
Action At the server console, log in to the server through the console (port A) using the root user ID and password. Bring the system to the OK prompt by typing:
# init 0
Insert SPFS CD disk#1 or SPFS DVD into the drive. At the OK prompt, restore the system by typing: OK boot cdrom - restore When prompted, accept the software license restrictions by typing:
ok
and pressing the Enter key. The system reboots. If restoring from DVD, you will be prompted to insert Volume 1 of the backup DVD into the drive. Insert the DVD on which you backed up the file systems. During the restore process, the system will prompt you for additional Volumes if more than one DVD was used during the backup of file systems. The restore process can take several hours to complete depending on the number and size of the files that are being restored.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
301
Although it can appear as if the system is hanging at times, do not interrupt the restore process. If you suspect an issue with the restore process, please contact your next level of support. 6 7 8 Eject the backup DVD from the drive. Ensure you are at the root directory level by typing:
# cd /
If the DVD drive tray will not open after you have determined that the DVD drive is not busy and is not being read from or written to, enter the following commands:
# /etc/init.d/volmgt stop # /etc/init.d/volmgt start
Then press the eject button located on the front of the DVD drive. 9 10 11 12 13 14 15 Remove the backup CD or DVD from the drive. If the SPFS-based server is hosting the CBM then do step 12. If the server is hosting anything else then you have completed this procedure. List the oracle groups by typing:
# groups oracle
If the output is install data dba or install dba data then do step 16. If the output is otherwise then do step 15. Correct the oracle groups by typing:
# usermod -g oinstall -G data,dba oracle
and pressing the Enter key. 16 17 18 Restore the oracle data using procedure Restoring the Oracle/MySQL data on an SPFS-based server. Reboot the server by typing:
# init 6
Restore the central security server. For more information, see "Restoring the Central Security Server" in Nortel IEMS Administration and Security ( (NN10336-611)) .
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
302
Procedure steps
ATTENTION
The recovery must be performed on the same unit where the backup was created.
Step 1
Action At the console connected to the inactive node, log in to the inactive node through the console (port A) using the root user ID and password. Bring the system to the OK prompt by typing:
# init 0
2 3
At the console connected to the active node, log in to the active node through the console (port A) using the root user ID and password. Bring the system to the OK prompt by typing:
# init 0
4 5 6 7
Insert SPFS CD disk#1 or SPFS DVD into the drive. At the OK prompt, restore the system by typing: OK boot cdrom - restore When prompted, accept the software license restrictions by typing:
ok
The system reboots. 8 When prompted, insert Volume 1 of the DVD on which you backed up the file systems into the drive. During the restore process, the system will prompt you for additional Volumes if more than one DVD was used during the backup of file systems. The restore process can take several hours to complete depending on the number and size of the files that are being restored. Although it can appear as if the system is hanging at times, please do not interrupt the restore process. If you suspect an issue with the restore process, contact your next level of support. 9 Ensure you are at the root directory level by typing:
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps # cd /
303
10
If the DVD drive tray will not open after you have determined that the DVD drive is not busy and is not being read from or written to, enter the following commands:
# /etc/init.d/volmgt stop # /etc/init.d/volmgt start
Then press the eject button located on the front of the DVD drive. 11 12 Remove the backup DVD from the drive. Remove the backup CD or DVD from the drive. Ensure the replicated file systems on the newly restored unit are active.
# cd /opt/nortel/sspfs/ha ./ActivateNode
The output from execution of the ActivateNode script is as follows, and the user must enter Y to the prompt:
This procedure may cause data loss. Are you sure you want to proceed? [Y/N]
13
14
Execute the following command to confirm that all replicated file systems are active.
# udstat /audio_files: ACTIVE repair DOWN dirty /data: ACTIVE repair DOWN dirty /data_ordata (/data/ordata): ACTIVE repair DOWN dirty /user_audio_files: ACTIVE repair DOWN dirty
Contact your next level of support if any of the file systems show STANDBY as the state. 15 16 17 Use the following steps to determine your next step. If the SPFS-based server is hosting the CBM then do step 21. List the oracle groups by typing:
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
304
18 19 20
If the output is install data dba or nstall dba data then go to step 20. Correct the oracle groups by typing:
# usermod -g oinstall -G data,dba oracle
Restore the data using procedure Restoring the Oracle/MySQL data on an SPFS-based server. After the data restore is complete, execute step 17 of this procedure.
21 22
Restore the central security server. See "Restoring the Central Security Server" in Nortel IEMS Administration and Security ( (NN10336-611)) .
23
The output from execution of the ActivateNode script is as follows, and the user must enter Y to the prompt:
This procedure may cause data loss. Are you sure you want to proceed? [Y/N]
24
25
Re-image the inactive node using the active nodes image. If required, see the procedure Cloning the image of the active node to the inactive node of a Core and Billing Manager cluster.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
305
Prerequisites
when you are removing the card from service along with procedure Unlocking a GWC card (page 457) to reboot a GWC and force a software download as part of fault clearing activity to determine if a problem is temporary or persistent when you apply or remove a patch to the GWC software using the Network Patch Manager (NPM) and create a new GWC software image on the CS 2000 Core Manager when you are removing a GWC node from the CS 2000 GWC Manager database when you are replacing or upgrading hardware
If the card that you want to lock is active, switch call processing to its mate card in the node. The switch places the card in standby mode. If required, follow procedure "Invoking a manual protection switch (warm SWACT)" in Nortel Gateway Controller Administration and Security ( (NN10213-611)) . When the card is in standby mode, disable (busy) services on the card. Follow procedure "Disabling (Busy) GWC card services" in Nortel Gateway Controller Administration and Security ( (NN10213-611)) .
Procedure steps
At the CS 2000 GWC Manager client
Step 1
Action At the CS 2000 Management Tools window, click the Gateway Controller folder from the Device Types directory tree in the far left frame. From the Contents of: GatewayController frame, select the GWC node that contains the card you want to lock.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
306
Select the Maintenance tab to display maintenance information about the node.
4 5
Click the Card View button for the card you want to lock. At the CS 2000 SAM21 Manager client in the card view, select the States tab, then click the Lock button to lock the card.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
307
The card is locked when you see the text "Application locked successfully" in the History display. The lock icon (circled in the following figure) should also be present on the card graphic at the left of the screen:
If necessary, repeat this procedure for the next GWC card in the node.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
308
CAUTION
Possible service interruption
Do not lock the active Call Agent.
Procedure steps
At the CS 2000 SAM21 Manager client
Step 1
Action From the Shelf View, right click on the card and select Lock from the context menu.
Lock is also available from the States tab of the Card View window. 2 Do not confirm a lock warning. The warning is only available for the active Call Agent. Wait for the lock icon to appear on the selected card.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
309
Job aid
The CS 2000 SAM21 Manager client responds to an active Call Agent lock with the prompt show in Figure 20 "Call Agent lock warning" (page 309). Do not click Yes. The inactive Call Agent is located in the other CS 2000 SAM21 Manager shelf and a lock request does not provide a prompt when the Call Agent is inactive.
Figure 20 Call Agent lock warning
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
310
Prerequisites
Collect the following information beforehand to ensure that you have the information ready for input for this procedure. Also acquire the root user ID and password.
ATTENTION
Also gather this information prior to an upgrade. System variable Hostname IP address Netmask Router Unit 0 IP address Daily backup (up to four) Default gateway IP IP of primary cluster unit 0 Format = where MM minutes (00 - 59) DNS domain IP address DNS search domain(s) Timezone SPFS serial number Timezone = Timezone (# echo $TZ) Located on label on back of SPFS DNS server(s) Remote backup server Actual value Description
DNS variables apply only when a DNS server has been configured.
Procedure steps
Installing the remote backup server on a Geographic Survivability standby server
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
311
Step 1
and press the Enter key. Go to step 5. 3 Log in to the remote server through the console (port A) and when prompted, enter the root or emsadm user ID and password. Bring the system to the {0} OK prompt by typing:
# init 0
4 5
Enter the following command to verify that the auto-boot option is set to true.
{0} ok printenv auto-boot?
Insert the SPFS0** DVD into the DVD drive on the remote backup unit. where ** refers to the release (for example, 091 for SN09.1)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
312
Example responses
{{0} ok boot cdrom - rbackup Boot device: /pci@1e,600000/ide@d/cdrom@0,0:f File and args: -r SunOS Release 5.8 Version Generic_108528-18 64-bit Copyright 1983-2001 Sun Microsystems, Inc. All rights reserved. Hardware watchdog enabled Configuring /dev and /devices Using RPC Bootparams for network configuration information. Skipping interface bge3 Skipping interface bge2 Skipping interface bge1 Skipping interface bge0 / This CD contains software that under the terms of your license agreement with Nortel Networks is licensed for use only on Nortel Networks OAM&P servers. Type "ok" to acknowledge this restriction on your use of the software. Any other input will halt the installation of this software. Type "ok" to acknowledge this restriction on your use of the software. Any other input will halt the installation of this software.
Type OK and press Enter to acknowledge restriction on your use of the software.
Searching for configuration file(s)...
Enter the serial number located in the rear view of the SPFS and press Enter.
*** REMOTE Backup INSTALL *** Begin entering installation information Please select a server profile for this system rbackup: Remote Automatic Backup standby system Enter the server profile for this system: ( rbackup )
10
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
313
11
Enter No to not select the default settings. The N240 server must be connected to the network and must have access to the default gateway. This allows you to enter the servers settings for the installation. Enter site-specific information in response to the following prompts. (See the information entered in the table at the beginning of this procedure.)
Enter Enter Enter Enter Enter the hostname for this system. the IP address for <hostname>. the subnet mask for this network. the IP address for this networks router. the timezone for this system [US/Eastern].
12
The default timezone is US/Eastern. Enter ? for a list of supported time zones.
Will this system use DNS? [yes]
Enter yes or no. If you answer yes, you are prompted for the DNS domain name, name server IP addresses, and the search domains. You can enter several name servers and search domains. Enter a blank line to stop the entry. 13 Click OK to accept current settings. The installation of the DVD takes approximately 60 minutes. If necessary, press Enter to redisplay the login prompt on screen after the following message displays:
NOTE: NOTE: SSPFS Installation is now completed. NOTE: HostName> console login:
14 15 16 17 18 19 20
Log in to the server using the root user ID and password. (At this point, the root password has been reset to "root.") Remove the SPFS DVD from the server. If necessary, enter
eject cdrom
Enter the number next to the Configuration option in the menu. Enter the number next to the Succession Element Configuration option in the menu. Enter the number next to the PSE Application Configuration option in the menu. Enter the number next to the Configure PSE option in the menu.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
314
21 22
Enter the primary/cluster IP address of CS 2000 Management Tools server. This is the address of the NPM server. Enter Y to confirm the IP address. Ignore the following error message if it displays.
Cant configure PSE on remote backup unit to enable NPM
23 24 25
Enter X to exit each level until you have exited from the cli tool. Start the PSE server.
# pse start
Verify that the server has started. If the server does not start, contact your next level of support.
# pse status
26 27 28
If an SPFS MNCL CD is to be installed, see the documentation included with the CD for complete installation instructions. Enter NPM on the CS 2000 Management Tools server and follow patching procedures to apply all relevant patches. To forward SNMP traps on the server to the IEMS, perform the procedure "Configuring the destination for SNMP traps" in Nortel ATM/IP Solution-level Configuration ( (NN10409-500)) .
--End--
Variable denitions
Variable <server> Value Name of the N240 server.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
315
ATTENTION
This procedure is for use with Geographic Survivability only.
Procedure steps
Step 1 2 Action Determine how to log in to the Sun Netra 240 (N240) server that is installed as the remote backup server. To log in using SSH, type:
ssh -l root (server>
and press the Enter key. Go to step 4. 3 To log in using Telnet, type:
telnet <server>
and press the Enter key. 4 5 6 7 When prompted, enter the root password. Start the command line interface tool by entering:
cli
Enter the number next to the Configuration option in the menu. The system displays the Configuration menu. Enter the number next to the Remote Backup option in the menu. Response:
Remote Backup Configuration 1 - rbackup_display (Display Remote Backup Configurati on) 2-rbackup_config (Remote Backup Configuration) 3-rbackup_exec (Execute Remote Backup Now) 4 - rbackup_cancel (Cancel Running Remote Backup) X-exit
Select
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
316
The system responds with the IP address of the primary server that is currently configured as the remote server, and the times that are currently configured for automatic backups. 9 Enter the unit 0 IP address of the primary server to be backed up.
<nnn.nnn.nnn.nnn> is alive
10 11 12
If the system prompts you to accept the ssh key Enter yes. Go to step 13. If the system does not prompt you to accept the ssh key then go to step 13. Enter the first time for a daily backup to occur. You can configure up to four times for daily backup to occur. Response:
Enter a time for a daily backup to occur (HH:MM):
Response:
Enter a second time for a daily backup to occur (HH:MM) or enter "x" to stop provisioning backup times:
13 14 15 16 17 18 19 20 21 22 23
If you want to enter another time for a remote backup to occur enter a second time for a daily backup to occur. Go to step 16. If you do not want to enter another time for a remote backup to occur enter x. Go to step 20. If you want to enter another time for a remote backup to occur, enter a third time for a daily backup to occur. Go to step 18. If you do not want to enter another time for a remote backup to occur enter x. If you want to enter another time for remote backup to occur enter a fourth time for a daily backup to occur. Go to step 20. If you do not want to enter another time for a remote backup to occur enter x. Enter a bandwidth limit in KB/s ( 0 means no limit ) [0]: To commit changers go to step 24. To exit enter quit. To re-enter setting enter anything other than ok or quit. Enter
ok === "rbackup_config"completed successfully
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
317
24
Variable denitions
Variable <nnn.nnn.nnn.nnn> <server> HH MM Value IP address that you entered. Name of the N240 server. HH is hours. Valid values are 00 to 23. MM is minutes. Valid values are 00 to 59.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
318
ATTENTION
This procedure is for use with Geographic Survivability only.
Procedure steps
Step 1 2 Action Log in to the Sun Netra 240 (N240) server that is installed as the remote backup server. To log in using ssh, type:
ssh -l root <server>
and press the Enter key. Go to step 4. 3 To log in using telnet, type:
telnet <server>
and press the Enter key. 4 5 When prompted, enter the root password. Start the command line interface tool by entering:
cli
The system responds by displaying a menu. 6 7 Enter the number next to the Configuration option in the menu. The system displays the Configuration menu. Enter the number next to the Remote Backup option in the menu. Response:
Remote Backup Configuration 1-rbackup_display (Display Remote Backup Configuration) 2-rbackup_config (Remote Backup Configuration) 3-rbackup_exec (Execute Remote Backup Now) 4 - rbackup_cancel (Cancel Running Remote Backup) X-exit
Select
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
319
Response:
Current settings: Target system is: <nnn.nnn.nnn.nnn> Back up times are: <Time 1>...<Time n>
Variable denitions
Variable server <nnn.nnn.nnn.nnn> <Time 1>...<Time n> Value Name of the N240 server. IP address of the remote server. Set of times at which automated backups occur.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
320
ATTENTION
This procedure is for use with Geographic Survivability only.
Procedure steps
Step 1 2 Action Log in to the Sun Netra 240 (N240) server that is installed as the remote backup server. To log in using ssh, type:
ssh -l root <server>
and press the Enter key. Go to step 4. 3 If you want to log in using telnet, enter:
telnet <server>
and press the Enter key. 4 5 When prompted, enter the root password. Start the command line interface tool.
cli
The system responds by displaying a menu. 6 7 Enter the number next to the Configuration option in the menu. The system displays the Configuration menu. Enter the number next to the Remote Backup option in the menu.
Remote Backup Configuration 1-rbackup_display (Display Remote Backup Configuration) 2-rbackup_config (Remote Backup Configuration) 3-rbackup_exec (Execute Remote Backup Now) 4 - rbackup_cancel (Cancel Running Remote Backup) X-exit
Select
3-rbackup_exec (Execute Remote Backup Now)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
321
An automatic backup is made. Pressing 4 during execution of the backup halts the process and performs necessary clean-up operations.
10
Variable denitions
Variable <server> Value Name of the N240 server.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
322
Procedure steps
Step 1 2 Action Log in to the Sun Netra 240 (N240) server that is installed as the remote backup server. To log in using SSH, enter:
ssh -l root <server>
and press the Enter key. Go to step 5. 3 To log in using Telnet, enter:
telnet <server>
and press the Enter key. Go to step 5. 4 5 To log in to the remote server console then go to step 5. Enter:
less /var/adm/messages
Variable denitions
Variable <server> Value Name of the N240 server.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
323
Prerequisites
If the server was previously a CBM and contains billing files not already sent to a downstream billing server, using the cluster server, copy these files to a downstream server before performing this procedure. Otherwise these files and the billing records will be lost. Contact next level of support for assistance. Both cluster units 0 and 1 must be in the firmware mode. (Enter command init 0 to perform this action) In this procedure, unit0 server of the cluster is used as a remote backup server. Use the same hostname and IP address that was used to configure the remote backup server.
WARNING
If both the remote backup unit and the high-availability (HA) pair are in service after a switch command, both units vie for dominance. This conflict results in a loss of communication to the core.
Procedure steps
Step 1 Action Follow procedure Installing the remote backup server (page 310). Use only one automated schedule and make sure to select a time that will not be invoked shortly. 2 3 Follow procedure Performing a manual backup of the target server (page 320). Bring down the machine currently active by following procedure Two-server (cluster) configuration in Shutting down an SPFS-based server in . Follow procedure Initiating a switch over to the remote backup server (page 325) to bring the services back to unit0 of the cluster. Follow procedure Cloning the image of one server in a cluster to the other server in the document . If the server was previously a CBM and contains billing files not already sent to a down-stream billing server, use the remote backup server to copy these to a downstream server before
5 6
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
324
performing Installing the remote backup server (page 310). Otherwise these files and the billing records are lost. Contact next level of support for assistance. 7 8 Reinstall the backup server following procedure Installing the remote backup server (page 310). Reconfigure the backup server following procedure Scheduling automatic backups on the remote server (page 315).
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
325
ATTENTION
When completed, this procedure reboots the remote backup server as the unit0 of the cluster.
Prerequisites
Prior to starting this procedure, shut down the Cluster machine. The user must be logged in as the root or emsadm user to initiate the switch command. For more information, see the section Two-server (cluster) configuration in Shutting down an SPFS-based server in document Nortel ATM/IP Solution-level Fault Management ( (NN10408-900)) , , and .
CAUTION
If configuration, provisioning, patching or other write-type operations occurred since the last remote backup, the remote backup system can be out of sync compared to the data in network elements and/or the primary OAM system. Take actions before initiating the switchover to a remote backup OAM server (that is, response to a geographic or other prolonged outage of the primary OAM system) to halt or prevent write-type operations by OSSs and operations personnel until an in-sync status is achieved. When initiating a switchover to a remote backup OAM server, do not execute configuration, provisioning, patching or other write-type operations through the remote backup OAM system until out-of-sync conditions are cleared.
Procedure steps
Step 1 Action Log in to the server by typing
> telnet <server>
and pressing the Enter key. 2 3 When prompted, enter your user ID and password. Change to the root user.
su - root
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
326
4 5 6
When prompted, enter the root password. Invoke the switch by typing:
$ /opt/sspfs/rbks/switch
Variable denitions
Variable <server> Value IP address or host name of the SPFS-based remote backup server.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
327
ATTENTION
Perform this procedure immediately after a site failure occurs to limit effects to Call Processing.
This procedure is required to workaround a deficiency in the architecture when the active in-service and SYSB (out of service) GWC send messages to a gateway provisioned off the GWC causing mismatches in the gateway resulting in loss of call processing. This procedure will impact every element sitting off the ERS 8600. There are two options for this procedure. The first option will disable all dynamic routing on the chassis and no traffic will be able to leave or enter the VLANs configured on that chassis when using Open Shortest Path First (OSPF). Static routes and Local communication is still possible. The second option will disable the WAN Edge Interface. All traffic, whether through dynamic or static routes will not be able to leave or enter the chassis.
Procedure steps
At the Ethernet Routing Switch 8600 Workstation
Step 1 2 3
Action To use option 1, go to step 3 To use option 2, go to step 5 Disable OSPF by typing
#config ip ospf disable
and pressing the Enter key. 4 5 This procedure is complete. Identify the WAN Edge VLAN ID by typing
#show vlan info basic
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
328
Variable denitions
Variable <vlan_id> <port_range_1> Value is the VLAN ID obtained in step 4 is a range of ports used for WAN Edge
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
329
Disabling and re-enabling the router ports on the Ethernet Routing Switch 8600
Use this procedure when a failure occurs, such that Sites A and B are isolated from each other, to disable and re-enable the Ethernet Routing Switch 8600 router ports that are connected to the GWC in the inactive site.
Prerequisites
You must know the list of router ports connected to the GWCs.
Procedure steps
Step 1 2 3 Action At the Ethernet Routing Switch 8600 Workstation, confirm which site has the active Call Agent. Confirm that the GWCs in the active site are active. Disable the ports to the GWC units on the inactive site by typing:
#config ethernet <port_range_1> state disable #config ethernet <port_range_2> state disable ... #config ethernet <port_range_n> state disable
and pressing the Enter key. 4 If you disabled OSPF in the procedure Disabling Ethernet Routing Switch 8600 WAN Edge routing on page 205, re-enable it by typing:
#config ip ospf enable
and pressing the Enter key. 5 If you disabled the ports used for WAN Edge connectivity, re-enable those ports by typing:
#config ethernet <port_range_1> state enable #config ethernet <port_range_2> state enable ... #config ethernet <port_range_n> state enable
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
330
7 8
Allow at least one minute to pass to ensure that internal GWC audits set the units to inactive. Check the time by typing:
#date
9 10 11 12 13
If the inactive building is accessible, perform steps 10 through 12 to verify the inactive GWC units have gone inactive. Connect the terminal to the console port on the GWC. Enter PMDEBUG and perform the T, N, and A commands. Confirm that the state is inactive. if not, repeat steps 3 through 8 on those ports. Re-enable the ports to the GWC units in the inactive site by typing
#config ethernet <port_range_1> state enable #config ethernet <port_range_2> state enable ... #config ethernet <port_range_n> state enable
Variable denitions
Variable <port_range> Value is a range of ports connected to the GWC (example: 3/1-3/48)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
331
The network elements should be backed up before powering down if time permits.
Network power down task flow This task flow shows you the sequence of procedures you perform to power down the network.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
332
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
333
Powering down an MG 9000 device (page 335) Powering down the IPmedia 2000 shelves (page 337) Powering down the SAM16 shelves (page 338) Powering down a Border Control Point (page 339)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
334
Powering down the MCS servers (page 340) Powering down the MDM workstation (page 341) Powering down an SPM device (page 342) Powering down a Media Gateway/Multiservice Switch 7400 (page 345) Powering down a Media Gateway/Multiservice Switch 15000 (page 346) Powering down a USP (page 347) Powering down the SAMF frame (page 348) Powering down the Call Control Frame (page 350) SAM21 Shelf Controller security and administration fundamentals (page 352) Powering down the Border Control Point Manager (page 355) Powering down the Border Control Point Manager (page 355) Powering down a Netra 240 server (page 356) Powering down the SDM (page 357) Powering down a Ethernet Routing Switch 8600 (page 358) Powering down the XA-Core (page 360) Shutting down the IEMS server (page 359) Performing a partial power down of the MG 9000 (page 363) Performing a partial power down of the Media Gateway/Multiservice Switch 15000 (page 364) Performing a partial power down of the USP (page 368) Performing a partial power down of the Ethernet Routing Switch 8600 (page 369) Performing a partial power down of the Netra 240 server (page 370) Performing a partial power down of the XA-Core (page 371)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
335
Prerequisites
The optimal power threshold should be 43.7 volts. The minimum is 42.5 volts.
Procedure steps
Step 1 Action
ATTENTION
It is recommended to only unseat the fuses to allow for rapid restoration.
At the MG 9000 frame, remove the CE (SB) fuses from the frame. See Job aid (page 335) to identify the fuse locations. 2 3 Remove the TB fuses from the frame. Remove the CU fuses from the frame.
--End--
Job aid
The following figure identifies the fuse locations.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
336
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
337
Procedure steps
Step 1 2 Action At the CCF or SAMF frame, open the front door of the CCF/SAMF frame. At the EBIP at the top of the frame, turn the breaker associated the Media Server to the OFF position.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
338
Procedure steps
Step 1 2 Action At the CCF or SAMF frame, open the front door of the CCF/SAMF frame. At the EBIP at the top of the frame, turn the breakers associated the SAM16 to the OFF position.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
339
Procedure steps
Step 1 Action At the frame housing the Border Control Point, power down the shelf by switching the breaker on the back of the chassis to the OFF position. For AC powered shelves, push the rocker located on the top of the AC outlet to apply power to the chassis. It should be switched to the "0" position. For DC powered shelves, switch the circuit breakers at the EBIP that provide power to the shelf being powered to the OFF position. The breaker will have the 0 side (bottom) of the breaker depressed.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
340
Procedure steps
Step 1 Action At the frame housing the MCS servers, power down the MCS servers by turning off the circuit breakers that provide power to the servers.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
341
Prerequisites
It is recommended that you shut down the MDM workstation while on battery power.
Procedure steps
Step 1 Action At the workstation terminal window, enter the following commands to shut down it:
# /bin/sync # /bin/sync # /sbin/init 0
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
342
Prerequisites
MG 4000s may or may not be located at the same location as the Call Server and therefore power conservation may or may not need to be considered separately from the core location.
Procedure steps
Step 1 Action
ATTENTION
This procedure is not necessary for the IW-SPM or DPT-SPM.
At the MAP terminal, post and busy the trunks for the SPM or MG 4000 to be powered down at the TTP level.
>MAPCI;MTC;TRKS;TTP;POST D SPM <spm_no>;BSY ALL
ATTENTION
This procedure is not necessary for the IW-SPM or DPT-SPM.
3 4 5
ATTENTION
The VT15P and DS3P carriers may not both exist.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
343
ATTENTION
The VT15P and DS3P carriers may not both exist.
7 8
ATTENTION
This step only applies to an MG 4000 or IW-SPM.
9 10 11 12 13 14 15 16 17
Power down the SIM cards using the switch on the front. Then pull the fuses from the PCIU. Unseat hardware as required to avoid water damage and shorts.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
344
Variable denitions
Variable <spm_no> <rm_type> <inactive_rm_number> <inactive_cem_number> Value is the number of the SPM/MG 4000/IW-SPM/D PT-SPM is the type of RM to select (OC3, DSP, and VSP) is the number of the inactive RM is the number of the inactive CEM
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
345
Prerequisites
Procedure steps
Step 1 Action
WARNING
If primary power to the shelf assembly is on, never set the individual power supply switches to the standby position. This can cause an overload to the on-line power supply.
ATTENTION
The front panel LED of each power supply remains lit for approximately 2 minutes after the power supply loses power.
At the chassis, turn off the circuit breakers for the outlets that supply power to your switch. 2 After a couple of minutes, verify the LEDs for the power supplies, cooling unit, and each processor card faceplate are off.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
346
Prerequisites
Procedure steps
Step 1 Action At the chassis, breaker interface panels (BIPs) are located at the top of the chassis. Power down all five breakers simultaneously on each BIP, starting at the left. Shut off the BIPs in the following order:
ATTENTION
This action shuts down the upper shelf.
BIP B2 BIP A2
ATTENTION
This action shuts down the lower shelf.
BIP B1 BIP A1
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
347
Prerequisites
The USP backup data should be up-to- date. No datafill changes should be made between the final data snapshot and the powering down of the USP. All users need to be logged out of the USP GUI and CLI.
Procedure steps
Step 1 Action
CAUTION
The following steps will take the USP out of service. In a redundant network configuration, the USPs mated STP will handle traffic after the USP is powered down. End office isolation will occur if the USP is configured as a Signaling Gateway.
ATTENTION
The USP bulk-input feature and the CLI can be used to automate these tasks in the event there are a large number of links in question (see the "Bulk Data Input of System-db Tables" section of the USP CLI Interface Specification Manual)
Inhibit and deactivate all links. 2 At the USP Manager, ensure that the LEDs on slot 13 and slot 16 front panel cards are not lit, and immediately turn off the A/B Power buttons on the control shelf. Turn off the A/B Power buttons located on the backside of each extension shelf. In the event that access to the USP is not available, it will be necessary to disconnect the power source that feeds the USP.
--End--
3 4
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
348
Procedure steps
Step 1 2 3 Action At the Session Server unit, perform procedure Lock the SIP gateway application on the active Session Server unit. Perform procedure Suspend the SIP gateway application on the active Session Server unit.
ATTENTION
The state of the SIP Gateway application is saved when the unit is powered off. When the unit is powered up, the SIP Gateway application initializes in the same state in which it was powered down.
Perform procedure Halt (shutdown) a Session Server unit on the inactive Session Server unit. 4 5 Perform procedure Power-Off a Session Server unit on the inactive Session Server unit.
ATTENTION
The state of the SIP Gateway application is saved when the unit is powered off. When the unit is powered up, the SIP Gateway application initializes in the same state in which it was powered down.
Perform procedure Halt (shutdown) a Session Server unit on the active Session Server unit. 6 7 8 9 10 11 Perform procedure Power-Off a Session Server unit on the active Session Server unit. At the Policy Controller Unit, perform procedure Halt (shutdown) a Policy Controller unit on the inactive Policy Controller unit. Perform procedure Power-Off a Policy Controller unit on the inactive Policy Controller unit. Perform procedure Halt (shutdown) a Policy Controller unit on the active Policy Controller unit. Perform procedure Power-Off a Policy Controller unit on the active Policy Controller unit. Lock all active GWC cards.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
349
12 13 14 15 16 17
At the CS 2000 GWC Manager, lock all inactive GWC cards. At the CS 2000 SAM21 Manager, lock any inactive cards other than the SC cards. Lock any active cards other than the SC cards. Lock the inactive SC card. Lock the active SC card. At the SAMF frame, power down the SAMF frame BIP breakers. At the top of the cabinet, turn off the breakers that supply power to the two SAM21 shelves and the Session Server (if equipped).
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
350
Procedure steps
Step 1 Action At the Session Server unit, perform procedure Lock the SIP gateway application on page 66 on the active Session Server unit. Perform procedure Suspend the SIP gateway application on page 68 on the active Session Server unit.
2 3
ATTENTION
The state of the SIP Gateway application is saved when the unit is powered off. When the unit is powered up, the SIP Gateway application initializes in the same state in which it was powered down.
Perform procedure Halt (shutdown) a Session Server unit on page 58 on the inactive Session Server unit. 4 5 Perform procedure Power-Off a Session Server unit on page 71 on the inactive Session Server unit.
ATTENTION
The state of the SIP Gateway application is saved when the unit is powered off. When the unit is powered up, the SIP Gateway application initializes in the same state in which it was powered down.
Perform procedure Halt (shutdown) a Session Server unit on page 58 on the active Session Server unit. 6 7 8 9 10 11 12 13 Perform procedure Power-Off a Session Server unit on page 71 on the active Session Server unit. At the CS 2000 GWC Manager, lock all inactive GWC cards. Lock all active GWC cards. At the CS 2000 SAM21 Manager, lock the inactive USP-lite card. Lock the active USP-lite card. Lock the inactive Compact Call Agent (CCA) card. Lock the active CCA card. At the front panel of the STORM SAM-XTS units, if your CCF is equipped with SAM-XTS units, disconnect the power to each
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
351
unit using the main power switch located on the front panel. For figure, see Job aid (page 351). 14 At the CCF, power down the CCF BIP breakers. At the top of the cabinet, turn off the breakers that supply power to the Session Server, the two SAM21 shelves, and the Storm SAM-XTS units or Storm Disk Array.
--End--
Job aid
The following figure helps you to power down the Call Control Frame.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
352
Navigation
Security and administration strategy (page 352) Tools and utilities (page 352) User administration (page 352) Device administration (page 353)
physical security of the client workstation physical security of the SAM21 shelf security mechanisms available through the client workstation operating system SAM21 SCs only allow login connections from the CS 2000 Core Manager, the host that provides the CS 2000 Management Tools, and the mate SAM21 SC.
User administration
The SAM21 SCs do not implement any custom security features. The CS 2000 SAM21 Manager client authenticates users when the application is started.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Device administration
353
Adding a user
The SAM21 SCs do not provide user administration. The CS 2000 SAM21 Manager provides authentication, but not user administration. For procedures on how to add a user to the CS 2000 SAM21 Manager client workstation, contact your system administrator or refer to the operating system reference documentation.
Setting privileges
The SAM21 SCs and the CS 2000 SAM21 Manager client do not implement user privilege administration.
Removing a user
To prevent user access to the CS 2000 SAM21 Manager client, contact your system administrator to disable the user account on the workstation.
Device administration
Administration of the SAM21 CSs is done through the CS 2000 SAM21 Manager client. This section lists the administration tasks that you can perform.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
354
Only lock a circuit pack that is in service. If a SAM21 SC is unlocked and then re-locked before the SAM21 SC boots, flash memory can be corrupted and require a replacement SAM21 SC. Check that a SAM21 SC is Unlocked and Enabled from the Status tab of the Card View window.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
355
Prerequisites
You must have the root user password for N240. You must have the IP addresses for N240.
Procedure steps
Step 1 2 Action At the N240 server, Telnet or SSH into the server and logon as the root user. Enter
#init 0
to shutdown the server and to close the Telnet or SSH connection. 3 At this point, the server can be turned off at the circuit breakers.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
356
Prerequisites
You must have the root user password for the N240 server. If you are shutting down the N240s, you need IP addresses for both the active and inactive units.
Procedure steps
Step 1 2 Action At the N240 server, Telnet or SSH into the inactive server and logon as the root user. Type
#init 0
The server shuts down gracefully and the Telnet or SSH connection is closed. 3 4 Telnet or SSH into the active server and log in as the root user. Type
#init 0
The server shuts down gracefully and the Telnet or SSH connection is closed. 5 Turn off both servers at the circuit breakers.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
357
Procedure steps
Step 1 2 Action At the VT100 terminal, login as the root user. Shutdown the SDM by typing
# Shutdown -f
At the SDM frame, turn off the two breakers at the top of the SDM frame.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
358
Procedure steps
Step 1 Action At the Ethernet Routing Switch 8600 frame, If you have an 8010co Chassis unit and an optional breaker interface panel (BIP) installed, set the BIP circuit breakers to the off position. See Installing the Breaker Interface Panel for the 8010co Chassis, 312755-E. 2 3 Turn the power supply switch to the off position. See Job aid (page 358) for figure. Disable the incoming power from the DC inpt power source. You may need to switch a circuit breaker or turn off the DC input power source.
--End--
Job aid
Figure 24 Turning the power supply switch to the off position
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
359
Prerequisites
The shutdown process checks for the authenticity and the permissions of the user who invokes operation. You must be authorized to perform this procedure.
Procedure steps
Step 1 2 3 4 5 6 Action Stop all the schedulers. Notify the registered shutdown observers. Shut down all the subprocesses (sub-modules), which execute specific tasks. Disconnect all database connections. If it was started by IEMS, shut down the web server . Exit (the main process). To shut down the IEMS server, kill the related Java application shell, press Ctrl+C. The shutdown process checks for the authenticity and the permissions of the user invoking the shutdown operation, and is allowed only if the user has the proper permissions.
Shutting down the IEMS server from the command line
To shutdown the server using the servman command, start the command line interface and type /opt/servman/bin/servstop IEMS System response Stopping group using servstop The I-EMS Server on host "192.168.4.176" was successfully shutdown IEMS Stopped
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
360
in the event of an emergency, for example, flooding or fire to protect equipment if the available functioning voltage at the power distribution center (PDC) falls below -43.75 V dc when instructed by the next level of support
Procedure steps
CAUTION
This procedure results in a complete loss of subscriber service. Nortel recommends that before you perform this procedure, you should contact emergency Technical Assistance Services (ETAS) of Nortel or contact your next level of support.
Step 1
Action During emergency shutdown of the switch, at your current location, notify network management personnel of the impending service interruption. Notify emergency services (police, fire, ambulance) of the impending service interruption. If the switch must be shut down immediately due to dangerous environmental conditions then go to step 5. If there is time (one half hour or more) to shut down the switch in an orderly fashion then go to step 7.
2 3 4 5
ATTENTION
Turning off the switch in this manner should be done only if absolutely necessary, as current arcing may occur.
Turn off the power to the power distribution centers (PDCs) by disconnecting the power feeds at the power room. 6 7 You have completed the procedure. Using office records, identify the peripheral modules that host emergency services (such as fire, police, and ambulance), so that these peripherals can be shut off last.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
361
ATTENTION
Older peripheral modules, such as line modules and digital carrier modules, use universal tone receivers resident in maintenance trunk modules.
Turn off the power converter for each maintenance trunk module shelf, except those identified in step 7 as essential for emergency service lines. 9 10 Turn off the inverters for all MAPs and printers, except the operator MAP and one printer. Turn off the power converters on all digital trunk controllers and trunk modules, except those identified in step 7 as essential for emergency service communications. Turn off the power converters on each line module shelf, except those identified in step 7 as hosting emergency service lines. Turn off the power converters on all line concentrating modules, except those identified in step 7 as hosting emergency service lines. Turn off the power converters on all line group controllers and line trunk controllers, except those identified in step 7 as essential for emergency service communications. Turn off the power on all remaining peripheral modules, leaving essential service peripheral modules until last.
11 12
13
14 15
ATTENTION
To power down the ENET and the LPP, unseat and then reseat the power converters.
Turn off the power for the network modules, link peripheral processor, and input/output controllers (IOCs). 16 Turn off the power for all remaining devices, including the inverter that supplies the operator MAP, and any external printers, tape drives, or disk drives. Turn off the power for the XA-Core. Power down both SIM cards in the XA-Core. In each SIM card you must turn off three breakers. Turn off the power for the message switch. Power down one plane by turning off the power converters. Power down the second plane by unseating and then reseating the power converters.
17
18
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
362
19
Turn off the power to the power distribution centers (PDCs) by discontinuing the A and B feeds at the power room.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
363
Procedure steps
Step 1 2 3 4 Action At the MG 9000 Manager, determine the inactive Supercore, ITP, and ITX cards. Lock each inactive card using the MG 9000 Manager. Take the cards offline using the MG 9000 Manager. After the cards are offline, they can be manually unseated.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
364
0 3 4 7 8 11 12 15
The high level steps of this procedure include the following activities: ensuring that the LAPS ports are active on the powered-up cards and inactive on powered-down cards ensuring the active PBG port is on a powered-up card ensuring the PBG port that connects to the SAM21 shelf controller is active ensuring the correct DS3 card remains active switching off selected power breakers
Procedure steps
Step 1 Action Ensure that the active lines are on the cards to remain powered up by typing
> d laps/* nearEndRxActiveLine, workingLine, protectionLine
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
365
If there are any LAPS ports on a card that will be powered down, a protection switch needs to be performed. To switch from the protection line to the working line, type
> switch -protectionworking laps/x
One of the HIOPs on the XA-Core may be powered down for power conservation reasons. Determine which PBG port the HIOP corresponds to by typing
> d pbg/* working line
For example, see Job aid (page 366). 4 5 6 Determine which PBGs are connected to the XA-Core by typing
>d atmif/* remoteatmif
Match the PBG port number with the AtmIf number. Display the Sonet ports to determine whether the remote-end HIOP cards are still in service by typing
>d lp/10 sonet/3 operational
followed by typing
>d lp/11 sonet/0 operational
Determine the states of the losAlarm and rxAisAlarm. If the losAlarm is On, then the corresponding HIOP card is likely powered down. If the rxAisAlarm is On, the HIOP card is probably locked.
Verify that this HIOP card is selected to be powered down for power conservation. The other Sonet port should have no alarms raised under the operational group. Select AtmIFs with matching instance numbers. For example, if pbg/1100 and 1003 are provisioned, type
>d atmif/1100 vcc/* vcd tm atmService Category >d atmif/1003 vcc/* vcd tm atmService Category
10
Determine which PBG port is active by displaying the corresponding AtmIf vcc traffic counts. Type
>d atmif/X vcc/Y txcell, rxcell
11
The active PBG port will have incrementing txcell and rxcell counts. Ensure that the active PGB ports workingLine is on one of the cards that will remain in-service (0, 3, 4, 7, 8, 11, 12, or 15).
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
366
12
If the PVG ports workingLine is not on one of the cards that will remain in-service, perform a Swact of the SAM21 Shelf Controller. Verify that the other PBG port to this SAM21 shelf now has incrementing txcell and rxcell counts. For a 4pDS3Ch or 12pDS3 card pair in slots 14 and 15, type
>d -p lp/14 activeCard, mainCard, spareCard Lp/14
13 14 15
Ensure that the active card is in the set of cards that will remain in-service. If not, switch Lp activity by typing
>switch lp/14
. 16 Ensure that the activeCard from step 14 matches the MG 9000 equipment that will remain in-service during the power conservation activity. If not, the MG 9000 equipment that is powered down will need to be switched. Manually switch off the following breakers on the upper shelf:
17
18
--End--
Variable denitions
Variable activeCard mainCard spareCard Value Shelf Card/14 Shelf Card/14 Shelf Card/15
Job aid
Example response
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid Pbg/* +=====+--------------------+--------------------------------------| Pbg | working |Response +=====+--------------------+--------------------------------------| 1003|Lp/10 Sonet/3 | | 1100|Lp/11 Sonet/0 |
367
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
368
Procedure steps
Step 1 Action At the USP frame, unseat the front panel and back panel CC cards in Slot 18, beginning with the CC Slot 18 card on each of the extension shelves. Unseat the front panel and back panel CC card in Slot 18 on the Control shelf. Open the RTC system node provisioning and maintenance window. Click on the System Mgmt item on the main menu to open the System Configuration window. Click on the icon for the control CAM shelf to open the shelf_name window. Click on the icon for an RTC system node. An operational-state of enabled indicates that the card is in-service. An activity state of inactive indicates that the associated card is the inactive RTC. 7 8 At the USP frame, ensure the LEDs on the slot 13 and 16 front panel cards are not lit. Unseat the inactive RTC card identified in step 6, its associated SCSI (front panel), and the associated cards on the back panel.
--End--
2 3 4 5 6
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
369
Procedure steps
Step 1 Action At the chassis, each chassis has its own BIP circuit breaker. Set the BIP circuit breaker to the off position to work in simplex mode. Turn one of the three DC power supplies off. This will leave 2 DC power supplies to support the remaining chassis.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
370
Prerequisites
You must have the root user password for the N240. You must have the IP address for the inactive unit.
Procedure steps
Step 1 2 Action At the N240 server, Telnet or SSH into the inactive server and logon as the root user. Type
#init 0
to shutdown the server and to close the Telnet or SSH connection. 3 At this point, the inactive server can be turned off at the circuit breaker.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Prerequisites
371
Prerequisites
Take into consideration the configuration and condition of your switch. Also take into consideration the expected period of the power outage, and the quantity of reserve power available. Continue as follows:
ATTENTION
If you receive a warning message for a loss of service if you busy a plane or unit, do not continue. Clear the problem that can cause a loss of service before you busy the plane or unit. Also you can leave both planes or units of that subsystem in service.
Complete the number of steps of this procedure as your set of conditions needs. For example, if you expect power to restore, you can decide to leave important systems to operate in a duplex mode. The message switch (MS) is an example of a system that can have a duplex mode. Equally for reliability, you can decide to leave both units in service on peripheral modules required for emergency service lines. This procedure requires a condition when this procedure instructs you to busy down one plane or unit of a system. The condition is that the mate plane or unit you leave in service, is fault free and can operate normally. When possible, take the same plane or unit out of service on each subsystem (for example, ENET plane 0, LIM unit 0, MS 0). This action decreases the possible result of error and reduces recovery time.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
372
Procedure steps
CAUTION
Potential service interruption or extended outage
Nortel recommends that you call Emergency Technical Assistance Services (ETAS) of Nortel. Also call your next level of support before you perform this procedure.
CAUTION
Potential loss of service or extended outage
This procedure is only for conservation of emergency backup power. Do not use this procedure or sections for equipment maintenance purposes.
Step 1
Action At your current location, use office records to identify and record the power converters that supply the MAPs and printers for the switch. Turn the power off on all power converters identified in step 1. Do not turn the power off for the power converters which supplies the operators MAP and one printer connected to IOC 0. At the MAP terminal, to confirm that an office image is available to reload the switch if a total turn off of power becomes necessary, type: >AUTODUMP STATUS . For example, see Job aid (page 374).
4 5 6
If an office image is available then go to step 7. If an office image is not available then go to step 6. Record the office image. Perform the procedure, "How to record an XA-Core office image on a disk" in this document. Return to this step when you complete the procedure to record the office image.
ATTENTION
Cards that have an effect on service include digitone receiver cards (NT2X48), centralized automatic message accounting (CAMA) cards (NT2X66), and digital recorded announcement machine (DRAM) cards.
Turn off power for all maintenance trunk modules (MTM) in the office. Do not turn power off if the MTM contains cards that have an effect on service. To turn off power perform the procedure,
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
373
"Emergency shutdown of maintenance trunk modules" in Recovery Procedures () , 297-8021-545. 8 Turn off power for one of the line module controllers (LMC) (NT2X14 shelf) in each double-bay line module (LM) pair in the office. To turn off the power, perform the procedure, "Emergency shutdown of one half of a line module pair" in Recovery Procedures () , 297-8021-545. Turn off power for one unit of all line concentrating modules (LCM) in the office. To turn off the power, perform the procedure, "Emergency shutdown of one unit of LCMs" in Recovery Procedures () , 297-8021-545. Turn off power for one unit of all line group controllers (LGC), line trunk controllers (LTC), and digital trunk controllers (DTC) in the office. To turn off the power, perform the procedure, "Emergency shutdown of one LGC, LTC, and DTC unit" in Recovery Procedures () , 297-8021-545. Turn off power for one plane of all network shelves in the office. For ENET, perform the procedure, "Emergency shutdown of one enhanced network plane" in Recovery Procedures () , 297-8021-545. For JNET, perform the procedure, "Emergency shutdown of one junctored network plane" in Recovery Procedures () , 297-8021-545. If you removed power from a complete network frame in steps 12 or 13, turn off power for the cooling fans for the frame. To turn off power, remove the correct power fuses from the PDC. Busy and power down one local message switch in the link peripheral processor (LPP). Perform the procedure, "Emergency shutdown of one LIM unit on each LPP" in Recovery Procedures () , 297-8021-545. Turn off power for one unit of all CCS7 message switch and buffers (MSB7) in the office. To turn off power, perform the procedure, "Emergency shutdown of one unit of MSB7s" in Recovery Procedures () , 297-8021-545. Turn off power for one message switch (MS) plane. To turn off power, perform the procedure, "Emergency shutdown of one DMS SuperNode MS plane" in Recovery Procedures () , 297-8021-545. If your office has a remote oscillator shelf (NT3X9507), busy the clock related to the MS number. The MS number is for the MS that you turned off power in step 17. To busy the clock, perform
10
11 12
13
14
15
16
17
18
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
374
the procedure, "Emergency shutdown of one remote ascillator shelf plane" in Recovery Procedures () , 297-8021-545.
--End--
Job aid
Example of a MAP response:
Successful Image: 990215_CM Taken: 1999/03/17 21:47:32:04.138 WED. On Volume: F17LIMAGE Successful Image: 990215_MS Taken: 1999/03/17 21:47:32:04:138 WED. On Volume: F17LIMAGE SCHEDULED-Image Dump is ON. Next scheduled dump is MONDAY at 22:30 hours. Next image to be dumped on F02LIMAGE.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
375
Network power up
This chapter describes restarting the network for the Nortel Carrier VoIP solutions. Network power up task flow This task flow shows you the sequence of procedures you perform to power up the network.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
376
Network power up
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
377
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
378
Network power up
Powering up the XA-Core (page 380) Powering up the Message Switch (page 396) Powering up the Enhanced Network (page 399) Powering up an Ethernet Routing Switch 8600 (page 401) Powering up the Link Peripheral Processors (page 402) Powering up the SDM (page 405) Powering up a Netra 240 server (page 406) Powering up the Border Control Point Manager (page 407) Powering up the SAMF frame (page 408) Powering up the Call Control Frame (page 410) Powering up a USP (page 412) Powering up a Media Gateway/Multiservice Switch 15000 (page 414)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
379
Powering up a Media Gateway/Multiservice Switch 7400 (page 415) Powering up the MDM workstation (page 417) Powering up an SPM device (page 418) Powering up the MCS servers (page 420) Powering up a Border Control Point (page 424) Powering up the IPmedia 2000 shelves (page 425) Powering up the SAM16 shelves (page 426) Powering up an MG 9000 device (page 427) Restoring certificates and Pre-shared Keys for IPSec (page 432) Restoring the MG 9000 manager system (page 436) Restoring files from a DVD-RW (page 437) Reloading or restarting a SAM21 Shelf Controller (page 440) Restarting or rebooting a GWC card (page 441) Performing a partial power up of the Netra 240 server (page 443)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
380
Network power up
Procedure steps
CAUTION
Call ETAS or your next level of support
In the event of a dead system, call the Emergency Technical Assistance Services (ETAS) of Nortel. Also call your next level of support before you perform this procedure.
DANGER
Risk of electrocution
Do not touch the cabinet wiring. Connections with unshielded cabinet wiring can result in electric shock. Only qualified power maintenance personnel can perform the voltage measurements in step 3.
Step 1
Action At the PDC, when possible after detection of the power outage, remove all the fuse holders for the line concentrating equipment (LCE) talk batteries, trunk module equipment (TME) talk batteries and PDC filter fuses from the correct PDCs
ATTENTION
The location of the fuse holders in the fuse panel can vary, depending on your office configuration. For help in locating the fuse holders, see the fuse assignment diagram for your office. Also call your next level of support to help you locate the fuses.
2 3 4
If the switch power is -48 V dc then go to step 4. If the switch power is -60 V dc then go to step 8. Continue when you know of restored power at the power plant for your office. Power maintenance personnel must check for restored power at each PDC. At the rear of each PDC, measure the dc voltage across the A feed bus and the battery return plate. Repeat the dc voltage measurement for the B feed bus. Power is correct when the voltage on each feed is -48 V dc.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
381
ATTENTION
Power can be at a nominal potential of -48 V dc. Under conditions that are not normal, the operating voltage can range from -43.75 V dc to -55.8 V dc. A not normal condition is a commercial power failure
5 6 7
If the switch has power retrieval then go to step 11. If the switch has no power retrieval then go to step 10. Continue when you know of restored power at the power plant for your office. Request power maintenance personnel to check for restored power at each PDC. At the rear of each PDC, measure the dc voltage across the A feed bus and the battery return plate. Repeat the dc voltage measurement for the B feed bus. Power is correct when the voltage on each feed is -60 V dc.
ATTENTION
Power can be at a nominal potential of -60 V dc. Under conditions that are not normal, the operating voltage can range from -57.4 V dc to -67.7 V dc. A not normal condition is a commercial power failure.
8 9 10
If power has power retrieval then go to step 11. If power has no power retrieval then go to step 10. At the power room, for help in restoring power to the PDC, call the personnel responsible for maintenance of power at your site. When power restores to the PDC, return to this point. At the PDC, inspect the alarm indication fuses for the XA-Core and the network cabinets. If there are blown fuses, go to step 14. If there are no blown fuses, go to step 22. Replace the blown cartridge fuse in the rear of the fuse holder. Make sure that the amperage of the replacement cartridge fuse matches the amperage marked on the PDC. Remove the blown alarm-indication fuse from the front of the fuse holder. Re-insert the fuse holder, with the alarm-indication fuse removed, into the PDC. Get an alarm-indication fuse for replacement. Insert the alarm-indication fuse that is for replacement into the fuse holder. If the replacement fuse is not successful and blows repeatedly, go to step 21. If the replacement fuse is successful, go to step 22. Call your next level of support for help.
11 12 13 14
15 16 17 18 19 20 21
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
382
Network power up
Continue when you complete replacement of all blown fuses and restored power to the XA-Core and network cabinets. 22 If a second person is available to help in the recovery, continue this procedure with two sets of tasks. Request the second person to restore power from the pdc to the peripheral module frames. The second person restores power by use of step 82 through step 97 of this procedure. While the second person restores power, you recover the core and network by completion of step 23 through step 65. If one person is available, recover the core and network first. If the switch has a remote oscillator shelf, go to step 25. If the switch does not have a remote oscillator shelf, go to step 26. At the remote oscillator shelf, turn on the power converters for the shelf. At the XA-Core cabinet, determine if the switch is a SuperNode switch or a SuperNode SE switch. If the switch is a SuperNode switch then go to step 29. If the switch is a SuperNode SE switch then go to step 32. Turn on circuit breakers for shelf interface module (SIM) A. Turn on the three switches for circuit breakers A1, A2, and A3. The SIM A card is in slot 3R of the XA-Core shelf. Turn on circuit breakers for shelf interface module (SIM) B. Turn on the three switches for circuit breakers B1, B2, and B3. The SIM B card is in slot 16R of the XA-Core shelf. Go to step 33. Turn on circuit breakers for shelf interface module (SIM) A. Turn on the two switches for circuit breakers A1 and A2. The SIM A card is in slot 3R of the XA-Core shelf. Turn on circuit breakers for shelf interface module (SIM) B. Turn on the two switches for circuit breakers B1 and B2. The SIM B card is in slot 16R of the XA-Core shelf. If all the power converters do not have power, go to step 35. If all the power converters have power, go to step 37. To power up the frame perform the procedure, "Clearing an Ext FSP DPCC cabinet major alarm" in Alarm Clearing and Performance Monitoring Procedures, 297-8001-543 (North American market) or 297-9051-543 (International market). When you completed the procedure, return to this point. 36 37 Go to step 27. At the XA-Core reset terminal, monitor the XA-Core reset terminal to determine if the switch has booted.
23 24 25 26 27 28 29
30
31
32
33 34 35
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
383
When the switch boots, the XA-Core reset terminal displays a response to indicate a boot in progress. The response also displays different diagnostic messages and alphanumeric addresses. When the switch has completely booted, an A1 appears on the RTIF display. 38 39 40 If the response has an A1 then go to step 40. If the response has no A1 after approximately 15 min then go to step 180. At the MAP terminal, see the Caution statement.
CAUTION
Extended service interruption
The exact log on procedure can vary, depending on your office configuration. If you need additional help, call the personnel responsible for the next level of support.
ATTENTION
The log on message indicates that you have to manually log on. An automatic log on can occur if the office parameters have automatic log on.
41 42 43
If the log on is not automatic, then go to step 43. If the log on is automatic, then go to step 47. Press the break key. Example of a MAP response
?
44
45
46
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
384
CAUTION
All customers must follow the sequence of steps set out in this procedure
Do not interrupt this procedure at this point to clear an alarm. If a TOD critical alarm appears under the APPL level in the alarm banner, and if the system uses Network Time Protocol (NTP), you must complete all steps in the sequence shown. You will clear the TOD critical alarm by completing step 58. (For information on NTP, see step 58.)
47
48
49 50 51
If the system time is not correct then go to step 51. If the system time is correct, then go to step 53. Enter the correct time (by use of the 24 hour clock).
>SETTIME hh mm
52
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
385
53 54 55 56
Determine if the system date is correct. If the system date is not correct, then go to step 56. If the system date is correct, then go to step 58. Enter the correct date.
>SETDATE dd mm yyyy
57
58
If the system uses Network Time Protocol (NTP) as the timing reference, and if a TOD critical alarm appears under the APPL level in the alarm banner, perform the procedure titled "How to check and adjust the XA-Core TOD" in the Nortel XA-Core Fault and Performance Management ( (297-8991-510)) . By performing that procedure, you will clear the TOD critical alarm. Return to this point when finished.
ATTENTION
In the German market only, switches can use Network Time Protocol (NTP) as the timing reference for the time-of-day clock. The system uses Network Time Protocol if the value of the SNTP_CLIENT office parameter in table OFCENG has been set to Y. For information about the office parameter, see the section titled "XA-Core data schema overview" in the Nortel XA-Core Reference Manual ( (297-8991-810)) .
59
At the MAP terminal, access the NET level of the MAP display.
>NET
386
Network power up
60 61 62 63
Repeat step 60 and step 61 for each JNET shelf. When all JNET shelves recover, continue this procedure. Determine if there are additional input output controller (IOC) and maintenance and administration position (MAP) terminals to recover. If recovery of additional IOCs and MAP terminals is not complete, then go to step 66. If recovery of additional IOCs and MAP terminals is complete, then go to step 79. Restore power to all remaining power inverters in the office. At the IOC, locate the IOC for recovery. If recovery is for an IOC, then go to step 70. If recovery is for an IOM, then go to step 74. Turn on the power converters on the IOC.
64 65 66 67 68 69 70
ATTENTION
The version of IOC determines if the IOC has one or two power converters.
71 72 73 74 75
While you press the reset button on one of the IOC power converters, lift the related circuit breaker to turn on the FSP. Release the reset button. Repeat step 67 through step 72 for each IOC for recovery. Access the input output device (IOD) level of the MAP display.
>IOD
Access the IOC level of the MAP display for the IOC for recovery.
>BSY ioc_no
76 77 78
Repeat step 75 and step 76 for each IOC or IOM for recovery. Log on to additional MAP terminals as required.
ATTENTION
Step 44 through step 46 describe how to log on to the MAP terminal.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
387
79
At the PDC, step 80 through step 104 describe how to restore power from the PDC to Series I and Series II peripheral module frames. If the PDC power to the PM frames does not restore correctly, then go to step 82. If the PDC power to the PM frames restores correctly, then go to step 105. Get a 100-W, 120-V light bulb installed into a socket that has pigtail leads or tool number T000655 (CPC number NTA0600512), that has a fuse holder-style connector instead of pigtail leads for easier insertion
80 81 82
83
WARNING
Possible equipment damage or extended service interruption
Use correct fuses. When you return fuses to the PDC, make sure that the amperage of the fuses is correct. The fuse amperage must match the amperage marked on the PDC.
At the first empty fuse slot in the PDC, connect the leads of the capacitor charging tool. Connect the leads across the contacts for the fuse holder until the lamp decreases brightness. If you use a charging tool with a fuse holder-style connector, insert the connector into the slot. Insert the connector until the lamp decreases brightness. 84 85 86 Remove the capacitor charging tool and immediately insert again the correct fuse holder into the slot. Repeat steps 83 and step 84 for the LCE talk battery, TME talk battery, and PDC filter fuse holders you removed. Determine if any alarm-indicating fuses blew.
ATTENTION
The fuse alarm-indicator lamp lights when an alarm-indicating fuse blows.
87 88 89
If no alarm-indicating fuses have blown, then go to step 89. If any alarm-indicating fuses have blown, then go to step 180. Locate a fuse holder with a blown alarm-indicating fuse.
ATTENTION
You can replace blown fuses in any order.
90
The cartridge fuse in the fuse holder has blown. Remove the fuse holder from the PDC.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
388
Network power up
91
Replace the blown cartridge fuse in the rear of the fuse holder. Make sure that the amperage of the replacement cartridge fuse matches the amperage marked on the PDC. Remove the blown alarm-indicating fuse from the front of the fuse holder. Insert again the fuse holder, with the alarm-indicating fuse removed, into the PDC. Get a replacement alarm-indicating fuse. Insert the replacement alarm-indicating fuse into the fuse holder. Determine if the alarm-indicating fuse blows.
92 93 94 95 96
ATTENTION
The fuse alarm indicator lamp lights when an alarm-indicating fuse blows.
If the alarm-indicating fuse is not blown, then go to step 99. If the alarm-indicating fuse is blown, then go to step 89. Determine if you replaced all the blown alarm-indicating fuses. If you have replaced all the blown alarm-indicating fuses then go to step 102. If you have not replaced all the blown alarm-indicating fuses then go to step 89. Determine if the fuse alarm indicator lamp lit. If the fuse alarm indicator lamp is not lit, then go to step 105. If the fuse alarm indicator lamp is lit , then go to step 180. At the PM frames, select a peripheral module (PM) frame to power up.
ATTENTION
The PM frames can power up in any order.
Locate the frame supervisory panel (FSP) and the power converters on the frame. Determine if the FSP for the frame has fuses or circuit breakers. If the FSP has fuses, then go to step 110. If the FSP has circuit breakers, then go to step 124. Determine if the power converters have Power Reset buttons or Power Reset switches. If the power converters have power Reset buttons, then go to step 113. If the power converters have power Reset switches, then go to step 117.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
389
113
To turn on each power converter press and hold its Power Reset button for 2 s.
ATTENTION
The Converter Fail light goes off when the power converter turns on.
Determine if all the power converters turn on correctly, indicated by all the Converter Fail lights going off. If all the Converter Fail lights are off, then go to step 127. If all the Converter Fail lights are not off, then go to step 130. To turn on each power converter pull out the power switch and toggle it to the Power Reset position.
ATTENTION
The Converter Fail light goes off when the power converter turns on.
Determine if all the power converters turns on correctly, indicated by all the Converter Fail lights are off. If all the Converter Fail lights are off, then go to step 127. If all the Converter Fail lights are not off, then go to step 130. Turn on each power converter by toggle the circuit breaker to the ON position. Press and hold the Power Reset button for 2 s. Release the circuit breaker and the Power Reset button.
ATTENTION
The Converter Fail light goes off when the power converter turns on.
124 125 126 127 128 129 130 131 132 133 134 135
Determine if all the power converters turn on correctly, indicated by all the CONVERTER FAIL lights are off. If all the Converter Fail lights are off, then go to step 127. If all the Converter Fail lights are not off, then go to step 130. Determine if all PM frames turn on. If all the PM frames turn on yes, then go to step 136. If all the PM frames turn on no, then go to step 130. Determine if an attempt was made to power up the remaining PM frames. If power up has not been attempted, then go to step 133. If power up has been tried and failed, then go to step 135. Power up the next PM frame. Go to step 106. To power up the remaining PM frames perform the correct procedures in Alarm Clearing and Performance Monitoring
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
390
Network power up
Procedures () , 297-8001-543 (North American market) or 297-9051-543 (International market). 136 At the MAP terminal, access the SRSTATUS level of the MAP display.
>MAPCI;MTC;SRSTATUS
137
From the MAP display, determine the recovery status of the Series I and II PMs.
ATTENTION
Series I PM recovery status displays to the right of the word "SER1" in the MAP display. Series II PM recovery status displays to the right of the word "SER2" in the MAP display. Recovery status can be one of pending, in progress, complete, or failed.
If the recovery status is not zero, then go to step 140. If the recovery status is zero, then go to step 142. Determine from office records or other office personnel which PMs to manually recover first. To manually recover the PMs in the required order, perform the correct alarm clearing procedures in Alarm Clearing and Performance Monitoring Procedures () , 297-8001-543 (North American market) or 297-9051-543 (International market).
142
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
391
CAUTION
Loss of billing data
Different billing systems than automatic message accounting (AMA) or additional billing system, can be in your office configuration. Call your next level of support to determine if other billing systems are in your office, and if you require recovery action.
Access the device independent recording package (DIRP) level of the MAP.
>IOD;DIRP
143
Determine the state of the recording volumes for the billing system.
>QUERY subsystem ALL
144
If the state of the recording volumes for the billing system has no volumes allocated, as indicated by the word NONE under the state header on the MAP display, then go to step 150. If the state of the recording volumes for the billing system has any volume is IN ERROR, as indicated under the REGULAR VOLUME(S) header on the MAP display, then go to step 147. If the state of the recording volumes for the billing system has all volumes are READY, as indicated under the REGULAR VOLUME(S) header on the MAP display then go to step 151.
145
146
ATTENTION
Different billing systems than automatic message accounting (AMA) or additional billing system, can be in your office configuration. Call your next level of support to determine if other billing systems are in your office, and if you require recovery action.
147
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
392
If the volumes reset passed, then go to step 151. If the volumes reset failed, then go to step 180. Perform the procedure, "Allocating recording volumes in the DIRP utility" in Routine Maintenance Procedures () , 297-8001-546 (North American market) or 297-9051-546 (International market). When you complete the procedure, return to this point. Determine the state of the DLOG recording volumes.
>QUERY DLOG ALL
151
152
If the state of DLOG recording volumes is no allocated volumes, as indicated by the word NONE under the state header on the MAP display, then go to step 158. If the state of DLOG recording volumes is any volume is IN ERROR, as indicated under the REGULAR VOLUME(S) header on the MAP display then go to step 155. If the state of DLOG recording volumes is all volumes are READY, as indicated under the REGULAR VOLUME(S) header on the MAP display then go to step 159.
153
154
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
393
ATTENTION
Different billing systems than automatic message accounting (AMA) or additional billing system, can be in your office configuration. Call your next level of support to determine if other billing systems are in your office, and if you require recovery action.
If the volumes reset passed, then go to If the volumes reset failed, then go to step 180. Perform the procedure, "Allocating recording volumes in the DIRP utility" in Routine Maintenance Procedures () , 297-8001-546 (North American market) or 297-9051-546 (International market). When you complete the procedure, return to this point. Determine from your next level of support if you need to recover other DIRP systems (for example, JF, OM). If you need to recover other DIRP systems, then go to step 162. If you do not need to recover other DIRP systems, then go to step 168. Perform the correct procedures in Alarm Clearing and Performance Monitoring Procedures () , 297-8001-543 (North American market) or 297-9051-543 (International market). When you complete the procedures, return to this point. Determine if DIRP logs generated.
>LOGUTIL;OPEN DIRP
If DIRP log generated a yes, then go to step 166. If DIRP log generated a no, then go to step 168. See the Log Report Reference Manual () , 297-8001-840 (North American market) or 297-9051-840 (International market), and take the correct action. When you complete the log report activities, return to this point. Turn off priority.
>PRIORITY OFF
167 168
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
394
Network power up SRSTATUS 0 Quit OVERALL STATUS Pend: 0% Inprg: 0% Comp: 100% Fail: 0% 2 View_ View: SYSTEM 14:08:30 3 List_ Pend InPrg Comp Fail Pend InPrg Comp Fail 4 MS 0 0 2 0 IOD 5 5 30 2 5 NET 0 0 6 0 Other 21 3 13 3 6 SER1 0 0 41 0 7 SER2 0 0 39 0 8 SER3 0 0 37 0 9 10 MTC: 11 STATUS:
Determine the status of the switch recovery. If the status of the switch recovery is any Series III PMs that failed recovery, then go to step 174. If the status of the switch recovery is any Series I or II PMs that failed recovery, then go to step 176. If the status of the switch recovery is any IODs or other devices and services that failed recovery, then go to step 179. If the status of the switch recovery is that the system has completely recovered, then you have completed the procedure. Manually recover the PMs. For more information, see Recovery Procedures, 297-8001-545 (North American market) or 297-9051-545 (International market). When you complete the procedure, return to this point. Go to step 168. Determine from office records or other office personnel which PMs you can recover first. To manually recover the PMs in the required order perform the correct alarm clearing procedures in Alarm Clearing and Performance Monitoring Procedures () , 297-8001-543 (North American market) or 297-9051-543 (International market). When you complete the procedure, return to this point. Go to step 168. To manually recover IODs and other devices and services, perform the correct procedure in this document. Also you can see your site-related operating procedures.
178 179
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
395
180
For additional help, call the personnel responsible for the next level of support.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
396
Network power up
Procedure steps
Step 1 Action At the XA-Core frame, locate the NT9X31 and NT9X30 power converters for message switch 0 (MS 0) in slots 33F and 36F on the MS 0 shelf. Turn on the NT9X31 and NT9X30 power converters in slots 33F and 36F of the MS 0 shelf at the same time. Lift and release the power switches located on the faceplates of the converters. Locate the NT9X31 and NT9X30 power converters for MS 0 in slots 1F and 4F on the MS 0 shelf. Turn on the NT9X31 and NT9X30 power converters in slots 1F and 4F of the MS 0 shelf at the same time. Lift and release the power switches located on the faceplates of the converters. Locate the NT9X31 and NT9X30 power converters for message switch 1 (MS 1) in slots 33F and 36F on the MS 1 shelf. Turn on the NT9X31 and NT9X30 power converters in slots 33F and 36F of the MS 1 shelf at the same time. Lift and release the power switches located on the faceplates of the converters. Locate the NT9X31 and NT9X30 power converters for MS 1 in slots 1F and 4F on the MS 01shelf. Turn on the NT9X31 and NT9X30 power converters in slots 1F and 4F of the MS 1 shelf at the same time. Lift and release the power switches located on the faceplates of the converters. At the MAP terminal, access the SRSTATUS level of the MAP.
>MAPCI;MTC;SRSTATUS
3 4
5 6
7 8
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps SRSTATUS 0 Quit OVERALL STATUS Pend: 0% Inprg: 0% Comp: 100% Fail: 0% 2 View_ View: SYSTEM 14:08:30 3 List_ Pend InPrg Comp Fail Pend InPrg Comp Fail 4 MS 0 0 2 0 IOD 5 5 30 2 5 NET 0 0 6 0 Other 21 3 13 3 6 SER1 0 41 0 0 7 SER2 0 39 0 0 8 SER3 0 37 0 0 9 10 MTC: 11 STATUS:
397
10
ATTENTION
MS recovery status displays to the right of the word "MS" in the MAP display. Recovery status for each MS can be one of pending, in progress, complete, or failed.
11 12 13 14
If the recovery status is MS failed recovery then go to step 20. If the recovery status is MS continues to have pending recovery then go to step 14 If the recovery status is a status other than failed or pending recovery then go to step 15 Wait until both MSs are either in recovery development or have completed recovery. If neither MS indicates pending recovery, go to step 10. Access the MS Clock level of the MAP display.
>MAPCI;MTC;MS;CLOCK
15 16 17 18 19
If the SYNC command is successful then go this procedure is complete If the SYNC command is failed then continue. Record the reason for synchronization failure, as shown in the MAP response. Repeat the try to synchronize the MS clocks later, after networks and PMs are in service.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
398
Network power up
20
For additional help, call the personnel responsible for the next level of support.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
399
Procedure steps
Step 1 2 3 4 5 6 7 8 Action At the ENET frames, locate the NT9X31 power converters in slots 1F and 33F on the ENET shelves. To turn on the NT9X31 power converters lift and release the power switches located on the faceplates of the converters. Locate the NT9X30 power converters in slots 4F and 36F on the ENET shelves. Turn on the NT9X30 power converters lift and release the power switches located on the faceplates of the converters. Determine if all the converters have power. Power is indicated by all the Converter Off lights going off. If all the power converters have not power then go to step 8. If all the power converters have power then go to step 9. Power up the ENET frame. Perform procedure, "Clearing an Ext FSP DPCC cabinet major alarm" in the document, Alarm and Performance Monitoring Procedures () , 297-8021-543. When you complete the procedure, return to this point. 9 At the MAP terminal, access the SRSTATUS level of the MAP display.:
>MAPCI;MTC;SRSTATUS
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
400
Network power up SRSTATUS 0 Quit OVERALL STATUS Pend: 0% Inprg: 0% Comp: 100% Fail: 0% 2 View_ View: SYSTEM 14:08:30 3 List_ Pend InPrg Comp Fail Pend InPrg Comp Fail 4 MS 0 0 2 0 IOD 5 5 30 2 5 NET 0 0 6 0 Other 21 3 13 3 6 SER1 0 41 0 0 7 SER2 0 39 0 0 8 SER3 0 37 0 0 9 10 MTC: 11 STATUS:
10
From the MAP display, determine the recovery status of the network.
ATTENTION
Network recovery status displays to the right of the word "NET" in the MAP display. Recovery status can be one of pending, in progress, complete, or failed.
11 12 13 14 15 16 17
If the status of any network element is failed then go to step 16 If the status of any network element is pending then go to step 14 If the status of any network element is any other status then this procedure is complete. Continue when there are no network elements that continue to be pending recovery. Go to step 10. Manually recover the ENET. Perform procedure "Recovering the Enhanced Network" in Recovery Procedures () , 297-8021-545. Go to step 9.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
401
Procedure steps
Step 1 Action At the Ethernet Routing Switch 8600 frame, turn the DC input power source on or reset the power source circuit breaker to provide power to the power supply. Turn the power supply switch for all three DC power supplies to the on position.
ATTENTION
You must turn on two of the power supply units within 2 seconds of each other. If you wait longer to turn on the second power supply, one of the power supplies could shut down. To correct this condition, turn off both power supplies, wait at least 30 seconds, and then turn on both power supplies again within 2 seconds.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
402
Network power up
ATTENTION
Throughout this procedure, LPP is used to refer to both the LPP and FLPP.
Procedure steps
Step 1 2 3 4 5 Action If PDC has not power restored to the LPPs then go to step 3 If PDC has power restored to the LPPs then go to step 7. If there are blown PDC fuses supplying the LPP,then go to step 5. If there are novisible blown fuses then go to step 6. Replace the blown fuses.
ATTENTION
If fuses blow repeatedly, call your next level of support for help.
When PDC power restores to the LPPs, continue this procedure at step 7. 6 Call the personnel responsible for maintaining power at your site, or see your next level of support for help. When PDC power restores to the LPPs, continue this procedure. 7 8 9 10 11 Locate the LPPs for recovery. At the LPP cabinet, release the locking levers and gently unseat the NT9X74 cards on the link interface shelves(LIS). Refer to Toggle the switch on each NT9X30/NTDX16 power converter cards on the link interface module (LIM) unit shelf. Re-toggle the switch on each NT9X30/NTDX16 power converter cards on the link interface module (LIM) unit shelf. If all of the cards did not turn on the power correctly, go to step 13. All of the CONVERTER OFF lights go off when the power turns on correctly. 12 If all of the cards turned on the power correctly, go to step 15. All of the CONVERTER OFF lights go off when the power turns on correctly.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
403
13
To power up the frame, perform the procedure "Clearing and Ext FSP LPP cabinet major alarm" in Alarm Clearing and Performance Monitoring Procedures (297-8991-543) for North American markets or (297-9051-543) for International markets. Go to step 8. Slide each of the NT9X74 cards back into its slot. Gently push on the upper and lower edges of the faceplate of each card until the card is locked into the shelf. Close the locking levers on each card. Repeat steps 5 through 17 for each LPP in your office. Continue this procedure when power has been restored to all LPPs. Access the SRSTATUS of the MAP display>MAPCI;MTC;SRS TATUS
SRSTATUS 0 Quit OVERALL STATUS Pend: 0% Inprg: 0% Comp: 100% Fail: 0% 2 View_ View: SYSTEM 14:08:30 3 List_ Pend InPrg Comp Fail Pend InPrg Comp Fail 4 MS 0 0 2 0 IOD 5 5 30 2 5 NET 0 0 6 0 Other 21 3 13 3 6 SER1 0 41 0 0 7 SER2 0 39 0 0 8 SER3 0 37 0 0 9 10 MTC: 11 STATUS:
14 15 16 17 18 19
20
If the recovery status of the Series III PMs is not zero, go to step 22.
ATTENTION
Series III PM recovery status displays to the right of the word "SER3" (line 8) in the MAP display. Recovery status can be pending, in progress, complete or failed.
21 22
If the recovery status of the Series III PMs is zero, this procedure is complete. To manually recover the PMs, perform the procedure "Recovering Link Peripheral Processors" in Recovery Procedures, (297-8021-545).
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
404
Network power up
Job aid
Card NT9X74 NT9X30 NTDX16 Slot position 7F, 32F 4F, 36F 1F, 4F, 33F, 36F
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
405
Procedure steps
Step 1 2 3 4 5 Action At the SDM frame, turn on the two top breakers at the top of the SDM frame. Wait a few moments until the green lights on each card are steady. At the VT100 terminal, log on as root and type
# sdmmtc
Verify that all hardware is in service. Verify that rootvg and datavg logical volumes are mirrored at the storage level.
ATTENTION
There will likely be alarms at the connection and application levels. The general status of the SDM will be in-service trouble (IsTb).
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
406
Network power up
IEMS Core and Billing Manager Communication Server 2000 Management Tools applications MG 9000 Manager
Procedure steps
Step 1 2 Action At the SDM frame, turn on the circuit breakers to bring the duplex N240 servers into service. To help with recovery, turn the power on to what was the active server first. If there is no record of this, turn on the Unit 0 server before the Unit 1 server. After power is restored, the servers recover on their own.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
407
Procedure steps
Step 1 Action At the frame housing the Border Control Point Manager, for DC powered shelves, switch the circuit breakers at the EBIP that provide power to the servers being powered to the ON position. The breaker will have the | side (top) of the breaker depressed.
ATTENTION
The DC powered N240 does not have a Power ON/OFF switch and can only be powered ON/OFF from the breaker at the EBIP.
For AC powered shelves, push the rocker located on the top of the AC outlet to apply power to the servers. It should be switched to the "|" position. After the breakers are switched ON at the power source, power up the servers by switching the breakers on the back of the chassis to the ON position. Verify the system is powered up correctly by observing the LEDs on the front of the N240.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
408
Network power up
Procedure steps
Step 1 Action At the SAMF frame, power up the SAMF BIP breakers. At the top of the cabinet, turn on the breakers that supply power to the two SAM21 shelves and the Session Server (if equipped). At the Communication Server 2000 SAM21 Manager, if necessary, unlock the previously active Shelf Controller (SC). See Unlocking a SAM21 Shelf Controller (page 353). 3 4 5 If necessary, unlock the previously inactive SC. See Unlocking a SAM21 Shelf Controller (page 353). If necessary, unlock any remaining locked cards on the SAM21 other than the GWC cards. At the Communication Server 2000 GWC Manager, if necessary, unlock all previously active GWC cards. See Unlocking a GWC card (page 457). 6 7 If necessary, unlock all previously inactive GWC cards. See Unlocking a GWC card (page 457). At the Policy Controller unit, determine which physical or logical unit you want to become active and power that unit up first by executing procedure Power-On and boot a Policy Controller unit.. After the active unit has begun call processing, power up the mate unit by executing procedure Power-On and boot a Policy Controller unit.. At the Session Server unit, determine which physical or logical unit you want to become active and power that unit up first by executing procedure Power-On and boot a Session Server unit.. Log onto the active unit to monitor the status of the SIP Gateway application using procedure View the operational status of the SIP Gateway application.. Verify that the Administrative state of the SIP Gateway application becomes Unlocked and the Operational state becomes Enabled. If the active unit comes up in any state other than Unlocked:Enabled, see Nortel Session Server Trunks Administration and Security ( (NN10346-611)) ,
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
10
Procedure steps
409
11
Complete one or both of the following procedures:Unsuspend the SIP gateway application OR Unlock the SIP gateway application
12
After the active unit has begun call processing, power up the mate unit by executing procedure Power-On and boot a Session Server unit.. From the active unit, verify that the SIP Gateway application databases on the both units have synchronized using procedure Verify synchronization status of Session Server units. Monitor the system for an appropriate period according to your site guidelines. If you experience problems with call processing, see Nortel Session Server Trunks Administration and Security ( (NN10346-611)) ,
13
14
15
Perform the following procedures in order: Invoke a maintenance SwAct of the Session Server platform AND Inhibit a system SwAct (Jam)
16
Contact your next level of support or Nortel GNPS for further assistance.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
410
Network power up
Procedure steps
Step 1 2 3 4 Action At the CCF, determine the configuration of your CCF. If the CCF configuration includes a STORM disk array and two STORM cards in the Call Agent shelf then go to step 4. If the CCF configuration includes two STORM SAM-XTS units then go to step 5. Power up the CCF BIP breakers. At the top of the cabinet, turn on the breakers that supply power to the STORM disk array and the two SAM21 shelves. Go to step 8. 5 Power up the CCF BIP breakers. At the top of the cabinet, turn on the breakers that supply power to the two SAM21 shelves and the STORM SAM-XTS units. At the front panel of the STORM SAM-XTS units, if necessary, power on the STORM SAM-XTS units using the main power switch located on the front panel.
7 8
At the console, monitor the boot progress of the unit until booting is complete. At the Communication Server 2000 SAM21 Manager, if necessary, unlock the previously active Shelf Controller (SC). See Unlocking a SAM21 Shelf Controller (page 353).
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
411
9 10
If necessary, unlock the previously inactive SC. See Unlocking a SAM21 Shelf Controller (page 353). If necessary, unlock the previously active Compact Call Agent (CCA) card. See Unlocking the Call Agent (page 456). If necessary, unlock the previously inactive CCA card. See Unlocking the Call Agent (page 456). If necessary, unlock the previously active USP-lite card. If necessary, unlock the previously inactive USP-lite card. If necessary, unlock any remaining locked cards on the SAM21 other than the GWC cards. At the Communication Server 2000 GWC Manager, if necessary, unlock all previously active GWC cards. See Unlocking a GWC card (page 457). If necessary, unlock all previously inactive GWC cards. See Unlocking a GWC card (page 457).
--End--
11 12 13 14 15
16
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
412
Network power up
Powering up a USP
Use this procedure to power up a Universal Signaling Processor (USP). The shelf uses the USP RTC-12 to initially get back into service.
Prerequisites
The Communication Server 2000 Core and USP ABS server must be in-service.
Procedure steps
Step 1 Action At the USP shelf, ensure all cards are seated in the USP shelf with the exception of the RTC-15 (slot 15) front panel card on the Control shelf. Power up both A/B buttons on the control and extension shelves. RTC-12 should SCSI boot and it should come into service within 1-2 minutes of being powered up. At the USP Manager, after RTC-12 is up, use the USP Manager to uninhibit and activate all links, where applicable. See Configuring Links. Verify logs and alarms to ensure no unexpected problems and perform test calls. Verify that all cards on all shelves are up and running with the exception of RTC-15. Ensure that the System Date/Time is set to the current local time, if not using SNTP. See Administration: Date and Time. At the USP frame, after all cards are up and properly running, reseat RTC-15. RTC-15 should SCSI boot and perform a database synchronization with RTC-12. RTC-15 should come into service in approximately 5-7 minutes. At the USP Manager, verify that RTC-15 is in-service and inactive. Verify logs and alarms again to ensure no unexpected or undesired events and perform test calls. Verify that the USP can communicate with the ABS. From the USP Manager main menu, click on the Administration button and select ABS Settings. Select the Test button to initiate a series of tests to ensure the USP can properly communicate with the ABS.
2 3 4
5 6 7 8
9 10 11
12
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
413
13
Check the USP logs for the test results. Troubleshoot ABS connectivity in the event that the tests fail.
ATTENTION
To assist with troubleshooting, the Logs message includes the nature of the test failure.
14 15
Take a snapshot of the USP, and copy the snapshot to the ABS using the File Manager. Using the ABS CM GUI, modify the site information to point to the new snapshot.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
414
Network power up
Procedure steps
Step 1 Action At the chassis, locate the breaker interface panels (BIP). All the BIPs are at the top of the chassis.
ATTENTION
If the CP card or its disk becomes unserviceable, the backup-up provisioning data and software are downloaded from the MDM backup copy and SDS site, respectively.
Switch the breakers on all the BIPs to the ON position. The system should come up on its own.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
415
Prerequisites
Procedure steps
Step 1 2 3 Action At the chassis, disengage all processor cards from the backplane of the shelf. Engage the minimum number of processor cards required, based on the number of power supplies in the shelf. If you have one power supply, engage a CP in slot 0 and at least one FP. Do not engage more than seven FPs. Ensure that all other processors are disengaged. 4 5 6 7 If you have two power supplies, engage at least one CP and two FPs. Or, engage two CPs and at least one FP. Turn on the circuit breakers for the outlets that supply power to your switch. Verify that the LEDs on all power supplies are red. Apply power to one of the power supplies by setting its power switch to the on position. Verify that the power supply LED is green. Normal operation is not guaranteed if there are no processor cards in the shelf. If you have not installed any processor cards before you switch on a power supply, the system does not supply power to the shelf and the power supply LED remains red. 8 9 Verify that the cooling unit LED is green and is operational. You should be able to hear the fans start to rotate. If necessary, apply power to a second and third power supply by setting their power switches to the on position. Verify that the LED for each power supply is green.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
416
Network power up
If you install eight or more FPs, you must use a second power supply. If your system contains fewer than eight FPs, the system uses the second power supply for redundancy. 10 11 12 If necessary, engage the remaining processor cards. Verify the LED color on each FP in appropriate. Verify that the appropriate LEDs are illuminated on termination panels.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
417
ATTENTION
If your MDM is an N240 server, see the procedure "Powering up a Netra 240 server."
Prerequisites
Procedure steps
Step 1 2 Action At the workstation, switch the workstation on. After the workstation has started, log on with an MDM user account and ensure that the MDM toolset is available.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
418
Network power up
Procedure steps
Step 1 2 3 4 5 6 Action At the MAP terminal, restore power to the frame and re-insert the fuses. Reseat the SIM cards if they were unseated and turn them on one at a time using the switches on the front. Reseat any unseated cards in the shelves. Post the SPM/MG 4000/IW-SPM/DPT-SPM.
>MAPCI;MTC;PM;POST SPM <spm_no>
Select and RTS the SRM, OC3 RMs, ATM RMs, and/or GEM RMs.
>QUIT;SELECT <rm_type> ALL;BSY ALL;LOADMOD ALL;RTS ALL
ATTENTION
To RTS the ATM, use the BSY FORCE and RTS FORCE commands.
Post and recover the trunks on the SPM at the TTP level. (Press the Enter key after each of the following commands.)
>QUIT ALL;MAPCI;MTC;TRKS;TTP;PRADCH;POST AD SPM <spm_no>;BSY ALL;RTS ALL >QUIT ALL;MAPCI;MTC;TRKS;TTP;POST D SPM <spm_no>;BSY ALL;RTS ALL
9 10
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
419
Variable denitions
Variable <spm_no> <rm_type> Value The number of the SPM/MG 4000/IW-SPM/D PT-SPM. This is either SRM, OC3, ATM, or GEM.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
420
Network power up
Procedure steps
Step 1 Action
ATTENTION
The following shows the preferred order, which is not the way the system comes up if all servers are powered on at the same time.
At the frame housing the MCS servers, power on the database server by turning on the circuit breakers that provide power to the servers. It takes from 5 to 8 minutes for the server to boot and load Oracle. 2 3 After the DBSvr powers on, telnet to the box and run a spot check to make sure it is ready. Make sure Oracle is loaded. The best way is to telnet to the database. Type: sqlplus and fill in the username and password (in other words, log on as user, not root). If Oracle is running, you should receive the SQL> command prompt. Type quit to exit from sqlplus. Verify that the snmp service is running on the database box. Type: ps -ef | grep snmp. You should see three snmp processes:
/usr/local/sbin/snmpd -f udp:161 /bin/sh /opt/app/oracle/product/9.2.0/bin/dbsnmpwd /opt/app/oracle/product/9.2.0/bin/dbsnmp
The snmp processes are not critical to getting the system up. However, if any of the snmp processes are not running, you will not be able to see accurate reporting of this box on the System Management Console. 5 To stop or start oracle, telnet to the database as sysadmin and then su to root. Go to the following directory: /etc/init.d. The stop command is ./dbora stop. The start command is ./dbora start. These commands will severely affect service. Verify that the SysMgr processes on the MgmtSvr/AcctMgr box are running to control the MCS components from the System
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
421
Management Console. Telnet to the MgmtSvr/AcctMgr box as nortel. 7 Make sure that the SysMgr processes are running by typingmeinit -p. You should see three processes running. If any of these processes are not running, you will not be able to launch the System Management Console:
Rel2.0 NTme_pids mgmtsvr mgmtsvr.3 Rel2.0 NTme_pids tsscma tsscma.5 Rel2.0 NTme_pids tssfpma tssfpma.6
To stop or start the SysMgr, go to the following directory: IMS/m gmtsvr/bin/mgmtsvr/ and type: ./MgmtSvrShutdown.pl. This will kill any (or all) of the SysMgr processes. To start the SysMgr, stay in the same directory and type: MgmtSvrConfigSetup.pl. This will try to start all three processes. You will see a messages on the telnet session that say something like "starting tsscma waiting 30 seconds." The tssfpma process will start next, followed by the mgmtsvr processes. If you want to watch the SysMgr startup logs, after performing step 3b above, go to the following directory: /var/Rel2.0/mgmtsvr. Then type: tail -f mgmtsvr.3.log. Verify that the snmp process is running on the MgmtSvr/AcctMgr box by typing ps -ef | grep snmp. You should see one process running: /usr/local/sbin/snmpd -f udp:161. If this process is not running or you get an alarm from the System Management Console that there is a problem with the MgmtSvr/AcctMgrs snmp, then kill the process. The process will automatically restart in about 30 seconds.
10
11
12
Restart the AcctMgr process. If any of the components send an alarm in the System Management Console indicating that the component cannot communicate with the primary or backup CAM, then again restart the AcctMgr process. Right click on the Accounting component in the System Management Console and select Restart. Confirm the request. A progress box will pop up and then disappear when the restart has begun. While the Accounting module is rebooting, the other MCS component will throw alarms to the System Management Console, but those will go away when the Acct component is fully operational. Now verify the state of the other machines and the MCS components on them. Launch the System Management Console as you normally would. Expand the navigation tree fully so that each component is fully exposed.
13
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
422
Network power up
14
Restart the AppSvr. If the clients are having trouble communicating with the AppSvr, then restart the AppSvr. Right click on the AppSvr component in the System Management Console and select Restart. Confirm the request. You should see a progress window and it disappears after the restart has begun. To view AppSvr startup logs, telnet to the AppSvr box (we recommend that you telnet as nortel) and go to the following directory: /var/Rel2.0/appsvr. Type: tail -f appsvr.0.log. Verify that the AppSvrs snmp process is running. This step is the same as step 3e above. Restart the IPCM. If the i2004 phones are not responding to client registers, the hollow blinking icons in the i2004 display, then restart the IPCM component. Right click on the IPCM component in the System Management Console and select Restart. Confirm the request. You see a progress window that will disappear after the restart has begun. To view IPCM startup logs, telnet to the IPCM box (recommend as nortel) and go to the following directory: /var/Rel2.0/esm. Type: tail -f esm.1.log. Verify that the IPCMs snmp process is running. This step is the same as step 3e above. Most likely, the Provisioning Module and the WebClient Module are deployed on the same box as the IPCM. Therefore, you only need to verify the snmp service on the box once. Restart the Provisioning component. The best way to see if the Provisioning Module is behaving properly is to log on to the ProvClient. Launch a web browser to the IP address of the box where the Provisioning Module is deployed, such as http:192.168.0.10/prov. Log on as admin/admin and attempt any of the List options in the navigation tree, for example, List Devices or List Users. If the browser doesnt respond properly, then restart the Provisioning Module by right clicking on the Provisioning component in the System Management Console and select Restart. Confirm the request. Restart the WebClientMgr component. Log on to the Personal Agent through your browser, such as http://192.168.0.10/pa. Log on as a subscriber, such as myusername@mydomain.com, and enter the subscribers password. The Personal Agent should appear for that subscriber. If it does not, or the interface does not seem to respond correctly, restart the Provisioning Module described in step 4e above. (If youve already restarted it once and the ProvClient interface seems ok, but the Personal Agent interface is still not working correctly, it is probably a configuration issue.
15 16
17
18
19
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
423
20
Launch the WebClient GUI from inside a subscribers Personal Agent page. Attempt to log on to the WebClient when the System Management Console completes loading.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
424
Network power up
Procedure steps
At the Ethernet Routing Switch 8600 Workstation
Step 1
Action At the frame housing the Border Control Point, for DC powered shelves, switch the circuit breakers at the EBIP that provide power to the shelf being powered to the ON position. The breaker will have the 1 side (top) of the breaker depressed. For AC powered shelves, push the rocker located on the top of the AC outlet to apply power to the chassis. It should be switched to the "|" position. After the breaker is switched ON at the power source, power up the shelf by switching the breaker on the back of the chassis to the ON position. Observe the System, Power/Fan, Slot (filled) and Host Card LEDs as the unit is being powered. After power on is complete, they should all have a steady green color.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
425
Procedure steps
Step 1 2 3 Action At the CCF or SAMF frame, open the front door of the CCF/SAMF frame. At the EBIP at the top of the frame, turn the breaker associated the Media Server to the ON position. Ensure that the Green Power LED on the front of the Nortel Media Server 2000 is on (right hand side of the card). After 1 to 2 minutes, the Green LED on the left hand side of the card lights. 4 Verify the status of the LEDs on the front of the Nortel Media Server 2000 . 5 6 Red Fail LED is off (left hand side of the board). ACT LED is on (left hand side of the board).
ATTENTION
It may take up to 60 seconds for the ACT LED to light up.
7 8 9
Blue LED does not remain on (right hand side of the board). PWR LED is on (right hand side of the board). Both Ethernet Link LEDs are on (middle of the board).
ATTENTION
The Ethernet LEDs are Green when indicating a connection, and Orange when indicating activity.
ATTENTION
Check the Media Server 2000 to make sure it is set to an unlocked state.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
426
Network power up
Procedure steps
Step 1 2 Action At the CCF or SAMF frame, open the front door of the CCF/SAMF frame. At the EBIP at the top of the frame, turn the breakers associated the SAM16 to the ON position.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
427
Procedure steps
Step 1 Action At the Subnet View of the MG 9000 Manager, select the MG 9000 to be re-commissioned. A warning window indicating that your connection to the gateway is down may appear. This will not block the saving of services information. Save a copy of the MG 9000 Manager SLoA provisioning data as follows (youll be re-entering this data later): To Save the SLoA services provisioning data, at the Frame (Element) view, select the "Save SLOA services" option from the Actions menu.
2 3
An acknowledgment window appears in response to this request. The acknowledgment window contains the name of the two files (with .html and .text suffixes) where your data had been stored on the MG 9000 Manager. 4 Record the file name provided in the Save acknowledgement window for later use.
Save a copy of the MG 9000 Manager PLoA provisioning data as follows (youll be re-entering this data later):
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
428
Network power up
To save the PLoA services provisioning data, at the Frame (Element) view, select the "Save PLOA services" option from the Actions menu. An acknowledgment window appears in response to this request. The acknowledgment window contains the name of the files where your data had been stored on the MG 9000 Manager.
Record the file name provided in the Save acknowledgement window for later use.
Obtain the IP address of the MG 9000 Manager server (not the mid-tier or client). At the Subnet View, select Configuration->View/Modify NE Properties from the menu bar.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
429
10
FTP to the address recorded in step 5 and copy the files in the /tmp directory to a secure location, such as your desktop or permanent server before proceeding.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
430
Network power up
11
At the MG 9000 frame, power up the shelves in the frame by reinstalling the fuse modules on the IBIP. Restore all fuses left to right in the frame. Connect a laptop PC to the active DCC card and launch the local craft interface (LCI). Wait for the restart to complete and for all cards initialize, typically takes 5 minutes. The Administrative State of all common cards will be Locked. From the LCI, Unlock the active DCC OC-3/DS1 IMA card by selecting the Admin. State Unlock radio button. Wait for the status at the LCI to change to Enabled, Online, and Unlocked. From the LCI continue to unlock the common cards in the following order until all are unlocked and online:
12 13
14
15
active ITX active ITP OC-3 carriers on the active OC-3 DCC card inactive DCC inactive ITX inactive ITP OC-3 carriers on the inactive OC-3 DCC card
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
431
16 17 18
Initiate rediscovery by sending a cold start trap from the LCI. At the MG 9000 Manager, monitor the MG 9000 network element and wait until the discovery process is complete. Wait for the network element to show it is operational, then check for dial tone.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
432
Network power up
Prerequisites
Perform regular full system backups and critical data backups according to corporate policy. This measure ensures that the current backup version is restored, and that a disaster recovery process is executed quickly without additional outages. Ensure that the next level of support is available for assistance during the recovery process. Have immediate access to the Certificate Manager. Restore the Oracle data on the Communication Server 2000 Management Tools. Also perform this restore procedure for all other SPFS-based servers, including the MDM and MG 9000 Manager. For more information, see Nortel Carrier Voice over IP Disaster Recovery Procedures ((NN10450-900)) .
Procedure steps
Step 1 2 3 4 Action Reinstall the WEBPKPROXY certificate for the Certificate Manager. Launch the Certificate Manager GUI. Unmanage and delete all WEBPKPROXY certificates. Clean up directory /data/NTPkmgr/cert/export on the device on which the Certificate Manager resides by deleting all files in this directory.
>rm -f /data/NTPkmgr/cert/export/*
Do not delete the directory. Delete only the files within it. 5 Recreate and re-install all WEBPKPROXY certificates. This step ensures that directory /data/pkclient/certificates is cleaned up on each management system. In addition, this step re-enables
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
433
the Communication Server 2000 Management Tools server the MG 9000 Manager system the MDM
Check that call processing traffic between each Gateway Controller (GWC) and the MG 9000 is not affected. Ensure that the connection between the GWC and the media gateways is still enabled.
CAUTION
Check the list of all certificates in the Certificate Manager. Ensure that each MG 9000 displays the same certificate serial number as the ones listed in the Certificate Manager. Do not rely on the displayed information in the VMG, as communication between the manager and the media gateways can be compromised. An outage can occur if the Certificate Authority serial numbers between the GWC and the media gateways do not match. Contact your next level of support to resolve this conflict.
See procedure Viewing certificate using Certificate Manager in NTP NN10453-100, Nortel Carrier VoIP Security Service Implementation Guide, for details. 8 Contact your next level of support within the next off-peak maintenance window to assist you with performing the following tasks:
to download the certificates to the network element to busy and return to service the inactive GWC node to perform a warm SWACT to revoke the certificate for the MG 9000 VMG or the MG 15000 that this GWC control(s) to busy and return to service the newly active side to clear any GWC307 alarms manually download all certificates on the MG 9000 VMG
During the next available maintenance window, ensure that all Gateway Controller alarms and all network gateway alarms are cleared. Launch the MG 9000 Manager web-based GUI.
10
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
434
Network power up
11
From the Subnet View, select Configuration > View/Modify NE Properties The NE Properties View displays.
12 13
Display the Properties menu of each MG 9000 node to verify that the Push and Pull Certificates match. Click on the NE Security tab. The NE Security View displays.
14
Compare the serial numbers in fields Last Pulled Certificate and Last Pushed Certificates against their Certificate Manager counterparts. If the serial numbers differ, click the Download Certificates button to match the pushed and pulled certificates. Then click the Refresh button.
15
From the menu bar, select Services > Switched Lines Services Manager. The Switched Lines Services Manager displays.
16 17
Select the VMG for which security is to be provisioned. Click the GW Security Config tab for each GWC card with a certificate. The GW Security Config screen displays.
18
Compare the serial numbers in fields Last Pulled Certificate and Last Pushed Certificates against their Certificate Manager counterparts. If the serial numbers differ, click the Download Certificates button to match the pushed and pulled certificates. Then click the Refresh button.
19 20
Verify that the call processing traffic between each GWC and media gateway has not been affected. Perform a full system backup on all management systems and network gateways.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Job aid
435
Job aid
This procedure applies to IEMS, Gateway Controller (GWC), Communication Server 2000 Management Tools, GWC Manager, MDM, MG 15000 and MG 9000 Manager in CVoIP IP Disaster Recovery and IP Security applications. Therefore, see the appropriate NTPs when directed as follows:
Nortel CVoIP IPSec Service Configuration ( (NN10453-100)) for use with WEBPKPROXY certificates for the Certificate Manager Nortel Carrier VoIP Disaster Recovery Procedures ((NN10450-900)) for details and procedures for scheduled system backups. (See Prerequistes.) If Pre-shared Keys are used for authenticating call processing traffic using IPSec, verify that the following conditions exists after a system recovery:
Nortel Carrier Voice over IP Fault Management Logs Reference Volume 2 ( (NN10275-909v2)) for details on alarm CMT301 Nortel Gateway Controller Fault Management ( (NN10202-911)) for details on alarm GWC307
When backing up devices that use certificates provided by the IEMS/Certificate Manager and/or Pre-shared Keys for IPSec, always note the versions of the data in the back-up image of the device. Due to the timing sensitivity of the certificates, a potential conflict can occur during restore between the version of the certificate in the back-up image versus the version of the certificate provided by the Certificate Manager. If the restored data has expired or is not current, call processing traffic that is secured and/or authenticated using IPSec can be affected. This condition can result in additional outages.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
436
Network power up
Procedure steps
Step 1 Action Unmanage and delete the WEBPKPROXY certificates for the MG 9000 manager system. Performing this step disables all communication between the Certificate Manager and the MG 9000 manager system. See Nortel CVoIP IPSec Service Configuration ( (NN10453-100)) for details. 2 Recreate and re-install the WEBPKPROXY certificates for the MG 9000 manager system only, generate the WEBPKPROXY certificate using the IP address and FQDN of the MG 9000 manager system. This step is required for cleaning up the /data/pkclient/certificates directory on each management system. See Nortel CVoIP IPSec Service Configuration ( (NN10453-100)) for details. 3 Install the WEBPKPROXY certificate for the MG 9000 manager system only. See Nortel CVoIP IPSec Service Configuration ( (NN10453-100)) for details. 4 Register the WEBPKPROXY certificate for the MG 9000 manager system only. This step is required to re-enable communication between the Certificate Manager and the MG 9000 Manager System. See Nortel CVoIP IPSec Service Configuration ( (NN10453-100)) for details. 5 Revoke all MG 9000 NE certificates and verify delivery of all MG 9000 certificates. See Nortel CVoIP IPSec Service Configuration ( (NN10453-100)) for details. 6 Perform a full system backup on all management systems and network gateways
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
437
Procedure steps
Step 1 Action At the Communication server Management Tools server insert the DVD-RW into the DVD tray. If the Communication server Management Tools server is a pair of Sun Microsystems Netra 240s, put the DVD in the unit with the USER LED lit.
Volume management software on the Communication server Management Tools server operating system mounts the DVD-RW.
3 4
At the MAP enter the DISKADM level and busy the volumes to copy:
> DISKADM <sd0x>; BSY <volname> > QUIT
The BSY command fails if applications have open files on any volume for the device. A busied disk may affect data recording or retrieval for applications. Consider to RTS the volume as soon as possible or convenient after copying the files. If applications are active and writing to the volume, determine if the application can ROTATE the disk writing activity to a backup volume. If unsure, contact your next level of support. 5 At a Communication server Management Tools server terminal secure copy the files from the DVD-RW on the Communication
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
438
Network power up
ATTENTION
The first time this command is issued, the secure copy program provides a prompt to exchange keys. Confirm the exchange with a "y." The secure copy program provides a progress indicator during the copy. Wait for the copy to complete and the $ prompt to return.
At the MAP enter the DISKADM level and RTS the volume:
> DISKADM <sd0x>; RTS <volname> > QUIT
List the contents of the volume so the file can be added to the Image Table of Contents (ITOC) in the next step:
> SCANF SD00IMAGE1 The contents of the volume are printed.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
439
ATTENTION
To determine if a free position is available, use the LBF CM command. If all positions are used, clear the file in position 15. Determine which volume the position 15 is on, use SCANF to list that volume, and then use the CBF CM FILE <image_name> command.
10
At a Communication server Management Tools server terminal unmount the DVD-RW, eject it, and exit from root privilege:
$ eject cdrom
11 12
At the Communication server Management Tools server remove the DVD-RW from the tray and close the tray. Store the DVD-RW according to office procedure.
--End--
Variable denitions
Variable <alr_flag> Value If this image should be booted for the next restart, specify ALR. Otherwise, leave the field blank. The file name for a single image file, or a wildcard expression such as "*_CM" An integer between 0 and 15, and is a free position in the ITOC. The absolute path to the files on the STORM unit to place the files such as /nfsserv/3pc/cs/sd00/image1 SD00 or SD01 The IP Address of the STORM unit such as 172.18.96.6 The name of the volume such as TEMP
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
440
Network power up
Procedure steps
Step 1 Action From the Shelf View, right-click on the card and select Lock.
Wait for the lock icon to appear on the selected card and the other SAM21 SC to indicate that it is in simplex (alarm 2C on the other SAM21 SC). Right-click on the card again and select Unlock. The card resets, downloads software, and reboots.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
441
Prerequisites
To reduce the risk of service interruption, first busy the GWC applications on specific GWC nodes using the Communication server 2000 GWC Manager. For more information, see Nortel Gateway Controller Administration and Security ( (NN10213-611)) .
Procedure steps
Step 1 Action At the Communication server 2000 SAM21 Manager client from the Shelf View, right-click the GWC card you want to reboot and select Card View.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
442
Network power up
2 3
At the Card View, select the States tab. Click the Lock button to lock the card.
4 5 6
Wait until the Card Status is Locked and the History window indicates that: Application locked successfully. Click the Unlock button. Monitor the reboot process. Wait until the Card Status changes to Unlocked and the History window indicates Bootloaded successfully.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
443
Prerequisites
You must have the root user password for the N240. You need to IP address for the inactive unit.
Procedure steps
Step 1 2 Action At the N240 server, the N240 server is inactive. Turn on the circuit breakers. The server should recover on its own and start syncing with the active server.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
444
Network power up
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
445
XA-Core 3rd Party Core Message Switch ENET Ethernet Routing Switch 8600 LPP SDM, Communication server 2000 Core Manager, or Core and Billing Manager IEMS Communication server 2000 Management Tools MG 9000 Manager Border Control Point Manager SAM21 shelf and cards CICM USP Media Gateway 15000/7400 or Multiservice Switch 15000/7400 MDM workstation SPM, MG 4000, IW-SPM, DPT-SPM MCS servers Border Control Point MS 2000 Series or UAS
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
446
Navigation Performing a partial power down recovery of the USP (page 447) Performing a partial power down recovery of the XA-Core (page 448) Performing a partial power down recovery of the Ethernet Routing Switch 8600 (page 451) Performing a partial power down recovery of the Media Gateway/Multiservice Switch 15000 (page 452) Performing a partial power down recovery of the SPM (page 453)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
447
Procedure steps
Step 1 2 Action At the USP frame reseat the front panel and back panel CC card in Slot 18 on the Control shelf. Reseat the front panel and back panel CC cards in Slot 18, beginning with the CC Slot 18 card on each of the extension shelves. Reseat the inactive RTC. The inactive RTC should SCSI boot and perform a database synchronization with the active RTC. It should come into service in approximately 5-7 minutes. 4 5 6 At the USP Manager verify the reseated RTC is in-service and inactive. Verify logs and alarms to ensure there were no unexpected or undesired events. Perform test calls.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
448
Procedure steps
WARNING
Potential extended equipment outage
Nortel recommends that you perform this procedure under the supervision of Emergency Technical Assistance Services (ETAS) of Nortel. Also call your next level of support before you perform this procedure.
CAUTION
Potential loss of service or extended outage
This procedure is only to restore normal operation after the performance of measures for emergency power conservation. Do not use this procedure or parts of this procedure for equipment maintenance purposes.
Step 1
Action At your current location use office records to identify and record the power converters which supply the MAPs and printers of the switch. Restore power to all the power converters identified in step 1that supply power to the MAPs and printers of the switch. At the MAP restore power to one side of the remote oscillator shelf (NT3X9507), If you turned off power to save emergency backup power. To restore power, perform the procedure
2 3
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
449
"Restoring the remote oscillator shelf to duplex operation" in Recovery Procedures () , 297-8021-545. 4 Restore power to one message switch (MS) shelf if you turned off power to save emergency backup power. To restore power, perform the procedure "Restoring the MS duplex operation" in Recovery Procedures () , 297-8021-545. Restore power to one unit of the CCS7 message switch and buffer (MSB7) if you turned off power to save emergency backup power. To restore power, perform the procedure "Restoring the MSB7 to duplex operation" in Recovery Procedures () , 297-8021-545. Restore power to one link interface module (LIM) unit on one or more link peripheral processors (LPP) if you turned off power to save emergency backup power. To restore power, perform the procedure "Restoring LPP LIMs to duplex operation" in Recovery Procedures () , 297-8021-545. If you removed power from one or more network frames, restore power to the changed frames at the power distribution center (PDC). Restore power to one or more junctored network (JNET) shelves if you turned off power to save emergency backup power. To restore power, perform the procedure "Restoring the junctored network to duplex operation" in Recovery Procedures, 297-8021-545. Restore power to one or more enhanced network (ENET) shelves if you turned off power to save emergency backup power. To restore power, perform the procedure "Restoring the enhanced network to duplex operation" in Recovery Procedures () , 297-8021-545. Restore power to one or more units of a line group controller (LGC), line trunk controller (LTC), or digital trunk controller (DTC). Restore power if you turned off power to save emergency backup power. To restore power, perform the procedure "Restoring LGCs, LTCs, and DTCs to duplex operation" in Recovery Procedures () , 297-8021-545. Restore power to one or more units of a line concentrating module (LCM) if you turned off power to save emergency backup power. To restore power, perform the procedure "Restoring LCMs to duplex operation" in Recovery Procedures () , 297-8021-545. Restore power to one or more line modules (LM) if you turned off power to save emergency backup power. To restore power, perform the procedure "Restoring line modules to duplex operation" in Recovery Procedures () , 297-8021-545.
10
11
12
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
450
13
Restore power to one or more maintenance trunk modules (MTM) if you turned off power to save emergency backup power. To restore power, perform the procedure "Returning maintenance trunk modules to service" in Recovery Procedures () , 297-8021-545. To clear any alarms on the MAP display, perform the correct alarm clearing procedure. The correct alarm clearing procedure is in this document or in the Alarm and Performance Monitoring Procedures () , 297-8021-543.
--End--
14
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
451
Performing a partial power down recovery of the Ethernet Routing Switch 8600
This procedure is used to perform a partial power down recovery of an Ethernet Routing Switch 8600.
Procedure steps
Step 1 Action At the chassis turn the DC input power source on, or reset the power source circuit breaker to provide power to the power supply. Turn the power supply switch to the on position.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
452
Performing a partial power down recovery of the Media Gateway/Multiservice Switch 15000
This procedure is used to perform a partial power down recovery of a Media Gateway/Multiservice Switch 15000.
Procedure steps
Step 1 Action At the frame manually switch on the following breakers on the lower shelf:
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Variable definitions
453
Procedure steps
Step 1 Action At the MAP post the SPM to recover by typing:
>POST SPM <spm_no>
and pressing the Enter key. 2 3 Manually reseat the inactive CEM. Post, busy, and return the inactive CEM to service by typing:
>POST CEM <inactive_cem>; BSY; RTS
and pressing the Enter key. 4 5 Manually reseat each RM. Select, busy, and return each inactive resource module to service by typing:
>POST <rm_type> ALL; BSY ALL; RTS ALL
and pressing the Enter key. 6 Perform step 5 for each RM type.
--End--
Variable denitions
Variable rm_type spm_no Value is SRM, OC3, ATM, GEM, DSP, VSP, ALM, or DLC is the number of the SPM to be powered down
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
454
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
455
Unlocking the Call Agent (page 456) Unlocking a GWC card (page 457)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
456
Procedure steps
Step 1 Action At the Communication server 2000 SAM21 Manager client from the Shelf View, right click on the card and select Unlock from the context menu.
Unlock is also available from the States tab of the Card View window. The card resets, downloads software, and reboots. 2 Wait for the lock icon to disappear. Do not perform any patching activities on the Call Agent until ten minutes have passed.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
457
after replacing a GWC card as part of a fault clearing activity when a new software load is available when you complete reprovisioning a GWC card or GWC node (a node is made up of unit 0 and unit 1 GWC cards) and you want the card or node to use the new provisioning values when you applied or removed a patch to the GWC software using the Network Patch Manager (NPM) and saved a new GWC software image to the Communication server 2000 Core Manager
Prerequisites
The card must be locked. If IP addresses for the card that you want to unlock and its mate are not contiguous, you cannot unlock the card. You must correct these addresses using procedure "Manually re-provisioning GWC cards" in Nortel Gateway Controller Configuration ( (NN10205-511)) before attempting to unlock the card. If the GWC hardware does not support the service profile assigned to the GWC node, the unlock process fails and the system generates the CMT305 critical alarm. For the description of each profile and its hardware requirement, see Nortel Gateway Controller Configuration ( (NN10205-511)) .
Procedure steps
Step 1 Action At the Communication server 2000 GWC Manager client at the Communication server 2000 Management Tools window, click the Gateway Controller folder from the Device Types directory tree in the far left frame. From the Contents of: GatewayController frame, select the GWC node that contains the card you want to unlock.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
458
Select the Maintenance tab to display maintenance information about the node.
Click the Card View button for the card you want to unlock. The Communication server 2000 SAM21 Manager opens. If a card is locked, all fields display the value <unknown>. At the Communication server 2000 SAM21 Manager in the card view, select the States tab, then click the Unlock button to unlock the card. If you want to display the status of all cards in the shelf, select Shelf View from the View menu.
If the IP addresses for the selected card and its mate are not contiguous, the system displays the following error message:
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Procedure steps
459
Follow procedure "Manually re-provisioning GWC cards" in Nortel Gateway Controller Configuration ( (NN10205-511)) to correct these addresses, then repeat this procedure. If the GWC service profile and the GWC hardware are not compatible, the unlock operation fails and the system generates the CMT305 critical alarm. For the description of the alarm and the required action, see Nortel Carrier Voice over IP Fault Management Logs Reference Volume 2 ( (NN10275-909v2)) . 6 Observe the system response in the History window. The card is unlocked when you see the text Application unlocked successfully . 7 If required, repeat this procedure for the next GWC card until all the GWC cards have been unlocked and brought into service. Each GWC node has two cards.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
460
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
461
Reference
The reference chapter describes information about the disaster recovery solutions.
Backup operations
There are two types of backup operations for components of CVoIP solutions: service data backup and fileset level backup. Perform service data backups once on each day during a period of low activity. Table 27 "Service data backup components" (page 461) lists components that are typically backed up using a service data backup.
ATTENTION
Point 1Service data from the SAM21, GWC, and MG 9000 is not backed up locally. This data is backed up at the manager. The manager is backed up to DVD using CS 2000 Core Manager or Core and Billing Manager (CBM) or CS 2000 Management Tools procedures. Point 2Service data from the IW SPM and MG 4000 is automatically backed up when the CS 2000 is backed up. Point 3Perform MDM backups by copying the files to tape or an off-product storage system through UNIX commands or CRON jobs. The script purgeTempData.sh in the /opt/nortel/iems/current/bin directory purges all event, alarm, and performance data from the IEMS database. After you purge the data you cannot retrieve it. The purge deletes all events, alarms, and performance data. To reduce the time taken to backup or restore, you must stop the IEMS server and execute the purgeTempData.sh script to purge the events, alarms, and performance data from the database.
NETWORK INTELLIGENCE
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
462
Reference
Table 27 Service data backup components (contd.) Component CS 2000 Call Agent SAM21 GWC Session Server Policy Controller Ethernet Routing Switch 8600 UAS MS 2000 Series Procedure (s) Backing up an XA-Core office image from disk to tape (page 119) Backing up Call Agent (page 126) none (See "Point 1" (page 461) ) none (See "Point 1" (page 461) ) Backing up the SST (page 129) Performing a manual backup of the Policy Controller database (page 162) Saving the Ethernet Routing Switch 8600 boot configuration file (page 136) Backing up UAS configuration files (page 160) For the following MS 2000 Series procedures, see Nortel Media Server 2000 Series Configuration ( (NN10340-511)) Displaying the MS 2000 Series node configuration: [***Configuring Automated INI file backup on a Media Server 2000 node] [***Changing the SNMP community string passwords] [***Changing the SNMP community string password for a Media Server 2020 node] Backing up all MS 2000 Series node INI files: [***Backing up INI files for all nodes] Configuring the MS 2000 Series CLUI tool: [***Displaying Media Server 2010 node current configuration] [***Displaying Media Server 2020 node current configuration] Backing up the APS-specific Oracle database and application files (page 150)
APS USP Real-time Transport Protocol (RTP) Media Portal CICM CORE NETWORK Multiservice Switch or Media Gateway devices GATEWAYS MG 15000 MG 9000 MG 4000 IW SPM
none (See "Point 1" (page 461) ) none (See "Point 2" (page 461) ) none (See "Point 2" (page 461) )
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
463
NETWORK MANAGEMENT CS 2000 Core Manager Core Element Manager CS 2000 Management Tools Starting or stopping the automated synchronous backup restore manager service in Nortel ATM/IP Solution-level Administration and Security ( (NN10402-600)) Starting or stopping the automated synchronous backup restore manager service (page 107)Performing a backup of Oracle/MySQL data on an SPFS-based server (page 156) Performing a backup of Oracle/MySQL data on an SPFS-based server (page 156) Performing a backup of file systems on an SPFS-based server (page 164) Starting or stopping the automated synchronous backup restore manager service in Nortel ATM/IP Solution-level Administration and Security ( (NN10402-600)) Starting or stopping the automated synchronous backup restore manager service (page 107) Performing a backup of Oracle/MySQL data on an SPFS-based server (page 156) IEMS centralized security server MG 9000 Manager Taking a full IEMS backup (page 169) Performing a backup of file systems on an SPFS-based server (page 164) Starting or stopping the automated synchronous backup restore manager service (page 107) MDM Backing up an SPFS-based security client Performing a backup of oracle data on an SPFS-based server Performing a backup of file systems on an SPFS-based server Starting or stopping the automated synchronous backup restore manager service
Fileset level backups back up the software and configuration data. Perform this type of backup after hardware changes, software updates or patches, or major configurations. You must perform a fileset backup before a major upgrade. Table 28 "Fileset level backup components" (page 463) provides a list of components with fileset level backup procedures.
Table 28 Fileset level backup components Component Network intelligence Procedures
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
464
Reference
Table 28 Fileset level backup components (contd.) Component GWC USP Gateways Multiservice Switch or Media Gateway devices Network management CS 2000 Core Manager orCore and Billing Manager (CBM) Creating system image backup tapes (S-tapes) manually Configuring SBA backup volumes on the core Starting or stopping the automated synchronous backup restore manager service. Performing a backup of file systems on an SPFS-based server. Configuring SBA backup volumes on a billing stream. Performing a backup of file systems on an SPFS-based server (page 164) Configuring SBA backup volumes on the core Restoring Multiservice Switch or Media Gateway service data (page 23) Procedures Create a backup of the GWC load file in Nortel Carrier Voice over IP Upgrade and Patches ( (NN10440-450)) . Administration: USP Manual Backup
ATTENTION
Point 1Service data from the SAM21, GWC, and MG 9000 is not backed up locally. The data is backed up at the manager. The manager is backed up to DVD using CS 2000 Core Manager or Core and Billing Manager (CBM) or CS 2000 Management Tools procedures. Point 2Service data from the IW SPM and MG 4000 is automatically backed up when the CS 2000 is backed up.
Table 29 Component backup frequency Component Network Intelligence CS 2000MSC Server 1000 XA-Core Call Agent Weekly Backup Frequency
Weekly
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
465
Table 29 Component backup frequency (contd.) Component SAM21 GWC Session Server CS 2000 CS LAN UAS MS 2000 Series APS RTP Media Portal CICM USP Core network Multiservice switches Gateways MG 9000 MG 4000 IW SPM See "Point 1" (page 464) See "Point 2" (page 464) See "Point 2" (page 464) Full backup - monthly, and prior to all migrations and patch operations Backup Frequency See "Point 1" (page 464) See "Point 1" (page 464) Weekly Full backup - monthly, and prior to all migrations and patch operations and configuration changes Daily Daily Daily Daily Daily Full system backup - once a week. Modified (differential) backup - once a day
Network management CS 2000 Management Tools MG 9000 Manager IEMS Nortel MDM Daily
Store CDs and DVDs at a temperature between 5 and 25 C and a humidity level between 10% and 80%. Do not touch the surface of a CD or DVD with your hands and do not mark the surface with a pencil, pen, or marker. Use a cotton cloth or disk cleaning kit to clean CDs and DVDs.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
466
Reference
ATTENTION
If checksum errors occur, perform a visible inspection of the disk to check for scratches or damage. If no scratches are visible, clean the disk using a cotton cloth or disk cleaning kit and retry.
State
Connectivity Host Name Date Time since last reboot System Power-On Time System booted from Last restart cause Last power event cause Current version Platform IP Address Platform EM Client IP Address
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
467
Table 30 System information page fields (contd.) Field Server Location Host Name Description Physical location of the unit. Name of the unit.
Table 31 "Node maintenance page fields" (page 467) lists information about Node maintenance page fields.
Table 31 Node maintenance page fields Field Operation State Activity Jam State Maintenance Actions (active unit only) Description Indicates the operational state of the NCGL software. Indicates the activity state of the platform software. Indicates if the inactive unit is Jammed. Maintenance page for performing SWACT and to Jam. See the maintenance and manual cold SWACT procedures in this document.
Table 32 "Chassis information page fields" (page 467) lists information about Chassis information page fields.
Table 32 Chassis information page fields Field Chassis information: Self Test Chassis information: Chassis Subsystems Description Status of the self test performed on the platform at bootup. Status of the platform hardware subsystems including the memory, CPU, all drives, network connection, power supplies, cooling and other I/O connections. Indicates the 1, 5 and 15 minute load averages for the CPU utilization. Indicates the 1 minute CPU load average utilization threshold value. When the set threshold value is exceeded, the appropriate minor, major or critical alarm is raised. Indicates the 5, 20, and 30 minute CPU utilization average. When the threshold value is exceeded, an alarm is raised.
CPU Information: load average CPU Information: load average threshold values
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
468
Reference
Table 32 Chassis information page fields (contd.) Field Chassis Utilization: alarm threshold values Description Indicates the 5, 20, and 30 minute CPU utilization average threshold value. When the set threshold value is exceeded, an alarm is raised. Total number of processes (non-threaded) that are running on the SST Platform. Number of defunct or terminated NCGL zombie processes.
A zombie process is a process that has terminated either because it has been killed by a signal or because it has called an exit() and whose parent process has not yet received notification of its termination. A zombie process exists solely as a process table entry and consumes no other resources. Process Information-zombie: minor alarm threshold value Maximum number of zombie processes allowed to be run by the CPU before a minor alarm is raised indicating that the set threshold has been exceeded. Maximum number of zombie processes allowed to be run by the CPU before a major alarm is raised indicating that the set threshold has been exceeded. Maximum number of zombie processes allowed to be run by the CPU before a critical alarm is raised indicating that the set threshold has been exceeded. Total amount of RAM installed on the mother board of each SST unit. Both units must have the same amount. Amount of memory available unallocated for use. Amount of memory available for programs. Indicates the threshold amount of available memory (in Mbytes) that the system must drop below before a minor alarm is raised. Indicates the threshold amount of available memory (in Mbytes) that the system must drop below before a major alarm is raised. Indicates the threshold amount of available memory (in Mbytes) that the system must drop below before a critical alarm is raised.
Memory Information: Total Memory (MB) Memory Information: Free Memory (MB) Memory Information: Available memory (MB) Memory Information: minor alarm threshold value Memory Information: major alarm threshold value Memory Information: critical alarm threshold value
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
469
Table 33 "Network connectivity page fields" (page 469) lists information about Network connectivity page fields.
Table 33 Network connectivity page fields Field Unit 0,1 Links Unit 0,1 Status Unit 0,1 Activity Unit 0,1 Maintenance Description Indicates which Ethernet IP links are installed on the units (each unit has two links). Status of the Ethernet links. Activity status of the Ethernet links, either active or inactive. Indicates the maintenance actions that can be performed on the Ethernet links, either Lock, Unlock or Swlink. Status of the PTP links between both units in the node. Network IP address of the SST unit. IP address of the local (active) SST unit. IP address of the mate (inactive) SST unit. IP address of the active or inactive Ethernet port 0. IP address of the active or inactive Ethernet port 1. IP address of the active or inactive PTP link.
Unit 0,1 PTP Links status Unit IP Active IP Inactive IP Port 0 IP Port 1 IP PTP IP
Table 34 "Disk Services page fields" (page 469) lists information about Disk Services page fields.
Table 34 Disk Services page fields Field RAID Array Status: Name RAID Array Status: Size (GB) RAID Array Status: State Description Indicates the name of each RAID-1 array in the system. Indicates the size of the partition in Gigabytes. Indicates a high level state for the array: - ".": indicates the array is functioning normally. - Missing: a disk was removed from the array. - Failed: a disk in the array has failed and needs to be replaced. - Rebuilding: the array is in the process of rebuilding to a fault-tolerant mode.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
470
Reference
Table 34 Disk Services page fields (contd.) Field RAID Array Status: Disk 0 RAID Array Status: Disk 1 RAID Array Status: Status Description Service status of disk 0. Service status of disk 1. Indicates the status of the array. Values are: - The array is operating normally - Missing - Failed - Rebuild Indicates the disk number in the array, 0, or 1. Total capacity of the disk drive in Gigabytes. Installation state of the disk. Indicates whether a hard disk can be inserted into the RAID array. Indicates the status of individual filesystems on the disk array. Indicates the name of the filesystem on the disk array. Some filesystem names are reserved. Indicates the results of any tests run on the filesystems. Tests are run approximately every 10 minutes to verify that all of the basic filesystem operations are working on each of the filesystems. Total amount of disk space (in MB) allocated for this filesystem. Total amount of disk space (in MB) in use on this file system. Total amount of disk space (in %) in use on this file system. Percentage of total disk space (in MB) free for use on this filesystem. Amount of disk space (in %) available for use by platform processes and applications. maximum amount of disk space (in %) that can be used before a minor alarm is raised indicating that the set threshold has been exceeded.
Disk Maintenance: Disk Number Disk Maintenance: Disk Size (GB) Disk Maintenance: Disk State Disk Maintenance: Disk Action Filesystem Information: Monitor Filesystem Information: Filesystem Name Filesystem Information: Test Results
Filesystem Information: Total Space (MB) Filesystem Information: Total Space Used (MB) Filesystem Information: Total Space Used (%) Filesystem Information: Total Space Available (MB) Filesystem Information: Total Space Available (%) Filesystem Information: Minor Alarm Threshold (%)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
471
Table 34 Disk Services page fields (contd.) Field Filesystem Information: Major Alarm Threshold (%) Description Maximum amount of disk space (in %) that can be used before a major alarm is raised indicating that the set threshold has been exceeded. Maximum amount of disk space (in %) that can be used before a critical alarm is raised indicating that the set threshold has been exceeded. Name of the volume group in the array. Total size of the volume group in the array. Amount of volume group space, in Gigabytes, currently allocated to filesystems. Amount of volume group space (in %) currently allocated to filesystems. Amount of unallocated volume group space, in Gigabytes, available for filesystems. Amount of unallocated volume group space (in %) available for filesystems.
Volume Group Information: Volume Group Name Volume Group Information: Volume Group Size (GB) Volume Group Information: Total Space Allocated (GB) Volume Group Information: Total Space Allocated (%) Volume Group Information: Total Space Available (GB) Volume Group Information: Total Space Available (%)
Table 35 "Services page fields" (page 471) lists information about Services page fields.
Table 35 Services page fields Field Network Services: Number of Active Command Line Sessions Network Services: Number of Clients with Active Web Sessions NTP Information: Server1 - Server 3 NTP Information: Total Number of Servers Description Number of command line interface (CLI) sessions (both remote and local) on the unit Number of clients running one or more web GUI sessions IP address of up to three Network Time Protocol (NTP) servers in the network, and the status of the connection Number of NTP servers registered with the CS-LAN network
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
472
Reference
Table 35 Services page fields (contd.) Field NTP Information: Accessible Servers NTP Information: Synchronized Servers Description Number of NTP servers accessible to the SST Number of NTP servers to which the unit is synchronized
Table 36 "Administration page fields" (page 472) lists information about Administration page fields.
Table 36 Administration page fields Field Bootload Management: Bootload Bootload Management: Maintenance Description Load ID for the NCGL platform software load Indicates if the Bootload is the default. You can choose a new default bootload if there is more than one load available. Additional loads can come from maintenance releases. File transfer protocol or source location for the platform software upgrade: FTP, Anonymous FTP, HTTP, HTTPS, Local File, or Local CDROM If a logon ID is required to access the upgrade platform load from another server in the network, a logon ID can be entered here. If a password is required to access the upgrade platform load from another server in the network, a password can be entered here. If it is required to access the upgrade platform load from another server in the network, an IP address can be entered here. The target upgrade load path and filename is entered here. The Upgrade button initiates a platform NCGL upgrade. Used to execute the Reboot, Halt, Rebootmate, and Haltmate functions. These are service affecting commands.
Software Upgrade: File Software Upgrade: ActionUpgrade button Server Maintenance (active and inactive units)
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
473
Device audits
After a dead office recovery, several data mismatches may be present in the network. An audit must be performed to clear up these data mismatches. Table 37 "Device audit procedures" (page 473) lists the audit procedures available for each component and where those procedures are located.
Table 37 Device audit procedures Component CS 2000 Core Manager or Core and Billing Manager GWC Procedures available Performing a system audit Location Nortel CS 2000 Core Manager Fault Management ( (NN10082-911))
Perform a GWC line data integrity audit Perform a GWC trunk data integrity audit Perform a CS 2000 data integrity audit Perform a GWC V5.2 data integrity audit Performing an audit of the MG 9000 provisioning data
MG 9000
474
Reference
devaddr is the IP address of the device and has the format n.n.n.n. id is the userID for a Passport and the READ community string for a Passport 4400/4460. pw is the user password for a Passport and the WRITE community string for a Passport 4400/4460. viewname is the name of the view file for a selective backup. -cdir is the backup directory used for this backup operation. -f indicates that the backup information is obtained from a file. backup_info_file is the name of the file containing backup information. Each line in the file has the same format as the -full, incr, or -view options. -chost indicates the remote Controller to be used in place of the Controller running on the local host. controller_address is the address of the Controller and has the format host[:port] -nolog indicates that output messages are discarded. -log indicates that output messages go to stdout/stderr or to a log file. The default is stdout/stderr. logfile is the name of the file to which output messages go. If not specified, the messages go to the file mbrbackup.log in the current directory (where the tool runs). -nc indicates the number of concurrent backups to be performed. By default, SNMP Devices Backup tries to back up 5 different devices concurrently. This parameter is useful when you are backing up a large number of devices. #_concurrent_connections is the number of concurrent backups.
--End--
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
475
./<devtype>/<devname>/timestamp>.<dataset>/\<datafiles. ..>
Note: The period (.) represents the SNMP Devices Backup and
Restore root directory.
where devtype is the device type (PP4400, or PP4460). devname is the device name timestamp has the format yyyymmddhhmmss dataset is the dataset name. datafiles... are the names of one or more files that are backed up. Note: The period (.) represents the SNMP Devices Backup and
Restore root directory.
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
476
Reference
Carrier VoIP Nortel Carrier Voice over IP Solutions Disaster Recovery Procedures NN10450-900 06.02 3 October 2008
Copyright 2008 Nortel Networks
Carrier VoIP