White paper

BMC ProactiveNet Performance Management: Disaster Recovery Best Practices

May 2010

www.bmc.com

BMC Software, Inc., Confidential

Contacting BMC Software You can access the BMC Software website at http://www.bmc.com. From this website, you can obtain information about the company, its products, corporate offices, special events, and career opportunities. United States and Canada Address BMC SOFTWARE INC 2101 CITYWEST BLVD HOUSTON TX 77042-2827 USA Telephone 713 918 8800 or 800 841 2031 Fax 713 918 8000

Outside United States and Canada Telephone (01) 713 918 8800 Fax (01) 713 918 8000

If you have comments or suggestions about this documentation, contact Information Design and Development by email at doc_feedback@bmc.com.

Copyright 2010 BMC Software, Inc. BMC, BMC Software, and the BMC Software logo are the exclusive properties of BMC Software, Inc., are registered with the U.S. Patent and Trademark Office, and may be registered or pending registration in other countries. All other BMC trademarks, service marks, and logos may be registered or pending registration in the U.S. or in other countries. All other trademarks or registered trademarks are the property of their respective owners. The information included in this documentation is the proprietary and confidential information of BMC Software, Inc., its affiliates, or licensors. Your use of this information is subject to the terms and conditions of the applicable End User License agreement for the product and to the proprietary and restricted rights notices included in the product documentation.. Restricted Rights Legend U.S. Government Restricted Rights to Computer Software. UNPUBLISHED -- RIGHTS RESERVED UNDER THE COPYRIGHT LAWS OF THE UNITED STATES. Use, duplication, or disclosure of any data and computer software by the U.S. Government is subject to restrictions, as applicable, set forth in FAR Section 52.227-14, DFARS 252.227-7013, DFARS 252.227-7014, DFARS 252.227-7015, and DFARS 252.227-7025, as amended from time to time. Contractor/Manufacturer is BMC Software, Inc., 2101 CityWest Blvd., Houston, TX 77042-2827, USA. Any contract notices should be sent to this address.

BMC Software, Inc., Confidential

Customer Support
You can obtain technical support by using the Support page on the BMC Software website or by contacting Customer Support by telephone or email. To expedite your inquiry, please see Before Contacting BMC Software.

Support website
You can obtain technical support from BMC Software 24 hours a day, 7 days a week at http://www.bmc.com/support. From this website, you can: Read overviews about support services and programs that BMC Software offers. Find the most current information about BMC Software products. Search a database for problems similar to yours and possible solutions. Order or download product documentation. Report a problem or ask a question. Subscribe to receive email notices when new product versions are released. Find worldwide BMC Software support center locations and contact information, including email addresses, fax numbers, and telephone numbers.

Support by telephone or email

In the United States and Canada, if you need technical support and do not have access to the Web, call 800 537 1813 or send an email message to customer_support@bmc.com. (In the Subject line, enter SupID:<yourSupportContractID>, such as SupID:12345.) Outside the United States and Canada, contact your local support center for assistance.

Before contacting BMC Software

Have the following information available so that Customer Support can begin working on your issue immediately: Product information o Product name o Product version (release number) o License number and password (trial or permanent) Operating system and environment information o Machine type o Operating system type, version, and service pack o System hardware configuration o Serial numbers o Related software (database, application, and communication) including type, version, and service pack or maintenance level Sequence of events leading to the problem Commands and options that you used Messages received (and the time and date that you received them) o Product error messages o Messages from the operating system, such as file system full o Messages from related software

BMC Software, Inc., Confidential

License key and password information

If you have a question about your license key or password, contact Customer Support through one of the following methods: E-mail customer_support@bmc.com. (In the Subject line, enter SupID:<yourSupportContractID>, such as SupID:12345.) In the United States and Canada, call 800 537 1813. Outside the United States and Canada, contact your local support center for assistance. Submit a new issue at http://www.bmc.com/support.

BMC Software, Inc., Confidential

Contents
Introduction ............................................................................................................... 6 Setup.......................................................................................................................... 7 High-level disaster recovery architecture .................................................................. 7 Preparing the primary server and integrations .......................................................... 8 Setting up the primary server ................................................................................ 8 Configuring the Remote Event Cell (remote computer) ..................................... 12 Configuring the BMC Remedy ITSM (IBRSD) server ....................................... 12 Configuring IBRSD on the primary server ......................................................... 12 Preparing the secondary server ............................................................................... 13 Offline backup requirements ................................................................................... 14 Backup frequency ................................................................................................ 14 Backup files ......................................................................................................... 15 Verification of backup ......................................................................................... 16 Steps to follow during disaster recovery ................................................................. 16 Steps on the secondary server ............................................................................. 16 Steps on the BMC Remedy ITSM system........................................................... 17 Steps on remote cell systems ............................................................................... 17 High availability and disaster recovery ................................................................... 17

BMC Software, Inc., Confidential

White paper

BMC ProactiveNet Performance Management: Disaster Recovery Best Practices

Introduction
This white paper provides guidance and a procedure for putting a disaster recovery (DR) plan in place for BMC ProactiveNet Performance Management infrastructure implementation. In a broad sense, disaster recovery can be handled by using the High Availability feature of operating systems (OSs) as documented in the installation guide. The guidance in this white paper is based on stand-alone systems and establishing a DNS alias name to redirect the events and data to a secondary server during a disaster. It recommends using a storage replication solution that can ensure the data consistency between a cell database and a BMC ProactiveNet Performance Management Server (Sybase) database while the BMC ProactiveNet Performance Management Server is up and running. It describes the files that are critical for a successful BMC ProactiveNet Performance Management Server recovery, when an enterprise is ready to shut down the BMC ProactiveNet Performance Management Server for a minimal amount of time to perform an offline backup. The procedure has been tested in-house using a DNS alias name. However, customers can extend this procedure with proper modification and testing for the following scenarios: Change the IP address and host name of the BMC ProactiveNet Performance Management Server Bind the server to a virtual IP address and host name Configure BMC ProactiveNet Performance Management for disaster recovery and high availability by using another IP address and host name

The procedure is applicable to versions 8.0.00 and 8.1.00 of BMC ProactiveNet Performance Management on Microsoft Windows 2003 and Windows 2008.

Setup
Prepare two servers for BMC ProactiveNet Performance Management disaster recovery that are identical on the following characteristics: OS version (for example, Windows 2003 or Windows 2008 Enterprise Server) Hardware configuration Available in the same domain Time stamp

High-level disaster recovery architecture

This following diagram shows BMC ProactiveNet Performance Management configured with two servers, a primary server and a secondary server. Both servers are in the same domain. The DNS alias name will resolve to the secondary server during a disaster so that the monitoring of devices and BMC product integrations continue to function.

BMC Software, Inc., Confidential

Preparing the primary server and integrations

This section provides installation and configuration steps for preparing a BMC ProactiveNet Performance Management primary server. The preparation of the primary server ensures that few steps need to be performed on the secondary server during a disaster. This section also describes the steps involved to integrate remote event cells and BMC Remedy IT Service Management (ITSM). Note: Throughout this white paper, the primary server is referred to as bppm_primary, the secondary server is referred to as bppm_secondary, and the logical DNS alias name is referred to as bppm.

Setting up the primary server

1. Establish a DNS alias (for example, bppm) on the DNS Server for the primary server (for example, bppm_primary). 2. Enter the DNS alias name in the etc\hosts file.
127.0.0.1 localhost bppm

3. Install BMC ProactiveNet Performance Management Server and note the following information that needs to be same and is required during the installation of BMC ProactiveNet Performance Management Server on the secondary server: Installation location (for example, C:\Program Files\ProactiveNet) Email information (Sender email, Administrator email, SMTP server) PATROL Proxy information BMC Atrium CMDB / BMC Remedy ITSM information

4. License the BMC ProactiveNet Performance Management Server. 5. Edit the Apache configuration files, and change the server name to the alias name.
C:\BMC\ProactiveNet\pw\ApacheGroup\Apache\conf\httpd.conf ServerName bppm C:\BMC\ProactiveNet\pw\ApacheGroup\Apache\conf\httpd-ssl.conf ServerName bppm:443

6. Execute the following pw commands. The local BMC ProactiveNet Performance Management agent and selfmonitoring configuration use physical host names and IP addresses. These need to be changed to more generic settings to minimize the number of changes required during failover. The pw commands apply the necessary changes. You can execute these commands either sequentially or by keeping them in a batch

BMC Software, Inc., Confidential

file. Optionally, you can make these changes by using the BMC ProactiveNet Performance Management Admin console.
call pw remote edit ProactiveServer 127.0.0.1 call pw device edit currentservername n genericname call pw monitor edit -f agent=ProactiveServer -A TARGETIP=127.0.0.1 call pw monitor edit -f all,mtype=SybaseASAIntelliscope -old SRCIP=ProactiveServer -A TARGETIP=127.0.0.1 -A INSTANCE_NAME="ProactiveNet DB" call pw monitor edit -f all,mtype=SybaseQuery -old SRCIP=ProactiveServer -A TARGETIP=127.0.0.1 -A INSTANCE_NAME="ProactiveNet DB Query" call pw monitor edit -f all,mtype=WindowsFileSystem -old SRCIP=ProactiveServer -A TARGETIP=127.0.0.1 -A INSTANCE_NAME="ProactiveNet Root Drive" call pw monitor edit -f all,mtype=WindowsSystem -old SRCIP=ProactiveServer -A TARGETIP=127.0.0.1 -A INSTANCE_NAME="ProactiveNet Windows Host" call pw monitor edit -f all,mtype=WebURL -old SRCIP=ProactiveServer -A URL="http://localhost" -A INSTANCE_NAME="ProactiveNet WEB UI" call pw monitor edit -f all,mtype=AgentStatus -old INSTANCE_NAME="ProactiveNet Agent Status" -A TARGETIP=127.0.0.1

Note: Executing the pw commands that update monitors produces messages such as No monitor instance is updated. You can use the Admin console to verify that monitor instances, devices, and agents have been updated with the proper local host IP address (127.0.0.1). 7. Stop the BMC ProactiveNet Performance Management Server services. 8. Change the following services to manual: BMC ProactiveNet Database Engine (ASANYe_ProactiveNetDB) BMC ProactiveNet Server (ProNetServer)

9. Use regedit to make the following changes: a. Rename the mcell_pncell_serverName service to match the BPPM_<bppm>. b. Change the folder name of the service in the registry. c. Update the Display Name and imagepath (-n argument) keys. 10. Edit the pronet.conf file located in the bppmInstallDirectory\pw\custom\conf directory, and change the following entries to use the DNS alias name (for example, bppm):
Pronet.rate.cell.cellName=pncell_<bppm> Pronet.jserver.cell.CellName=pncell_<bppm> Pronet.gateway.gatewayName=pn_<bppm> NGP_Server_Name=<bppm>

11. Locate the following files and change all instances of cell name and machine name in the files to use the DNS alias name (for example, bppm).
Cell name entry pncell_<bppm_primary> to pncell_bppm Machine name entry bppm_primary to bppm

BMC Software, Inc., Confidential

File location pw\pronto\conf\cellinfo.list

Entries cell.EM pncell_bppm Production * mc bppm:1828

pw\pronto\conf\pnjserver.conf pw\pronto\data\admin\admin.dir

COMDefine pronet.rate.cell.cellName=pncell_bppm admin ias_Admin 0 bppm:3084 cell pncell_bppm mc bppm:1828

pw\pronto\data\cell\admin.cellconfi name="pncell_bppm" g.xml pw\server\etc\mcell.dir cell pncell_bppm mc bppm:1828 gateway.pn_server pn_server_bppm mc bppm:1851 admin ias_Admin 0 bppm:3084 gateway.ibrsd IBRSD mc bppm:3115 ServerName = pncell_bppm

pw\server\etc\mcxa.conf

pw\admin\etc\mccscommunication\ mccs_host_0=bppm mccs.prop pw\server\etc\pncell_primary\mcell. ServerHostName=bppm conf 12. Rename the following folder names to use the DNS alias name (for example, bppm) instead of the primary server name: bppmInstallDirectory\pw\server\etc\pncell_bppm bppmInstallDirectory \pw\server\log\pncell_bppm bppmInstallDirectory \pw\server\tmp\pncell_bppm bppmInstallDirectory \pw\server\var\pncell_bppm

13. Rename the database file in the bppmInstallDirectory \pw\sybase directory to use the DNS alias name (for example, storm_<bppm>). 14. Edit the - dbCmdLine.conf file located in the bppmInstallDirectory \pw\sybase directory to use the database name that you specified in the preceding step (for example, storm_<bppm>). 15. Change the Data Sources (ODBC) connection settings for Sybase SQL Anywhere. You access these settings from Administrative Tools. a. Double-click the Data Sources (ODBC) icon and then click the System DSN tab. Note that there are two entries for Adaptive Server Anywhere 9.0. Make the

BMC Software, Inc., Confidential

10

changes to both of these entries. b. Click Configure. c. Click the Database tab. d. Change the value of Server name to use the new host name (for example, "storm_<bppm>"). e. Click the Network tab. f. If an entry exists, change it to use the new IP address. If it is blank, leave it blank. 16. Create a - failover.conf file in the bppmInstallDirectory\pw\pronto\conf directory and set the logical server name to the DNS alias name
C:\BMC\ProactiveNet\pw\pronto\conf\failover.conf Logical Server Hostname=<bppm>

Note: Do not insert any spaces between the Logical Server Hostname, the equal sign (=), and <bppm>. 17. Edit the getMcellPID.bat file in the bppmInstallDirectory\pw\pronto\bin directory to use the DNS alias name. 18. Edit the AdminUser.properties file in the bppmInstallDirectory\pw\custom\conf directory to use the DNS alias name. 19. Restart the computer for the changes to take effect. (You must restart the computer for the registry changes to take effect.) 20. Start the BMC ProactiveNet Performance Management Server by using the Windows Services option. 21. Verify that the dbCmdLine.conf file at bppmInstallDirectory\pw\sybase uses the correct database name (for example, storm_bppm) to have a successful start of BMC ProactiveNet Database Engine. 22. Update the following monitors for monitoring the health and performance of the BMC ProactiveNet Performance Management Server device itself: a. In the Admin console, open the Sybase Query folder under the BMC ProactiveNet Performance Management Server device, right-click the monitor instance beneath it that targets BMC ProactiveNet Performance Management Server, and select to edit it. Update the value for Database Name so that it matches that name of the database file. Then click OK. b. Open the Sybase ASA Intelliscope folder, right-click the monitor instance beneath it that targets BMC ProactiveNet Performance Management Server, and select to edit it. Update the value for Database Name so that it matches that name of the database file. Then click OK. 23. Launch the user console using the DNS alias name (for example, http://bppm) and validate the self monitoring using the flashcheck option.

11

BMC Software, Inc., Confidential

Configuring the Remote Event Cell (remote computer)

1. At a command prompt, run the following command to flush out the cache to get the DNS alias name (for example, bppm):
Ipconfig /flushdns

2. Edit the bppmInstallDirectory\pw\server\etc\mcell.dir file, and enter the cell details to use the DNS alias name.
Cell pncell_bppm mc bppm:1828

3. Restart the remote cell.

Configuring the BMC Remedy ITSM (IBRSD) server

This section provides the steps to follow on the BMC Remedy ITSM system. Perform the required installations and configurations in the Integration for BMC Remedy Service Desk User Guide on the BMC Remedy AR System server. 1. Edit the IBRSD.dir file located in the bmcARInstallDirectory\conf\ibrsd directory to include the cell and gateway details. The cell name reflects the DNS alias name instead of the primary server name.
Cell pncell_bppm mc bppm:1828 Gateway.ibrsd IBRSD mc bppm:3115

2. At a command prompt, run the following command to refresh the cache to get the DNS alias name.
Ipconfig /flushdns

3. Restart the AR Plugin server (arplugin.exe).

Configuring IBRSD on the primary server

This section describes the steps to perform to incorporate the changes required for the smooth functioning of BMC ProactiveNet Performance Management and IBRSD (BMC Remedy ITSM) integration. All these steps are required on the primary server. 1. Edit the bppmInstallDirectory\pw\integrations\ibrsd\conf\IBRSD.dir file and provide the BMC ProactiveNet Performance Management Server name, the cell name, and gateway details.
Cell pncell_bppm mc bppm:1828 Gateway.ibrsd IBRSD mc bppm:3115

2. Edit the bppmInstallDirectory\pw\integrations\ibrsd\conf\IBRSD.conf file and provide the BMC Remedy ITSM host name and credentials. 3. At a command prompt, run the Encrypt Utility that is available in the bppmInstallDirectory\pw\integrations\ibrsd\bin directory. The syntax is

BMC Software, Inc., Confidential

12

EncryptUtility f <Absolute passfile path> -l <Absolute logfile path> -p <password>. For example:
EncryptUtility f C:\Program Files\ProactiveNet\pw\integrations\ibrsd\conf l C:\Program Files\ProactiveNet\pw\integrations\ibrsd\log p **

4. Restart the BMC ProactiveNet IBRSD service. 5. Using the Admin console, create a propagation policy for routing events to IBRSD. 6. Verify that the incidents are getting created.

Preparing the secondary server

This section provides the procedure for setting up the secondary server (bppm_secondary). This procedure is applicable only in an environment where a DNS alias name is used to redirect to the secondary server during a disaster. 1. Enter the DNS alias name in the etc\hosts file as follows:
127.0.0.1 localhost bppm <ipaddress of secondary server> <bppm_primary>

2. Install the BMC ProactiveNet Performance Management Server and provide the same information that you provided for the primary server: Installation location (for example, C:\Program Files\ProactiveNet) Email information (Sender email, Administrator email, SMTP server) PATROL Proxy information BMC Atrium CMDB / BMC Remedy ITSM information

3. License the BMC ProactiveNet Performance Management Server. 4. Change the following BMC ProactiveNet Performance Management services to manual: BMC ProactiveNet Database Engine (ASANYe_ProactiveNetDB) BMC ProactiveNet Server (ProNetServer)

5. Use regedit to make the following changes: a. Rename the mcell_pncell_serverName service to match the BPPM_<bppm>. b. Change the folder name of the service in the registry. c. Update the Display Name and imagepath (-n argument) keys. 6. Change the Data Sources (ODBC) connection settings for Sybase SQL Anywhere. You access these settings from Administrative Tools. a. Double-click the Data Sources (ODBC) icon and then click the System DSN tab.

13

BMC Software, Inc., Confidential

Note that there are two entries for Adaptive Server Anywhere 9.0. Make the changes to both of these entries. b. Click Configure. c. Click the Database tab. d. Change the value of Server name to use the new host name (for example, "storm_<bppm>"). e. Click the Network tab. f. If an entry exists, change it to use the new IP address. If it is blank, leave it blank. 7. Restart the computer. 8. Shut down BMC ProactiveNet Performance Management Server on the primary server. 9. Kill the pproxsrv.exe and pwtray.exe processes, if they are present. 10. Transfer the configuration changes made in the primary server to the secondary server by copying the entire bppmInstallDirectory\pw folder from the primary server to the secondary server (bppmInstallDirectory\pw folder). 11. Start BMC ProactiveNet Performance Management Server services by using services.msc. 12. Verify that the dbCmdLine.conf file in the bppmInstallDirectory\pw\sybase directory reflects the correct database name; otherwise, the BMC ProactiveNet Database Engine service might not start. 13. Verify that the devices, events, event history, policies, detail diagnostics, and so on are available. Verify that device and event relationships are present.

Offline backup requirements

BMC recommends that you use a storage replication solution that can ensure data consistency between the cell database and the BMC ProactiveNet Performance Management Server (Sybase) database while BMC ProactiveNet Performance Management Server is up and running. On an enterprise where minimal downtime of BMC ProactiveNet Performance Management Server is tolerated, BMC recommends shutting down the BMC ProactiveNet Performance Management Server and copying the entire BMC ProactiveNet Performance Management installation directory. Other than the binaries, the installation directory has the cell database, Sybase data and log files, and configuration files (listed below) that are critical for bringing back the BMC ProactiveNet Performance Management Server on a secondary server during a disaster.

Backup frequency
The more current the backup, the more current the recovery. It is required to bring back all the files and database data as of the current date and time. Because the backup is

BMC Software, Inc., Confidential

14

offline, you can choose the time for daily or weekly backups that can have minimal downtime on monitoring devices and business services.

Backup files
The following files and directories are critical, and they need to be current for a successful recovery of BMC ProactiveNet Performance Management Server.

Configuration files
Adapters, instances, and integration information are available in the following files: pw\monitors\monitors\_* pw\pronto\usr_* pw\pronto\conf pw\pronto\data\admin pw\pronto\data\cell pw\custom\conf pw\integrations\ibrsd\conf pw\server\etc pw\server\etc\pncell_primary pw\server\log pw\server\tmp pw\server\var pw\admin\etc\mccscommunication

KB files
The KB files are available in the bppmInstallDirectory\pw\server\etc\pncell_serverName\kb directory.

Cell DB file
The cell database mcdb file is located in the bppmInstallDirectory\pw\server\var\pncell_serverName directory.

Database files
The following database files are located in the bppmInstallDirectory\pw\sybase directory: Transaction Log: pronto.log Data file: storm_hostName.db

15

BMC Software, Inc., Confidential

Verification of backup
The following two tests have been verified in-house to validate the procedure. However, BMC recommends that you validate this procedure on a test environment before implementing it on production. Validated the backups and procedure that can bring up the secondary server Performed the backups from the secondary server and transferred to the primary server to bring the primary server back up

Steps to follow during disaster recovery

During a disaster, you perform the following steps to make the BMC ProactiveNet Performance Management Server available on a secondary server along with integrations: 1. On the DNS server, make the necessary changes so that the DNS alias redirects to the secondary server. 2. Make BMC ProactiveNet Performance Management Server files and database files current on the secondary server. 3. Make the necessary changes on the secondary server. 4. Refresh the cache to get the secondary server IP address on servers that run remote event cells. 5. Refresh the cache to get the secondary server IP address on the BMC Remedy ITSM system.

Steps on the secondary server

This section provides the steps to perform on the secondary server that takes the place of the primary server during a disaster. 1. Where no storage replication solution is used, transfer the latest bppmInstallDirectory\pw files from the backup media onto the secondary server. 2. Restart the secondary server computer for the changes to take effect. 3. At a command prompt, run the Encrypt Utility that is available in the bppmInstallDirectory\pw\integrations\ibrsd\bin directory. The syntax is
EncryptUtility f <Absolute passfile path> -l <Absolute logfile path> -p <password>. For example:
EncryptUtility f C:\Program Files\ProactiveNet\pw\integrations\ibrsd\conf l C:\Program Files\ProactiveNet\pw\integrations\ibrsd\log p **

4. Restart the BMC ProactiveNet IBRSD service.

BMC Software, Inc., Confidential

16

5. Launch the user console by using the DNS alias name (for example, http://bppm) and validate the self monitoring by using flashcheck option.

Steps on the BMC Remedy ITSM system

This section provides the steps to perform on the BMC Remedy ITSM system. 1. At a command prompt, run the following command to refresh the cache so that the transactions can be redirected to new BMC ProactiveNet Performance Management Server.
ipconfig /flushdns

2. Restart the AR Plugin server (arplugin.exe). 3. Verify that the new incidents are getting created as applicable.

Steps on remote cell systems

This section provides the steps to perform on computers where remote event cells are running. 1. At a command prompt, run the following command to refresh the cache so that the transactions can be redirected to new BMC ProactiveNet Performance Management Server.
ipconfig /flushdns

2. Restart the remote cell. 3. Verify that the events are getting propagated from the remote cell to the BMC ProactiveNet Performance Management cell on the secondary server.

High availability and disaster recovery

The procedure and guidance provided in this white paper are implied on high availability systems where the Windows cluster feature is used to ensure the continuity of BMC ProactiveNet Performance Management Server. In a high availability environment, the files are shared rather than replicated. The procedure of having a DNS alias name is taken care by the logical name established by the Windows clustering service.

133400

17

BMC Software, Inc., Confidential