--------------------------------------------------------------------------------

From: mezher.mezher@ericsson.com [mailto:mezher.mezher@ericsson.com]

Sent: Tuesday, February 28, 2006 2:38 PM
To: Mezher Mezher (LB/ELC)
Subject: APG 40 Both Nodes Replacement

The following is a Primus(R) eServer solution:

ID: SCS453720
Domain: primus_owner
Solution Class: 4.X

Goal APG 40 Both Nodes Replacement

Fact Network: CDMA

Fact Network: GSM

Fact APG40

Fact Node: AXE MSC

Fact Node: AXE BSC

Symptom APG40 Laboratory Node Preparation

Symptom APG 40 Both Nodes Change

Symptom To Upgrade a New APG40 from Backup of other APG40 at required SW

Level

Symptom Both APG Nodes are faulty and to be replaced with new nodes

Symptom APG40 to be upgraded from CM4214 to CM4216 on Non Live Traffic

Carrying Node

Symptom Actual cloning using Ghost Multicasting is Not Possible

Symptom There is no suspected Hardware Fault on APG40 Nodes,which are to be

cloned.

Symptom It is acceptable to customer to loose old information

Cause New APG node is not on required Software Level & hence needs to be
upgraded .
Ex: APG node along with new APZ(say 2 12 40)is not on required minimum level of
software.

Cause Critical SW Failure on a Live APG40 Node

Fix Solution:

CONDITIONS:-
- The procedure is to be performed locally at site
- Two Nos of Monitor, Keyboard & Mouse are required
- VERIFIED Backup of Another APG 40 node at desired level is available in two
different DAT, one for each A and B node.
-Backup of all RELFSWX required files from the L: drive should be taken and safely
stored on DAT. These backups will need to be transferred on the new node once in
service.
-After completing this procedure, the Node will be exactly the image of the node, from
which Backup was taken .Hence all IP, User Name, Password Need to changed as per
customer requirement.
-All required site-specific information has been gathered from the live system, and a DT
file has been prepared to map these changes on the new node once in service.
-This procedure should only be performed by someone with Advance Working
Experience in APG40 and if performed in A live Network it must be performed under
close supervision of Ericsson Support.
-Knowledge of Cable and other connections on APG is must.
-Following this procedure improperly may leads to APG40 Completely Faulty.

PROCEDURE:-
1.1.1 Power-up of replacement/un configured nodes.
The two nodes are received, and the following steps will be used to power-up these nodes
such that they don't interfere with each other with regards to PDC/BDC negotiation.
1) Insert both nodes into the rack.
2) Connect a LCT, mouse and keyboard to each node.
note: At this point, only the LCT, mouse, and keyboard are connected on each node. The
nodes should not be interconnected amongst each other (SCSI, Ethernet, heartbeat, etc·)
3) Power up both nodes simultaneously by inserting the power cable in its
appropriate location in each node.
4) Wait for the nodes to come up. When the Windows NT logon screen appears, login
to each node as "Administrator" user.
5) We will now prepare DAT restore of online backups on laboratory nodes
6) Open an MS-DOS command tool. Startà runà cmd . This should be done on both
nodes.
7) Delete the contents of the D: Drive by issuing the following commands in the MS-
DOS command tool opened. This should be done on both nodes.
C:\> d: (This will change working drive to the D: drive)
D:\> del *.* (answer Y at the prompt)
D:\> rd /s /q .
8) Ensure the D: drive is empty. This should be done on both nodes.
D:\> dir (should result in no file listing)
9) Insert DAT tape labelled NodeA in the tape drive of the node on the left hand side.
10) Insert DAT tape labelled NodeB in the tape drive of the node on the right hand
side.
11) Start the Windows NT Backup Tool.
Startà Programs à Administrative tools (Common) à Backup
12) Expand the tape window, and wait while the tape driver is busy until the label of
the tape is shown. Ensure the tape label of the node on the left hand side reads Node A,
and the one on the right hand side reads Node B. This should be done on both nodes.
1.1.2 Execute DAT restore of online backups on laboratory nodes
13) Now that the contents of the DAT tape are visible, from the tape window select
the backup to be restored by clicking the checkbox next to it (it should be the only one
available).
14) Click on the "Restore" button. A new window will appear. Check that the restore
will be made to the D: drive and choose the alternatives "Restore Local Registry" and
"Restore File Permissions" by clicking in the appropriate checkboxes. Press "OK" to start
the transfer. If asked to replace files, click "Yes" to all. This should be done on both
nodes. This step should take about 40 minutes.
15) Wait until the copying is finished and then close the tape Backup tool. This
should be done on both nodes and can be done simultaneously.
1.1.3 BURserver manipulations
Now that the online backups have been stored on the D: drive, we must ensure that the
BUR server service is started such that we are able to execute the "burrestore" command.
NOTE :- IF YOU HAVE COPIED THE ZIP FILE THEN UNZIP IT USING
FOLLOWING COMMAND :-
pkzip25 -extract -dir filename.zip
Note: All steps in this section must be done on both nodes.
16) Open an MS-DOS command window.
17) Stop the burserver with the following command
C:\> net stop burserver
note: If no BUR is installed on the new node, then the following error (1060) will be
presented on the screen. Ignore that and continue.
18) Still from the MS-DOS command window, execute the following step to register
the Burserver as a service and then start it.
C:\> D:
D:\> cd Program Files
D:\Program Files> cd AP\ACS\Bin
D:\Program Files\AP\ACS\bin> burserver -service
D:\Program Files\AP\ACS\bin> net start burserver

NOTE :IF "NET START BURSERVER" IS NOT SUCCESSFUL GO TO CONTROL

PANEL AND ENABLE THE CHECKBOX FOR "ALLOWING SERVER TO
INTERECT WITH DESKTOP" FOR "BURSERVER" SERVICE.
1.1.4 Unrestored and power down
At this point the "bur" commands will be able to function since we have registered the
burserver as a service.
The next step would be to use the burrestore command to complete our restore.
19) From the MS-DOS command window, execute the following command
C:\> burrestore -stay
Imagename: nodeA_20040113
Date: 1/13/04 11:11:26 PM
Node: Lab_NodeA
Status: OK (7807 files and 945 directories copied, 0 locked files)
Restore from this backup [y=yes, n=no]: y
The currently running system will be stored on the backup partition.
Enter a description for the currently running system.
Image name: old_system
Partition 2 & 3 successfully swapped
What we did here was prepare the system to swap the D: and C: partitions. We have also
asked the system (with the -stay operand) to not reboot after the swap.
note: This command should be executed on both nodes.
20) Shutdown the node.
Startà Shut Down à Shut Down the computer
This should be done on both nodes.
21) Wait for both nodes to shutdown.
22) When the nodes have reached shutdown state remove the power cables from both
nodes simultaneously.
1.1.5 Cross Connections, power up and BIOS setup
At this point, we will make all required physical connections in order to cross connect
both nodes. It is assumed that the engineer on site is knowledgeable with regards to
where each cable ought to be connected.
23) Using a SCSI cable, connect the upper SCSI connector of NodeA to the lower
SCSI connector of NodeB. Using another SCSI cable, connect the lower SCSI connector
of NodeA to the upper SCSI connector of NodeB.
24) Connect all heartbeat Ethernet cables, serial cables and external cables.
25) The LCT, keyboard and mouse should still be connected, as we have not
disconnected them.
26) This next step will consist of powering up both nodes simultaneously. It is
important to be somewhat quick once powering up both nodes, as we will need to enter
the BIOS on both nodes as soon as they are powered up.
27) Power up both nodes simultaneously by inserting the power cable in its
appropriate location in each node.
28) Both nodes will start powering up.
29) When you see the blue "Adaptec I2O" symbol displayed on the screen press
CTRL-A to enter the Adaptec SmartROM configuration menu. This should be done on
both nodes.
note: There are two Adaptec host adapters. Be sure to press <CTRL-A> when the second
Adaptec host adapter is initializing, as described above (blue)
1.1.6 Data Disk Restore
At this point, both nodes are in BIOS mode.
We need to follow the AP, System Data Disk Restore OPI to ZAP and format the data
disks.
note: All the following steps will be done on NodeA (PDC)
30) Highlight the controller symbol (marked with #0) in the left column using
<Arrow down>.
31) Go to the configuration page of the controller by pressing <ALT-C>.
32) Make the Cluster Enable Box visible by pressing <ALT-F10><D>
33) Uncheck the Cluster Enable Box. Use <TAB> to select box and <SPACE> to
uncheck.
34) To save changes, press <ALT-S>. Confirm changes by selecting "Yes" and
pressing <ENTER>. (If necessary, press <ESC> to return from the file menu).
35) Highlight the controller symbol (marked with #0) in the left column using
<Arrow down>.
36) To activate the Zap Drive menu item in the Action menu press <ALT-F10><Z>.
(No visible change will be seen on the screen).
37) Open list of SCSI drives by pressing <ENTER>.
38) The RAID information and partition tables of the drives listed under the two
SCSI busses are to be deleted with the following procedure. Move to the first hard disk
device symbol (looks like a box) with the cursor down key.
39) Press <ALT-A> to enter the Action Menu in the menu bar and enter the Zap
Drive box by pressing the <Z> key.
40) Select the "Yes" button with the tab key to zap the drive and press <ENTER> to
confirm selection.
41) Select the "Yes" button in the next menu with the tab key to delete the partition
table and press <ENTER> to confirm selection.
42) Press <ENTER> to confirm that the changes will be taken into effect at the next
reboot.
43) Select next drive and repeat the previous 4 steps (39-42) until all 6 drives have
been zapped.
44) Press <ALT-F> to enter the file menu. Press <X> to exit and reboot. Confirm the
action with OK.
45) After the reboot, enter the SMOR bios again by pressing <CTRL-A> when the
blue "Adaptec" symbol and the text "I2O" is displayed on the screen.
46) Bring down the RAID menu by pressing <ALT-R> and select Create by using
<Arrow up>/<Arrow down> and select with <ENTER>.
47) From the new window that appears, select RAID-1 and use <TAB> key to go to
the "OK" button. Press <ENTER>.
48) Choose two disk drives which differ in the second number (SCSI channel) but
are equal in the third number (SCSI ID). Move using <Arrow up>/<Arrow down>, and
select by pressing <SPACE>. The selected disk drives will be highlighted and checked.
49) Use <TAB> to select "Done" followed by <ENTER>.
50) A new window comes up. Select Copy from (0,0,x,0) to (0,1,x,0).
51) Use <TAB> to select OK button followed by <ENTER>.
52) Repeat the previous 6 steps (46-51) until all drives are set into RAIDs. (The
procedure should be performed three times in total).
53) Set settings into hardware by pressing <ALT-F><S>. Confirm changes by
pressing <ENTER>.
54) The RAID will start building. Do not wait for completion of the RAID build.
55) Highlight the first RAID-1 entry in the left column using <Arrow up>/<Arrow
down> . (The RAID symbol looks like a cylinder).
56) Press <ALT-R> and from the menu select Stop Build.
Repeat the procedure on the previous step for all other RAID-1 entries. (The procedure
should be performed three times in total).
57) Highlight the first RAID-1 entry in the left column using <Arrow up>/<Arrow
down>.
58) Press <ALT-F10><O>. A new command, Make Optimal, will appear in the
Action menu. (No visible change will be seen on the screen).
59) Press <ALT-A> and select Make Optimal using <Arrow up>/<Arrow down>
followed by <ENTER>.
60) Repeat the procedure in the previous step for all other RAID-1 entries. (The
procedure should be performed three times in total).
61) Set settings into hardware by pressing <ALT-F><S>. Select "OK" followed by
<ENTER>.
62) Go to the configuration page of the controller by selecting the controller symbol
(marked with #0) and then pressing <ALT-C>.
63) Make the Cluster Enable Box visible by pressing <ALT-F10><D>.
64) Check the Cluster Server Enable box using <TAB> to move and <SPACE> to
check.
65) Save changes by pressing <ALT-F>. Confirm by selecting "Yes" and then press
<ENTER>.
66) Exit and reboot the system by pressing <ALT-F><X>. Confirm exit by pressing
<ENTER>.
67) Logon to the PDC as a user with administrative rights.
68) Start the Control Panel by clicking Start->Settings->Control Panel.
69) Double click on Devices.
70) Select Cluster Disk and then click on the "Startup..." button.
71) Note: If the message "The service database is locked" is presented, wait two
minutes and then try again.
72) Change Startup Type to Manual and then left click "OK". Confirm changes by
clicking "Yes".
73) Close the Devices window by clicking "Close".
74) In the Control Panel window, double click on Services.
75) Select the Cluster Server service and then click on the "Startup..." button.
76) Change Startup Type to Manual and then click "OK".
77) Select the ACS_FCH_Server service and then click on the "Startup..." button.
78) Change Startup Type to Manual and then click "OK".
79) Close the Services window by clicking "Close".
80) Reboot the node by clicking Start->Shutdown... and select Restart the computer.
81) When the system comes up again, logon to the PDC as a user with administrative
rights.
82) To enter the directory bur, use command:
 C:\> cd bur
83) To list all the files in directory, use command:
C:\BUR> dir *.ddi
84) Find the file with the most recent time and date stamp. Replace the <filename> in
the next step with the actual name of the file.
85) To format and create directories on the data disks, use command:
C:\BUR> burbackup -restoredatadiskinfo c:\bur\<filename>.ddi
note: Confirm execution of burbackup by typing "y" followed by <ENTER>. Wait for the
prompt to return.
86) All the data disk drives should have been successfully formatted.
87) Start the Control Panel by clicking Start->Settings->Control Panel.
88) Double click on Devices.
89) Select the Cluster Disk and then click on the "Startup..." button.
90) Change Startup Type to System and then click "OK".
91) Close the Devices window by clicking "Close".
92) In the Control Panel window, double click on Services.
93) Select the Cluster Server service and then click on the "Startup..." button.
94) Change Startup Type to Automatic and then click "OK".
95) Select the ACS_FCH_Server service and then click on the "Startup..." button.
96) Change Startup Type to Automatic and then click "OK".
97) Close the Services window by clicking "Close".
98) Set the timezone on the system.
99) Reboot the node by clicking Start->Shut Down and select Restart the computer.
100) Logon to the PDC as a user with administrator rights.
101) Verify that the cluster server service has started by entering the command
"prcstate"
note: Error messages can popup if the cluster is not started. Retry this step a number of
times up to 3 minutes.
102) Should everything go well, the prcstate output will show as active. At this point,
power-up NodeB (the BDC) by exiting the SMOR Bios. <ALT-F><X>. Confirm exit
selecting "Yes" and by pressing <ENTER>
103) Logon to the BDC as a user with administrative rights.
104) Verify that the cluster server service has started by entering the command
"prcstate". The expected printout is "passive".
note: Error messages can popup if the cluster is not started. Retry this step a number of
times up to 3 minutes.
105) Set the time zone on the system
106) Backup both nodes in Hard Disk & Dat Tape

1.1.7 Check page file settings and F: partition creation

The F: partition will probably not have been created on the replacement nodes. This
section will outline the required steps to be followed to ensure the successful creation of
the F: partition. This section should be done on both nodes.
107) Start the Control Panel (Startà Settings à Control Panel)
108) Double click on the System icon to view System Properties. Choose the
Startup/Shutdown tab.
109) Verify the ·Write Debugging Information to· setting.
Note: If you get an error message then press OK.
110) If the debugging information is written to F:\MEMORY.dmp and the F: drive is
created then the rest of this section can be skipped and you can proceed to section 2.1.8.
If not, continue with the next steps.
111) Use the Explorer to see which label the C: partition has.
112) Is the label of the C: partition "ntserv"?
113) If the C:\ partition label is ntserv, skip the next step. If it is not, the next step
MUST be performed.
114) In order to proceed a restore operation has to be made so that the C: partition
has the label "ntserv".
Caution: It is assumed that you have already followed step 106 to backup the C-drive.
Failure to do so will render the system unusable.
Execute the command ·burrestore· and enter ·y· when prompted. The system will
reboot. Wait until the system reboots. Once back to the Windows NT login screen, login
as administrator. Use the Explorer to see which label the C: partition has. At this point,
the partition must show ·ntserv·. Close the window and then continue.
115) Start the Disk Administrator tool. STARTà Programs à Administrative Tools
(Common) à Disk Administrator. Answer ·yes· if you get a popup menu.
116) Select the free space on the system disk that has the E:, C: and D: partitions.
117) On the partition menu, click create. Answer Yes on the pop up menu. In the
"Create primary partition" menu, enter the size of 4000 of the new partition and click OK.
Answer Yes on the pop up menu. It will show the size 4001 MB.
118) Select the new partition and on the Tools menu, click Assign Drive Letter. In the
Assign Drive Letter menu, select the drive letter "F", and click OK.
119) Select the new partition and on the Partition menu, click on "Commit Changes
Now" Answer Yes on the pop up menu. Answer OK on the pop up menu.
note: Do not follow the recommendation, ·It is recommended that you update the
emergency repair configuration information and create a new Emergency Repair disk· if
you are prompted with it.
120) Select the new partition, and on the Tools menu click Format. In the Format
menu, select NTFS as File system, enter the text "ntdump" as Volume Label, and select
the Format Options "Quick Format". Then press Start. Answer OK in the next pop up
menu and click OK when the format is completed (takes a couple of seconds). Close the
Format Menu. Close the Disk Administrator.
121) Start a command prompt window and change partition with the command:
C:\> F:
122) Create the ACS\LOGS directory with the command:
F:\> mkdir ACS
F:\> cd ACS
F:\ACS> mkdir logs
F:\ACS> cd logs
F:\ACS\logs>
123) Close the command window
124) Start the Windows NT Explorer (STARTà Programsà Windowsà NT Explorer)
125) Right click on ntdump (F:) and click on properties. Choose the security tab and
then click on Permissions. Mark the checkbox, "Replace permissions on Subdirectories".
Verify that the checkbox "Replace Permissions on Existing Files" is marked. Press the
add button. Wait for the new window "Add users and Groups". Click on Administrators
and press the Add button. Change type of access to Full Control and press OK. Click on
·Everyone· and press the Remove button. Press the OK button and answer Yes on the pop
up menu.
126) Choose the Security tab again and then click on Permissions. Verify that the
checkbox "Replace Permissions on Subdirectories" is not marked. Verify that the
checkbox "Replace Permissions on Existing Files" is marked. Press the Add button. Wait
for the new window "Add users and groups". Click on SYSTEM and press the Add
button. Change "Type of Access" to Full Control and press OK. Press the Add tab. Wait
for the new window "Add users and Groups". Click on everyone and press the Add
button. Check that the "Type of Access" is Read and press OK. Press the OK button.
Press OK on the Properties menu, Close Windows NT Explorer.
NOTE: DO NOT FORGET TO PERFORM THIS PROCEDURE ON BOTH NODES.

Note This procedure can also be used to restore APG40 in a live network if there is

1.No valid backup is available for the node and hence using the backup of other APG is
the only option left.

OR

2.Valid backup is available, but both nodes are to be replaced simultaneously.

.However in addition to above conditions following should also fulfill:-

-For the duration of Procedure all CP-AP communication will be unavailable .
Take all precautions for this downtime for charging etc.
-Prepare The Procedure & Data Transcript to be used after the activity including the
following (But not Limited To):-

-IP Change
-User Administrator Change
-Restore of RELFSW files
-STS objects and measurements
-CDH Destinations
-RDT Parameters
-Billing center Definitions
-FTP Directories
-AFP Files
-CPF Files
-Antivirus Definitions
-Other Site Specific Changes.