Sie sind auf Seite 1von 18

<RNC & Other Data Elements Health Check MOP>

RNC & other Data elements basic


troubleshooting

Document Type: MOP


Document Number: (ALU-HSD-20062011-1)

Prepared By

Reviewed By

Authorized By

Pravin
Kokate

Kishan S Rawat

Hiren Joshi

Release On :- 20-June-2011
Validity: Till Further Notification

Page 1 of 18

ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

Index
1.
2.
3.
4.
5.

Active Sessions in RNC


VRRP status on both the extreme switches
Port Status on Both Extreme Switches
Port Utilization on Both Extreme Switches
Connectivity of PDSN & RAN (external & Internal) devices with
RNC
a. Checking connectivity of PDSN & AAA from Ext Switch
b. Checking connectivity of PDSN & AAA from TP
6. Extreme Switch logs for fault analysis
7. Flx AP & TP alarms.
8. EVDO PCMD file
9. AP & TP card status check
10. HSD cells alarms
11. ROP analysis
a. PDSN & AAA unreachable logs from ROP
12. Live DO sessions in RNC
13. Other Data element health check
a. C10K router
b. 3400 Switch
c. MLS 4503 Switch
d. 4948 Switch

Page 2 of 18

ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

RNC Health Check MOP


Check Active Sessions in RNC
omp_delhi2 rakesh> telnet flx201 ----- Login into flx
Trying 97.241.151.12...
Connected to flx201.
Escape character is '^]'.
login: root
Password:
Last login: Mon Nov 22 11:56:34 from flx202
*************************************************
************ flx01 -> 1xEV-DO R32.0***********
*************************************************
R32 SU2 TMP E
You have new mail.
TERM is set to "vt100".
WARNING: YOU ARE SUPERUSER !!
[flx201]-> tools --- access required folder
/flx/ONEXEVc/current/release/tools
[flx201]-> cd tss
/flx/ONEXEVc/current/release/tools/tss
[flx201]-> su onexev
[flx201]-> ./show_pdsns
This command should be executed only on HOC AP (flx202).
Exiting ./show_pdsns... ---- Whenever such error comes ,please access flx202
[flx201]-> rsh flx202 --- login to flx202 from 201
Last login: Mon Dec 13 15:37:20 from apomp

*************************************************
************ flx02 -> 1xEV-DO R32.0***********
*************************************************
R32 SU2 TMP E
TERM is set to "vt100".
[flx202]-> tools
/flx/ONEXEVc/current/release/tools
[flx202]-> cd tss
/flx/ONEXEVc/current/release/tools/tss
[flx202]-> su onexev
Password:1xevadm
[flx202]-> ./show_pdsns

Check the Sessions on all TP cards as below, please ensure below points.
1. TPs should take equal session load
2. All TPs must be reachable with both the PDSNs (Mumbai &
Chennai).
Page 3 of 18

ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

***************************************************************************************
DATE : 13 Dec 2010

TIME : 15:47:10

***************************************************************************************
+-------------------------------------------------------------------------------------+
| TP ID : 321
|
| TP IP Address : 172.12.38.9
|
| TP Status : Up
|
+-----------------+---------+------------------+-------+------------+-----------------+
| PDSN IP Address | Type | In File |
State | SPI | Time Drift | Sessions |
+-----------------+---------+------------------+-------+------------+-----------------+
| 10.239.200.128 | PDSN | yes | Reachable
| 100 |
-14 | 432 |
| 10.239.201.128 | PDSN | yes | Reachable
| 100 |
-14 |
0|
+-----------------+---------+------------------+-------+------------+-----------------+
***************************************************************************************
+-------------------------------------------------------------------------------------+
| TP ID : 322
|
| TP IP Address : 172.12.38.10
|
| TP Status : Up
|
+-----------------+---------+------------------+-------+------------+-----------------+
| PDSN IP Address | Type | In File |
State | SPI | Time Drift | Sessions |
+-----------------+---------+------------------+-------+------------+-----------------+
| 10.239.200.128 | PDSN | yes | Reachable
| 100 |
-14 | 430 |
| 10.239.201.128 | PDSN | yes | Reachable
| 100 |
-14 |
0|
+-----------------+---------+------------------+-------+------------+-----------------+
***************************************************************************************
+-------------------------------------------------------------------------------------+
| TP ID : 323
|
| TP IP Address : 172.12.38.11
|
| TP Status : Up
|
+-----------------+---------+------------------+-------+------------+-----------------+
| PDSN IP Address | Type | In File |
State | SPI | Time Drift | Sessions |
+-----------------+---------+------------------+-------+------------+-----------------+
| 10.239.200.128 | PDSN | yes | Reachable
| 100 |
-14 | 402 |
| 10.239.201.128 | PDSN | yes | Reachable
| 100 |
-14 |
0|
+-----------------+---------+------------------+-------+------------+-----------------+
***************************************************************************************
+-------------------------------------------------------------------------------------+
| TP ID : 324
|
| TP IP Address : 172.12.38.12
|
| TP Status : Up
|
+-----------------+---------+------------------+-------+------------+-----------------+
| PDSN IP Address | Type | In File |
State | SPI | Time Drift | Sessions |
+-----------------+---------+------------------+-------+------------+-----------------+
| 10.239.200.128 | PDSN | yes | Reachable
| 100 |
-14 | 407 |
| 10.239.201.128 | PDSN | yes | Reachable
| 100 |
-14 |
0|
+-----------------+---------+------------------+-------+------------+-----------------+
***************************************************************************************
+-------------------------------------------------------------------------------------+
| TP ID : 326
|
| TP IP Address : 172.12.38.14
|
| TP Status : Up
|
+-----------------+---------+------------------+-------+------------+-----------------+
| PDSN IP Address | Type | In File |
State | SPI | Time Drift | Sessions |

Page 4 of 18

ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

+-----------------+---------+------------------+-------+------------+-----------------+
| 10.239.200.128 | PDSN | yes | Reachable
| 100 |
-14 | 415 |
| 10.239.201.128 | PDSN | yes | Reachable
| 100 |
-14 |
0|
+-----------------+---------+------------------+-------+------------+-----------------+
***************************************************************************************
+-------------------------------------------------------------------------------------+
| TP ID : 328
|
| TP IP Address : 172.12.38.16
|
| TP Status : Up
|
+-----------------+---------+------------------+-------+------------+-----------------+
| PDSN IP Address | Type | In File |
State | SPI | Time Drift | Sessions |
+-----------------+---------+------------------+-------+------------+-----------------+
| 10.239.200.128 | PDSN | yes | Reachable
| 100 |
-14 | 404 |
| 10.239.201.128 | PDSN | yes | Reachable
| 100 |
-14 |
0|
+-----------------+---------+------------------+-------+------------+-----------------+

Check vrrp status on both the extreme switches


[flx202]-> telnet switcha
Trying 172.12.38.60...
Connected to switchA.
Escape character is '^]'.
telnet session telnet0 on /dev/ptyb0
login: root
password:newsys11
ExtremeXOS
Copyright (C) 2000-2006 Extreme Networks. All rights reserved.
Protected by US Patent Nos: 6,678,248; 6,104,700; 6,766,482; 6,618,388; 6,034,957;
6,859,438; 6,912,592; 6,954,436; 6,97
7,891; 6,980,550; 6,981,174; 7,003,705; 7,012,082.
=============================================================
Press the <tab> or '?' key at any time for completions.
Remember to save your configuration changes.
* X450a-UNC21_A.1 # sh vrrp
Note- VRIDs must be in Master backup mode, if VRID 200 is Master on Extreme
Switch A then same will be Backup in Extreme Switch B
Please check the same in Extreme Switch B
VLAN Name VRID Pri Virtual IP Addr State Master Mac Address
CABINET(En) 0200 254 172.12.38.62 MSTR 00:00:5e:00:01:c8
CABINET(En) 0201 253 172.12.17.158 BKUP 00:00:5e:00:01:c9
EXTERNA(En) 0002 254 172.12.1.39
MSTR 00:00:5e:00:01:02
EXTERNA(En) 0024 253 172.12.9.39
BKUP 00:00:5e:00:01:18
En-Enabled, Ds-Disabled, Pri-Priority, T-Advert Timer, P-Preempt
TP-Tracked Pings, TR-Tracked Routes, TV-Tracked VLANs
* X450a-UNC21_A.2 #
Page 5 of 18

ALUMS TEC RAN

TP/TR/TV/P/T
1 0 0Y0
1 0 0Y0
0 0 0Y0
0 0 0Y0

<RNC & Other Data Elements Health Check MOP>

Check Port status on both the extreme switches


* X450a-UNC21_A.2 # show port information
Note: Check the port status; it should be Active/Ready. All used ports should indicate
E at the beginning, it means Enabled. If D observed then that port is not in use & kept
Disabled. Please check the same for Extreme switch B as well.
Port

Flags

Link
Link Num Num Num Jumbo QOS Load
State ELSM UPS STP VLAN Proto Size profile Master
============================================================
1
Em-la---e--fMB- active - 2 1 1
1 9216 none 1 a
2
Em-la---e--fMB- active - 2 1 1
1 9216 none 1 a
3
Em-la---e--fMB- active - 2 1 1
1 9216 none 1 a
4
Em-la---e--fMB- active - 2 1 1
1 9216 none 1 a
5
Em------e--fMB- active - 19 1 1
1 9216 none
6
Em------e--fMB- active - 21 1 1
1 9216 none
7
Em------e--fMBready - 0 1 1
1 9216 none
8
Em------e--fMBready - 0 1 1
1 9216 none
9
Em------e--fMB- active - 11 1 1
1 9216 none
10
Em------e--fMB- active - 7 1 1
1 9216 none
11
Em------e--fMBready - 0 1 1
1 9216 none
12
Em------e--fMBready - 0 1 1
1 9216 none
13
Em------e--fMB- active - 48 1 1
1 9216 none
14
Em------e--fMBready - 0 1 1
1 9216 none
15
Em------e--fMBready - 0 1 1
1 9216 none
16
Em------e--fMBready - 0 1 1
1 9216 none
17
Em------e--fMB- active - 40 1 1
1 9216 none
18
Em------e--fMB- active - 47 1 1
1 9216 none
19
Em------e--fMBready - 0 1 1
1 9216 none
20
Em------e--fMBready - 0 1 1
1 9216 none
21
Em------e--fMB- active - 20 1 1
1 9216 none
22
Em------e--fMB- active - 20 1 1
1 9216 none
23
Em------e--fMBready - 0 1 1
1 9216 none
24
Em------e--fMBready - 0 1 1
1 9216 none
25
Em------e--fMB- active - 12 1 1
1 9216 none
26
Em------e--fMB- active - 8 1 1
1 9216 none
27
Em------e--fMBready - 0 1 1
1 9216 none
28
Em------e--fMBready - 0 1 1
1 9216 none
29
Em------e--fMB- active - 21 1 1
1 9216 none
30
Em------e--fMBready - 0 1 1
1 9216 none
31
Em------e--fMBready - 0 1 1
1 9216 none
32
Em------e--fMBready - 0 1 1
1 9216 none
33
Em------e--fMB- active - 18 1 1
1 9216 none
34
Em------e--fMB- active - 34 1 1
1 9216 none
35
Em------e--fMBready - 0 1 1
1 9216 none
36
Em------e--fMBready - 0 1 1
1 9216 none
37
Em-la---e--fMB- active - 2 1 1
1 9216 none 1 a
38
Em-la---e--fMB- active - 2 1 1
1 9216 none 1 a
39
Em-la---e--fMB- active - 2 1 1
1 9216 none 1 a
Page 6 of 18

ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

40
Em-la---e--fMB- active - 2 1 1
1 9216 none 1 a
41
Em------e--fMBready - 0 0 0
0 9216 none
42
Em------e--fMBready - 0 0 0
0 9216 none
43
Em------e--fMBready - 0 0 0
0 9216 none
44
Em------e--fMBready - 0 0 0
0 9216 none
45
Em-la---e--fMB- active - 3 0 2
0 9216 none 45 a
46
Em-la---e--fMB- active - 3 0 2
0 9216 none 45 a
47
Dm------e--fMBready - 0 0 2
0 9216 none
48
Dm------e--fMBready - 0 0 2
0 9216 none
49
Em------e--fMBready - 0 0 0
0 9216 none
50
Em------e--fMBready - 0 0 0
0 9216 none
=============================================================
> indicates Port Display Name truncated past 8 characters
Flags : a - Load Sharing Algorithm address-based, D - Port Disabled,
e - Extreme Discovery Protocol Enabled, E - Port Enabled,
g - Egress TOS Enabled, j - Jumbo Frame Enabled,
l - Load Sharing Enabled, m - MACLearning Enabled,
n - Ingress TOS Enabled, o - Dot1p Replacement Enabled,
P - Software redundant port(Primary),
R - Software redundant port(Redundant),
q - Background QOS Monitoring Enabled,
s - diffserv Replacement Enabled,
v - Vman Enabled, f - Unicast Flooding Enabled,
M - Multicast Flooding Enabled, B - Broadcast Flooding Enabled
L - Extreme Link Status Monitoring Enabled
* X450a-UNC21_A.3 #

Check Port utilization on both the extreme switches


Note- If Rx & Tx packets observed on particular port means that port is in use for
traffic. This is useful to check on which ports PDSN & Router traffic is going. In
below example Port 45 is connected towards Router through MLS & port 46 is
connected towards PDSN through MLS-CCR Router, & they are taking traffic means
Extreme switch A is active fro traffic.
* X450a-UNC21_A.3 # show port utilization
Link Utilization Averages
Mon Dec 13 17:27:47 2010
Port Link Receive
Peak Rx
Transmit
Peak Transmit
Status packets/sec packets/sec packets/sec packets/sec
=========================================================================
1
A
4756
11425
2492
9349
2
A
1640
4784
853
2564
3
A
5464
15852
3492
8335
4
A
1278
8262
777
2937
5
A
0
1
4
2048
6
A
0
1
4
2048
7
R
0
0
0
0
8
R
0
0
0
0
9
A
383
4157
409
8204
10
A
612
4162
610
8144
11
R
0
0
0
0
12
R
0
0
0
0
Page 7 of 18

ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

13
A
9673
21775
8991
19767
14
R
0
0
0
0
15
R
0
0
0
0
16
R
0
0
0
0
=========================================================================
> indicates Port Display Name truncated past 8 characters
Link Status : A-Active, R-Ready, NP-Port Not Present, L-Loopback
Link Utilization Averages
Mon Dec 13 17:27:48 2010
Port Link Receive
Peak Rx
Transmit
Peak Transmit
Status packets/sec packets/sec packets/sec packets/sec
=========================================================================
17
A
8724
21696
8061
20074
18
A
9089
15496
8496
16600
19
R
0
0
0
0
20
R
0
0
0
0
21
A
0
4
6
1608
22
A
0
3
6
1608
23
R
0
0
0
0
24
R
0
0
0
0
25
A
433
8182
484
4153
26
A
596
3656
618
7191
27
R
0
0
0
0
28
R
0
0
0
0
29
A
1
6777
6
6050
30
R
0
0
0
0
31
R
0
0
0
0
32
R
0
0
0
0
=========================================================================
> indicates Port Display Name truncated past 8 characters
Link Status : A-Active, R-Ready, NP-Port Not Present, L-Loopback
Link Utilization Averages
Mon Dec 13 17:27:54 2010
Port Link Receive
Peak Rx
Transmit
Peak Transmit
Status packets/sec packets/sec packets/sec packets/sec
=========================================================================
33
A
8185
20234
7594
18269
34
A
1
7625
5
6592
35
R
0
0
0
0
36
R
0
0
0
0
37
A
968
6458
556
2663
38
A
6254
16718
3510
9305
39
A
5378
13267
3405
9328
40
A
2290
7069
1117
3263
41
R
0
0
0
0
42
R
0
0
0
0
43
R
0
0
0
0
44
R
0
0
0
0
45
A
11108
25200
17477
29861
46
A
11288
26525
19405
29223
47
R
0
0
0
0
48
R
0
0
0
0
=========================================================================
> indicates Port Display Name truncated past 8 characters

Page 8 of 18

ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

Check connectivity of PDN & RAN (external & Internal)


devices with RNC
Note- UNC IP sheet & all data elements IP sheet are required for the same. If not
available then get it from respective SMEs or IPBH team. Login into extreme
switch & give ping command. Please follow correct syntax.
Command syntax ping <Destination IP> from <Internal PDN cabinet Sw IP>
* X450a-UNC21_A.1 # ping 10.239.200.128 from 172.12.17.156
Ping(ICMP) 10.239.200.128: 4 packets, 8 data bytes, interval 1 second(s).
16 bytes from 10.239.200.128: icmp_seq=0 ttl=61 time=22 ms
16 bytes from 10.239.200.128: icmp_seq=1 ttl=61 time=21 ms
16 bytes from 10.239.200.128: icmp_seq=2 ttl=61 time=20 ms
16 bytes from 10.239.200.128: icmp_seq=3 ttl=61 time=20 ms
--- 10.239.200.128 ping statistics --4 packets transmitted, 4 received, 0% loss
round-trip min/avg/max = 20/21/22 ms
* X450a-UNC21_A.2 # ping 10.239.201.128 from 172.12.17.156
Ping(ICMP) 10.239.201.128: 4 packets, 8 data bytes, interval 1 second(s).
16 bytes from 10.239.201.128: icmp_seq=0 ttl=60 time=30 ms
16 bytes from 10.239.201.128: icmp_seq=1 ttl=60 time=30 ms
16 bytes from 10.239.201.128: icmp_seq=2 ttl=60 time=31 ms
16 bytes from 10.239.201.128: icmp_seq=3 ttl=60 time=30 ms
Check Connectivity for below elements
1. Mumbai & Chennai PDSN (Mumbai PDSN ip 10.239.200.128 & Chennai
10.239.201.128, same can also be taken from HDRF tab in OMC GUI)
2. All AAA servers (Mumbai AAA IPs- 220.224.134.162 & 220.224.134.170
Banglore AAA IPs- 220.224.135.162 & 220.224.135.170. same can also be
taken from HDRF tab in OMC GUI)
3. External RAN C10K & RDN CCR Router VRRP & physical Ips (same can be
taken from EXTERNAL RAN sheet)
4. External PDN RDN CCR Router VRRP & physical Ips (same can be taken
from EXTERNAL PDN sheet)

Page 9 of 18

ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

Note: Incase of any issue in any TP related to connectivity in ./show_pdsns as per


earlier point, the connectivity from AP & TP also can check with ping command.
[flx201]-> telnet F20.D1.S4 --- Login into TP card (D=Drawer S=Slot of TP in
drawer)
Trying 172.12.38.9...
Connected to F20.D1.S4.
Escape character is '^]'.
VxWorks login: lucent
Password:password
-> ping "10.239.200.128" ---- Ping Destination IP address
PING 10.239.200.128 (84=20+8+56 bytes)
64 bytes from 10.239.200.128: icmp_seq=0. time=16. ms
64 bytes from 10.239.200.128: icmp_seq=1. time=16. ms
64 bytes from 10.239.200.128: icmp_seq=2. time=16. ms
64 bytes from 10.239.200.128: icmp_seq=3. time=16. ms
64 bytes from 10.239.200.128: icmp_seq=4. time=16. ms
64 bytes from 10.239.200.128: icmp_seq=5. time=16. ms
64 bytes from 10.239.200.128: icmp_seq=6. time=16. ms
64 bytes from 10.239.200.128: icmp_seq=7. time=16. ms
64 bytes from 10.239.200.128: icmp_seq=8. time=16. ms
64 bytes from 10.239.200.128: icmp_seq=9. time=16. ms
64 bytes from 10.239.200.128: icmp_seq=10. time=16. ms
64 bytes from 10.239.200.128: icmp_seq=11. time=16. ms

Take extreme switch logs for analysis of faults, check for the
time stamp.
* X450a-UNC21_A.1 # show log
12/13/2010 19:14:40.27 <Info:AAA.authPass> Login passed for user root through telnet (172.12.38.1)
12/13/2010 19:10:50.29 <Info:AAA.logout> User root logout from telnet (172.12.38.3)
12/13/2010 19:10:47.07 <Info:AAA.authPass> Login passed for user root through telnet (172.12.38.3)
12/13/2010 19:05:36.65 <Info:AAA.logout> User root logout from telnet (172.12.38.3)
Page 10 of 18

ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

12/13/2010 19:05:32.42 <Info:AAA.authPass> Login passed for user root through telnet (172.12.38.3)
12/13/2010 19:00:21.67 <Info:AAA.logout> User root logout from telnet (172.12.38.3)
12/13/2010 19:00:17.41 <Info:AAA.authPass> Login passed for user root through telnet (172.12.38.3)
12/13/2010 18:55:54.41 <Info:EPM.cpu> CPU utilization monitor: process aaa consumes 86 % CPU
12/13/2010 18:55:52.29 <Info:AAA.logout> User root logout from telnet (172.12.38.1)
12/13/2010 18:55:06.63 <Info:AAA.logout> User root logout from telnet (172.12.38.3)
12/13/2010 18:55:02.39 <Info:AAA.authPass> Login passed for user root through telnet (172.12.38.3)
12/13/2010 18:49:52.05 <Info:AAA.logout> User root logout from telnet (172.12.38.3)
12/13/2010 18:49:47.76 <Info:AAA.authPass> Login passed for user root through telnet (172.12.38.3)
12/13/2010 18:44:38.25 <Info:AAA.logout> User root logout from telnet (172.12.38.3)
12/13/2010 18:44:35.04 <Info:AAA.authPass> Login passed for user root through telnet (172.12.38.3)
12/13/2010 18:44:34.49 <Info:EPM.cpu> CPU utilization monitor: process aaa consumes 90 % CPU
12/13/2010 18:39:25.16 <Info:AAA.logout> User root logout from telnet (172.12.38.3)
12/13/2010 18:39:22.58 <Info:AAA.authPass> Login passed for user root through telnet (172.12.38.1)
12/13/2010 18:39:20.87 <Info:AAA.authPass> Login passed for user root through telnet (172.12.38.3)
12/13/2010 18:34:08.86 <Info:AAA.logout> User root logout from telnet (172.12.38.3)
12/13/2010 18:34:06.29 <Info:AAA.authPass> Login passed for user root through telnet (172.12.38.3)
12/13/2010 18:28:54.23 <Info:AAA.logout> User root logout from telnet (172.12.38.3)
12/13/2010 18:28:51.28 <Info:AAA.authPass> Login passed for user root through telnet (172.12.38.3)
12/13/2010 18:23:40.54 <Info:AAA.logout> User root logout from telnet (172.12.38.3)

Check alarms on APs & TPs


Login into OMP & refer below steps for the same.
omp_delhi2 mahesh> telnet flx201
Trying 97.241.151.12...
Connected to flx201.
Escape character is '^]'.
login: root
Password:
Last login: Wed Jun 15 03:46:01 from apomp
*************************************************
************ flx01 -> 1xEV-DO R34.0 **************
*************************************************
R34 SU1 Cft G
You have new mail.
TERM is set to "vt100".
WARNING: YOU ARE SUPERUSER !!
[flx201]-> util
/flx/ONEXEVc/current/release/util
[flx201]-> su onexev
[flx201]-> ./CLI
Error: Cannot execute APCLI on this AP.
Please invoke evcstat to identify the APCLI AP.
Exiting..... --- Some times process will be in other AP
[flx201]-> rsh flx202 ----Login into other AP
Last login: Mon Jun 20 15:06:39 from apomp
*************************************************
************ flx02 -> 1xEV-DO R34.0 *************
*************************************************
R34 SU1 Cft G
TERM is set to "vt100".
[flx202]-> util
/flx/ONEXEVc/current/release/util
[flx202]-> su onexev
Password:
[flx202]-> ./CLI
Page 11 of 18

ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

AP CLI> op:tp 321,status --- To get the status for TP


M 12 OP:TP 321, STATUS! IP COMMAND ACCEPTED
ORIGINATING COMMAND #8.113181
2011-06-20 15:12:25 REPORT #000001 FINAL
06/20/11 15:12:25 #000001
AP CLI>
M 12 OP:TP 321, STATUS! COMPLETED
EQUIP STATE: EQUIPPED
FLEX STATE: ACTIVE
ORIGINATING COMMAND #8.113181
2011-06-20 15:12:25 REPORT #000001 FINAL
06/20/11 15:12:25 #000002
AP CLI> op:tp 321,alarm -- To get the alarms for TP
M 12 OP:TP 321, ALARM! PF
ORIGINATING COMMAND #8.178717
2011-06-20 15:12:37 REPORT #000001 FINAL
06/20/11 15:12:37 #000003
AP CLI>
M 12 OP:TP 321, ALARM! COMPLETED
No alarms associated with TP 321
ORIGINATING COMMAND #8.178717
2011-06-20 15:12:37 REPORT #000000 FINAL
06/20/11 15:12:37 #000004
AP CLI> op:ap 201,status ---To get the alarms for TP
M 12 OP:AP 201, STATUS! IP COMMAND IN PROGRESS
ORIGINATING COMMAND #8.244253
2011-06-20 15:12:45 REPORT #000001 FINAL
06/20/11 15:12:45 #000005
AP CLI>
M 12 OP:AP 201, STATUS! COMPLETED
AP 201 FLEXSTATE=ACTIVE
AP 201 EQUIPSTATE=EQUIPPED
ORIGINATING COMMAND #8.244253
2011-06-20 15:12:45 REPORT #000001 FINAL
06/20/11 15:12:45 #000006
AP CLI> op:ap 201,alarm
M 12 OP:AP 201, ALARM! PF
ORIGINATING COMMAND #8.309789
2011-06-20 15:12:53 REPORT #000001 FINAL
06/20/11 15:12:53 #000007
AP CLI>
M 12 OP:AP 201, ALARM! COMPLETED
No alarms associated with AP 201
ORIGINATING COMMAND #8.309789
2011-06-20 15:12:53 REPORT #000000 FINAL
06/20/11 15:12:53 #000008
AP CLI>

Page 12 of 18

ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

Go to Hdrfframe to check alarms on APs & TPs. Just right click on AP or TP & give
Poke command OP: ALARM. Check for all Boards. Incase of any alarm take help
from SME to clear the same.

Page 13 of 18

ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

Check for PCMD files are getting generating or not


omp_delhi2 rakesh> cd /omp-data/logs/evdo_pcmd
omp_delhi2 rakesh> ls -ltr
total 455432
-rw-r----- 1 root omptech 5652033 Dec 13 09:59 10121310.EVDOPCMD.gz
-rw-r----- 1 root omptech 8409494 Dec 13 10:59 10121311.EVDOPCMD.gz
-rw-r----- 1 root omptech 9227777 Dec 13 11:59 10121312.EVDOPCMD.gz
-rw-r--r-- 1 root omptech 3145 Dec 13 12:06 err_activity_loga
-rw-r----- 1 root omptech 10509355 Dec 13 12:59 10121313.EVDOPCMD.gz
-rw-r----- 1 root omptech 11198514 Dec 13 13:59 10121314.EVDOPCMD.gz
-rw-r----- 1 root omptech 10902058 Dec 13 14:59 10121315.EVDOPCMD.gz
-rw-r----- 1 root omptech 11087851 Dec 13 15:59 10121316.EVDOPCMD.gz
-rw-r----- 1 root omptech 11075361 Dec 13 16:59 10121317.EVDOPCMD.gz
-rw-r----- 1 root omptech 71023114 Dec 13 17:59 10121318.EVDOPCMD
-rw-r----- 1 root omptech 66110114 Dec 13 18:59 10121319.EVDOPCMD
-rw-r----- 1 root omptech 17761939 Dec 13 19:16 10121320.EVDOPCMD --- Check for
current Time stamp & file size.

All TP card & flx AP status


[flx201]-> tools
/flx/ONEXEVc/current/release/tools
[flx201]-> cd tss
/flx/ONEXEVc/current/release/tools/tss
[flx201]-> su onexev
[flx201]-> ls
ps_evc
show_pdsns stat_machine
[flx201]-> ./stat_machine
+-----------------------------------------------------------------------------+
| Host Name | Host ID | Drw/Slot | State | Type of Unit | Release |
+========================================================================
+========================================================================
| flx201
| 84e6b9c4 | 1/3 | ACT | ------------ | EV32.0.0002TE|
| F20.D1.S4 | 321
| 1/4 | ACT | ------------ | EV32.0.0002TE|
| F20.D2.S4 | 323
| 2/4 | ACT | ------------ | EV32.0.0002TE|
+-----------------------------------------------------------------------------+
| flx202
| 84ea6a1c | 2/3 | ACT | LEAD
| EV32.0.0002TE|
| F20.D1.S5 | 322
| 1/5 | ACT | ------------ | EV32.0.0002TE|
| F20.D2.S5 | 324
| 2/5 | ACT | ------------ | EV32.0.0002TE|
+========================================================================
| flx203
| 84e6989c | 3/3 | ACT | DB
| EV32.0.0002TE|
+-----------------------------------------------------------------------------+
| flx204
| 84e6a1ec | 4/3 | ACT | ------------ | EV32.0.0002TE|
| F20.D3.S5 | 326
| 3/5 | ACT | ------------ | EV32.0.0002TE|
| F20.D4.S5 | 328
| 4/5 | ACT | ------------ | EV32.0.0002TE|
+========================================================================
NOTE: Any 0.0.0.0 in HOST column for TP indicates - UNABLE TO GET TP HOST
INFORMATION
[flx201]->

Page 14 of 18

ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

Check for BTS alarm for HSD cell sites


Login into OMP TICLI & give command OP: ALARM & search for below points
1. To check URC card out of service search for DO TRAFFIC LINK
2. To check EVM card alarms search for MODEM

Check for Critical ROP massages


Note- With the help of below steps HSD outage can be prevented, grep below
attached alarm to check the connectivity of RNC with all cell sites. Incase too many
cells found with such error then escalate the issue to SME. This is service affecting
alarm & need close monitoring.
omp_delhi2 rakesh> cd /omp-data/logs/OMPROP1
omp_delhi2 rakesh> ls
101115.APX.gz 101119.APX.gz
101209.APX 101213.APX
101116.APX.gz 101120.APX.gz
101210.APX 101214.APX
101117.APX.gz 101121.APX.gz
101211.APX nohup.out
101118.APX.gz 101122.APX.gz
101212.APX

101123.APX.gz 101127.APX.gz 101201.APX.gz 101205.APX.gz


101124.APX.gz 101128.APX.gz 101202.APX.gz 101206.APX.gz
101125.APX.gz 101129.APX.gz 101203.APX.gz 101207.APX.gz
101126.APX.gz 101130.APX.gz 101204.APX.gz 101208.APX.gz

omp_delhi2 rakesh> cgrep -R "LOST COMM WITH BACKUP EVDO RNC OHM" 101214.APX| more
12 REPT:CELL 177 CDM 3, CCU(EVM) 6, HEH
SUPPRESSED MSGS: 0
ERROR TYPE: ONEBTS MODULAR CELL ERROR
CLEARED: LOST COMM WITH BACKUP EVDO RNC OHM
02 03 00 00 00 00 00 00
00 00 00 00 00 00 00 00
00 00 00 00
12/14/10 13:12:26 #309384
08 REPT:CELL 161 CDM 3, CCU(EVM) 6, HEH
SUPPRESSED MSGS: 0
ERROR TYPE: ONEBTS MODULAR CELL ERROR
CLEARED: LOST COMM WITH BACKUP EVDO RNC OHM
02 03 00 00 00 00 00 00
00 00 00 00 00 00 00 00
00 00 00 00
12/14/10 15:08:14 #359809
omp_delhi2 rakesh>

Page 15 of 18

ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

Check connectivity of AAA & PDSN with RNC in ROP


Note- To check the connectivity between RNC to AAA/PDSN give below command.
This is service affecting alarm & need close monitoring.
omp_delhi2 rakesh> cd /omp-data/logs/OMPROP1
omp_delhi2 rakesh> cgrep -R "UNREACH" 101214.APX| more
**05 REPT: EVDO: TP 323, ALARM
AAA 220.224.134.162: UNREACHABLE
PERCEIVED SEVERITY: MAJ, CAUSE: Loss Of Signal
2010-12-14 02:05:54 REPORT #000001 FINAL
12/14/10 02:05:36 #127011
**05 REPT: EVDO: TP 323, ALARM
AAA 220.224.134.170: UNREACHABLE
PERCEIVED SEVERITY: MAJ, CAUSE: Loss Of Signal
2010-12-14 02:06:02 REPORT #000001 FINAL
12/14/10 02:05:44 #127027
*C05 REPT: EVDO: TP 323, ALARM
ALL AAA SERVERS UNREACHABLE
PERCEIVED SEVERITY: CRIT, CAUSE: Loss Of Signal
2010-12-14 02:06:02 REPORT #000001 FINAL
12/14/10 02:05:44 #127028
**06 REPT: EVDO: TP 326, ALARM
AAA 220.224.134.162: UNREACHABLE
PERCEIVED SEVERITY: MAJ, CAUSE: Loss Of Signal
2010-12-14 02:06:18 REPORT #000001 FINAL

Checking Live Sessions in RNC


Access below path
cd /omp-data/logs/evdo_pcmd
ls ---Get the latest hour file & give below
11062003.EVDOPCMD.gz 11062006.EVDOPCMD.gz
11062004.EVDOPCMD.gz 11062007.EVDOPCMD.gz
11062005.EVDOPCMD.gz 11062008.EVDOPCMD.gz
11062012.EVDOPCMD err_activity_loga

command
11062009.EVDOPCMD.gz
11062010.EVDOPCMD.gz
11062011.EVDOPCMD

tail -f 11062012.EVDOPCMD | nawk -F";" '{ print $5,$6,$28,$29 }'


Please ensure latest Hour & date in above command
1
0
0
0
1
1
1
1
1
0
1
1
0
1
1
1
0
0
0
1
0

00a000001fcfdc45
000000004cf23a49
00000000ff403a7b
00000000ff402942
00a000002d3d9b84
00a000002d3d9a69
00a000002d3c9ee7
00a000001a2a74cf
00a000001f9b83d6
0000000006cf4a01
00a000001fe217a3
00a000001fcfe136
0000000006d162fa
00a000001f72977f
00a000001fe23698
00a000001f773bb0
000000000551bd11
0000000002fcb8e6
000000004cf23a64
00a000001fd00511
0000000012d5c4f2

Page 16 of 18

172.12.38.16
172.12.38.11
172.12.38.12
172.12.38.12
172.12.38.14
172.12.38.12
172.12.38.10
172.12.38.14
172.12.38.16
172.12.38.14
172.12.38.14
172.12.38.12
172.12.38.9
172.12.38.10
172.12.38.12
172.12.38.14
172.12.38.12
172.12.38.11

10.239.200.128 --- PDSN IP


10.239.200.128
10.239.200.128
10.239.200.128
10.239.200.128
10.239.200.128
10.239.200.128
10.239.200.128
10.239.200.128
10.239.200.128
10.239.200.128
10.239.200.128
10.239.200.128
10.239.200.128

172.12.38.9 10.239.200.128
172.12.38.11 10.239.200.128
ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

Health Check of other data elements


Login into Router, MLS 4503, 4948SW, 3400SW etc & give below commands to
carryout health check.
Show interface description
Show interface <port details>
Refer below case of 4503 SW as an example
omp_delhi2 rakesh> telnet 97.241.151.132 --- login into data element
Trying 97.241.151.132...
Connected to 97.241.151.132.
Escape character is '^]'.
User Access Verification
Username:
Password:

DLHIM002WLJ01#show interface description


Note- After giving this command checks for status & protocol for all used
ports. It should be Up Up.
Interface
Vl1
Vl300
Vl301
Vl441
Vl443
Vl445
Vl446
Gi1/1
Gi1/2
Gi1/3
Gi1/4
Gi1/5
Gi1/6
Gi1/7
Gi1/8
Gi1/9
Gi1/10
Gi1/11
Gi1/12
Gi1/13
Gi1/14
Gi1/15
Gi1/16
Gi1/17
Gi1/18
Gi1/19
Gi1/20
Po2
Po3
Po4

Page 17 of 18

Status
up
up
up
up
up
up
up
up
down
down
down
down
up
up
up
up
down
down
up
up
up
up
down
down
down
up
up
up
down
up

Protocol Description
up
up
mgmt-vlan
up
ospf-vlan
up
External-RAN-vlan
up
External-EVDO_DO-AP-vlan
up
External-PDN-vlan-MCN-2
up
Inter-RNC-Hand-Off-Delhi-MCN-2-v
up
*****connected to DLHIM002WLH02*****
down UNC-5 AP
down UNC-5 AP
down UNC-5 AP
down UNC-5 AP
up
UNC-6 AP
up
UNC-6 AP
up
UNC-6 AP
up
UNC-6 AP
down *****Inter MLS Switch Connectivity*****
down *****Inter MLS Switch Connectivity*****
up
*****Inter MLS Switch Connectivity*****
up
*****Connected to DLHIM002WLI03*****
up
*****Connected to DLHIM002WLI04*****
up
*****connected to RDN via ODF*******
down *****connected to RDN via ODF*******
down *****connected to UNC-5 Extreme switch UNC-5*****
down *****connected to UNC-5 Extreme switch UNC-5*****
up
*****connected to UNC-5 Extreme switch UNC-6*****
up
*****connected to UNC-5 Extreme switch UNC-6*****
up
****Etherchannel For Inter MLS Switch Connectivity*****
down ****Etherchannel For C4503 to Extreme Switch Connectivity*****
up
****Etherchannel For C4503 to Extreme Switch Connectivity*****

ALUMS TEC RAN

<RNC & Other Data Elements Health Check MOP>

DLHIM002WLJ01#show interface Gi1/15 ----to check the status of particular port


here Gi1/15 is taken as an example
Note- Check for reliability & errors highlighted below. Reliability should be 255/255
& errors must be zero & should not increment. Incase of any issue contact IPBH
team.
GigabitEthernet1/15 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet Port, address is 0023.04c2.3c0e (bia 0023.04c2.3c0e)
Description: *****connected to RDN via ODF*******
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseLH
input flow-control is on, output flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 1y15w, output never, output hang never
Last clearing of "show interface" counters 1y15w
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 5000 bits/sec, 6 packets/sec
169087492448 packets input, 115728289236192 bytes, 0 no buffer
Received 13294571 broadcasts (12496651 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 input packets with dribble condition detected
102695137035 packets output, 27737328287557 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
DLHIM002WLJ01#

----------END----------

Page 18 of 18

ALUMS TEC RAN

Das könnte Ihnen auch gefallen