Beruflich Dokumente
Kultur Dokumente
snip -
(142.0.0.2,225.0.0.1), Flags:
3d09h
Last Used: never
‘SW Forwarding Counts: 0/0/@
SH Replication Counts: 0/0/0
SH Failure Counts: 0/0/0/0/0
TenGigé0/5/0/1 Flags: A, Up:3409h
Gigabitetherneto/4/o/3.102 Flags: MS EG, Up:23:19:39 € Traffic will be
forwarded
Gigabitetherneto/4/o/10.101 Flags: NS EG, Up:3d06h
RP/0/RSPO/CPUO: asr9k-3#
Now check if packets are getting forwarded in hardware using mfib commands:
RP /0/RSPO/CPUD:asr9k-3ashow fib route statistics 225.0.0.1 142.0.0.2 detail
IP multicast Forwarding Information Base
Entry flags: ¢ - Directly-Connected check, $ - Signal, D - Drop,
Ik - Inherit Accept, IF - Inherit From, MA - MOT Address,
ME = MOT Encap, MO - MOT Decap, NT - MOT Threshold Crossed,
Ma - MOT interface handle, cD ~ Conditional Decap,
OT - MOT Decap True
Interface flags: F - Forward, A - Accept, IC - Internal Copy,
NS - legate Signal, OP - Don't Preserve, SP - Signal Present,
EG - Egress, EI - Encapsulation interface, MI - MOT Interface
‘Su/Hi Forwarding/Replication Counts: Packets in/Packets out/aytes out
SW Failure Counts: RPF / TTL / Empty Olist / Encap RL / other
1_ASR9000 Multicast troubleshooting document, Cisco Systems Inc
Hii Drop Counts: Ingress 7 Egress
Hu Forwarding Rates: bps In/pps In/bps out/pps out
}-0.2,225.0.0-1), Flags:
257
‘SH Forwarding Counts: 0/0/20
SW Replication counts: w/o/e
SW Failure Counts: 0/0/0/0
Hu Forwarding Counts: W/A /W/A /W/A
Hu Replication Counts: N/A /M/A /W/A
Hu Drop counts: N/A /W/A
Hu Forwarding Rates: N/A /N/A /N/A /u/A
TenGigeo/5/0/1 Flags: A, Up:00:12:
Gigabitetherneto/4/o/3.102 Flags: WS, Up:00:12:57
Gigabitetherneto/4/o/10.101 Flags: NS, Up:00:12:57
RP/0/RSPO/CPUO: asr9k-3#
P/0/RSPO/CPUO:asrSk-3#show mfib route statist 225.0.0.1 142.0.0.2 detail loc
@/4/cPu0(aa.
-0.2,225.0.0-1), Flags:
3do9h
Last Used: never
‘SH Forwarding Counts: 0/0/20
‘SH Replication Counts: 0/0/0
SW Failure Counts: 0/0/0/0
Hui Forwarding Counts: 0/169493467/7796699482
Hu Replication Counts: 0/736054404/33858506724
Hu Drop counts: 0/0
Hu Forwarding Rates: N/A /N/A /725395/1970
TenGigeo/5/a/1 Flags: A, Up:3desh
Gigabitetherneto/a/o/3.102 Flags: WS EG, Up:23
Gigabitetherneto/4/o/10.101 Flags: NS EG, Up:3de6h
P/0/RSPO/CPUO: asrok-3#
28
© Verifying hardware mroute/mfib Table
Following command output shows the whether packets forwarded in
hardware for _a given (S,G
‘ASR9000 Multicast troubleshooting document, Cisco Systems Inc 1
7
P/0/RSPO/CPUO:asrok-3show mFib hardware route stati 225.0.0.1 142.0.0.2 loc 0/5/CPUO
Le Type: Typhoon ASK-NOD160-SE
np 10 R: Received
Forwarded P: punted to cpu
1D: Ingress Drop ED: egress Drop
182.0.0.2 Group: 225.0.0.1 Mask:64
rytes)/P(packet s)/I0(packets)/ED(packets)
‘€ THis Ne is receiving traffic from wire
No OLIST interfaces found for this route
P/0/RSPO/CPUO:asr9k-3#
P/@/RSPO/CPUO:asr9k-3¥show mfib hardware route stati 225.0.0.1 142.0.0.2 loc 0/4/CPUO
Le Type: Trident A9K-40Ge-E
ae & © peceivelForwarded
Ingress Drop
Punted to CPU
Egress Drop
0:
10)
Source: 142.0.0.2 Group: 225.0.0.1 Mask:64
NP R(packets:bytes)/F(packets
rytes)/P(packet s)/I0(packets)/EO(packets)
1.0 /0/ 0 € This NP 4s sending traffic out on wire
10/0 /0 € This NP is sending traffic out on wire
2 Gio/a/e/10.101 434208:19973568,
3 Gio/a/o/3.102 443309:20392214
-going interface on the NP2
'going interface on the NP3
P/0/RSPO/cPUO:asr9k-3#
1 _ASR9000 Multicast troubleshooting document, Cisco Systems Inc
7 Tracing the packet through Fabric and ASICs:
At very high-level path taken by a packet is as follows. Only Trident LC has bridge asic.
PHY > ingress NP > Bridge > FIA > Fabric > FIA > Bridge > Egress NP PHY
Lets check the counters in each asic using show commands to trace the packet getting
forwarded.
7.1_Ingress NP counters:
WP/0/RSPO/CPUD: asrok-3eshow controllers np counters npo Location @75/@PUS
Node: 0/5/cPUO:
Show global stats counters for Pe, revision v2
Read 23 non-zero WP counters:
offset counter Framevalue Rate (pps)
21 MDF_TX_FABRIC 23279 4001
29° PARSE_FAB_RECEIVE_CNT 7 1
33. PARSE_INTR_RECEIVE_CNT 43 7
41 PARSE_ENET_RECEIVE_CNT 23278 4001
P/0/RSPO/CPUO: asr9k-3#
7.2__Ingress FIA countersseeeeceee prgig tetenentee
category: count-0
From Unicast xbar[0] 6
From unicast xbar[1] 4
From unicast xbar[2] @
From unicast xbar[3] °
From Multicast xbar[o] e
From Multicast xbar(1]} °
From Multicast xbar(2] °
From Multicast xbar[3] °
To unicast xbar[o] 3
To Unicast xbar[1] 3
To unicast xbar[2] °
To unicast xbar{3] °
To multicast xbar[o] 17829 &
To multicast xbar[2] 17820 €
To multicast xbar[2] 8
To multicast xbar[3] °
To Line Interface[0] °
To Line interface(2) 10
From Line Interface[o] °
From Line Interface(1] 35668
Ingress drop: @
Egress drop e
ASR9000 Multicast troubleshooting document, Cisco Systems Inc 1.
9
Total drop? e
seveneeee praca tenensnees
RP/0/RSPO/CPUO:asrOk-3#
7.3__Egress LC FIA counters:
B70/#SPO/CPUO:asr9k-Sashow Controllers Fabric fia stats Location 0/4/CPUS
sesneeseee FIA-g seeeeeeees
Category: count-0
From Line Interface[o] 6
From Line Interface[1] 2
To Line Interface(o] 6
To tine tnterface(t] 63807
‘To unicast xbar[o]
To unicast
To Unicast
To unicast
To multicast
To multicast
To multicast
To multicast
From unicast
From unicast
From unicast
From unicast
From Multicast
From Multicast
From Multicast
From Multicast
xbar[2]
xbar[2]
xbar{3]
xbar0]
xbar{2]
xbar{2]
xbar{3]
xbar [0]
xbar{]
xbar{2]
xbar[3]
Xbar[o]
xbar[1]
xbar[2]
xbar[3]
331753
331752
€ From Fabric
€ From Fabric
Ingress drop:
Egress drop
Total drop:
7.4_Egress NP counters
P70/8SPO/CPUO:asr9k-34show controllers np ports all location 0/4/CPUS
Node: 0/4/cPvO:eee ee
Gigabitetherneto/4/o/30 - Gigabitetherneto/4/0/39
oon ae
1 © © Gigabitetherneto/4/0/20 - Gigabitetherneto/4/0/29
2 1 © Gigabitetherneto/a/o/10 - Gigabitetherneto/4/0/19
3 1 0 _Gigabitetherneto/4/o/o - cigabitetherneto/a/o/9
RP/0/RSPO/CPUO:asrok-3#
RP/0/RSPO/CPUO:asr9k-34show controllers np counters np2 location 0/4/cPUO
Node: 0/4/cPU0:
show global stats counters for NP2, revision v3
Read 16 non-zero NP counters:
Offset” Counter Franevalue Rate (pps)
24 PARSE_LooPaack RECEIVE. cwT 3674751 2000
30 MOOTFY_ENET_TRANSATT ci 11349784 4000
38 MODIFY _NCAST FLO, LOOPBACK_cnT 3674751 2000
p/0/RSPO/CPUO:asr9k-38show controllers np counters np3 location 0/4/cPuO
Node: 0/4/CPU0:
2__ASR9000 Multicast troubleshooting document, Cisco Systems Inc
show global stats counters for NP3, revision v3,
Read 19 non-zero NP counters:
offset counter Framevalue Rate (pps)
23 PARSE_FABRIC_RECEIVE_CNT 3604472 2000
24 PARSE LOOPBACK_RECEIVE_CNT 47 °
30 MODIFY_ENET_TRANSMIT_CHT s604a25 2000
np /0/RSPO/CPUO: as ok-3#
8 Debugging multicast Process Memory:
This section outlines some basic commands, which can help to troubleshoot whether system
is running with low memory, memory leak and or process crashes.
+ Collect the output of “show process memory location <>” and check if memory in
dynamic column increasing. If so, this might be sign of one or more process holding up
memory and not releasing it. Please take few snap shot of the output. Output shows
memory utilization for only multicast process.
1 70/#SPO/CPUB:asrSK-S¥shOn processes wenory location 073/cPUs | In ipva_wfwd
1D Text Data Stack Dynamic Process
237 475136 192512 «126976 12337352 ipva_mfwd_partner
RP/0/RSPO/CPUO:asr9k-3#
9 Debugging Multicast and related processes:
‘+ Following outputs were taken from one of the Service request for the explanation
purpose. This output shows one of the multicast process blocked
* Check if multicast process is blocked:
R70/8SPO/CPUO:on-bari-vord#sh processes blocked Location o/rspo/cpud
nid Pid Tid ane State Tinelnstate Blocked-on
6554916397 1 ksh Reply 6572:57:32:0613 16396 devc-conaux
6555457362 devb-umass Reply 0:00:07:0644 45096 io-usb
A F
96 40996 2 —_umass-enum Reply 6572:57:36:0954 1 kernela , es
52 69677 2 attachd Reply 6572 49196 eth_server
52 69677 3 attachd Reply 6572 24595 mquéue
65582 57390 2 devb-umass Reply :00:37:0655 45096 io-usb
87 69080 G net Reply 49196 eth_server,
51 69684 2 attach server Reply 6572:57:33:0215 24595 mqueue
65594 69690 2 devb-umass Reply @ 0158 45096 io-usb
3349917781 ttp_server Reply 29R6:32:74:0177 74505 mquoNe
pai 258252 2 Ipts_fm Reply 0485 245891 Ipts_pa
65741 39502797 1 exec Reply 1:13:10 258244 _deve-vty
353° 1835278 6 ipv4_mfwd_partner hutex 3238 69654-56184 #2 attach_server
65844 6035072 1 exec Reply a kernel
65045 61145397 1 ‘more Repl} 0065 24593 pipe
‘ASR9000 Multicast troubleshooting document, Cisco Systems Inc 2
1
GSH46 61145398 1 show processes Reply
P/0/RSPO/CPUO:0n-bari-vorié
T
Kernel
+ Above output shows that ipv4_mfwd_partner process blocked on attach. server. Lets check
status of the attach server process
* Check state of the attach server process and ipv4_mfwd_partner
'R970/8SPO/CPUOTOn-barL-vOriash processes attach server
Thu May 10 15:42:26.745 EOT
dob Id: 51 €2== J0b 10
PID: 69684 €=.
Executable path: /asr9k-0s-4.0.1/bin/attach_server
Instance #: 1
version 10: 00.00.0000
Respawn: oN
Respawn count: 1
Max. spawns per minute: 12
Last started: Wed Aug 10 15:02:46 2011
Process state: Run
Package state: Normal
core: MAINMEH
Max. core:
Level: 39
MaintrodeProc: on
Placenent: tione
startup_path: /pkg/init.d/attach_server. init
Process cpu time: 0.012 user, 0.000 kernel, 0.012 total
PID
JID TID CPU Stack pri state Timeznstate _HR:MN:SS:MSEC
S110 20K 10 Receive 6576:30:04:
Si 2 -1-20K «55 Reply —__6576:39:31:0128 _@:00:20:0000 attach_server
RP/0/RSPO/CPUD:0n-bari-vorisshow processes ipv
Thu May 10 15:44:38.446 EOT
Job Id: 353
PID: 1835278
Version 1D: 00.00.0000
Respawn: ON
Respawn count: 1
Max. 2
Fri Aug 19 21:3
Run
Normal
MaTiOEM
°
199
None
9 2011
327 @:00:00:0012 attach_server
/install /iosxr-mcast-4.0.1/startup/ipva_nfud. startup
NaveProcess cpu time: 222629.394 user, 60507.334 kernel, 263336.728 total
31D TID CPU Stack pri state TimeInState | HR:MN:SS:MSEC NAME
3531 @ 76K 10 Receive a:ee:e0:e0a0 48:58:31:
ipva_mfedl_parn
353 6 1 76K 10 Mutex 20:
ipva_mfud_partner
2__ASR9000 Multicast troubleshooting document, Cisco Systems Inc
50
31:0794 28:39:32:0556
Collect output of “show follow process