IOS XR Operational Tools

2005 Cisco Systems, Inc. All rights reserved. Cisco Public 1

 Presentation Goals

Introduce debugging tools available in IOS-XR

Provide a chance to practice using the tools
Short lab time after each set of commands

XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 2
 Agenda

Processes
Memory
System Monitoring

XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 3
 Process Management Terms

Process
An executable portion of code run within its own memory space

Thread
Sub-process managed by the parent process
Responsible for executing a portion of the overall process
A process may contain one or more threads
Only runs when the parent process is allocated runtime
Example - OSPF thread which handles hello receipt and
transmission, one per neighbor

Dynamically Linked Library

Reusable section of code that can be loaded at runtime

XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 4
 Process Identification

Job ID (JID)
Assigned when process first run
Globally unique identifies the process
No change if process restarts
Process ID (PID)
Assigned when instance of process is created
Globally unique identifies the process
New PID assigned if process restarts
Thread ID (TID)
TID numbering restarts at 1 for each process
Name
Some commands allow process identification by name
XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 5
 show process command options
RP/0/RP0/CPU0:CRS#show processes ?
Reference process by job id
<0-4294967295> job id
WORD Name of the executable
aborts Show process aborts
all Show process data for all processes
blocked Show detail for reply/send/mutex blocked processes.
boot Show process boot info
cpu Show CPU use per process Display process on other nodes
distribution Show distribution of processes
dynamic Show process data for dynamically created processes
failover Show process failover info
family Show process family information.
files Show file and channel use per process
location location to display
log Show process log
mandatory Show process data for mandatory processes
memory Show memory use per process
pidin Show processes using QNX pidin command
searchpath Show the search path
signal Show signal use for processes.
startup Show process data for processes created at startup
threadname Show thread names.

XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 6
 show process command output
RP/0/RP0/CPU0:CRS#show process ospf
Job Id: 262 JID and PID
PID: 209102
Executable path: /disk0/hfr-rout-3.3.1/bin/ospf
Instance #: 1
Automatic restart?
Version ID: 00.00.0000
Respawn: ON
Respawn count: 1
Max. spawns per minute: 12
Last started: Thu Jul 20 15:39:20 2006
Process state: Run
Package state: Normal
Started on config: cfg/gl/ipv4-ospf/proc/1/ord_z/config
core: TEXT SHAREDMEM MAINMEM
Max. core: 0
Placement: ON
startup_path: /pkg/startup/ospf.startup
Ready: 13.338s Thread IDs
Available: 17.353s
Process cpu time: 2.702 user, 0.188 kernel, 2.890 total SNMP Process
JID TID Stack pri state HR:MM:SS:MSEC NAME contains 8 threads
which operate under
262 1 60K 10 Receive 0:00:02:0672 ospf JID 288
262 2 60K 10 Receive 0:00:00:0012 ospf
262 3 60K 10 Receive 0:00:00:0012 ospf
XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 7
 Monitoring Top Processes
Note: Remote execution & auto updating display
RP/0/RP0/CPU0:CRS#monitor processes location 0/2/CPU0
125 processes; 493 threads; 2201 channels, 3365 fds
CPU states: 93.1% idle, 5.8% user, 1.0% kernel
Memory: 2048M total, 1607M avail, page size 4K

JID TIDS Chans FDs Tmrs MEM HH:MM:SS CPU NAME

57 11 19 13 6 12M 0:18:53 2.95% eth_server
70 12 52 9 3 624K 0:01:47 1.74% qnet
1 20 129 94 1 0 502:07:10 1.02% procnto-600-smp
200 6 16 17 4 564K 0:14:43 0.85% qsm
125 6 34 30 6 976K 0:25:48 0.13% egressq
135 3 7 8 2 124K 0:03:00 0.03% hbagent
172 4 25 27 7 276K 0:02:07 0.03% mstats_svr
171 16 66 40 14 684K 0:00:16 0.03% pse_driver
184 2 12 49 3 872K 0:00:08 0.03% pifibm_server
185 5 30 28 5 544K 0:00:08 0.00% pla_server
137 5 22 53 9 524K 0:00:21 0.00% hfr_pm
248 9 32 16 13 2M 0:23:24 0.00% wdsysmon
101 5 22 57 8 552K 0:00:01 0.00% arp
205 3 5 10 1 152K 0:00:10 0.00% SNMP Process
reddrv_listener
195 1 8 34 3 204K 0:00:00 0.00% contains 8 threads
ppp_socket
which operate under
178 5 19 36 5 440K 0:00:00 0.00% nfsvr JID 288
74 10 114 22 4 568K 0:00:00 0.00% sysmgr
224 2 6 7 2 236K 0:00:00 0.00% sysmgr
XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 8
 Stopping, Restarting, and Crashing Processes
Note: Lab use only

process shutdown shut down and do not restart

Graceful termination
process start start a shut down process
process restart restart a process
Works if process is running or shutdown
Graceful termination
process crash simulate a crash
Process will restart automatically if Respawn=ON
Non-graceful termination

XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 9
 Process Restart Example
RP/0/RP0/CPU0:CRS#show proc qnet
Job Id: 74 Initial respawn count shows
PID: 32795 process hasnt restarted
Executable path: /hfr-os-3.3.1/sbin/qnet
Instance #: 1
Args: transport=enet,conn_est_retries=3
Version ID: 00.00.0000
Respawn: ON Restart a few times
Respawn count: 1

RP/0/RP0/CPU0:CRS#process restart 74 PID changes, JID stays same

RP/0/RP0/CPU0:CRS#process restart 74
RP/0/RP0/CPU0:CRS#show proc 74
Job Id: 74 Respawn count increases
PID: 7061531
Executable path: /hfr-os-3.3.1/sbin/qnet
Instance #: 1
Args: transport=enet,conn_est_retries=3
Version ID: 00.00.0000
Respawn: ON Reason for restart
Respawn count: 3
Max. spawns per minute: 12
Last started: Thu Aug 31 07:13:37 2006
Process state: Run (last exit due to SIGTERM)

XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 10
 Process Shutdown
RP/0/RP0/CPU0:CRS#process shut ospf

RP/0/RP0/CPU0:CRS#show process ospf

Job Id: 262
PID: 6983888
Executable path: /disk0/hfr-rout-3.3.1/bin/ospf
Instance #: 1
Version ID: 00.00.0000 Process not running
Respawn: ON
Respawn count: 4
Max. spawns per minute: 12
Last started: Thu Aug 31 07:02:32 2006
Process state: Killed (last exit status : 1)
Package state: Normal
Registered item(s): cfg/gl/ipv4-ospf/proc/
core: TEXT SHAREDMEM MAINMEM
Max. core: 0
Placement: ON
startup_path: /pkg/startup/ospf.startup
Ready: 6.368s
Available: 10.385s

RP/0/RP0/CPU0:CRS#process start ospf

XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 11
 XR Processes Mini-lab
Try the process commands on your lab router

show process
process shutdown shut down and do not restart
Graceful termination
process start start a shut down process
process restart restart a process
Works if process is running or shutdown
Graceful termination
process crash simulate a crash
Process will restart automatically if Respawn=ON
Non-graceful termination
monitor process real time list
Sorted by memory or CPU usage
? to adjust parameters
XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 12
System Monitoring and Online Help

2005 Cisco Systems, Inc. All rights reserved. Cisco Public 13

 System Monitoring

Monitoring
show tech
show system verify
monitor interface
monitor controller
Online Help
describe
man

XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 14
 show tech variations
RP/0/RP0/CPU0:CRS#show tech ?
bcdl Output show commands of interest for bcdl debugging
cef Output show commands of interest for CEF debugging
file Specify a valid file name (e.g. disk0:tmp.log)
gsp Output show commands of interest for gsp debugging
lrd Output show commands of interest for lrd debugging
ospfv3 Output show commands for OSPFv3 debugging
password Include password in output
pfi Output show commands of interest for PFI debugging
placement Gather lots of information about process placement
platform show tech-support platform output
rib Show IP RIB related information
routing Routing show tech-support output
terminal Send output to terminal

XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 15
 Monitor One Interface
monitor interface command
RP/0/RP1/CPU0:CRS1# monitor interface mgmtEth 0/rp1/CPU0/0
CRS1 Monitor Time: 00:00:52 SysUptime: 16:33:16
MgmtEth0/RP1/CPU0/0 is up, line protocol is up Encapsulation ARPA

Traffic Stats:(5 minute rates) Delta

Input Packets: 313326 17
Input pps: 5
Input Bytes: 34467898 1033
Input Kbps: 3
Output Packets: 37633 4
Output pps: 0
Output Bytes: 2034463 244
Output Kbps: 0
Errors Stats:
Input Total: 1 0
Input CRC: 0 0
Input Frame: 0 0
Input Overrun: 0 0
Output Total: 0 0
Output Underrun: 0 0
Quit='q', Freeze='f', Thaw='t', Clear='c', Interface='i',
Detail='d', Brief='b', Next='n', Prev='p'
XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 16
 Monitor All Interfaces
monitor interface all command
RP/0/RP1/CPU0:CRS1# monitor interface all
CRS1 Monitor Time: 00:00:55 SysUptime: 16:35:50
Interface Encap Input Output Input Output
pps pps Kbps Kbps
MgmtEth0/RP0/CPU0/0 ARPA 4 0 2 0
MgmtEth0/RP1/CPU0/0 ARPA 4 0 2 0
POS0/4/0/0 HDLC 1230 0 542210 0
POS0/4/0/3 HDLC 0 0 0 0
POS0/4/0/6 PPP 0 0 0 0
POS0/4/0/7 HDLC 0 1230 0 542210

Quit='q', Freeze='f', Thaw='t', Clear='c', Next set='n', Prev set='p'

XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 17
 Online Help
Requires -doc package

RP/0/RP1/CPU0:CRS#man keyword mpls

Following Commands Matched for 'mpls'
FEATURE COMMAND
is-is mpls traffic-eng
is-is mpls traffic-eng router-id (is-is)
is-is show isis mpls traffic-eng adjacency-log
is-is show isis mpls traffic-eng advertisements
is-is show isis mpls traffic-eng tunnel
ospf mpls traffic-eng area (ospf)
ospf mpls traffic-eng router-id (ospf)
ospf show ospf mpls traffic-eng
quality-of-service match mpls experimental topmost
quality-of-service set mpls experimental topmost
RP/0/RP1/CPU0:CRS#man command show cef ipv4
COMMAND
show cef ipv4
DESCRIPTION
To display the IPv4 Cisco Express Forwarding (CEF) table, use the show
cef ipv4 command in EXEC mode.
show cef ipv4 [prefix [mask] | type instance] [detail] [location node-id]

XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 18
 Details about commands (permissions)
describe command

RP/0/5/CPU0:GSR1# describe show install active

Package:
c12k-base
c12k-base V3.5.0[00] Base Package for 124xx
Vendor : Cisco Systems
Desc : Base Package for 124xx
Build : Built on Tue Jun 12 20:36:49 UTC 2007
Source : By zamboni in
/auto/srcarchive2/production/3.5.0/c12k/workspace for c2.95.3-p8
Card(s): RP, DRP, DRPSC, OC3-POS-4, OC12-POS, GE-3, OC12-POS-4,
OC48-POS, E3-OC48-POS, E3-OC12-POS-4, E3-OC3-POS-16, E3-OC3-POS-8, E3-
OC3-POS-4, E3-OC48-CH, E3-OC12-CH-4, E3-OC3-CH-16, E3-GE-4, E3-OC3-ATM-4,
E3-OC12-ATM-4, E5-CEC, E5-CEC-v2, SE-SEC
Restart information: Required permissions
Default:
parallel impacted processes restart

Component:
installmgr V[main/472] On the box installation program

User needs ALL of the following taskids:

pkg-mgmt (READ) Cisco Public 19

XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved.
 XR Monitoring and Help Mini-lab
Try the following commands on your lab router

Monitoring
show tech
show system verify
monitor interface
monitor controller
Online Help
describe
man

XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 20
 Verifying the System

SUMMARY STEPS

1. admin
9. show context
2. show platform [node-id]
10. show memory summary detail location all
3. show version
4. show running-config 11. show memory heap summary {job-id | all}
5. show logging 12. top processes
6. show environment 13. show running-config
7. show context 14. show system verify start
8. exit show system verify report
15. show {ipv4 | ipv6} interface brief
16. Show install active

XR Training / lwigley 2005 Cisco Systems, Inc. All rights reserved. Cisco Public 21
 Presentation_ID
XR Training / lwigley
2004, Cisco Systems,
2005 Cisco Systems, Inc.
Inc. All
All rights
rights reserved.
reserved. Cisco Public 22 22