CCNP TSHOOT 642-832 Network Maintenance

Maintenance is no doubt an important component to network stability and thats why we see
it covered on the troubleshooting exam. This is perfect multiple-choice style content as it is
difficult to test your understanding of maintenance concepts and methodologies on a
simulation-focused exam like TSHOOT. Keep that in mind as you walk through the following
topics.
Good troubleshooting reduces the time an outage lasts, good maintenance
minimizes outages themselves.
Maintenance Methodologies
Several well known maintenance models have been defined by a number of organizations.
Many organizations use parts of several instead of adopting one method completely, but it is
important as a network engineer to understand what models exist and how they translate
into improving your organization. A documented maintenance strategy is worth its weight in
gold.
IT Infrastructure Library (ITIL)
ITIL focuses on creating a technology service framework within an organization and aligning
it closely with the organizations requirements and processes. Note that ITIL is a large and
comprehensive approach that was developed specifically for IT professionals.
FCAPS
FCAPS is an IT maintenance model created by ISO that categorizes network management
into five parts. FCAPS is an acronym using the first letters of the five categories it includes.
Fault management
Preventive maintenance
Minimizing network downtime
Configuration management
Both hardware and software installation and configuration
Change control
Inventory management
Accounting management
Capacity planning
Cost efficiency
Performance management
Maximize performance on existing network investments
Security management
Confidentiality, integrity, availability (CIA)
Authentication, authorization, accounting (AAA)
Encryption
Intrusion detection/prevention
Cisco Lifecycle Services
Cisco has come up with their own maintenance model, sometimes also referred to as
PPDIOO, or Prepare, Plan, Design, Implement, Operate, and Optimize. This model is
specifically focused on deploying and operating Ciscos product families.
Telecommunications Management Network (TMN)
TMN was developed by ITU-T and is a tailored version of FCAPS specific to the
telecommunications industry.
Once the model has been selected, its parts should inform an IT organizations processes
and standard procedures. After all, a model is meaningless unless it affects how a business
operates.

After the maintenance model components have defined an organizational processes (ex.
automated config backups, manual security audits, etc.), tools should be selected to carry
out those processes. FTP could be used for configuration backups for example.
Network Maintenance Core Tasks
Whatever model an IT organization chooses, there a some functions that should be included
every time. These include:
Managing adds, moves, and changes
Installing and configuring new network devices
Replacing failed hardware
Software backup
Configuration backup
Troubleshooting failure scenarios
Software upgrades
Network performance monitoring
Capacity planning
Creating/updating network documentation
Documentation
Up-to-date, clear, and complete infrastructure documentation is crucial to reduce recovery
times and maintain a robust networked environment. Different levels of detail are
appropriate for different audiences, but some common details that should be documented
include:
Production configurations
Inventory (including serial numbers, support info, etc.)
Circuit information
Network drawings
IP address assignments
Another important component to network documentation is a performance baseline, or
snapshot. It captures the expected performance of your network systems like link
bandwidth, WAN jitter and delay, and port status. This is a tremendous help during
troubleshooting efforts because without knowing what normal levels are, detecting abnormal
traffic behavior becomes very subjective.
IOS Tools
Configuration
Configurations should be backed up periodically or after changes are made. One of the
simplest methods is to save the configuration as a text file on a remote TFTP or FTP server.
TFTP and FTP servers are available on all modern operating systems and free, open source
offerings are widely available.
Adding the date to the saved configuration can make rolling back changes easier in the
future. Heres an example of a router saving its configuration to a local TFTP server:
RouterA# copy run tftp
Address of name of remote host []? 10.10.1.35
Destination filename [routera-config]?routera
Syslog
Syslog is a tool that collects alerts from network devices and stores them on a common log.
Obviously, this can be very handy when you need to troubleshoot an issue across many
devices.
Know that every syslog message contains two parts, a severity level and a facility. The
severity level goes from 0 to 7 with 0 being the most severe to 7 being simply informational.
Syslog Priority (highest to lowest):
0. Emergency (highest)
1. Alert

2.
3.
4.
5.
6.
7.

Critical
Error
Warning
Notice
Informational
Debug (lowest)

NTP
Alerting is important, but if the timestamps that are included are not correct, then the alerts
are unreliable (and next to useless). NTP stands for Network Time Protocol and is used to
keep accurate and consistent time on all network devices. NTP works by using pulling the
current time from a time server, each of which are assigned by stratum. Stratum 1 clocks
are synchronized directly with an atomic clock, stratum 2 clocks get their time from stratum
1 clocks, etc.
Configuring NTP is easy just point the device to the proper time server:
Switch(config)# ntp server ip_address_of_ntp_server
To verify:
Switch# show ntp status
One last note for NTP, it is important to consider the time zone that each device is set to.
Make sure you have it consistent (ex. local time zones, GMT, HQ time zones, etc)
Archive
Cisco has developed a built-in configuration backup and restore feature, called archive. The
archive function maintains a copy of the current configuration as well as a set of past
configurations. If a configuration change is made with unpleasant results, the switch or
router can roll back to a previous configuration relatively easily.
There are several keywords available inside archive configuration mode. Here is a list of
some of the most common:
path
Specifies where you want the backup configuration stored (ex. flash, tftp server, etc.)
Example:
archive
path flash://routerc
OR
archive
path tftp://192.168.1.22/routerc.txt
write-memory
When the write-memory keyword is configured, a backup of the configuration will be
automatically saved every time the configuration is manually saved.
time-period
Sets the maximum time allowed before another backup is automatically saved
When the archive function backs up a configuration, it appends a -1, -2, -3, etc. to the end of
the file name depending on how many have already been saved. It will count up to 14
(represented as filename-14) and then cycle back to 1. If your time-period is set too
frequently, then youre backups may be written over too often.

CCNP TSHOOT 642-832 The Art of Troubleshooting

There are two elements to good troubleshooting preparation and technique. Preparation
comes in the form of documentation, change control, and understanding of the environment.
The second part, technique, is just as important.
There are a number of methods to tackle the same problem. To be honest, Cisco doesnt
promote a specific approach for the CCNP TSHOOT exam. The important part is that you
are consistent and your troubleshooting methodology follows a structured approach.
Structured Troubleshooting
What Cisco calls structured troubleshooting simply means you use a system to solve a
problem by collecting information about the problem, forming a hypothesis, and then test it.
The structured approach also is helpful when the hypothesis you create fails. It may rule out
many more scenarios and likely leads to the next hypothesis to test. The recovery time for a
structured troubleshooting approach is usually much less than randomly changing
configurations or settings in a hurry to try and get things working.
There are several common structured troubleshooting approaches, with these being the
most common:
Bottom-Up
Start with the OSI physical layer and work your way up.
Top-Down
Start with the OSI application layer and work your way down.
Follow-the-Path
Consider the path a packet would take from source to destination, checking each
node/device/configuration along the way.
Spot-the-Difference
This is where configurations are compared between what is currently running and what the
expected configurations should be.
Move-the-Problem
Move a device to see if the problem moves with it.
Use the Scientific Method
The first step whenever you encounter a technical problem is to define the problem. This
will involve collecting input from those experiencing the issue directly things like the
Internet is down or my email is slow or I cant get to my Facebook account when I
should be processing TPS reports You get the idea. Keep in mind that you will need to
understand that they are explaining the symptoms its your job to determine the problem
behind the symptoms.
After you have identifies the problem, its time to trim it down. Whats the scope? How
many users are affected? What changed? When did it happen? Is it a constant problem or
intermittent?
Now this is where your tool bag of structured troubleshooting methodologies should come
out. Try one that you think best matches your hypothesis of the root issue and work your
way through it. Did your test work? If not, continue through the layers, the path, or
whatever approach you are using.
When you find a test that is successful and determine that it in fact is the root cause, make
sure to communicate the problem and recovery to all stakeholders and update any
necessary documentation. These are small, simple tasks but they are rarely done
consistently.
If a configuration change was the culprit, think about your current change control policy and
ask if it needs to be updated.

CCNP TSHOOT 642-832 Layer 2 Troubleshooting

Poor Switch Performance
Most performance issues on switches are related to one of three errors:
1. Cabling and port problems (layer 1)
2. Duplex mismatches between switch ports and an attached device
3. TCAM issues.
Physical layer Troubleshooting Commands
show interface
show interface counters
show interface counters errors
Look for the following errors:
FCS-Err
Usually a cabling issue.
Xmit-Err
The transmission buffers are full. This is sometimes seen when switching from a fast link to
a slower one.
Undersize, Giants
The transmitting NIC may have problems.
Single-Col, Multi-Col, Late-Col, Excess-Col
All of these are collision types, which can point to a duplex mismatch. Cisco recommends
setting all interfaces, switch and server, to auto.
Spanning Tree
Spanning Tree Protocol is a loop prevention mechanism to allow redundant Ethernet network
connections. Here is an important summary of how each switch determines Spanning Tree
port roles:
1. Each switch periodically transmits BPDUs that include its bridge ID, current root bridge,
and cost to that root bridge. Additionally, each switch starts assuming it is the root bridge.
2. If a switch receives a BPDU from another switch with a different root, it does a
comparison. If the BPDU has a lower advertised root, the switch changes its root to match
and recalculates the cost to the new root. The port that received the BPDU is now the root
port all others become designated ports.
3. If a switch receives two BPDUs with the same root, it then compares costs and uses the
port with the lowest cost. The port with the higher cost is blocked also called a nondesignated port.
To quickly review STP costs, below is a list of link costs based on interface speed.

After the whole process, there will be only one root bridge with each nonroot switch having only one root port.
To see the status of spanning tree, do a show spanning-tree vlan vlan-id.

To view sent/received BPDU information for a switch, do a show spanning-tree interface

interface detail.
Broadcast Storms
Broadcasts storms can occur due to Spanning Tree misconfigurations and/or rogue switches
being added which closes a loop. Regardless, a broadcast storm will be obvious when the
switch slows way down, becoming unresponsive, and all the links light up solid green.
The CLI may be very slow to respond if you still have remote access to it, so often times to
fastest way to fix the problem is to physically begin pulling redundant links.
Troubleshooting EtherChannels
EtherChannel issues usually fall into one of three categories:
1. Every port participating in an EtherChannel must have identical speed, duplex, access or
trunk settings. If an EtherChannel isnt forming, check each port configuration.
2. Both sides of the EtherChannel must be configured as a bundle directly or be using a link
aggregation protocol (LACP or PAgP). If one side is configured as an EtherChannel and the
other side is not, look for error-disabled EtherChannel ports on the EtherChannel-enabled
switch.
3. If traffic is only flowing over a single link in a bundle, it is likely that the hash algorithm
should be adjusted to use different seed values. Also note that link bundles should be used
in even numbered pairs like 2, 4, 8, etc.
VLANs
When troubleshooting issues that you suspect are related to VLAN logic, you should first
make sure you have tested for physical layer issues like bad cabling, a power failure, or
bad switch ports. Also, check that you are not dealing with an issue with the switch itself
things like software bugs, loops, or ARP problems.
VLAN issues usually come in the form of misconfigured VLANs, improper VTP mode, trunk
issues, or native VLAN mismatches.
Switch Tables
It is important that you understand what show commands display information on what
switch tables. These will come in handy when you are isolating a switching issue.
MAC Address Table (MAC-to-port mapping)
show mac-address
VLAN Assignments (VLAN-to-port mapping)
show vlan
Trunk Assignments
show interface switchport
show interface switchport trunk
show etherchannel
Troubleshooting Inter-VLAN Routing
Routing between VLANs can be done on either a router, or a layer 3 switch but the data
plane is different depending on the platform you are using.
Either way, show ip cef displays the CEF forwarding table and show adjacency will show you
the layer 2 headers used in forwarding.
Keep in mind that routers always use layer 3 information to pass traffic between ports.
Switches can either use MAC address forwarding (for layer 2 forwarding), SVIs for inter-VLAN
routing, or layer 3 routed ports. The last category, routed ports do not run layer 2 protocols
like Spanning Tree very important.
Last thing to remember SVIs will only go into down state when all interfaces within that
particular VLAN are down.
HSRP, VRRP, & GLBP
First hop redundancy protocols allow a layer 2 segment to have two gateway routers for
redundancy, while still only showing a single gateway IP and MAC address.

The three FHRPs Cisco supports are HSRP, VRRP, and GLBP.
HSRP is one of the original FHRPs that was developed by Cisco and is proprietary. One
router is active and another is a backup (using HSRP keepalives to maintain connectivity).
HSRP is extremely popular and you should make sure to understand how it works for the
TSHOOT exam. Check out the High-Availability page to learn more.
VRRP is another gateway redundancy protocol that is an open standard and very similar to
HSRP.
GLBP is Cisco proprietary; its primary advantage is its ability to automatically load balance
between gateway routers.

HSRP
HSRP is the primary FHRP covered on the TSHOOT exam, so lets go through the basics one
more time.
HSRP is configured using the standby command under interface configuration mode.
Routers in the same HSRP group share a common MAC and virtual IP address. The standby
configuration statements define the HSRP group as well as the virtual IP in use.
Each HSRP-enabled router has a default HSRP priority of 100 (remember, highest wins). If
another router joins the group with a higher priority it will still not become the active router
unless the preempt command is applied.
An example HSRP configuration could look something like:
Router(config)# interface gig1/1
Router(config-if)# ip address 192.168.1.2
Router(config-if)# standby 4 ip 192.168.1.1
Router(config-if)# standby 4 priority 200
Router(config-if)# standby 4 preempt
To show the current HSRP status, issue either show standby or show
standby brief depending on the level of detail you require.

CCNP TSHOOT 642-832 Layer 3 Troubleshooting

Before we get into the layer 3 troubleshooting methods, we first need to make sure we have
a basic understanding of how routers and multilayer switches route traffic. Three tables are
used: the routing table, ARP table, and CEF mappings.
The routing table pairs network prefixes with the routers preferred next hop address or
interface. Packets are routed based on the output of the routing table by first matching the
longest prefix and then using other IGP-specific metrics. The show ip route command
displays the contents of the routing table.
After the router has determined what the next-hop address is, the router then needs to
translate that into a layer 2 MAC address. The ARP table is exactly what this is for. The
show ip arp command will display the current ARP pairings.
Lastly, CEF is used in layer 3 switches to optimize routing and layer 2 headers. To view the
CEF entries, use the show ip cef command.
Troubleshooting Any Routing Protocol
Regardless of what routing protocols are in use, there are some common troubleshooting
steps that can be applied. First, try to ping the destination to determine reachability. Next,
look at the routing table to make sure a route to the destination exists. Finally, run a
traceroute from the source towards the destination to see where the last reachable hop is.
For further digging, the show ip protocols command gives some very helpful information
on the current routing protocols in use (like timers, AS numbers, etc.).
Routing Protocol Troubleshooting Methodology
There are three key questions that can be extremely helpful when troubleshooting a routing
issue regardless if you are running EIGRP, OSPF, or BGP.
1. Is the route being advertised properly?
2. Is the route being received?
3. Is there a more desirable route being used (longer prefix or lower
administrative distance)?
Now lets dissect each of these for the major routing protocols one at a time.
EIGRP
First, verify connectivity to the remote networks using pings and by taking a look at the local
routing table.
As a reminder, EIGRP stores its information in three different tables: the EIGRP interface
table, neighbor table, and topology table.
EIGRP Interface Table
The EIGRP interface table displays interfaces participating in the local EIGRP processes. Use
the show ip eigrp interface command to display its contents.
EIGRP Neighbor Table
The EIGRP neighbor table contains a list of discovered EIGRP neighbors. Use the show ip
eigrp neighbors command to display its contents.

EIGRP Topology Table

The topology table contains a complete list of EIGRP-learned routes. Use the show ip eigrp
topology command to display its contents.
Is the EIGRP route being advertised properly?
Remember those three troubleshooting questions listed above? Lets start with the first one
is the route being advertised properly?
The first step is to identify the router that is connected to the destination subnet as it should
be advertising the route out. There are two simple ways to check if that router is advertising
the routes properly.
First, do a show run | section eigrp. This will display the running EIGRP configuration,
including what networks are being advertised with the network statements.
Another option is to do a show ip protocol. The nice thing about this command is that it
displays the EIGRP network statements. Remember, EIGRP only advertises subnets of
interfaces that match an EIGRP network statement.
Is the EIGRP route being received?
Routers must be EIGRP neighbors for the routing information to be shared. To check this,
issue a show ip eigrp neighbors on the two routers exchanging hellos. You should see the
neighbor listed on each device.
You can also perform a debug ip eigrp packets to make sure hellos are being sent out
from each router.
If all of that looks good, look at the EIGRP running configuration and make sure the AS
numbers match, the timers are close, and that any authentication configurations are the
same.
Next, issue a show ip eigrp interface to make sure the interfaces you expect are
participating in the EIGRP process. Lastly, route maps or distribution lists could be blocking
routing traffic. Do a show ip protocols to display any distribute lists.
Is there a more desirable route being used?
Its possible that EIGRP knows about the route, but it is not being used in the routing table.
If a more desirable path is known, that will be used instead. Compare the EIGRP topology
table to the local routing table.
OSPF
These steps for troubleshooting OSPF are very similar to EIGRP. First, verify that there is a
problem using pings and by taking a look at the routing table.
OSPF stores its information in three different tables: the OSPF interface table, neighbor table,
and link-state database.
OSPF Interface TableThe OSPF interface table displays interfaces participating in the local
OSPF processes. Use the show ip ospf interface command to display its contents.
OSPF Neighbor TableThe neighbor table contains a list of discovered OSPF neighbors. Use
the show ip ospf neighbors command to display its contents.
OSPF Link State Database
The link state database contains the received LSAs. Use the show ip ospf database
command to display its contents.
Is the OSPF route being advertised properly?
The first step is to identify the router that is connected to the destination subnet as it should
be advertising the route out. There are two simple ways to check if that router is advertising
the routes properly.
First, do a show run | section ospf. This will display the running OSPF configuration,
including what networks are being advertised with the network statements. Another option
is to do a show ip protocol.

Remember, OSPF only advertises subnets of interfaces that match an OSPF network
statement.
Is the OSPF route being received?
Routers must be OSPF neighbors for the routing information to be shared. To check this,
issue a show ip ospf neighbors on the two routers. You should see the neighbor listed on
each device.
You can also perform a debug ip ospf adj to show any issues that would prevent the
routers from forming an adjacency.
OSPF is more particular about matching protocol variables than EIGRP. OSPF requires that all
of the following parameters match between devices:
Bidirectional communication
AS number
Timers
Common area type
Common subnet prefix
Authentication
The OSPF protocol values can be seen using the show ip ospf interfaces command.
Lastly, route maps or distribution lists could be blocking routing traffic. Do a show ip
protocols to display any distribute lists.
Is there a more desirable route being used?
Its possible that OSPF knows about the route, but it is not being used in the routing table. If
a more desirable path is known, that will be used instead. Compare the OSPF topology
table to the local routing table. Take the time to check each hop along the expected path
and look at the routing tables on each router.
BGP
BGP stores its information in two tables: the BGP neighbor table and the BGP table.
BGP Neighbor Table
The neighbor table contains a list of known BGP neighbors. Use the show ip bgp
neighbors command to display its contents.
BGP Table
This table contains all the received BGP prefixes as well as their associated attributes lists.
Perhaps most importantly, it also shows the BGP best path to each destination. Use the
show ip bgp command to display its contents.
Are the BGP routers neighbors?
BGP neighbors must be administratively assigned on each router running BGP. If the routers
are not neighbors, BGP routing and network information will not be passed between them.
Start by doing a show ip bgp neighbors. If the expected BGP peers do not show up in the
output, make sure they have L3 connectivity using a simple ping test. If you need to
investigate further, a debug ip bgp updates should show the BGP hellos and
advertisements.
Remember that BGP requires bidirectional communication as well as matching AS numbers
and authentication. The show run or show ip bgp command will display that information.
Also, consider that route maps or distribution lists could be blocking routing traffic. Do a
show ip protocols to display any distribute lists.
Is the BGP route being advertised?
As with the other routing protocols, make sure that the router connected to the destination
subnet is advertising the route out. There are two simple ways to check if that router is
advertising the routes properly.

Perform a show run | section bgp to look at the neighbor statements. You should also
keep in mind that BGP will only advertise routes when (1) they are defined using neighbor
statements and (2) the router knows about the route from another source.
Route Redistribution
Route redistribution can be a tricky situation to troubleshoot, but understanding the
following concepts should be helpful.
1. Redistributed routes require an existing entry in the routing table. If the
redistributing router does not have a routing table entry for the route being redistributed, it
will not work. Seems simple, but it should checked right away.
2. Routing loops are a common problem with multi-router routing redistribution.
Use a single router to perform the redistribution if possible.
3. Understand that redistributed routes lose their native metric information.
When redistributing into EIGRP, a default metric MUST be set or no route will be imported.
When redistributing into OSPF, all routes will be imported as classful unless the subnets
keyword is appended to the end of the redistribution statement.