Introduction to Network Troubleshooting

Ann Harding April 2005

Preparation
Learn the network
Topology
\Network_Diagrams\PhysicalNetwork\HEAnet-PhysicalNetworkMap.vsd \Network_Diagrams\Multicast\multicast-map.vsd

Protocols
IPv4, IPv6, Multicast

Routing
Policy, protocols https://www.hea.net/docs/bgp.html, https://www.hea.net/docs/v6bgp.html, https://www.hea.net/docs/igp-costs.html

Equipment
http://www.cisco.com/en/US/customer/support/index.html http://www.juniper.net/customers/support/

Understand normal network behaviour

https://www.hea.net

START Define Problem Gather Facts FINISH Document Facts Has anything changed? The Internetsymptoms? is broken What are theResolved What is happening? Problemtroubleshooting Check possible causes as you go Re-evaluatetime you change Section down on Netsaint the Check each often, where? XYZ issome checking: When, how possibilities Do invasive impact Limit Narrow problem definition Change one variable at a time Referaredown a website to possible I cant get to RANCID, Whatprepared to roll back Routers, Syslog, causes? Be at a source device Identify what youfact-gathering Start Refer to testsManagement in need to check Network Y

Consider Possibilities Troubleshooting Methodology Create Action Plan Implement Action Plan Observe Results
Do Problem Symptoms Stop?

Iterate Process

ATM links
Two main setups
Eircom Symphony ATM Service
Client and HEAnet have two different physical access circuits Eircom provision the PVC through their network Client side of the PVC may or may not have an ATM switch present

HEAnet ATM over Esat links

Virtual path and physical path similar Telco unaware of ATM aspect Client side of the PVC may or may not have an ATM switch present

ATM links
Docs reference:
https://www.hea.net/docs/clientlinks-atm-eircom.html https://www.hea.net/docs/clientlinks-atm-pointtopoint.html

Determine which switch and router the circuit is on and telnet to them. See if you can ping the other end of the link from the router Check the logs using show log on the relevant switch and access router and look for linkup/linkdown messages.
If the interim links are down, there is nothing for the Virtual Circuit to run over.

Check the ATM VC on the router and switch

show atm vc to check the status of the virtual circuits.

Check the interfaces on the router and switch using show interface

ATM links
Ls1010 interface example
ATM0/0/0 is up, line protocol is up Hardware is oc3suni This means the interface hardware is active. Description: STM-1 DIAS (Esat HEAN0013) Line protocol is up means that keepalives are successful. MTU 4470 bytes, sub MTU 4470, BW 155520 Kbit, DLY 0 usec, rely If three successive keepalives fail, line protocol will go down. 255/255, load 1 /255 Encapsulation ATM, loopback not set, keepalive not supported Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters never Queueing strategy: fifo Output queue 0/40, 0 drops; input queue 0/75, 0 drops 5 minute input rate 19000 bits/sec, 65 packets/sec 5 minuteNeither input nor outputbits/sec, 400 packets/sec output rate 168000 errors should be accumulating. If in doubt, input, counters and check 0 interface again. 456031409 packets clear the2694828197 bytes, the no buffer Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 89 input errors, 93 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 1839217443 packets output, 2989243967 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 output buffer failures, 0 output buffers swapped out

ATM Links
Router interface example
mantova>sh int atm3/1.8 ATM3/1.8 is up, line protocol is up Hardware is ATM OC12 LC, address is 0011.21b3.bf80 (bia ATM and protocol may show up, even if there is a problem with the circuit 0011.21b3.bf80) as it refers only to the connection between the router and the LS1010. Description: DIT 90Mbps (CBR AVC502353) Where OAM management is being configured, this will pull this link down in Internet address is 193.1.194.41/30 the event of a break anywhere on the VC path. MTU 4470 bytes, BW 90000 Kbit, DLY 80 usec, reliability 255/255, txload 22/255, rxload 4/255 Encapsulation ATM 1096645466 packets input, 298037834034 bytes 1552827776 packets output, 1414170448777 bytes 239959 OAM cells input, 241916 OAM cells output

Serial and Gigabit Ethernet links

Docs reference
https://www.hea.net/docs/clientlinks.html#clientlinks-serial https://www.hea.net/docs/clientlinks-gigabit.html

Determine what router the circuit is on and telnet to it. Test to see if you can ping the other end of the link Check the logs and look for linkup or linkdown messages, or OSPF transitions. Check the interface for the affected circuit using the show interface command

Serial interface example

Serial1/0 is up, line protocol is up Hardware is M4T Description: NQAI (DS0858730) InternetThis means the interface hardware is active. address is 193.1.194.25/30 MTU 1500Line protocol64 up means that keepalives are successful.255/255, bytes, BW is Kbit, DLY 20000 usec, reliability If three successive keepalives fail, line protocol will go down. txload 1/255, rxload 23/255 Encapsulation HDLC, crc 16, loopback not set Keepalive set (10 sec) Last input 00:00:04, output 00:00:00, output hang never Last clearing of "show interface" counters 7w3d Queueing strategy: fifo Neither input nor output input queue 0/75, 0 drops Output queue 0/40, 261 drops;errors should be accumulating. 5 minute If in doubt, clear thebits/sec, 0 packets/sec input rate 6000 counters and check the interface again. 5 minute output rate 0 bits/sec, 0 packets/sec 3779614 packets input, 363897481 bytes, 3 no buffer Received 0 broadcasts, 0 runts, 0 giants, 0 throttles The number of carrier transitions should not be incrementing. 534 input errors, 300 CRC, 0 frame, 0 overrun, 0 ignored, 234 abort Clear counters and check. 3868414 packets output, 1209721840 bytes, 1 underruns 1 output errors, 0 collisions, 34 interface resets 0 output buffer failures, 0 output buffers swapped out 68 carrier transitions DCD=up DSR=up DTR=up RTS=up CTS=up

IP connectivity
Ping cisco output
! successful reply . timeout waiting for reply U destination unreachable N network unreachable P protocol unreachable Q source quench M could not fragment ? Unknown packet type

IP connectivity
trace(route|rt) cisco(linux|windows)
Terminates when destination responds or max TTL exceeded Cisco version includes AS routing info Look at msec values, not how slow the info appears on your screen! Remember hostnames are determined by DNS

IP connectivity
TCP/IP show commands
show ip route [address] | show route [address]
Shows entries in routing table Shows protocols that derive the route Shows administrative distance and metric of the route Shows the next hop router address Shows the interface where the route can be reached

IP connectivity
hyperion>sh ip route 193.1.219.90 Routing entry for 193.1.219.0/24 Known via "ospf 1", distance 110, metric 20 Tag 1000, type extern 2, forward metric 1 Last update from 193.1.196.174 on GigabitEthernet3/0/0, 00:04:06 ago Routing Descriptor Blocks: * 193.1.196.122, from 193.1.195.33, 00:04:06 ago, via GigabitEthernet3/3/0 Route metric is 20, traffic share count is 1 Route tag 1000 193.1.196.174, from 193.1.195.33, 00:04:06 ago, via GigabitEthernet3/0/0 Route metric is 20, traffic share count is 1 Route tag 1000

IP Connectivity
TCP/IP show commands
show ip protocols
What routing protocols are running Frequency of updates Redistribution information Routes for which the routing process is injecting routes

IP Connectivity
ROUTE SOURCE Connected Interface Static route EIGRP summary route External BGP Internal EIGRP IGRP OSPF IS-IS RIP EGP ODR External EIGRP Internal BGP Unknown not entered into Cisco routing table DEFAULT VALUE 0 1 5 20 90 100 110 115 120 140 160 170 200 255

IP connectivity
TCP/IP show Commands
show ip access-list | show firewall [filter <name>]
Gives ACL number/name and policies and counts

show ip interface | show interface detail

Gives all addresses on interface Multicast group membership information MTU ACLs applied

show ip traffic | show interface queue

Shows information on reasons for packet discards

Exercises
Use show ip route to trace the path from Titan to Cyclops
Compare what you find out to the output of trace

Use a show command to find out if there are ACLs on NUIMs link
How would you check how many matches?

Check for errors on the UCDs primary connection

Are there any dropped packets or likely circuit problems?

On Deimos, what type of routes are being redistributed into which routing protocols?

Reading
Cisco Internetwork Troubleshooting HEAnet Documentation
https://www.hea.net/docs/

http://www.cisco.com/warp/public/535/4.html
Cisco TCP/IP tutorial