Beruflich Dokumente
Kultur Dokumente
Network Troubleshooting
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Student Objectives
The Network Troubleshooting module presents a systematic methodology for troubleshooting, how to use the troubleshooting tools provided in Extreme Networks switches, and examples of how to identify common problems. Upon completion of this module, the successful student will be able to:
Identify the network management and maintenance systems that prepare you to troubleshoot. Describe the Layered approach used for troubleshooting. Identify useful commands for finding errors in OSI Layers 1, 2, and 3. Interpret the output of show commands. Use correct to troubleshoot problems at the Physical, Data lLnk, and Network Layers. List the steps required to perform systematic troubleshooting.
10
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
11
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Overview
Troubleshooting is an important part of managing any network. This module presents a systematic methodology for troubleshooting, how to use the troubleshooting tools provided in Extreme Networks switches, and examples of how to identify common problems. The troubleshooting topics covered are:
Gathering and maintaining the information and tools necessary to troubleshoot the network. Organizing and applying a systematic method of troubleshooting. Identifying which commands to use to systematically troubleshoot a network problem. Describing how to test the operation of the switch.
12
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Figure 2: Overview
13
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Create an illustration of the physical connectivity of the network and keep it up to date. Create an illustration of the logical connectivity of the network and keep it up to date. Set up a Trivial File Transfer Protocol (TFTP) server and:
Store copies of the active and previous versions of configuration files and images, so that you can access them quickly in case of a problem. Make a backup of each switch configuration file on a regularly scheduled basis. Update the stored configuration files after making any changes.
Maintain a contact list of vendors, service suppliers, and users to inform in case of a problem. Store user names, passwords, and SNMP default community strings in a safe location and change (or remove) the default values. Record network problems, symptoms, and the resolution.
Establish baselines for network response time, and traffic statistics. Test connectivity and response time on a regularly scheduled basis and compare the response time to the baseline. Monitor traffic statistics on a regularly scheduled basis and compare the statistics to the baseline. Use Extreme Networks EPICenter to
Monitor alarms and events. Verify the physical and logical topology. Monitor network, VLAN, and port statistics. Set traffic thresholds. Store a syslog of switch activity.
Provide a clean/conditioned environment (air and power) and provide proper ventilation depending on hardware type. Some switches vent front to rear, others vent side to side. Software maintenance such as installing new software to fix bugs or add functionality.
14
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
15
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
16
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
17
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
First check the physical connectivity. Is the connector seated properly? Ensure that the physical medium connecting to the interface port is fault free. Isolate the problem by swapping cables with ports. Does the problem follow? Check the status of the LEDs. Use loop back plugs or a loop back cable to verify the link. Be aware of the surrounding environment for potential electro-mechanical interference (EMI).
Verify that the correct cables are being used. Distinguish between:
Media type Length of the cables Crossover cable versus straight cable
Verify that the port settings are configured properly. Common problems in port settings are:
Auto negotiation of speed and duplex is set incorrectly. The port is administratively disabled. Speed and duplex settings do not match the settings of the connected device. Load sharing configuration is set improperly.
18
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
19
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
The description for port LEDs in Table 1 applies to ports on the SummitX450 family of switches, Summit 200, Summit 300, and Summit 400.
The description for port LEDs in Table 2 applies to ports on the BlackDiamond 8800 series, BlackDiamond 10808 series, and non-POE ports on the BlackDiamond 12804 switches.
The description for port LEDs in Table 3 applies to POE ports with power enabled on the BlackDiamond 12804.
20
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
21
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
22
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
23
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Primary Information
The show ports configuration command output shows:
The port number and virtual router. The port state as enabled (E) or disabled (D). The link status as ready (R), active (A), or not present (NP). If auto negotiation of speed and duplex is enabled (ON) or disabled (OFF). The link speed configured and if auto negotiation is enabled, the actual speed. The duplex mode configured and if auto negotiation is enabled, the actual duplex as half or full. Link media type for the primary and if configured, redundant port. Examples of media types are:
Unshielded Twisted Pair (UTP) Multimode Gigabit Interface Connector (MGBIC) Small Formfactor Pluggable (SFP)
Use this information to verify that the intended ports are enabled and active. Also verify that the speed, duplex and frame size match the other end of the physical connection. If the switch supports user-created virtual routers, verify that the VLAN router interface (RIF) is configured on the correct VR. Commands such as ping and tftp require that the VR be specified. The media type is useful if redundant fiber and copper ports are supported. In this case the port number is the same for both the fiber and copper port.
Secondary Information
The command output also shows:
If Ethernet flow control is enabled symmetrically in both directions (SYM) or asymmetrically in one direction (ASYM). 10Gbps ports can be asymmetrical. The load sharing master port if the port is participating in link aggregation.
Use this information to verify that the port settings and media type match the other end of the physical connection.
24
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
25
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Transmit Collisions (TX Coll). The total number of collisions seen by the port, regardless of whether a device connected to the port participated in any of the collisions. Transmit Late Collisions (TX Late Coll). The total number of collisions that have occurred after the ports transmit window has expired. This could indicate a duplex setting mismatch. Transmit Deferred Frames (TX Deferred). The total number of frames that were transmitted by the port after the first transmission attempt was deferred by other network traffic. Transmit Errored Frames (TX Errors). The total number of frames that were not completely transmitted by the port because of network errors (such as late collisions or excessive collisions). Transmit Lost Frames (TX Lost). The total number of transmit frames that did not get completely transmitted because of buffer problems (FIFO underflow). Transmit Parity Frames (TX Parity). The bit summation has a parity mismatch.
Receive Bad CRC Frames (RX CRC). The total number of frames received by the port that were of the correct length, but contained a bad FCS value. This could indicate a duplex setting mismatch. Receive Oversize Frames (RX Over). The total number of good frames received by the port greater than the supported maximum length of 1,522 bytes. Receive Undersize Frames (RX Under). The total number of frames received by the port that were less than 64 bytes long. Receive Fragmented Frames (RX Frag). The total number of frames received by the port were of incorrect length and contained a bad FCS value. Receive Jabber Frames (RX Jabber). The total number of frames received by the port that was of greater than the support maximum length and had a Cyclic Redundancy Check (CRC) error. Receive Alignment Errors (RX Align). The total number of frames received by the port that occurs if a frame has a CRC error and does not contain an integral number of octets. In a shared Ethernet environment, alignment errors could be the result of collisions. Receive Frames Lost (RX Lost). The total number of frames received by the port that were lost because of buffer overflow in the switch.
26
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
27
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Collision statistics for each port. The numbers 1 to 16 represent the number of collisions and subsequent retransmissions encountered prior to successfully transmitting the packet. Collision statistics are applicable only for half-duplex links.
Collisions are primarily a problem for shared Ethernet environments where devices are attached through a hub. If you see collisions, it indicates that someone has connected a hub to the switch port.
28
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
29
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
NOTE
Monitoring port utilization is useful to identify broadcast storms caused by loops in the network. Extreme Networks switches are non-blocking. So, links can be utilized up to 100% when there is a network loop.
Enter the show ports utilization command then toggle the screen using the space bar. The first two screens in the output show transmitted and received port byte and packet counts. Use this information to verify that the ports are passing traffic. The third screen in the show port utilization command output shows:
The port number, link status, and link speed. The current and peak port utilization received as a percentage of the maximum bandwidth. The current and peak port utilization transmitted as a percentage of the maximum bandwidth.
Use this information to verify that there are no Layer 2 forwarding loops. If a Layer 2 forwarding loop exists it causes a broadcast storm. During a broadcast storm the port utilization reaches close to 100%.
NOTE
If a forwarding loop exists it may take a minute of two for the utilization to show the broadcast storm.
To clear the port utilization counters, enter the following command: clear counters ports
30
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
31
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
The EDP interval timers and which ports have EDP enabled.
Use this information to verify that EDP is enabled on the intended ports and that the timers match the other end of the connection. The show edp ports all command output shows:
The SNMP sysname of each EDP neighbor on every active port. The MAC address of each EDP neighbor. The slot and port number of the other end of the connection. The age of the EDP entry. The number of VLANs supported on this link.
Use this information to verify that the ports are physically connected to the intended EDP neighbor on the intended port. You can also use this command for troubleshooting at the Data Link Layer because you get information about the VLANs on all neighboring switches.
32
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Figure 13: Displaying Connectivity and Configuration Information for Neighboring EXTR Switches
33
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Port Number Link Status The current status of the link. Options are:
Ready (the port is ready to accept a link) Active (the link is present at this port) Disabled (D): The link is disabled at this port. Not Present (NP): The link is not present at this port.
Transmit Collisions: The total number of collisions seen by the port, regardless of whether a device connected to the port participated in any of the collisions. Transmit Late Collisions: The total number of collisions that have occurred after the ports transmit window has expired. Transmit Deferred Frames: The total number of frames that were transmitted by the port after the first transmission attempt was deferred by other network traffic. Transmit Errored Frames: The total number of frames that were not completely transmitted by the port because of network errors (such as late collisions or excessive collisions). Transmit Parity Frames: The bit summation has a parity mismatch.
34
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
35
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Switch 1 Symptoms
On a switch running in half duplex mode there are a lot of late collisions. The attached switch, running in full duplex mode, sends frames if there is data. It does not detect if the media is already used or not.
Switch 2 Symptoms
On a switch running in full duplex mode there are a lot of CRC errors. The attached switch, running in half duplex mode stops transmitting data each time it detects a collision. The illustration shows Switch 1, running in half duplex mode and Switch 2, running in full duplex mode. The show ports txerrors command on Switch 1 shows the late collisions. The show ports rxerrors command on Switch 2 shows the CRC errors. The following port receive error information is collected by the system:
Port Number Link Status The current status of the link. Options are:
Ready (the port is ready to accept a link) Active (the link is present at this port) Disabled (D): The link is disabled at this port. Not Present (NP): The link is not present at this port.
Receive Bad CRC Frames (RX CRC)The total number of frames received by the port that were of the correct length, but contained a bad FCS value. Receive Oversize Frames (RX Over)The total number of good frames received by the port greater than the supported maximum length of 1,522 bytes. Ports with jumbo frames enabled do not increment this counter. Receive Undersize Frames (RX Under)The total number of frames received by the port that were less than 64 bytes long. Receive Fragmented Frames (RX Frag)The total number of frames received by the port were of incorrect length and contained a bad frame check sequence (FCS) value. Receive Jabber Frames (RX Jabber)The total number of frames received by the port that were longer than the supported maximum length and had a Cyclic Redundancy Check (CRC) error. Receive Alignment Errors (RX Align)The total number of frames received by the port that have a CRC error and do not contain an integral number of octets. Receive Frames Lost (RX Lost)The total number of frames received by the port that were lost because of buffer overflow in the switch.
36
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
37
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
There might be protocol mismatches because of the different L2 protocols. The Ethernet II, IEEE 802.3, or LLC/SNAP setting causes the mismatch. A wrong driver in the Ethernet card of a server might also cause a mismatch.
Addressing Problems
Start by answering the following questions to identify possible addressing problems. Are there permanent MAC address entries in the Forwarding Database (FDB) which are incorrect? Is there a faulty or misconfigured device that is responding to all ARP requests on a VLAN or network? When troubleshooting spanning tree, answer the following questions:
Which mode of spanning tree are you using? Does your spanning tree domain layout and your VLAN layout work together? Is the VLAN in the correct spanning tree domain? Is spanning tree enabled for the VLAN?
VLAN Problems
When troubleshooting VLAN problems verify the following:
The ports are properly configured on the VLAN. Consistent VLAN tags are used.
If VLAN tags are not properly configured, your 802.1Q link cannot work. Remember that VLAN names are only locally significant but you should use a consistent naming scheme, to prevent confusion and obstruct troubleshooting.
The VLAN name is not a reserved word or the name includes special characters that are not allowed or the switch will not accept the name.
VLAN names consist of alphanumeric characters, must begin with an alphabetical letter, no spaces are allowed, and special characters are only allowed if the VLAN name is enclosed by quotation marks.
The protocol filter is set to ANY or the desired protocol filter when using protocol based VLANs. When the filter is set to NONE, protocol based VLANS will not work. The protocol filter is set to NONE when a special protocol filter is deleted. The port you are adding is not already added to a different VLAN untagged or you will receive an error message.
38
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
39
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
40
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
41
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Primary Information
The show fdb command output shows:
The Ethernet MAC address of each learned device in the forwarding database. The VLAN where the device is a member. The port where the address is learned.
For troubleshooting Layer 2 and Layer 3, verify that the expected device is physically connected to the port and is being learned. Verify that the entry is not a black hole. Verify that the device is in the right VLAN(s).
Secondary Information
The show fdb command output also shows:
The age of the FDB entry. The FDB statistics on total learned, static, locked, permanent, dynamic, dropped, locked, and locked with a timer. The age timer and VPLS age timer.
If the entry is learned dynamically (d) or configured statically (s). If the entry is permanent (p). If the entry is locked down or locked with a timer (l, L). If the entry is on a port being used as the mirror port (M). If the entry is created by NetLogin (n).
Use the following command to clear dynamic FDB entries: clear fdb {<mac_address> | locked-mac <mac_address> | vlan <vlan name> | ports <portlist>}
42
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
43
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
If the VLAN is enabled or disabled. If the VLAN is tagged and what the tag is. The virtual router where this VLAN is assigned. The IP address and subnetwork mask of the router interface of this VLAN. If loopback mode is enabled. The tagged and untagged ports assigned to this VLAN. Flags indicating if each port is:
active (*). disabled (!). part of a load sharing group (g). blocked for this VLAN (b).
For Layer 1 problems verify that the port is enabled, active, and part of or not part of a load sharing group. For Layer 2 problems verify that the port is:
For Layer 3 related problems, verify that the following are correct:
44
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
45
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
The IPv6 address if one is assigned. The Spanning Tree Protocol Domain (STPD) name if one is assigned. The VLAN protocol filter name if one is assigned. If the NetLogin security feature is enabled. The name of the QOS policy file used to assign quality of service to packets on this VLAN if one is assigned. Flags indicating if each port is authenticated (a) or unauthenticated (u) for this VLAN by Netlogin.
For Layer 1 problems verify the port is authenticated if applicable. For Layer 2 problems verify that the VLAN:
is in the correct Spanning Tree domain. has the correct protocol filter if applicable. is correctly configured for security and QOS features if applicable.
46
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
47
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
48
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
49
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Every device must have a unique host address. Devices with the same subnetwork address have to be within the same VLAN. When devices are in separate VLANs, those VLANs must have different subnetwork addresses and the devices providing connectivity between them must be configured to forward Layer 3 traffic. Every device involved in Layer 3 forwarding must have a route to reach the network of the destination IP address or have a default route to the right gateway.
Routing Advertisements
Extreme Networks switches do not advertise routes if:
The VLAN does not have an IP address assigned to it. IP Forwarding is not enabled for that VLAN. The VLAN has not been added to the Routing Protocol. The Routing Protocol has not been enabled globally.
Does the switch discover its neighbors and establish adjacencies? Are all routers in the same area? Is OSPF enabled on the VLAN and globally? Are the timers correct? Are the Link State Databases synchronized? Is a link state advertisement (LSA) missing? Is SPF running too often because of flapping links? Is authentication used, ECMP enabled, and the Router ID unique?
In a stable network, the Link State Database (LSDB) does not change much. One way to determine whether the entries in the LSDB are changing is to monitor the checksum and SPF runs. These are indicators of how often the LSDB is changing. Using multiple areas there are issues concerning stub area, NSSA, default routes, ABRs, ASBRs, Virtual Links, Route Summarization and more.
50
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
51
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
The name of the VLAN. The IP address and subnetwork mask of the router interface. If IP forwarding (f) is enabled. If OSPF is enabled on the VLAN (o). If RIP is enabled on the VLAN (r). If the VLAN loopback is enabled (L).
52
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
53
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
The name of the VLAN. The IP address and subnetwork mask of the router interface. If IP forwarding (f) is enabled. If the interface is enabled (E) and active (U).
The interface is up IP forwarding is enabled The interface has the correct IP address and subnetwork mask
54
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
55
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
The destination network. The next hop gateway. The preferred route for unicast and multicast traffic. The duration of time this route has been in the routing table.
Use this information to verify that the destination network is in the routing table. Verify where the packets are being sent for a particular designating network. Also verify if the preferred route is the intended route. If a destination network is unreachable, check the following:
Every router in the path knows the destination network The default gateway is in the correct direction The return path is correct
Verify that directly connected routes are used when IP route sharing is enabled using Equal Cost Multi Paths (ECMP). For IP route sharing directly connected routes should be used instead of other routes even if the cost or relative route priority of the other routes has been manually set. The type of route entry flags show:
The route is Dynamic (D). The route is a Black hole (B). The route is a Gateway (G). The route is a Host Route (H). The route is a Label Distribution Protocol (LDP) Label Switch Path (LSP). The route is an indirect LDP LSP. (i) The route is a multicast (m) or unicast (u) route. The route is a Longest Prefix Match (LPM) route (P). The route is modified (R). The route is a Static route (S). The route is a Resource Reservation Protocol (RSVP) Traffic Engineering (TE) LSP. (T) The route is an indirect RSVP-TE LSP. (t) The route is up (U).
56
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
57
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Directly connected interfaces (d) Border Gateway Protocol (bg), External BGP (be), or Internal BGP (bi) BootP (bo) Core Based Tree (CBT) multicast routing protocol (ce) Down Interface (df) Distance-vector Multi Routing Protocol (dv) Intermediate System - Intermediate System (ISIS) level-1 external (e1) (ExtremeWare only) Intermediate System - Intermediate System level-2 external (e1) (ExtremeWare only) ISIS level-1 internal (i1) and ISIS level-2 internal (i2) (ExtremeWare only) Hardcoded (h) Internet Control Message Protocol (ICMP) (i) Multicast Border Gateway Protocol (MBGP) (mb), MBGP external (mbe), or MBGP internal (mbi) Multicast Open Shortest Path First (MOSPF) (mo) OSPF external LSA type 1 (o1) and OSPF external LSA type 2 (o2) OSPF intranetwork (oa) and OSPF internetwork (or) OSPF external Autonomous System (oe) Protocol Independent Multicast - Dense Mode (pd), PIM-Sparse Mode (ps) Routing Information Protocol (RIP) (r) Route Advertisement (ra) Static (s) Server Load Balance (SLB) virtual server IP (VIP) (sv) (ExtremeWare only) Unknown (un)
Use this information to determine the source of the routes. This is useful for the following:
When using route redistribution When troubleshooting routing loops When unexpected routes are showing up in the routing table
Directly connected routes have a relative route priority of 10. Relative route priorities cannot be modified for directly connected routes or black hole routes. Redistribution of RIP routes into OSPF results in OSPF external routes with different relative priorities.
58
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
59
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Each Ethernet MAC address that has been mapped by ARP. The IP address mapped to the MAC address. If the entry is marked incomplete, it means the switch is arping for an IP address that is not responding. The port on which the MAC address is located. The VLAN on which the MAC address is located. The virtual router associated with the VLAN on which the MAC address is located. A list of all duplicate IP addresses detected.
Use this information to verify that each IP address is mapped to a single MAC address. Duplicate IP addresses are a very common misconfiguration and can cause confusing and unpredictable behavior. Also verify that the IP address is in the correct VLAN and subnetwork. To remove dynamic entries in the IP ARP table, enter the following command: clear iparp {<ip address> | vlan <vlan name>}
Secondary Information
The show iparp command output also shows:
If the ARP mapping entry is statically configured. The age of the entry in seconds. Statistics for:
The current number of dynamic and static entries. The current number of pending entries. The ARP request is sent but no response has been received so far. The number of ARP requests received (in request), ARP responses transmitted (out response), ARP requests sent (out request), and ARP responses received (in response). The number of failed requests. A request was sent but no response was ever received. The number of received ARP responses that were rejected due to some protocol error in the packet. For all ARP responses that were rejected what is the count, port, IP address, and interface.
The maximum number of ARP entries and ARP pending replies configured or supported on this switch. Whether ARP address checking and ARP refresh are enabled. Checking checks if the ARP Request source IP address is within the range of the local interface. Disabling ARP refresh stops the sending of any ARP requests before the ARP entry times out. The ARP entry age timeout.
Use this information to verify that ARP is working correctly. Also use this to verify that the proxy ARP server is responding
60
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
61
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
If OSPF is enabled. The router ID and whether it is configured or automatically assigned. If this router is an ASBR or ABR. If the router ID is configured or automatically selected.
Use this information to verify that OSPF is enabled globally on the router. Verify that the router ID is configured correctly. Virtual links are built using the router ID. Verify that the router is acting in the right role: Interior Router, ABR, or ASBR.
62
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
63
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
64
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
65
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Packet statistics and error statistic for IP. Packet statistics and error statistic for the Internet Control Message Protocol (ICMP). Packet statistics and error statistic for the Internet Group Management Protocol (IGMP).
Use the IP statistics to identify if errors are causing the protocol to fail or converge slowly. Use the ICMP statistics to identify if errors are causing connectivity failures or ping response failures. Use the ICMP statistics to identify if errors are causing problems with multicast routing.
The IP address and network mask of the router interface Packets and Bytes in and out Multicast and Broadcast packets in and out Packets with errors and discarded packets Packets received of a unknown protocol type
66
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
67
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
NOTE
If you ping a router interface that is configured on the switch and is active, the router interface responds with an ICMP echo response even if IP forwarding is not enabled. The router interface may appear to be forwarding packets at Layer 3 even though it is not.
Use the following command to trace the routed path between the switch and a destination end station.: traceroute {vr <vrid>} {ipv4 <host>} {ipv6 <host>} {ttl <number>} {from <from>} {[port <port>] | icmp} Traceroute works by increasing the time-to-live (TTL) value of each successive batch of packets sent. The first three packets have a time-to-live (TTL) value of one (implying that they make a single hop). The next three packets have a TTL value of 2, and so on. The command output shows the IP address of the router interface at each hop along the path and the number of milliseconds it took each packet to receive an ICMP time exceeded packet from that interface. The command continues until the user enters Ctrl-c to abort the command. The traceroute command is useful in determining which route a packet is taking when multiple routes are available. It is also useful in identifying firewalls that may be blocking access to a site. If the trace route cannot reach the destination the command output identifies the reason why. Failures can be due to:
ICMP network unreachable - There is no path. Check the routing table. Make sure you are specifying the correct virtual router. Make sure IP forwarding is enabled. ICMP host unreachable - Make sure you entered the correct address. ICMP fragmentation needed - The packet is too large for one of the routers in the path. Enable fragmentation on the router. ICMP source route failed - Source routing is seldom supported. Transmit error - Check IP statistics for IP errors.
68
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
69
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
NOTE
Part of the output is omitted from the illustration. What is displayed is the overview of the router interfaces because that is what is important.
For the VLAN named noipforwarding, IP forwarding is not enabled. This is the cause of the problem.
70
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
71
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
72
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
73
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
A link going down. A user logging in. A command entered on the command line. The software executing a debugging statement.
A timestamp of when the event was logged. The timestamp is in the format yyyy-mm-dd,hh:mm:ss. The sending devices IP Address. The IP Port number used for receiving log messages. 23 in the example. The Syslog Facility number 7 is the default for Extreme Networks devices. The protocol reporting the event. OSPF in this example. The actual log message.
Start Intra area SPF Area 0.0.0.10. (The start of an SPF run for area 0.0.0.10) Start IntraArea Route Table update SPF area 0.0.0.10. (The start of the routing table update) End IntraArea Route Table update area 0.0.0.10. (The end of the routing table update) End IntraArea Spf Area 0.0.0.10. (The end of an SPF run for area 0.0.0.10) NOTE
The syslog facility can be set to different values and the receiver can be programmed to treat different facility levels uniquely.
74
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
75
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Timestamp - records the month and day of the event, along with the time (hours, minutes, seconds, and hundredths of a second). Severity Level - indicates the urgency of a condition reported in the log. Severities include critical, error, warning, notice, info, debug-summary, debug-verbose, and debug-data. Component, Subcomponent, and Condition Name - describes the subsystem in the software that generates the event. This provides a good indication of where a fault might be. Message - A description of the event occurrence. If the event was caused by a user, the user name is also provided.
Use this information in the same way you use the remote system logging messages. Logging of CLI configuration commands must be enabled. The switch log overwrites existing log messages in a wrap-around memory buffer.
76
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
77
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Ask questions to clearly define the problem. Gather information about the network. Consider escalating the problem to someone with additional expertise. The goal is to reach a resolution as fast as possible. Determine if the problem can be consistently reproduced or if it is intermittent. Document the history of the problem. Identify when the problem was first noticed and what events preceded the problem occurrence. Locate information about similar problems. This may indicate possible sources of the problem. Develop theories about what could be wrong. Start with simple ideas and advance to more complex issues. Test theories starting with the most inexpensive and simple, then test the more expensive and complex theories until the solution is found. Always document the problem, theories, tests, and end resolution. Retain the problem documentation and make it available for future troubleshooting.
78
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
79
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
What are the symptoms? Is the problem reproducible? How long has the problem been happening? How often does the problem happen? When does the problem happen? Is the issue affecting only one device or a large portion of the network? Is the issue isolated within a VLAN or does it also affect Layer 3 functions? How many users are affected? Which applications are affected? Does the problem seem to be related to the network load? Does the problem seem related to a new network installation or change? Was the network previously operating properly?
How long was the network operational prior to this issue arising?
80
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
81
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
A detailed network diagram with all connected devices, network addresses, and physical media types
You have to know your network in a stable situation, so that you know what to expect when looking at the status of the network.
Availability reports Records of recent changes made to the network Current status of devices and connections Event and error logs
Keep manuals and release notes of implemented equipment ready to use. Make sure that everybody involved in troubleshooting has access to this information.
82
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
83
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Consider Escalation
Before you escalate a problem to technical support, ask the following:
What is the impact on the users? What is the impact on the resiliency of the network? What is the cost per day of the problem? Does the problem seem to be in an area you have expertise in? How much time has already been spent on troubleshooting? What is covered in your support contract?
Escalating does not mean you have to stop troubleshooting. Escalation is most effective when you provide a good problem description and all relevant network information.
Your company name and the phone number of the contact Serial number(s) of the switch chassis and additional module(s) Service contract number Detailed problem description Output of the following commands:
show version show switch show configuration show tech-support or show diagnostics show log run diagnostics extended
84
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
85
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Testing Theories
A layered approach gives structure to the troubleshooting process. Use the OSI model to structure tests. Start testing the lowest layers first then work your way up the OSI layers. For example: 1. Use the link Light Emitting Diodes (LEDs), show commands, and physical inspection of the cable to verify the Physical Layer. 2. Use the ping command, verify packet statistics, verify port settings, and verify the Layer 2 forwarding tables to test Layer 2 connectivity. 3. Use an end-to-end ping command, trace route command, verify Layer 3 configuration, and verify Layer 3 route tables to test Layer 3 connectivity. 4. Use telnet, FTP, TFTP, or SNMP to test protocols at the top three layers. If the test results support the theory, go on to implementing a solution. If the test results do not support the theory, go back to test a new theory.
86
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
87
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Implementing a Solution
After you implement a solution, recheck the status of the problem. If there is still a problem, go back to either gather more information or test different theories. The symptoms of the problem might change as a result of the troubleshooting actions. If you did not improve the situation by your action, consider undoing the actions.
88
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
89
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
The symptoms The time and frequency of the symptom The theories tested The theory that proved to be correct The solution The root cause of the problem Any ideas of how to recognize this same problem in the future How long the network operation was affected
90
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
91
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
Summary
The Network Troubleshooting module presented basic concepts of network troubleshooting and the show commands to use. You should now be able to:
Identify the network management and maintenance systems that prepare you to troubleshoot. Define the steps to perform systematic troubleshooting. Describe the layered approach for troubleshooting. Identify useful commands for finding errors at each layer. Interpret the output of show commands.
92
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
93
Copyright 2008 Extreme Networks, Inc. All rights reserved. Extreme Networks, Inc. confidential and proprietary information -BETA ~FOR REVIEW ONLY.
Network Troubleshooting
94