Beruflich Dokumente
Kultur Dokumente
OAM Best Practices in Mission-Critical MPLS, IP, and Carrier Ethernet Networks
A variety of Operations, Administration, and Management (OAM) protocols and tools have been developed recently for MPLS, IP, and Ethernet networks, which provide the unparalleled power to proactively manage networks and customer Service-Level Agreements (SLAs). This paper reviews the OAM tools available in MPLS, IP, and Ethernet networks at various layers and describes best practices for choosing the right OAM tool to use for particular network deployments.
SERVICE PROVIDER
CONTENTS
Overview ............................................................................................................................................................................................................................................. 3 OAM Layering ................................................................................................................................................................. 3 OAM Tools and Network Layers .................................................................................................................................... 4 Layer 2 OAM Tools .......................................................................................................................................................................................................................... 5 Layer 2 Trace ................................................................................................................................................................. 5 Port Loop Detection ....................................................................................................................................................... 6 Unidirectional Link Detection ........................................................................................................................................ 7 Single-Link LACP Keep-Alive .......................................................................................................................................... 8 IEEE 802.1ag CFM ......................................................................................................................................................... 9 Continuity Check Messages (CCM)...................................................................................................................... 11 Loopback Messages (LBM).................................................................................................................................. 11 Linktrace Messages (LTM) ................................................................................................................................... 11 Brocade Implementation of 802.1ag: ................................................................................................................. 12 Hierarchical Fault Detection using 802.1ag ....................................................................................................... 12 IEEE 802.1ag Configuration Example ................................................................................................................. 13 IEEE 802.1ag CFM versus ITU-T Y.1731 OAM .................................................................................................... 15 ITU-T Y.1731 Performance Management ................................................................................................................... 15 IEEE 802.3ah Ethernet First Mile (EFM) Link OAM .................................................................................................... 16 Layer 2 OAM Summary ................................................................................................................................................ 17 MPLS OAM Tools ...........................................................................................................................................................................................................................18 LSP Ping ....................................................................................................................................................................... 18 LSP Traceroute ............................................................................................................................................................. 19 LSP Ping and LSP Traceroute Considerations ............................................................................................................ 19 BFD for RSVP-TE LSPs ................................................................................................................................................. 20 MPLS OAM Summary ................................................................................................................................................... 21 IP and VRF OAM Tools .................................................................................................................................................................................................................22 IP and VRF Ping ............................................................................................................................................................ 22 IP and VRF Traceroute ................................................................................................................................................. 22 BFD for OSPFv2, OSPFv3, IS-IS, and BGP4 ................................................................................................................ 23 IP and VRF OAM Summary .......................................................................................................................................... 25 Summary .........................................................................................................................................................................................................................................26
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
2 of 26
SERVICE PROVIDER
OVERVIEW
A variety of OAM tools have been developed in recent years for MPLS, IP, and Ethernet networks. These tools provide unparalleled power for an operator to proactively manage networks and customer ServiceLevel Agreements (SLAs). These OAM tools address fault detection, fault verification, and fault isolation and provide proactive detection of service degradation, service performance monitoring, and SLA verification. In MPLS, IP, and Ethernet networks, Operations, Administration, and Management (OAM) and Provisioning (OAM&P) encompasses the Management Plane (see Figure 1), represented by Network Management Systems (NMS) and Element Management Systems (EMS), and the Network Plane, represented by Network Elements (NE) and the OAM tools that run across NEs. This white paper reviews the OAM tools available in MPLS, IP, and Ethernet networks at various layers of the networking stack and recommends and reviews best practices for choosing the right OAM tool to use for a particular network deployment.
Management Plane (NMS, EMS)
OAM&P
OAM Layering
OAM tools can be classified into three main types based on the OAM layer (Figure 2): Service Layer OAM. Tools applicable to services on an end-to-end basis Network Layer OAM. Tools applicable to services over a particular network Transport Layer OAM. Tools applicable to the transport layer of the network
Service Layer OAM Network Layer OAM Transport Layer OAM
Figure 2. OAM layers These OAM layers are hierarchical in nature. For example, in Figure 3 the Service Layer OAM for Operator A can be seen as a Transport Layer OAM for the service provider, who sees the service provided by Operator A as a transport tunnel for the customer.
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks 3 of 26
SERVICE PROVIDER
NOTE: The terms customer, service provider, and operator are commonly used to reflect the business relationships that often exist among organizations and individuals. An operator provides a single Layer 2 or Layer 3 backbone network to a service provider. An operator can be identical to, or a part of the same organization as, a service provider. The best OAM tools to use at a particular network layer depend on the type of network. For example, in Figure 3, Operator A has an MPLS network and uses MPLS OAM tools, while Operator B has an Ethernet network and uses Ethernet OAM tools.
Service Provider
Customer network Site 1 MPLS Operator A Network Ethernet Operator B Network Customer network Site 2
Service OAM
MPLS OAM (Operator A) Ethernet OAM (Operator B)
Link OAM
Link OAM
Link OAM
Figure 4. Each network layer has its own best-suited OAM tools
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
4 of 26
SERVICE PROVIDER
Layer 2 Trace
Layer 2 Trace is a Brocade proprietary OAM tool that traces the traffic path in a VLAN. Layer 2 Trace is run on demand using a CLI command. Layer 2 Trace can be used to trace a particular IP, MAC, or hostname in a given VLAN. The Layer 2 Trace command (trace-l2) probes the entire Layer 2 topology and displays the input or output ports of each hop in the path, the round trip travel time of each hop, and each hop's Layer 2 protocol (such as STP, RSTP, 802.1w, SSTP, metro ring, or route-only). Figure 5 shows an example of Layer 2 Trace command (trace-l2) executed for the given network configuration. The probed Layer 2 information is discarded after 10 minutes or when a new trace-l2 command is issued. Layer 2 Trace can also display hops that form a forwarding loop in a VLAN. Figure 6 is an example in which the active topology for VLAN 2 forms a forwarding loop. In this case, Layer 2 Trace on VLAN 2 detects the forwarding loop and issues the indicated warning message. Layer 2 Trace configuration considerations: The devices that will participate in the Layer 2 Trace protocol must be assigned to a VLAN and all devices on that VLAN must be Brocade devices that support the Layer 2 Trace protocol. Devices that do not support the Layer 2 Trace protocol simply forward Layer 2 Trace packets without a reply and are transparent to the Layer 2 Trace protocol. The destination for the packet with the trace-l2 protocol must be a device that supports the Layer 2 Trace protocol. The destination cannot be a client, such as a personal computer, or devices from other vendors.
SERVICE PROVIDER
Loose mode. Detects Layer 2 forwarding loops for a given VLAN or a VLAN group. Loose mode floods test packets to the entire VLAN or VLAN group. See Figure 7.
NetIron(config)#vlan 20 NetIron(config-vlan-20)#loop-detection NetIron(config)#vlan-group 10 NetIron(config-vlan-group-10)#add-vlan 1 to 100 NetIron(config-vlan-group-10)#loop-detection
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
6 of 26
SERVICE PROVIDER
SERVICE PROVIDER
With Single-Link LACP Keep-Alive, LACP PDUs are exchanged between the two nodes to determine if the connection between the devices is still active. If no LACP PDUs are received from the other node after 3 lacp-timeout periods, a timeout event occurs and the port is blocked. The LACP keep-alive PDUs can be sent every 1 second (lacp-timeout short) or every 30 seconds (lacptimeout long). Since a timeout is declared after missing 3 consecutive LACP keep-alive PDUs, a timeout can be declared in 3 seconds or 90 seconds, depending on the selected LACP keep-alive PDUs interval. To configure single-link LACP keep-alive timeout intervals:
NetIron(config)# lacp-timeout short | long
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
8 of 26
SERVICE PROVIDER
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
9 of 26
SERVICE PROVIDER
Service Provider
Customer network Site 1 MPLS Operator A Network Ethernet Operator B Network Customer network Site 2
ME
Customer MA
ME
Service Provider MA
Operator A MA
ME
Operator B MA
ME
Switch
Up MEP
Figure 13. Using up and down MEPs to include or exclude the path inside a switch IEEE 802.1ag CFM supports Continuity Check Messages (CCM), Linktrace, and Loopback Messages, which are described in the following sections.
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
10 of 26
SERVICE PROVIDER
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
11 of 26
SERVICE PROVIDER
Figure 17. Example of 802.1ag hierarchical fault detection (refer to the numbered items below)
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
12 of 26
SERVICE PROVIDER
To simplify this example, the service provider level is not shown. If it were, the service provider would be represented by the overall network from Operator A in Location 1 through Operator B to Operator A in Location 2. The following is an example of how this fault can be detected at the different levels of the hierarchy: 1. 2. 3. 4. 5. The customer detects a service continuity fault using CCMs. Using Linktrace, the customer finds that the fault is beyond the MIPs at the border of Operator A. Provider A detects a service continuity fault using CCMs. Using Linktrace, Provider A determines that the fault is inside Operator Bs network. Operator B detects a service continuity fault using CCMs.
Operator B uses MPLS OAM tools to determine the location of the fault in its MPLS network. See the MPLS OAM section for details on MPLS-specific OAM tools. This statement is included here to emphasize the fact that you need to use the appropriate OAM tools for the type of network being used. In this case, Operator B has an MPLS network and needs to use MPLS OAM tools. Operator A has a Layer 2 Ethernet network and can use 802.1ag CFM. Note that Operator Bs MPLS network is required to support 802.1ag CFM messages over VPLS and VLL to allow customers and Operator A to use 802.1ag end-to-end. 1 Note that the customer, Operator A, and Operator B can concurrently and independently detect the continuity fault and run Linktrace to determine the location of the fault. The steps above are numbered to allow for easy reference to the respective actions depicted in Figure 17. The numbering does not imply an ordered sequence of events. That is, Operator A does not have to wait for the customer to tell it that the service is broken before it runs its own Continuity Check. Note that the CCMs shown in Figure 17 can be set up to run continuously to detect potential continuity faults or they can be set up on demand as needed.
Brocade supports 802.1ag CFM over VPLS and VLL to allow Ethernet OAM to function end-to-end over an MPLS core network.
13 of 26
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
SERVICE PROVIDER
Customer CCM @ 10 sec Service provider CCM @ 10sec 4 7 1/1 CE1 1/1 PE1 7 7 4 7 VLL MPLS PE2 4 7 2/1 7 2/1 CE2
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
14 of 26
SERVICE PROVIDER
MEP 3
ETH-DM
MEP 2
Besides CFM and other functionality, ITU-T Y.1731 also includes Performance Management, which is addressed in this paper.
2
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
15 of 26
SERVICE PROVIDER
Figure 22 shows an example of the Y.1731 delay measurement between MEP3 and MEP2 shown in Figure 21. The command sends a selectable number (default is 10) of delay measurement PDUs (ETH-DM), which are time-stamped in hardware at the source and destination MEPs to achieve high-precision measurement independent of software delays. The command averages the individual measurements and lists the resulting minimum, average, and maximum delays.
SERVICE PROVIDER
802.3ah OAM
802.3ah OAM
Figure 23. IEEE 802.3ah EFM OAM Figure 24 shows an example of the output of an 802.3ah EFM OAM show command. Note that the show command displays not only local link OAM information, but also remote link OAM information.
Supports
Generation
Manual No
Automatic No
Automatic No
Automatic Yes
Standard
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
17 of 26
SERVICE PROVIDER
LSP Ping
LSP Ping provides OAM functionality for MPLS networks based on RFC 4379. LSP Ping is used to detect data plane failure and to check the consistency between the data plane and the control plane. LSP Ping verifies that packets that belong to a particular Forwarding Equivalence Class (FEC) actually end their MPLS path on a Label Switching Router (LSR) that is an egress for that FEC. LSP Ping sends MPLS echo requests following the same data path that normal MPLS packets would traverse (Figure 25). LDP LSP Ping and RSVP LSP Ping are supported, as shown in Figure 26 and Figure 27 respectively.
MPLS Network PE LSP LER LSR Echo Request Echo Reply LER P PE
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
18 of 26
SERVICE PROVIDER
LSP Traceroute
LSP Traceroute provides OAM functionality for MPLS networks based on RFC 4379. LSP Traceroute is used to isolate a data plane failure to a particular router and to provide LSP path tracing. With LSP Traceroute, an echo request packet is sent to each transit LSR and the LER. The echo request follows the same data path that normal MPLS packets would traverse. A transit LSR or an LER receiving the echo request checks that it is indeed a transit LSR or LER for this path and returns echo replies (Figure 28). LDP LSP Traceroute and RSVP LSP Traceroute are supported, as exemplified in Figure 29 and Figure 30, respectively.
MPLS Network PE LSP LER LSR LER P PE
Echo request
Echo replies
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
SERVICE PROVIDER
Transit-originated detour. The user can initiate a Ping or Traceroute operation on a transit-originated, detour LSP. Because the session name does not uniquely identify a session on a transit LSR, the user needs to specify the entire session ID (including the tunnel end-point, tunnel ID, and extended tunnel ID) for the detour LSP to which the LSP Ping or Traceroute command is applied. LSP re-optimization. If LSP re-optimization occurs while the Ping or Traceroute is in progress, the echo request will be sent out on the current LSP instance until the new instance is created.
BFD allows for the detection of a forwarding path failure in 300 milliseconds or less (depending on the configuration).
MPLS Network PE LSP LER LSR LER P PE
BFD
Figure 31. BFD for RSVP-TE operation BFD for RSVP-TE LSP should be used selectively to monitor unreliable paths such as those through nonMPLS devices, for example, optical switches. In Figure 32, for example, the LSP traverses optical switches. The optical switches keep the links to the MPLS routers up even in the event of a failure between the optical switches. This would prevent the MPLS routers from supporting path switchover (since, as far as the MPLS routers are concerned, the link between them is up). BFD for RSVP-TE LSP would detect the LSP path failure and would trigger a path switchover. 3 Since a link failure will trigger FRR directly, the only benefit of using BFD for RSVP-TE LSP when there are no optical switches (or other transport types that would prevent MPLS routers from detecting the physical path as down) would be to detect control plane failures.
In configurations in which there is no alternative path, the LSP is brought down and the BFD session is deleted. The LSP then follows the normal retry procedures to come back up.
3
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
20 of 26
SERVICE PROVIDER
Failure
Figure 32. BFD for RSVP-TE LSP used to monitor paths through non-MPLS devices BFD for RSVP-TE LSP can be enabled or disabled on the fly at the global MPLS level 4 (see Figure 33) or for each individual RSVP LSP (see Figure 34) without affecting the LSP operational status. In addition, BFD for RSVP-TE LSP parameters can be changed on the fly without changing the state of the BFD session.
Supports
Generation Standard
Manual Yes
The number of BFD sessions supported by the system must be taken into account when enabling BFD for RSVPTE globally.
4
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
21 of 26
SERVICE PROVIDER
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
22 of 26
SERVICE PROVIDER
Source router
Destination router
BFD
BFD
BFD
Failure
Link is up
SERVICE PROVIDER
Figure 42 shows an example of BFD configuration. BFD can be enabled or disabled for all interfaces or per interface for use with OSPFv2 (that is, IPv4), OSPFv3 (that is, IPv6), and IS-IS, as shown in Figure 43, Figure 44, and Figure 45, respectively.
Figure 43. Enabling/disabling BFD for OSPFv2 for all interfaces (top) or per interface (bottom)
Figure 44. Enabling/disabling BFD for OSPFv3 for all interfaces (top) and per interface (bottom)
Figure 45. Enabling/disabling BFD for IS-IS for all interfaces (top) and per interface (bottom)
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
24 of 26
SERVICE PROVIDER
BFD for BGP4 supports single-hop and multi-hop BFD on Ethernet, POS, and Virtual Interfaces. BFD for BGP4 can be enabled or disabled at the global BGP router level, for each individual peer, or for a peer group, as shown in Figure 46, Figure 47, and Figure 48, respectively.
Supports
Generation Standard
OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks
25 of 26
SERVICE PROVIDER
SUMMARY
This paper reviewed OAM tools available for MPLS, IP, and Ethernet networks at various layers of the stack and reviewed best practices for choosing the right OAM tool to use in a particular network deployment. These tools provide unparalleled power for an operator to proactively manage networks and customer Service Level Agreements (SLAs). These OAM tools address fault detection, fault verification, and fault isolation; enable proactive detection of service degradation; and provide service performance monitoring and SLA verification.
2010 Brocade Communications Systems, Inc. All Rights Reserved. 11/10 GS-BP-356-00 Brocade, the B-wing symbol, BigIron, DCFM, DCX, Fabric OS, FastIron, IronView, NetIron, SAN Health, ServerIron, TurboIron, and Wingspan are registered trademarks, and Brocade Assurance, Brocade NET Health, Brocade One, Extraordinary Networks, MyBrocade, VCS, and VDX are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names mentioned are or may be trademarks or service marks of their respective owners. Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the United States government. OAM Best Practices in Mission Critical MPLS, IP, and Carrier Ethernet Networks 26 of 26