Beruflich Dokumente
Kultur Dokumente
Ethernet has become the most successful and widely deployed LAN transport technology
in the world. While other technologies such as Token Ring and Fiber Distributed Data
Interface (FDDI) have become obsolete, Ethernet has more than 100 million clients deployed
today, making it the standard interface for most network-capable devices.
The last 20 years have seen significant development of Ethernet technology from 10-Mbps
shared operation over thick yellow cable, to switched operation over unshielded twisted
pair (UTP) at 10, 100, and 1000 Mbps. However, the most significant development from a
wide-area networking perspective has been fiber optic transmission at 100 Mbps, 1 Gbps,
and 10 Gbps at transmission distances from 2 kilometers (km) up to 2000 km using
long-haul dense wavelength-division multiplexing (DWDM) systems.
In parallel to these developments, Ethernet switching technologies have evolved from simple
2-port devices with forwarding rates typically in the thousands of packets per second to
today’s modern switching systems that support high port densities, forwarding rates in the
millions of packets per second, and sophisticated wire-rate services. Additionally, the
economics of Ethernet make it an attractive proposition as a WAN transport for service
providers.
Another advantage of Ethernet as a transport is its support of class of service (CoS) that
allows up to eight classes of service to be defined. From an enterprise perspective, the
ubiquitous nature and sympathetic CoS characteristics of Ethernet make Ethernet as a WAN
technology very attractive because the Ethernet WAN can be seen as an extension of the
campus LAN.
Ethernet uniquely supports true multipoint communications. Most WAN technologies such
as Frame Relay or ATM offer point-to-point connectivity only, which are complex to
configure as more connections are added. If a Virtual Private LAN Service (VPLS) is used, its
inherent broadcast nature makes newly added locations ready to use because the routing
protocol will automatically detect neighboring devices and form routing adjacencies.
Although Metro Ethernet is often thought to be analogous to Transparent LAN Services,
Ethernet can be used as a broadband access technology to point-to-point Layer 2 VPN,
Layer 3 VPN, and Internet services.
For many of these same reasons, service providers are beginning to consider Ethernet a core
technology, if not a core service, for their WAN and metropolitan-area network (metro)
service offerings as well as a mechanism for offering numerous other services. When coupled
This paper will introduce Ethernet as a metro networking technology and describe the various technologies and
services that can be enabled over an Ethernet infrastructure, including integration with established or emerging
technologies such as Resilient Packet Ring (RPR) and SONET/SDH.
Most, if not all, service providers are looking closely at Ethernet as a technology within their service networks.
For example, incumbent local exchange carriers (ILECs); post, telephone, and telegraphs (PTTs); and interexchange
carriers (IXCs) are considering Ethernet, either as a pure Layer 2 transport mechanism, or for offering IP VPN
services to complement their existing Frame Relay or ATMs services. A number of independent operating companies
(IOCs) are using Ethernet as a broadband technology for delivery of multiple services to the residential customer.
Additionally, there is an increasing trend involving the use of Ethernet as a backhaul instead of ATM for xDSL
applications. Cable companies themselves are using Ethernet as a means of aggregating cable headend systems. These
applications will be further discussed later in this paper.
But it is first important to understand what is encouraging the interest and demand for Ethernet in the service
provider space. These factors can be summarized as:
• Incremental services
• Bandwidth
• Availability of fiber
• The enabling applications
Incremental Services
Ethernet is of interest to the service provider only if it provides a means for incremental revenue. In the past, Ethernet
was offered only as a custom service to large customers who had the requirement and were willing to pay for it. Now,
service providers are looking to Ethernet as a core transition medium for three main reasons: to enable new services,
to align business objectives and network infrastructure, and to customize the network for the end customer.
Ethernet first gained traction in the service provider space in 2001 due to the increased demand for transparent LAN
service. This service, while originally viewed as a niche service, has recently seen an upswing. Ethernet, by its nature,
allows a provider to offer this service, incremental to existing leased line or Frame Relay service, for additional
revenue. Another benefit of Ethernet, particularly in tandem with MPLS, is the ability to converge multiple services
onto a common transport medium. This is one of the primary benefits of a connectionless service interface as opposed
to the current connection-based, point-to-point services offered today.
The speed and bandwidth characteristics of Ethernet (discussed in the next section) allow the service provider to offer
incremental and customized services more easily than previously possible. For instance, a particular end customer
might be looking for a voice service as well as a point-to-multipoint capability for video. The provider could offer a
voice service, while at the same time taking advantage of the broadcast nature of video and the intelligence of IP to
offer multicast video. All of this is run over a high-bandwidth infrastructure, helping mitigate the need for very
granular quality of service (QoS).
One of the clear advantages of Ethernet is its ability to scale in bandwidth and speed. Whereas Ethernet began its life
as a shared 10-Mbps medium over thick coaxial cable, it can now scale to switched 10-Gbps per link over fiber, with
100 Mbps and 1 Gbps as interim steps. As Internet and Web-hosting traffic continue to grow, the ability to scale
bandwidth on a flexible platform, such as the Cisco® Catalyst® 6500 Series switches or Cisco 12000 Series routers,
becomes extremely attractive. For example, a 100-Mbps connection to a data center server can be easily upgraded
to a 1-Gbps connection, should that bandwidth be required. This is important because large networking servers can
now manage 100 Mbps to 1 Gbps of sustained throughput. Similarly, a network architect might also consider
upgrading the backbone links to 10 Gigabits Ethernet, something that can be done easily without costly equipment
upgrades or SONET ring resizing.
In addition, the costs for Ethernet interfaces are often considerably less than those for more traditional technologies
such as packet over SONET (POS). Ethernet lacks some POS characteristics, such as 50-milliseconds (ms) link
restoration, that make POS so attractive to a service provider. However, many providers are willing to consider
Ethernet as a lower-cost means of connectivity, particularly if a SONET/SDH infrastructure is not already in place.
Product Availability
Another trend that is clearly helping both Ethernet and IP is the availability of Ethernet interfaces on a wider variety
of equipment. Whereas five years ago, Ethernet could only be found on network interface cards (NICs), hubs, and
switches, it is now found, in relatively high densities, across most switching and routing platforms. What is even more
interesting are the other devices that can now connect to an Ethernet and/or IP network. These include storage arrays,
WiFi access points, cable headend equipment, gaming equipment, and cameras.
The cost of Ethernet interfaces is dropping dramatically as they become more integrated into the end-user devices.
Most PCs now come with an onboard 100-Mbps or 10/100/1000 interface. NICs can be purchased inexpensively.
An Ethernet interface on anything from a Sony Playstation to a video-on-demand quadrature amplitude
modulation (QAM) modulator can be included easily and at little cost to the equipment manufacturer. The sheer
availability of Ethernet is increasing bandwidth for the enterprise and home user, thereby requiring the service
provider to increase its own bandwidth.
Availability of Fiber
Ethernet is designed to run over two media types: copper and fiber. Ethernet is usually sent over copper cable because
the vast majority of Ethernet connections are from a switch to an end station. Longer-distance Ethernet transmissions
at gigabit or tens-of-gigabits rates require fiber-optic cable. Fortunately for the service provider, much of that fiber
already exists. In the 1990s many service providers, utilities, and government entities began trenching and laying fiber
into the ground. While the expected demand for that fiber has not grown as quickly as expected, the infrastructure,
to a large extent, exists and is waiting to be used.
While much of this fiber has been “lit up” with time-division multiplexing (TDM) or private-line access to their
customer base, many service providers have a large portion of dark fiber waiting to be used. Many providers are
looking to this fiber to carry incremental service—this is where Ethernet broadband, often referred to as Ethernet to
the X (where “X” is “business” or “residential”), and Metro Ethernet can be deployed to offer new and innovative
high-bandwidth services. Many providers have fiber access to many large business parks and some residential areas
(though a residential offering is more common in high-density residential areas featuring many apartments). There
The use of copper for Ethernet service delivery is becoming increasingly attractive, particularly considering the sheer
volume of copper already in the ground. Copper today provides the majority of last-mile technology for telephone
and serial WAN connections, both for residential and business services. Fortunately, innovations in Ethernet, as well
its usage in retrofitting existing technologies, have helped enable copper access to Ethernet networks. Technology
such as Ethernet over very-high-bit-rate DSL (VDSL) is one mechanism that has been deployed in Europe and in
parts of the United States today. In addition, other technologies such as Digital Subscriber Line Access
Multiplexers (DSLAMs) with Ethernet uplinks and asymmetric DSL2 (ADSL2) are also aiding in the adoption of
Ethernet in the last mile.
Many enterprise network managers are seeing their data volume dramatically increase. This is due to a number of
reasons, such as more peer-to-peer traffic, larger file transfers (either FTP or e-mail), and simply more use of the
Internet as a business tool. Another, arguably more relevant factor that contributes to higher bandwidth, even though
the data volumes may be low, is application transaction delay. Some modern applications actually require transaction
delays in the 50-ms range that can only be delivered using high-speed transport. However, these applications will
typically not compel a provider to dramatically rearchitect its network; instead, they will change the way in which
services including bandwidth are delivered. The providers, understandably, are looking for the “next big thing.” The
emerging applications may tip the balance in the service provider networks toward higher bandwidth and Ethernet.
Storage
Enterprises are now more dependant than ever on their data, both transactional and backup. This has made storage
networking a critical business requirement. While many transactional storage applications require synchronous
storage using dark fiber or DWDM, many backup, network-attached storage (NAS) and disaster-recovery solutions
can make excellent use of Ethernet as a transmission medium in the metro between data centers. Service providers
are beginning to notice this opportunity because it not only requires bandwidth, but a guaranteed service-level
agreement (SLA), something that they can charge back to their customers.
Video
Video providers, such as cable companies, are looking for less expensive ways of delivering data to their end users.
As many of these cable headend devices become Ethernet-attached to the aggregation layer of the network, more
bandwidth is required to transfer these now-digital files. In addition, multiple video streams per link can dramatically
increase bandwidth requirements, thereby generating more interest in cost-effective transmission. Ethernet’s
simplicity and affordability is appealing to video and cable providers.
The Cisco metro solution delivers the most comprehensive multilayer service portfolio for providers to quickly scale
customers and revenue. This portfolio can enable high-value services at any layer while providing a growth path to
a broader and more profitable service offering. Cisco Systems® has developed its metro solution to tightly link into
enterprise environments that require integrated voice, video, and content applications. This integration allows service
providers to link to enterprise applications based on Cisco AVVID (Architecture for Voice, Video and Integrated
Data). The perspectives provided in this paper come from the considerable experience and success Cisco has achieved
in helping enterprise companies design and deploy their converged data networks.
Standards Development
Another important reason for Ethernet in the service provider space has been the activity within the standards bodies.
In 2001, the Metro Ethernet Forum (MEF) group was formed with the express mission of furthering the adoption
of service provider Ethernet, and Ethernet technologies in general, in the WAN space. Specifically, MEF has been
looking at the Ethernet services and service definitions, gathering agreement between Ethernet vendors and service
providers about how a particular service should behave. This allows Ethernet vendors and service providers to
best understand what the service will look like to the end customer. Finally, MEF has been evaluating carrier-class
Ethernet. This initiative seeks to specify the architecture, protocols, and management for Ethernet-based
transport networks.
MEF has been an excellent forum for discussion of these services and capabilities. It is the standards bodies, however,
such as the IETF, the International Telecommunication Union (ITU), and the IEEE, that have begun work on the
standardization of these particular protocols. The IETF has been specifically involved in the development of a
multipoint Ethernet service architecture called Virtual Private LAN Service (VPLS) and Virtual Private Wire
Service (VPWS). The IEEE, which governs Ethernet and bridging standards with the 802.3 and 802.1 committees,
respectively, has been working on standards such as Resilient Packet Ring (RPR) (with the 802.17 committee) and
Provider Bridges (802.1ad).
Cisco has also been heavily involved in the “Ethernet over Transport Architecture” initiative. The original initiative
created the “G.etna” draft, which is now further segmented into G.etoa, G.ethna, G.esm, and G.ethsrv. The
following work items are under study group:
• G.eota (Ethernet over transport architecture)
• G.ethna (Ethernet-layer network architecture)
• G.esm (Ethernet over transport Ethernet service multiplexing)
• G.ethsrv (Ethernet over transport service characteristics)
• G.smc (service management channel private line)
• G.enni (Ethernet over transport network interface)
• G.euni (Ethernet over transport user interface)
• G.eequ (Ethernet equipment)
The service provider environment is very diverse; there are numerous technologies and protocols available for use in
service creation and service delivery. The first step to understanding how Ethernet as a technology plays in the metro
is to understand how it fits into the general technology scheme. Figure 1 shows the hierarchy and interrelation
between the metro technologies.
Figure 1
Ethernet Related to Service Provider Technologies
IP / MPLS Control
Resilient
Packet SONET / SDH Switched
Ring Ethernet
CWDM / DWDM
Fiber
As stated earlier, the critical enabler for Ethernet as a transmission media is the availability of fiber. For that reason,
fiber is shown as the foundation for all the technologies in Figure 1. How that fiber is used, meaning what
transmission technology is implemented, depends on a number of factors, such as installed base, how the fiber is laid
out, density of customers in a particular area, and the service being offered.
At this juncture, it is very important to point out that “metro Ethernet” or “service provider Ethernet” does not
necessarily imply that Ethernet is used end-to-end. As Figure 1 shows, Ethernet is one possible option as a transport
medium; however, Ethernet can also be run over SONET or RPR. This is a critical distinction in the discussion of
service provider Ethernet: Ethernet as the customer User-Network Interface (UNI) versus Ethernet as the transport.
As the Figure 1 demonstrates, both are possible.
Access into the service provider network can also be provided via Ethernet, either from a business or a residence.
This technology is generally referred to as Ethernet in the First Mile (EFM). The media choices are straightforward:
There is either copper access, such as existing telephone wire or Category 5 unshielded twisted pair, or fiber.
Many multi-dwelling units have copper wiring that has been in existence for quite some time. The ability to make
use of this for Ethernet is one of the main attractions of Long-Reach Ethernet (LRE). LRE, which is offered on the
Cisco Catalyst 2950 LRE Series switches, supports between 5 and 15 Mbps of Ethernet transmission over existing
voice-grade cabling with a drive distance of up to 5000 feet, making it an ideal solution for in-building connectivity
Fiber cabling to the residence or multi-dwelling unit simplifies the distance and bandwidth limitations that are found
in LRE and xDSL. As stated earlier, the amount of fiber deployed every year by service providers, utilities, and local
governments continues to increase year after year. High-density fiber aggregation devices are required to support such
technologies as 1000Base-LX and 100Base-LX10. Cisco offers solutions for both “ends” of the fiber. The Cisco
ONT 1031 Media Converter provides customer-premises fiber termination. This serves as a demarcation between
the service provider network and customer network with environmental hardening for outside placement, if required.
It provides one 1000BASE-LX Gigabit Ethernet port for the service provider and one 10/100/1000BASE-T copper
port for connectivity to the end-customer network.
Cisco Catalyst switches serve as the “end point” for customer aggregation, either residential or business. For
example, the Cisco Catalyst 2950 and Catalyst 3550 series switches provide a low-cost customer premises
equipment (CPE) device to aggregate lower-density (less than 48 customers) 100Base-TX or LRE-connected users
within a building. The Cisco Catalyst 4500 Series switches are optimized for residential aggregation, featuring
high-density 1000Base-LX and 100Base-LX10 line cards that provide aggregation for many customers onto a single
platform, where intelligent services can then be applied. Many providers offering business services use Cisco Catalyst
6500 Series switches and Cisco 7600 Series routers as the aggregation and/or POP device to provide VPN service,
QoS, and advanced services.
The existing fiber plant a provider has access to will determine almost everything about the physical transmission
medium used. Unlike enterprise fiber plants, a service provider fiber plant cannot be installed, removed, or altered
without a major effort, one that usually involves pulling fiber and the associated trenching that may also incur
substantial costs. For that reason, a fiber plant is installed with the intention to make no alterations other than to
provide access to the customer site itself. It is important to understand that the logical topology and the physical
topology will differ significantly because traditional transmission technologies such as SONET/SDH rings rely on the
physical layout of the fiber plant.
Main HQ
Branch
Office
Figure 2 shows the typical physical topology of a service provider’s network. The topology is laid out in a ring
configuration with three distinct tiers. The access ring provides access out to the customer premises. In many cases,
an access ring will support only a single customer, although there are cases where many customers could be supported
per ring, depending on the bandwidth allocated per access ring. The access rings are terminated at ring collector
nodes, which are interconnected via the access-collector ring. The access-collector ring aggregates multiple access
rings into a single fiber distribution frame. Finally, these access-collector rings are aggregated at a CO point of
presence (POP). Each CO POP is typically interconnected over a high-speed (OC-48 or 192 ring) inter-office and/or
regional ring.
The physical topology determines a number of important factors, each of which affects the design of the Ethernet
service network. These factors include:
• Amount of fiber available—In some cases, a provider has installed plenty of fiber cores. This means that some of
the cores can be used for traditional TDM traffic while others can be “sold” to the data portion of the provider
as dark fiber.
• Section of the business—The transmission group in a service provider may be the group offering the Ethernet
service. In that case, they almost always want to use the infrastructure already there, namely SONET/SDH. The
service offered will most likely be an Ethernet private line with an end-to-end guaranteed SLA.
• Customer density—If there are few customers, it might make more sense to multiplex Ethernet over an OC-3
circuit than to run Gigabit Ethernet to each customer. Conversely, if there are numerous customers, using Ethernet
switching over dark fiber might make the most sense.
The question now arises as to what technologies are available to offer Ethernet over the transmission medium. Once
the provider has decided on the service offering and understands the implications on the transmission network, the
options can be considered.
Wavelength-division multiplexing (WDM) technology uses the physical characteristics of light to multiplex
wavelengths (lambdas), or streams of light, at different frequencies on the same fiber. There are two main types
of WDM technology: coarse and dense. Coarse WDM, or CWDM, uses wider spacing between frequency bands,
allowing for fewer wavelengths, but at a lower cost because the granularity of the lasers doesn’t have to be tightly
controlled. Dense WDM, or DWDM, is more sophisticated and, with a smaller spacing between wavelengths, allows
for more lambdas on the same fiber.
The second component of the CWDM solution is the optical add/drop multiplexer (OADM). These OADMs are
designed to take in the “colored” light from the GBIC and multiplex it with the other wavelengths over the trunk.
Each site would have an OADM, either single lambda or multilambda, depending on the number of wavelengths
added or dropped. Figure 3 shows a typical CWDM Ethernet ring configuration. For the DWDM GBICs, the
Cisco ONS 15216 Metropolitan/Regional DWDM System can be used for the add/drop component. Either way, the
capability and architecture of the network is the same.
Figure 3
CWDM/DWDM Ethernet Ring
Multiple l
Passive Single l
CWDM Multiplexer Passive
(Headend) CWDM Multiplexer
0/1
2/1
0/2
2/2
East-Facing GBIC
West-Facing GBIC
Because each wavelength is essentially a point-to-point link, the logical topology is a point-to-point network across
a physical ring. This is much like an Ethernet over SONET deployment in that the physical and logical topologies
vary. Redundancy within CWDM is handled by the edge devices because CWDM simply provides virtual
point-to-point Ethernet connections across the physical-ring topology. The edge devices may use 802.1 Spanning Tree
DWDM
DWDM is a far more robust technology, providing more wavelengths than a CWDM solution. This enables a service
provider to better use the fiber plant by increasing the bandwidth potential of the installed fiber. DWDM is also
amplifiable to provide greater drive distances between nodes. Most DWDM solutions, including the Cisco ONS
15530 DWDM Multiservice Aggregation Platform and Cisco ONS 15540 ESP and ESPx extended services
platforms, provide support for up to 32 wavelengths, with each wavelength capable of supporting a single 10-Gbps
stream, multiplexed 1-Gbps stream, or Fibre Channel traffic. Figure 4 shows how DWDM can be used in a service
provider network to support Ethernet.
Figure 4
DWDM in a Service Provider Environment
GRID Computing
Facility
Service
Provider
POP
xDSL
ATM
Frame
Relay
Si
Super Computing
Metro
Facility
DWDM
32 x 10 GE
Wavelengths
Systems that support DWDM are typically more complex. To design a scalable DWDM network, the network
architect must consider the dispersion and amplification characteristics of the network due to fiber as well as distance.
While CWDM has similar issues, only a few frequencies in the C-band can be amplified. In DWDM, amplification
and dispersion compensation must be strategically placed throughout the network to ensure signal integrity.
Due to the large amount of fiber in the ground today, most service providers are not using WDM strictly for fiber
savings. However, they are using WDM technology to efficiently scale bandwidth and provide rapid service
deployment. This is seen with the Cisco ONS 15540, which is often deployed in storage applications for data center
disaster recovery (both in enterprises and service providers). The Cisco ONS 15540 uses 10-Gigabit Ethernet
technology that, coupled with DWDM technology, delivers up to 320 Gbps of bandwidth over a single pair of fiber.
This capacity is then available for the service provider to offer LAN and storage area network (SAN) extension
between sites while making efficient use of the available fiber.
Another emerging application for DWDM and Ethernet is video on demand (VoD), which not only has large
bandwidth requirements, but also needs to be distributed to multiple distributed hubs located around the ring. In this
application, video traffic is sent from an Ethernet-connected VoD server over WDM and then distributed to the
headends serving customers. This gives the VoD provider the price advantage of Ethernet within their infrastructure
and the high bandwidth over single fiber gained from WDM.
Data-Optimized SONET/SDH
In many cases, particularly with an incumbent provider, the installed fiber is carrying TDM traffic. This traffic is
essential a service provider’s business. These service providers already have SONET/SDH networks deployed and
have been offering services off them for a number of years. As providers look to offer incremental data services with
Ethernet over their existing infrastructure, they require the flexible technology that is found with a multiservice
provisioning platform (MSPP).
An MSPP is a platform that can transport traditional circuit-based TDM traffic such as voice, but also supports
native data services (such as switched Ethernet) that offer efficient and cost-effective services over SONET/SDH
infrastructure. The ability to integrate both capabilities allows the service provider to deploy a flexible, cost-effective
architecture that makes the best use of its existing investment while providing incremental services.
There are two ways of implementing data over this type of an infrastructure. First, the Ethernet-connected customer
could be mapped into a STS/VC circuit in a point-to-point fashion. This is often called “book-ending” because the
circuit originates on an Ethernet card in one location, such as a corporate headquarters, and terminates on another
Ethernet card at a satellite office. This one-to-one mapping does not, however, make the best use of bandwidth and
can be very inefficient. Consequently, Ethernet deployments over SONET/SDH infrastructure have traditionally been
expensive.
Technology supported on the Cisco ONS 15454 MSPP provides data optimization via two main mechanisms. First,
the Cisco ONS 15454 ML Series line card provides integrated Ethernet switching in the SONET/SDH platform.
Ethernet switching can then be used to multiplex customers into the SONET infrastructure. For example, say a
provider wants to offer service to 15 customers in a multitenant unit. Using the Cisco ONS 15454, the provider can
connect each customer via a 100-Mbps or 1-Gbps interface on the Cisco ONS 15454 ML Series line card. By using
the rate limiting and policing capabilities of the ML Ethernet switch, the provider can give a committed access rate
to each customer. That traffic is then mapped on the back end to STS-1, STS-3, or STS-12 circuits on the SONET/
SDH ring. This traffic can then be terminated at a service platform, which will be discussed later.
Figure 5
Resilient Packet Ring Operation
RPR Ring
In a multiservice provisioning platform-enabled SONET/SDH ring, RPR is an overlay and is processed by the
multiservice component of the platform on a common circuit. Therefore, a single 1-Gbps circuit can support many
customers connected on ML Series cards on numerous Cisco ONS 15454 platforms around the ring. This provides
excellent bandwidth utilization while still providing fair access to all nodes on the ring. To gain further bandwidth
and ring efficiencies, an oversubscription factor can be applied to the ring, which is determined by the service defined
by the provider as well as the SLA offered to the end customer.
There are a number of other advances in data-optimized SONET/SDH. One technology, Virtual Concatenation,
provides the ability to group several noncontiguous STS or virtual circuit fragments into a single larger virtual STS.
This virtual STS is referred to as a Virtual Concatenated Group (VCG) because it is made up of a group of smaller
STS, or VT, levels. Virtual Concatenation is used in conjunction with another scheme, Link Capacity Adjustment
Scheme (LCAS), which allow members of a VCG to be dynamically added or subtracted to provide additional
bandwidth as required. These schemes have now been finalized by the ITU (G.707 defines Virtual Concatenation and
G.7042 defines LCAS).
It is important at this juncture to point out that, in most cases, the Ethernet topology (Figure 6), or the view from
the spanning tree or IP and MPLS perspective, may look considerably different than the physical SONET/SDH
network. Review Figure 2 and then evaluate Figure 6 for comparison.
One of the most obvious differences between the physical infrastructure and logical one is the fact that, although the
physical network is laid out in a ring, the Ethernet network is a full mesh of point-to-point circuits. To access a
customer, for example, a single Ethernet connection will be mapped to a circuit, such as a 10-Mbps port to a DS-3
circuit. Using the SONET/SDH ring protection mechanism means that there does not necessarily need to be dual
physical connections on the Ethernet side, because redundancy can be handled by SONET/SDH. Another important
consideration is the routing protocol. Its topology depends on the routers in the network and what adjacencies and
access exists. Therefore, a network administrator has to be familiar with the underlying transport as well as the
Ethernet and routed network.
Although Ethernet can be overlaid on top of an existing transport infrastructure, such as data-optimized SONET/
SDH or DWDM, Ethernet switching can also be run over dark fiber end to end. There are some clear advantages to
this, namely the simplicity of Ethernet and the cost associated with Ethernet switching equipment. It is, however,
important to consider the unique characteristics of Ethernet when using it as a transmission medium. By using
SONET/SDH, for example, resiliency is provided by the transport mechanisms, such as bidirectional line switched
ring (BLSR) or unidirectional-path switched ring (UPSR) in SONET. Ethernet, over dark fiber, must use the schemes
inherent in Ethernet.
It is important to understand at this juncture that Ethernet at Layer 2 is radically different than traditional Layer 2
WAN and MAN protocols such as Frame Relay and ATM. One of the main differences is that Frame Relay and ATM
both offer an intelligent forwarding mechanism (essentially a routing protocol) at Layer 2. For ATM, this protocol
is Private Network Node Interface (PNNI). Frame Relay uses a proprietary routing protocol that uses PNNI,
although most Frame Relay cores are ATM-based. Switched Ethernet at Layer 2 has no such intelligence. It follows
Native Ethernet uses two main topologies: point-to-point and ring (Figure 7). The choice between the two is often
predetermined by what exists in the ground (SONET/SDH or fiber layout) and cost. If a SONET/SDH network exists
already, then the Ethernet overlay network could be either hub-and-spoke over a physical SONET/SDH ring or could
use a shared ring mechanism like RPR. The hub would reside at the POP and provide aggregation for the spokes,
which are customer premises equipment (CPE).
Figure 7
Ethernet Topologies
Metro
POP
If fiber rings exist, there are two methods of connecting Ethernet devices together: either “daisy chaining” devices
together in a ring, as shown in Figure 7, or using CWDM to provide virtual point-to-point Gigabit Ethernet circuits
over the physical ring as described earlier. The first method, used in many networks today, deploys native Ethernet
rings using the IEEE 802.1 Spanning Tree protocols. Ethernet rings are usually considered for deployment if the
following conditions exist: there is no SONET or SDH already installed, and multiple customer premises are
physically connected over the same fiber ring. Ethernet rings are particularly attractive to startup Ethernet providers
who have installed their own fiber and are cost-conscious, and may consider Ethernet a low-cost alternative to a
transport infrastructure (SONET, SDH, or DPT). However, there are a number of issues to consider when dealing
with Ethernet ring scenarios.
Figure 7 shows a sample network architecture using Ethernet rings. From a technology perspective, Ethernet
switching at Layer 2 is not optimized in a ring configuration. This is in part due to the operation of the Spanning
Tree Protocol that is required by Ethernet to prevent loops in the network. Convergence times may be as short as 2
seconds with Rapid Spanning Tree Protocol [802.1w] and as long as 50 seconds with Spanning Tree Protocol
[802.1d].
So far, this paper has focused on how Ethernet can be used as a transmission medium from a customer site into the
service provider’s network. However, intelligence is a critical component of a carrier Ethernet network, bringing
availability, QoS for SLA enforcement, and network security to the service provider network.
The intelligence provided, typically in the POP or data center, allows the service provider to offer not just connectivity
with an Ethernet User-Network Interface (UNI), but ISP peering, data center connectivity and aggregation, and
intelligent inter-POP connectivity.
Carrier Ethernet must be built into a robust infrastructure capable of providing either the services a provider can
offer to customers or a backbone infrastructure to support those services. Although the applications may vary, the
services themselves are the same. The primary services are:
• Availability and redundancy
• Quality of service
• Security
Many service providers today use IP or MPLS to scale their networks, either to offer Internet access or to provide an
MPLS VPN service to their customers. Those providers not using IP or MPLS today almost always have a roadmap
for how they will get there. The use of IP or MPLS (which requires an IP foundation) brings another set of resiliency
mechanisms that may be used to augment the availability of the network. Cisco IOS® Software, which has been
developed over the past decade and deployed in most every service provider network in the world, brings a robust
Cisco pioneered Tag Switching in 1997, which became the foundation for today’s MPLS. Cisco also innovated
numerous MPLS features, including Fast Reroute and Traffic Engineering, to better manage bandwidth and
availability, and MPLS VPN to provide highly available and efficient use of network resources as well as sophisticated
services. Cisco supports IP routing on all of its platforms based on Cisco IOS Software, ranging from the Cisco 800
and 1700 series routers to the Cisco 7600 and 12000 series routers , and also including switching platforms such as
the Cisco Catalyst 3550, Catalyst 4500, and Catalyst 6500 series and the Cisco ONS 15454 ML Series. MPLS is
supported across most Cisco IOS routers, from the Cisco 3600 Series to the Cisco 12000 Series routers, and is also
available on the Cisco Catalyst 6500 Series switches.
Quality of Service
QoS provides a means of guaranteeing bandwidth and service to one type of traffic over another. It allows service
providers to effectively manage their bandwidth usage and offer tiered traffic service levels. As service providers look
to differentiate themselves and offer SLAs to their customers, QoS becomes an important component in the network.
Many providers may choose to use a packet-based network as the transport medium. Cisco switches and routers
provide a number of mechanisms to manage how QoS is applied to a packet as it traverses a network. First, the
service provider may provide a certain data rate out to the customer. The provider can use the policing capability of
Cisco routers and Cisco Catalyst switches to limit the customer’s traffic and provide burst capabilities. These
functions are analogous to Committed Information Rate (CIR) and Peak Information Rate (PIR) capabilities within
Frame Relay and ATM networks. Cisco provides several mechanisms that support granular policing, from kilobits
per second to megabits per second. Once the traffic is in the provider’s network, a number of mechanisms can be
employed. First, some traffic types, such as voice, have specific latency and jitter requirements. Strict priority queuing,
when used in conjunction with Weighted Round Robin (in the switching platforms) and Class-Based Weighted Fair
Queuing (in the routers), provides for bandwidth guarantees while also providing traffic differentiation based on IP
precedence or differentiated services code point (DSCP). QoS marking, reclassification, and queuing is handled in the
hardware of Cisco Catalyst switches and Cisco 12000 and 7600 series routers without incurring performance
degradation.
Perhaps the easiest way to guarantee a customer’s traffic is to provide an end-to-end physical connection, such as an
Ethernet-based private line. The Cisco ONS 15454 allows mapping of an Ethernet UNI directly to an STS on a
SONET network that guarantees end-to-end bandwidth within the network. However, many providers may be
interested in multiplexing multiple customers’ traffic streams onto a single STS to realize greater bandwidth
efficiencies within the network. By using the Cisco ONS 15454 ML Series line card to an STS circuit, several customer
traffic streams can be aggregated and QoS policies applied such that if congestion is experienced, QoS can manage
the congestion and help ensure that SLAs are met. By using Resilient Packet Ring (RPR) on the Cisco ONS 15454
ML Series line card, fairness around the ring can be provided via RPR ring access fairness algorithms.
It is important to remember that Ethernet in itself, due to its simple forward and filter mechanisms, is not inherently
secure. A number of mechanisms in Cisco switches and routers have been put in place to ensure that the network is
as secure as possible. One simple mechanism, port security, can be enabled to limit the number of MAC addresses
that are learned per port. A static entry of a MAC address per port is also possible. This prevents unauthorized end
stations from “appearing” on the provider’s network. Cisco has also been helping lead the adoption of IEEE 802.1x
port-based user authentication. Once the MAC address has been learned by the switch, 802.1x will authenticate the
user attempting network access. This provides an additional layer of network security by ensuring that only allowed
and authenticated devices are learned on the service provider network.
At Layer 3, Cisco provides a rich set of access control lists (ACLs), all supported in hardware, that enforces policies
on Layer 3 and 4 information. The switching mechanism used by Cisco switches and routers is called Cisco Express
Forwarding. While this in itself is not a network security mechanism, the fact that it uses topology-based switching
means that certain Internet worms that generate thousands of IP flows will not adversely affect network performance.
Cisco uses Cisco Express Forwarding across its entire line of routing and switching products. At Layer 2, Cisco also
provides sophisticated protection mechanisms to protect Spanning Tree Protocol, such as BPDUGuard and
RootGuard, which enhance the overall security and availability of the network.
If Ethernet is being offered as the UNI, the transmission medium, whether SONET, WDM, or Ethernet, gets the end
customer’s traffic into the provider’s network. When offering a VPN service to the enterprise, service providers need
to put mechanisms in place to create the VPN and scale it across the service provider’s infrastructure. To accomplish
this, a number of mechanisms can be used.
Tag Stacking
Tag Stacking, also known as Q-in-Q, provides a means of tunneling one set of VLAN tags by inserting a second
VLAN tag into the frame. This allows a service provider who is running a native Ethernet switched network (in other
words, a network build entirely on Ethernet and Spanning Tree Protocol without IP or MPLS) to transparently tunnel
the customer’s VLANs from one site, through the service provider “cloud,” to the second site. There are a few benefits
to this type of approach. The first benefit is that the service provider and customer VLAN tags do not need to match.
This eases the VLAN management and distribution for the service provider. Another benefit is that this approach is
inherently multipoint and provides a service that, to the enterprise customer, looks like an Ethernet segment. Because
Ethernet is inherently a broadcast-based medium, many providers consider offering an Ethernet service that uses
these Ethernet capabilities. It should be noted that Tag Stacking is also less expensive and complex than MPLS
solutions and is very attractive from that perspective, although Spanning Tree Protocol limits the extent to which
a Tag Stacking network can grow.
Cisco is working within the IEEE 802.1 committee to standardize a mechanism of scaling bridging in the service
provider space. This draft, called Provider Bridges, is currently being evaluated in the IEEE and is also being
coordinated with efforts in other standards bodies, including ITU and IETF.
In this case, the “spigot” delivered to the customer is a provisioned or managed Ethernet connection. Using Cisco IOS
routers such as the Cisco 2600, 3600, or 3700 series, the customer-edge device is mapped into an Ethernet VLAN
that defines membership of a particular MPLS VPN. The Ethernet tagged frames will be transported over the
transmission infrastructure to a Cisco Catalyst 6500 Series Supervisor Engine 720, or Cisco 7600 Series or 12000
Series provider-edge router where the Ethernet VLAN is subsequently mapped to a particular MPLS VPN. At this
point, advanced MPLS features such as traffic engineering and MPLS fast reroute can be used to better engineer the
customer’s network, adding potential service revenue to the provider.
The Cisco EoMPLS solution, based on an IETF draft standard, is an extension of MPLS, which naturally
complements the VLAN capability inherent in Layer 2 architectures. In its simplest form, EoMPLS provides an
emulated wire that is used to transport Layer 2 traffic across an MPLS-enabled Layer 3 core. This allows the service
provider the best of both worlds: the scalability of an MPLS core without having to worry about Spanning Tree
Protocol, and a Layer 2 transparent service offering.
A VPLS is an architecture that describes how Virtual Switch Instances may be interconnected using pseudo-wires to
form an emulated LAN service. A Virtual Switch Instance behaves, functionally at least, much the way a LAN switch
would. This architecture functions as an overlay onto the MPLS cloud. Cisco is committed to supporting VPLS as a
service offering on its switch and router platforms.
Residential Ethernet
In many places around the world, construction of new multi-dwelling units and turnover of infrastructure, such as
laying new fiber, is allowing Ethernet to become an alternative for broadband access. Instead of using telephone lines
for DSL or the coaxial cable for cable modems, fiber is run to the multi-dwelling units or, in some cases, directly to
the residence. Many providers, such as some PTTs and competitive LECs in Europe and Asia or Independent
Operating Companies (IOCs) in the United States, are looking to offer a “triple play” service to the residential
customer, bundling voice services, video (cable TV and VoD), and data services (Internet access and Internet gaming).
Although voice and Internet access typically require little bandwidth of the network, video often requires significant
bandwidth, hence the interest in Ethernet to the home.
Multicast
Internet access is handled by high-speed connectivity coupled with intelligent routing protocols such as OSPF and
BGP. Voice, which is not bandwidth-intensive, requires QoS to meet its latency and jitter needs. The third service in
the residential market, particularly one over Ethernet, is video. And to scale delivery of video and ensure the best uses
of the available bandwidth, the service provider’s network must support multicast.
Multicast, in its simplest definition, provides intelligent forwarding of IP video stream to the destinations that have
requested that stream. Unlike bridged broadcasts, which flood to all users in a VLAN or bridge domain, multicast
uses Layer 3 to request streams that may be available across an IP network to be forwarded to the requester.
Additional Layer 2 mechanisms, such as Internet Group Management Protocol (IGMP) snooping, in the Cisco
switches then ensure that the multicasts (which at Layer 2, by default, are treated as broadcasts) are forwarded
intelligently to their destination.
Cisco has worked over the past 10 years to develop numerous innovations related to multicast forwarding and
routing protocols. Multicast replication and forwarding is performed in hardware at data rates exceeding tens to
hundreds of millions of packets per second in Cisco switches and high-end routers (Cisco 7600 and 12000 series).
Forwarding, though, is only part of the equation. The other is the complex task of determining how to send multicast
data from the source to only those destinations that have requested the multicast stream.
Multicast VLAN Registration (MVR) is another Cisco innovation that increases the efficiency of multicast transport
and is important for residential providers. MVR involves the creation of separate, dedicated VLANs constructed
specifically for multicast traffic distribution. Each Cisco Catalyst switch that receives an MVR stream will examine
each multicast group and internally bridge the multicast VLAN traffic to a particular subscriber that has requested
the multicast stream. This is yet another feature Cisco has developed to help providers offer new and incremental
services to their customers.
Service Selection
As Ethernet moves from a niche residential application to a more mainstream broadband-access mechanism, the
ability of the provider to create and deploy services and bill customers accordingly is of tremendous interest to the
provider. Today, many providers are aggregating Ethernet UNIs at the customer premises back to an aggregation
switch or router in the POP. It is now possible to add more system intelligence by including Cisco Service Selection
Gateway (SSG) software on the Cisco Catalyst 6500 Series and Cisco 7600 Series.
Both the Cisco Catalyst 6500 Series and the Cisco 7600 Series are optimized for aggregating Ethernet traffic that has
been deployed in numerous Ethernet service networks and service provider backbones. By coupling the Cisco SSG
software with the Multiprocessor WAN Application Module (MWAM), a line card dedicated to broadband service
selection in the Cisco Catalyst 6500 Series and Cisco 7600 Series, the service provider can support such service
deployments as PPP Terminated, L2TP Access Concentrator, and L2TP Network Server. This allows for termination
of the session and then application of the appropriate service.
The Cisco MWAM is a complete subscriber-management platform, providing basic subscriber services as well as
complete service-profile assignment and network routing. It enables the service provider to offer a number of services
to their subscribers. These include integrated voice and data, IP television and VoD, differentiated Internet access
with QoS, and VPN access. This allows for the “triple play” to be offered to residential users. The provider can also
offer more advanced services to the residence, such as online gaming and multimedia services.
Cisco Systems has more than 200 offices in the following countries and regions. Addresses, phone numbers, and fax numbers are listed on the
Cisco Web site at www.cisco.com/go/offices
Argentina • Australia • Austria • Belgium • Brazil • Bulgaria • Canada • Chile • China PRC • Colombia • Costa Rica • Croatia
Czech Republic • Denmark • Dubai, UAE • Finland • France • Germany • Greece • Hong Kong SAR • Hungary • India • Indonesia • Ireland
Israel • Italy • Japan • Korea • Luxembourg • Malaysia • Mexico • The Netherlands • New Zealand • Norway • Peru • Philippines • Poland
Portugal • Puerto Rico • Romania • Russia • Saudi Arabia • Scotland • Singapore • Slovakia • Slovenia • South Africa • Spain • Sweden
S w i t z e r l a n d • Ta i w a n • T h a i l a n d • Tu r k e y • U k r a i n e • U n i t e d K i n g d o m • U n i t e d S t a t e s • Ve n e z u e l a • Vi e t n a m • Z i m b a b w e
All contents are Copyright © 1992–2003 Cisco Systems, Inc. All rights reserved. CCIP, CCSP, the Cisco Arrow logo, the Cisco Powered Network mark, Cisco Unity, Follow Me Browsing, FormShare, and StackWise are
trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, and iQuick Study are service marks of Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCNA, CCNP,
Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, the Cisco IOS logo, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Empowering the Internet Generation, Enterprise/Solver,
EtherChannel, EtherSwitch, Fast Step, GigaStack, Internet Quotient, IOS, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, LightStream, MGX, MICA, the Networkers logo, Networking Academy, Network
Registrar, Packet, PIX, Post-Routing, Pre-Routing, RateMUX, Registrar, ScriptShare, SlideCast, SMARTnet, StrataView Plus, Stratm, SwitchProbe, TeleRouter, The Fastest Way to Increase Your Internet Quotient,
TransPath, and VCO are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and certain other countries.
All other trademarks mentioned in this document or Web site are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company.
(0304R) ETMG 203150—JR 10/03