White

PAPER

10 Gigabit Ethernet WAN PHY

Introduction
The introduction of 10 Gigabit Ethernet (10 GbE) WAN PHY into the IP/Ethernet networking community has led
to confusion over the applicability between 10 GbE WAN PHY and OC-192c/STM-64 interfaces. Both of these
interfaces support transmission at 9.6 Gbps, support SONET/SDH framing, and can be connected to SONET/SDH
ADM or DWDM gear. So what is the difference? The answer can be boiled down to two things: L2/L3 protocol
support, and link diagnostic and maintenance capabilities.

Support for Layer 2 and Layer 3
SONET/SDH interfaces deployed in routers have
been designed to support Packet over SONET/SDH
(POS) as defined in RFCs 1619, 1662 and 2615.
Even though most IP packets originate from Ethernet
networks as packet-only interfaces, support for
transmission and switching of Ethernet frames on
POS is explicitly excluded. POS interfaces actually
strip off the Ethernet header and add their own link
layer header, for example PPP or HDLC framing. This
design aspect of POS interfaces has led to multiple
efforts to carry Ethernet over SONET/SDH: ITU X.86
and T1X1 for OC-3 through OC-48, and 10 GbE
WAN PHY for OC-192.
While the cost difference between OC-192c POS
and 10 GbE WAN PHY has created the perception
that POS interfaces actually have greater function-
ality, this is a myth. By eliminating the need to
re-encapsulate Ethernet into PPP, 10 GbE WAN
PHY supports IP as well as direct transmission and
switching of Ethernet frames more efficiently and
cost effectively than OC-192 POS.
Rich Diagnostic and Maintenance
Capabilities
The misperception exists that if you want rich
diagnostic and maintenance capabilities across
OC-192 links you need to buy OC-192 POS
interfaces. The truth is that 10 GbE WAN PHY
provides equivalent, and in some cases enhanced,
diagnostic and maintenance functionality. The
reason for the misperception lies in the methods
used for the various functions. SONET approaches
diagnostics from the perspective of managing a
TDM link where all management information is
carried outside of the TDM data. 10 GbE approaches
the problem from a data link perspective where
much of the diagnostic information is carried
within the data frames. The 802.3ae standards body
recognized this fact when designing the WAN
PHY specification and included only the SONET
diagnostic and maintenance functionality not
already present in Ethernet.

Figure 1. Framing overhead

© 2007 FORCE10 NETWORKS, INC. [ PA G E 1 O F 5 ]

 Fault Management Functionality Ethernet Features TDM/SONET Equivalent

Link availibilty, proper cable Link detect, interface up/down Loss of signal, signal fail

Noisy link or faulty Tx/Rx Encoding error Coding/framing error

Configuration mismatch of faulty Tx Giants, jabber NA (fixed frame length)

Long links (collision), noisy link or faulty Tx/Rx Runts NA (fixed frame length)

Noisy link or faulty Tx/Rx CRC B1, B2, B3, FEBE/REI, signal degrade

Link quality NA (asynchronous, uses CRC) Errored seconds

Noisy link or faulty Tx/Rx NA (no pointers) Loss of pointer

Noisy link or faulty Tx/Rx NA (asynchronous, no clock) Loss of synch

Remote alarm/defect Keep-alive AIS, RDI

Fault protection/redundancy Link aggregation (LAG) APS (linear)

Fault protection/redundancy NA (no ring topology) APS (UPSR/BLSR)

Table 1. Fault management features

As shown in Table 1, 10 GbE WAN PHY makes mechanisms for many features (Table 1). Much of the
use of the SONET overhead bits to provide: difference between SONET and Ethernet can be
boiled down to the fact that SONET is synchronous,
• Same diagnostics capability
it relies on counting a specific number of clock
- Loss of signal
cycles to identify the start of a frame, and Ethernet is
- Framing errors/loss of frame
asynchronous, frames may come at any time. The
- Coding violations
synchronization diagnostics in SONET, for the most
(Line and Path RDI and Line and Path AIS)
part, do not apply in the Ethernet world. If the SONET
- Errored seconds
link does lose synchronization, the Ethernet layer
- Severely errored seconds
detects that the link is down without having to look
- Delay triggers (currently not supported in FTOS)
at the SONET synchronization overhead bits.
- Loss of synchronization (not needed)
• Same error/performance monitoring
SONET APS and 10 GbE LAG
- B1, B2, B3
- J0, J1 ( J1 is currently not visible in the FTOS CLI) One area of concern is 10 GbE WAN PHY’s lack of
support for SONET automatic protection switching
• Same defect/alarms
(APS). In the POS world, support for APS was an
- FEBE
interesting feature because there is no equivalent in
- RDI
the IP world. Nevertheless, APS on POS interfaces is
- AIS
significantly different from the APS supported on
SONET add-drop multiplexers that must participate
Ethernet has built in mechanisms for detecting link
in a ring topology. POS interfaces only support Linear
state and frame errors so it does not need to rely on
APS, or the ability to switch from a failed primary
the SONET capabilities to provide this information
point-to-point connection to an idle backup.
even though it can use both Ethernet and SONET

© 2007 FORCE10 NETWORKS, INC. [ PA G E 2 O F 5 ]

Protection Fast Failover Time
Linear APS does have the advantage of 50 ms
Protection of Any Media
switchover time versus the approximately 100 ms it
The IEEE 802.3ae working group did not define Linear
takes for LAG to detect a failed link and switch. In the
APS for 10 GbE WAN PHY because Ethernet already
voice world, 50 ms restoration of a SONET link that
has a more generalized approach to handling this kind
could be carrying anywhere from thousands to millions
of link redundancy, 802.1ad Link aggregation (LAG).
of revenue-generating telephone connections is impera-
Unlike APS, since LAG operates at Layer 2, primary
tive. A link failure that caused any connections to be
and backup links do not need to traverse the same
dropped is a direct hit to the bottom line. In the data
Layer 1 physical media — one link may be running
world the difference between 50 ms and 100 ms is like
10 GbE LAN PHY over dark fiber and the backup link
splitting hairs. In neither case will the link failure cause
can be 10 GbE WAN PHY running over SONET or
loss of IP sessions or connections
DWDM. This gives 10 GbE a major advantage over
APS that only operates over SONET.
Summary
N+1 Protection
The myth that 10 GbE WAN PHY is a less capable
Another advantage of using LAG for protection switch-
version of OC-192 POS has been proven to be just a
ing is its ability to simultaneously protect multiple
myth. With support for both switching of Ethernet
circuits. LAG incorporates load-balancing capabilities
frames and IP packets, 10 GbE WAN PHY offers greater
that enable up to 16 links to be combined into a
flexibility in utilizing 10 Gbps links. The link diagnostic
single logical connection.
and management functionality of OC-192 POS has been
incorporated into 10 GbE WAN PHY giving the latter
equivalent capabilities. With the addition of LAG-based
protection switching available on 10 GbE, the link
protection capabilities and topology options surpass
those available with SONET OC-192.

Figure 2. Protection mechanisms in SONET and Ethernet

© 2007 FORCE10 NETWORKS, INC. [ PA G E 3 O F 5 ]

Appendix: WAN PHY Diagnostic CLI Command Example

Force10#show controllers TenGigabitEthernet 6/0

Interface is TenGigabitEthernet 6/0

SECTION
LOF = 0 LOS = 0 BIP(B1) = 0

LINE
AIS = 0 RDI = 0 FEBE = 0 BIP(B2) = 0

PATH
AIS = 0 RDI = 0 LOP = 0 FEBE = 0 BIP(B3) = 0

Active Defects: NONE

Active Alarms: NONE

Alarm reporting enabled for: B1-TCA LAIS LRDI B2-TCA B3-TCA

Framing is SONET, AIS-shut is enabled

Scramble-ATM is enabled, Down-when-looped is enabled
Loopback is disabled, Clock source is line, Speed is Oc192
CRC is 32-bits, Flag C2 is 0x1a, Flag J0 is 0xcc, Flag S1S0 is 0x0

Appendix: Use of Overhead Bits in SONET and 10 GbE WAN PHY

Byte Description OC-192c SONET 10 GbE WAN PHY

A1 Framing Frame alignment Fixed: 0xF6

A2 Framing Frame alignment Fixed: 0x28

J0 Section trace User programmable section message

Z0 Growth Undefined Unused: 0xCC

B1 Section BIP-8 Bit interleave parity for section error monitoring of previous frame

E1 Orderwire Optional 64kbps channel between STE Unused: zero

F1 Section user channel Optional for data communication Unused: zero

D1-3 Section DCC Optional for data communication Unused: zero

Table 2. Section overhead

© 2007 FORCE10 NETWORKS, INC. [ PA G E 4 O F 5 ]

 Byte Description OC-192c SONET 10 GbE WAN PHY

H1, 2 Pointer byte Location of SPE in each STS relative to the pointer (in bytes). Also used for frequency justification.

H3 Pointer action Used to adjust the fill of input buffers in case of negative Fixed: 0x00,
pointer justification. Also used for frequency justification. no pointer action required

B2 Line BIP-1536 Bit interleave parity for line error monitoring of previous frame

K1 Linear APS Request (bits 1-4): specifies automatic protection switching request type Fixed: 0x0, no request
or Ring APS
(not used by WAN PHY) Channel (bits 5-8): channel to switch from Fixed: 0x1, working channel

K2 Linear APS Channel (bits 1-4): channel to switch to Fixed: 0x0, working channel
or Ring APS
(not used by WAN PHY) Redundancy (bit 5): 1+1 or 1:n Fixed: 0 1+1

Status (bits 6-8): indicates line AIS (111), line RDI (110), bi-directional (101), Indicate line RDL, AIS
unidirectional (100), rest undefined or set to zero

D4-12 Line OCC Optional for data communication Unused: zero

S1 Synch Synchronization status Fixed: 0x0,

no line synch required

Z1, 2 Growth Undefined Unused: zero

M1 FEBE Reports detected block errors (B2) to far end

E2 Orderwire Optional 64kbps channel between LTE Unused: zero

Table 3. Line overhead

Byte Description OC-192c SONET 10 GbE WAN PHY

J1 Path trace 16 J1 bytes in consecutive WIS frames make up a user programmable path message that
repeats. First byte is 0x00 for no message.

B3 Path BIP-8 Bit interleave parity for path error monitoring of previous frame

C2 Path signal label Specifies payload type in SPE Fixed: 0x1A

G1 Path status REI/FEBE (bits 1-4): reports detected block errors (B3) for far end
RDI (bit 5): remote defect indicator

F2 Path user channel Optional for user communication Unused: zero

H4 Multiframe indicator Used for VT structured payload Unused: zero

Z3, 4 Growth Undefined Unused: zero

N1 TCM Optional tandem connection maintenance Unused: zero

Table 4. Path overhead

Force10 Networks, Inc. © 2007 Force10 Networks, Inc. All rights reserved. Force10 Networks and E-Series are registered trademarks, and Force10, the
Force10 logo, Reliable Business Networking, Force10 Reliable Networking, C-Series, P-Series, S-Series, EtherScale, TeraScale,
350 Holger Way FTOS, SFTOS, StarSupport and Hot Lock are trademarks of Force10 Networks, Inc. All other company names are trademarks of
San Jose, CA 95134 USA their respective holders. Information in this document is subject to change without notice. Certain features may not yet be generally
available. Force10 Networks, Inc. assumes no responsibility for any errors that may appear in this document.
www.force10networks.com
WP24 807 v1.2
408-571-3500 PHONE
408-571-3550 FACSIMILE

© 2007 FORCE10 NETWORKS, INC. [ PA G E 5 O F 5 ]