T E C H N O L O G Y

W H I T E

P A P E R

Synchronization over Ethernet Networks

While packet networks, for example, Ethernet, IP and MPLS networks, have been optimally designed for data services, they are increasingly being called upon to support other services, including those traditionally delivered on a circuit-switched infrastructure. Many of these services and their interfaces have stringent timing requirements to ensure correct operation. This document provides a comprehensive review of Synchronization over Ethernet techniques together with best practices and pragmatic deployment considerations. Technical background information is included to assist in the selection and engineering of appropriate timing and time distribution capabilities, for effective service delivery and transport, within communications networks.

Table of contents
1 2 1. Introduction 2. Ethernet Timing and Time technologies

3 2.1 Synchronous Ethernet 4 2.2 Timing and Time over packet technologies 8 3. Packet delay variation

8 3.1 Causes, measurements and filtering 9 3.2. Packet transfer delay variation examples 10 4. Deployment Options

10 4.1 Leased line replacement 10 4.2 Rollout of Synchronous Ethernet 12 4.3 End-to-end Timing Over Packet 13 13 5. OAM&P Aspects 6. Background information

13 6.1 Timing distribution and clock hierarchy 15 6.2 Timing applications 16 6.3 Continuous bit delivery over SDH/SONET networks 16 6.4 Continuous bit delivery over packet networks 17 6.5 Packetized data delivery services 17 6.6 Service clock vs. retiming 18 6.7 Oscillator Stability and PLL 18 6.8 Performance Metrics 19 6.9 Mobile base-station frequency reference 20 6.10 Special note on Synchronous Ethernet over copper 21 21 7. Conclusions 8. Abbreviations

1. Introduction

Observe due measure, for right timing is in all things the most important factor.
Hesiod, Greek didactic poet (~800 BC)

The growth in Internet usage and the increasing number of applications that demand higher bandwidth are trends that are forcing changes in todays telecommunication networks. The traditional circuit switched networks widely deployed today were designed to support the transport of traditional voice circuits; they are not optimal for transporting large volumes of data. Many operators are now looking to transform their networks from the circuit switched paradigm to an IP/Ethernet paradigm, leveraging MPLS for determinism, OAM tools, etc. The TDM-based networks were originally used for transporting voice traffic, but later evolved to provide for data services. To better handle data traffic, operators are migrating from these TDM-based networks to packet-based networks. Key drivers for this migration are the commercial benefits, including rapid service creation and delivery, together with reduced capital and operating expenses. While packet networks (e.g., Ethernet, IP, and MPLS networks) have been optimally designed for data services, they are now being called upon to support other services including voice transport. Ideally, all devices will migrate to support packet interfaces, but realistically there will be a long migration period where both the legacy and packet interfaces will need to be supported by the packet network. Therefore, there is a continuing need to transport circuit switched (TDM) services in addition to providing support for legacy synchronous interfaces for ATM/ML-PPP/FR/etc. Also, some applications have evolved which rely on the traditional TDM interfaces not for their traffic transport but for their inherent synchronization characteristics. The most significant of these applications is the mobile base-stations use of the T1/E1 ports from the network as accurate frequency references used to drive the RF carriers and to facilitate accurate handovers between base-stations. These services and interfaces have stringent timing requirements to ensure correct operation. Packet networks, however, were never required to support timing services; as a result, network operators now must find a way to provide these timing services over the packet infrastructure. Possible solutions range from the creation of overlay synchronization networks to the deployment of expensive distributed reference clocks. These approaches are contrary to the original value proposition of a packet network and, therefore, the challenge for vendors and carriers is to adapt methods that will allow packet networks to support time-sensitive services and applications in a cost-effective and scalable manner. Several technologies are emerging to meet these timing needs. These range from changing the Ethernet physical layer to provide synchronization reference distribution (along the lines of SDH and SONET) to Timing Over Packet (ToP) techniques such as adaptive clock recovery on Circuit Emulation Services, enhanced NTP and IEEE1588v2. The latter two technologies also address the requirement for the distribution of highly accurate time for applications that need time-of-day or phase accuracy. When service providers look toward these new technologies, they need to view the transport of timing and time1 as a network service. These services will have network planning considerations along with OAM&P tools in order to ensure proper operation within the network.
1

 ote on terminology: Within this document, and generally in the industry, the term timing is used for discussing the delivery or use N of a frequency. Phase is used when there is a requirement for time alignment over a relatively short (< 1 second) period of time. Time or Time of Day is used when there is a requirement for time alignment over long periods of time (anything from a number of seconds to years).

Synchronization over Ethernet Networks | Technology White Paper

This document discusses these technologies and some of the service related aspects that must be considered when planning the provision of timing and time services over Ethernet core networks. While it is not necessary to be a expert in synchronization to follow the contents of this document, some level of understanding of legacy synchronization networks is useful. For those who need a refresher, Section 6 contains an overview of some of the key terms and concepts relating to network synchronization.

2. Ethernet Timing and Time technologies

Traditionally, reference timing has been distributed using synchronous physical layer technologies. This means that the devices driving the interfaces have the ability to both transmit the data using a timing reference and to recover the timing from the received data for reference purposes. Many technologies include this capability. For example: PDH, SDH, SONET, POS, PDH Microwave, DSL, GPON, and WDM. Traditional Ethernet does not fall into this class since it uses local interface timing rather than a timing reference for its transmissions. The Ethernet interfaces either need to be modified to act as synchronous physical layer interfaces (Synchronous Ethernet) or a higher layer timing distribution protocol needs to be used (Timing Over Packet). These two approaches are depicted in Figure 1.

Figure 1. Layer 1 vs Timing Over Packet Synchronization distribution

Primary reference clock

Primary reference clock

Primary reference clock Clock recovery and regeneration points Synchronization distribution over Physical Layer 1 Synchronization distribution using timing over packet

Synchronous Ethernet is an enhancement of a legacy Ethernet interface and includes the ability to relay accurate timing information along with the Ethernet frames over the physical media. It is the highest performing solution for timing over Ethernet, but may cause some issues for complete network rollout.

Synchronization over Ethernet Networks | Technology White Paper

Several ToP techniques exist that allow for the transport of timing information without the need for a synchronous physical layer. These include NTP, IEEE1588, and ACR. Network Time Protocol (NTP), defined in IETF RFC 1305, has been used for many years to allow distributed devices to synchronize with respect to Time of Day. It uses timestamps embedded in packets to accomplish this synchronization. This protocol has been deployed in networks for over 20 years and has been proven to provide reliable time distribution to accuracies in the order of milliseconds. This level of accuracy is often acceptable for alarm time-stamping, billing and statistics; however, for applications like E1/T1 timing and mobile base-station phase and frequency references, accuracies of approximately 1 microsecond are required. IEEE1588 has, at its core, a timestamp distribution mechanism very similar to NTP. Version 1 was targeted at time distribution over an Ethernet LAN while Version 2 adds some key capabilities to address the distribution over the WAN environment. Both versions were designed to allow higher accuracy time distribution than possible with NTP. Circuit Emulation Service (CES) implementations use the constant bit rate of the TDM interfaces to generate packets. Adaptive Clock Recovery relies on this constant rate of packet generation to recover the original timing information. This is primarily intended to allow for the transport of the originating E1 or T1 service clock, but if the originating interface is known to be locked to the PRC traceable reference, CES can then be used to distribute a highly accurate timing reference. 2.1 Synchronous Ethernet During early 2006, several European Telecom companies started an initiative within the ITU-T to define the requirements for having the traditional Ethernet interfaces meet comparable timing performance targets to those of SDH/SONET interfaces. They recognized that the Layer 1 relaying of synchronization information would potentially be the most reliable form of synchronization transfer and would not have any impact from the packet delivery load over the interface. By deploying a network of Synchronous Ethernet interfaces, or a hybrid of SDH/SONET and Synchronous Ethernet, the network provider can ensure the delivery of the same quality of timing references through the network as is currently achieved using SDH/SONET only. This can then be used for TDM services at the network edge or as a timing reference into Mobile base-stations for carrier frequency derivation. Synchronous Ethernet uses the physical layer of the Ethernet link to distribute the clock among nodes in the network. In an analogous manner to SONET/SDH, each node has a local or system clock which determines the outgoing clock rate of each interface. The system clock is derived from the incoming clock at one of its input interfaces or from a dedicated timing interface such as a BITS port. Synchronous Ethernet is based on the same architectural structure as SONET/SDH. It is important to note that Synchronous Ethernet works at layer 1 and is concerned only with the precision of the timing of signal transitions to relay and recover accurate frequencies. It is not impacted by the traffic load. For this reason, it has been shown to have a performance equivalent to that seen in SDH/SONET networks. Figure 2 shows a test environment created within the Alcatel-Lucent laboratories to analyze the performance of a long chain of a total of 20 Service Routers and Service Aggregation Routers using Synchronous Ethernet as the reference frequency distribution method. For this test, six of the devices were deployed within a thermal chamber to introduce extreme temperature variations into the environment. Even with this long chain and a 75 C temperature range, the timing reproduced at the end of the network was two orders of magnitude better than the network limit as defined by the ITU-T (see Figure 2 and Figure 3).

Synchronization over Ethernet Networks | Technology White Paper

Figure 2. Synchronous Ethernet test environment

20 nodes

2.048 MHz

Monitor

E1 SynchE 7705 SAR #1 7750 SR #2 SynchE 7750 SR #3 SynchE 7705 SAR #4 SynchE

...

SynchE 7705 SAR #18 7705 SAR #19

SynchE 7705 SAR #20

The Synchronous Ethernet specification also includes the same ability to relay timing source quality level information as is found in the Synchronization Status Messages (SSMs) of PDH and SDH/SONET. Since Synchronous Ethernet uses Ethernet OAM messages for this purpose, there is discussion within the standards bodies of these SSMs being expanded to support many new features to assist in the management of Synchronization Distribution. Alcatel-Lucent is actively involved in these discussions and is developing applications to make use of these capabilities.

Figure 3. Synchronous Ethernet MTIE performance over Alcatel-Lucent Service Routers

2.2 Timing and Time over packet technologies While Synchronous Ethernet is a Layer 1 enhancement providing for the relay of timing information completely independent of user traffic volume, the remaining technologies to be discussed are all packet based and are impacted by the user traffic. This means that the packets carrying the timing information must compete for network resources with all of the other data services and the routing protocol packets. Significant work has been conducted over the past several years which has led to implementations of these technologies being able to meet the required performance targets over relatively noisy environments where variable queuing and processing delays can be introduced. Since changes in network conditions with respect to traffic load can affect the performance of these technologies, there is the need to provide more management and monitoring ability for these services than might be required for Layer 1 technology like Synchronous Ethernet. Alcatel-Lucent has architected its Timing Over Packet solutions to include the OAM&P capabilities to allow these technologies to be deployed with success.

Synchronization over Ethernet Networks | Technology White Paper

As indicated above, all of these ToP technologies are affected by traffic load in the network. The key parameter that relates to the performance is the variation in the Packet Transfer Delay (PTD) across the network or the Packet Delay Variation (PDV). Section 3 provides details on PDV. The following sections provide an overview of the prevailing Timing and Time over packet technologies.
2.2.1. Adaptive Clock Recovery with Circuit Emulation Service

Adaptive Clock Recovery (ACR) is used in conjunction with circuit emulation services. It is intended to ensure that the timing used to transmit the TDM data out from the packet network is tracking the timing used to inject the TDM data into the packet network. If these two timings are not matched, then the packet to TDM IWF will either overflow with data coming from the packet network faster than can be transmitted or it will underrun as it transmits data faster than can be replenished from the packet stream. This recreation of the originating TDM service timing is performed based on the expected rate of delivery of the packets. At the packet to TDM IWF as shown in Figure 4, adaptive methods adjust a local frequency reference to ensure that the rate of data being transmitted by the packet to TDM IWF matches the rate of data reception at the TDM to packet IWF. This can be accomplished in multiple ways. One well understood method is to make use of the CES jitter buffer level. This buffer is sized to accommodate the expected PDV from the network. Playout of data from this buffer starts once a threshold level is reached. If the depth of the buffer begins to drop over time, it means that the local timing reference is running too quickly and needs to be slowed down to match the originating service clock. Conversely, if the buffer begins to grow, the local reference needs to be speeded up. By monitoring the level over long periods, the effects of PDV can be averaged out and the originating service clock can be matched. An alternate method can use the inter-arrival times of the packets. Since the packet generation rate should be known (e.g., 1 packet per millisecond) the local frequency reference can be adjusted to ensure that the average packet inter-arrival time is the expected duration. Any delay variation between packets will be reflected as a variation in the short-term frequency of the recovered clock. The trick is to set the averaging time to be long enough to remove PDV but not so long that the local timing reference drifts due to environmental conditions.

Figure 4. Adaptive clock recovery

Adaptive clock recovery Recovered TDM timing based on adaptive clock recovery TDM

CE TDM IWF

Packet switched network

IWF

CE

TDM service clock

Synchronization over Ethernet Networks | Technology White Paper

The adaptive clock techniques implemented in the Alcatel-Lucent Service Routers and Service Aggregation Routers have been finely tuned to ensure optimal performance given noisy network environments. Adaptive clock recovery can be used for both service clock recovery and, given a PRC traceable reference at the TDM to packet IWF, for Synchronization timing distribution.
2.2.2. Differential Clock Recovery with Circuit Emulation Service

In the differential timing method, which is often referred to as Differential Clock Recovery (DCR), both the encapsulation and de-capsulation interworking functions (IWFs) have access to a common reference clock. DCR is used exclusively for service clock recovery. The encapsulation side inserts a timestamp with each transmitted packet and the de-capsulation side retrieves the timestamp and uses that as the main input parameter to the clock recovery subsystem to control the timing of the outgoing TDM stream. ATMs Synchronous Residual Time Stamp (SRTS) is an example of this differential timing method. The differential timing method handles the network impairments on the PSN much better than adaptive timing methods. This is because it does not rely directly on the arrival time of each packet but instead uses the timestamps to control the outgoing bit rate. The issue with differential timing is the requirement for a common timing reference to be available at both ends of the network as shown in Figure 5.

Figure 5. Differential clock recovery

Differential timing messages Recovered TDM timing based on differential timing messages TDM

CE TDM IWF

Packet switched network

IWF

CE

Synchronization network TDM service clock

Synchronization network

PRC

PRC

The two PRCs may also originate from the same source

2.2.3. NTP

NTP has been in use since the early 1980s and continues to provide Time of Day synchronization to devices connected over the public Internet and private Internets. It works extremely well in allowing edge devices to be synchronized to within tens of milliseconds of UTC. It was not designed for highly accurate frequency distribution, as is now being considered for telecommunication applications, nor for the highly accurate phase requirements of the TDD mobile technologies. However, mated with a high quality oscillator and using long time constants to filter PDV, NTP has been shown to meet the target MTIE performances in lab trials and currently in some live deployments. While the protocol is fully defined in RFC 1305, including a recovery algorithm, vendors can implement a different algorithm if desired. This may be desirable given the much more stringent performance targets of the telecom application. In addition, the message rate from a single client is usually

Synchronization over Ethernet Networks | Technology White Paper

restricted by the NTP servers to sub 1 Hz message rates (one message per minute in locked mode is common). This protocol was defined in order to avoid overloading servers in the public internet environment while still allowing millisecond accuracies using low cost TCXO technology as the core of the recovery algorithm. For private internets, the message rate can be increased to allow for more accuracy. The protocol uses four timestamps: two from the server and two from the client. Using these four timestamps, and assuming symmetric delays in the server-to-client and client-to-server directions, the client can synchronize its time of day to that of the server. The PDV is averaged out by using many sets of timestamps. While the protocol was designed to synchronize a clients time of day with that of the server, it can also be used for frequency-only synchronization. Generally speaking, the assumption of symmetric delay does not create a problem for frequency recovery, but it can be a significant factor for highly accurate time of day and phase recovery. The timing industry needs to study this aspect in more detail to ensure that accuracies in the low microseconds can be maintained across an entire network. Alcatel-Lucent Service Routers currently use NTP and SNTP to distribute and recover time of day for management purposes.
2.2.4. IEEE1588v2

IEEE1588v2 and its Precision Time Protocol (PTP) message exchange is another mechanism that can be used to synchronize time and timing within a network2. Version 1 of this standard is currently being used in the LAN environment of industrial manufacturing. It uses a very similar concept of time-stamped packets between master and slave network elements to NTP but includes some enhancements such as higher packet rate and hardware-based time-stamping to improve on the accuracies of the recovered time. IEEE1588v1 has demonstrated accuracies in the one microsecond range in the LAN environment. However, when applied to the noisier WAN environment, it cannot guarantee this performance. Version 2 was created to try to address this noisier environment. Two significant concepts within IEEE1588v2 are the boundary clock and the transparent clock. The boundary clock is a device which has at least one slave port recovering timing/time from an upstream master and it then uses this recovered timing/time as a basis for one or more master ports toward downstream slave ports. The boundary clock can then be used both for scaling purposes and as an intermediate device to break up the PDV between the grandmaster and the slave devices. The transparent clock is a device that participates in IEEE1588v2 but does not perform any timing/time recovery. The transparent clock measures the residence time of each PTP message as the message transits the node and updates the message with this residence time. Since most of the PDV is caused by queuing within the nodes, the transparent clock can remove this unknown. If every device between the master port and the slave port performs as a transparent clock, then the actual transit time for each message can be measured and corrected. In an ideal IEEE1588v2 network, every device along the path from the Grandmaster clock to the slave clock is IEEE1588v2 aware and acts as a boundary or transparent clock. However, many implementations have been developed that can meet the performance targets in environments where there are a number of non-IEEE1588v2-aware devices between the master and slave clocks.

 he terms IEEE1588, 1588, and PTP have been used relatively interchangeably within the industry. However, PTP is the messaging T protocol, and IEEE1588 is the standard that defines this protocol and its use within the network. Within this document the term IEEE1588v1 refers to the standard as specified in IEEE Std 1588 TM -2002 and the term IEEE1588v2 refers to the standard as specified in IEEE Std 1588 TM -2008

Synchronization over Ethernet Networks | Technology White Paper

3. Packet delay variation

3.1 Causes, measurements and filtering The variation in the time it takes packets to transit a network is referred to as PDV. PDV is an important factor that impacts all ToP techniques like ACR, NTP, and IEEE1588. As discussed earlier, it does not have a significant impact on DCR. If the packet delay through the packet network is constant, the arrival rate of packets at the destination node is constant. It will be relatively easy for the clock recovery module to recreate a stable frequency, phase, and time of day from the delivered packets. If the packet delay varies, then the recovery algorithm is more rigorously exercised. The algorithm has to deal with both sudden and slow changes in packet delivery based on reroutes or slowly increasing loads in the network, as well as possible changes in the master timing due to reference switches (needs to be followed) and/or drift in the local oscillator at the core of the local clock (needs to be removed). Therefore, a clock recovery design must incorporate a filtering capability to recognize and remove these effects. There are several causes of packet delay variation on a packet network. Some examples include: Random delay variation (e.g., packet arrivals at queuing points) Low frequency delay variation (e.g.,day/night traffic load patterns) Systematic delay variation (e.g., store and forward mechanisms in the underlying transport layer) Routing changes and congestion effects The most significant factors in the PDV are: the number of nodes (which contain queuing points) the speed of the interfaces (GE vs. FE, for example) technology of the interfaces (Ethernet, DSL, Wave) the packet size, randomness, and load of the aggregate traffic in the network In order to know whether a particular ToP implementation will meet the performance targets in a given network deployment, it is desirable to characterize the limits on the PDV that the implementation can support and to also measure the network against these limits. Unfortunately, ToP implementations as they exist today cannot always quantify the PDV metric or the specific limit. There have been several proposals for how to measure the PDV (peak-to-peak level, TDEV, minTDEV, MAFE, etc.) but since implementations use different filtering techniques, it is not clear that one single PDV metric will apply to all solutions. Even within one implementation there may be factors built into the timing recovery algorithm which filter the PDV in multiple ways. In these cases, one single PDV metric would not be sufficient to define the algorithms behaviour. When a PDV metric (or set of PDV metrics) is available, the target deployment network will need to be measured to ensure that the PDV metrics are not exceeded. It may be difficult to create all the network conditions over which the solution must operate, so some scenarios will have to be estimated. It will be essential for any deployed solution to provide some indication of its performance so that it can be monitored to ensure that performance limits are being met. In the ACR implementation on the Alcatel-Lucent Service Routers (and Service Aggregation Routers), the algorithm filters all the packets received over a short time period and identifies the one packet which experienced the minimal delay in transiting the network. Only these filtered packets are used to drive the clock recovery algorithm.

Synchronization over Ethernet Networks | Technology White Paper

This mechanism works well in environments where there are some packets which transit the network without experiencing any queuing. These packets then have a consistent network transit time. By using only these packets as input to the phased-locked loop (PLL) portion of the clock recovery algorithm, the local frequency can be tuned to match that used at the packet origination point (i.e., the TDM to packet CES IWF point). The ACR implementation on the Alcatel-Lucent Service Routers includes metrics on both the consistency of the packet filtering and the stability of the PLL. Over time, thresholds will be defined to provide alarms when the recovered clock stability is indicating that the performance targets are at risk. 3.2. Packet transfer delay variation examples For illustrative purposes, Figures 6 to 8 show some end-to-end packet transfer delay (PTD) data from testing across a network of AlcatelFigure 6. PTD Histogram for 4-node network with 80% aggregate Lucent Service Routers. The PDV is the variation of this PTD. Figure 6 shows the histogram of the PTD experienced by circuit emulation traffic through a four-node network at 80% loading. Figure 7 shows the histogram of the PTD for an eight-node network that was loaded to 20% capacity. The important element of the histogram for the minimally delayed packets is the relative height of the left-most point. This provides an indication of the consistency of the minimal delay that will be seen by the filtering. The test result for the eight-node network shows a strong component on the left of the data. This translates as a high probability that in one filtering period there will be a packet with a minimal PTD so the clock recovery will work very well. The four-node 80% test result shows a much weaker component on the left of the data. This means that there will be a much lower probability that in one filtering period there will be a packet with a minimal PTD. Minimal delay packet filtering algorithms will not perform as well in the four-node test network as in the eight-node test network. This shows that loading is a significant factor in PDV and the resulting ToP performance.

Figure 7. PTD Histogram for 8-node network with 20% aggregate traffic

Synchronization over Ethernet Networks | Technology White Paper

Figure 8 shows two recordings of the individual PTDs seen during a 16-hour test. PDV can be read from the height of the lines. In this test, the traffic loading in the network is gradually increased from 20% to 80% and then decreased. The blue data shows the PTDs experienced by the timing packets when all packets in the network have equal priority. The red data shows the PTDs experienced by the timing packets when the timing packets are given a higher priority than the other traffic in the network. These results show the benefit of using packet prioritization to control the delivery of timing packets and reduce the PDV.

Figure 8. Comparison of PTD in 3-node network with and without QoS

4. Deployment Options
4.1 Leased line replacement An Ethernet network can provide traditional E1/T1 interfaces and perform circuit emulation across the network. This configuration could replace the traditional leased E1/T1 services offered by wireline carriers. In this deployment, the Circuit Emulation Service (CES) must meet the same performance characteristics of the SDH/SONET transport networks. The principle concern is the transport of the E1/T1 bits from one location to another. This means that support for service clock transport is required. This service clock transport support can be provided using either an ACR or DCR timing method with the CES. When access to a traditional synchronization distribution network with BITS devices is available, then the preferred clock recovery technique is DCR (as it is immune to PDV). If such a distribution network was not available, then the option is to either use DCR on the E1/T1 circuits along with a dedicated ToP technique to distribute the common reference or to use ACR on the individual E1/T1 circuits 4.2 Rollout of Synchronous Ethernet
4.2.1. SyncE SDH interop

If a network provider is providing Ethernet ports on a SDH/SONET backbone, then there is likely to be an option to upgrade only the Ethernet interface modules from legacy Ethernet to Synchronous Ethernet capabilities. The SDH/SONET equipment will already support a clock architecture conformant to the SDH/SONET requirements and should be able to operate in Hybrid mode between SDH/SONET synchronization distribution and Synchronous Ethernet distribution. In this environment, the Synchronous Ethernet interfaces can be rolled out only where needed at the network edge. This allows for a phased transition from the SONET/SDH network over to an all Synchronous Ethernet backbone while providing immediate support of Synchronous Ethernet services at the network edge.

10

Synchronization over Ethernet Networks | Technology White Paper

Figure 9. Synchronous Ethernet ports timed from SONET network Source

SONET network

Primary reference clock

Legacy Ethernet network

Primary reference clock Clock recovery and regeneration Figure 10. Synchronous Ethernet ports timed from BITS at edges of points network Synchronization distribution over Physical Layer 1 Primary reference clock Synchronization distribution using timing over packet Primary reference clock

Legacy Ethernet network

Primary reference clock Clock recovery and regeneration points Synchronization distribution over Physical Layer 1 Synchronization distribution using timing over packet

4.2.2. SyncE with ToP

The ToP technologies have to be concerned with the PDV between the timing master and the timing slave points. Using SyncE as the synchronization distribution technique over the core links (and then ToP only over the last mile links) can be a cost-effective deployment option while ensuring performance is maintained. This scenario is depicted in figure 11. Since traffic loading is lower at the network edge than near the core hub, the smaller number of core links can be upgraded to Synchronous Ethernet. This avoids the high PDV that can occur when the links are run with high loading, while at the network edge ToP can maintain performance when there are fewer links and lower loading. As there are many more links at the network edge and usually multi ple transmission technologies, this permits a cost-effective and consistent solution over the last mile.

Synchronization over Ethernet Networks | Technology White Paper

11

Figure 11. Combination of synchronous Ethernet and Timing Over Packet

PRC

Cell site access Cell site gateway (xDSL, GPON, microwave, Legacy Ethernet) Access aggregator

Aggregation network (Optical synchronous Ethernet) Mobile aggregation site gateway

Core network BSC/RNC

BTS/NodeB

Timing provided through synchronous L1 technology Timing provided using timing over packet technology

4.3 End-to-end Timing Over Packet As discussed above, the principle concern with the Timing Over Packet techniques is the control of the PDV. The larger the network span between the master and slave agents, the larger the peakto-peak PDV and the greater the trend to Gaussian distribution. In all Timing Over Packet implementations there will be an engineering limit to the network span over which it will work. There are three methods to address this. The first is to use distributed masters where GPS-based masters are placed within the network to ensure that the span between the master and slave agents is reduced to an acceptable limit. The second method is to use boundary clocks within the network between the primary master and the edge slave devices. The third method, applicable to IEEE1588v2 deployments only, is to use transparent clocks within the network. The use of distributed timing masters, or boundary clocks using a central primary master, are similar concepts. Before the availability of network elements incorporating boundary clock functionality, the use of distributed timing masters will be more common. While the distributed GPS-based masters are operating in plesiochronous mode, the difference in frequencies at the various receivers is so small (0.01 ppb) that there should be no real impact from this mode of operation. Once the networking elements are upgraded to support boundary clock functionality, the use of one central GPS-based timing master and then operating through boundary clocks to reach the network edges will become more common. In this case, the end-to-end PDV is replaced by several segments of low PDV, but with the tradeoff of operating several clock recovery control loops in series. More analysis is needed to determine if the use of boundary clocks will become more common. More data needs to be collected on the tradeoffs between using boundary clocks to reduce the PDV of a single large network span into multiple master-slave spans. Is there some optimal number of internodal links that provide enough PDV reduction while not introducing too many recovery steps? At least for frequency distribution, preliminary testing has shown some benefits to having boundary clocks in every network element in order to ensure the PDV is minimized for each master-slave association.

12

Synchronization over Ethernet Networks | Technology White Paper

5. OAM&P Aspects
As Synchronous Ethernet interfaces roll out in a network and are used for Synchronization distribution, the methods and procedures developed by the SONET/SDH timing experts in operation of the distribution network will be relevant to the new environment. New capabilities unlocked by enhancements to the SSM of Synchronous Ethernet will become available to assist and expand on the management of the distribution network. These capabilities can be integrated into dedicated management systems or incorporated as subsystems within the overall service management agents of the network. The intention will be to develop a comprehensive system to allow for the analysis of the distribution network to look for optimizations and analyze network scenarios. In the case of the newer concept of ToP, there is an even greater requirement for management capabilities to control and monitor the performance. These capabilities start in the clock recovery slaves that run in the edge devices but also apply to the management of the timing masters and the delivery paths between the masters and slaves. The implementations within the slaves should be capable of indicating some level of confidence in the accuracy and stability of the recovered timing and time of day. This will have to be based on the stability of the output of the PDV filtering algorithm. If the algorithm is having trouble generating a consistent output, then this will be reflected in variations of the recovered frequency. Since this situation will be impacted by network loading, this needs to be monitored on a constant basis and indication given when stability drops. These indications can then be correlated to changes in network conditions, and corrective action can be performed in order to reduce the variability seen by the slave. These OAM&P capabilities need to be included in the solutions as they are provided. The Alcatel-Lucent Service Routers ACR implementation includes an inherent performance monitoring capability along the lines described above. A network manager can monitor performance and run statistical collection continuously, or on demand, to investigate network characteristics and performance.

6. Background information
6.1 Timing distribution and clock hierarchy In ideal deployments, all network elements derive timing from one single master clock, usually a Primary Reference Clock (PRC). The timing from that clock is relayed throughout the network using the hierarchical distribution defined by ITU-T and Telcordia. In this mode, all clocks will be able to maintain the same long term frequency and will just add some level of short term frequency deviation around this centre frequency (see metrics section below). If some segment of the network temporarily loses its traceability to the one central clock, then the long term frequency of that segment may start to drift from the central clocks frequency and this may begin to cause slips when the two different timing domains are crossed. The timing characteristics of the central PRC as well as those of the network element and BITS clocks have been specified within the series of recommendations G.811, G.812, and G.813. The ITU-T also ran simulation models to define the worst case limits of short term noise that can be experienced on an end-to-end connection across the entire network composed of these clocks. These limits are covered in the G.823/824/825 series of recommendations along with the maximum deployment model as shown in Figure 12. These limits derived from the worst-case network clock distribution coupled with a series of SDH/SONET network elements pointer adjustments, and the E1/T1 encapsulation and de-capsulation.

Synchronization over Ethernet Networks | Technology White Paper

13

Figure 12. Clock hierarchy ITU-T Model

Hierarchy PRC ITU-T Recommendation G.803 denes the sychronization reference chain

G.812 type I

G.812 type I

G.813 option 1 Number of G.812 type I clocks 10 G.813 option 1 Number of G.813 option 1 clocks 20

G.813 option 1

G.813 option 1

G.813 option 1

G.813 option 1

G.812 type I

G.812 type I

G.812 type I

G.813 option 1

Total number of G.813 clocks in a synchronization trail should not exceed 60.

G.813 option 1

G.813 option 1

In a similar fashion, the limit for the new Synchronous Ethernet-based clock has been provided in recommendation G.8262. Work is ongoing at the ITU-T on the deployment models and clock limits for Timing Over Packets techniques within the recommendations G.8261, G.8263, G.8264, and G.8265. While using a single clock as the timing source for the entire network is ideal, it may be operationally impractical. In practice, networks will often run with multiple PRC clocks. This configuration is termed plesiochronous operation and is acceptable since the PRCs are so highly accurate that the worst-case slip rate when using two separate PRCs is one slip every 72 days (see Table 1 for clock hierarchy).

14

Synchronization over Ethernet Networks | Technology White Paper

Table 1. Clock hierarchy

North America Stratum Level ITU-T Clock Level Free-run Accuracy Holdover Stability Pull-in / Hold-in Range Wander Filtering Phase Transient (Re-arrangement)

1 (PRS) 2 Not Defined 3E

G.811 (PRC) G.812 Type II G.812 Type I G.812 Type III

1 x 10-11 0.016 ppm Not defined 4.6 ppm

N/A 1 x 10
-10

N/A /day 0.016 ppm 0.01 ppm 4.6 ppm

N/A 0.001 Hz 0.003 Hz 0.001 Hz

N/A MTIE < 150 ns MTIE < 1 s MTIE < 150 ns Phase slope 885 ns/s

2.7 x 10 /day
-9

1.2 x 10-8 for initial 24 hours of holdover

G.812 Type IV

4.6 ppm

3.7 x 10-7 (or <255 slips) for initial 24 hours of holdover

4.6 ppm

3 Hz

MTIE < 1 s Phase slope 61 s/s Objective: MTIE < 150 ns Phase slope 885 ns/s

Not Defined SMC

G.813 Option 1 G.813 Option 2

4.6 ppm 20 ppm

2 x 10-6 /day 4.6 x 10

-6

4.6 ppm 4.6 ppm or 20 ppm

1 Hz 0.1 Hz

MTIE < 1 s MTIE < 1 s Objective: MTIE < 150 ns Phase slope 885 ns/s

for initial 24 hours of holdover

Not defined

32 ppm

NA

32 ppm

None

No requirement

6.2 Timing applications Todays timing requirements come from several sources. One is the traditional E1/T1 and SDH/SONET telecommunications networks bit delivery. Another is the physical layer transmission of data that must ensure data is successfully delivered between a transmitter and a receiver connected through a physical layer. A third is mobile technologies which require a highly accurate reference frequency to be available at the base-stations in order to ensure that their radio frequencies transmissions are within specification. The phase accuracy requirement is also a strong requirement of the mobile base-stations. For wireless technologies that use Time Division Duplexing (TDD) technology, the alignment of the TDD frames between base-stations transmissions must be controlled to ensure minimal interference. Currently, the applications requiring ToD are generally OAM type applications: alarming, statistics, and event logging. These applications currently expect accuracies in the ms range. Looking forward, the availability of highly accurate (1 s) time, new applications or efficiencies could begin to appear in the industry. In addition to the above, legacy Layer 2 transmission technologies have their own timing accuracy and stability requirements. If these legacy transmission interfaces are to be supported over the allEthernet network, then these accuracy and stability requirements must continue to be met.

Synchronization over Ethernet Networks | Technology White Paper

15

6.3 Continuous bit delivery over SDH/SONET networks The original purpose of the European and North American/Japanese PDH3 technologies and the follow-on SDH and SONET technologies was to provide the end-to-end delivery of a continuous stream of bits. This was primarily voice traffic coming from PCM coders within telephony systems. This traffic then expanded to include Fax, Modem traffic and then ISDN data traffic. As this constant bit rate (CBR) traffic is processed by network elements, the timing used to transmit the data into a network element must be matched to transmit the data out of the network element. If these are not matched, the network element will experience either a data overrun (when too much data has been collected) or data underrun (when the transmitter has sent all the data and no new data is available to send). In SDH/SONET network elements, these overrun and underrun events are known as slips. 6.4 Continuous bit delivery over packet networks When using packet technologies to provide Layer 1 CBR services, the network must meet the endto-end timing constraints defined in G.8261. These constraints were developed based on the limits defined for the SDH and SONET networks. The conversion between constant bit rate data and packetized data is known as circuit emulation service (CES). The application of CES allows a packet network to transport legacy CBR traffic. Instead of using dedicated circuits, the CBR traffic is packetized and aggregated with other CBR or VBR traffic onto a single packet infrastructure. The packets are generated at a constant rate based on the rate of data reception at the PDH or SDH/SONET interface. A CES interworking function (IWF) can support several independent packetization instances. The CBR data from each input port is processed and forwarded within a packet stream. At the remote end, the packets are deconstructed and the original CBR data is sent onward (see Figure 13 for a typical application scenario).

Figure 13. TDM interface service clock regeneration

Digital switch

E1 TDM

Packet network

E1 TDM

Digital switch

End customer clock used to time E1 toward network

IWF encodes the TDM stream including timing information into packets

IWF uses timing information from packets to regenerate the service clock received at network ingress

Since the TDM data is being received at the ingress CES IWF at a constant rate, the packets will be generated into the packet network at a constant rate. At the egress CES IWF, the transmission rate of the TDM data must match the rate of reception at ingress or data overrun or underrun will occur. The transport and re-creation of the clock from the ingress port to the egress port over CES is referred to as service clock transport.

F or the remainder of the document, the term PDH will be used to cover both European PDH and North American /Japanese PDH. 

16

Synchronization over Ethernet Networks | Technology White Paper

6.5 Packetized data delivery services The previous section discussed the original purpose of the SDH and SONET networks to support CBR bit delivery. But these networks and Ethernet networks also support the delivery of packetized data over the physical interfaces. When delivering packetized data, the timing requirements can be relaxed significantly due to two factors. First, the network is used to multiplex data of one interface to more than one destination. In this case, it is impossible to control the amount of data being directed to a single interface so the expectation is that there will be periods of congestion. The data will have to be stored during this congestion for transmission later. Secondly, data services can take advantage of the fact that when there is no data, nothing needs to be transmitted. For data services, the interface timing becomes only a means to ensure the proper delivery of the data across the interface. There is no requirement to ensure that the timing used to inject the data into the network at one side is matched to transmit the data out of the network at the other side. This applies to ATM cells, HDLC/PPP/FR frames, IP packets and Ethernet Frames, as depicted in Figure 14. In the case of ATM/HDLC/FR/PPP emulated services over a packet network, the data is received at the IWF already in packet form (e.g., ATM cell units). These have less stringent timing requirements than the TDM CES service and they do not require service clock regeneration. The use of idle line conditions and packet buffering can compensate for short term differences in network ingress and egress clocks.

Figure 14. ATM interface retiming

PRC

Idle ll cells Packet network

Idle ll cells

Digital switch

E1 ATM

E1 ATM

Digital switch

End customer clock used to time E1 toward network

Network time reference used to time E1 out of network

6.6 Service clock vs. retiming There are two common options for the source of the clock used to transmit the data out of the packet network. These are either a regenerated copy of the clock used to inject data into the network (service clock) or an independent clock available at the network egress node (retiming). It can be acceptable to use retiming when it is known that the originating clock and the retiming clock are within well defined performance margins of each other. For PDH interfaces, this occurs if both clocks have been designed using the clock distribution guidelines defined by the ITU-T and Bellcore standards bodies for telecommunications deployments. Generally, this means that both clocks are traceable back to primary reference clocks via a chain of second- and third-tier clocks with constrained noise. This ensures that, over the long term, the frequencies are very close4 and that the maximum wander that can be accumulated is known so that temporary buffers used at the retiming point can be properly sized.
4

 he worst-case frequency difference between two PRCs is 0.02 parts per billion. This leads to an accumulated phase difference of T 125 us after 72 days.

Synchronization over Ethernet Networks | Technology White Paper

17

In the case of CES IWF, the buffering requirements from the retiming are on the order of tens of microseconds and are easily covered by the normal millisecond range buffering capability built into most CES services at network egress. 6.7 Oscillator Stability and PLL Most clock recovery circuitry makes use of Phase-Locked Loop (PLLs). In a PLL, there is either a Voltage Controlled Oscillator (VCO) or a Digitally Controlled Oscillator (DCO) that is adjusting a reference provided by a crystal oscillator in order to ensure that the phase error between the adjusted frequency and the reference frequency is minimized. This technology is impacted by both the stability of the input reference and the stability of the frequency output by the crystal oscillator. If the crystal oscillator is of high quality, the DPLL can be configured to trust the local oscillator more and adjust slowly to differences seen with the reference frequency. If the crystal oscillator is lower quality, then the DPLL should not rely on it and should adjust more quickly to differences seen with the reference frequency. The key parameter for this quality designation is the oscillators frequency stability. When dealing with ToP solutions where the short term stability of the reference frequency will be questionable due to the level of PDV, it is preferable to use a highly stable oscillator to allow the DPLL to filter out short term variations in the reference frequency and look at the longer term average frequency. In general, an Oven-Controlled Crystal Oscillator (OCXO) is suggested for these solutions. 6.8 Performance Metrics When analyzing signals for their timing accuracy and stability, the preferred metric is the Time Error (TE). This metric is a measurement of the time between the occurrence of a significant event in a reference signal and the occurrence of the same significant event in a test signal. This significant event is usually declared when the signal crosses a threshold in one direction. For example, it could be when a sinusoidal signal transitions from positive to negative values or when a TTL signal crosses +100mV in the positive direction. Figure 15 shows two E1 signals where the significant event is the crossing of a threshold set at half the height of the positive excursion of an E1 signal. As can be seen, this crossing does not occur at exactly the same time for both signals. The difference is the TE. When an entire series of Time Errors is collected, then they may be processed into several other metrics including the Maximum Time Interval Error (MTIE), Fractional Frequency Offset (FFO), Time Deviation (TDEV), Allan Variance (ADEV), and many more. The most common metric discussed in relation to the performance requirements is the MTIE. This metric uses the Time Interval Error, which is simply the difference in Time Error for a given interval and then the MTIE is the worst-case TE for any interval within a fixed observation window at any point in the measurement time (see Figure 16). MTIE limits were defined to provide performance targets for individual network nodes and also network spans to ensure that slip buffers at international gateways had controlled slip rates.

Figure 15. Time error between a reference and a test signal

Ref Threshold

Signal

Threshold

18

Synchronization over Ethernet Networks | Technology White Paper

Figure 16. TIE, MTIE and observation time

Time delay with respect to reference

MTIE(S)

TIE(S)

Observation time Measurement time

Time

6.9 Mobile base-station frequency reference All mobile technologies have a requirement to ensure that their radio frequency transmissions are within a given range of the true frequencies. For the most part, this accuracy requirement is listed as 50 parts per billion. What this means is that when the base-station is transmitting a carrier at 900 MHz, then it must be somewhere between 899,999,955 Hz and 900,000,045 Hz when compared to an atomic standard. The requirement stems from the fact that in a mobile deployment, there will be relative movement between the handsets and the cell towers. The Doppler Effect causes shifting to the frequency as perceived by the receiver. The developers of mobile technologies had to define a worst-case frequency error based on expected deployments. They identified that most of this error would come from the Doppler Effect but some of this error would also come from the base-stations frequency accuracy. Table 2 shows the frequency offsets that were defined and how they were budgeted between the Doppler Effect and originating frequency accuracy.

Table 2. Frequency accuracy budget for mobile systems

Type Mobile must tolerate offset GSM 900 MHz 295 Hz GSM 1800 MHz 340 Hz WCDMA 2100 MHz 591 Hz /A

Macro

Velocity and Doppler Base-station accuracy (0.05ppm)

250 km/h and 250 Hz 45 Hz 205 km/h and 205 Hz 90 Hz

130 km/h and 250 Hz 90 Hz 80 km/h and 160 Hz 80 Hz

250 km/h and 486 Hz 105 Hz 196 km/h and 381 Hz 210 Hz

Femto

Velocity and Doppler Base-station accuracy (0.1ppm)

Traditionally, base-stations achieve this frequency accuracy with one of three options: a highly accurate free-running oscillator that is calibrated on a periodic basis (several months) a local oscillator tuned to an input from a GPS receiver a local oscillator tuned to an E1 or T1 input

Synchronization over Ethernet Networks | Technology White Paper

19

The GPS option is the only viable option for mobile technologies using TDD since, in addition to the frequency accuracy, there is a phase accuracy requirement that cannot be met using the E1/T1s. For FDD systems, it is much more common for the backhaul E1/T1s to be used. These interfaces must have timing that: Is traceable back to a PRC/PRS to ensure long term frequency accuracy of 0.01 ppb Has a known limit to the amount of jitter and wander in relation to the PRC/PRS With these two requirements met, the base-station clock regenerator can filter the signal presented on the T1 or E1 interface in order to meet the requisite targets. The length of time for this filtering depends primarily on the quality of the oscillator used within the base-station clock regenerator. If a highly stable oscillator is incorporated into the base-station clock, then an E1/T1 that only meets the limits of a Traffic Interface as defined in G.823/824 can be used at the input to the base-station. If a more cost-effective base-station clock design is used, then the E1/T1 must be more stable and must meet the limits of one of the Synchronization Interfaces defined in G.823/824. The 3GPP specification does not specify which of the Traffic or Synchronization Interface limits applies; consequently, both types of base-stations exist in practice. Table 3 shows the length of time a given signal needs to be filtered to ensure it meets the frequency accuracy requirements. The 50 ppb target is often stated as a target but is really the target for the output of the base-station clock. In order to meet this on its output, the input may have to be more accurate. A common conversion is to target 16 ppb to allow for noise in the clock circuitry. The real requirement for a given base-station will be somewhere in between these two values.

Table 3. Filtering time of signal to meet frequency accuracy requirements

T1/E1 stability Filter time to ensure 50 ppb Filter time to ensure 16 ppb

G.823 Traffic Interface G.8261 Deployment Case 1 2.048 MHz G.823 SEC Synchronization Interface G,824 Traffic Interface G.8261 Deployment Case 1 1.544 MHz G.824 Synchronization Interface

350 s 85 s 40 s 170 s 42 s 5.2 s

1150 s 280 s 130 s 510 s 134 s 23 s

6.10 Special note on Synchronous Ethernet over copper There is a specific challenge when looking at Synchronous Ethernet 1000BASE-T over copper interfaces (e.g. 1000BASE-T). On copper-based Ethernet, due to the method used to transfer information in both directions on a single wire, the transmit timing from each side needs to be associated. This isaccomplished on initial power up through a negotiation procedure where one side of the link is declared the master for timing and the other side becomes the slave and uses loop timing derived from the master side. A Synchronous Ethernet interface should be capable of relaying the timing from a nodes central clock out of any synchronous Ethernet interface. This would be a problem if a copper-based Ethernet interface had negotiated to be running in slave mode. Today renegotiation of the Ethernet timing generally causes a data transmission hit and thus would be unacceptable. Since copper-based Ethernet is generally only intra-facility, it is easier to rely on an existing BITS intra-facility timing distribution system or to just switch interfaces to optical fiber when switching to Synchronous Ethernet.

20

Synchronization over Ethernet Networks | Technology White Paper

7. Conclusions
Synchronization is as fundamental to network operation as power and grounding. It is a basic requirement in a network of any scope so that disparate elements may work co-operatively to correctly transfer traffic between user equipment. Nowhere is this truer than in a mobile (wireless) network. A lack of accurate synchronization can negatively impact the user quality of experience via failed handovers, dropped calls and data retransmissions. In turn, this can lead to reduced customer loyalty or increased customer churn the bane of any operator. Traditionally, synchronization has been distributed over the SDH/SONET network and via T1/E1 connections to remote cell sites and also via satellite systems. These techniques will persist in some cases. With the advent of the massive transformation of many carrier networks to cost-efficient, scalable and service-rich All-IP network architectures over Ethernet media, new techniques for timing distribution have come to the fore. Techniques (such as Synchronous Ethernet) and ToP capabilities (such as Adaptive Clock Recovery and IEEE1588v2) can be used to ensure reliable network operation through this transformation. Alcatel-Lucent has been engaged in the research and engineering of synchronization solutions for several decades with deep research and development capabilities and considerable practical deployment experience. In many IP transformation projects worldwide, Alcatel-Lucent is bringing a powerful combination of strong network and management products, comprehensive consulting services, and synchronization knowledge to facilitate successful project completion.

8. Abbreviations
ACR ATM BITS BSC BTS CES DCR DSL FDD FLL GPON GPS GSM GSM IP IWF LAN MAFE MPLS MTIE MTSO NTP NTR Adaptive Clock Recovery Asynchronous Transfer Mode Building Integrated Timing Supply Base-Station Controller Base Transceiver Station Circuit Emulation Service Differential Clock Recovery Digital Subscriber Line Frequency Division Duplexing Frequency-locked Loop Gigabit Passive Optical Network Global Positioning System Global System for Mobile communications Global System for Mobile communications Railway Internet Protocol Interworking Function Local-area network maximum average frequency error Multiprotocol Label Switching Maximum Time Interval Error (see G.810) Mobile Telephone Switching Office Network Time Protocol Network Timing Reference PDV PLL POS PPB PRC PSN PTD PW QoS RAN RNC RTP SDH SONET SRTS TCXO TDEV TDD TDM ToD ToP UMTS UTC WAN WCDMA WDM Packet Delay Variation Phase-locked Loop Packet over SONET Parts Per Billion Primary Reference Clock Packet Switched Network Packet Transfer Delay Pseudo-wire Quality of Service Radio Access Network Radio Network Controller Real time Protocol Synchronous Digital Hierarchy Synchronous Optical Networks Synchronous Residual Time Stamp Temperature Compensated Crystal Oscillator Time Deviation Time Division Duplexing Time-Division Multiplexing Time of Day Timing Over Packet Universal Mobile Telecommunications System Coordinated Universal Time (French acronym) Wide-area Network Wideband Cell Division Multiple Access Wave Division Multiplexing

OAM&P  Operations, Administration, Maintenance & Provisioning OCXO PDH Oven Controlled Crystal Oscillator Plesiochronous Digital Hierarchy

Synchronization over Ethernet Networks | Technology White Paper

21

www.alcatel-lucent.com

Alcatel, Lucent, Alcatel-Lucent and the Alcatel-Lucent logo are trademarks of Alcatel-Lucent. All other trademarks are the property of their respective owners. The information presented is subject to change without notice. Alcatel-Lucent assumes no responsibility for inaccuracies contained herein. Copyright 2009 Alcatel-Lucent. All rights reserved. CPG2896090291 (07)