Timing and Synchronization for Quasi-Real-Time Systems Using IEEE 1588v2 Over Ethernet

Marc Cohn
Micrel, Incorporated www.micrel.com LAN Solutions Business Unit San Jose, California, USA marc.cohn@micrel.com
Abstract Ethernet and IEEE 1588 are continuing to emerge in a wide range of industries. Increasingly, they are replacing industry-specific interconnects, delivering performance and cost efficiencies. Initiatives including Industrial Ethernet, IEC 61850, and LXI are all adopting commercial off- the-shelf technologies to achieve broader communications at far lower costs. However, significant investments that are warranted for large automation systems become prohibitively expensive for a diverse set of less, ambitious Quasi-Real-Time (QRT) applications that also require distributed communications and timing. This paper proposes an approach to address the communications and timing/synchronization requirements for QRT systems, characterized by simple and low-cost devices interconnected and synchronized over a real-time network. To achieve the aggressive cost targets, a highly integrated attachment device is introduced that integrates Ethernet communications, IEEE 1588v2 distributed synchronization, and precision I/O for local synchronization in a single, energy efficient device. Systems issues are also raised affecting the applicability for applications that can exploit a QRT network. Keywords: IEEE 1588v2, Real-Time Ethernet, Precision I/O, Synchronized I/O, Industrial Ethernet, Quasi-Real-Time

Industrial Ethernet standards such as Ethernet/IP [2], Profinet [3], and PowerLink [4] seeking to capitalize upon the benefits of Commercial Off-The-Shelf (COTS) technologies, are aggressively deploying Ethernet integrated with IEEE 1588. Similarly, the power systems automation industry has specified an important standard to guide Power Substation Automation. IEC 61850 [5] breaks new ground in adopting Ethernet, IEEE 1588, and responsive fault tolerance schemes for a comprehensive communications framework for Substation Automation Systems (SAS). Industrial and Power Systems Automation market leaders are aggressively promoting Industrial Ethernet standards, as well as IEC 61850. However, significant up-front investment and comprehensive software stacks render the standards too expensive for many applications. Ultimately, there are many lower-end applications that would benefit from reliable communications and tight synchronization, but without the rich set of services and objects defined by the standard frameworks. II. QUASI-REAL-TIME (QRT) NETWORKS

I.

INTRODUCTION

Ethernet, the predominant Local Area Network standard, is emerging in a number of non-traditional applications, leveraging the tremendous momentum of an installed base in excess of one billion ports. Economic realities have motivated the adoption of Ethernet in such diverse environments ranging from the factory floor, in-vehicle, to electrical power substations. While all very different, such applications share an unmistakable and pervasive trend: point-to-point interconnects and industry-standard buses are being replaced by robust, Ethernet-based networks. Increasing intelligence, counter-balanced by the everpresent need for cost reduction, has necessitated integration of communications and timing/synchronization, germane to Industrial, Automotive, and Power Systems automation. Market and technology forces influenced the IEEE to upgrade a relatively obscure standard into prominence. Thus, IEEE 1588-2008 [1] is gaining market momentum as the basis for next-generation real-time automation systems and, in the process, is transforming the target industries.

To distinguish from the large, hierarchical automation networks, we introduce QRT systems and networks, which correspond to a broad set of relatively mid- to low-end applications built upon an integrated services network. By merging timing and communications into a unified, low-cost network, applications processing may be distributed, communications and synchronization jitter reduced, and greater precision may be achieved [6]. QRT systems are characterized by: Standardized, high-performance network architecture providing communications and synchronization Relatively low-end devices (~$100s - $1,000s): Sensors, Actuators, Motor Drives, etc. Embedded CPUs, running the host application Relatively small number of nodes per sub-network (typically less than 30) Message latencies of less than 1 ms (short, <200 byte messages) Sub-100 s synchronization jitters (typically better)

978-1-61284-893-8/11/$26.00 2011 IEEE

As with most successful technologies, availability of lowcost, standardized building blocks spawn a diverse and broad set of applications. The convergence of Ethernet and IEEE 1588 has paved the way for precision timing to catalyze development of QRT systems; by capitalizing upon the benefits of COTS, proven communications and synchronization technologies, and of course, reduction of life-cycle costs. The key to enabling such applications is availability of highly integrated silicon that simplifies QRT node designs, and adapt to the widely varying interface requirements, including: Comprehensive support for predominant standards including IEEE 802.3 (Ethernet) and IEEE 1588/PTPv2 distributed timing and synchronization Multi-megabit communications (typically 100 Mbps) Standardized copper and fiber media options to accommodate varying cable reaches Flexible topologies: o o Centralized, star-wired topologies to exploit the benefits of structured wiring Distributed, daisy-chained topologies (i.e., rings and linear buses) to facilitate dynamically changing network configurations Hybrid topologies, exploiting the benefits of both Centralized and Distributed topologies

Systems performance is driven by a complex combination of these factors, making it difficult to apply generic rules of thumb for QRT network configuration. Communications latency, synchronization jitter, ring recovery delays, etc. are all inter-related, c the need to meticulously analyze the applications (with respect to the network), in order to achieve the desired systems behavior. For instance, decreasing ring recovery times may not yield appreciable benefits if the synchronization acquisition time is orders of magnitude higher. Increasing network bandwidth will not necessarily address overcome the issue, nor will increasing the processing speed necessarily improve network availability. The challenge is architecting a QRT network architecture sufficiently broad to encompass a range of applications at a very low cost. Systems designers may then exploit standardized building blocks that attain the best of both worlds- the highperformance only achievable by silicon, at a cost-effective price. One prospective application for QRT networks is the emerging IEC 61850 standard for Power Substation Automation. IEC 61850 defines a hierarchical communications architecture, based on Ethernet and IEEE 15888 technology. The lowest layer is referred to as the Process Bus, which interconnects Intelligent Electronic Devices (IED, as defined in the standard) that provides real-time communications. Relays, switches, and other electrical power gear are controlled by the Process Bus, which is considered missioncritical by substation operators. The standard recognized the need for a multi-service, scalable, robust, and highperformance network to achieve the explicit goal of reducing overall operations costs- a key benefit of the QRT networks. III. COST REALITIES- INTEGRATED ETHERNET/IEEE 1588V2 NETWORKS

Optional, network fault tolerance mechanisms to maintain communications availability Support for distributed synchronization, which may be extended to locally connected devices Hardware assist for communications and synchronization to minimize the processing demands on the embedded processors Rich set of I/O capabilities to accommodate a range of devices with varying operating characteristics

There are a number of complementary technologies that are also needed to enable QRT systems/network design: Real-Time, Systems Timing References, referred to in IEEE 1588 as Grand Master Clocks; highly accurate timing standards for synchronizing distributed nodes Robust Ring Recovery protocols capable reconfiguring around common network faults, typically < 10 milliseconds Streamlined Real-Time Operating Systems (RTOS), to ensure efficient communications, synchronization, and applications processing Network management to enable the availability, performance, and configuration of the network to be monitored for real-time and long-term analysis

Migrating from relatively simple serial interconnects and buses to an integrated Ethernet/IEEE 1588v2 network in a costeffective manner can be challenging. Considering the cost difference between an RS-485 interface and Ethernet port, it remains cost-prohibitive to directly connect low-end devices to Ethernet networks. Typically, the Ethernet MAC and PHY functions are implemented in silicon in accordance with the IEEE 802.3 standard [7]. Ethernet LAN are commercially available from multiple vendors. IEEE 1588v2 implementations are far less common, and are typically implemented in FPGAs, Physical Layer (PHY) transceivers, or Micro-Controller Units (MCUs). Integrating Ethernet switching along with IEEE 1588v2 (and the associated logic) is not a trivial exercise, and requires additional communications and synchronization processing (in the host CPU). Ethernet/IEEE 1588 attachments will multiplex multiple devices (through I/O pins) in order to achieve cost parity with point-to-point systems (see Figure 1). Migrating to a network offers greater flexibility and a compelling range of benefits.

Figure 1. Per-Device Cost Comparison Serial Interconnect vs. Serial Bus vs. Ethernet Network

Networks enable a rich set of communications services such as multicast, Virtual LANs, and Quality of Service that offer unprecedented applications flexibility. Integrating IEEE 1588 realizes sub-s synchronization of nodes distributed throughout the network cost-effective and reliable. Far greater bandwidths, often enumerated in orders of magnitude, are readily achievable with the well-established 10/100 Ethernet standards. In addition, cabling can be simplified as well, to accommodate frequently changing operational needs. The costs for an Ethernet/1588v2 attachment (assuming a distributed topology), include: IEEE 1588v2 Facilities o o o IEEE 1588v2 Precision Clock 2 Time Stamp Units (TSUs) for PTP packets PTP hardware support 2 integrated Media Access Control (MAC) 2 10/100BaseTX PHY Transceivers

Figure 2. KSZ84xx IEEE 3-port 10/100 / IEEE 1588v2 Switch Functional Block Diagram

Wire-speed, full-featured 3-port 10/100 Mbps switch Dual IEEE 1588v2 time stamp units, Precision Clock, and distributed synchronization facility o o Synchronization performance is improved as TSUs reside between the MAC and PHY Grand Master, Master, Slave, and Transparent Clock modes are supported Lowest power 100BaseTX PHY Transceiver (< 150 mW per port) IEEE 802.3FX optical transceiver support

Dual, low-power 10BaseT / 100BaseTX PHYs o o

3-port 10/100 Ethernet switch o o

Precision I/O (synchronized to the Precision Clock) PTP Software (typically running in the Host CPU)

Hardware support for communications and precision clock synchronization, which reduces the overall processing load on the host CPU Integrated I/O that can be synchronized to the overall system synchronization hierarchy Advanced power management including IEEE 802.3az Energy Efficient Ethernet (EEE) [9] Compact size through a single-chip (64-pin package, 10 mm x 10 mm) design

Cost challenges persist. Presently, an Ethernet/1588v2 attachment implemented in an FPGA, with a pair of external Ethernet PHY Transceivers, with an estimated cost measured in $20-30 USD or more. Assuming four to eight devices share a single Ethernet/1588 attachment, the per-device costs are on the order of $3 $5; multiples of the sub $1 USD cost for an RS-485 interface transceiver. IV. INTEGRATION OPPORTUNITIES

In order to reduce the Ethernet/IEEE 1588 attachment costs so as to be on par with serial interconnects, further integration is necessary. This is primarily achievable through an ASIC. As a result, Micrel Semiconductor recently introduced the KSZ84xx family of IEEE 1588v2-enabled Ethernet 3-port 10/100 switches [8], for QRT applications. This Industrialgrade platform represents a single chip, highly integrated 1588/Ethernet attachment, offering several tangible benefits:

Figure 2 depicts the high-level block diagram for the KSZ84xx switch, illustrating the functional blocks. The KSZ84xx is available with standard (MII or RMII) and generic host bus interfaces to support CPUs with and without embedded Ethernet MACs. The Precision GPIO facility enables multiple devices to share a single Ethernet/IEEE 1588 attachment. Precision GPIO is highly flexible and configurable to support a diverse set of devices with wide ranging operational and performance characteristics.

Figure 3. IEEE 1588v2 Systems Timing Hierarchy Figure 4. QRT Node Timing Interfaces

V.

PRECISION I/O AND LOCAL SYNCHRONIZATION

Devices connected to the QRT network through the Precision GPIO facility must be synchronized to the node, and hence system reference timing. Figure 3 illustrates the system timing hierarchy implied by IEEE 1588v2, where: A global, real-time reference (i.e., GPS) synchronizes a Grand Master Clock (GMC), the system reference Multiple Master Clocks (MCs), each corresponding to a distinct synchronization domain, are synchronized to the GMC Each MC synchronizes the set of Slave Clocks (SCs) comprising a particular synchronization domain. For Industrial Ethernet, SCs are typically daisy-chained together into a distributed topology. Locally connected nodes can source or sink data and/or control information through a Precision I/O interface that is synchronized with the local precision clock

A select set of applications may require synchronization performance far better than specified in the IEEE 1588-2008 standard. Section 1 of the standard specfies 1 s Jitter (variability) value. However, the IEEE Precise Networked Clock Synchronization Working Group recognized the need for sub-s performance for a range of specialized applications, which is explicitly called out in the standard. In the Industrial Automation arena, defacto standards such as EtherCat [11], and ProfinetIRT [12] have emerged to fill the need for the most stringent real-time applications. At 100 Mbps, these technology enable real-time system cycle times below 500 s; at 1 Gbps < 500 s is feasible [13]. Synchronization accuracies are measured in 10s of ns. EtherCat and ProfinetIRT use proprietary approaches that preclude the use of standard Ethernet MACs (even though both adopted the Ethenet 10/100/1000 Mbps PHY layer). For higher-end applications, IEEE 1588v2 over Ethernet implementations may be enhanced by proactively addressing the error sources that deteriorate synchronization performance. Higher accuracy oscillators, enhanced power filtering, etc. are likely to result in synchronization performance well-below 100ns. While the synchronization performance may be on par with the real-time protocols, overall systems performance for EtherCat and ProfiNet IRT will be more suitable for precision motion, and other real-time systems which require the highest performance. VI. LOCAL TIMING & I/O IMPLEMENTATION

The distinction is drawn between timing, where real-time is distributed, and synchronization, which relates to a frequency reference distribution across multiple, disparate equipment. Synchronization may be provided through standardized timing interfaces (see Figure 4): 1 PPS reference for time distribution 10 MHz reference for synchronized clock distribution

Because timing and synchronization is application-specific, versatility is required to tailor the timing interfaces to the needs for specific designs. Additional reference signals (such as 100 PPS timing output, 1 MHz and 5 MHz, etc.) may be necessary. Also, signal characteristics may also vary (e.g., pulse width, voltage level, etc.), necessitating configurability, without compromising synchronization in accordance with the systems timing hierarchy. Real-time distribution may be achieved through such standards as IRIG-B, published by the Inter-Range Instrumentation Group. The IRIG standard [10], defines internationally recognized Time Code Formats and sentences that reduce the need for real-time references. Real-time distribution is being considered for future versions of the KSZ84xx platform.

Micrels KSZ84xx Industrial Ethernet switch platform is an ASIC specifically designed to enable QRT networks. Figure 5 depicts how multiple devices share a single Ethernet/IEEE 1588 attachment through the KSZ84xx Precision I/O facility. Precision I/O pins are synchronized with the local precision clock, which in turn is synchronized to the system timing hierarchy. In-house testing yields synchronization jitters on the order of 100 ns or less (see Figure 6), even under 99%+ network loads. Such performance is attained through positioning the time stamp units, directly between the PHY and the MAC (on-chip). As a result, measurement and time stamp errors are significantly reduced at the node-level.

Figure 5. Multiple Devices Share a Common QRT Network Attachment to Attain Cost Parity with Serial Interconnects

Figure 6. Sub-100 ns Synchronization Jitter (Master to Slave Clocks)

Most available 1588 implementations partition the MACto-PHY interface on two distinct devices, compromising synchronization performance. Multi-chip implementations also prove more costly, consume higher power, require additional board space, and are typically less reliable than an ASIC. Another factor yielding improved synchronization performance is the tightly coupled Precision Timing Protocol (PTP) stack, developed by OnTime Networks, Oslo, Norway. Hardware-assisted PTP operations (especially periodic Transparent Clock corrections) are not only more efficient, but more importantly, conserve scarce Host CPU resources which can otherwise be dedicated to applications processing. In order to support diverse interface needs anticipated for QRT systems, a broad set of I/O capabilities has been provided, as summarized in Table I. The I/O implementation is based on a set of logical I/O Event Units that may be assigned to any of the available I/O pins. By decoupling Event Units from physical pins, more sophisticated I/O operations are possible. Input Event Units provide an efficient means of monitoring external events and conditions on I/O input pins. For each Input Event, a time stamp is generated whether for an Edge or Pulse sensed on that pin. Input Event Units may be cascaded together to monitor compound events (i.e., sequences of multiple pulses, edges, and in essence bit patterns), offloading the Host CPU from low-level I/O manipulation. Once detected, the time stamp corresponding to each received event will be queued up for the host CPU to retrieve. Such an approach decouples host processing from the I/O acquisition, further offloading the Host CPU. Improving host efficiency is especially significant for QRT systems where multiple events may occur virtually simultaneously. Upon retrieving each Input Events, the Host CPU must analyze the time stamps to determine pulse widths, the time intervals when the events occurred, etc. For Output Events, a flexible range of I/O Output Units and associated operations are supported. Each Output Event will be initiated (i.e., triggered) when the local precision clock reaches a pre-configured value corresponding to the event. Thus, all Output Events are timed in concert with the local clock, and in turn the overall systems timing hierarchy.

GPIO Output performance is governed by the Host CPU selection, host interface (MII, RMII, or Generic Host Interface), and complexity of the designated output operations. For each operation, the Host must configure the appropriate registers (through the management interface), and set the appropriate trigger time.

I/O Capability

Applications

Configurability

Notes

Inputs Edge Monitor Pulse Monitor Events Alarms Interrupt Control word sensing Timing (n x PPS) Positive or Negative Polarity Polarity Pulse Width May be cascaded for more complicated bit patterns May be used as a 1 PPS input for local timing by an external system

Outputs Edge Generator Pulse Generator Positive or Negative Polarity Polarity Pulse Width May be cascaded for more complicated bit patterns May be used as a 1 PPS input for local timing by an external system

Waveform Generator

t x PPS Timing signal

Frequency Generator Register Mode

F MHz Reference Control Word outputs

Start time Cycle time # of Cycles Pulse Width Frequency, Duty Cycle Start time, Repetitions

1 Hz 12.5 MHz

Shifts out contents of a designated output register

TABLE I.

KSZ84xx Precision I/O Mechanisms

Similar to the Input Event Units, Output Event Units may be cascaded to enable more complex operations. For example, if a Logic Controller is required to send a 32-bit control word to request a Sensor reading, 2 Register Mode Output Events may be cascaded together. Once triggered (at the prescribed time), both of the Register Mode Output Units will sequentially shift out the 16-bit contents of user-data (contained in an internal register) onto a single I/O Output pin. In addition to the relatively simple Edge, Pulse, and Register modes, timing interfaces may be implemented using the highly configurable Output Trigger Unit, such as the common 1 PPS signal that distributes precision timing. The 1 PPS Output may be configured to comply with the GPS Interface Control Document (ICD) for the Precision Time and Time Interval (PTTI) Interface [14], or an alternative specification. The Waveform Generator Output Unit may be configured to address a number of applications including: N x PPS Output Interval (where N = 1 for 1 PPS) Start Time Pulse Width Number of pulses (if not continuous)

REFERENCES
[1] Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems, IEEE Standard 1588, 2008. ODVA, Ethernet/IP Technology Overview, http://www.odva.org/ PI, Profinet- A Rich, Powerful, Ethernet Solution for Automation, http://www.profibus.com/technology/profinet/ EPSG, Ethernet PowerLink, http://www.ethernet-powerlink.org/ Communication networks and systems in substations - ALL PARTS, IEC 61850, 2011. Ken Harris (January, 2009), An Application of IEEE 1588 to Industrial Automation, Rockwell-Automation Publication: 1756-WP005-EN-E Available: http://samplecode.rockwellautomation.com/idc/groups/literature/docume nts/wp/1756-wp005_-en-e.pdf IEEE Standard for Information technology - Telecommunications and information exchange between systems - Local and metropolitan area networks - Specific requirements Part 3: Carrier Sense Multiple Access with Collision Detection (CSMA/CD) Access Method., IEEE 802.3., 2008. Micrel Inc., KSZ84xx Data Sheet, 2011 IEEE 802.3az-2010 IEEE Standard for Local and Metropolitan Area Networks - Specific requirements Part 3: Carrier Sense Multiple Access with Collision Detection (CSMA/CD) Access Method and Physical Layer Specifications Amendment 5: Media Access Control Parameters, Physical Layers, and Management Parameters for Energy-Efficient Ethernet Inter-Range Instrumentation Group SERIAL TIME CODE FORMATS, IRIG Standard 200-98, Telecommunications and Timing Group of the Range Commanders Council, 1998. EtherCat Technology Group, EtherCat- Ethernet for Control Automation Technology, http://www.ethercat.org PI, Profinet- A Rich, Powerful, Ethernet Solution for Automation, http://www.profibus.com/technology/profinet/ Gunnar Prytz (2008), 13th IEEE International Conference on Emerging Technologies and Factory Automation, pp. 408-415 Available: http://www.ethercat.org/pdf/english/ETFA_2008_EtherCAT_vs_PROFI NET_IRT.pdf GPS User Equipment III ICD for the Precision Time and Time Interval (PTTI) Interface, GPS ICD-GPS-060, GPS Navstar Joint Program Office (US Govt.), 2002.

[2] [3] [4] [5] [6]

[7]

[8] [9]

[10]

Multiple timing signals may be initiated with different rate, which provides a low cost, yet accurate timing reference. High-frequency reference signals may also be generated, using the Frequency Generator Output Unit. Frequency Outputs may be configured from 1 Hz to 12.5 MHz, on one or more I/O output pins. While the frequency is synchronized, phase is not; the output may be used for syntonization. VII. CONCLUSIONS Quasi-Real-Time systems, characterized by low-end devices interconnected over a distributed Ethernet/IEEE 1588v2 network in timing-based systems, are made viable through availability of highly integrated, silicon-based implementations. Cost-effective, energy efficient network attachment implementations enable a range of distributed QRT monitoring, control, and automation applications that can exploit unprecedented communications and synchronization performance while reducing life cycle costs. While QRT network performance remains coupled closely with the applications, high-performance silicon such as Micrels KSZ84xx family renders IEEE 1588 over Ethernet as a viable option for a range of diverse applications that were previously cost-prohibitive. As costs are further reduced, QRT will make it feasible to continue to migrate down-market, replacing point-to-point interconnects.

[11] [12] [13]

[14]