Teleprotection over an IP/MPLS Network

Technical validation

ir. Lieven LEVRAU Alcatel-Lucent IP Division

April 4th, 2011

IP/MPLS-based Utility Networks Challenges for IP/MPLS in teleprotection Other design considerations Conclusion

Alcatel-Lucent 2011 All Rights Reserved

IP/MPLS-based Utility Networks

All Rights Reserved Alcatel-Lucent 2011

IP/MPLS-Based Utility Network

Power Generation
7705 SAR

Corporate HQ / NOC
OS6855 LAN IED Internet Omni PCX 5620 SAM SCADA

7750 SR

7750 SR

IP/MPLS Network
7750 SR 7705 SAR

Servers Collaboration Tool

NOC

Substation

Substation

Transmission

7750 SR

7705 SAR

7705 SAR IED OS6855

7705 SAR RTU LAN TPR

Video Surveillance

TPR

LAN

TDM

Video Surveillance

4 | MPLS Network for Teleprotection | Sep 2009

All Rights Reserved Alcatel-Lucent 2011

How IP/MPLS Address the challenges of the Utility Networks?

Reliability Traffic Isolation Multiple Legacy technologies Security Unified management Quality of Service

High Availability, Fast Reroute, end-to-end, IP/MPLS based - L2-VPNs, VLL, L3-VPNs, PWE FR, ATM, TDM, Serial, analog voice, synchronous, Traffic isolation, Rtng protocols authentication, L2/L3 Encryption + dynamic secure Key distribution, DoS End-to-end secure management of network and services Guaranteed bandwidth, low latency and jitter ` Enhanced QoS on different network layers

All Rights Reserved Alcatel-Lucent 2011

Challenges for IP/MPLS in teleprotection

All Rights Reserved Alcatel-Lucent 2011

Using MPLS Network for Teleprotection

Substation Substation

kV

G.703
MUX

TPR

t E&M RS-232 Ethernet IP/MPLS

TPR

G.703
MUX

E1/T1

E&M RS-232 Ethernet 7710 SR 7750 SR 7705 SAR

E1/T1

7705 SAR

Teleprotection relay (TPR) signals must be transferred reliably and fast with low latency End to end delay = telecom network latency + teleprotection equipment delay TDM over MPLS for legacy support (requires integration of legacy interface in MPLS node to limit and control end to end delay) VPLS for IEC 61850 (requires high reliable Layer 2 transport)

7 | MPLS Network for Teleprotection | Sep 2009

All Rights Reserved Alcatel-Lucent 2011

Main Challenges for Teleprotection

End to End Delay Jitter Asymmetry Resiliency Denial of Services Synchronization Quality of Service

Depending on vendor equipment May include packetisation/depacketisation Variation of delay in certain circumstances such as traffic burst on network Delay variations between transmit and receive The impact of a failure in the network shouldnt be noticed by application How DoS/DDoS can affect applications TDM applications need end to end synchronization Guaranteed bandwidth, low latency and jitter Enhanced QoS on different network layers

All Rights Reserved Alcatel-Lucent 2011

End-to-End TDM Transport Model (only left-to-right direction shown)

TDM Packets moving in this direction

DS1 Access Circuit

DS1 LIU

Data

Packetization
Si g

GigE

Packet Switched Network (PSN)

GigE

Jitter Data Buffer

Si g

DS1 LIU

DS1 Access Circuit

Packetization As TDM traffic from the Access Circuit (AC) is received, it is packetized and transmitted into the PSN Two modes of operation:
CESoPSN (RFC5086) for structured nxDS0/64k channels SAToP (RFC4553) for unstructured T1

Network Fixed delay

Packet transfer delay based on link speeds and distances from end to end

Playout TDM PW packets are received from the PSN and stored into its associated configurable jitter buffer Play-out of the TDM data back into the AC when its at least 50% full

Variable delay
the number of and type of switches queuing point in the switches

QoS is key to ensure effective service delivery

9 | MPLS Network for Teleprotection | Sep 2009

All Rights Reserved Alcatel-Lucent 2011

Latency
Using MPLS Network for Teleprotection
kV
Substation Substation

G.703
MUX

TPR

t E&M RS-232 Ethernet IP/MPLS

TPR

G.703
MUX

E1/T1

E&M RS-232 Ethernet 7750 SR 7750 SR 7705 SAR

E1/T1

7705 SAR

TPR relay signals must be transferred in < t T = 1 cycle at 50Hz or 60Hz = 20ms or 16ms End to end delay = telecom network latency + teleprotection equipment time latency = packetization delay + network delay + jitter buffer delay

The total end-to-end latency is calculated by summing the packetization delay (PD), network delay (ND) and jitter buffer delay (JBD) as shown here:
Total Latency = PD + ND + JBD
e.g. PD of 2 ms (16 T1 frames/packet), ND of 3 ms, JBD of 4 ms Total Latency = 2 + 3 + 4 = 9 ms

10 | Teleprotection over IP/MPLS - Validation - March2011

All Rights Reserved Alcatel-Lucent 2011

Using MPLS Network for Teleprotection

Latency

kV
Substation Substation

G.703
MUX

TPR

t E&M RS-232 Ethernet IP/MPLS

TPR

G.703
MUX

E1/T1

E&M RS-232 Ethernet 7750 SR 7750 SR 7705 SAR

E1/T1

7705 SAR

TPR relay signals must be transferred in < t T = 1 cycle at 50Hz or 60Hz = 20ms or 16ms End to end delay = telecom network latency + teleprotection equipment time

Telecom network latency = packetization delay + network delay + jitter buffer delay

11 | Teleprotection over IP/MPLS - Validation - March2011

All Rights Reserved Alcatel-Lucent 2011

Calculation of Latency for Teleprotection

Latency

Latency is mainly at the edge where low speeds are present (Serial / E1 / 100 FX) Latency in the core depends on number of nodes but mainly transmission delays Each node adds a maximum of :
150 s (eqpt latency) 10 s (transmission of 1500 Bytes over a Gigabit link) 3 s / km : speed of light transmission over fiber Example of a connexion between 2 TPRs : Over 1000km traversing 10 nodes : 4ms

12 | Teleprotection over IP/MPLS - Validation - March2011

All Rights Reserved Alcatel-Lucent 2011

Jitter
Jitter in normal operations may come from the equipments themselves This is minimal (less than 1% of router latency) and compensated by jitter buffer. Jitter happens in non normal conditions such as congestion. This can be solved by applying the correct QoS parameters to the node. The implementation should allow total control of the bandwidth required per application, minimizing jitter.`
Rate Limit TPR to 100Kbps Rate Limit CCTV to 6Mbps
TPR

TPR gets to EF class CCTV to BE Class

Per SAP policing

Per interface queuing

Ingress
Make Sure that no application can go higher than expected bandwidth. Via Rate limit per SAP

Egress
Ensure that TPR application always gets the priority

H-QoS
13 | Teleprotection over IP/MPLS - Validation - March2011 All Rights Reserved Alcatel-Lucent 2011

Impact of failure
Todays teleprotection applications were developed towards SDH/PDH 50ms failover time. The impact of a failure (node or link) can have huge impact in case the failure exceeds the 50ms. MPLS FRR technology allows 50ms failover time in any failure scenario. These 50ms apply only in the core of MPLS network, but not in the case of the Access switch failing.

14 | Teleprotection over IP/MPLS - Validation - March2011

All Rights Reserved Alcatel-Lucent 2011

Conclusion on network failure impact

Failover in the backbone can be limited to 50ms with FRR FRR has to be implemented in the first node to minimize failure risks. All ALU Service routers support FRR. Integrating TDM in MPLS (as in 7705 SAR) brings the FRR to the application and limits the failure risks (compared to a 2 box solution) Providing extended rapid failover scenarios may be key in many applications. Support very rapid convergence time even in case of non direct connectivity (BFD), for example with Microwave links Implementing MC-LAG, G.8032 and BGP-MH for IEC 61850 greatly enhances total availability of the applications.

All Rights Reserved Alcatel-Lucent 2011

Other design considerations for Teleprotection

16 | Teleprotection over IP/MPLS - Validation - March2011

All Rights Reserved Alcatel-Lucent 2011

Packet flow asymmetry

LSPs are unidirectional Topology changes or mis-configuration may result in different path being used in both directions. Only trouble shooting tools may discover that. 5620 CPAM allows this detection and raises alarms.
Bypass actual path Green and Red LSP follow different Path Active Path Bypass Logical Link

Tracking of LSP status and history

17 | Teleprotection over IP/MPLS - Validation - March2011

Tracking of LSP Path for a given service

All Rights Reserved Alcatel-Lucent 2011

Synchronization
Information need to be delivered with time precision from an application standpoint.
Many TDM applications require clock synchronization (as SDH) Some applications require Time of Day type of synchronization

Packet networks can deliver clocking through different techniques :

18 | Teleprotection over IP/MPLS - Validation - March2011

All Rights Reserved Alcatel-Lucent 2011

Summary

All Rights Reserved Alcatel-Lucent 2011

Conclusion
Teleprotection is the most stringent application that can be transported over networks due to :
Low delays requirements, very low jitter requirements Impact of a failure in the application Alcatel-Lucent demonstrates that its IP/MPLS can be used as the next generation network for such applications with : Native TDM integration (incl. interfaces, synchronization, ) Very high resistance to potential failures in network Several management tools to anticipate, control and trouble shoot network IP/MPLS is the foundation to prepare the Smart Grid data explosion, and manage the transition of existing mission critical applications
20 | Teleprotection over IP/MPLS - Validation - March2011 All Rights Reserved Alcatel-Lucent 2011

www.alcatel-lucent.com