Sie sind auf Seite 1von 140

STANDARDS

IEEE Recommended Practice for 
Network Communication in Electric 
Power Substations 

IEEE Power and Energy Society 

Developed by the 
Power System Communications and Cybersecurity Committee 

IEEE Std 1615™‐2019 
(Revision of IEEE Std 1615‐2007) 

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615™-2019
(Revision of IEEE Std 1615-2007)

IEEE Recommended Practice for


Network Communication in Electric
Power Substations

Developed by the

Power System Communications and Cybersecurity Committee


of the
IEEE Power and Energy Society

Approved 5 September 2019

IEEE SA Standards Board

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
Abstract: Recommended practices for communication and interoperation of devices connected
on an electric power substation Internet Protocol (IP) network are provided. An introduction to the
concepts that need to be mastered as well as specific recommendations to follow when deploying
the technologies are provided for the power engineer new to IP networking. Direction and
requirements to facilitate interoperable electric utility information networks are provided for
equipment manufacturers and system integrators.

Keywords: Distributed Network Protocol 3 (DNP3), Ethernet, fiber-optic, IEC 60870-5, IEC
61850, IEEE 1615™, intelligent electronic device (IED), Internet Protocol (IP), managed switch,
network, network devices, noise sources, non-operational data, operational data, RS-232, RS-
485, security awareness, Transmission Control Protocol (TCP), time synchronization, wireless
network

The Institute of Electrical and Electronics Engineers, Inc.


3 Park Avenue, New York, NY 10016-5997, USA

Copyright © 2019 by The Institute of Electrical and Electronics Engineers, Inc.


All rights reserved. Published 8 November 2019. Printed in the United States of America.

IEEE is a registered trademark in the U.S. Patent & Trademark Office, owned by The Institute of Electrical and Electronics
Engineers, Incorporated.

PDF: ISBN 978-1-5044-6137-5 STD23873


Print: ISBN 978-1-5044-6138-2 STDPD23873

IEEE prohibits discrimination, harassment, and bullying.


For more information, visit http://www.ieee.org/web/aboutus/whatis/policies/p9-26.html.
No part of this publication may be reproduced in any form, in an electronic retrieval system or otherwise, without the prior written permission
of the publisher.

2
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
Important Notices and Disclaimers Concerning IEEE Standards Documents
IEEE documents are made available for use subject to important notices and legal disclaimers. These
notices and disclaimers, or a reference to this page, appear in all standards and may be found under the
heading “Important Notices and Disclaimers Concerning IEEE Standards Documents.” They can also be
obtained on request from IEEE or viewed at http://standards.ieee.org/IPR/disclaimers.html.

Notice and Disclaimer of Liability Concerning the Use of IEEE Standards


Documents
IEEE Standards documents (standards, recommended practices, and guides), both full-use and trial-use, are
developed within IEEE Societies and the Standards Coordinating Committees of the IEEE Standards
Association (“IEEE SA”) Standards Board. IEEE (“the Institute”) develops its standards through a
consensus development process, approved by the American National Standards Institute (“ANSI”), which
brings together volunteers representing varied viewpoints and interests to achieve the final product. IEEE
Standards are documents developed through scientific, academic, and industry-based technical working
groups. Volunteers in IEEE working groups are not necessarily members of the Institute and participate
without compensation from IEEE. While IEEE administers the process and establishes rules to promote
fairness in the consensus development process, IEEE does not independently evaluate, test, or verify the
accuracy of any of the information or the soundness of any judgments contained in its standards.

IEEE Standards do not guarantee or ensure safety, security, health, or environmental protection, or ensure
against interference with or from other devices or networks. Implementers and users of IEEE Standards
documents are responsible for determining and complying with all appropriate safety, security,
environmental, health, and interference protection practices and all applicable laws and regulations.

IEEE does not warrant or represent the accuracy or content of the material contained in its standards, and
expressly disclaims all warranties (express, implied and statutory) not included in this or any other
document relating to the standard, including, but not limited to, the warranties of: merchantability; fitness
for a particular purpose; non-infringement; and quality, accuracy, effectiveness, currency, or completeness
of material. In addition, IEEE disclaims any and all conditions relating to: results; and workmanlike effort.
IEEE standards documents are supplied “AS IS” and “WITH ALL FAULTS.”

Use of an IEEE standard is wholly voluntary. The existence of an IEEE standard does not imply that there
are no other ways to produce, test, measure, purchase, market, or provide other goods and services related
to the scope of the IEEE standard. Furthermore, the viewpoint expressed at the time a standard is approved
and issued is subject to change brought about through developments in the state of the art and comments
received from users of the standard.
In publishing and making its standards available, IEEE is not suggesting or rendering professional or other
services for, or on behalf of, any person or entity nor is IEEE undertaking to perform any duty owed by any
other person or entity to another. Any person utilizing any IEEE Standards document, should rely upon his
or her own independent judgment in the exercise of reasonable care in any given circumstances or, as
appropriate, seek the advice of a competent professional in determining the appropriateness of a given
IEEE standard.
IN NO EVENT SHALL IEEE BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO:
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
OTHERWISE) ARISING IN ANY WAY OUT OF THE PUBLICATION, USE OF, OR RELIANCE
UPON ANY STANDARD, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE AND
REGARDLESS OF WHETHER SUCH DAMAGE WAS FORESEEABLE.

3
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
Translations
The IEEE consensus development process involves the review of documents in English only. In the event
that an IEEE standard is translated, only the English version published by IEEE should be considered the
approved IEEE standard.

Official statements
A statement, written or oral, that is not processed in accordance with the IEEE SA Standards Board
Operations Manual shall not be considered or inferred to be the official position of IEEE or any of its
committees and shall not be considered to be, or be relied upon as, a formal position of IEEE. At lectures,
symposia, seminars, or educational courses, an individual presenting information on IEEE standards shall
make it clear that his or her views should be considered the personal views of that individual rather than the
formal position of IEEE.

Comments on standards
Comments for revision of IEEE Standards documents are welcome from any interested party, regardless of
membership affiliation with IEEE. However, IEEE does not provide consulting information or advice
pertaining to IEEE Standards documents. Suggestions for changes in documents should be in the form of a
proposed change of text, together with appropriate supporting comments. Since IEEE standards represent a
consensus of concerned interests, it is important that any responses to comments and questions also receive
the concurrence of a balance of interests. For this reason, IEEE and the members of its societies and
Standards Coordinating Committees are not able to provide an instant response to comments or questions
except in those cases where the matter has previously been addressed. For the same reason, IEEE does not
respond to interpretation requests. Any person who would like to participate in revisions to an IEEE
standard is welcome to join the relevant IEEE working group.
Comments on standards should be submitted to the following address:

Secretary, IEEE SA Standards Board


445 Hoes Lane
Piscataway, NJ 08854 USA

Laws and regulations

Users of IEEE Standards documents should consult all applicable laws and regulations. Compliance with
the provisions of any IEEE Standards document does not imply compliance to any applicable regulatory
requirements. Implementers of the standard are responsible for observing or referring to the applicable
regulatory requirements. IEEE does not, by the publication of its standards, intend to urge action that is not
in compliance with applicable laws, and these documents may not be construed as doing so.

Copyrights
IEEE draft and approved standards are copyrighted by IEEE under U.S. and international copyright laws.
They are made available by IEEE and are adopted for a wide variety of both public and private uses. These
include both use, by reference, in laws and regulations, and use in private self-regulation, standardization,
and the promotion of engineering practices and methods. By making these documents available for use and
adoption by public authorities and private users, IEEE does not waive any rights in copyright to the
documents.

4
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
Photocopies
Subject to payment of the appropriate fee, IEEE will grant users a limited, non-exclusive license to
photocopy portions of any individual standard for company or organizational internal use or individual,
non-commercial use only. To arrange for payment of licensing fees, please contact Copyright Clearance
Center, Customer Service, 222 Rosewood Drive, Danvers, MA 01923 USA; +1 978 750 8400. Permission
to photocopy portions of any individual standard for educational classroom use can also be obtained
through the Copyright Clearance Center.

Updating of IEEE Standards documents


Users of IEEE Standards documents should be aware that these documents may be superseded at any time
by the issuance of new editions or may be amended from time to time through the issuance of amendments,
corrigenda, or errata. A current IEEE document at any point in time consists of the current edition of the
document together with any amendments, corrigenda, or errata then in effect.

Every IEEE standard is subjected to review at least every ten years. When a document is more than ten
years old and has not undergone a revision process, it is reasonable to conclude that its contents, although
still of some value, do not wholly reflect the present state of the art. Users are cautioned to check to
determine that they have the latest edition of any IEEE standard.

In order to determine whether a given document is the current edition and whether it has been amended
through the issuance of amendments, corrigenda, or errata, visit IEEE Xplore at http://ieeexplore.ieee.org/
or contact IEEE at the address listed previously. For more information about the IEEE SA or IEEE’s
standards development process, visit the IEEE SA Website at http://standards.ieee.org.

Errata
Errata, if any, for all IEEE standards can be accessed on the IEEE SA Website at the following URL:
http://standards.ieee.org/findstds/errata/index.html. Users are encouraged to check this URL for errata
periodically.

Patents
Attention is called to the possibility that implementation of this standard may require use of subject matter
covered by patent rights. By publication of this standard, no position is taken by the IEEE with respect to
the existence or validity of any patent rights in connection therewith. If a patent holder or patent applicant
has filed a statement of assurance via an Accepted Letter of Assurance, then the statement is listed on the
IEEE SA Website at http://standards.ieee.org/about/sasb/patcom/patents.html. Letters of Assurance may
indicate whether the Submitter is willing or unwilling to grant licenses under patent rights without
compensation or under reasonable rates, with reasonable terms and conditions that are demonstrably free of
any unfair discrimination to applicants desiring to obtain such licenses.

Essential Patent Claims may exist for which a Letter of Assurance has not been received. The IEEE is not
responsible for identifying Essential Patent Claims for which a license may be required, for conducting
inquiries into the legal validity or scope of Patents Claims, or determining whether any licensing terms or
conditions provided in connection with submission of a Letter of Assurance, if any, or in any licensing
agreements are reasonable or non-discriminatory. Users of this standard are expressly advised that
determination of the validity of any patent rights, and the risk of infringement of such rights, is entirely
their own responsibility. Further information may be obtained from the IEEE Standards Association.

5
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
Participants
At the time this draft recommended practice was completed, the Network Communications in Substations
1615 Working Group had the following membership:

Kevin Easley, Chair


Craig Preuss, Vice Chair
Ed Cenzon, Co-Secretary
Michael Dood, Co-Secretary

Farel Becker Chris Huntley Silvio Roesler


James Bougie Marc Laroix Eric Thibodeau

The following members of the individual balloting committee voted on this recommended practice.
Balloters may have voted for approval, disapproval, or abstention.

William Ackerman Efthymios Karabetsos Miriam Sanders


David Aho George Karady Steven Sano
Ali Al Awazi Stuart Kerry Bartien Sayogo
Jay Anderson Yuri Khersonsky Robert Seitz
Philip Beaumont James Kinney Devki Sharma
James Bougie Boris Kogan Mark Simon
Derek Brown Bruce Kraemer Veselin Skendzic
Gustavo Brunello Jim Kulchisky Jeremy Smith
William Byrd Chung-Yiu Lam Jerry Smith
James Cain Lawrenc Long Michael Smith
Paul Cardinal Elvis Maculuba Gary Smullin
Yesenia Cevallos Arturo Maldonado Yuyin Song
Christopher Chelmecki Pierre Martin Robert Stagg
Randall Crellin John McDonald Thomas Starai
Ratan Das C. Michael Miller Wayne Stec
Matthew Davis Jeff Mizener Ryan Stone
Michael Dood Jerry Murphy David Tepen
Kevin Easley R. Murphy Eric Thibodeau
Ronald Farquharson Bruce Muschlitz Michael Thompson
Kenneth Fodero Arthur Neubauer Mark-Rene Uchida
Mietek Glinkowski Michael Newman Joe Uchiyama
Jalal Gohari Paul Nikolich James Van De Ligt
Edwin Goodwin Joe Nims Roger Verdolin
Randall Groves James O’Brien John Vergis
David Haynes Lorraine Padden James Volk
Roger Hedding Richard Paes Khurram Waheed
Marco Hernandez Bansi Patel Andrew West
Werner Hoelzl Charles Pestell Kenneth White
Gary Hoffman Percy Pool Aaron Wilson
Chris Huntley Charles Rogers Robert Wilson
Richard Jackson Benjamin Rolfe Jian Yu
Innocent Kamwa Thomas Rozek Oren Yuen
Daniel Sabin

6
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
When the IEEE SA Standards Board approved this recommended practice on 5 September 2019, it had the
following membership:

Gary Hoffman, Chair


Ted Burse, Vice Chair
Jean-Philippe Faure, Past Chair
Konstantinos Karachalios, Secretary

Masayuki Ariyoshi David J. Law Annette D. Reilly


Stephen D. Dukes Joseph Levy Dorothy Stanley
J. Travis Griffith Howard Li Sha Wei
Guido Hiertz Xiaohui Liu Phil Wennblom
Christel Hunter Kevin Lu Philip Winston
Thomas Koshy Daleep Mohla Howard Wolfman
Joseph L. Koepfinger* Andrew Myles Feng Wu
John D. Kulick Jingyi Zhou

*Member Emeritus

7
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
Introduction

This introduction is not part of IEEE Std 1615-2019, IEEE Recommended Practice for Network Communication in
Electric Power Substations.

IEEE Recommended Practice for Network Communication in Electric Power Substations was first
published in 2007. Much of the guidance in the 2007 version of this document remains important and
necessary in continuing to support legacy equipment.

There has been a dramatic change in the networking technologies since 2007. Security requirements and
tools to address them have impacted design philosophies. Technical requirements unique to power industry
applications previously precluded certain uses of Ethernet and TCP/IP (Transport Control Protocol/Internet
Protocol), and these are being resolved through the creation of other standards.

Many new applications have created the need for higher bandwidth to support real-time operational
awareness and to make this data available to many users beyond the traditional “system operators”. These
include higher power system load factors, the introduction of distributed and variable generation, and the
use of synchrophasors to support various power system applications.

Historically, serial connectivity (e.g., TIA-232-F) and protocols based on serial technology have dominated
substation communications. Serial communication was taken to be inherently secure, both through limited
physical accessibility and through obscurity of proprietary protocols. Substation inputs and control signals
have been wired directly to protective relays, remote terminal units (RTUs), and recording/metering
equipment.

The ability of Ethernet and TCP/IP to serve the large set of needs of the industry securely, reliably, and
economically alters the landscape of electric utility communications. DNP3 IP is ubiquitous, and IEC
61850 (International Electrotechnical Commission) is designed as a network suite of functions.
Historically, protective relays have been selected by power-system engineers without regard for network or
protocol support, and communications have been accomplished using whatever adapters and converters
were required. Now, although the protective functionality obviously remains paramount, lack of necessary
network functionality may be used to disqualify equipment.

8
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
Contents

1. Overview .................................................................................................................................................. 11
1.1 Scope ................................................................................................................................................. 11
1.2 Purpose .............................................................................................................................................. 11
1.3 Preface ............................................................................................................................................... 11
1.4 Design philosophy ............................................................................................................................. 12
1.5 Factors in network design .................................................................................................................. 12

2. Normative references................................................................................................................................ 14

3. Acronyms and abbreviations .................................................................................................................... 15

4. Electric utility information and communications technology (ICT) ......................................................... 17


4.1 Serial communication architecture .................................................................................................... 19
4.2 Ethernet network communication architecture .................................................................................. 21
4.3 Other network technologies ............................................................................................................... 24
4.4 Interfacing serial equipment to an Ethernet-based network ............................................................... 24
4.5 Moving from a serial system to an Ethernet-based architecture ........................................................ 26
4.6 Recommendations for utility networks .............................................................................................. 29

5. Data flows on Ethernet networks.............................................................................................................. 29


5.1 Data flows .......................................................................................................................................... 29
5.2 Characterization of data flows ........................................................................................................... 32

6. Security issues and recommendations ...................................................................................................... 34


6.1 Threats ............................................................................................................................................... 35
6.2 Targets of attack ................................................................................................................................ 35
6.3 Cyber-attack techniques..................................................................................................................... 35
6.4 Recommendations and techniques to address threats in a utility information system ....................... 36

7. Recommendations for Ethernet-based networks ...................................................................................... 39


7.1 Media ................................................................................................................................................. 41
7.2 Reliability .......................................................................................................................................... 43

8. Substation network design ........................................................................................................................ 44


8.1 Selection of communication architecture........................................................................................... 44
8.2 Functional requirements .................................................................................................................... 49

9. Interface and processing ........................................................................................................................... 51


9.1 Communication interfaces ................................................................................................................. 51

10. Testing .................................................................................................................................................... 53


10.1 Test planning ................................................................................................................................... 53
10.2 Testing plan framework ................................................................................................................... 54
10.3 Testing ............................................................................................................................................. 55

Annex A (informative) Security issues and awareness ................................................................................ 60


A.1 Source of threats to the electric power system .................................................................................. 60
A.2 Targets of attack in the electric utility network ................................................................................ 61
A.3 Common cyber-attack techniques ..................................................................................................... 63
A.4 Techniques to address threats to the electric utility network ............................................................ 66

9
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
Annex B (informative) Protocols used in and to substations ........................................................................ 73
B.1 Services supported by the Internet Protocol Suite (IPS) ................................................................... 73
B.2 Name-to-address mapping using DNS .............................................................................................. 73
B.3 Managed host configuration using Dynamic Host Configuration Protocol (DHCP) ........................ 73
B.4 Bootstrapping an IP address using Bootstrap Protocol (BOOTP)..................................................... 73
B.5 Network management using Simple Network Management Protocol (SNMP) ................................ 73
B.6 Time synchronization using Network Time Protocol (NTP)/IEEE Std 1588 ................................... 74
B.7 Remote terminals using secure shell (SSH)/Telnet ........................................................................... 74
B.8 IED access using Hypertext Transfer Protocol (HTTP)/HTTPS ...................................................... 75
B.9 File transfer using File Transfer Protocol (FTP)/Secure File Transfer Protocol (SFTP) .................. 76
B.10 Rapid Spanning Tree Protocol (RSTP) ........................................................................................... 77
B.11 High-availability Seamless Redundancy/Parallel Redundancy Protocol (HSR/PRP) ..................... 77
B.12 IEEE 1815 (DNP3) ......................................................................................................................... 79
B.13 Protocol features ............................................................................................................................. 81
B.14 IEC 60870-5 .................................................................................................................................... 83
B.15 IEC 61850 ....................................................................................................................................... 86

Annex C (informative) Internet Protocol Suite (IPS) ................................................................................... 91


C.1 Introduction ....................................................................................................................................... 91
C.2 IP layers ............................................................................................................................................ 92
C.3 IP Addressing .................................................................................................................................... 95
C.4 IPv6 ..................................................................................................................................................104

Annex D (informative) Interconnection diagrams for communications to and within substations .............106
D.1 Single master station ........................................................................................................................106
D.2 Multiple master stations ...................................................................................................................107
D.3 Multiple master stations, multiple RTUs .........................................................................................108
D.4 Combination systems .......................................................................................................................109
D.5 Substation gateway connections (legacy to standard protocols) ......................................................110
D.6 Networked systems ..........................................................................................................................111
D.7 Network physical topologies ...........................................................................................................112

Annex E (informative) Serial communication channel analysis ..................................................................116


E.1 Introduction ......................................................................................................................................116
E.2 Specify the performance of a master station to RTU communication channel ................................116
E.3 Channel performance analysis procedure.........................................................................................117
E.4 Illustrative example ..........................................................................................................................117

Annex F (informative) Protocol models ......................................................................................................119


F.1 Master slave......................................................................................................................................119
F.2 Client-server model ..........................................................................................................................121
F.3 Peer-to-peer networks.......................................................................................................................122

Annex G (informative) Network management ............................................................................................123


G.1 Configuring network settings in IEDs .............................................................................................123
G.2 Network management in the electric utility information system .....................................................124
G.3 Tools ................................................................................................................................................124

Annex H (informative) Features of devices and media supporting Ethernet-based networks .....................128
H.1 General device features ....................................................................................................................128
H.2 Media features .................................................................................................................................131
H.3 Network devices ..............................................................................................................................133

Annex I (informative) Bibliography ............................................................................................................136

10
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Recommended Practice for
Network Communication in Electric
Power Substations

1. Overview

1.1 Scope

This document defines a recommended practice for the design, testing, and operation of communications
networks within, to, and from electric power substations. Security considerations are included in the above.
It does not establish a new underlying communications standard. Instead, this document presents guidelines
and best practices for designing these communication networks.

1.2 Purpose

This recommended practice provides direction for implementers who wish to produce interoperable and
secure communications for an electric power substation network. It also provides direction for users,
system integrators, and equipment manufacturers who need to establish requirements and design of
communications networks within, to, and from the electric power substation. This is meant to address both
the physical and logical elements of the network.

1.3 Preface

Network communications is an important component of substation design because of the reliance of many
utility applications on a variety of communication networks. This recommended practice recommends
proven methods to specify network requirements, specify network schema, design network schema, and
mitigate hazards and threats to networks in electric utility substations, while controlling the cost of the
network and maintenance of the network. It is impossible for this recommended practice to be inclusive of
all types of communication networks and protocols that could exist inside of substations. For example,
advanced metering infrastructure (AMI) equipment may be installed inside substations, provide no
interaction with the substation local area network (LAN), and be independently connected to the wide area
network (WAN). So while impossible to cover every possible technology and protocol, there are many
common technologies typical to many substations.

Ethernet and the Transmission Control Protocol/Internet Protocol (TCP/IP) are by far the dominant
technologies in office and home use. The cost-effectiveness of IP over Ethernet, including factors of

11
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

reliability, equipment availability, equipment costs, skilled labor available, and others, have made IP over
Ethernet an obvious consideration for substation use.

In fact, IP over Ethernet is widely used in modern substation design, and has been shown to meet the needs
very well-presuming proper design and implementation. Unfortunately, the ease of plugging some
intelligent electronic devices (IEDs) into a router/switch combination, and plugging the router into a
modem provided by an Internet service provider (ISP), may lead to such deployment in a substation
environment without proper design and/or implementation.

Since an ad hoc network can quickly provide a high level of verifiable functionality, it can easily allow
many important design requirements to be overlooked, such as those presented in this recommended
practice. Some of these may be identified with minimal understanding or research, but many are more
subtle and their importance may not be discovered until unreliable communications are observed, a power
system disturbance occurs (such as those described in IEEE Std 1613) and causes the failure of
communication equipment and its supported communications, simple IED and/or equipment failure occurs
due to improper temperature ratings or clogged fans, or a breach of security occurs. 1

The use of IP over Ethernet is not presumed in this recommended practice. However, most of the principles
here apply to any network technology, and are generally stated here in IP over Ethernet terms.

1.4 Design philosophy

Many decisions in network design are dependent upon the importance of the substation and the types and
volume of data. A very robust, high-speed network could be designed and deployed at all substations.
However, as cost is always a matter in proper utility operations, it is acknowledged that not all substations
warrant the expense. Regardless, there are minimum practices necessary for diligent design of any network.
Understanding the level of network performance required and elements for achieving the requirements are
provided in this section. There are cost breakpoints in designing a network. In general, it is not possible to
spend 10% more for a 10% “better” network. This and the cost of applying a high level of design effort to
each substation leads to the common practice of designing two to four “classes” of network, and then
evaluating for each new substation which class to employ.

Historically, most substation IEDs were chosen using evaluation criteria that took precedence over
communications capabilities. Relays were selected based on performance, function, and capability in their
primary (protective) role. Meters were primarily selected based upon metering functions and displays.
Frequently, communication capabilities were not included or not significantly weighted in the evaluation
process. The result is that these substations have one or more IEDs that lack the interface (physical or
protocol) preferred by communication engineers. These devices can still be placed on a network by using
an intermediate device performing required media and protocol conversions.

1.5 Factors in network design

Factors in network design are as follows:

a) Cost of initial development


1) Equipment
2) Design
3) Labor

1
Information on references can be found in Clause 2.

12
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

b) Continuing costs
1) Replacement of failed equipment
2) Labor
3) Recordkeeping

c) Risk of loss of data from substation (value of data)


d) Risk of loss of control to substation (value of control)
e) Risk of stolen data for profit
f) Risk of damage due to attack
1) External, terrorist
2) External, mischief
3) Internal, disgruntled employee
g) Scope of possible damage
1) Single substation or part of substation
2) System wide

h) Methods of attack

1) Physical access to equipment


2) Electrical or communications to equipment:
i) Wired
ii) Wireless

i) Network equipment
1) Managed and unmanaged switches
2) Media converters, modems, transceivers, etc.

j) Media:
1) Copper wired cables
2) Optical fiber cables
3) Wireless
4) Ancillary equipment such as cable trays, conduits, fiber distribution panels, patch panels,
terminations, etc.

k) Legacy equipment/systems:
1) Serial, TIA-232-F (commonly known as RS-232)
2) Serial, TIA-485-A/TIA-422-B (commonly known as RS-485/RS-422)
3) Modems
4) MARS radio
5) Transducers

13
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

2. Normative references
The following referenced documents are indispensable for the application of this document (i.e., they must
be understood and used, so each referenced document is cited in text and its relationship to this document is
explained). For dated references, only the edition cited applies. For undated references, the latest edition of
the referenced document (including any amendments or corrigenda) applies.

IEEE Std 487.1™-2014, IEEE Standard for the Electrical Protection of Communication Facilities Serving
Electric Supply Locations Through the Use of On-Grid Isolation Equipment. 2, 3

IEEE Std 525™, IEEE Guide for the Design and Installation of Cable Systems in Substations.

IEEE Std 802.1D™-2004 (Reaff 2011), IEEE Standard for Local and metropolitan area networks: Media
Access Control (MAC) Bridges.

IEEE Std 802.1Q™-2018, IEEE Standard for Local and Metropolitan Area Networks—Virtual Bridged
Local Area Networks.

IEEE Std 802.3™-2018, IEEE Standard for Ethernet.

IEEE Std 1613™, IEEE Standard Environmental and Testing Requirements for Communications
Networking Devices in Electric Power Substations.

IEEE Std 1646™-2004, IEEE Standard Communication Delivery Time Performance Requirements for
Electric Power Substation Automation.

IEEE Std 2030™-2011, IEEE Guide for Smart Grid Interoperability of Energy Technology and
Information Technology Operation with the Electric Power System (EPS), End-Use Applications, and
Loads.

TIA-232-F, Interface between Data Terminal Equipment and Data Circuit-Terminating Equipment
Employing Serial Binary Data Interchange. 4

TIA-334-C, Signal Quality at Interface between Data Terminal Equipment and Synchronous Data Circuit-
Terminating Equipment for Serial Data Transmission.

TIA-404-B, Standard for Start-Stop Signal Quality for Non-Synchronous Data Terminal Equipment.

TIA-422-B, Electrical Characteristics of Balanced Voltage Digital Interface Circuits.

TIA-485-A, Electrical Characteristics of Generators and Receivers for Use in Balanced Digital Multipoint
Systems.

TIA-530-A, High Speed 25 Position Interface for Data Terminal Equipment and Data Circuit-Terminating
Equipment, Including Alternative 26-Position Connector.

2
The IEEE standards or products referred to in Clause 2 are trademarks owned by the Institute of Electrical and Electronics Engineers,
Incorporated.
3
IEEE publications are available from the Institute of Electrical and Electronics Engineers (http://standards.ieee.org/).
4
TIA publications are available from the Telecommunications Industry Association (http://www.tiaonline.org/).

14
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

3. Acronyms and abbreviations


For the purposes of this document, the following acronyms and abbreviations apply. The IEEE Standards
Dictionary Online should be consulted for terms not defined in this clause. 5

ACSI abstract communication service interface


AES advanced encryption standard
AMI advanced metering infrastructure
APCI Application Protocol Control Information
ARP Address Resolution Protocol
ASDU Application Service Data Unit
ASN.1 Abstract Syntax Notation One
ATM asynchronous transfer mode
BER bit error rate
BOOTP Bootstrap Protocol
CIP Critical Infrastructure Protection
COMTRADE common format for transient data exchange
CRC cyclic redundancy check
CSMA carrier sense multiple access (Ethernet)
DA distribution automation
DCE data circuit-terminating equipment
DCS distributed control system
DES/3DES data encryption standard
DHCP Dynamic Host Configuration Protocol
DMS distribution management system
DNP/DNP3 Distributed Network Protocol, version 3.0
DNS domain name system
DoS denial of service
EMI electromagnetic interference
EMS energy management system
EPS electric power system
FTP File Transfer Protocol
GMS generation management system
GOOSE generic object oriented substation event
GPR ground potential rise
GPS global positioning system
GSSE generic substation status event

5
IEEE Standards Dictionary Online is available at: http://dictionary.ieee.org.

15
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

GUI graphical user interface


HDLC high-level data link control
HEMP high-altitude electromagnetic pulses
HMI human-machine interface
HSR high availability seamless redundancy
HTML hypertext markup language
HTTP Hypertext Transfer Protocol
IANA Internet Assigned Numbers Authority
ICMP Internet Control Message Protocol
ICT information and communications technology
IDS intrusion detection system
IED intelligent electronic device
IETF Internet Engineering Task Force
IP Internet Protocol
IPS Internet Protocol Suite
IPv6 Internet Protocol, version 6
ISO independent system operator
ISP Internet service provider
IT information technology
LAN local area network
LRU logical remote terminal unit
MAC media access control
MIB management information base (SNMP)
MMS manufacturing message specification
MTU maximum transmission unit
NAT network address translation
NERC North American Electric Reliability Corporation
NTP Network Time Protocol
OID object identifier
OSI open systems interconnection
PLC programmable logic controller
PPP Point-To-Point Protocol
PQDIF Power Quality Data Interchange Format
PRP Parallel Redundancy Protocol
PSTN public switched telephone network
QoS quality of service
RC-4 Rivest Cypher

16
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

RCP remote copy


RFC request for comment document
RFI radio frequency interference
RMON remote traffic monitoring
RS232 Recommended standard 232, or TIA-232-F
RS485 Recommended standard 485, or TIA-485-A
RSA (Rivest, Shamir, Adelman) encryption algorithm
RSTP Rapid Spanning Tree Protocol
RTO regional transmission organizations
RTU remote terminal unit
SCADA supervisory control and data acquisition
SCL substation configuration language
SCP secure remote copy
SDN software defined networking
SFTP Secure File Transfer Protocol
SGIRM Smart Grid Interoperability Reference Model
SNMP Simple Network Management Protocol
SNTP Simple Network Time Protocol
SPAN switched port analyzer
SSH secure shell
SV sampled value
SWC surge withstand capability
SYN synchronize
TCP Transmission Control Protocol
Telco telecommunication service company
UART Universal Asynchronous Receiver/Transmitter
UDP User Datagram Protocol
UHF ultra high frequency
VLAN virtual local area network
VPN virtual private network
WAN wide area network
WLAN wireless local area network

4. Electric utility information and communications technology (ICT)


Clause 4 first addresses the link between IEEE Std 2030-2011 and substation network design, and then
answers the following two questions in broad terms:

17
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

 What is the justification for moving from a traditional serial-communication architecture to a


network-based one?
 What is a general strategy for performing the upgrade?

In answering these questions, two technologies are considered for the upgrade, namely serial-to-network
conversion equipment and the transition from analog to digital channels.

IEEE Std 2030-2011 provides a roadmap in establishing the Smart Grid Interoperability Reference Model
(SGIRM). The SGIRM is a conceptual representation of smart grid infrastructure from three architectural
perspectives: electric power systems (EPS), communications, and information technology. EPS integration
with ICT achieves seamless operation of electric generation and delivery, while providing end-use benefits
and permitting two-way power flow with communication and control. Reliability of the EPS is increasingly
affected by the ICT infrastructure. Substation communication networks are actually network architectures
supporting communications inside and to electric power substations, accounting for a small portion of the
overall ICT. The safety and reliability of the EPS should prevail in the event of a disruption of the ICT
infrastructure. This recommended practice addresses this goal through providing recommended practices
for the design, installation, testing, and maintenance of the substation’s portion of the overall ICT
infrastructure.

IEEE Std 2030-2011 provides high level descriptions of the entities, or actors, that require communication
channels to and within the substation and terminate at data acquisition/control equipment in the substation.
Historically, there was only one actor with its own ICT [for example, a supervisory control and data
acquisition (SCADA) master connection using only a 1200 baud or 9600 baud modem on a leased-line
phone circuit]. Typically, actors were added with their own dedicated and independent ICT for each
required communication channel, such as with the addition of protection relay communications, remote
engineering access, metering, additional utilities, and other applications over time. Example actors are
aligned with IEEE Std 2030-2011 and are discussed in more detail in Clause 5.

As the amount of data required by these different applications grows (data volume), the ICT may not be
able to transport and process it in a timely fashion (information transfer time), without increasing
bandwidth (data versus speed). As the distance grows between intelligent electronic devices (IEDs) using
TIA-232-F technology, converting to other technologies that can support the distance is required [this may
be a simple change to serial fiber technology or TIA-485-A, or a more complex conversion to Ethernet that
may involve “serial/port/terminal servers” or true Ethernet between the devices, which may also need to
address fiber for distance or electromagnetic interference (EMI) isolation]. A networked architecture for the
ICT adds capabilities supporting improved reach (distance), information transfer time, latency,
synchronicity, information reliability, data volume, and security (confidentiality, integrity, and availability),
as well as scalability. Different communication architectures provide tradeoffs between these
characteristics.

IEEE Std 1615 can be applied as follows:

 After the ICT evaluation has been accomplished as outlined in IEEE Std 2030-2011, meaning a
communication architecture has been selected and will be applied to substations.
 During ICT evaluation of the classification of data flows as described in IEEE Std 2030-2011,
meaning that a communication architecture has yet to be selected.
 Without applying the IEEE Std 2030-2011 SGIRM, which may be a situation where the utility has
no overall strategic plans for an overall smart grid infrastructure and wants to start deploying
networks in substations to support different applications as either the first deployment of substation
communications or to upgrade from older serial technology that cannot support those applications.
The remainder of Clause 4 only applies when the latter two scenarios exist, because the remaining clauses
describe the limitations and advantages of serial communication architectures, network communication
architectures, and moving from a serial to network architecture.

18
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

4.1 Serial communication architecture

The historically dominant, serial communication architecture is illustrated in Figure 1 and consists of
equipment at one or more central locations (typically control centers), at substations, and along feeders.
This architecture has evolved over time to facilitate the interconnection of many different types of actors
(such as SCADA master stations, remote access, planning, maintenance, protection, non-operational data,
or other types of computers accessing data on demand) to IEDs using many different serial protocols and
channels. The serial channels to the substation are typically point-to-point (TIA-232-F) but can also be
point-to-multipoint (TIA-485-A). One or more serial channels will terminate at the RTU/data concentrator
function described by IEEE Std C37.2™ [B22], which typically supports several serial channels itself and
provides its collected data to each actor via a dedicated channel. 6

Figure 1 —Traditional serial-communication architecture

Table 1 describes characteristics that explain considerations necessary for traditional serial
communications.

6
The numbers in brackets correspond to those of the bibliography in Annex I.

19
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Table 1 —Serial Communication Architecture characteristics

Characteristic Serial Communication Architecture


Reach (distance) TIA-232-F is typically limited to distances of less than 30 m (100 ft), depending
upon the selected baud rate and cable quality (lower capacitance is better), which
can be extended using fiber optic transceivers or converting to TIA-485-A if this
is too short for the application. With a distance limitation over 300 m (1,000 ft),
TIA-485-A distance limitations are typically not relevant to most substations.
Information transfer time Lower speed serial technology limits the data rate. The data occurrence interval
and information transfer time (or data update rates) on serial are impacted by the
number of IEDs sharing a single communication channel, the amount of data each
IED is transmitting, and the processing time for the IED to decode the poll and
assemble the response. As serial systems grow over time, it is common to
accumulate many IEDs, both inside the substation and on feeders that connect to
the serial system via a small number of external channels. Radio links and
multipoint leased lines support the use of many devices in this fashion. However,
the tradeoff is response time, as all master/slave protocols utilize the channel in a
time-multiplexed manner. The available channel bandwidth for each IED is
proportional to the inverse of the number of devices using the channel, so
performance of the entire channel suffers as more and more IEDs are added to it.
Inside a substation, serial channels are similarly limited as the number of IEDs
connected to one channel increases. In addition, as point counts increase, this may
also require the end device more time to assemble a response and require more
time to transmit a complete response.
Latency (includes access, Depending on the amount of data, number of devices on each serial channel, and
propagation, and reception the serial channel speed, data may not be obtained in the desired interval.
delay)
Synchronicity and Serial channel performance may be somewhat predictable, but likely not
determinism deterministic nor synchronistic.
Information reliability Providing redundancy in serial systems requires careful design in order to get true
redundancy, which will likely double the number of data concentrators and
complicate the data collection and cabling design.

Fiber optic transceivers can be used to provide the required isolation from the
electromagnetic interference (EMI) environment described by IEEE Std 1613.
Careful selection of communication cable may also be required for the application
per IEEE Std 525.
Data volume As systems get larger, and more IEDs are deployed, the sheer volume of point data
that needs to move over the communication links increases.

Also, serial channels are static and typically implement protocols that allow only
communication where one device polls (the master) all of the others (the slaves)
for data.

It is often desirable to partition different types of data or data with different


priorities into multiple logical channels in a communication architecture. Whereas
SCADA protocols will generally allow multiple addresses on a serial channel,
single serial channels do not allow multiple protocols.

It is possible to interleave serial communications using proprietary methods to


provide two channels of communication on a single serial port.
Security (confidentiality, There is a general perception that serial communication is somehow secure and,
integrity, and availability) therefore, does not require any, or limited security. Serial communications may,
however, still be vulnerable to security concerns, including the interception of data
streams that could allow serial messages to be modified, forged, reordered, or
replayed.

IEEE Std 1711™ [B19] discusses methods of securing serial communications.

Security of serial communications may be cheaper and easier to obtain.


Table continues

20
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Characteristic Serial Communication Architecture


Scalability IED manufacturers provide information that frequently does not map into SCADA
protocols or at least cannot be transported well between different protocols (e.g.,
event logs, oscillography files, power quality data, or non-operational data
discussed in Clause 5). An engineer who wants to access this additional
information must either go out to the remote site to access the IED or more serial
communication channels that are dedicated for this purpose.

As IEDs become more sophisticated and flexible supporting more data flows as
discussed in Clause 5, they require more configuration. More often than not, this
configuration is accomplished via a dedicated “maintenance port” on the IED with
either a terminal program or proprietary software used to access the IED. Even a
moderately sized system quickly accumulates dozens of configuration channels
that must be planned, designed, tested, and managed.

As pure serial systems expand, the number of IEDs required to collect the data
scales linearly to some proportion of the IEDs. In addition, a relationship may
need to be created to have a master data concentrator that collects data only from
the data concentrators. For example, many data concentrators have only 16
“RS232” (TIA-232) serial ports, so as the number of IEDs expands, the number of
data concentrators increases by one concentrator for every 16 IEDs. In this
scenario, a substation of 80 IEDs requires a minimum of five data concentrators.

There are advantages to using serial communication architectures as follows:

 Some serial architectures may be seen as more reliable, more secure, and less expensive than
networks. Careful comparison is required, because it is easy to conclude that a small serial network
using copper serial cables will be cheaper and more reliable than an Ethernet LAN using fiber
cables and an improperly sized and specified Ethernet switch.

 For small systems with small point counts, low-speed serial systems for only SCADA channels can
still meet their intended functional requirements.

 Serial systems may also have a smaller impact on substation battery size due to smaller power
requirements of port-powered devices.

4.2 Ethernet network communication architecture

Figure 2 shows an Ethernet network architecture where all IEDs do not necessarily have to be on a LAN to
be accessible to the network. In Figure 2, the feeder communication circuits can use serial protocols that
transport the network packets [e.g., Point-To-Point Protocol (PPP)]. The remote IEDs are allowed to be part
of the network and therefore accessible by the data users.

21
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure 2 —Networked information system

Table 2 describes characteristics that explain considerations necessary for Ethernet communications.

22
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Table 2 —Ethernet Network Communication Architecture characteristics

Characteristic Ethernet Network Communication Architecture


Reach (distance) Ethernet network distances using copper cables span between TIA-232-F and
TIA-485-A distances, but cannot reach TIA-485-A distances without conversion
to optical fiber cable.
Information transfer time Ethernet networks can deliver information transfer times to support IEC 61850
Latency (includes access, generic object oriented substation event (GOOSE) message performance
propagation, and reception requirements.
delay)
Synchronicity and Requires dedicated network queue allocations.
determinism
Information reliability A network will typically allow more redundancy options that will leverage the
network, both in physical connections (two Ethernet ports) and various versions of
redundant operations. Just like serial systems, network redundancy results in more
data concentrators, but depending upon capability there may only need to be two.
Also, network redundancy may present somewhat simpler data collection and
cabling design.

Communication cables, especially copper Ethernet cables, need to be suitable for


their installation environment. These cables also require separation from control
cables in a cable tray and from fluorescent lights. Use of fiber optic transceivers
can be used to provide the required isolation from the EMI environment described
by IEEE Std 1613. See IEEE Std 525 for more information on the design
recommendations. Adding the appropriate cable trays to support the required
separation in existing substations may not be as straightforward and simple as it is
a greenfield substation.

The design of information technology (IT) networks is not the same as the design
of critical operations networks that must be on 24×7×365. While many
fundamental concepts overlap, the design of substation networks must be suitable
for that environment, which is much different from a commercial building
environment. Environmental conditions are also very different, such as
temperature, humidity, and vibration, which are addressed in IEEE Std 1613 and
IEEE Std 525.
Data volume Ethernet LAN in a substation, along with a high-speed digital interconnection to
the control center, greatly increases the available communication bandwidth.
SCADA communication protocols typically consume little bandwidth on a
100 Mbps network.

Networks can support multiple logical channels across multiple IEDs. Two IEDs
on a network can be transferring data via one protocol while two other devices
simultaneously transfer data via another protocol.
Security (confidentiality, Introducing networks into substations may require access control for substations,
integrity and availability) to better secure them in order to address new cybersecurity programs.

Network cybersecurity is a significant issue. The risks should be completely


identified and mitigated in order to adequately protect the substation IEDs and
network from cyber-attack. See IEEE Std 1686™ [B18] for cybersecurity features
that may be desirable. See Clause 6 for criteria to determine if the substation
network is properly secured.
Table continues

23
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Characteristic Ethernet Network Communication Architecture


Scalability The network layer protocol provides a direct link to IEDs from anywhere on the
network without worrying about what other protocols happen to be in operation on
the same network.

Depending on IED functionality, engineers could upload additional information


from IEDs with minimal impact (assuming adequate bandwidth and device
processing power) to other processes (e.g., SCADA) and without additional
hardware. Each IED merely has to provide another protocol port number for those
additional functions.

The added bandwidth and multi-protocol capabilities of IEDs mean that a separate
maintenance port may only be needed to initially configure the IED. After that, all
configuration activities could take place over the network from a central location.

Adding Ethernet networks to substations can introduce the following significant challenges:

 Training and organizational structure: Those who are familiar with IT networks are generally not
familiar with substations or the “always on”, critical nature of control systems. Those who are
familiar with substation IEDs may not be familiar with networking. In many cases, overlap does
not exist, so training the personnel on both sides is required, and some organizational realignment
may be needed to bridge the gaps prior to the start of design.

 Battery power: Networking devices can add substantial requirements to the substation battery. For
substation upgrades/retrofits, larger batteries may be required to support the networking equipment
during extended outages. Battery calculations should always be revisited during system design to
help ensure the battery system will support all IEDs in the substation automation system, including
substation networking devices (switches, routers, firewalls, serial servers, etc.).

4.3 Other network technologies

Token Ring, Fieldbus, Modbus Plus, Profibus, and other technologies are also used in electric power
substations. Many of the principles in this document are applicable across technologies, although other
technologies may not support their implementation.

4.4 Interfacing serial equipment to an Ethernet-based network

Substation IEDs are frequently chosen using evaluation criteria that take precedence over communications
capabilities. Relays are selected based upon performance, function, and capability in their primary
(protective) role. Meters are primarily selected based upon metering functions and displays. Frequently,
communication capabilities are not included or not significantly weighted in the evaluation process. The
result is that the substation has one or more new or existing IEDs that have one or more serial interfaces
with no network interface. These serial devices can still be placed on a network by using an intermediate
device with both a network connection and serial connection.

These intermediate devices can be classified into two general classes of serial-to-network converters—
those that are protocol-independent and those that are protocol-specific. Each has its place in a modern
electric power information system.

24
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure 3 —Serial-to-network converters

4.4.1 Protocol-independent converter

The protocol-independent converter is sometimes called a “terminal server” (also known as a “port server”
or “serial server”) and typically supports up to 16 serial ports but can be as high as 32 or even 48 ports. A
terminal server is used when both pieces of equipment to be interfaced understand the same protocol and
merely requires a physical conversion of network packets to/from a serial data stream. It is typically not
protocol aware or does not understand the single protocol being used.

4.4.2 Protocol-aware converter

When a serial server is protocol aware, that awareness is usually leveraged by the device to improve overall
connectivity and functionality. For example, when using addressable protocols such as Modbus and
Distributed Network Protocol, Version 3.0 (DNP3) (IEEE Std 1815), some serial servers will route
messages from the network to a specific serial port based upon the address contained in the protocol. On
the serial side, some of these devices will take a single serial message and send it to multiple IEDs on the
network side.

A protocol-aware converter can be two types. In the first type, the converter is aware of simple protocol
functionality and allows messages to be routed between the Ethernet network and serial ports and vice-
versa, without protocol conversion (but perhaps protocol encapsulation).

The second type is a more complicated converter used when two IEDs cannot communicate because they
use different protocols [i.e., one could be Modbus and the other DNP3 (IEEE Std 1815)]. This is a more
sophisticated converter because of how each network’s interface is handled and the amount of
configuration required. Some converters translate requests and responses on protocol 1 directly to/from
protocol 2. A prerequisite for this design is a one-to-one mapping between requests and data types of the
two protocols. More often, the IED lets both protocols run separately. Data and control are stored in a
generic format, and converted by the protocols as appropriate. This type of design requires much more
configuration information. This type of protocol-specific conversion is typically supported by a data
concentrator, but can also be accomplished in a simpler protocol converter.

25
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

4.5 Moving from a serial system to an Ethernet-based architecture

Moving from a serial communication architecture to an Ethernet-based network architecture could be


justified by one or more of the following needs:

 Ability to deploy and enable new technologies such as IEC 61850 communications [including
GOOSE, manufacturing message specification (MMS), and Process Bus].
 Faster data update rates: Ethernet networks provide higher speeds that can support faster update
rates than serial can support, typically independent of the number of IEDs on a substation LAN in
comparison to the number of IEDs on a serial channel.
 Important and timely data access: If a utility requires the ability to analyze faults, obtain other
power quality related data, obtain other asset management related data, or obtain files that contain
large amount of data, it will likely install IEDs that can provide this information. Depending on the
importance that a utility places on this data or the time-critical nature of this data, this could be the
sole justification for installing an Ethernet network because too many low-speed serial connections
are required using serial communications.
 Reduction of configuration and system management time: Reconfiguration and maintenance of
substation IEDs is an ongoing process. A common misconception is that access to the maintenance
facilities of IEDs is only done at commissioning. In practice, there are a number of reasons to use
the facilities on a more frequent basis, such as scheduled updates and system diagnostics. There can
be reasons for access that are not just maintenance. For example, it may be desirable for operators
or engineers to be able to dynamically change relay settings in response to an outage or seasonal
loads. Remote access is supported by serial and Ethernet-based networks. Remote access can be
used for more than maintenance activities, such as checking configurations, checking error logs or
reports, checking device health, obtaining device reports, and obtaining files. While cybersecurity
concerns exist with allowing remote access via either method, the benefits of reduced travel time
for widely dispersed assets can be very tangible. As bandwidth increases with Ethernet-based
networks, response times decrease and the amount of data that can be obtained increases over the
same time interval when compared to serial systems.
 Ability to multiplex channels of information: With serial communications, separate physical links
may be needed for SCADA, remote access, file retrieval, and to a substation human-machine
interface (HMI). A vendor may be able to support one or two of these connections over one serial
port with limited impact to either’s performance. An Ethernet-based network supports multiple
logical ports allowing concurrent access to an IED using a variety of protocols, provided the IED
supports multiple concurrent connections. Many IEDs still have limited resources and therefore
support a limited number of concurrent connections. For example, one IED may support up to five
concurrent connections of any protocol combination, but only two instances of IEEE 1815 (DNP3)
are supported. Another example may be an IED that supports only two concurrent IEEE 1815
(DNP3) connections and no other logical ports are used at all.
 Reduction of system expansion costs: Ethernet-based networks can also reduce overall system costs
when compared with a serial system. While serial devices may initially be cheaper, as system
functionality increases along with the number of IEDs, serial systems require more connections that
may offset the additional costs of installing an Ethernet-based network depending upon system
architecture and functionality. Initial deployment costs may linearly increase with serial systems as
system functionality increases. Expansion of serial systems will most likely be more expensive
because serial port densities are not as great as Ethernet port densities, and multiple serial ports are
most likely required to support multiple functions. Both serial and Internet Protocol Suite (IPS)
systems have scalability issues, but serial systems typically encounter these values at lower
numbers of IEDs. Serial ports on data concentrators or RTUs are limited to some small number,
typically 16, 32, or 48. Some of these connections may be a mix of single or multi-drop serial
connections. For serial systems with larger device counts beyond what a single data concentrator
can support, more data concentrators may be needed with master-slave relationships, and some
vendors provide tools to make this process “easier”. On Ethernet-based systems, typically one data

26
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

concentrator can support up to 128 concurrent connections for polling IEDs on an Ethernet-based
network. But in the end, Ethernet-based systems suffer from similar expansion problems as serial
systems, but they typically occur at larger IED counts.
 Ubiquity of Ethernet: Ethernet-based networks are widely supported and understood in IT
departments. A huge body of knowledge exists in the public domain for installation and
deployment. These networks can also be easily monitored and managed using tools that are
standards-based. Ethernet also supports a wide range of applications across the utility enterprise
without requiring a different or separate network.
 Redundancy: There are many forms of redundancy that can be implemented in serial and Ethernet-
based networks. Typically, redundant serial networks are more difficult to implement and more
complex because of the singular nature of each connection to each IED. Ethernet-based networks
may provide advanced redundancy features, not only for redundancy in the communication path to
the IED, but also to provide a range of simple to robust failover schemes.

By comparing costs with benefits, a business case can develop the financial justification for migrating to
Ethernet-based architectures. Some benefits may be easier to monetize, or easier to develop the cost
savings.

4.5.1 Strategy for upgrading

As will always be the case, the primary needs of the particular system drive the upgrade strategy. A utility
should use the process outlined in the SGIRM to develop an interoperable architecture that ensures the
upgrade can support the data sources that can give the most important information. Upgrading from a serial
communication architecture system to an Ethernet network architecture can be accomplished in two ways:
top-down or bottom-up.

4.5.1.1 Top-down approach

The following steps constitute a top-down approach:

a) Upgrade communications to the substation from serial to Ethernet-based network and add a
network interface to one or more substation IEDs.
b) Add an Ethernet-based network within the substation that connects all IEDs by upgrading them to
network connectivity or via serial servers.
c) Upgrade communications to the feeder IEDs and make them network accessible using serial servers
or upgrade them to network connectivity.

4.5.1.1.1 Advantages to top-down approach

The top-down approach has the following advantages:

 Adds bandwidth where it is needed most, the substation-to-utility communications.


 Improves access to the RTU/data concentrator and, potentially, other IEDs.
 Allows replacing analog lines with more robust digital channels.
 May allow inclusion or consolidation of other communications services on a single communication
channel (e.g., phone, video surveillance).
 May provide options for redundant communications to the substation.

27
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

 May reduce communications costs, depending upon consolidation of analog circuits and digital
circuit bandwidth.

4.5.1.1.2 Disadvantages to top-down approach

The top-down approach has the following disadvantages:

 May significantly increase communication costs if bandwidth requirements remain low.


 Initially only allows access to the RTU/data concentrator; this IED may not be the most important
data source or the highest bandwidth consumer.
 Requires upgrading equipment at the control center.
 Forces additions to workforce skill sets.

4.5.1.2 Bottom-up approach

The bottom-up approach constitutes the following steps:

a) Add a network within the substation and connect as many IEDs as possible to that network via
direct connections or serial servers.
b) Upgrade communications to the feeder IEDs and make them network accessible.
c) Upgrade communications to the substation and add a network interface to the RTU/data
concentrator if not already accomplished.

4.5.1.2.1 Advantages to bottom-up approach

The bottom-up approach has the following advantages:

 Most work can be limited to the substation with little effect on the SCADA master.
 Lower initial communication costs.

4.5.1.2.2 Disadvantages to bottom-up approach

The bottom-up approach has the following disadvantages:

 Does not allow access to systems outside of substations until later in the evolution.
 Does not improve response time from the control center.
 May require initial upgrade to networked IEDs or, depending on the protocol, the addition of a
serial/port/terminal server. If this feature is not available from the current vendor, replacement costs
are increased significantly.
 Forces additions to workforce skill sets.

4.5.2 Converting from analog leased to digital leased lines

Some utilities have relied on leased communication lines from a telecommunication service company
(Telco) provider for substation communications. There may be separate communication lines for SCADA,
protection, remote access, metering, equipment monitoring, etc. In many cases, the utilities did so with the

28
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

belief that the line was dedicated, end-to-end, to the sole function of their use. These lines handled analog
signals.

Analog lines are disappearing in the Telco world. A solid copper wire pair connecting two points
disappeared years ago. The copper pair delivered to the utility is really just wires to the nearest Telco
network center. From that point forward, the traffic on the wires becomes digital signals through the Telco
network. The Telco performs the A-to-D and D-to-A conversion and reserves a packet connection for that
traffic. So for all practical purposes, the connection is permanent and essentially dedicated but not
necessarily private.

Telcos charge different rates for digital service. For the equivalent of an analog leased line, 56 kbps, they
charge a higher base rate; however, they charge only for the distance to the nearest Telco digital hub.
Where the utility pays the end-to-end distance charges for an analog line, it will pay a lesser distance
charge for digital lines, potentially saving the utility Telco charges. Note this assumes the utility has
already installed any required Telco isolation equipment as per IEEE Std 487.1™-2014. Should a line that
is not equipped with isolation be converted, it loses its “grandfathered” status and would need to be
equipped for isolation at utility expense.

It is possible for a utility to convert leased lines from analog to digital, decrease the combined basic and
distance charge, and gain additional bandwidth. The additional bandwidth may permit added
communications connections for more than just SCADA or protection as previously discussed.

4.6 Recommendations for utility networks

All electric utility networks should use the IPS for the transfer of operational and non-operational data as
well as configuration management. In addition, the following are recommended:

 Each utility should develop a plan for private IP address allocation (see C.3.3).
 Private addressing should be employed on electric utility substation networks (see C.3.4).
 Substation information systems should not be connected to a public or utility-wide network unless
the security measures discussed in Clause 6 are evaluated and put in place.

5. Data flows on Ethernet networks


As previously discussed, Ethernet networks provide the capability of supplying multiple logical
connections to one physical IED. Each one of these logical connections produces a data flow across the
Ethernet network. According to IEEE Std 2030-2011, each data flow is categorized based upon data use
characteristics.

5.1 Data flows

This clause does not provide a detailed analysis of the data flows as described by IEEE Std 2030-2011.
Actors producing data flows to and within substations may include any combination of the following
applications producing those flows using the typical protocols and ports (see C.2.3.3) as shown in Table 3.
Many of these protocols are discussed in Annex B.

29
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Table 3 —Applications and data flows


Typical Protocols
IEC 61850
Secured DNP3 SSH HTTP SFTP
Not SNMP IEEE NTP IEEE GOOSE MMS SV IEEE Modbus Telnet HTTPS FTP
Application secured 1588 C37.118 1815
Supervisory control and X X X
data acquisition (SCADA)
Energy management X X
systems (EMS)
Distribution management X X
systems (DMS) or advanced
DMS (ADMS)
Generation management X X
systems (GMS)
Distribution automation X X
systems (DA), which may
be integrated with the
substation, integrated into
the DMS/ADMS, and/or
pass through
communications
Protection communications X X X
within the substation
Protection communications X X
outside the substation (to
another substation, for
example)
Equipment monitoring X X X
(transformer, breaker,
battery, etc.)
Power quality monitoring X X X X X
Metering X X X X
System planning and design X X X
Cyber access control and X X X
monitoring
Third party access (external X X X
entities to the utility, such
as vendors, independent
system operators, and other
utilities)
Disturbance monitoring X X X X X
Generation plant control X
systems
Distributed generation and X X
storage management
Remote access to IEDs for X X X X
maintenance activities
involving file retrieval,
settings review/changes,
event records, etc.
Asset X X X X X
management/condition
monitoring system
Time synchronization X X
LAN/WAN management, X X X X
monitoring and control
Physical security
monitoring
Corporate network access X X X
TCP Port 161, 319, 123 4712, N/A 102 N/A 20000 501 22 80 22
162 320 4713 23 443 20, 21

While the previous data flow list is extensive, a simpler approach characterizes data flows in substations
into two general categories: operational and non-operational. Some utilities obtain both types of data using

30
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

only one system, although this requires a narrow interpretation of non-operational data and does not include
many of the data flows previously listed.

One of the major reasons for moving to an Ethernet-based network is to allow the various users of the
substation data to have access to which is important to them. It is, therefore, important to consider all of the
departments in the utility and their data needs. Table 4 provides a general list of departments with the
general type of data that they may need.

Table 4 —Data type use by department


Department Operational Operational Non-Operational Examples
status control
Operations X X SCADA, EMS, DMS, GMS
Planning X X Load flow analysis and
forecasting
Feedback info to operations
Maintenance X X Equipment condition monitoring

Protection X X X Relay settings, oscillography


Protective relaying X Protective relaying and high-
speed status messages
Power quality X X Analysis of events, harmonic
content, sags, swells
Billing X X Uses subset of operational data
revenue kWh and kvarh
Power marketing X X Uses subset of operational data

A similar table can be constructed and distributed in the network planning stage to help identify the
shareholders for all data flows. This can then be expanded to address the more extensive data flows and
characterizations in IEEE Std 2030-2011 that will ultimately drive more comprehensive communication
architecture decisions.

5.1.1 Operational data

Operational data can be broken into status and control data needed by operations to monitor and control the
EPS. Operational data is delivered reliably so that proper and timely decisions can be made in the event of
an EPS event, such as an accidental change (fault, equipment failure, etc.) or planned change of operational
state. Also called “SCADA data”, some examples are given below.

Some operational data may require synchronicity, or for a specific action to occur at a specific time over
some segment of a utility’s portion of the EPS. This commonly occurs with accumulator freeze commands,
allowing energy readings to be reported at the same time across the system. This may also occur using
broadcasted freeze commands, requiring the ICT to support the receipt of the broadcast message by all end
devices at the same time.

5.1.1.1 Operational status data

Operational status data is information used by operators, system management personnel, and control
processes to keep the power system running. Operational data shows the current state of the power system
and is analog or discrete in nature. The data is supplied by IEDs on an interval basis [on a polled or report
by exception basis, see IEEE Std 1815 or IEC 61850 (summary information for these two standards is

31
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

provided in Annex B)]. Examples include breaker/switch status and loading, line loading, and equipment
status and loading. Status can have different time requirements, where discrete status data may need to be
updated on a quicker interval than analog data.

5.1.1.2 Operational control data

Operational data also includes control commands, which are messages sent by operators and control
processes to optimize, change, and restore the state of the EPS. Controls can be binary or analog (setpoint).
The decision to change state is generally based on the current or predicted state of the EPS. Control
commands require reliable, secure, and timely delivery. Controls can have different time requirements,
where controls initiated from protection messages may need to arrive in milliseconds and controls initiated
via system operators may be acceptable in several seconds.

5.1.2 Non-operational data

Non-operational data is information used by non-operational personnel to analyze events, evaluate the
performance of the power system and substation devices, configure devices, or any data that is not required
by operations. This data is often stored outside of the primary control and data-acquisition system, but may
also have a completely independent monitoring system. The data is typically accessed infrequently but can
involve a large amount of bulk data at once. One example is an oscillography waveform capture in
common format for transient data exchange (COMTRADE) (see IEEE Std C37.111™-2013 [B26], which
has been harmonized with IEC 60255-24-2013 [B6]). Another example would be power quality
disturbances (e.g., waveform samples, RMS samples, voltage sag events lists) or ten-minute harmonic data
logs of analog values stored in IEEE Power Quality Data Interchange Format (PQDIF) format (see IEEE
Std 1159.3™ [B15]). Other examples include configuration files, log files, text files, event files, and data
being sent to corporate data warehouses.

It is unlikely that non-operational data requires synchronicity.

5.2 Characterization of data flows

IEEE Std 2030-2011 characterizes data flows using the following data use categories:

a) Reach (distance requirement)


b) Information transfer time:
1) Data occurrence interval
2) Method of broadcast
3) Priority
c) Latency
d) Synchronicity
e) Information reliability:
1) Availability
2) Level of assurance
3) High altitude electromagnetic pulses (HEMP)/EMI
f) Data volume

32
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

g) Security:
1) Confidentiality
2) Integrity
3) Availability

Substation networks should be designed to appropriate system-level requirements defined by these data use
categories. At the substation level of the ICT, key performance is measured using requirements for
information transfer time (message delivery time), latency, and performance during EMI (substation noise).

5.2.1 Data occurrence interval, information transfer time, and latency

IEEE Std 1646-2004 provides guidance for data occurrence intervals (i.e., polling) and information transfer
time for many substation related data flows. The information transfer time defined in IEEE Std 2030-2011
as including both information technology and communications system processes aligns with IEEE Std
1646-2004 definition of application to application. But there is no direct method of testing conformance to
either without the IEDs defining when the time stamp is applied. This is ongoing work on IEEE Std
C37.237™ [B29].

Information transfer time, data occurrence interval, and latency requirements impact system performance.
Generalizations of IEEE Std 1646 requirements for operational and non-operational data flows are as
follows:

 Operational status data changes should be supplied to the operator or control process in the order of
seconds or less. Status data may require shorter intervals than analog data.
 Operational control data should be delivered to its command target as quickly as possible, typically
in the order of milliseconds to a second depending upon the application (i.e., protection versus
operator control). If a command cannot be delivered or acted upon in the required interval, the
application should report this to the operator or control process.
 Non-operational data should be delivered soon after it is requested, but can be delayed or
interrupted as necessary if operational or control data is being processed or bandwidth is limited.

The information transfer time supported by a substation network is highly dependent on the applications
utilizing the network. Some operational data flows, such as message exchange for protection relay
commands, require both short information transfer time (very high speed) and very low network latency on
the order of milliseconds. Other applications, such as file transfers and SCADA, do not require short
information transfer time but can benefit from high speed.

5.2.2 Substation noise sources (EMI)

Electric power substations are sources of electrical noise and transients that can disrupt communications.
Some operation and maintenance activities can make conditions much worse. Although power system
faults are often thought to be the predominant cause of these transients, that is not the case. Power system
faults in distribution, transmission, and generating plant substations may have magnitudes of tens of
thousands of amperes. These fault currents and their harmonics may be coupled to control cables that are
parallel to the power system bus carrying the fault current. In addition, such faults will cause a ground
potential rise (GPR) up to several kilovolts at the source substation, which can damage improperly-
terminated control cables. But these fault currents usually persist from a few cycles to a few seconds, and
some communication systems not involved in protective relaying applications may be able to tolerate
interruptions caused by these faults if no physical damage occurs. See IEEE Std 487™ [B13] for additional
information on protecting cables from GPR.

33
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Lasting much longer are the transients generated when high-voltage air-insulated disconnect switches are
opened. Most of these switches are non-load break—that is, they are designed to just interrupt the charging
current of a transformer, line, or bus section. Compared to the opening speed of circuit breaker contacts, the
contacts of these disconnect switches open very slowly. As they open and the current waveform crosses
zero, the current may remain at zero until the voltage builds up across the slowly opening contact and
flashes over to conduct again. This may happen many times before the contacts are open far enough so that
a flashover does not occur. Each time the interruption occurs, the series L-C circuit consisting of the bus
equivalent inductance and its equivalent capacitance to ground is energized and de-energized and the L-C
circuit oscillates at its fundamental frequency, which typically is in the 0.5–10 MHz range, and so couples
quite easily to control cables that run parallel to the high-voltage bus work. This coupling can result in
substantial transient voltages on the control cables and this led to the establishment of the requirements of
IEEE Std C37.90.1™ [B23], in which they are called oscillatory surge withstand capability (SWC) tests.
The specified test voltage is a damped oscillatory wave with the envelope decaying to 50% of peak value
between the third and sixth periods with a first peak of 2.5 kV.

The power system is not the only source of transients on control system wiring. To improve reliability, it is
common practice in transmission substations to have two high-voltage circuit breakers serving one system
element (transmission line or transformer). These are termed ring bus, breaker-and-a-half, or double-
breaker arrangements. When a fault occurs, both breakers must be sent trip signals from the protective
relay. To maintain separation of the two breakers’ control circuits, auxiliary interposing relays are
commonly used that have two or more output contacts. To minimize battery drain if continuously energized
(as they are in some schemes), these hinged armature auxiliary dc relays have coils with many thousand
turns of fine wire—a substantial inductance. If one attempts to de-energize one of these auxiliary relays
with a slow-acting control switch, the same arcing phenomenon occurs. The transient can be coupled to
adjacent control wiring in the control house and potentially damage components that cannot withstand the
very rapid rise time. This coupling can result in substantial transient voltages on the control cables, and led
to the establishment of the requirements of IEEE Std C37.90.1 [B23], in which they are called fast transient
oscillatory tests. In this case, the rise time is much faster—5 ns—to a peak of 4 kV, a duration of 50 ns with
bursts repeated for 1 s per each polarity.

Roving operators and maintenance workers often use hand-held transceivers (“walkie-talkies”) when in the
yard or in the control house. In the 1970s, several false trips occurred that were traced to such
transmissions. This gave rise to IEEE Std C37.90.2™ [B24] for protective relaying. Its application to
substation communications is defined in IEEE Std 1613.

In recent years, some control houses and control rooms have been built with flooring material that, when
walked on, can cause static electricity to build up on the human body. If the human then touches a
grounded object, the static is discharged onto that object. IEEE Std C37.90.3™ [B25] defines the required
immunity of protective relays to such discharges, and its application to IEDs on a substation LAN is
defined in IEEE Std 1613.

More information on the history of substation environmental conditions can be found in IEEE Std 1613.

6. Security issues and recommendations


This clause provides a list of cybersecurity threats to an electric utility information system, followed by
techniques to address these threats. It is an abstract of informative Annex A, which goes into much more
detail about each of the list items. The goal of this clause and its corresponding annex is to make the utility
engineer aware of the types of security issues that may arise in the course of deploying substation networks.

Clause 6 is not meant to be a comprehensive work on cybersecurity but only to give an overview of the
threats and some of the available methods to minimize these threats. IEEE standards addressing
cybersecurity include IEEE Std 1686 [B18] and IEEE Std C37.240™ [B30]. Other standards also address

34
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

cybersecurity, such as IEEE Std 1815 including secure authentication for DNP3, and IEEE Std 1711 [B19]
for securing serial-based communication channels.

6.1 Threats

The following are some of the individuals and entities that are motivated to attack the utility information
system:

 Disgruntled insiders
 Competing financial interests
 Nuisance hackers
 Terrorist organizations/foreign governments

6.2 Targets of attack

Some utility information system applications that may be targeted are as follows:

 SCADA
 Remote access
 Real-time protection

6.3 Cyber-attack techniques

Examples of some cyber-attack techniques are as follows:

 Mapping: The process of gathering information about a network with the intent of using this data to
exploit a weakness or do damage.
 Ping sweeper: An automated program to determine the IP addresses of machines on the network by
simply observing which addresses respond to a ping message with the intent of using this data to
exploit a weakness or do damage.
 Port scanner: The technique of sequentially contacting IP port numbers on a device and seeing how
it responds with the intent of using this data to exploit a weakness or do damage.
 War dialers: A computer program used to identify the phone numbers that can successfully make a
connection with a computer modem with the intent of using this data to exploit a weakness or do
damage.
 War drivers: The act of locating and possibly exploiting connections to wireless LANs while
driving with the intent of using this data to exploit a weakness or do damage.
 Denial of service (DoS): A deluge of counterfeit network traffic directed at a host computer. The
goal of this attack is to consume system resources and thereby make legitimate access to the
network or system difficult if not impossible. Some common forms are buffer overflow attack, TCP
SYN field attack, smurf attack, and viruses.
 Physical infrastructure attack: Damaging some portion of the communications infrastructure.

35
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

 Application layer threat: Attacking known weaknesses in computer programs or by replacing


applications with “Trojan horse” versions.
 Gaining access: The desire and ability to gain access to the physical network in order to take
control over some portion of the network or the cyber assets on the network. Some common
techniques are password attacks, packet sniffing, spoofing, and man-in-the-middle attacks.

6.4 Recommendations and techniques to address threats in a utility information


system

This subclause addresses some of the methods to mitigate threats.

6.4.1 External audit

Utilities should, at the very minimum, do a comprehensive security review of their operational networks.
Generally speaking, a qualified, reliable third party (outside of the organization) should be retained to
perform this review. It is absolutely critical that utilities do the following:

 Understand their exposure


 Develop a cost effective risk management strategy

On the basis of this audit, the utility should consider the mitigation approaches described in 6.4.2 to 6.4.18
(in all or in part, as appropriate).

6.4.2 Security policy

Every electric utility should create a network security policy to tell people what they can and cannot do
with cyber assets. Full management support as well as thorough understanding throughout the organization
is essential.

A network security policy should be enforced at the defensible boundaries of the network. The policy for
the cybersecurity of the systems in a substation should be a part of the larger plan for the cybersecurity of
the entire utility IT infrastructure. This policy should also be consistent with the policy for physical security
of the utility assets including penalties for violations.

6.4.3 Security awareness

All employees, contractors, and temporary workers should understand the network security policy and be
aware of their responsibility in maintaining the overall cybersecurity.

6.4.4 Password management

The following should be researched and analyzed for applicability to the utility before establishing a
company-wide password policy:

 Procedures for protecting password files and administrator accounts


 Random password generation, one-time passwords, and two-factor authentication

36
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

 Password expiration and renewal


 Procedures for removing employee, ex-employee, and contractor access
 Length and strength of acceptable passwords

6.4.5 Anti-malware policy

The anti-malware policy should specify solutions to prevent the introduction of malicious code on the
substation’s perimeter defense systems, the method for deploying new system elements, and who is
responsible for this activity. The policy should include the steps for reporting an incident, removing the
offending code, and testing for other infections.

6.4.6 Incident handling

The cybersecurity policy should cover the very practical steps that an organization needs to take when a
cybersecurity incident occurs. All employees should be responsible for reporting evidence of an intrusion
or attack.

6.4.7 Backup and recovery

An electric utility needs to have a formal policy for creating and recovering from backups. The policy
needs to emphasize the fundamental importance of backup and recovery processes. The policy should
include details of backup recovery test procedures including detail of frequency, records to be kept, and
personnel responsible. Responsibilities should be clearly documented.

6.4.8 Proprietary information

The cybersecurity management policy should reinforce a company’s formal information classifications and
specify the rules, guidelines, and procedures for the protection of that data. Proprietary information should
be regularly audited in terms of how it is handled, who has access to it, and the level at which it is
protected.

6.4.9 Activity logging and alarming

IEDs frequently have an external or virtual self-status, critical alarm, or some other programmable alarm
status that may indicate intrusion detection but also indicate whether the device is working properly. When
this alarm contact indicates a security indication, the corporate cybersecurity policy should ensure that the
contact status is logged, alarmed, and audited. See IEEE Std 1686 [B18].

6.4.10 Network connections outside of substation

Electric utilities should use private communication channels for network connections outside the
substation. If it is not possible to use a physically private line, strong encryption/authentication techniques
should be used to create a virtually private connection. All utility communications that are routed over
untrusted or public networks should use virtual private network (VPN) technology to protect the channels.

Utilities should also secure the network perimeter with virus scanners, firewalls, and intrusion detection
systems (IDS).

37
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

6.4.11 User access

IEDs or access control systems should implement access restrictions using hierarchies with different levels
of permission for viewing data and changing settings. Utilities should also use strong passwords and user
authentication, in addition to physical security, to guard against unauthorized access.

6.4.12 Connections into IEDs through the network

The following techniques should be used to discourage electronic intrusions and enable electronic
monitoring and trespass prosecution:

 Use secure dial-back, encrypting, or authenticating modems


 Have predetermined timed sessions
 Terminate sessions after a set period of inactivity
 Control remote and monitor access systems via notification of appropriate personnel
 Use warning or appropriate-use banners on login dialogs and screens

6.4.13 Secure modem communications

Secure modems should be used when serial dial-up or leased-line communications use untrusted (not
secure) networks.

6.4.14 Firewalls/connection to the enterprise

Firewalls should be used to guard against external threats and should be placed at various security
perimeters in the system.

6.4.15 Network IDS

IDS should be evaluated as a possible tool to monitor, detect, and respond to unauthorized activity by
company insiders and outsider intrusion.

6.4.16 Encrypting data transmission

Data encryption should be used when communications use untrusted networks. A key management system
and the corresponding policies should be implemented as appropriate for the utility.

6.4.17 Virtual local area networks (VLANS)

Substation switches with controlled per-port VLAN filters can isolate mission-critical traffic from untrusted
sources.

6.4.18 Untrusted networks

All networks should be considered untrusted (not secure) until a robust analysis has proven otherwise.

38
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

7. Recommendations for Ethernet-based networks


All substation networks (both intra-substation and inter-substation) should be designed and equipment
chosen to support the specific requirements for the data flows supported by the communications
architecture and suitable for the substation environment (see IEEE Std1613 for devices and IEEE Std 525
for communication cables), whether serial-based or Ethernet-based. If applications exist on Ethernet
networks that may interfere with each other with regard to timeliness of message delivery, then network
segmentation techniques such as VLANs, message prioritization, bandwidth limiting, subnets, relocating
IEDs higher in the architecture, and physically separate networks should be used on Ethernet-based
networks to reduce network congestion. Similar concepts can be applied to serial-based networks.

Network performance is negatively impacted by hubs, so managed switches are recommended. Network
segmentation can improve network performance, but can also be used to partition the network with respect
to departmental access.

All network switches and routers should conform to IEEE Std 1613 for immunity to electrical substation
noise and other environmental factors, as well as IEEE Std 525 for cables. If communications must
continue uninterrupted during the electrical transients defined in IEEE Std 1613, then it is recommended
that all devices involved in such communications conform to IEEE Std 1613 Class 2. See IEEE Std 487 for
additional information on protecting cables from GPR.

All network cables should be installed per IEEE Std 525.

Consideration should be given to the number of simultaneous connections that can be supported by an IED.
Some IEDs limit the total number of simultaneous connections, and this can affect the network architecture.

Every effort should be made to create a private network exclusively for the substation information system.
If any computers on this network require access to an external network, such as the internet or utility wide
area network, then a router/firewall should be used between the networks. This router/firewall should
support network address translation (NAT) and filtering that limits incoming and outgoing traffic to only
those ports required by the substation information system.

The control of all the access points into a serial-based or Ethernet-based network is often called the security
perimeter. Establishing a security perimeter in compliance with a combination of local/state/national
regulations [such as North American Electric Reliability Corporation (NERC) Critical Infrastructure
Protection (CIP)], IEEE Std 1686 [B18], IEEE Std C37.240 [B30], and others, is important for legal and
security reasons. The main driving force behind the deployment of a substation LAN is its ability to
simultaneously support multiple applications: SCADA, protective relaying, equipment monitoring, power
quality, engineering access, system planning, etc. These applications bring along distinctly different (and
sometimes contradictory) requirements such as reliable data transfer, two-way delivery confirmation, high
data volume, message priority enforcement, or guaranteed delivery time.

Over the years, Ethernet technology was widely believed to be inadequate for mission critical, hard real-
time applications, with multiple studies analyzing limitations of the carrier sense multiple access (CSMA)
technique (Skeie, Brunner, and Johannessen [B57] and Pozzuoli [B56]). Recently, however, fostered by the
explosive internet growth (Holmeide and Skeie [B5]), Ethernet has features that improve support for
predictable real-time network technology (Pozzuoli [B56]) for intra-substation networks.

Protective relaying is a well-known example requiring predictable (real-time) LAN behavior for both intra-
substation and inter-substation networks. This behavior can easily be achieved within individual substation
boundaries and may, when necessary, on a case-by-case basis, be expanded to carry limited number of
time-critical protection messages between substations (inter-substation communication). Network latency
(potentially from bursty protection-related traffic, resulting in network congestion and delays from network
queuing), channel asymmetry, jitter, and network redundancy can cause message delays that do not meet
specific protection application requirements.

39
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Real-time network capability inside substations is achieved by using managed Layer 2 Ethernet switches
and eliminating Ethernet hubs, which cannot provide modern quality of service (QoS) enhancements
necessary for real-time network operation.

Following is a short list of key technologies behind predictable real-time Ethernet for intra-substation
communications. It is given primarily as an explanation intended to help with substation LAN equipment
specification. Although extensive, the entire list [with the exception of items a), b), and j)] can be satisfied
by specifying a “managed Ethernet switch” device.

a) 100 Mbps (with 1 Gbps available) port speed 7


b) Full duplex port operation (IEEE Std 802.3™) and collision-free environment
c) Priority queuing support (IEEE Std 802.1D™)
d) VLAN support (IEEE Std 802.1Q™)
e) Loss-of-link management
f) Rapid-spanning tree algorithm support (IEEE Std 802.1D)
g) Back pressure and flow control (IEEE Std 802.3)
h) Multicast propagation control
i) Remote monitoring, port mirroring, and diagnostic support
j) EMI-hardening and extended temperature range operation
k) Support for media access control (MAC) filtering and port lockout

The use of managed switches is highly recommended in substations, as they substantially reduce network
congestion. Use of managed switches is a typical solution in time critical power systems protection
applications (see Clause 7 and Tengdin, Simon, and Sufana [B59]).

In order to support real-time protection over Ethernet, the substation network infrastructure needs to satisfy
the following minimum requirements:

 Ethernet hubs should be eliminated and replaced with Ethernet switches.


 All Ethernet switches should be managed.
 All switches should support 100 Mbps full duplex operation (10 Mbps support is optional).
 Priority queuing, VLAN, rapid spanning tree, and rapid spanning tree protocol should be supported.
 High-priority network traffic should be allocated to power system protection and high-speed
automation. High-priority traffic volume should be carefully planned, managed, and monitored
throughout the lifetime of the installation.
 Redundant network architecture is highly encouraged for critical applications that require high
availability.
 All Ethernet equipment should be rated for operation in the substation environment.
 The utility’s IT department should be involved early on in the project.
 Network security (especially external access to the substation LAN) should be addressed at the
planning stage and properly managed throughout the lifetime of the installation.

7Legacy networks may operate successfully at slower speeds depending on the functions implemented.

40
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

7.1 Media

The communication system uses media as part of the physical layer for the chosen protocol. Most systems
consist of a mixture of various media and devices with the goal of optimizing communication at each node.
Devices are used to provide interconnection between the various media. The IPS does not specify the link
and physical layers, which is generally considered to be Ethernet and includes the selection of media.

Several considerations come into play in the selection of media for the substation network. These include
the following:

 Required data speed/throughput


 Linear distance
 Physical routing
 EMI and GPR susceptibility
 Number of network-connected devices
 Substation wiring practices/procedures
 Initial installation cost
 Maintenance capabilities
 Vendor device support

As with most engineering decisions of this nature, the choice of media will require compromises on some
issues to achieve the most effective cost/benefit ratio for any given application.

Communication media choices may be divided into two categories: wired and wireless. Each media type
has distinct characteristics that determine the advantages and disadvantages of the media choice. In general,
the characteristics can be summed up as follows:

a) Copper:
1) Advantages: Slightly lower installation cost, widespread device support, easily installed and
maintained when using un-terminated (without connector) cable.
2) Disadvantages: EMI and GPR susceptibility, limited distance, low to medium
speed/throughput, low pulling specifications.

b) Optical fiber:
1) Advantages: EMI and GPR immunity, operation over long distances, high bandwidth, high
pulling specifications.
2) Disadvantages: Slightly higher installation cost, may require new skills for terminations.

c) Wireless:
1) Advantages: GPR immunity, cost is distance insensitive.
2) Disadvantages: EMI susceptibility (both Rx and Tx), cost, low speed/throughput, can be
limited by substation physical configuration, can be made less secure if not configured
correctly.

7.1.1 Media recommendations—wireless

Wireless local area networks (WLANs) can be installed in the harsh conditions of a substation. This
includes environments with extreme temperatures and where radio frequency interference (RFI)/EMI levels

41
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

exist. Therefore, it is imperative that the WLAN system components are up to the task and IEEE Std 1613
be applied to wireless devices installed in substations.

The designer should also match the performance capabilities of the WLAN with the required performance
of the application. When designing systems that utilize WLAN, the application utilizing the data being
transmitted over the WLAN should be designed so that failure does not result in disaster. Consideration
should also be given to testing WLAN devices in the environment that is to be the transmission medium,
which is the substation yard and/or control house. This will ultimately verify vendor performance claims
against system requirements under conditions that will impact WLAN performance, including rain, fog,
temperature extremes, dust, switching, faults, and any local interference.

7.1.2 Media recommendations—fiber versus copper

Although there are several factors that affect the selection of communication media in a substation, the
primary decision for each channel rests on the priority of the data, the proximity of the channel to a noise
source (see 5.2.2), and the existence of GPR. Substation networks must be reliable during outages and other
power system events.

Priority of data delivery includes the following considerations:

 If the messages must go through in a timely fashion, even during an event, then the communication
media must be noise free (e.g., protection messages, breaker status).
 If some message loss can be tolerated, the communication media can have a limited amount of
noise susceptibility (e.g., non-operational data that can be retransmitted, analog values that will be
updated at the next poll interval).
Consideration must also be given to the delays placed on the message to media converters, switches, and
routers.

Proximity of media to noise sources includes the following considerations:

 If copper communication media is close to a noise source or if it is not properly shielded, the
channel will be noisy.
 If copper communication media is far from a noise source and is properly shielded, the channel will
have very little noise.
 Fiber-optic cable is immune to the type of induced electrical noise found in a substation.

If communications must continue uninterrupted during the electrical transients defined in IEEE Std 1613,
then it is recommended that all communication links longer than 2 m be fiber.

Table 5 shows the recommended media given the above criteria.

Table 5 —Recommended communication media for a substation network

Priority of the data delivery

Critical Non-critical
Proximity of Far Fiber or copper Copper
media to noise
source Near Fiber Fiber

42
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

GPR affects the media selection, and fiber should be used in the following conditions:

 If the devices on the channel do not coexist on the same substation ground plane
 If substation fault or switching transient current can flow through the grounding conductors
between devices on the channel

A substation network media example is illustrated in Figure 4.

Figure 4 —Substation network media example

7.2 Reliability

Since the communications equipment will become increasingly important as greater use of the substation
LAN is implemented, it is important to consider the implications of communications equipment reliability
on the overall reliability of the substation LAN. One method to improve the availability of the substation
LAN is to require all communications equipment to be manufactured to the same reliability criteria as the
protective relays; that is, conform to IEEE Std 1613. This may not be feasible in all cases.

One option is to specify substation network equipment with dual redundant power supplies. This will
dramatically increase the mean time between failures for the communications equipment. In addition, using

43
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

one ac and one dc power supply can maximize the advantages of this configuration. This mix of sources
allows the communications equipment to be powered from the station dc battery supply until a voltage dip
occurs. At this time a seamless transfer to the ac supply can maintain operability of the communications
equipment. In this way, the dc voltage dips associated with the starting of large dc motors found in medium
and high voltage switchgear will not affect the network equipment.

Another option is to provide a separate dedicated battery system or provide an inverter or dc-dc converter
compliant to IEEE Std 1613 for the communications equipment power supply, and ensure that the
equipment is installed in a controlled environment (air conditioning and heaters) with at least temperature
monitoring and alarming.

8. Substation network design


The design of a communication system for substation automation includes the following steps:

a) Define messaging system endpoints (internal or external to the substation, or both).


b) Develop the communication system topology.
c) Determine device interoperability requirements.
d) Choose the communication media.
e) Define the communication architecture.
f) Determine the protocols and physical interfaces supported by every IED to be interfaced.
g) Determine the minimum protocols and physical interfaces that are required.
h) Determine if it is more cost effective to provide protocol and media converters or establish
dedicated networks for each combination, or a combination of approaches.

Each of these steps is related to each other. For instance, some topologies may be supported by only a few
types of communication media. Also, the communication topology may restrict the choice of
communication protocol or protocols. In an Ethernet-based communication architecture, a set of protocols
is chosen.

Survey all equipment to be connected, and consider likely future equipment, to determine what types of
ports and connectors are available. Ensure required protocols are available on available ports. Protective
equipment, in particular, may have been chosen for its primary function with little or no regard for
communications. The network designer may be able to influence communications options for equipment
otherwise specified outside communications control.

8.1 Selection of communication architecture

The communications architecture defines the following:

 Physical architecture—The physical connectivity between IEDs, computers, systems, and users
needed to support the specified functionality.
 Logical architecture—The logical connectivity between the IEDs, computers, systems, and users
needed to support the specified functionality, where a serial-based communication architecture has
an extremely limited logical architecture (protocol addresses and communication port settings).

44
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

The system architecture may include simple point-to-point connections and/or an array of complex
interconnected Ethernet-based and/or serial-based networks, depending on the goals and requirements for
the system. Some of the common architectures are illustrated in Annex D.

One approach to selecting a specific physical architecture is one that fits a particular concept, such as an
architecture that matches the physical layout of the substation, some other functionality, or process. In
addition, developing a suitable architecture often requires that the new and the existing be blended together
in a functional accommodation along with a vision toward a future architecture.

This definition stage may uncover the loss of some desired functionality resulting from limitations imposed
by connectivity, such as when IEDs only include RJ45 connections and optical fiber cable is required or
when IEDs only include serial ports and Ethernet ports are required. Resolving these limitations requires an
implementation plan to carry the existing architecture forward to a final architecture that supports the goals
of the system.

8.1.1 Selection of external communications interfaces

While a substation communication network may be fully contained within a single substation, few systems
are implemented without connections to the utility outside the substation through an external interface. As
with other portions of the system, these interfaces need to meet the functional and performance
requirements of the applications being supported over each external connection, at whatever location they
may reside. The availability of communications technology to support outside connections has both an
economic and performance impact on users and the utility.

Communications choices should withstand a critical performance weighted cost/benefit analysis. External
interfaces also expose the substation to the broader environment and, therefore, expose the substation to
hazards from uncontrollable actions by others as well as unauthorized access or intrusion. The interface
between the communications technology and the substation system, thus, becomes a critical component in
meeting the utility expectations.

External communications interfaces serve two distinctly different clients: those that use “real-time” or
“near real-time” monitoring and control for utility operation(s) at operation(s) and engineering center(s);
and session oriented users performing data retrieval, maintenance, and configuration activities. “Real-time”
or “near real-time” monitoring and control interfaces are usually maintained connections, dedicated to a
single user and/or purpose such as a SCADA, EMS, and DMS. This interface often maintains compatibility
between an existing utility EMS or DMS and the substation. Such an interface may take the following
forms:

 A traditional SCADA RTU with hardwired interfaces to substation equipment


 An advanced RTU where its interface is more of a communications gateway to IEDs and their
network(s)
 An RTU like function embedded within a substation host or an automation controller

The typical SCADA/EMS/DMS dedicated communications link operates with a serial-style protocol
(legacy) with low-speed communications technology that should be supported at least until system
upgrades provide for moving to newer communications technology. With analog pathways giving way to
digital communications links that incorporate packet-based protocols, support for carrying the legacy
messages within these protocols should be required. Network pathways are also capable of supporting
legacy messages embedded within TCP/IP protocol links along with other network traffic.

This is an obvious migration path and should be included in the growth plan for external interconnections.

45
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Other outside users typically connect through session oriented communications interfaces and links to
interact with the substation system and/or its IEDs, on demand. This interface is often to a separate
substation network connecting to IED serial “maintenance” ports, isolated from that supporting real-time
monitoring and may take the following forms:

 A public or private switched network, “dial-up”, connection to a modem with a port selector
mechanism to allow the user to connect to a targeted IED
 A dedicated interface device that serves as a connection point for IEDs and an access point for
outside users
 A dedicated WAN interface access device with enough intelligence to route messages to specific
IEDs and perform protocol translation or message “packing” if needed
 A WAN network connection to an internal substation TCP/IP network that interconnects the IEDs
through a gateway, network switch, or firewall

The configuration of the session oriented user interface is driven by the communications capabilities of the
IEDs and the economic justification for supported functionality. The designer/specifier should consider the
communications capability of IEDs in selecting them for the system as they may add hardware and
software to support the required access. Where the system has a population of serial IEDs, the
designer/specifier should develop a plan to accommodate network enabled IEDs as they replace serial
IEDs. The designer/specifier should also be aware of the potential security issues posed by providing
outside access to IED “maintenance” ports.

Any external interface may suffer from reliability issues related to messaging over long distances. In many
instances, such channels pass through media changes that can also impact performance and reliability. In
selecting the approach, the design process should assess the potential effects of the loss of any portion of a
communication path to at least the following considerations:

 Loss of power to an interface device


 Unauthorized access to an interface device
 Failure of an interface device (due to fault or other causes)
 Exposure of the channel media to physical damage
 Response of the channel owner to request for repair service
 The consequences of channel owners reconfiguring channels without notification to users
 Return time of the channel after temporary failure

8.1.2 Selection of internal communications interfaces

Communications interfaces internal to the substation are evolving from simple serial connections to highly
complex networked integrations. The selection of internal interface options is driven by two constraints.
The functionality of the system will determine the supported messaging and who the users will be. Often,
the IEDs selected for the system will constrain the functionality based on the communications technology
they support. Thus, a multi-user wide access architecture concept may be impeded by IEDs that only
support serial communication through one or more configurable ports. This will challenge the growth plan
for the substation integration.

46
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

8.1.2.1 Serial interfaces

IED serial interfaces are generally byte oriented although some legacy SCADA protocols are not. Serial
messaging may be a one-to-one (point-to-point) connection, or point-to-multipoint.

For IEDs providing communication ports conforming to the TIA-232-F standard, the designer/specifier
should be aware of at least the following potential issues using these ports:

 IEDs may not fully support connections to modems or computers that require channel control
signals to be available.
 IEDs may have additional signals and/or power conductors present in the port connector that can be
problematic to users.
 TIA-232-F does not specify isolation between the channel pathway and the communications port or
UART of the device. Isolation is advisable to maintain the reliability of the channel.
 TIA-232-F does not support multi-drop configurations. Additional hardware will be needed to
implement a multi-drop channel.
 TIA-232-F is limited to short distances depending upon the cable quality (capacitance) and baud
rate, typically less than 10 m.

For IEDs supporting multi-drop communication ports with a TIA-485-A interface, the design process
should address at least the following when using these ports:

 TIA-485-A channel control is generally “master/slave” where one device addresses others at a time
to control traffic. There is no mechanism defined in TIA-485-A to mitigate channel contention or
collisions. Some messaging schemes allow the transfer of channel control from one IED to another
to provide multiple device access. Here the serial protocol supports addressing for both sender and
receiver in each message.
 All IEDs on a TIA-485-A channel should use a common protocol. Channel control may become
more complicated by the use of multiple protocols.
 TIA-485-A specifies biasing, terminating, and linearity requirements. TIA-485-A is often deployed
with little regard for these, and often it is not observed to be detrimental. It is recommended that, in
spite of possibly common practice, biasing, terminating, and linearity requirements be implemented
per the standard.
 Many IEDs provide terminating resistors internally. When used, internal terminating resistors
should be checked to verify they are the correct value for the cable being used, and are applied
according to TIA-485-A.
 TIA-485-A does not specify isolation between the channel pathway and the communications port
or UART of the IED. Isolation is required to maintain the reliability of the channel and its
connected devices.
 Round-trip travel time for messages in a TIA-485-A network can impact the ability of a master to
poll all of the IEDs on the network that supports minimum update times. As the number of IEDs
increases, the polling interval decreases in order to support the minimum update time. As this
situation reaches a critical limit, it is important to configure the master polling time appropriately or
decrease the number of IEDs by increasing the number of channels being used.
Serial interfaces are supported by a communication process within the IED. This process may be based on
terminal emulation (e.g., VT100), a proprietary protocol, or standard protocol. The design process should
address at least the following port support characteristics:

 Integrating IEDs with terminal emulation interfaces beyond simple session oriented single user
connections requires custom software, and some form of communications-processing devices

47
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

deviceto extract information and perform controls. The communications processing device
emulates the activities of a human with a terminal to interact with the IED. Some IEDs will require
the communications processing IED to provide a dedicated port for each connected IED. While
these IEDs can be successfully integrated into a “system” this can be an expensive, single product
and short-lived solution.
 Protocol based interfaces can be easier to integrate, especially since they tend to support multiple
IEDs per channel (port) and are addressable. If an IED-specific protocol is based on a common
protocol, for example, Modbus or IEEE Std 1815 (DNP3), creating the bridge between the system
and the IEDs can be less expensive. Still, a communication processing interface will be needed,
although the supplier of such an IED may only need to configure the bridging software rather than
create new software from scratch. Some suppliers have access to a large library of such software
and may be able to create the bridge at a reasonable cost. The interface IED, however, can become
a significant risk as a single point of failure. This can also be a short-lived solution as the interface
software may leave the user with no path to integrate newer version IEDs without reinventing the
interface.
 Standard protocol-based interfaces can have a significant advantage. Some systems can
communicate with them directly without an interface IED. When an interface is required, creating
an interface connection is simplified significantly by the availability of the standard protocol.
However, some differing interpretations of the standard may lead to difficulties. It is more likely
new IEDs, or even IEDs of a different manufacturer, can use the interface, therefore, it may have a
longer useful life.

8.1.2.2 Ethernet-based network interfaces

Ethernet-based systems assemble messages in variable length packets. Each packet contains the message
data bytes in whatever form the sender and receiver understand. To traverse the network, a network
protocol “wraps” the message data in an envelope that network devices understand. Network devices do not
need to understand the contents of the message. The network protocol may carry data bytes representing
any kind of transaction, in any form, and of any size. Network devices will use as many packets as
necessary to complete the transaction.

While serial messaging connects the sender and receiver directly, network messages may pass through
intermediary devices to reach their destination. Moreover, the route taken by packets from the sender to the
receiver may change without changing any parameters at either the sender or receiver.

Network messaging assumes that any device may exchange messages with any other device. Special
message handling is required to “route” messages only between specific senders and receivers. This service
is performed, not by the devices, but by intermediary devices in the network pathway.

Unlike a serial pathway, networks can support multiple users and devices quasi-simultaneously. While only
one message packet can traverse the network at a time, the source and destination can be anyone with
different sources and destinations interspersed. Messages using multiple packets are not necessarily
contiguous.

8.1.2.3 Serial servers

It is possible to bridge the gap between network devices and serial devices with a network adapter,
typically called a serial server, terminal server, or port server. This IED connects to serial devices on one
port and Ethernet-based networks on another. When considered as part of the communications architecture,
the design process should at least consider that serial servers are as follows:

 A patch put in place to allow serial device to become part of an Ethernet-based system, while the
long-term plan evolves to replace them with Ethernet-enabled devices.

48
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

 A special functionality that is usually IED specific. Therefore, including them in a system design
adds significant costs and upkeep to the system. Just like any IED, not only will they require
periodic firmware updates, but a serial server may require version updates any time there is a
software or firmware change in the connected IEDs.
 A gateway or data concentrator supports a number of serial devices and communicates to them in
their native protocols. It makes the result of these transactions available to the network in whatever
form is compatible with the network. This function may reside in a special processor, sometimes
supplied by an IED supplier to bridge the gap in their product offering. Some RTU suppliers offer
this function as a natural extension of their RTU product offering. This function is sometimes
embedded within a substation processor that may be providing an HMI or data logging service.

8.2 Functional requirements

The basic functional requirement of any communications system is that it supports the various
communications methods that may be required or desired. If an “all new” installation is specified, a study
should be undertaken to determine the following:

a) All data sources and destinations


b) Expected routing for above including alternate routes where required
c) Delivery and refresh time for all data sets and control messages
d) Expected flow in the short and long term for normal and out of normal power system conditions
e) Appropriate media and technologies
f) Electrical and physical isolation requirements
g) Estimate of preliminary cost
h) Requirements for continuity of service
i) Privacy, security and access control requirements
j) Requirements for electrical isolation
k) Requirements for environmental withstand

If the communication system incorporates any part of an existing system, such as existing communications
interfaces built into existing facilities, microwave systems, wire-line systems, or optical fiber optic systems,
the functional requirements include detailed descriptions of these “pre-existing” conditions. They also
define how any pre-existing facilities are to be integrated into the communications system.

Regardless of what the final system may look like, a complete description of the functional requirements of
the system should be written, along with a plan to get from existing to the final configuration.

8.2.1 Reliability

It cannot be assumed that any given stream of bits to be sent between IEDs will be delivered without error
to the output at all times. A communication channel consists of an assembly of wires and cables, electronic
equipment, power supplies, terminations, towers, and other apparatus. Each individual communications
channel should be reviewed from end to end with a list of all equipment that is involved. Take into account
possible redundant paths and a combination of series and parallel-connected equipment. If there is any ac
dependent equipment in series with the channel, it should be assumed the channel will not function in the
event of a power failure or “blackout”.

49
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Reliability calculations can be complicated when public facilities are involved. Many telephone systems
include significant redundancy in common equipment. However, very few, if any, public telephone systems
are designed for 100% service. Dial-up circuits are particularly vulnerable to overloading (access denial)
during emergency. Dedicated virtual private networks can be more reliable, but there is no guarantee that a
higher priority user (government or public safety, for example) will not preempt the virtual network. Very
few public network facilities are ac independent.

In the reliability assessment, identify points in the network that represent a single point of failure. The
assessment should include how each element (or its function) is mitigated if its failure impedes critical
functions. Consider duplicating sensitive items with functional equivalents, but consider using different
suppliers so that a weakness in one device does not propagate to the redundant device. If an identified
failure impedes performance, determine if that impediment is tolerable or if replicating the equivalent
performance is not economically justifiable.

Consider segregation of pathways. The network can become vulnerable to failure if all communications
pass through a single device (e.g., communications isolator) or is routed through a common space such as
an exit cable, cable duct, communications tower, or even an equipment room. Segregating network cabling
within the substation may also afford some protection for accidental damage or a disaster. Segregating
pathways external to the substation reduces the system’s exposure to natural and manmade disasters.

Segregate LANs and WANs by critical function. It is important that critical functions be maintained under
all reasonable circumstances, while secondary functions may be compromised without undue effect on the
users. Times of stress on the utility system can bring significant traffic in secondary data transfers that
could slow the delivery of critical functions. The added expense to assure critical functions is often money
well spent during times of stress.

8.2.2 Performance

The system relies on the performance of the communications links to satisfy the requirements for data and
control both inside and outside of the substation. Assess the data flow over the network elements during
normal and stressed conditions of the power system. A network may be robust enough to handle normal
condition traffic but may bog down or “crash” during periods of high activity. Note that some systems rely
on abbreviated messaging techniques such as change reporting lowering traffic and, thereby, lower costing,
but even these techniques are vulnerable to overload during stress on the power system when many
parameters are changing very rapidly. This condition needs thorough evaluation in the design stage.

8.2.3 Establishing priority pathways

The most common measure of performance is the bit error rate (BER). A BER of 1 × 10−5 is typical of an
unconditioned voice grade channel, and it is recommended that this be the worst possible BER that is
acceptable. This means that, on average, 1 bit out of 10 000 transmitted bits will be in error. If this were an
evenly spaced distribution, it would not be that big of a problem and it would be fairly easy to develop an
error correcting code that would compensate for the errors. However, errors in communication circuits tend
to occur in bursts, therefore, most communication protocols contain techniques to at least detect errors and
initiate a retransmission. Extended outages of communication channels due to weather or other conditions
are usually better analyzed under the category of reliability.

8.2.4 Security

A communication channel should be immune to unauthorized access. Unauthorized access includes such
activities as taps, eavesdropping, bit substitution, spoofing, etc., or just random interference. A detailed
analysis of each channel is required to help ensure immunity. Inadvertent access is a common cause of

50
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

communication problems. Channel connections can appear in many locations, such as patch and connection
boxes, main frames, etc. A technician using probes to discover a noisy pair, or unused pair, can often cause
problems as the probes are run down the terminal blocks. Most often it is impossible to ensure that the
communications channel is highly secure. The security function is therefore usually applied to the
protocol(s) used on the communications channel.

9. Interface and processing


This clause provides recommendations for establishing interface and processing requirements for control
and data acquisition equipment.

9.1 Communication interfaces

IED network communication interfaces should meet the requirements specified in IEEE Std 1613, whether
serial-based or Ethernet-based. However, the following two characteristics are common to internal and
external modem interfaces and should be measured regardless of the configuration utilized:

 Surge withstand capability measured between data communication equipment and the
communication channel is defined in IEEE Std 1613.
 Channel BER is measured between data communication equipment and control and data acquisition
equipment. Due to the variety of channel and modem qualities available and in use, an average
value of 1 bit error in 104 bits is recommended for design and analysis purposes. For Ethernet
network systems, the BER objective should be below 1 bit error in 1010 bits.

9.1.1 Serial ports not connected to external data communication equipment (e.g., modems)

TIA-232-F and TIA-485-A are commonly used in the electric utility industry, and sometimes TIA-422-B.
TIA-530-A is a high-speed interface that may be used. Refer to TIA-404-B for asynchronous DCE, and
TIA-334-C for synchronous DCE.

9.1.2 Signal quality standards interface characteristics to internal data communication


equipment (e.g., modem) when the modem is provided as an integral part of the control
and data acquisition equipment

The following are the interface recommendations:

a) Signal impedance. All inputs and outputs should be balanced 600 Ω ± 10% whenever signal rates
require standard voice grade channels.
b) Signal level. Input (receive) levels may range down to −45 dBm and output (transmit) levels should
not exceed 0 dBm. The output level and receive sensitivity should be adjustable in steps of 4 dB or
less.
c) Signal stability. All inputs and outputs should be stable within ± 1 dB for at least one month
without adjustment.
d) Signal linearity. The output (transmit) level should be linear within ± 1 dB over the output level
range and frequency allowed. Input (receive) level linearity and delay distortion are not defined and
should be specified for each channel type and data rate required.
e) Signal distortion. All input and output signals should not contain RMS harmonics that exceed 2%
whenever signal rates require standard voice grade channels.
f) Signal carrier. Center frequency and bandwidth should be specified.

51
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

9.1.3 Protocol selection

As part of the design process, protocol(s) are selected between the following:

 External data flows to the substation, such as SCADA master(s) to RTU(s)


 Internal data flows within the substation, such as between the RTU and IEDs

Besides defining the protocols being used, the design should also address the following:

a) Number of channels
b) Channel considerations
c) Error detection techniques
d) Channel switching
e) Number of RTUs per channel and/or channels per RTU
f) Number of retries each attempt
g) Time out value(s) by message type
h) Communication error reporting, failure criteria, and recovery
i) Channel quality monitoring (normal and backup)
j) Channel diagnostic/test provisions
k) Equipment interfaces
l) Polling methods, such as report-by-exception or point scan, or both

Systems with report-by-exception functions need to support the bandwidth required whenever all data is
reported plus any periodic update purposes.

The communication hardware related performance capabilities used in the calculation of scan cycle must be
defined.

9.1.4 Channel considerations

Annex E provides a suitable method to calculate the routine loading of SCADA equipment, which may be
used to evaluate the adequacy of a channel configuration. The channel routine load factor is defined as the
worst-case fractional channel operating time required to complete all routine (i.e., periodic) data and
control transactions between the master station and the attached RTU(s). Where aperiodic transactions are
expected, for example, when report-by-exception techniques are used, estimates of average and peak
aperiodic events should be included in calculations of channel capacity needed.

It is recommended that the channel routine load factor, with fully expanded RTUs, should be less than 0.75.
The remaining channel capacity will then be available for occasional retransmissions and for event-driven
data and control transactions. To minimize the incidence of message failures due to channel errors, it is also
recommended that transaction lengths be limited to about 250 channel bit times. When the channel BER
reaches 1 in 10 000 bits, 2.5% of such transactions will fail. Automatic repetition (retry) of transactions that
fail may be implemented, but no more than three consecutive retries should be permitted. If the transaction
is not completed satisfactorily at the fourth attempt, the relevant RTU, or the channel itself, should be
declared at least temporarily inoperative.

52
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

9.1.5 Proprietary links

The communications link between the RTU network controller and I/O modules can be a proprietary design
of the RTU supplier. When these I/O modules are contained within single or multiple RTU cabinets using a
proprietary communication network, the definitions and descriptions of this link are beyond the scope of
this recommended practice. When these distributed I/O modules are installed outside the RTU cabinets,
outside in the substation yard, or use Ethernet networks, the appropriate serial or network requirements of
this recommended practice should apply.

10. Testing
Network communication tests are specific to the type of communication media being used (copper, fiber,
and wireless) and network (Ethernet, “RS232”, “RS485”, etc.). Successful testing usually results from
implementing the following three test phases:

a) Planning the test (refer to 10.1)


b) Creating the testing plan (refer to 10.2)
c) Performing and completing the testing plan (refer to 10.3)
The testing plan should include the following tests as required: 8

 Physical layer testing (refer to 10.3.1)


 Interoperability testing (refer to 10.3.2)
 Network testing (refer to 10.3.3)
 Performance testing (refer to 10.3.4)
 Cybersecurity testing (refer to 10.3.5)

Procurement, vendor, and design specifications should be considered in test planning.

10.1 Test planning

Proper test planning typically supports the successful completion of any testing because the following are
addressed during planning:

 The testing is included in the overall project plan and subsequently included in the project schedule
where its status is properly tracked.
 The time required to completely perform each testing phase is determined and included in the
project schedule (planning, creating, and performing).
 The tests can be prioritized such that any required changes due to unforeseen scheduling issues can
be easily identified and adjustments made to the plan.
 Definition of the testing, including the logistics of the testing, such as who is performing the tests
(and the requirements around certifications, qualifications, and experience), where the tests will be
performed (such as any of the following: a laboratory setting, a vendor’s location, and at the

8
The communication test method specified here is adapted from IEEE Std C37.115-2003 [B27], which has been administratively
withdrawn. Previous versions of IEEE Std 1615 have referenced IEEE Std C37.115. Much of the information in IEEE Std C37.115
remains relevant and valuable, so Clause 10 adapts and updates text from Clause 6 of IEEE Std C37.115.

53
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

installation site), what the expected results are for each test, and what equipment and tools are
required to support the tests (that could enable the automation of some testing procedures). For
example, the definition of the testing should address whether certain testing techniques, such as
penetration testing, are performed because they were included in the specified requirements and
what communication networks are involved in the penetration testing. A rigorous testing definition
can be structured into a testing plan framework.

10.2 Testing plan framework

The following should be included in a test plan for testing the communication network(s):

a) Revision block (refer to 10.2.1)


b) Introduction (refer to 10.2.2)
c) Approach (refer to 10.2.3)
d) Tests (refer to 10.2.4)
e) Action items (refer to 10.2.5)

NOTE 1—Testing of the communication network(s) is typically performed early in the system test, prior to the testing
of higher level functions/applications supported by each communication network. This approach should minimize the
amount of testing performed by reducing failures and re-tests at the higher levels. 9

NOTE 2—Testing plans may become fairly complex and modularized into a series of smaller testing plans. Master
testing plans may then be used to track a series of smaller testing plans.

10.2.1 Revision block

A revision block should be used to track revisions to the testing plan, submittal of the testing plan for
reviews, and final approval of the testing plan. The revision block can also be used to track the submittal of
testing results and submittal after completion of all action items.

10.2.2 Introduction

The testing plan introduction should include items such as the following:

a) Objectives (goals) of the testing plan, at a summary level. The objectives should address whether
the tests take place as a factory acceptance test, site acceptance test, and/or commissioning.
b) Test items, a summary list of what will be tested.
c) Test descriptions, a high-level description of the tests being performed on what equipment and/or
components, with justifications for anything that will not be tested. Refer to 10.3.
d) Resource requirements (people and tools) for the tests being performed. This includes the
environment from which the testing will occur.
e) Expected test duration (based upon budget constraints).

9
Notes in text, tables, and figures of a standard are given for information only and do not contain requirements needed to implement
this standard.

54
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

f) Any constraints and limitations of the testing plan and/or test results, such as some results that may
be sensitive because of their contents and some testing activities that may require signing of non-
disclosure agreements (e.g., cybersecurity testing).

10.2.3 Approach

The approach summarizes the tests to be performed, such as a simple list of all of the tests that includes a
name, short description, priority, expected duration, required test equipment, and a short description of the
test.

The approach may also identify the following items:

 A sample product is required for testing


 Instruction manuals detailing the operation of the product and installation specifications
 Vendor test configuration hardware to support a simulated target-operating environment (e.g.,
power supplies, sensors)
 Vendor test configuration software to support a simulated target-operating environment

10.2.4 Tests

This section of the testing plan includes the details for each test to be performed. Examples of tests are
discussed in 10.3. Each individual test should include information such as the sources of test data, inputs
and outputs, testing techniques, and priorities. These sections define the guidelines for requirements
analysis, scenario development, acceptance criteria derivation, and how to construct and execute the tests.

This section documents the ties between the tests and specific requirements. This section includes the
expected results and the actual results. In general, the test documentation should address the following:

 The repeatability of the test


 The types of failures that might be expected
 An evaluation of how the component(s) handled the failure (e.g., reboot, recovery, catastrophic
failure)
 Whether a re-test is required and/or an item is added to the action item list (punch list) to fix an
identified problem before the next re-test can occur

10.2.5 Action items (punch list)

This section of the testing plan identifies items encountered during testing that require resolution. This
section is used to track the action items until all are successfully completed.

10.3 Testing

Any complex network has components (e.g., an Ethernet network will likely have more than one Ethernet
switch, router, firewall, serial server, cable, ports). The tests may be constructed to identify whether the
correct component for each location has been selected [simply by type of communication media being used
(copper, fiber, and wireless)] and the layer 1 and/or layer 2 of the network (Ethernet, “RS232”, “RS485”,
etc.). Other tests confirm that the components can handle the expected amount of network traffic, distance
of network segments, and network topologies involved. The tests covered in this subclause are as follows:

55
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

a) Physical layer testing


b) Interoperability testing
c) Network testing
d) Performance testing
e) Cybersecurity testing

Higher-level errors should be monitored during testing. This is typically accomplished using network
analyzers and/or error statistics in the network components, such as Ethernet switches.

10.3.1 Physical layer testing

The conformance testing confirms that the network being tested at the physical layer (layer 1 and,
potentially, layer 2) reacts correctly to errors [e.g., frames with cyclic redundancy check (CRC) errors are
discarded] and that the thresholds on errors are within specifications under both static and dynamic
conditions. Physical layer tests (e.g., BER, power, signal level, pulse shape, and timing tests) ensure
conformance to specified values for the physical communication link (e.g., fiber, copper, or radio).

Static physical layer tests check the list of features and options claimed to have been implemented in the
network. It is acceptable to use results from conformance test labs for components making up the network
under test as an alternative to performing exhaustive tests on the physical layer (e.g., signal shape). A
subset of tests may be conducted to verify that the items, such as BER, are being met by the equipment in
the installed environment.

10.3.2 Interoperability testing

Interoperability testing verifies the data link connectivity (e.g., when each end of the communication link
has a component manufactured by different vendors, but claimed compliant to an industry standard such as
IEEE Std 802.3-2018). These tests may include the following (depending upon the communication link
technology):

 Address translation and mapping table functionality


 Protocol translation verification
 Protocol encapsulation and decapsulation
 Congestion indication translation
 Connection status management
 Link failure modes

10.3.3 Network testing

Network testing confirms that the higher-level protocols are functioning per their specification. For
example, on a serial network supporting only IEEE 1815 communications (DNP3), this test confirms that
the DNP3 devices are establishing a correct connection without unacceptable communication errors (e.g.,
timeouts, CRC errors).

For IP networks, testing of the end-to-end QoS for all required traffic is essential. The key network quality
parameters that affect IP QoS are packet delay, packet loss, and delay variation (jitter). IP tests should be

56
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

performed to monitor the data exchange between each end so as to address the following interoperability
problems:

 Routing table errors


 Failure of session connection negotiation due to terminal capabilities mismatch
 Configuration errors
 Packet loss
 Packet delay
 Packet jitter

In addition, tests for IP networks should also include tests that ensure proper (as designed) routing and
switching occurs across the network. It may be helpful for this test to use a network capture of all
communication traffic to evaluate whether all addresses are active as designed. The test may emulate
message traffic at the endpoints.

10.3.4 Performance testing

Performance tests help ensure that the network performs in accordance with the specified requirements. In
general, this includes network load tests and confirmation of message delivery times under varying network
load conditions.

Dynamic conformance tests check the communications system under load conditions. Tests for Ethernet
networks should load the communication network with background traffic so as to test the communication
system when it is moderately busy. Tests for serial networks, especially RS485 networks with multiple
IEDs, meet the polling rates required by the specified requirements.

In general, this recommended practice can only specify the testing of the message delivery time. Message
delivery time should be defined in the specified requirements so as to limit confusion in measuring
performance. One definition is the time required to send a message from the sending application interface
to its receipt at the receiver application interface. IEC 61850-5 [B12] includes a different specification of
message delivery time.

Functional performance can also be tested in parallel with message delivery times, but the functional
performance of higher-level functions supported by the communications network is outside the scope of
this recommended practice because it measures the time required to communicate, from the sender to the
receiver, the information needed to execute the specified function, plus the time to execute the function.
Such a function might be a protection function, which is outside the scope of this recommended practice.

Application response time testing can also be tested in parallel with message delivery times and functional
performance tests. This test measures how long it takes an application to complete a series of tasks, and
best represents the utility’s perception of the network system (application network operating system and
network components).

Feature testing can also be tested in parallel with message delivery times to verify individual commands
and capabilities of the application(s) being supported by the communications network(s). Functional testing
is outside the scope of this recommended practice because it verifies that the features work correctly.

Tests for message delivery time should be run at various network loads with real or emulated IEDs to
create a load versus response time curve for each network tested.

57
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

10.3.4.1 Regression testing

Regression testing should be performed when a new release of hardware or software (firmware) is released
for any device connected to the communications network (IEDs, Ethernet switches, converters,
transceivers, etc.) to compare the performance, reliability, and functionality to the existing network and
ensure that the product upgrades will not adversely impact the operational network.

10.3.4.2 Throughput testing

Throughput testing should be performed to measure data transfer rates to evaluate performance against the
specified requirements, which should include a maximum sustainable throughput on the communication
network(s) without impacting message delivery times. Throughput testing should then be run to determine
whether the communication network(s) function(s) at its capacity limits.

10.3.4.3 Reliability testing

Reliability tests should be run as required by the specified requirements. As required, the test is run over an
extended period (e.g., 24 h−72 h) and under medium to heavy network traffic. The reliability test monitors
the network for errors and failures.

Reliability testing may also check the communications network’s ability to correctly handle faulty traffic.
Graceful recovery from a loss of signal, loss of synchronization, or discarded traffic should be included in
faulty traffic tests to show that the hardware and software failures do not cause higher-level errors.

10.3.5 Cybersecurity testing

Any communication network may include elements of cybersecurity that may be tested using a variety of
tests, such as the following:

 Documentation and design review, which checks that the actual system implementation matches
the system as documented and/or designed. This relates also to system hardening, which confirms
that only the ports and services that are allowed in the system design and/or documentation are
enabled.
 Log review, which must be performed on a test system that is generating the specified logs as part
of its design. 10
 Ruleset review, which must be performed on a system that has devices with rulesets that define
how communication is allowed (e.g., a firewall).
 Network sniffing, which must be performed at proper levels in the communication network so that
all communication traffic can be observed and validated against the documentation and/or design.
Network sniffing may capture more information that just protocols and ports, but also active
systems, operating systems, and applications. Active network sniffing (e.g., sending specifically
crafted messages to devices on the network and observing the responses) should be carefully
evaluated for causing failure or other problems from which recovery could be very difficult.
Passive network sniffing, may be less problematic, but could still require physical and/or logical
reconfiguration of the network from its design.
 Penetration tests, which can be performed from inside, outside, and between the designed security
zones. Penetration testing attacks the network to identify methods for circumventing its security

10
Example logs are those that indicate successful and failed network access, network errors, malicious network traffic, outbound
connection attempts that could indicate some kind of network compromise, etc.

58
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

features. Realistic attacks are launched by using tools and techniques commonly used by attackers.
While penetration testing can be invaluable in evaluating whether security features are
implemented correctly, this type of testing could damage or render inoperable any part or portion of
the system being tested. Recovery from damage incurred from penetration testing could vary from
a simple reboot to complete replacement. Penetration testing is not recommended without proper
planning.

NOTE—Specific cybersecurity tests may be required by regulations (e.g., NERC CIP).

59
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Annex A

(informative)

Security issues and awareness

The electric power industry is increasing its reliance on both automated control systems and
microprocessor-based protection systems. The need to provide remote access into these systems, along with
the rise in computer attacks and international terrorism, has magnified the risk of unauthorized hostile
access into these automation and protection systems. By extending communication networks out to the
substation, electric power systems are facing an increased threat of sabotage and espionage via electronic
intrusion and computer hacking. The electric power industry must be aware of this threat and must take
steps to reduce risk and mitigate vulnerabilities. Protection, communication, and automation engineers
should take the necessary steps to minimize the likelihood that a person with hostile intent could adversely
affect the electric power system.

The purpose of this annex is to give the reader an overview of the threats that exist to their network, and
some assurance techniques that address the threats. This annex is not meant to be a comprehensive work on
cybersecurity, but only to give an overview of the threats and some of the available methods to minimize
these threats.

A.1 Source of threats to the electric power system

The following are some of the individuals and entities that are motivated to attack the electric power
system:

 Disgruntled insiders: Insiders are those who have legitimate access to company systems. They
typically have more opportunities and greater knowledge of the system than those who are outside.
Their motivation is typically either financial opportunity or revenge. The current widespread
practice of using third parties to outsource certain functions that formerly were done in-house blurs
the line between insiders and outsiders.
 Competing financial interests: As deregulated markets are created and more stakeholders
participate in the delivery of electric power, there is an increasing number of individuals or
corporations that can benefit from stealing proprietary information.
 Nuisance hackers: Nuisance hackers are incredibly common on the internet. They send out probes
looking for any interesting targets. They will send out automated viruses, Trojan horses, and attack
scripts with random targets; they will try to target data theft, electronic vandalism, and denial of
service. Many times they are just trying to exploit the vulnerabilities simply “because they can”.
The higher the profile of the target, the better. Professional hackers hide behind the nuisance and
recreational hackers. Their goal may be revenge, financial gain, or any number of potential
benefits. Electronic fraud and espionage were mostly recreational, but now are starting to be big
business.
 Foreign governments: There are countries that have government-sponsored information warfare
programs with both offensive and defensive applications. These countries see information warfare
as a way of leveling the playing field against a stronger military power. It is likely that these attacks
will be on basic infrastructure, which includes the electric power grids and the telecommunications
networks as primary targets.
 Terrorist organizations: Recent evidence acquired from seized computers points to increased
interest in distributed control systems (DCSs) and supervisory control and data SCADA equipment
associated with critical infrastructure. Evidence shows that members were visiting Web sites that

60
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

offer software and programming instructions for these devices. Established terrorist groups are
likely to view attacks against information systems as a means of striking at government,
commercial, and industrial targets from offshore, believing there is little risk of being caught. The
bottom line is there are motivated and well-funded individuals intent on harming their targeted
group or country. A successful attack on the electric power grid would accomplish this goal, as the
consequences of a successful attack are significant. These observations alone justify the need to
assess and mitigate the vulnerabilities in our electric power infrastructure.

A.2 Targets of attack in the electric utility network

There is no shortage of individuals and entities that might want to attack the electric power industry. One
reason why electric power industry has become more vulnerable to attacks in recent years is because of the
need to access valuable information from the substation to better operate and maintain the electric grid. The
recent widespread use of automation systems in substations with interconnected intelligent electronic
devices (IEDs) tends to open up additional outside access points into the substation. There are also more
joint-use substations with the break-up of utilities into various operating companies. Regional regulatory
agencies such as regional transmission organizations (RTOs) and independent system operators (ISOs)
have access to large numbers of assets. Each of these organizations could be using independent public
network access into the substations. This could further blur who is responsible for the overall network
security of the facility. Contracting third-party companies to do remote maintenance can add to the
confusion.

Figure A.1 shows some areas that may be targeted.

61
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure A.1—Points of vulnerability to the electric utility network

A.2.1 SCADA

SCADA could be vulnerable to cyber-attacks for a number of reasons including:

 Insecure communication links between the master station and the substation.
 Undocumented vendor backdoors into the EMS (energy management system), substation remote
terminal unit (RTU), IED, or network.
 Default or weak passwords.
 Devices that do not support strong passwords.
 Operating systems that are not security hardened.
 Poorly implemented engineering access into both the master station and the RTUs
 SCADA and IED protocols that are not designed with security support.

62
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

 Modern SCADA and IED protocols are becoming more standard and the documentation and
knowledge base is easy to obtain.

A.2.2 Remote access

Remote access is typically user-driven communication; it is typically not automated. It may be used to
verify or change settings in a remote IED; it may be used to check the device’s status or to download event
reports. Common implementations include dialup modems (publicly accessible) and TCP/IP-based
communication over a WAN. Remote access will typically be carried out over publicly accessible network
infrastructure. Dialup modems provide remote access to the vast majority of sites (substations, etc.) in the
electric power industry. Since dialup connections are implemented over the public switched telephone
network, anybody can send data to the modem. Only electronic security measures can prevent access by
attackers. Obscurity is not an acceptable security measure. For larger sites, it is increasingly common to see
TCP/IP-based WAN solutions. In such solutions, typically all communications “conversations” between
two sites are multiplexed over the same WAN connection. Common WAN media solutions include leased
lines, fiber, and microwave radio.

Sometimes automatic remote access is utilized to obtain metering, maintenance, operating, and engineering
information from the remote system. This information can be sensitive to the corporation, and safeguards
must be taken to protect it against attacks. It is not only important to protect and control access into the
network, but it is also critical to protect transmission of the data. Even if the data is not of critical
importance, sending passwords without encryption provides information attackers can use to access the
network.

A.2.3 Real-time protection

Even though real-time protection is not usually considered vulnerable to attacks, as substation
communication networks begin to be used for these applications, this will become more of a concern.
Protection applications using networks are generally peer-to-peer relay schemes. The applications include
various pilot schemes, direct transfer trip, and current differential. These schemes can be executed over
many different types of media including power-line carrier, fiber or wire, and radio (typically spread-
spectrum). Currently, real-time protection communications are not highly interconnected. These
connections are typically a single pair of relays, positioned at opposite ends of a transmission/distribution
line, exchanging protection data.

A.3 Common cyber-attack techniques

Some of the various attack techniques that are in widespread use today can be listed and defined.

A.3.1 Mapping

The process of gathering information is known as mapping. The attackers are in a data-gathering phase.
They are collecting information such as assigned IP addresses, phone numbers, physical location of
company assets, employee names, operating systems, and the services being used. They may try to trick
insiders into thinking that the attacker is in fact another insider to obtain some of this information. The
purpose is clear—the more one knows about the target before attacking, the higher the probability of
success. With this information, attacks can be more focused and are less likely to be detected. Some of the
tools used to “map” a target are as follows.

 Ping sweeper: An automated program to determine the IP addresses of machines on the network by
simply observing which addresses respond to a ping message.

63
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

 Port scanner: Port scanning refers to the technique of sequentially contacting IP port numbers on a
device and seeing how it responds. The responses can be used to determine what services they
might offer (i.e., FTP, Telnet, or HTTP).
 War dialers: A war dialer is a computer program used to identify the phone numbers that can
successfully make a connection with a computer modem. The program automatically dials a
defined range of phone numbers and logs and enters in a database those numbers that successfully
connect to the modem. Some programs can also identify the particular operating system running in
the computer and may also conduct automated penetration testing. These programs can run through
a predetermined list of common user names and passwords in an attempt to gain access to the
system.
 War drivers: War driving, also called access point mapping, is the act of locating and possibly
exploiting connections to WLANs while driving. War driving requires only a vehicle, a computer,
a wireless Ethernet card set to work in promiscuous mode, and some kind of an antenna. Because a
wireless LAN may have a range that extends well beyond the fence of a substation, an outside user
may be able to intrude into the network, obtain a free internet connection, and possibly gain access
to substation network and other resources.

A.3.2 Denial of service (DoS)

A DoS attack consists of a deluge of counterfeit network traffic directed at a host computer. The goal of
this attack is to consume system resources and, thereby, make legitimate access to the network or system
difficult if not impossible. Typically, the loss of service is the unavailability of a particular network service,
such as SCADA, or the temporary loss of all network connectivity and services. A DoS attack can also
destroy programming and files in a computer system. A DoS attack is a type of security breach to a
computer system that does not usually result in the theft of information or other security loss. However,
these attacks can cost the target utility a great deal of time and money.

Some common forms of DoS attacks are as follows:

 Buffer overflow attacks: The most common kind of DoS attack is simply to send more traffic to a
network address than the programmers who planned its data buffers anticipated someone might
send. The attacker may be aware that the target system has a weakness that can be exploited or the
attacker may simply be trying the attack to see if it succeeds.
 SYN attack: When a connection is initiated between a Transmission Control Protocol (TCP) client
and server in a network, a very small buffer space exists to handle the usually rapid “handshaking”
exchange of messages that sets up the session. The session-establishing packets include a SYN
field that identifies the sequence in the message exchange. An attacker can send a number of
connection requests very rapidly and then fail to respond to the reply. This leaves the first packet in
the buffer so that other, legitimate connection requests cannot be accommodated. The packet in the
buffer will eventually time out after a certain period without a reply, but the effect of many of these
bogus connection requests is to make it difficult for legitimate requests for a connection to get
established.
 Smurf attack: The attacker sends an Internet Protocol (IP) ping request to a receiving site. The ping
packet specifies that it be broadcast to a number of hosts within the receiving site’s local network.
The packet also indicates that the request is from another site, the target site that is to receive the
DoS. (Sending a packet with someone else’s return address in it is called spoofing the return
address.) The result will be numerous ping replies flooding back to the unaware, spoofed host. If
the flood is great enough, the spoofed host will no longer be able to receive real traffic.
 Viruses: Computer viruses, which replicate across a network in various ways, can be viewed as
DoS attacks where the victim is not usually specifically targeted but simply a host unlucky enough
to get the virus. Depending on the particular virus, the DoS can be hardly noticeable ranging all the
way through disastrous.

64
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

A.3.3 Physical infrastructure attacks

This attack can be as simple as damaging a fiber-optic cable.

A.3.4 Application layer threats

Attackers often try to exploit well-known weaknesses in software residing on servers, such as FTP,
hypertext markup language (HTML), and HTTP. By exploiting these weaknesses, attackers can gain access
to a device with the privileges of the account running the application. Trojan horse attacks are implemented
using bogus programs that an attacker substitutes for common programs. These programs may provide all
the functionality that the normal applications or services provide, but they also include malicious code that
could have a logic bomb or provide backdoor access to the attacker at a future time.

A.3.5 Gaining access

A hacker’s goal is not just to deny legitimate access to a system, but also they often desire to take control
over some, if not all, of the network assets. The most direct way to gain access to a network device is to
acquire the login information for that system. There are a number of ways to gather this kind of
information, as follows:

 Password attacks: Password attacks can be implemented several ways including Trojan horse
programs, IP spoofing, packet sniffers, and brute force attacks. IP spoofing and packet sniffer-
methodology can result in directly obtaining user accounts and password information. Most of the
time, password attacks are done using brute force methods. These brute force attacks are performed
with a dictionary program that runs across the communication network and attempts to login to a
shared device. These dictionary programs are widely available on the Web. Once in, and with
correct privileges, the attacker can create a back door for future access. A dictionary attack is a
method of breaking into a password-protected computer or IED by systematically entering every
word in a dictionary as a password. A dictionary attack can also be used in an attempt to find the
key necessary to decrypt an encrypted message or document. Dictionary attacks work because
many computer users and businesses insist on using ordinary words as passwords. Dictionary
attacks are rarely successful against systems that employ multiple-word phrases, and unsuccessful
against systems that employ random combinations of uppercase and lowercase letters mixed up
with numerals. In those systems, the brute-force method of attack (in which every possible
combination of characters and spaces is tried up to a certain maximum length) can sometimes be
effective, although this approach can take a long time to produce results.
 Packet sniffing: Packet sniffing is a process that runs on a network-attached device that passively
receives all data link-layer frames passing by. Thus, by setting a device to promiscuous mode, any
device on the network can be used to receive all passing Ethernet frames. These frames, in turn, can
be sent on to application programs that extract application-level data. For example, packet-sniffer
programs could be used to extract login information from a Telnet session between a user and an
IED. Once the passwords to user accounts are obtained, the attackers will have full access to those
devices. Packet-sniffing programs are freely available at various Web sites and are easy to use.
 Spoofing: Spoofing is the interception, alteration, and retransmission of a message that is done in
such a way as to deceive the receiver of the message. An IP spoofing attack occurs when an
attacker outside a trusted network pretends to be a trusted computer. Normally an IP spoofing
attack is limited to the injection of data passed between a client and server application or a peer-to-
peer connection. To enable bidirectional communication, the attacker must change all routing
tables to point to the spoofed IP address. Successful spoofing could result in getting access to
sensitive data and/or the ability to perform control commands that would be accepted by the
receiving equipment.

65
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

 Man-in-the-middle attacks: A man-in-the-middle attack requires that the attacker have access to
network packets that come across the network. The man-in-the-middle attack is where the attacker
puts himself transparently, for illegitimate purposes, in the middle of two devices that are having a
legitimate conversation. This is done in such a way that the devices on either end continue to think
they are talking to each other but instead are talking to the attacker. This will allow to the attacker
to access critical data and information about other users on the network, and to possibly send
control commands.

A.4 Techniques to address threats to the electric utility network

Security experts agree that a “defense in depth” strategy provides a more robust solution than one that relies
on a single parameter for protection. Security people often use the following concepts to describe a robust
solution:

 Confidentiality: This ensures that the contents of a message are not exposed to anyone but the
intended recipient.
 Availability: This ensures that the system can service the needs of users and processes as needed.
 Integrity: This ensures that the message received is the same as the message sent.
 Authorization: This ensures that the user or process sending the message has the rights to do so.
 Authentication: This ensures that the sender of a message is who they claim to be.

These concepts can be put in place to reduce the chance of a successful attack. These concepts make it hard
for an attacker to access the channel from the “outside”. Applying technology to the problem without an
overall plan does little good. The first goal should be to establish security policy and security awareness
throughout the organization.

A.4.1 Security policy

Creating and enforcing a network security policy is of critical importance in protecting cyber assets.
Without a security policy, a company has no way of enforcing good practices. Full management support, as
well as thorough understanding throughout the company, is essential.

A security policy tells people what they can and cannot do with a company’s technology assets. It also sets
out what is necessary and required to protect these assets (backup strategy, anti-virus deployment, incident
handling, disaster recovery, etc.). Fundamentally it should include who is responsible, whom it affects,
when it expires, and what is required. A good policy should be short, concise, and easy to understand. It is
very important to audit and review the policy on a regular basis to ensure it is still relevant and meets the
company’s needs. A good policy will balance security with usability. A security policy cannot make it
impossible to access the data or make it too expensive. The policy should consider different levels of
security depending on criticality of data and access.

A network security policy should be enforced at the defensible boundaries of the network. This is often
called perimeter security. The policy for the cybersecurity of the systems in a substation should be a part of
the larger plan for the cybersecurity of the entire utility IT infrastructure. This policy should also be
consistent with the policy for physical security of the utility assets including penalties for violations. For
this reason, some utilities have put both physical and cybersecurity under the same managerial
responsibility.

66
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

A.4.2 Security awareness

It is important for all users to understand and follow the security policy. Educating the user to the risks will
result in fewer policy violations and a feeling of being more involved in security. Security is not just about
technology; it has a very important human component. That is, an electric utility needs to establish policies
and awareness that complement the technical aspects, resulting in the entire solution. A direct benefit of
aware users comes when considering social engineering attacks. For example, if users know not to give
their passwords to other people, a potential attack may be circumvented.

Any security training should be tailored to the individual organization using various media and presentation
types to ensure the concepts reach all employees. This training should be repeated at regular intervals to
provide refreshers and to demonstrate the commitment of upper-level management to the development of a
security-aware workforce.

A.4.3 Password management

Passwords are a first line of defense when it comes to controlling access to devices and systems. It is
important to know the idiosyncrasies and limitations of account administration when it comes to operating
systems, database servers, and applications. The following should be researched and analyzed before
establishing policy in a formal way:

 Procedures for protecting password files and administrator accounts


 Random password generation, one-time passwords, and two-factor authentication
 Password expiration and renewal
 Procedures for removing employee, ex-employee, and contractor access
 Length and qualities of acceptable passwords

Vulnerability to passwords or decryption-key assaults can be reduced to near zero by limiting the number
of attempts allowed within a given period of time, and by wisely choosing a strong password or key. For
example, if only three attempts are allowed and then a period of 30 seconds needs to elapse before the next
three attempts are allowed, and if the password or key is a long, meaningless jumble of letters and
numerals, a system can be rendered nearly immune to dictionary attacks and practically immune to brute-
force attacks.

Many devices ship with default passwords or none at all. It is imperative that the passwords in these
devices be reset in accordance with the organization’s password policies. Whenever possible, strong
passwords should be utilized.

A.4.4 Anti-virus

An effective cybersecurity policy considers where vulnerabilities exist for an organization’s resources
before formalizing processes and procedures. This is especially true when the utility network is connected
to the public internet. Once weaknesses are identified, the policy should specify solutions to prevent the
introduction of malicious code on the company’s perimeter defense systems, the method for deploying new
system elements and who is responsible for this activity. It is also important to analyze what will transpire
once a virus is detected, and the procedure to be followed during the handling of an incident. An effective,
formal anti-virus policy clearly states the simple steps required to report an incident.

67
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

A.4.5 Incident handling

The policy should cover the very practical steps that an organization needs to take when a cybersecurity
incident occurs. All employees should be responsible for reporting evidence of an intrusion or attack.
Documented incident handling tasks are aimed first at securing information assets and minimizing the
damage. Accomplishing these tasks requires that the incident be reported as soon as it happens.

A.4.6 Backup and recovery

With the potential for malicious attacks, hardware failures, and unintentional human error, it is extremely
important to have a formal policy for creating and recovering from backups. The policy needs to emphasize
the fundamental importance of backup and recovery and recovery test processes. Responsibilities should be
clearly documented.

A.4.7 Proprietary information

Every organization has sensitive information that it does not want exposed to outsiders. Cybersecurity
management policy should reinforce a company’s formal information classifications and specify the rules,
guidelines, and procedures for the protection of that data. Proprietary information should be regularly
audited in terms of how it is handled, who has access to it, and the level at which it is protected.
Incorporating the areas of password management, anti-virus, backups, and proprietary information
protection into an organization’s cybersecurity policy can help to establish some common best practices.

A.4.8 Activity logging and alarming

Many IEDs have alarm outputs (these may be hardwired contacts or “soft” contacts) for access, passwords,
and setting events. These alarm contacts should be monitored, as they are important not only for intrusion
detection but also ensuring that the device is working properly. Logging of the alarm events and suspicious
activities is essential for the purpose of determining trends. These files should be regularly audited. It is
important that the logging of activity into the companies’ networks include keeping track of those devices
that access the network. Things that might be important to log include the following:

 Identity of the user


 Login and logout times
 Failed login attempts
 Level of access obtained into the device
 Configuration changes and reason for change

A.4.9 Network connections

When possible, private communication channels should be used. If it is not possible to use a physically
private line, strong encryption/authentication techniques should be used to create a virtually private
connection. Implement access hierarchies with different levels of permission for viewing and setting
devices. Use password, access restrictions, and user authentication to guard against unauthorized access.
Secure SCADA and substation automation systems with virus scanners, firewalls, and anti-intrusion
detection systems. Limit access to information through the design of the communication systems and
devices.

68
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

A.4.10 Connections into IEDs through the network

Use warning banners on login dialogs and screens to discourage electronic intrusions and enable electronic
monitoring and trespass prosecution. Use secure dial back, encrypting, or authenticating modems. Have
predetermined, timed sessions. Also terminate sessions after a set period of inactivity. Control remote
access and monitor access systems via notification of appropriate personnel. In some cases, this may
include alerting SCADA operators.

A.4.11 Secure modem communications

There are a number of categories of secure modems. In increasing cost and complexity, they are as follows:

 Modem key/lock combinations: They only work in pairs (or groups) to ensure that communication
is limited to similarly configured modems. An authorized connection is established when the
dialing modem key synchronizes with the dialed modem lock. These modems provide
authentication with each other, but the data transmission is not encrypted.
 Programmable modems: These modems have on-board security features that permit the creation
and assignment of passwords to incoming or outgoing dialed connections. They can also be
configured to dial back to pre-assigned numbers. The programmable secure modem establishes an
authorized connection but does not encrypt the transmitted data packets.
 Secure encrypting modems: Like the key/lock combinations, these modems work between pairs (or
groups) of similarly configured modems. Unlike the key/lock modems, encrypting modems not
only restrict access but they also encrypt all data transmission between them to safeguard against
eavesdropping or a man-in-the-middle attack.

Secure modems should be used when communications use public-access networks.

A.4.12 Firewalls/connection to enterprise

Firewalls should be used to guard against external threats and are placed at various security perimeters in
the system. A firewall is a device that has a set of rules specifying what network traffic it will allow or deny
between different networks. A firewall can be implemented using a router that filters out unwanted traffic,
or it can be a much more complicated system that includes both hardware and software.

A.4.13 Network intrusion detection systems (IDS)

Whereas a firewall is concerned with guarding unauthorized access into the network, an IDS concentrates
on the internal side of the network. It is like a burglar alarm system for the network that is used to detect
and alert for malicious events. The intent is to determine if insiders or external users are using the system
inappropriately. Intrusions often have predefined signatures (similar to virus signatures) of bad events that
are patterns associated with misuse of the network. When IDS sensors detect bad events, they can alert by
email, paging, or logging the occurrence. IDS sensors can report to a central database that correlates their
information to view the network from multiple sites. IDSs serve three essential security functions: they
monitor, detect, and respond to unauthorized activity by company insiders and outsider intrusion. IDSs use
policies to define certain events that, if detected, will issue an alert. They do not fully guarantee security,
but they can play an important role in the overall security of the networks.

69
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

A.4.14 Encrypting data transmission

As mentioned previously, there are two facets of securing a network: securing access to the network and
securing the movement of data. Typically a lot of thought goes into securing access to the network but
often the securing of the transmission of data is neglected. That is, authentication is required to gain access
to the network but the transmission of data is not protected from viewing by anyone on the network. A
network is really not secure unless both the access to it and the data transmission are protected. Encryption
is the process of transforming data into a form that is unreadable by anyone without a secret key. Its
purpose is to ensure that the data sent over the network is kept confidential. It is also important to ensure
that the data was not altered or that any substitution was not done in transit by an attacker. This is typically
called data integrity. The output of the encryption process is a function of both the message and an
encryption key. This encryption process must be completely reversible by an authorized individual or
computer with access to the secret decryption key. This is one way to ensure confidentiality of the data in
transit.

There are two broad categories of encryption algorithms that use very different approaches:

 Shared key (symmetric): A shared key encryption uses the exact same key for encryption and
decryption. Thus, both the sender of encrypted messages and the intended recipient must have
knowledge of the value of the key in order to have a “conversation”. This means of encryption can
be fast because the mathematics needed to create the encrypted data from a shared key is not that
intense. Thus, it works very well in a real-time system. The main disadvantage of the shared key is
that it is difficult to remotely exchange keys or start an exchange with an unknown party and
authenticate that the party is who he says he is. A symmetric algorithm provides confidentiality of
data as long as the key remains a secret. The best-known types of symmetric encryption are the data
encryption standard (DES), triple DES (3DES), the advanced encryption standard (AES), and
Rivest Cipher (RC-4).
 Public-private key (asymmetric): Asymmetric algorithms use two different keys: a public key and a
private key. The public key is used to encrypt, and the private key is used to decode it back to clear
text. Note that the encryption key and the mathematically related decryption key are different
values. Furthermore, the relationship of the encryption and decryption key is such that one cannot
be derived from the other (e.g., one cannot ascertain the value of the decryption key if they know
the value of the encryption key). Key management is extremely flexible due to the fact that it is
safe to disclose a public key to anybody who wishes to communicate with the computer. All such
individuals can safely use the same public key to encrypt all messages to this computer. Knowledge
of this encryption key does not allow an individual to decrypt the message. All public key
algorithms rely on very complex mathematical relationships in order to form an appropriately
related pair of keys. Most public key algorithms use key sizes on the order of 1024 bits. These
values are incredibly huge. The operations required to perform the encryption and decryption
operations must be performed using these huge key values. Because of this, they are extremely
computationally expensive and as result they might not be able to be utilized in a real-time system
with embedded devices. RSA (Rivest, Shamir, Adelman) is a public key encryption algorithm that
was invented by Ron Rivest, Adi Shamir, and Len Adelman.

A.4.15 Digital signatures

Digital signatures are meant to prove that a piece of information came from a certain individual or entity. It
is an electronic signature that cannot be forged. It verifies that the message originated from the machine
whose signature is attached to it and that message has not been altered since it was signed. This is one way
to provide message authentication.

70
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

A.4.16 Hash algorithms

Hash algorithms are used to create a “fingerprint” of a piece of information or file. The use of hash
algorithms will help assure that data has not been altered in transit. A strong cryptographic hash algorithm
has the property that it is very unlikely two input messages produce the same hash output. This fingerprint,
called a hash or message digest, is used to verify that the message has not been altered. If the message
changes, so will its hash. This is one way to provide message integrity.

A.4.17 Virtual private network (VPN)

A VPN is a protected network session formed across unprotected channels such as the internet. It is
commonly referred to as a “tunnel”. VPN devices allow two networks, physically separated by a possibly
great distance and connected through an insecure network (i.e., the internet) infrastructure to communicate
in a very secure manner (see Figure A.2).

Figure A.2—Virtual private network (VPN)

They effectively combine much of the traffic filtering capability of a firewall with the security of strong
encryption and authentication. Most VPN devices support many simultaneous tunnels, which allow secure,
bidirectional communication between many users with a single VPN device at each remote site. The
security offered by these tunnels comes in the form of strong authentication and encryption. Tunneling is
the process of encapsulating an encrypted packet inside an IP packet for secure transmission across an
inherently insecure IP network. A VPN device will encrypt an entire IP packet and embed it within the data
field of an unencrypted IP packet. In effect, the protocols, by encrypting data at the sending end and
decrypting it at the receiving end, send the data through a tunnel that cannot be entered by data that is not
properly encrypted. An additional level of security involves encrypting not only the data, but also the
originating and receiving network addresses.

An effective VPN will provide the following functions, which have been mentioned previously, to provide
security across a public network:

 Access control: Restrict access to the network to only legitimate users.

71
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

 Authentication: Unambiguously determine the origin of a message.


 Confidentiality: Do not expose the contents of that data to an attacker.
 Data integrity: Ensure that message alterations or substitutions are evident.

The goal is to provide a secure link with security similar to that of having private or leased lines, but at a
much lower cost.

Even private communications networks could be insecure if the entire end-to-end network is not directly
controlled by the utility. Communications networks that utilize third party switching infrastructure should
be given additional analysis. It is critical that each utility perform an in-depth analysis and verify network
isolation before declaring a network to be isolated. All utility communications that are routed over
untrusted or public networks should use VPN technology to protect the channels.

72
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Annex B

(informative)

Protocols used in and to substations

This annex provides a brief history of common protocols used in and to substations, focusing primarily on
those that are standardized either via a standards development organization or the Internet Engineering
Task Force (IETF). Not every protocol used on every system can be covered and for those included, only
summary information is provided.

B.1 Services supported by the Internet Protocol Suite (IPS)

The IPS supports many services and protocols used for a variety of applications.

B.2 Name-to-address mapping using DNS

The domain name system (DNS) is a distributed database that is used by other applications to map between
hostnames (e.g., www.ieee.org) and IP addresses. This allows users to specify human readable names of
other computers when initiating data transfer and it allows users access to computers that may have
dynamic IP addresses. The IPS must know the IP address of a computer before initiating a TCP connection
or User Datagram Protocol (UDP) datagram transmission. DNS uses the UDP transport layer. The DNS
service must be provided on a network by a DNS server, and each computer must know the address of this
server to use the DNS.

B.3 Managed host configuration using Dynamic Host Configuration Protocol


(DHCP)

The DHCP provides a means for delivering host-specific configuration parameters (e.g., IP addresses at
startup) and a mechanism for managing IP addresses on a network. A designated DHCP server can allocate
IP addresses to other computers on the network so that the address pool for that network can be managed
from a single location. Each client computer queries the server for its unique set of IP address information.

B.4 Bootstrapping an IP address using Bootstrap Protocol (BOOTP)

The BOOTP is a UDP network protocol used by a network client to automatically obtain an IP address
during the booting process of computers or operating systems running on them. The BOOTP servers assign
the IP address from a pool of addresses to each client. A typical use of BOOTP is to enable “diskless
workstation” computers to obtain an IP address prior to loading any advanced operating system.

B.5 Network management using Simple Network Management Protocol (SNMP)

Figure B.1 shows the basic layout of the SNMP components.

73
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

MIB: Management information base: Specifies the elements that a particular agent maintains in human-readable form.
OID: Object ID: The unique key-value pair that denotes each element in the agent.
SNMP: Simple Network Management Protocol.

Figure B.1—SNMP components

The manager application hides most of the details of the protocol and the data exchange from the user. It
allows browsing and information retrieval for a wide variety of types of devices spread out over a large
network. All of the device elements are organized in a hierarchical tree at the lowest level that the
application displays in a user-friendly manner. The MIB name and number space is coordinated by the
Internet Assigned Numbers Authority (IANA) to ensure unique elements across all vendors and equipment.

Given the scalable nature, speed, and complexity of modern Ethernet, the network management task can
easily grow to become a major obstacle. A wide variety of standardized tools is available to accomplish this
task. By using SNMP for device-based management, port mirroring, and remote traffic monitoring
(RMON) based upon SNMP for traffic flow monitoring.

B.6 Time synchronization using Network Time Protocol (NTP)/IEEE Std 1588 [B17]

If NTP is used, IEEE Std 1686 [B18] requires either version 3 or version 4. Security for IEEE Std 1588™
[B17] is not yet specified, but should be addressed in future revisions of the standard.

B.7 Remote terminals using secure shell (SSH)/Telnet

Telnet has been deprecated by IEEE Std 1686 [B18]. Use of SSH is recommended.

Some devices and most computers allow users to remotely log in and interact with the operating system.
This facilitates running programs, checking the status of processes, and viewing/updating configuration
parameters. The most common program for general purpose use is the IPS Telnet application. It is a client-

74
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

server application with broad support across operating systems. The downside of Telnet is that it transmits
all of the data and passwords “in the clear”, therefore, it is not appropriate to use over unsecured networks.
However, there are several very secure replacements for remote login applications that encrypt the data and
can be used when one needs to protect against eavesdropping. Table B.1 lists the remote login options,
along with the tradeoffs.

Table B.1—Remote terminal options

Program Application Advantages Disadvantages


Telnet Between nearly all Ubiquitous; works Insecure; requires an
computers across operating additional process on
systems; fast; data substation devices
separate from normal
utility communications
SSH Between most computers Very secure; fast; data Requires an additional
separate from normal process on substation
utility communications devices; requires key
management on hosts
DNP3 virtual Between DNP3 devices Does not require Not universally
terminal separate process on supported; only supported across
substation devices computers running DNP3

B.8 IED access using Hypertext Transfer Protocol (HTTP)/HTTPS

Figure B.2 shows the basic layout when using HTTP (deprecated by IEEE Std 1686 [B18]) and HTTPS.

Figure B.2—HTTP (WEB) components

The Web browser provides a common display program for pages constructed by the Web server. Each
embedded Web server can provide any set of pages in any format the vendor chooses. The format of the
pages is based on the HTML; however, many additional languages and formats are generally supported for
displaying information on modern Web browsers. Each piece of substation information system equipment
is accessed separately by IP address or, if DNS is provided in the system, by a name.

75
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

B.9 File transfer using File Transfer Protocol (FTP)/Secure File Transfer Protocol
(SFTP)

FTP has been deprecated by IEEE Std 1686 [B18] and only SFTP is recommended.

There are a number of techniques for transferring files between hosts on an electric utility IP network. The
selection depends on the user processes (applications) available on the two hosts. The most common for
general purpose use is the FTP. It is a client-server application with broad support across operating
systems. The downside of FTP is that it transmits all of the data and passwords “in the clear”, therefore, it
is not appropriate to use over unsecured networks. However, there are several very secure replacements for
FTP applications that encrypt the data and can be used when one needs to protect against eavesdropping.
Table B.2 lists the file transfer options along with the tradeoffs.

Table B.2—File transfer options


Program Application Advantages Disadvantages
FTP Between nearly all computers Ubiquitous; works across Insecure; requires an additional
operating systems; fast; data process on substation devices
separate from normal utility
communications
SFTPa Between most computers Very secure; fast; data Several different varieties;
separate from normal utility requires an additional process
communications on substation devices; requires
key management on hosts
RCP Between most computers Implemented on major Insecure; requires an additional
operating systems (e.g., process on substation devices
Linux®/Unix® and
Windows®b) data separate
from normal utility
communications
SCPa Between most computers Very secure; data separate Requires an additional process
from normal utility on substation devices; requires
communications key management on hosts
DNP3 file transfer Between DNP3 devices Does not require separate Not universally
process on substation supported; only supported
devices across computers running
DNP3
IEC 60870-5 file Between IEC 60870-5 devices Does not require separate Not universally
transfer process on substation supported; only supported
devices across computers running IEC
60870-5
IEC 61850 file transfer Between IEC 61850 devices Does not require separate Only supported across
process on substation computers running IEC 61850
devices
Legend:
DNP3: Distributed Network Protocol, version 3.0 SCP: Secure remote copy
FTP: File Transfer Protocol SFTP: Secure File Transfer Protocol
RCP: Remote copy
a
Not related to FTP.
b
This information is given for the convenience of users of this recommended practice and does not constitute an endorsement
by the IEEE of these products. Equivalent products may be used if they can be shown to lead to the same results.

76
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

The recommended file transfer mechanisms are SFTP and SCP such as those found in one of the SSH
implementations (Barrett and Silverman [B1]). There are implementations available for most operating
systems in both freely distributable and commercial product form. Other protocol-specific file-transfer
methods may be appropriate if the necessary security enhancements are standardized in the future.

B.10 Rapid Spanning Tree Protocol (RSTP)

While very powerful in terms of bandwidth and its real-time capabilities, an Ethernet-based network would
be useless without the ability to survive network equipment failures. This is accomplished by adding
hardware redundancy (redundant Ethernet switches) and using advanced network topologies. Extensive
description of redundant network topologies can be found in Pozzuoli [B56], Galea and Pozzuoli [B4], and
Galea [B3].

For the purposes of this discussion, it is sufficient to note that multiple communication paths will
automatically be established as soon as redundancy gets introduced into the network. While multiple
communication paths are desirable (in general terms), their very presence jeopardizes the ISO Layer 2
switch-based network operation.

Unless specifically disabled, multiple paths will result in packets being forwarded indefinitely through all
circular paths, leading to fast traffic build-up and complete network failure. This problem can be addressed
by introducing the “spanning tree” algorithm, which enables multiple Ethernet switches to self-organize,
discover, and temporarily disable redundant network connections. Such connections are kept in “hot
reserve” and can be activated on demand. A rapid-spanning tree algorithm fine-tunes the original spanning
tree implementation (which could take anywhere from several seconds to a few minutes to fully restore the
network depending on the size of the network and other factors) by speeding it up to the 5–500 ms level.

Most vendors offering substation-hardened Ethernet switches also provide a proprietary mode to restore
traffic in a few milliseconds.

B.11 High-availability Seamless Redundancy/Parallel Redundancy Protocol


(HSR/PRP)

With the introduction of high-speed digital technology into the substation environment, the need for
redundant, seamless fail-over technology became critical. In response to this need, the IEC published
specification for highly available automation networks. HSR and PRP are covered in Technical Report IEC
61850-90-4 [B12].

Generally speaking, the decision to use one versus the other boils down to a question of economics. HSR is
less expensive, but is also less capable, in that a significant outage can isolate IEDs from the network. The
duplicate network design of PRP is more expensive than HSR, but is also more robust. Accordingly, PRP
may be more appropriate in complex, “high-value” environments like high-voltage substations.

It is expected that, in the future, IED will have HSR/PRP capabilities available.

B.11.1 HSR

With HSR, the nodes on the network are arranged in a ring. Each of the nodes in the ring has two identical
interfaces (Port A and Port B). When a frame is sent, the source node sends two copies via Port A and B.
The packets are sent in opposite directions on the ring. Each node on the ring relays the frame from Port A
to Port B (except if it has already forwarded the packet). The receiving node accepts the first packet and
drops the duplicate. In the event that the network is damaged, the redundant packet will still reach one of
the two ports.

77
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

This approach ensures that no frames are lost (bumpless switchover) in the event that the ring is
interrupted. However, if a large section of the network is compromised, or if multiple nodes are disabled,
several of the remaining nodes could be isolated from the network. Connection to legacy devices and to
other networks is handled via a redundancy box (i.e., a redbox). HSR is a flexible topology that allows
multiple rings, rings within rings, and connection to PRP rings (see Figure B.3). The solution is cost
effective, in that ring topology is not duplicated.

Figure B.3—HSR overview

B.11.2 PRP

With PRP, there are two parallel networks (Network 1 and Network 2). Like HSR, each of the nodes on the
network has identical interfaces (Ports A and B). Each interface is connected to one of the parallel rings.
For example: Port A is connected to Network 1; Port B is connected to Network 2. When a frame is sent,
the source node sends a copy of the frame on each network. The receiving node processes the first packet
received and discards the duplicate packet. If one of the rings is compromised, traffic will reach the nodes
via the second ring.

PRP allows seamless switch over, with no frames lost. PRP requires a duplicate ring, which makes it more
expensive than HSR. However PRP can tolerate more damage to the network and is therefore preferred for
mission critical networks. Finally, PRP supports any topology and is not limited to ring. See Figure B.4 for
an overview of PRP structure.

78
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure B.4—PRP overview

B.11.3 Software defined networking (SDN)

SDN is another control plane technology improving on RSTP and is compliant to IEEE Std 802.3-2018.
The control plane is abstracted from the network appliance and centralized in software called the flow
controller. Doing this allows the network engineer to proactively traffic engineer all flows on the network
or use reactive controller configuration as traffic appears. Packets are forwarded based on a lookup table
filled with match and instruction pairing. As each packet comes into the switch it is compared to these
tables to identify if there are any matches and once a match is found the instructions are executed on that
packet. Network appliances are simplified and performance is increased. Packets are inspected at multi-
layers covering layer one through four, and only packets that match a flow are forwarded, increasing
cybersecurity and traffic shaping capabilities. Since the switch is simplified to a look up table, the network
heal times are greatly improved as well with microsecond level healing. Traffic engineering is done by
merging the engineering controls for physical and logical packet delivery into flows. SDN allows for
proactive network engineering planning event response to any link failure. SDN is an open standard
enabling wide interoperability. SDN achieves microsecond network heal times but, if that is still not fast
enough, may be used in conjunction with a PRP infrastructure. There are commercially available substation
hardened SDN switches on the market.

B.12 IEEE Std 1815 (DNP3)

DNP3 represents a fundamental shift in the master/slave protocol paradigm. It is a protocol that distributes
intelligence throughout the network by encouraging intelligent slave devices, thus reducing the emphasis

79
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

on polling by getting more efficient information throughput through the collection of SCADA event
information.

DNP3 is capable of performing well over multiple communication media, such as the following:
 Serial modem
 Dialup
 Radio
 Satellite
 Digital cellular
 LAN (TCP/IP)

B.12.1 History

A detailed history of DNP3 is available in IEEE Std 1815.

B.12.1.1 1990 to1991

 DNP 1.0, 2.0 created by Westronic Inc.

B.12.1.2 1993 to1996

 DNP3 Basic 4 published and placed into public domain by Harris


 DNP User’s Group formed 25 October 1993
 DNP Technical Committee formed
 Publication of DNP subset definitions document
 www.dnp.org created

B.12.1.3 1998 to 2014

 DNP3 conformance test documents—beta release


 IEEE Std 1379 published—IEEE Std 1379™ [B16], IEEE Recommended Practice for Data
Communications Between Remote Terminal Units and Intelligent Electronic Devices in a
Substation
 DNP3 LAN-WAN standard version 1.0 approved and published by DNP Technical Committee
1999–2006
 DNP3 level 1 and 2 test procedures (official release)
 IEEE publishes IEEE Std 1379-2000, IEEE Recommended Practice for Data Communications
Between Remote Terminal Units and Intelligent Electronic Devices in a Substation, subsequently
withdrawn
 IP networking version 2.0 approved and published by DNP3 Technical Committee
 Master station test procedures (currently in draft stage)
 New objects created
 DNP3 secure authentication

80
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

DNP3 was standardized by the IEEE in 2010 and updated in 2012.

B.13 Protocol features

This subclause covers some protocol features relevant to IEEE Std 1615, where much more detailed
information can be found in IEEE Std 1815.

B.13.1 Polling schemes

DNP3 addresses a unique way of collecting data from the outstations—it requires that the outstation be
intelligent with its responses. The following are three advanced methods of polling:

a) DNP3 integrity poll, returns all DNP data, requests all events (changes) and a static (snapshot) of
entire DNP3 database - database sync of master and remote.
b) DNP3 event polling, returns all events only (change) data.
c) Unsolicited reporting, where outstations report events (changes) as they occur without the master
having to ask for them. DNP3 can also act like legacy protocols, by polling for data one type at a
time and asking for a specific variation and quantity of points.

There is also the ability to poll for a range of specific data, provided this capability is supported by the
outstation.

B.13.2 Object-based application layer

DNP3 includes the following objects at the application layer:

 Function code independent of data:


 Read, write, select/check/operate, freeze and read, time sync.
 Objects are data complete with properties/attributes.
 Objects available for standard SCADA data:
 Binary inputs (status), binary outputs (control), analog inputs, analog outputs (setpoints),
counters (accumulators).
 Objects available for extended SCADA and process control data:
 Floating point, virtual terminal, file transfer, strings, data sets.

DNP3 is a very large and flexible protocol and it is unrealistic to expect or require that all DNP3 outstations
implement the entire protocol. In some implementations, only a small portion of the protocol may be
required for basic outstations, so DNP3 defines several subsets, as follows:

 Level 1: Simple IEDs.


 Level 2: IEDs and simple RTUs. Includes everything in Level 1 plus:
 Freeze requests on binary counter objects
 Reads of all binary input variations
 Supports unsolicited responses with static data

81
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

 Variation 0 of most objects supported


 IEEE Std 1379 [B16] recommends this level for substation applications
 Freeze requests for analog input and frozen counters are not supported
 Level 3: Everything in Level 2, plus other data types and function codes that were found to be
useful after the first original subsets were introduced, intended for RTUs and data concentrators.
 Level 4: Everything in Level 3, intended for communications between a master station and a data
concentrator/RTU.
Complete information on these levels is found in IEEE Std 1815.

B.13.3 DNP3 system architecture

DNP3 is specified to operate over 8-bit asynchronous serial channels and the IPS. Any system (i.e.,
traditional or networked) that supports either of these basic transport methods can support DNP3. Since all
DNP3 link layer messages are composed of 8-bit octets with fixed starting characters, DNP3 is particularly
well suited for network-to-serial conversion.

B.13.4 IPS and services

DNP3 requires both transport layers (TCP and UDP) of the IPS. DNP3 defines three types of TCP end
points: initiating, listening, and dual. Initiating and listening end points correspond to the client/server
notation commonly used in TCP. A dual end point listens for TCP connection requests and can also initiate
a connection if necessary (e.g., to send an unsolicited response). UDP can be used for all messages or for
just broadcast messages. IP addresses are configured directly into the outstation.

B.13.5 Data classification

DNP3 is optimized for the transfer of status and control data. Recently, additional features such as file
transfer and data sets were added.

B.13.6 Security

As of 2012, DNP3 provides secure authentication.

B.13.7 System level requirements

DNP3 provides a network-based time synchronization protocol that can be used on a LAN. DNP3 provides
time tags with resolutions to 1 ms and has no specification of clock accuracy.

B.13.8 Error detection

DNP3 has extremely robust bit error detection. The link layer can reliably detect up to five error bits
anywhere in the DNP3 stream. This exceeds the error detection of Ethernet and the IPS. The error detection
properties of the 16-bit link layer cyclic redundancy checks (CRCs) protect against inadvertent control
operations or incorrect data due to noise.

82
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

B.13.9 Network management

Network management tools are outside the scope of DNP3. A profile exists in Wireshark 11 for decoding
DNP3, and protocol analyzers are on the market that will decode DNP3 packets. File transfer and virtual
terminal services are provided by the protocol.

B.13.10 Media and device alternatives

DNP3 supports any media and device infrastructure that can transport IP packets.

B.13.11 Substation LAN requirements

The details of the communication network are outside the scope of DNP3. DNP3 is typically not used for
low-latency, high-speed protective relaying messages and therefore these requirements are not applicable.
DNP3 is very good at operating over all types of heterogeneous IP networks including those with PPP
serial links.

B.14 IEC 60870-5

B.14.1 History

To gain an understanding of IEC 60870-5-104, it is necessary to place it in the context of the entire suite of
IEC 60870-5 standards developed by Working Group 03 of the IEC TC 57 committee.

The base standards of IEC 60870-5 were originally defined as a framework for protocols to be used over
traditional telecontrol (master/remote communications between different facilities) channels with low
bandwidth (up to 19.2 kb/s). This framework includes standards for transmission frame formats (IEC
60870-5-1), link layer services (IEC 60870-5-2), application data structure (IEC 60870-5-3), application
information elements (IEC 60870-5-4), and basic telecontrol application functions (IEC 60870-5-5).

Based on this framework, the working group then went on to define profiles in 1992. The profiles released
in the form of Companion Standards are discussed below in this subclause.

B.14.1.1 IEC 60870-5-101—Companion standard for basic telecontrol tasks

This standard is intended to be used for standard master/remote SCADA functionality over fixed lines, and
contains the traditional functionality of real-time control, status, analog, and RTU management. The first
edition of this companion standard was published in 1994 and was upgraded and improved by the
edition 2.0 published in 2003.

IEC 60870-5-101 is one of two communication protocols selected in IEEE Std 1379-2000 [B16].

B.14.1.2 IEC 60870-5-102—Companion standard for the transmission of integrated totals in


electric power systems

This profile is similar in nature to IEC 60870-5-101 but is designed for meter reading.

11
This information is given for the convenience of users of this recommended practice and does not constitute an endorsement by the
IEEE of these products. Equivalent products may be used if they can be shown to lead to the same results.

83
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

B.14.1.3 IEC 60870-5-103—Companion standard for the informative interface of protection


equipment

This companion standard has its roots in a national initiative in Germany worked out by the VDEW/ZVEI
organization as a recommendation. It was developed in parallel to IEC 60870-5-101, based on the same
framework with the goal to access the data of protection relays within substations in a standardized way. In
a second step, the access of disturbance records of protection relays was added. One of the first activities of
the working groups formed to develop a future-proof communication structure within substation
automation systems (now known as IEC 61850 [B12]) was to bring out this VDEW/ZVEI recommendation
as an IEC standard, since it was already a “de facto standard” worldwide.

The VDEW/ZVEI recommendation was improved by adding functions to access configuration data of the
protection devices as well (generic functions) and was published as IEC 60870-5-103 in 1997.

B.14.1.4 IEC 60870-5-104 [B10]—Network Access for IEC 60870-5-101 using standard
transport profiles

Facing the fact that more and more power utilities worldwide started to replace the traditional
communication infrastructure of fixed lines between substations and control centers by WANs with better
flexibility and higher bandwidth, in 1996 the TC 57 Working Group 03 started the work on IEC 60870-5-
104 [B10]. The major goal was a smooth migration of existing solutions based on 60870-5-101 into the
new world of “networking”. The application layer of IEC 60870-5-104 [B10] is functionally identical to
IEC 60870-5-101, but maps its functionality on the broadly used TCP/IP transport profile instead of the
link layer specified in the original IEC 60870-5 suite. With TCP/IP over Ethernet, various network types
can be utilized [including X.25, frame relay, ISDN, and asynchronous transfer mode (ATM)] by using
standard network nodes like routers. A specific sublayer was defined to compensate for the weaknesses of
TCP/IP used for telecontrol purposes. Special care was taken during the definition of that sublayer in order
to avoid losing any IEC 60870-5-101 [B7] functionality concerning the integrity of the process data (e.g.,
protection against undetected loss or duplication of messages); therefore, only the “connection-oriented”
TCP service was selected.

The definition of IEC 60870-5-104 [B10] was done in parallel to the enhancements and improvements of
IEC 60870-5-101 (Edition 2.0) [B7], to offer a well-adjusted solution with high capabilities for smooth
migration to the power utilities.

The publication of IEC 60870-5-104, Edition 1.0 took place in 2000. Table B.3 shows the protocol
structure of the end system.

Table B.3—Selected standard provisions of IEC 60870-5-104 [B10]


Selection of application functions of IEC 60870-5-5 according to IEC Initialization User Process
60870-5-101 [B7]
Selection of ASDUs from IEC 60870-5-101 [B7] and IEC 60870-5-104 [B10]
APCI (Application Protocol Control Information) Application (layer 7)
Transport Interface (user to TCP Interface)
RFC 793 (TCP) [B34] Transport (layer 4)
RFC 791 (IP) [B32] Network (layer 3)
RFC 1661 (PPP) [B41] RFC 894 (Transmission of IP Datagrams Data Link (layer 2)
RFC 1662 (PPP in HDLC-like framing) [B42] over Ethernet Networks) [B36]
X.21 (serial line) IEEE Std 802.3 (Ethernet) Physical (layer 1)
NOTE—Layers 5 and 6 are not used.

84
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

B.14.2 Functionality

At the base level, IEC 60870-5-104 [B10] inherited from IEC 60870-5-101 [B7] the typical functionality
one would expect in a master/remote protocol such as control, status reads, analog reads, accumulators
handling (freeze/read/clear), etc. In addition, more advanced features that were being contemplated at the
time of IEC 60870-5-101’s [B7] development in the mid-1990s are also available. These include the
following:

 Multi-window sizing messaging capabilities with configurable window size


 Unsolicited reporting (same as in IEC 60870-5-101 [B7])
 Handling of virtual devices [logical remote terminal units (LRUs)]
 “Dual mode stations” (controlled and controlling), capable of communicating all process values in
both monitor and control directions
 Originator address for the handling of multiple controlling stations (control centers)
 Parameter loading for measurands (threshold, smoothing, limits)
 Generic file transfer can be used to transport configuration files, sequence-of-event logs, sampled
waveform data, disturbance records from protection relays, or any other file data
 Handling of redundant network connections between control center and RTU

B.14.3 System architecture

IEC 60870-5-104 [B10] conforms to 4.2 using a networked communication architecture. Because IEC
60870-5-104 [B10] uses a standard transport profile (TCP/IP) and interface (Ethernet), it can be used either
in LAN or WAN environment.

B.14.4 IPS and services

IEC 60870-5-104 [B10] uses an Ethernet-based network. Supported RFCs are IP and TCP. Configuration
of IP addresses is outside of the specification, but may be done using the RFCs listed in the table.

B.14.5 Data classification

IEC 60870-5-104 [B10] is designed to reliably transfer operational, and control data within the time
requirements specified in 6.1.4 of the IEC standard. This can even be done over low-speed networks, since
the protocol is based on IEC 60870-5-101 [B7]. Non-operational data is supported through file transfers as
outlined in C.2.

B.14.6 Time synchronization

IEC 60870-5-104 [B10] typically can achieve time synchronization over the network in the area of ±10 ms.
In order to achieve higher time accuracy, the additional use of NTP is recommended.

85
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

B.14.7 Network management

Network management is currently outside the scope of IEC 60870-5-104 [B10]. Advanced network
analyzing tools supporting IEC 60870-5-104 [B10] are available on the market. For file transfers, IEC
60870-5 file transfer is supported.

B.14.8 Media and device alternatives

IEC 60870-5-104 [B10] is media and device independent.

B.14.9 Substation LAN requirements

The details of the communication network are outside the scope of IEC 60870-5-104 [B10]. The different
features discussed in Annex H can be used to build efficient and reliable communications networks.

B.15 IEC 6185012 [B12]

As described in IEEE Std 2030.100 [B21], IEC 61850 [B12] is more than a communication protocol. IEC
61850 [B12] specifies the following:

 A comprehensive object model comprising models for the complete substation, both the switchgear
itself as well as the automation f unctions like protection functions
 An abstract model including services for information exchange between devices
 The mapping of this abstract model on real communication protocols
 A configuration language supporting the exchange of configuration information between IEDs
 A conformance test specification

The goals of IEC 61850 [B12] include: to promote the interoperability of IEDs from one or several
manufacturers, to support different philosophies and allow a free allocation of functions, and to follow the
progress in communication technology.

An example of a basic system architecture of IEC 61850 [B12] is shown in Figure B.5. These are logical
interfaces. IEC 61850 [B12] does not specify how these interfaces are mapped on communication
networks. A typical example would be to map interfaces 1, 3, and 6 on a station bus and interfaces 4 and 5
on a process bus. The process bus may be restricted to one bay, while the station bus might connect the
whole substation. However, it may be possible as well to map interface 4 on a point-to-point link
connecting the sensor and, for example, the relay.

12
IEC 61850 [B12] Communication Networks and Systems in Substations was prepared by IEC technical committee 57, working
groups 10, 11, and 12. It was published between 2002 and 2005. IEC 61850 in total consists of 14 parts. IEC 61850 has incorporated
UCA 2.0 published by IEEE as TR1550.

86
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure B.5—Interfaces within a substation automation system

B.15.1 The information model according to IEC 61850 [B12]

The core element of the information model is the logical node. A logical node can be considered a
container for function-related data. Logical nodes contain data. Data and data attributes represent the
information. The name of a logical node class is standardized and is always comprised of four characters.
Basically, one can differentiate between the two following kinds of logical nodes:

 Logical nodes representing information of the primary equipment (e.g., circuit breaker—XCBR, or
current transformer—TCTR). These logical nodes implement the interface between the switchgear
and the substation automation system.
 Logical nodes representing information of substation automation functions. Examples are any kind
of protection function (e.g., distance protection—PDIS) or the measurement unit—MMXU.

B.15.2 Information exchange

To make the standard future-proof, IEC 61850 [B12] has introduced the concept of the ACSI (abstract
communication service interface). The ACSI is defined in IEC 61850-7-2 [B12] and is a set of abstract
models and service specifications that describe the information exchange.

The major information exchange models defined in IEC 61850-7-2 [B12] are as follows:

 Read-and-write data
 Control
 Reporting
 GOOSE
 Sampled values (SV)

The first three models, read-and-write data, control, and reporting, are based on a client/server relationship.
The server is the device that contains the information while the client accesses the information. Read-and-
write services are used to access data or data attributes. These services are typically used to read and

87
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

change configuration attributes. Control model and services are a specialization of a write service. Their
typical use is to operate disconnectors, earthing switches, and circuit breakers. The reporting model is used
for event-driven information exchange. Information is spontaneously transmitted when the value of the data
has changed.

The last two models, GOOSE and SV, are based on a publisher/subscriber concept. (In IEC 61850 [B12],
the term peer-to-peer communication is used.) They are used for the exchange of time-critical information.
The device being the source of the information publishes the information. Any other device that needs the
information can receive it. These models use multicast communication (the information is not directed to
one single receiver).

Generic substation status event (GSSE) is a model used to transmit status information in a fast way to
multiple devices. Instead of using a confirmed communication service, the information exchange is
repeated regularly. Applications of GSSE services are the exchange of position information from switches
for the purpose of interlocking, or the transmission of a digital trip signal.

GOOSE is a model to rapidly transmit information to multiple devices in a way similar to GSSE. GOOSE
messages are data sets that include digital status points as well as digital analog values. Like GSSE, a non-
confirmed communication service is used and the information exchange is repeated regularly. An
application of a GOOSE service is the exchange of measured values from various meters for the purpose of
performing distributed event recording.

The model for the transmission of sampled values is used when a waveform needs to be transmitted using a
digital communication. In the source device, the waveform is sampled with a fixed sampling frequency.
Each sample is tagged with a counter representing the sampling time and transmitted over the
communication network. The model assumes synchronized sampling; that is, different devices sample the
waveform at exactly the same time. The counter is used to correlate samples from different sources. Such
an approach creates no requirements regarding variations of the transmission time.

IEC 61850 [B12] does not define communication protocols—it uses existing protocols instead. The use of
these protocols is defined in the mappings (IEC 61850-8-1, IEC 61850-9-1, and IEC 61850-9-2). The
mappings define how the abstract models and services (according to IEC 61850-7-2) are implemented
using the models and services of the communication protocols used.

The mappings currently defined in IEC 61850 [B12] differentiate between the client/server services and the
publisher/subscriber services. The client/server services utilize the full seven-layer communication stack,
using manufacturing message specification (MMS) and TCP/IP. The publisher/subscriber services are
mapped onto a reduced stack, basically directly accessing the Ethernet link layer with specific, standardized
EtherTypes.

Client/server based services use the following communication protocols:

 Application layer: MMS (ISO 9506 [B46]) and association control service elements (ISO/IEC 8649
[B49] and ISO/IEC 8650 [B50])
 Presentation layer: Connection-oriented presentation (ISO/IEC 8822 [B51] and IEC/ISO 8823-1
[B52]) ASN.1 using basic encoding rules (ISO/IEC 8824-1 [B53] and ISO/IEC 8825-1 [B54])
 Session layer: Connection-oriented session (ISO/IEC 8326 [B47] and ISO/IEC 8327-1 [B48])
 Transport layer: ISO transport on top of TCP (IETF RFC 1006 [B38]), Internet Control Message
Protocol (ICMP) (IETF RFC 792 [B33]), and TCP (IETF RFC 793 [B34])
 Network layer: IP (IETF RFC 791 [B32]) and Address-Resolution Protocol (ARP) (IETF RFC 826
[B35])
 Data link layer: Transmission of IP datagrams over Ethernet (IETF RFC 894 [B36]) and
CSMA/CD (IEEE Std 802.3-2018)

88
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

 Physical layer: electrical 10BASE-T/100BASE-T or fiber-optic transmission system 100BASE-FX


(IEEE Std 802.3-2018)

Optionally, an open systems interconnection (OSI)-T profile can be used in addition.

Publisher/subscriber-based services use the following communication protocols:

 Presentation layer: ASN.1 using basic encoding rules (ISO/IEC 8824-1 [B53] and ISO/IEC 8825-1
[B54])
 Data link layer: Priority tagging/VLAN and CSMA/CD (IEEE Std 802.1Q-2018 and IEEE Std
802.3-2018)
 Physical layer: Electrical 10BASE-T/100BASE-T or fiber-optic transmission system 100BASE-FX
(IEEE Std 802.3-2018)

B.15.3 Configuration management

IEC 61850-6 [B12] defines the substation configuration language (SCL). The purpose of SCL is to define a
file format that can be used to exchange configuration information between tools. The SCL supports the
design, engineering, and commissioning of the substation during its whole life cycle. It starts with the
formal specification of the substation (single-line diagram, functionality in terms of logical nodes) in the
system specification description file. The capability of the IEDs (logical nodes and communication services
that can be supported) that will be used to implement the substation automation system is described in the
IED-capability description file. These files are used as input for the system configuration tool. With the
system configuration tool, the complete design and engineering of the future substation is made. The output
of the system configuration tool is the substation configuration description file, which contains the
complete configuration information for the substation. This file can be used as input to the IED
configuration tool. With the IED configuration tool, additional IED-specific information can be added and
the configuration download for a specific IED is created.

B.15.4 System architecture

The scope of this document through this version is limited to within substations. When using IEC 61850
[B12] within substations, the bottom up approach for upgrading is appropriate.

B.15.5 IPS and services

IEC 61850 [B12] uses the Ethernet-based networks. IP and TCP are supported. IP addresses are directly
configured in the device through the substation configuration language.

B.15.6 Data classification

The different data classes and their requirements are specified in IEC 61850-5. IEC 61850 [B12] has a
more distinguished classification of data than 5.1 (operational/control/non-operational).

B.15.7 Time synchronization

IEC 61850 [B12] supports real-time data with delivery requirements of a few milliseconds. IEC 61850
[B12] typically uses switches that support priority tagging. For time synchronization, Simple Network
Time Protocol (SNTP) is specified, but this likely requires a special implementation in order to meet the

89
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

time synchronization performance requirements. For the synchronization of devices requiring high
accuracy (1 µs), a direct-wired time synchronization network originating from a global positioning system
(GPS) satellite clock is used. Reference IEEE Std 1588 [B17] and IEC 61850-90-3 [B12].

B.15.8 Network management tools and practices

This was outside the scope of IEC 61850 [B12], but is now addressed by IEC 61850-90-4 [B12], which
adds SNMP.

B.15.9 IEC 61850 media and device alternatives

IEC 61850 [B12] supports copper according to 10/100BASE-T specification, and fiber according to
100BASE-FX specification. With Edition 2, gigabit Ethernet using optical fiber is now supported.

B.15.10 IEC 61850 substation LAN requirements

The details of the communication network used to be outside the scope of IEC 61850 [B12], but are now
addressed by IEC 61850-90-4 [B12].

90
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Annex C

(informative)

Internet Protocol Suite (IPS)

C.1 Introduction

This annex introduces the IPS and some of the services relevant to this recommended practice and provides
a description of IP addressing as well as Internet Protocol, version 6 (IPv6).

The IPS is platform-independent and supported universally around the world. It is highly scalable, that is,
LAN to WAN, and many quality implementations exist for both embedded and workstation operating
systems. The growth of the internet has fueled the large availability of equipment and has proved that the
IPS is capable of transporting tremendous quantities and types of data. With some tailoring, the security
protocols and mechanisms developed, and being developed, for the internet are applicable to electric utility
networks.

The most attractive reasons for implementing an IPS are as follows:

 Simplified integration between existing networks


 Leverage existing equipment and standards

The IPS is composed of several protocols that provide data transport services and a number of applications
for network-related activities. It is commonly referred to as “TCP/IP,” after the two most important
protocols in it: the TCP and the IP, which were also the first two defined. However, the IPS refers to the
entire set of protocols developed by the internet community and is, therefore, used here.

The Internet protocols are defined by request for comment (RFC) documents available from the Internet
Engineering Task Force (IETF) 13 as public domain works. Although some of the RFCs specify the internet
protocols, many of the RFCs are for information or application guidance only and are not official standards.
While the IPS can be roughly fitted to the OSI model that describes a fixed set of seven layers and some
like to use this model, others believe that fitting the IPS to the OSI model does more to confuse than to
help. Note that the IPS does not define the data link and physical layer because the IPS was designed to
accommodate any underlying network technology.

Table C.1 is a partial list of the internet protocols pertinent to this clause.

13
IETF documents are available at http://www.ietf.org.

91
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Table C.1—Primary internet protocols for this recommended practice


Name RFC Description
IP 791 Network layer of the IPS.
TCP 793 Connection-oriented transport layer of the IPS.
UDP 768 Connectionless transport layer of the IPS.
DNS 1034, 1035 Distributed database used to map between hostnames and IP addresses.
ARP 826 Provides address mapping IP and the link layer (e.g., Ethernet)
BOOTP 951 Basic service allowing hosts to query for their IP address
DHCP 2131 Service allowing hosts to query for their IP address and other initial parameters at startup
from a server. Replaces BOOTP.
1918 Address allocation for private internets.
1006 ISO Transport service on top of the TCP
Legend
ARP: Address Resolution Protocol IPS: Internet Protocol Suite
BOOTP: Bootstrap Protocol ISO: International Organization for Standardization
DHCP: Dynamic Host Configuration Protocol TCP: Transmission Control Protocol
DNS: Domain name system UDP: User Datagram Protocol
IP: Internet Protocol

C.2 IP layers

Network technology has a set of common standards for defining communication details. They are common
knowledge to the network professional but may be foreign to the utility power professional. The common
framework for describing communications is the layered communications stack.

In the world of computer systems, the various functions are described as being layers in a stack that are
needed to complete the interaction between computers. Normally there are seven layers described.

In the substation, to ensure data exchange between IEDs, a communication system topology is required
along with a communication architecture that supports them. The communication architecture is composed
of several communication protocols/standards that ensure that data is transmitted between IEDs. A set of
protocols/standards are layered over each other, each layer offering a set of services to the layers above and
below to enable data transmission from layer to layer. Data starts at the top of the stack and is transferred
from layer to layer until it reaches the bottom of the stack, where the data is transferred over the physical
media. Once the data is received, the data is again transferred from layer to layer, each layer adding its own
information in the message. As shown in Figure C.1, some layers are responsible for data exchange
between IEDs, while other layers are responsible for data exchange between application functions.

IED 1 IED 2

Function Function
Function to function
communication

Communication Subsystem Communication Subsystem


IED to IED
communication

Data Transmission System

Figure C.1—IED-to-IED communication

92
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

A chosen set of protocols forms a stack, where a substation communication network will often use up to
four layers. In the simplest implementation with a point-to-point communication, a two-layer stack can be
used. More complex systems have additional layers in the stack.

Complex networks often use the internet stack as shown in Figure C.2.

Application Layer

Transport Layer

Network Layer

Link and Physical Layer

Figure C.2—Internet stack

The data link and physical layer control the transmission/reception of data in a suitable format for the
communication media. The specification of the physical layer includes both mechanical and electrical
characteristics. This layer also detects some types of errors and notifies the data link layer when such errors
are detected. Some examples of physical layer standards are: TIA-232-F, TIA-422-B, and Ethernet for
high-speed networks. The link is in charge of transmitting messages over the physical connection, detecting
errors and, in some implementations, correcting some types of errors. It detects frame errors, controls the
flow of data between IEDs, and ensures the correct sequence the received message.

The network layer routes messages between nodes that are transparent to the upper layers. The network
layer ensures that the data will be transmitted from end-to-end over the network. The network layer also
handles network addressing and congestion control.

The transport layer provides a network-independent message transfer facility. The transport layer provides
for making connections between messaging partners when a messaging session is opened to connect
partners or for transporting messages where a connection-less session is supported.

The application layer offers a set of services and data manipulation for substation automation. This is the
segment that is specific to the data transfer and IEDs. This segment determines the interoperability between
IEDs. To be interoperable, devices understand the specific application layer protocol carried by the network
messaging protocols.

Figure C.3 shows the various layers protocols in the IPS (see Stevens [B58] for additional description).

93
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure C.3—IPS protocols and layers

C.2.1 Link layer

The link layer is not really a part of the IPS. The link layer passes packets from the network layer of one
device to the network layer of another. On one end, the link layer adds a frame header to prepare it for
transmission and then transmits the frame over a physical medium. On the other end, the link layer receives
data frames, strips off the frame headers, and hands the received packets to the network layer.

C.2.2 Network layer

The network layer solves the problem of getting packets across a single network or from the source
network to the destination network. This generally involves routing the packet across a network of
networks, known as the internet.

In the IPS, IP provides the addressing and basic delivery service. An IP data packet is called a datagram.
The IP layer does not maintain any state information about successive datagrams. However, there is enough
information in the datagrams to perform packet fragmentation in the network and reassembly at the
destination. An IP datagram encapsulates transport layer data packets.

Matching the IP address to a specific network device is accomplished by the ARP. It is this protocol that
links the OSI network layer to the OSI link layer. Each hardware device connected to a network is assigned
a hardware address called a media access control (MAC) address. The ARP “resolves” these addresses and,
therefore, associates the IP software address with the device hardware address.

C.2.3 Transport layer

The transport layer service allows applications to use a standard set of function calls in order to
communicate on a variety of networks without worrying about different network interfaces. In the IPS this
function is achieved by the connection oriented TCP and the connectionless UDP.

94
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

C.2.3.1 Connection oriented—TCP

The TCP provides a connection-oriented transport service for the IPS. This means that two computers must
negotiate a connection before data can be transferred. For TCP, one end is termed the server and it listens
for connection requests. The other end is termed the client, which initiates connection requests. Once the
connection is established, it may be held indefinitely or closed by either end. TCP also provides a number
of mechanisms for enhancing the reliability of the data transfer. For example, TCP uses acknowledgements,
sequencing, and dynamic timeout and retry of packets.

C.2.3.2 Connectionless—UDP

The UDP provides a connectionless transport service for the IPS. Two or more computers can exchange
UDP datagrams without having to negotiate a connection. UDP does not provide any reliability
enhancements aside from an optional checksum of the data. Any applications that utilize UDP must provide
their own assurance mechanisms as necessary. UDP datagrams may be sent to multiple destinations using
broadcast or multicast addressing.

C.2.3.3 Ports

To facilitate a plurality of applications within a device (e.g., on a PC with a single IP address), TCP and
UDP transport layer protocols have their own identifiers called ports (0–65 535). Port numbers below 1024
are reserved for standard applications such as HTTP, FTP, Telnet, etc. and are referred to as “well-known
ports”. Port numbers from 1024 through 49 151 may be used by ordinary applications. They are called
“registered ports” because the IANA allows users to register them for common knowledge. “Dynamic
and/or private ports” are those from 49 152 through 65 535. 14

C.2.3.4 TCP/UDP comparison

TCP provides features that ensure the correct delivery of messages; however it does so at the expense of
increased network traffic (often by two to ten times), and the potential for increased latency (TCP can delay
messages by hundreds of milliseconds). TCP is a good choice for “long-term” communications such as file
transfers (e.g., event records, e-mail) and data acquisition that can tolerate short delays (e.g., most SCADA
data).

UDP provides a simple, low overhead message delivery service. However it does not provide any features
that ensure the messages arrive at their destination. This must be handled by the application. UDP is a good
choice for substation applications that require timely communications (e.g., time synchronization,
switching commands) and can tolerate or compensate for network errors.

C.3 IP Addressing
Every device/computer on an IP network must have a unique IP address. For most devices, this means one
unique address, as most devices have one interface to the network. For routers and gateways (devices that
move data from one network to another), this requirement dictates two or more unique addresses depending
on the number of networks. 15

14
IANA information at http://www.iana.org.
15
Currently, IP addresses are 32 b long. However, a movement is underway to slowly phase in 128-b addresses, because the number of
devices connected to the global internet continues to grow rapidly and may eventually exceed the available 32-b address space. These
efforts parallel address assignment and routing initiatives that better utilize the existing address space. The new 128-b address scheme
(termed IPv6) also provides additional features such as quality of service (QoS), auto-configuration, and security headers.

95
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

C.3.1 Structure

An IP address is composed of three parts (see Figure C.4). The Network ID is determined by the first bytes
of the address and depends on the particular range of addresses allocated for use on the global internet. The
Subnet ID allows a large network to be divided into smaller networks, thus simplifying the routing. The
Host ID refers to the particular computer on this smaller network. The combination of the three parts gives
each computer a unique address in the overall network.

Figure C.4—IP address structure

The subnet mask is required for all computers and determines the boundary in the 32-b address between the
subnet ID and host ID. Using the host ID and subnet mask, each computer can determine whether a packet
destined for another computer can be sent directly or whether it needs to be routed through another network
to reach its destination. Valid subnet masks always have only ones on the left and zeroes on the right. For
example, the mask 255.255.255.192 has 26 “1” bits and 6 “0” bits for a total of 64 addresses on the subnet.

Any IP address can be written with the notation AAA.BBB.CCC.DDD/NN where AAA.BBB.CCC.DDD is
the IP address and NN is the number of “1” bits in the subnet mask.

C.3.2 Special cases

A computer on a network cannot be assigned a host ID of zero or all ones (-1). These are reserved for
special use. A host ID or subnet ID of 0 can never be used as a destination address, and is typically used in
network diagrams and configuration programs to refer to a range of addresses. For example, the address
192.168.5.0/24 describes all of the computers with host IDs 1 through 254 on the subnet 192.168.5 with a
subnet mask of 255.255.255.0. The all-ones host ID for this network of 255 (hex FF) can never be used as a
source address, and is used to indicate a broadcast packet. For example, a destination address of
192.168.5.255/24 will cause a packet to be routed to all computers on the 192.168.5 subnet (assuming
broadcasting is enabled for that network).

C.3.3 Address allocation

There are two ways that a network device can obtain its IP address. The first is to have a discrete method
for setting the value in the device (usually via a user interface and/or configuration file). The second is for
the device to query a server on the network for a pre-assigned or dynamic address. The latter type of
allocation uses either BOOTP or DHCP.

For more guidance on address allocation, see IETF RFC 1918 [B43] or Mansfield [B55]. Other address
allocation guidance for substations implementing IEC 61850 [B12] can be found in Clause 8 of IEC 61850-
90-4 [B12].

C.3.3.1 Discrete address allocation

Figure C.5 shows an example of a network where all of the devices have an individually-defined address.

96
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure C.5—Discrete address allocation

Discrete allocation requires that the network administrator keep track of which addresses have been used so
that all of the devices have a unique address, as required by IP. Peer-to-peer communication is
accomplished by entering the specific IP address of the peer device (again, usually via a user interface
and/or configuration file). Discrete allocation is the most straightforward method and is recommended.
Discrete allocations, in most cases, simplify troubleshooting routing problems.

C.3.3.2 Centralized address management

If the network grows larger or the need arises to be able to connect and remove devices from the network
frequently, it may be more efficient to employ a centralized address management scheme. In Figure C.6,
the devices C and D query the address server using DHCP at startup to obtain their IP addresses.

97
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure C.6—Centralized address allocation via DHCP

By expanding on this example, all of the devices on a subnet can obtain their address in this manner. Most
office networks employ centralized address allocation for user workstations to ease the addition of new
computers. DHCP servers can be configured to assign unique addresses from a range to ensure that no two
computers get the same address. This is referred to as dynamic address allocation, and is shown in Figure
C.7. Servers on such a network are assigned a static address so that the services they provide are
maintained at a fixed address. A DHCP response can provide devices with the addresses of various network
servers in addition to the IP address.

Figure C.7—Dynamic allocation via DHCP

For most substation networks, all devices require a fixed IP address and do not support DHCP and/or
BOOTP. Peer-to-peer communication is often required as well as data-reporting channels (e.g., data
concentrator to IEDs), which are most easily configured with predefined, known IP addresses. A DHCP

98
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

server can assign a unique but fixed IP address based on the hardware address (e.g., Ethernet MAC
address) of each device on the subnet. All of the address assignment bookkeeping is handled at the DHCP
server rather than in a separate database. The address server can allocate a range for dynamic devices (e.g.,
configuration laptop computers) that can join the network and be assured of a unique address.

C.3.3.3 Summary of address assignment advantages, disadvantages, and


recommendations

Discrete address assignment has the following advantages:

 Requires only simple configuration parameter(s) at each IED


 No server required on network
 More straightforward for small networks

Discrete address assignment has the following disadvantages:

 Requires maintaining an off-line database of assigned addresses


 Database can become complicated for large networks
 IEDs cannot be added to the network without consulting database

Centralized address assignment has the following advantages:

 All IP address bookkeeping done in DHCP server configuration rather than in a separate database
 Scales well from small to large networks
 Devices can be added to the network without changing or consulting the configuration

Centralized address assignment has the following disadvantages:

 Requires a server application on the network to handle DHCP protocol. If the DHCP server fails,
networked devices will not be able to reboot.
 Requires that substation IEDs support services such as DHCP, DNS, and BOOTP.
 More complicated for small networks.
 Extra parameters of DHCP not needed for substation networks.

Each utility should develop a plan for address allocation that addresses the following:

 Can be expanded for future upgrades.


 Makes intuitive sense for the logical and geographical organization of the substations.
 Can be effectively communicated to the personnel responsible for installation.
 If the substation networks are to be incorporated into an enterprise network structure, the IP address
allocation scheme should be designed to coexist within the larger framework.

99
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

C.3.4 Private address space

The IANA has reserved three address ranges that can be used for private IP networks with no danger of
conflicting with public internet addresses (see Table C.2). All of these ranges can be further divided into
subnets.

Table C.2—IP address space for private Internets


Address range Network notation
10.0.0.0 through 10.255.255.255 10/8
172.16.0.0 through 172.3 1.255.255 172.16/12
192.168.0.0 through 192.168.255.255 192.168/16

C.3.4.1 Sub-network

A LAN can be configured with devices and nodes that are shared but retain a degree of isolation from one
another. The pieces of network are sub-networks. The devices and nodes could interchange messages but,
by virtue of configuration, they do not. In effect, the devices do not know of the existence of the other
devices. Sub-networks reduce messaging traffic to devices that do not need to interoperate.

C.3.4.2 Segments

There are physical and logic limitations that constrain the size of a LAN. These are managed by defining
LAN segments. A segment serves a small geographic area whose center is a device like a switch or router
that passes messages within the segment members and to other segments. This limits messages passing the
outside and compensates for the distance restriction imposed by the media. Network designers often
establish LAN segments to logically group devices together. For example, in a substation one network
segment may be configured for relays and another for meters. Or one segment might be configured for
primary relays and another for backup relays. Configuring segments is a technique to manage the risks of a
LAN failure.

C.3.5 Subnetworks

Network based communication may use a telecommunication system that can provide multiple
communication paths between two IEDs as opposed to a simple common bus arrangement.

Figure C.8 shows communication between A and B can be done through nodes C, D, or E. This provides a
more robust pathway that is both fault and loading tolerant.

100
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure C.8—Message routing between multiple nodes

To allow the data exchange between IEDs on a network, additional information is needed to route the data
through the network. Each message contains addressing information used to help nodes to route the
message to its destination.

A substation communications system may require many segments and links to make a complete physical
media connection between all IEDs. The system may involve media conversion, protocol translation,
gateways, and access devices.

101
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure C.9—Enterprise level network example

Figure C.9 illustrates the network implementations at the enterprise level. Figure C.10 illustrates the
possible complexity of a substation network.

102
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure C.10—Substation network example

The communications network connecting IEDs within the substation may also be connected to, or in
parallel with, a network to serve enterprise users. While many automation system designers envision a
single multi-function network within the substation connected to the enterprise for outside users, it is likely
the system will be comprised of networks and sub-networks within the substation and a variety of
connection options for enterprise users. The configuration of this assemblage, the network topology, is the
“road map” that enables communication. Intersecting points on the map may require devices to change
media, change protocol, extend connection distance, and/or control access. There may also be requirements
to support legacy devices and systems as well.

Connecting substation IEDs to users dispersed throughout the utility and, possibly, the outside world
imposes a different set of communication requirements. Typically, substation-to-enterprise communications
have been a dedicated low-speed pathway used primarily for real-time applications. To add enterprise-wide
users requires more connectivity than simple point-to-point pathways. Some solutions to this requirement
rely on the PSTN to provide users in diverse locations a means to connect to the substation. A more robust
solution for adding remote sites lies in network technology where the substation-to-enterprise traffic is
carried over a WAN. Network technology can employ any number of different media including wire line,
optical fiber, ultrahigh frequency (UHF), and microwave radio, and the public telecommunications network
to connect enterprise users to substation networks or network servers. Networks afford a high-speed
connection more suitable to multiple users and varied applications.

103
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

C.4 IPv6

In the early 1990s, it was recognized that the internet was running out of IP addresses. This situation was
caused by the rapid growth of internet-based applications and devices. These addresses are a fundamental
component of the internet. It was deemed critical that the IPv4 addressing mechanism be upgraded so as to
manage this shortfall. This issue was recognized in the early 1990s. In 1996, a series of RFCs were
released, defining IPv6.

With IPv4, IP addresses are usually displayed as 4 numbers, from 0 to 255, separated by a decimal point
(representing a 32-b number). This length provides for a total of 4.3 × 109 possible unique addresses. With
IPv6, addresses are represented by 8 groups of 16-b values. A colon separates each group. Within each
group are four hexadecimal digits. This yields an address length of 128 b and allows for 3.4 × 1038
addresses.

C.4.1 Differences between IPv4 and IPv6

Along with the additional addresses, there are significant differences between the two versions. For
example, IPv6 uses a different packet format from IPv4 designed to reduce the overhead incurred by
routers for the packet header processing. This ensures that the two protocols are not interoperable.
However, in most aspects of IP stack, there are little differences. The increased address length and different
packet header format offer some useful advantages over IPv4.

C.4.1.1 Larger address space

While the increased length of the IPv6 address spaces allows for a far greater number of addresses, it also
allows for simplified address allocation and more efficient route aggregation.

C.4.1.2 Multicasting

With multicasting, a packet can be sent to many destinations in a single step. In IPv4, this is an option and
can be challenging for an organization to set up. A globally routable multicast group assignment can be
difficult to acquire and inter-domain multicasting solutions can be hard to implement. IPv6 is a new
protocol and has multicast services built-in from the startthere are no broadcast addresses.

Network discovery functions can now build upon this functionality. Hosts that run these servers listen to
the multicast addresses and other hosts will not be bothered when a client sends IP packets to these
addresses.

C.4.1.3 Simplified router processing

While the packet header in IPv6 packets is larger than IPv4, it is somewhat simpler, allowing for more
efficient processing by IPv6 aware routers. In addition, the IPv6 header is not protected by a checksum.
Integrity is provided by link-layer and higher-layer error detection.

C.4.1.4 Jumbograms

IPv6 can support jumbogramspackets as large as (232 − 1) octets. IPv4 is limited to a payload of (216 − 1)
octets. Links with a high maximum transmission unit (MTU) may have improved performance with
jumbograms.

104
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

It is true that substation communications networks are (or should be) isolated from the internet. However,
as new networks are established and older ones upgraded, IPv6 has to be considered, if only for the added
functionality it provides.

105
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Annex D

(informative)

Interconnection diagrams for communications to and within substations

This annex includes block diagrams of different master station and RTU connections.

D.1 Single master station

A single master station-to-single RTU requires only one communication channel as shown in Figure D.1.

MASTER STATION RTU

Figure D.1—Single master station, single RTU

A single master station-to-multiple RTU(s) uses one communication channel per RTU as shown in Figure
D.2. This is also known as “star” connectivity.

RTU 1

MASTER STATION RTU 2

RTU n

Figure D.2—Single master station, multiple RTU(s), radial circuit

With multiple RTU(s) in a multi-drop circuit, as shown in Figure D.3, one circuit is shared, which requires
addressing or other identification method to determine for which device each message is intended. It also
has the possibility of collision when two devices transmit at the same time. A common mitigation method
is that no device except the master transmits unless in response to a request from the master.

106
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

MASTER STATION

RTU 1 RTU 2 RTU n

Figure D.3—Single master station, multiple RTU(s) multi-drop circuit

D.2 Multiple master stations

The use of two master stations with multiple RTUs on a multi-drop circuit is shown in Figure D.4.
Collision avoidance is more complex.

MASTER
STATION 1
RTU 1

RTU 2

RTU n
MASTER
STATION 2

Figure D.4—Dual master stations, multiple RTUs, multi-drop circuit

An RTU with multiple ports may communicate with one master on each port as shown in Figure D.5.
When this method is used, the RTU must be able to differentiate between the two masters and respond to
requests correctly.

107
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

MASTER
STATION 1

RTU

MASTER
STATION 2

Figure D.5—Master station, single dual-ported RTU, radial circuit

D.3 Multiple master stations, multiple RTUs

Figure D.6 shows communications between multiple master stations, which may be located in different
control centers, connected to different sets of RTUs.

RTU 1

MASTER
STATION 1

RTU n

RTU 1

MASTER
STATION M

RTU m

Figure D.6—Multiple master stations, multiple single ported RTUs

Figure D.7 adds direct connect connections between each RTU and each master and removes master station
to master station communications. Addressing and media access schemes should be used, as appropriate.

108
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

RTU 1

MASTER
STATION 1

RTU n

RTU 1
MASTER
STATION M

RTU m

Figure D.7—Multiple master stations, multiple dual ported RTUs

D.4 Combination systems

Sub-master stations may be added as shown in Figure D.8. These are often used as “data concentrators,”
and may be used to collect data from many RTU(s) or other devices, and forward to a master station.

RTU 1

SUB-MASTER STATION
RTU n

MASTER STATION

RTU 1

RTU m

Figure D.8—Single master station, single sub-master station, multiple RTUs

109
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure D.9 shows one master station with multiple sub-masters. Here, the sub-masters (only two are
shown) collect data from multiple RTUs and each sub-master has a single connection to the master station.

RTU 1

SUB-MASTER STATION 1

RTU n

MASTER STATION

RTU 1

SUB-MASTER STATION 2

RTU m

Figure D.9—Single master station, multiple sub-master stations, multiple RTU(s)

D.5 Substation gateway connections (legacy to standard protocols)

A gateway/data concentrator may be used to collect data using one protocol and then communicating that
data to the master station using a different protocol. Refer to Figure D.10.

IED 1

IED 2
GATEWAY / DATA
MASTER STATION
CONCENTRATOR

IED n

Figure D.10—Single master station, substation gateway/data concentrator

Figure D.11 illustrates multiport gateway/data concentrators may serve multiple IED/master station
combinations.

110
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

IED 1

MASTER STATION 1

IED 2
GATEWAY / DATA
CONCENTRATOR

MASTER STATION 2
IED n

Figure D.11—Dual master station, substation gateway/data concentrator

D.6 Networked systems

Ethernet or other network technology may replace serial connections and modems as shown in Figure D.12.

IED 1

LAN

WAN
MASTER STATION ROUTER ROUTER SWITCH IED 2

IED n

Figure D.12—Single master station, substation WAN/LAN connection via routers (firewall
not shown)

Ethernet or other network technology may replace serial connections and modems as shown in Figure D.13.

111
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

MASTER STATION ROUTE IED

LA

WA ROUTE SWITCH IED

MASTER STATION ROUTE


IED

Figure D.13—Multiple master station, substation WAN/LAN connection via routers

D.7 Network physical topologies

Network topologies are either logical or physical. A logical topology is the way that data passes over a
network from one IED to the next without regard to the physical IED interconnection. The physical
topology of a network maps the IEDs of the network and the connections between them. There are two
major groups of topologies: point-to-point and point-to-multipoint. Point-to-point connection only connects
two IEDs together. Point-to-multipoint networks have several major network topologies for
communications.

D.7.1 Point-to-point networks

A point-to-point connection is an individual communication channel between two IEDs. It is the simplest
solution to provide data exchange between two IEDs.

Many IEDs may connect point-to-point to a multi-ported controller or data concentrator that serves as the
substation automation system communications hub. In early implementations, these connections were
simple TIA-232-F serial pathways similar to those between a computer and a modem. TIA-232-F is one of
the first standards used to connect terminal devices to leased wire lines, where the telephone system
supplied the interconnecting device—a modem. Figure D.14 illustrates a more modern TIA-232-F point-to-
point connection.
Substation
IED Server
A1

IED
A2

IED
A3

IED
A4

IED
X1

IED
X2

Figure D.14—Point-to-point IED to server connections

112
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

TIA-232-F does not support multiple IEDs, so IED addresses are not required (unless required by the
protocol). When the media is copper, TIA-232-F is typically used for short distances, with a limitation of
about 50 ft, depending upon the cable quality (capacitance) and baud rate. Cables with lower capacitance
allow transmission over longer distances than cables with high capacitance. The longer a cable, the more
the capacitance increases and the baud rate decreases. Eventually, the cable will be too long to function
even at a very low baud rate. To extend the distance, fiber optic transceivers can be used or TIA-232-F to
TIA-485-A converters or TIA-232-F to TIA-422-B converters can be used. To electrically isolate TIA-232-
F cables in a substation environment, the designer/specifier should use fiber optic transceivers to convert
the electrical signal to optical and back again. Other options are to provide a device that provides optical
isolation and surge suppression. Some IEDs support a serial TIA-232-F or TIA-485-A fiber optic port(s).
When using fiber optic transceivers in IEDs labeled as “RS232” or “RS485,” similar transceivers should be
used at either end to ensure compatibility.

TIA-422-B (formerly EIA-422 and RS-422) is similar to TIA-232-F (not addressable, point-to-point)
except it is two pairs: one outbound and one inbound. TIA-422-B is intended to allow longer distances (up
to about 4000 ft, as opposed to TIA-232-F, which is 50 ft).

D.7.2 Point-to-multipoint networks

Many substation control systems use point-to-multipoint IED connections. IEDs that share a common
protocol can usually support the same communication pathway wherein they share the channel.

With a point-to-multipoint topology, many IEDs are connected together on the same physical media. This
communication supports only one data exchange at a time, and some protocol is needed to allow one IED at
a time to use the communication media. In a multi-drop configuration, the master broadcasts a message that
is received by all the devices connected to the media. The IED with the address responds and the master
waits for the answer before polling the next IED; therefore, timeout settings can delay the polling of the
other IEDs on the same channel.

When IEDs on a shared physical channel use a protocol that supports unsolicited reporting (such as DNP3),
the unsolicited responses by IEDs can disrupt polling by causing collisions on the shared physical channel
and cause communication trouble. Subclause 9.2.10 of IEEE Std 1815™-2012 [B20] suggests some
optional collision avoidance techniques.

D.7.2.1 Bus topology

A bus topology has each IED connected to the same physical media as shown in Figure D.15.

IED IED IED IED

Figure D.15—Bus topology

Figure D.16 illustrates a TIA-485-A communication bus seen in a substation. TIA-485-A is the most
common point-to-multipoint bus. It uses a twisted shielded pair copper cable. The TIA-485-A standard
states to use termination resistors at both ends of the line that are equal to the characteristic impedance of
the cable, typically 120 Ω for twisted shielded pair cable. Termination resistors may not be required for
shorter cable runs and baud rates slower than 115 kbps. Good TIA-485-A cables have a braided shield,

113
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

nominal impedance of 120 Ω, and a capacitance of 12 to 16 picofarads per foot. Channel length is typically
4000 ft maximum. TIA-485-A buses support up to 32 devices on the channel. Repeaters can be added to
increase the number of IEDs by 32 devices and cable length by 4000 ft for each repeater. The longer the
bus, the more likely communication errors will occur because of reflection and noise on the cable.

Generally, the longer the bus is the slower the baud rate is. TIA-485-A may run as fast as 1.0 Mbps,
although most operate closer to 19.2 kbps or slower. The TIA-485-A bus should be linear, end to end.
Stubs or taps will cause reflections and communication trouble. TIA-485-A devices are wired in a “daisy
chain” arrangement. The channel direction is turned around when each device takes control of the bus
while transmitting.

Substation
Server
RS-232/485

IED
Y1

IED
Y2

IED
Y3

IED
Y4

Figure D.16—TIA-485-A IED communications bus with converter

TIA-422 can also be used point-to-multipoint, but only one master is allowed.

Ethernet is typically a logical bus topology now that has a physical star topology.

D.7.2.2 Star topology

In a star topology, such as Ethernet, each IED is connected to a special node at the center that can be
passive, providing a path for the message to traverse, or active, regenerating the electrical signal. Hubs
simply repeat any message on all ports. More intelligent hubs are switches, which route a message to the
port where the targeted IED is connected. The data exchange on each segment is isolated with full duplex
and because separate optical fibers or twisted pair wires are used for data transmitted and received,
collisions are avoided. This creates a situation where each IED appears to be on a single “virtual” LAN, so
that collision and contention problems are minimized or eliminated. Figure D.17 is an example of the use of
star topology

IED IED IED IED

Hub/switch

Figure D.17—Star topology

114
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

D.7.2.3 Ring topology

In a physical ring topology, each IED is connected to the next with the entire network forming a closed
circle as shown in Figure D.18. Each IED is isolated from all but two IEDs. Each IED has a physical
connection to two adjacent IEDs. Ring networks are less efficient than star networks because data travel
through more IEDs before reaching its destination.

One exception to a physical ring network is a token ring network, which is a logical ring topology, because
it uses a physical star or bus network. Token ring networks are deprecated as they have been superseded by
other network topologies discussed in this recommended practice such as HSR and PRP.

IED

IED IED

IED

Figure D.18—Simple ring topology

115
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Annex E

(informative)

Serial communication channel analysis

E.1 Introduction

The responsiveness of a SCADA system is limited primarily by the data throughput and latency
characteristics of the data communication network that connects the master station to its RTUs (and any
attached IEDs). It is therefore essential to be able to check that the network design will support acceptable
system performances.

This annex describes a two-step procedure to identify any necessary changes, either to the network design
or to the SCADA system specifications. The procedure addresses only bandwidth and delivery-time issues
in the communication network. All other network implementation issues such as connectivity, cost,
security, reliability, maintainability, and expandability are resolved separately. It applies to any network
configuration where one or more RTUs are connected to one master station communication network port.

The required communication performances in terms of the SCADA system data volumes and required
update times are documented in E.2. Subclause E.3 verifies that the real or virtual data communication
channel provided by the communication network between a master station and its connected RTUs can
provide the required performances. Finally, E.4 provides an example of the application of this procedure.

E.2 Specify the performance of a master station to RTU communication channel

a) List all required types of master station “on-line” functions that involve data communication with
an RTU (and any attached IEDs). These functions include the following:
1) Remote control of RTU internal and external IEDs
2) Reporting of measured and processed data values (status, analogs, counters, etc.) from RTUs
and their local IEDs to the master station
3) Transfer of analog output values from the master station to RTU and IED processors
NOTE—Transfers of configuration, events, or other files to and from the RTU and IED processors are
considered to be “off-line” or “setup” functions that do not affect the performance of the system.
b) Specify the required repetition (update) interval for every on-line function that is to be executed
periodically. Typical values are as follows:
1) Two seconds, 10 s, and 30 s for acquisition (upload) of measured values
2) Thirty seconds for delivery (download) of generator automatic control data
3) Fifteen min, 30 min, and 60 min for upload of counter data (each transaction is typically
preceded by a counter control function)
4) Hourly and daily for transfers of processed data
c) Specify the maximum acceptable execution time limits (referred to as T below) for all on-line
functions, both periodic and event-driven. Typical values are as follows:
1) One second for any device control function

116
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

2) One second for upload of any data transferred on demand (for example, status data acquired
“by-exception”)
3) Less than 50% of the update interval for periodic functions (some functions may require a
lower limit to provide sufficient time for master station processing between data updates)
4) No specific limit for file transfers
d) List, for each connected RTU and any attached IEDs, the data size in bytes (information bytes
exclusive of communication overheads) to be transferred in each update cycle of each periodic
function. Allowances should be made for likely future expansions of these units.

E.3 Channel performance analysis procedure

a) Estimate (or measure) the typical time, TC, required to execute a device control function. This time
includes the operating times for all channel equipment while servicing all layers of the
communication protocols used, in a channel that is ready, idle, and operating with no
communication errors.
b) Check that TC is much less than the limit value, T, specified in step c)1) of E.2. above. T is a
fundamental network design parameter that limits the worst-case network transport delay. For
example, the value of one second listed as typical for T inherently precludes routing any part of the
network via a geosynchronous satellite.
c) If TC exceeds about 25% (a recommended practical limit) of T, the communication subsystem
design should be revised to reduce TC or the specified value of T should be increased. Reducing
TC relative to T increases the channel time available for transfers of other data. Preferably, TC
should be less than about 20% of T. However, T cannot exceed the fraction specified in step c)3) of
E.2 of the shortest periodic update interval specified in step b)1) of E.2.
d) To meet the limit value T for completing the execution of a device control function, the channel
time required to execute any uninterruptible communication function should be less than (T – TC).
This time interval is typically too short to support the transfer of the largest files listed in step d) of
E.2. Such files are therefore divided into multiple segments for delivery.
e) Estimate (or measure) the maximum file segment size that can be transferred in time (T – TC)
when the channel is open, idle, and error-free.
f) Calculate, from the entries in step d) of E.2 above, the number of data segments required to be
transferred during each execution of each periodic function.
g) Calculate the total number of periodic data segments to be transferred, and thus the required total
transfer time for routine data, during any 60 min period of operation of the RTU communication
channel.
h) Check that the total channel time required to transfer all periodic data is less than about 45 min, for
a channel loading by routine data of less than a practical target value of about 75%. Unused
channel time is then available for other, future, functions.
i) Check that the data transfer processes will meet the execution time requirements of step c)of E.2.

E.4 Illustrative example

The results of each step in the analysis procedure, when applied to a network servicing one large-capacity
RTU, are as follows:

E.3 a) Result: The measured value of TC is approximately 400 ms in the initial communication network
configuration.

E.3 b) Result: The initial value of TC is about 40% of T.

117
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

E.3 c) Result: As the initial value of TC exceeds 25% of T, network changes were required to reduce
data delivery times. These changes reduced the value of TC to 180 ms.

E.3 d) Result: With T set at 1.0 s, the nominal value of (T – TC) is 820 ms.

E.3 e) Result: The measured value of the maximum file segment size is about 0.2 kB.

E.3 f) Results for the example RTU:

Periodic function Data segments


2 s measurements (1800 transfers/h): 1
10 s measurements (360 transfers/h): 2
30 s measurements (120 transfers/h): 1
30 s controls (120 operations/h): 1
15 min counters (4 transfers/h): 1
60 min processed data in (1 transfer/h): 30
60 min processed data out (1 transfer/h): 10
24 h processed data 0

E.3 g) Result: From item f), the total number of data segments to be transferred per hour is as follows:

(1800 + 2 × 360 + 120 + 120 + 4 + 30 + 10), that is, 2804.

E.3 h) Result: Assuming a convenient data segment transfer rate of one/second, the total channel time
required per hour is 2804 s, or nearly 47 min. This calculated channel time represents a channel loading for
routine traffic of about 78%. This high value can be accepted as it includes sufficient time (180 msec.) for
the execution of one device control function in every one-second interval. It also provides the maximum
length of 820 ms for the transfer of every file segment although most will be shorter. In addition, there are
796 (3600–2804) unused one-second time slots available during each hour for future functions.

E.3 i) Result: Assuming, for simplicity, that the time required to execute one device control function
(180 ms) is reserved in each one-second time slot, then:

 The time required to upload any short data file on demand is available in each of the majority of
one-second time slots that are not pre-empted by device controls.
 Transfers of all 2 s measurements can be completed in the first 820 ms of each time slot, that is, in
41% of their update interval.
 Transfers of all 10 s measurements can be completed in 3.82 s, that is, in 38% of their update
interval.
 Transfers of all 30 s measurements can be completed in 5.82 s, that is, in 19% of their update
interval.
 Transfers of all 30 s generator control commands can be completed in 7.82 s, that is, in 26% of
their update interval.
 Transfers of all 15 min counter data can be completed in 10 s, that is, in about 1% of their update
interval.
 Transfers of all 60 min data can be completed in 400 s, that is, about 11% of their update interval.
 These results show acceptable performance of the example RTU and network configuration.

118
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Annex F

(informative)

Protocol models

Three of the most popular protocol models are discussed in this annex.

F.1 Master slave

Master slave communications occur when the master controls all of the traffic on the channel. There are
two different types of masters—dedicated master and token-passing master. This terminology is historically
applied to serial communications, where media access is typically controlled by a single master on a
channel, it being the only device allowed to transmit except in response to a specific request. The master
requests data from a specific IED (even if only one IED exists on the channel), and the IED does not
transmit without having been queried; although exceptions do exist and are still referred to as
“master/slave”.

Polling schemes involve no network contention because access to the medium is granted in an orderly
fashion with every device taking its turn.

F.1.1 Centralized

With centralized polling, all IEDs are addressable and the master IED will send out messages only
addressed to a single slave. Each device has a different address as defined in the protocol being used. All
IEDs receive the message, but only the addressed IED responds. The master communicates to each IED
one at a time so as to prevent communications collisions. Figure F.1 is a block diagram of a master/slave
(client/server) configuration. Figure F.2 shows that only the addressed IED will respond to each master
request.

Figure F.1—Centralized polling

119
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure F.2—Centralized polling scheme

F.1.2 Token passing

With token passing, each IED acts as a repeater of a message called a token and each IED can be both a
master (requesting data from other IEDs) and a slave (sending requested data to other IEDs). The token
may contain some data that is copied by the receiver. If the token contains no data, then an IED can use it
and fill in its information contained in the token.

Programmable logic controller (PLC) communications and some other control systems use token passing
schemes to give control to IEDs along the bus. A token is passed from IED to IED along the
communications bus that gives the IED the authority to transmit messages. While the IED has the token, it
may transmit messages to any other IED on the bus. Different schemes control the amount of access time
each “pass” allows. When the token is lost or an IED fails, the token is regenerated. Therefore, token
schemes have a mechanism to recapture order. The most significant advantage of a token passing scheme is
that it is deterministic. That is, every IED is guaranteed an opportunity to transmit data, and the total time
to accumulate a specific amount of data from each device can be calculated. Applications that require timed
responses or completion of control sequences will benefit from this determinism.

120
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure F.3—Token ring access flowchart

F.2 Client-server model

This is the most popular model for network applications. Each device may instantiate multiple instances of
both clients and servers, allowing connections to multiple other IEDs.

The characteristics of a server are as follows:

 It is passive, similar to the slave in master-slave communications


 It waits for requests from clients
 Upon receipt of requests, it processes them and then sends a response

The characteristics of a client are as follows:

 It is active, similar to the master in master-slave communications


 It sends requests to servers
 It waits for and receives server replies

Master-slave and client-server communications are similar. The biggest difference is that, generally, there
is only one master, whereas there can be multiple clients.

121
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

F.3 Peer-to-peer networks

There is a growing trend in IED communications to support peer-to-peer messaging. Here, each IED has
equal access to the physical media and can message any other IED. Thus, each IED is both a client and a
server. This is substantially different than master-slave communications, even when multiple masters are
supported. A peer-to-peer network provides a means to prevent message collisions, or to detect them and
mitigate the collision. In this configuration, each IED can communicate to each other in an unsolicited
manner.

122
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Annex G

(informative)

Network management

As with a traditional system, a network-based electric utility information system requires a set of tools to
commission, monitor, and maintain the communication channels and end devices. The difference is that a
network-based approach allows connectivity to a much larger portion of the overall system from a central
location or from the substation. This facilitates faster troubleshooting and reconfiguration. This clause
includes a discussion of some of the common tools and practices used to configure and monitor network-
based substation devices.

Care should be taken when implementing these tools and practices as they may expose network and/or
device vulnerabilities. Network management tools and practices should be used in accordance with
company IT security policies and with regard to the threats defined in 6.3.

G.1 Configuring network settings in IEDs

All IEDs share one common trait: they must be given an IP address before they can communicate on the
network. One way of assigning an IP address to an IED is via DHCP over the network from a central
server. However, many substation IEDs require a fixed IP address and thus do not support DHCP. Some
IEDs will have a default IP address that can be used to connect to the IED over the network with a vendor
maintenance or configuration tool. Some IEDs will support the preceding method and/or require the use of
a serial maintenance or configuration port to assign the IP addresses to the IED as shown in Figure G.1.

Figure G.1—Vendor-specific configuration tool

Once the initial IP address is entered, the serial cable can often be removed. Access to the IED continues
through the network interface as shown in Figure G.2.

Figure G.2—Local network access for configuration

123
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

To improve security, some vendors only allow certain information to be changed through the network
interface. For example, passwords and critical parameters may require local maintenance port access.

Vendor-specific tools are the primary means for accessing all the information available in utility
information system equipment. Newer utility protocols continue to add features to enable the transfer of all
types of data including text, floating point values, and files. Virtually all vendors provide their own
programs to configure, retrieve, and view device information in a way that they see as providing the most
value or product distinction. This trend highlights the importance of being able to access all the devices
from a central location and take advantage of the huge amount of information available.

Figure G.3—WAN access

G.2 Network management in the electric utility information system

Network infrastructure devices such as switches, routers, and gateways (protocol converters) can provide
status information as well as features for managing their portion of the network. This information can
generally be accessed from the network using one of two methods: via the SNMP or via HTTP.

Both methods can be used at the same time if the wide area network is composed of some devices that
support SNMP and some that support HTTP.

G.3 Tools

This clause discusses network tools that should be part of any utility network engineer’s toolbox. Note that
G.3.1 and G.3.2 assume the network is not configured to block ICMP messages.

G.3.1 Ping

The ping program sends an ICMP echo request to an IP host on the network and waits for a reply. If the
reply is received, the round-trip time is printed. If no reply is received, a message indicating a timeout

124
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

condition is printed. There are several responses to ping, and each response provides a different meaning.
Request time out means that the ping was sent to the device but no answer was received. A destination host
unreachable means that the ping could not be sent because there was no route to the host or the MAC
address was not known and the message could not be sent. A general error means that the device initiating
the ping has no network connectivity. Virtually all computer operating systems that provide interactive use
have a version of the ping program. All devices that provide the IPS will respond to the ICMP echo request,
because the response does not require any other applications to be running.

Use the ping program to verify that:

 A device has a correctly configured IP address and is on


 The network route to the device is setup correctly

Examples

% ping 10.10.1.5 <——Linux/Unix

C:\>ping rtu17.scadalan.net <——DOS and Windows

Note that ping provides a vulnerability to the network by allowing a simple process to discover a
rudimentary map of the network. Disabling ping makes it more difficult for an attacker to discover devices
for attempts to compromise.

G.3.2 Traceroute

The traceroute program uses UDP datagrams and ICMP responses to map all of the hosts between two
points on an IP WAN. It prints out a list of all the intermediate and destination network interfaces along
with the round-trip time for each of the three test datagrams. If a ping fails, traceroute can often show the
node where the failure took place.

Use the traceroute program to:

 Show and verify the network path to another host


 See where the last point a packet is successfully routed on a failed link

Examples

% traceroute 10.200.4.67 <——Linux/Unix

C:\>tracert www.ieee.org <——DOS and Windows

G.3.3 Network analyzers

Network analyzers are also known as network monitors or packet sniffers. Network analyzers provide an
independent view of the bits and bytes on a network to help troubleshoot configuration or implementation
problems. For example, the following problems would be candidates for network analysis:

 Failing to make TCP connections.


 UDP datagrams being lost between a control center and substation.

125
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

 TCP connection is made, but the process responsible for handling the higher level data (e.g., utility
protocol) is not receiving any data.
 A utility is trying to get two pieces of equipment to communicate. At the site, both equipment
vendors claim they have implemented the protocols per the specification but the data transfer is not
as expected.
Figure G.4 shows how a network analyzer works on a network. As the IP packets pass between computer 1
and computer 2, the monitor computer captures all the traffic for display.

Figure G.4—Network analyzer operation

Capturing IP packets in this fashion is acceptable, because an Ethernet hub broadcasts all messages on its
segment to all hosts. If the Ethernet segment is based on a switch, however, the monitor computer will not
see any of the packets because a switch, by design, only passes the Ethernet frames to the destination port.

Unmanaged switches cannot overcome this (Figure G.5, Example A) and a “test” hub must be used on the
segment that is to be monitored (Figure G.5, Example B). A managed switch can typically be enabled to
either behave as a hub or to pass all frames to a specific port (Figure G.5, Example C). This feature is
called port mirroring or switched port analyzer (SPAN).

126
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure G.5—Monitoring with switches

Table G.1 shows the recommended network analyzers.

Table G.1—Network analyzers


Program Description
TCPdumpa, b Command line based; available for any computer
Wireshark® a, c GUI-based; available for most computers

a
This information is given for the convenience of users of this recommended practice and does not
constitute an endorsement by the IEEE of these products. Equivalent products may be used if they
can be shown to lead to the same results.
b
TCPdump is available from the Web site http://www.tcpdump.org.
c
Wireshark® is available from the Web site http://www.wireshark.org.

The most important feature of a network analyzer is its filtering ability. When viewing network traffic,
there is a tremendous amount of data that is not applicable to the problem at hand. Both of the
recommended analyzers have extensive filtering capabilities with the GUI-based tool adding color
highlighting of the packet fields as well.

It is also worth mentioning that some protocol-specific tools have network monitoring built in.

The recommended remote terminal mechanism is SSH such as can be found in one of the SSH
implementations (Barrett and Silverman [B1]). There are implementations available for most operating
systems in both freely-distributable and commercial product form.

127
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Annex H

(informative)

Features of devices and media supporting Ethernet-based networks

H.1 General device features

The following general features will be supported in most devices supporting Ethernet-based switching and
routing functions, except for hubs.

H.1.1 Port speed and fiber-optic interface

Original 10 Mbps Ethernet has been practically replaced with 100 Mbps. With the CAT-6 (twisted pair)
physical layer TIA-568 [B60], both speeds are still available and are supported automatically by the
associated hardware interface (10/100/1000). In the case of the fiber-optic interface, the speed selection is
not automatic, resulting in virtual obsolescence of the 10 Mbps option. It should also be noted that
connection between 10 Mbps and 100 Mbps network segments is not seamless. It requires bridging, and
needs to be avoided if any real-time traffic is being exchanged between the two network segments.

H.1.2 Full duplex operation and collision-free environment

Modern switches offer a full-duplex interface capable of simultaneously transmitting and receiving remote
device traffic. This capability doubles the available network bandwidth (200 Mb/s), while eliminating the
possibility of local packet collisions. Collisions involving traffic from different ports are avoided by storing
(within the switch) all of the traffic that cannot be immediately forwarded. This storage keeps all of the
incoming pipelines open, while making it possible to optimally schedule/combine the outgoing port traffic.
Internal switch bandwidth should be specified as non-blocking or dimensioned to support simultaneous
operation of all ports (several Gbps in the case of an 8 ports 100 Mbps switch), thus maximizing the overall
data throughput.

The primary function of an Ethernet switch is to establish a direct connection between the sender and the
receiver (based on the individual device MAC address). Individual packets are forwarded only between the
two communicating ports without affecting the bandwidth available to other ports.

H.1.3 Priority queuing

Because it is a multipurpose network, Ethernet often needs to carry vastly different types of network traffic.
This is best illustrated by thinking of a power system substation application in which mission-critical IEC
GOOSE traffic (e.g., command to trip a breaker) coexists with SCADA and device management type traffic
(e.g., event oscillography retrieval). It is obvious that different messages have different delivery time
requirements, making it necessary to separate the incoming traffic into different priority queues for
transmission or forwarding.

The priority queuing mechanism is described in IEEE Std 802.1D-2004. Priority is accomplished by
inserting a four-byte “tag” into the standard Ethernet frame header. Actual tag position is shown in Figure
H.1.

128
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Figure H.1—Layer 2 tagged Ethernet MAC header

The supported number of priority queues varies among switches, with the minimum of two needed to claim
IEEE Std 802.1D support. Queuing strategy is usually configurable, with “strict priority” being one of the
options (no low-priority messages are forwarded until all of the high-priority traffic has been processed).

Separating the traffic into priority classes makes it possible to ensure bandwidth will be available for
mission-critical protection type applications and SCADA, thus eliminating congestion and providing a
more firm guarantee for high-priority traffic delivery.

Separation of traffic into multiple priority classes makes it possible to calculate and control the worst-case
latency encountered by the highest priority real-time traffic, thus creating the determinism needed for real-
time network operations.

H.1.4 VLAN support

VLAN is a mechanism for partitioning the network into multiple “virtual” domains. It uses the 12-b VLAN
tag field shown in Figure H.1 to divide the network into a maximum of 4094 individual domains
(VID=000, FFF denotes special cases). VLAN can also be configured in a managed switch on an individual
physical port basis in order to support devices without tag insertion capability. It can also be used to
enhance network security by separating protection and control traffic into different VLAN domains.

H.1.5 Loss-of-link management

Because of the high reliability expected from power system automation devices, some manufacturers have
equipped their products with a redundant Ethernet port interface. This interface normally provides a “stand
by” channel capable of taking over the network function in case of the primary port failure.

While detecting the loss of receiving fiber is relatively trivial (end device notices the absence of “link”
pulses), detection of the outgoing (transmit) fiber failure is more involved. In the case of outgoing link
(from the IED) failure, the receiving device (Ethernet switch) will know that communication with the IED
is missing, but the IED will be under the impression that everything is OK because it continues to receive
link pulses. As a result, IED will not switch over to its backup port.

This problem is resolved by adding intelligence to the Ethernet switch, which disables the outgoing link
pulse stream after detecting the incoming link failure. Additional features may be used to enhance the
switch learning capabilities immediately following a “loss-of-link” event (flushing of the MAC address
table, port flooding, etc.). Exact recovery procedures are not fully standardized, and remain
manufacturer/product specific, thus requiring verification before actual field deployment in case of port
failure or to support redundant networks in the substation.

129
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

H.1.6 Flow control

The CSMA technology in Ethernet was originally used to provide network access arbitration and collision
detection, and appears to be unnecessary in the “collision-free” Ethernet switch environment. This
appearance is deceptive.

In the Ethernet switch-based environment, the PAUSE function can be used to implement flow control on
all full-duplex links if the attached station supports it. A PAUSE function consists of a special frame
containing a timer value that is sent by the data link control layer of a switch. When the station receives a
PAUSE frame, it must stop transmitting until either the timer expires or is canceled by the receipt of a
PAUSE frame from the switch with a timer value of 0. The station may also send PAUSE frames to the
switch if, for example, it is receiving data faster than an application can process it.

In general, the use of the PAUSE function may be unnecessary because an overrun link partner can simply
discard frames it cannot process or forward. In this case, higher-level protocols such as TCP will assume
the network is congested because of frame loss and/or delay and reduce the rate of transmission
automatically. Support for the PAUSE function is determined when the link is connected using the same
negotiation technique used to establish speed and duplex.

H.1.7 Multicast propagation control

Basic switching is very effective in managing the directly addressed Ethernet traffic. However, its benefits
disappear when exposed to a “broadcast” and “multicast” rich environment.

The multicast concept is a technology used by power system protection applications to simultaneously send
analog values, state changes, or commands to several peers. Instead of using multiple, individually-
addressed messages, this is accomplished by sending a single multicast message to the switch, which in
turn forwards it to all outgoing ports. Receiving devices are simply configured to listen to particular
multicast addresses, thus making it possible to discard the unwanted network traffic.

Because of indiscriminate output port flooding, extensive use of multicast traffic could jeopardize the
operation of large substation networks. Partitioning multicast traffic and limiting it to individual “multicast
domains” can eliminate this problem and optimize system performance.

Multicast domain partitioning can be achieved by using the VLAN mechanism described in IEEE Std
802.1Q-2018. Careful network planning and relay/Ethernet switch setting configuration may be required in
order to take full advantage of this mechanism.

H.1.8 Environmental hardening

Ethernet networks are well positioned to become an indispensable part of every substation. As such, they
will be involved in critical SCADA and protection traffic delivery and are expected to provide the highest
levels of reliability/availability. All hardware components used within the substation environment should
be environmentally hardened and adequately tested. The list of applicable environmental requirements is
given in IEEE Std 1613 and IEC 61850-3 [B12], where edition 1 is deprecated.

H.1.9 MAC filtering and port lockout

MAC filtering allows a switch to filter the traffic allowed on a port to one or more MAC addresses. This
feature can be used to enhance system security, but also requires that the MAC addresses of all devices be
known so the switch can be programmed. Some switches can be set in a mode where they will learn the

130
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

MAC addresses on a port, thus allowing a port to be locked down after a device has been connected to the
port.

Port lockout is another feature that enhances system security by programing a switch to effectively “close,”
“ignore”, or “not allow” any traffic on a port that is locked out. This feature requires switch programming
and is useful when trying to limit connections to the substation LAN when a physical port is available.

H.2 Media features

H.2.1 Wired networks

See IEEE Std 525 for a complete discussion of wired cabling options and recommendations for substation
communications networks.

H.2.2 Wireless networks

While the use of WLANs in the industrial world is working its way to the factory floor, WLAN use within
the substation or control house is still in the developmental stages for anything more than basic or non-
critical applications. Because of limited bandwidth when compared to wired LANs, implementers of
WLAN technology should first classify the applications running on the WLAN according to at least the
following:

 Low- or high-importance for proper operation of the substation function


 Low-speed or high-speed traffic
 Can accept possible time delays (latency) in the transmission of data
 Low-volume messages or high-volume messages (referring to message size)
 Infrequent or frequent messages (number of transmissions per unit time)
 Purpose (control, status, analogs, etc.)
 Degree of risk if the communication is compromised or delayed due to interference or
compromised security

The applications that run on a substation LAN could also run on a WLAN. However, a WLAN may be
more typically used to extend the substation LAN out to the substation yard. Thus, a WLAN may be used
to establish connections for at least the following:

 Breaker monitoring, status, and control


 Switch status
 Transformer status and temperature
 Transformer tap-changer status and control
 Transmission of metering values
 Intrusion detection

For these applications, the use of certain WLAN technologies may be preferred to other types of
technologies due to bandwidth, potential interference, or other constraints. In addition, the WLAN
technology should be capable of performing in the substation environment.

131
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Technologies are available on simple radio equipment that has traditionally been used in industrial
environments for short-range telemetry and control purposes. The technology often takes the form of a
digital signal on a licensed frequency. This is usually a very narrowband emission and is used for low data
rates such as 9.6 kbps or less. This technology might be suitable for many of the control functions related to
the substation, provided that the bandwidth limits and device addressing capability can handle all of the
requirements. The licensing aspect removes much of the risk of interference, especially that which
originates from a consumer’s unlicensed devices.

Other technologies, such as the unlicensed IEEE Std 802.11™ [B14] series of spread-spectrum broadband
devices, offer much higher bandwidth capacity. Because the technology is unlicensed, there is a risk of
potential interference from nearby users (unintentional as well as intentional). It is difficult to quantify and
guarantee the availability of this technology, which may make it suitable for only specific applications. The
technology may be quite adequate for carrying security video from the substation yard, as well as extending
the substation WLAN to other non-critical devices located in the substation yard. Metering applications,
which usually need a continuous communication path, could operate over these sorts of links. Adding
device control using this technology could present a considerable risk that the link would not be present and
interference-free at the time it is required.

Spread spectrum radios have advantages of some degree of immunity to noise, though this is not a
guarantee of perfect operation during substation disturbances. This subject will need much more detailed
investigation.

It should be noted that there are frequency-hopping radios—which is another way to achieve a spread
spectrum transmission—that recognize unusable frequencies and will either skip them or automatically
change to another set of frequencies. While this may seem to be a good solution to the interference
problem, it must be recognized that this operation takes a little time during which communications over the
link may be interrupted and cause intolerable latency to the application. A person intent on harm could
present enough disturbing signals to keep the radios looking for channels and, as a result, effectively
achieving a DoS attack for the WLAN.

The commonly used unlicensed frequencies include spectrum in the vicinity of 915 MHz, 2.4 GHz, and
5.8 GHz. There are industrial and scientific allocations at other frequencies, both higher and lower. An area
of study that has yet to be explored is the propagation of these wavelengths (nominally 33 cm, 13 cm, and
6 cm) in substation situations, particularly with respect to noise from corona and possibly electrical arcs.

Because wireless technologies were originally introduced for the commercial and office world, the
technology may also be susceptible to the harsh substation environment. Because WLAN equipment is
networking equipment, when installed in a substation IEEE Std 1613 should apply. Note that the impact of
WLAN equipment on other equipment in the substation, particularly relays and other protective devices,
has not yet been addressed in IEEE or IEC standards. However, both standards groups are looking at
extending the frequency ranges in IEEE Std C37.90.2 [B24] and IEC 61000-4-3 [B11] to cover the 2.4–
5 GHz frequency range. In the meantime, the designer should confirm that the power densities for WLAN
technologies are not sufficient enough to pose problems for equipment that already meets these standards
for other frequency ranges.

132
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

H.3 Network devices

H.3.1 Hubs and switches

Hubs and switches provide the electrical interconnection of nodes on a modern Ethernet network. 16 The
following three types of devices are used:

 Unswitched hubs—Forward all traffic from a network segment to all other segments.
 Unmanaged switches—Forward traffic only to the segment containing the destination node.
 Managed switches—Forward traffic only to the destination segment with some security constraints
such as MAC address filtering, VLAN, QoS, and prioritization features. Managed switches also
support remote diagnostics and port lockout.

H.3.2 Media converters

Media converters are used to convert either from copper to fiber or from one fiber wavelength/transmission
rate/mode-type to another. All types of copper and fiber signaling schemes allow the detection of loss of
connection. Media converters sometimes transmit a null output stream upon loss of input connection and
sometimes cease transmission entirely. In addition, some converters cease transmissions on a port if the
corresponding receiver on that port indicates a loss of connection. The reason that problems exist is because
some devices emulate loss of connection even when a physical connection has been established (laptop
computers with power-down circuitry are a prime example of this). Generally, system operation is more
predictable if intermediate devices ignore the loss of connection (high-layer protocols will often reliably
indicate connection losses).

Media converters add additional failure points to the substation LAN and also require external power. Due
to the additional failure modes and the potential impact on station battery capacity, their use should be
minimized.

H.3.3 Routers

Routers are networking devices that allow interconnection of LANs. The interconnection of LANs is
known as a wide area network (WAN). The router’s primary task is to move or “route” IP packets between
a LAN port and the appropriate WAN port (or between multiple WAN ports). The router often functions as
a physical layer translator during this data movement. For example, the LAN devices may be
communicating via Ethernet while the WAN port operates using frame relay or ATM. Some vendors
provide a single device that incorporates the features of both a router and an Ethernet switch.

Routers also allow networks to be segmented in the event of a failure of other networked components. The
use of routers at the access point of the substation provides utilities with the ability to isolate improperly-
operating devices and prevent them from failing the entire network.

H.3.4 Modems

Modems (a concatenation of the term MODulator/DEModulator) are devices that translate data between
digital and analog domains. Their purpose is to enable digital devices to communicate over a channel better
suited to analog signals. They are available for the public switched telephone network (PSTN) as well as

16
The original Ethernet design, as well as early IEEE 802.3 networks, used a single coaxial cable for interconnecting devices. Starting
with 10BASE-T, IEEE 802.3 networks are constructed with twisted pair wiring or fiber-optic cables in a physical star/logical bus
network structure. This type of topology can tolerate wiring failures better than coaxial networks and has led to significant data rate
increases as the technology has advanced.

133
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

other media such as analog leased lines. In the simplest form, this device provides a translation between
each digital data bit and a short analog tone. Other modem devices are available that convert and transmit
digital data over leased telephone lines, power-line carriers, and wireless systems. Each of these modem
types has different characteristics, but they share some common features. Modems can be classified as
being either half-duplex or full duplex. Half-duplex systems are incapable of simultaneously receiving and
transmitting information on the analog portion, while full-duplex systems possess this capability. Half-
duplex systems need to control media access, because transmission during reception cannot be permitted.
Media control mechanisms must be compatible between all devices connected to the analog media. Some
media control mechanisms involve additional digital signaling (flow control), which needs to be compatible
with the digital equipment.

H.3.5 Firewall, authentication, and security devices

While routers are designed to ensure transparent access between LANs, the reality is that restrictions to this
transparency should be made. Three types of devices are used to accomplish this restriction: firewalls,
authentication, and security devices.

H.3.5.1 Firewalls

Firewalls are designed to control inbound and outbound network traffic. Placed between the internal
(trusted) LAN and the external (insecure) WAN, firewalls analyze data packets and allow or deny these
packets, based upon a predefined set of rules. Essentially, there are three types of firewall mechanisms
discussed in this subclause: packet filters, stateful firewalls, and application-level firewalls.

H.3.5.1.1 Packet filters

Packet filters inspect the contents of the data packet traveling between the WAN and the LAN. Often the
packet filter is looking at the source and destination address of its protocol (like TCP or UDP) as well as
the port number. Packets that do not meet specified criteria are discarded. These types of filters are unaware
of the connection “state” (i.e., whether or not the packet is part of an existing stream of traffic). Effectively,
this kind of firewall treats each packet in isolation and has no memory of the previous packet. A packet
filter, therefore, has no way of knowing if a packet is part of an existing connection, is attempting to create
a new connection, or is just a random packet. As a result, packet filters are open to “spoofing” attacks,
where an attacker masquerades as a legitimate entity.

H.3.5.1.2 Stateful firewalls

Stateful firewalls are somewhat more sophisticated than packet filters. These systems track the state of the
network connections (such as the UDP communication or TCP packet streams). The firewall holds specific
attributes of each connection in memory. When a session is established, intensive checking is performed,
and UDP streams (or TCP connections) that satisfy predetermined security policies are entered in the
firewall’s dynamic state tables. These attributes may include the sequence numbers of the packets in the
stream, IP addresses, and ports in use. Sessions that do not meet these security policies are denied. Stateful
firewalls are more efficient than packet filters, because the firewall only checks the state table, rather than
matching packet contents against a (possibly large) set of firewall rules.

H.3.5.1.3 Application-level firewalls

Application-level firewalls understand the difference between the different types of traffic going through it.
Unlike stateful firewalls, application-level firewalls can control applications or services. For example, an

134
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

application-level firewall would recognize the difference between HTTP traffic used for a web page and
HTTP traffic used for FTP. Packet filtering and stateful firewalls would handle all HTTP traffic the same.
While more secure than packet firewalls, application-level firewalls are slower than stateful firewalls. In
addition, application-level firewalls tend to be more complex to set up and manage.

Often the firewall and router functions are placed within the same physical device. However, stand-alone
firewall devices are often preferred for larger and more complex networks.

H.3.5.2 Authentication systems

Authentication systems are used to identify legitimate users to the firewall and/or the internal network.
Typically these systems rely upon two factors:
 Something you have (a token of some form)
 Something you know (a password or passphrase)

In most implementations, the token generates a highly randomized 8-digit number every 60 seconds. This
token is synchronized with the authentication server. After the user logs into a system, the authentication
server takes over and issues a “challenge”, requesting the random number from the token. If the user enters
the number correctly, the user is allowed into the system. If the user fails to enter the correct number, the
user is given two more chances. If the user exhausts all three tries, the user is locked out of the system.

The token can be a hardware device (in the form of a keyfob) or a software application that can be used on
laptop/desktop or a smartphone. Should the token get out of sync with the server, there is a process to re-
sync these systems.

H.3.5.3 Security devices

Security devices are designed to block various categories of unauthorized data transfers, for example, those
that do not originate from authorized devices, those that are not destined to authorized devices, transfers of
excluded services (such as email), or executable code. For security devices to successfully perform their
function, they must be carefully programmed as appropriate for their intended function and their
environment.

H.3.6 Terminal servers

Terminal servers (also known as “port servers”) allow low-speed serial devices to communicate through a
LAN. This is accomplished by placing the serial data (possibly with additional flow control information) on
the LAN, typically using TCP/IP. Terminal servers almost always operate in master/slave mode where one
device (the master) communicates with pre-defined slave devices. It is typically not possible for the serial
devices to dynamically select the destination of the serial data. There is no general agreement between
terminal server vendors on the higher-level protocols used to transport the data and, therefore, the vendor
supplies both the master and slave hardware or software ends. Some terminal servers include PC software
allowing the serial devices attached to the terminal server to appear as emulated local communication ports
on the PC.

In contrast to those using TCP/IP, terminal servers using UDP/IP only need to be configured with the
destination IP and UDP-port addresses (several destinations may be supported if desired); furthermore, the
serial data is simply placed in the UDP datagram’s payload bytes allowing different vendors’ products,
including PC virtual-COM port emulators, to be interoperable.

135
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

Annex I

(informative)

Bibliography

Bibliographical references are resources that provide additional or helpful material but do not need to be
understood or used to implement this recommended practice. Reference to these resources is made for
informational use only.

[B1] Barrett, D.J., and R.E. Silverman, SSH, the Secure Shell: The Definitive Guide. Sebastopol, CA:
O’Reilly Media, Inc., 2001.
[B2] Buchanan, Jr., R. W., The Art of Testing Network Systems, 1996, John Wiley & Sons, Inc.
[B3] Galea, M., “Rapid Spanning Tree in Industrial Networks.” 17
[B4] Galea, M., and M. Pozzuoli, “Redundancy in Substation LANs with the Rapid Spanning Tree
Protocol (IEEE Std 802.1w).” 18
[B5] Holmeide, Ø., and T. Skeie, “VoIP Drives Real Time Ethernet.” 19
[B6] IEC 60255-24:2013, Measuring relays and protection equipmentPart 24: Common format for
transient data exchange (COMTRADE) for power systems. 20
[B7] IEC 60870-5-101, Telecontrol equipment and systemsPart 5-101: Transmission
protocolsCompanion standard for basic telecontrol tasks.
[B8] IEC 60870-5-102, Telecontrol equipment and systemsPart 5-102: Transmission
protocolsCompanion standard for the transmission of integrated totals in electric power systems.
[B9] IEC 60870-5-103, Telecontrol equipment and systemsPart 5-103: Transmission
protocolsCompanion standard for the informative interface of protection equipment.
[B10] IEC 60870-5-104 Ed.2 (2006), Telecontrol equipment and systems—Part 5-104: Transmission
protocols—Network access for IEC 60870-5-101 using standard transport profiles.
[B11] IEC 61000-4-3 Ed.3.0 (2006), Electromagnetic compatibility (EMC)—Part 4-3: Testing and
measurement techniques—Radiated, radio-frequency, electromagnetic field immunity test.
[B12] IEC 61850-SER (2005), Communication networks and systems in substations—All Parts.
[B13] IEEE Std 487™, IEEE Standard for the Electrical Protection of Communications Facilities Serving
Electric Supply LocationsGeneral Considerations. 21, 22
[B14] IEEE Std 802.11™, IEEE Standard for Information technology—Telecommunications and
information exchange between systems—Local and metropolitan area networks—Specific requirements—
Part 11: Wireless LAN Medium Access Control (MAC) and Physical Layer (PHY) Specifications.
[B15] IEEE Std 1159.3™, IEEE Recommended Practice for the Transfer of Power Quality Data.

17
This document is available at http://www.ruggedcom.com.
18
This document is available at http://www.ruggedcom.com.
19
This document is available at http://www.ontimenet.com.
20
IEC publications are available from the International Electrotechnical Commission (http://www.iec.ch) and the American National
Standards Institute (http://www.ansi.org/).
21
The IEEE standards or products referred to in Annex I are trademarks owned by the Institute of Electrical and Electronics Engineers,
Incorporated.
22
IEEE publications are available from the Institute of Electrical and Electronics Engineers (http://standards.ieee.org/).

136
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

[B16] IEEE Std 1379™, IEEE Recommended Practice for Data Communications Between Remote
Terminal Units and Intelligent Electronic Devices in a Substation.
[B17] IEEE Std 1588™-2008, IEEE Standard for a Precision Clock Synchronization Protocol for
Networked Measurement and Control Systems.
[B18] IEEE Std 1686™, IEEE Standard for Intelligent Electronic Devices Cyber Security Capabilities.
[B19] IEEE Std 1711™, IEEE Trial-Use Standard for a Cryptographic Protocol for Cyber Security of
Substation Serial Links.
[B20] IEEE Std 1815™, IEEE Standard for Electric Power Systems CommunicationsDistributed
Network Protocol (DNP3).
[B21] IEEE Std 2030.100™, IEEE Recommended Practice for Implementing an IEC 61850-Based
Substation Communications, Protection, Monitoring, and Control System.
[B22] IEEE Std C37.2™, IEEE Standard for Electrical Power System Device Function Numbers,
Acronyms, and Contact Designations.
[B23] IEEE Std C37.90.1™, IEEE Standard for Surge Withstand Capability (SWC) Tests for Relays and
Relay Systems Associated with Electric Power Apparatus.
[B24] IEEE Std C37.90.2™, IEEE Standard for Withstand Capability of Relay Systems to Radiated
Electromagnetic Interference from Transceivers.
[B25] IEEE Std C37.90.3™, IEEE Standard Electrostatic Discharge Tests for Protective Relays.
[B26] IEEE Std C37.111™-2013, IEEE Standard Command Format for Transient Data Exchange
(COMTRADE) for Power Systems.
[B27] IEEE Std C37.115™, IEEE Standard Test Method for Use in the Evaluation of Message
Communications Between Intelligent Electronic Devices in an Integrated Substation Protection, Control,
and Data Acquisition System.
[B28] IEEE Std C37.118™, IEEE Standard for Synchrophasors for Power Systems.
[B29] IEEE Std C37.237™, IEEE Standard for Requirements for Time Tags Created by Intelligent
Electronic DevicesCOMTAG.
[B30] IEEE Std C37.240™, IEEE Standard Cybersecurity Requirements for Substation Automation,
Protection, and Control Systems.
[B31] IETF RFC 768, User Datagram Protocol, 1980. 23
[B32] IETF RFC 791, Internet Protocol, 1981a.
[B33] IETF RFC 792, Internet Control Message Protocol, 1981.
[B34] IETF RFC 793, Transmission Control Protocol, 1981c.
[B35] IETF RFC 826, Ethernet Address Resolution Protocol, 1982.
[B36] IETF RFC 894, A Standard for the Transmission of IP Datagrams over Ethernet Networks, 1984.
[B37] IETF RFC 951, Bootstrap Protocol, 1985.
[B38] IETF RFC 1006, ISO Transport Service on top of the TCP Version, 1987.
[B39] IETF RFC 1034, Domain Names—Concepts and Facilities, 1987.
[B40] IETF RFC 1035, Domain Names—Implementation and Specification, 1987.
[B41] IETF RFC 1661, The Point-to-Point Protocol (PPP), 1994.
[B42] IETF RFC 1662, PPP in HDLC-like Framing, 1994.

23
RFC documents may be obtained from the Internet Engineering Task Force at http://www.ietf.org.

137
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
IEEE Std 1615-2019
IEEE Recommended Practice for Network Communication in Electric Power Substations

[B43] IETF RFC 1918, Address Allocation for Private Internets, 1996.
[B44] IETF RFC 2030, Simple Network Time Protocol (SNTP) Version 4 for IPv4, IPv6 and OSI, 1996.
[B45] IETF RFC 2131, Dynamic Host Configuration Protocol, 1997.
[B46] ISO 9506, Industrial Automation SystemsManufacturing Message SpecificationPart 2: Protocol
Specification. 24
[B47] ISO/IEC 8326, Information technologyOpen Systems InterconnectionSession service
definition.
[B48] ISO/IEC 8327-1, Information technologyOpen Systems InterconnectionConnection-oriented
Session protocol: Protocol specification.
[B49] ISO/IEC 8649, Information technologyOpen Systems InterconnectionService definition for the
Association Control Service Element.
[B50] ISO/IEC 8650, Information technologyOpen Systems InterconnectionConnection-oriented
protocol for the Association Control Service Element.
[B51] ISO/IEC 8822, Information technologyOpen Systems InterconnectionPresentation service
definition.
[B52] ISO/IEC 8823-1, Information technologyOpen Systems InterconnectionConnection-oriented
presentation protocol.
[B53] ISO/IEC 8824-1, Information technologyAbstract Syntax Notation One (ASN.1): Specification of
basic notation.
[B54] ISO/IEC 8825-1, Information technologyASN.1 encoding rules: Specification of Basic Encoding
Rules (BER), Canonical Encoding Rules (CER) and Distinguished Encoding Rules.
[B55] Mansfield, N., Practical TCP/IP. Designing, using and troubleshooting TCP/IP networks on Linux
and Windows. London: Addison-Wesley, 2003.
[B56] Pozzuoli, M., “Ethernet in Substation Automation Applications—Issues and Requirements,”
DistribuTECH 2004. 25
[B57] Skeie, T., C. Brunner, and S. Johannessen, “Ethernet in Substation Automation,” IEEE Control
Systems Magazine (June 2002): 43–51.
[B58] Stevens, W.R., TCP/IP Illustrated, Volume 1: The Protocols. Reading, MA: Addison-Wesley, 1994.
[B59] Tengdin, J.T., M.S. Simon, and C.R. Sufana, “LAN Congestion Scenario and Performance
Evaluation,” IEEE Paper WPM 99CB36233, page 919.
[B60] TIA 568, Commercial Building Telecommunications Cabling Standard. 26

24
ISO publications are available from the International Organization for Standardization (http://www.iso.org/) and the American
National Standards Institute (http://www.ansi.org/).
25
This document is available at http://www.ruggedcom.com.
26
TIA publications are available from the Telecommunications Industry Association (http://www.tiaonline.org/).

138
Copyright © 2019 IEEE. All rights reserved.

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.
RAISING THE
WORLD’S
STANDARDS
Connect with us on:
Twitter: twitter.com/ieeesa
Facebook: facebook.com/ieeesa
LinkedIn: linkedin.com/groups/1791118
Beyond Standards blog: beyondstandards.ieee.org
YouTube: youtube.com/ieeesa

standards.ieee.org
Phone: +1 732 981 0060

Authorized licensed use limited to: University of Glasgow. Downloaded on November 30,2019 at 10:43:17 UTC from IEEE Xplore. Restrictions apply.

Das könnte Ihnen auch gefallen