Beruflich Dokumente
Kultur Dokumente
Executive Summary
Over the past decade, two trends have converged that have caused many utilities to re-evaluate
their substation communications infrastructure. One trend is the migration of the electrical
Substation Communications grid from a reliable, but inflexible system to the “Smart Grid,” which promises adaptability and
Design Legacy to IEC 61850 efficiency. It also requires the two-way communication of data, something that is not possible
Best Practices with traditional electrical grids.
The other trend is the increasing adoption by industry of Ethernet networking technologies
Eric Byres, P. Eng., ISA Fellow
CTO and VP Engineering
for their communications. The ARC Advisory Group estimates that the adoption of industrial
Tofino Security, a Belden Brand Ethernet networks is growing at a >12 percent plus CAGR (Compound Annual Growth Rate).
eric.byres@belden.com
As a result of these trends, many utilities are faced with having to design and implement
Tim Wallaert, BSEE, MBA communications infrastructures that are unlike anything they have been involved with before.
Director – Vertical Markets, Energy
Belden If you are an engineer, field technician, manager or IT professional involved with such a project,
tim.wallaert@belden.com you may be wondering where to begin. Alternatively, if you are working with an automation
vendor or system integrator, you may want to become better informed to contribute to the
project’s success.
To help you, we have consolidated our learning and experience from helping hundreds of
customers around the world design robust substation communications networks. The result is an
easy-to-follow process that will assist you when designing new or upgraded communications
systems. Here are the ten best practices discussed in this white paper:
Table of Contents
1 – Segment Operational Networks
Executive Summary...............................1
2 – Add Serial Communications Devices to Ethernet Infrastructure
The Foundation of the Smart Grid: 3 – The Importance of Power over Ethernet
Modern Two-Way Communications
Networks.................................................2
4 - Time Synchronization
5 – Selecting Appropriate Switches and Routers for the Environment
Reliable Substation
6 – Building Multiple Layers of Security
Communications is Vital - 10 Best
Practices.................................................2-7 7 – Adding Communications Infrastructure between Master, Backup and Substations
8 – Choosing the Correct Cables, Jackets and Connectors
Conclusion..............................................7
9 - Vendor Selection
References..............................................8 10 – Good Project Management
This white paper will walk you through each of the best practices, explaining the importance of
each one and providing guidance on how to apply it to your needs.
The Foundation of the Smart Reliable Substation largest segment of the substation automaton
Grid: Modern Two-Way Communications is Vital revenue, followed by retrofits of existing
Communications Networks Today, the majority of utilities are still
transmission substations (7.1 percent CAGR)
and automation of distribution substations
Electric utilities are constantly searching for using technologies ranging from modem
(7.0 percent CAGR).
the most efficient, reliable and cost-effective connectivity to serial bus technology to ‘talk’
ways to deliver electricity to customers. to their substations to gather important, The large investments in substations being
A vision for doing just that and more is needed information. In order to integrate undertaken today will only pay off if the
provided with the smart grid model. with the smart grid, substations need to be resulting communications infrastructure is
upgraded to modern Ethernet and IP-based high performing, reliable and secure.
One definition of the smart grid, based on systems.
work from the U.S. Department of Energy, is: Through years of experience in automating
The magnitude of this effort is huge. For substations around the world, we have
“A modernized electrical grid that example, in the U.S., there are roughly developed 10 best practices for reliable
uses information and communications 14,000 transmission substations and 48,000 substation communications. These practices
technology to gather and act on distribution substations. Upgrading these are part of our Substation Communications
information in an automated fashion stations to communicate within a smart grid Legacy to IEC 61850 Design Checkup,
… to improve the efficiency, reliability, will not happen overnight and will require a a process we use with utilities to ensure
economics, and sustainability of significant capital outlay. that network upgrades benefit from proven
the production and distribution of
techniques.
electricity.” According to research completed by Pike
Research in 2012, the global market for Here are the 10 best practices:
The promises of the smart grid are exciting on substation automation will continue to
many fronts. It pledges to: grow, increasing from $2.7 billion in 2012 to 1. Segment Operational Networks
• Integrate renewable energy sources into $4.3 billion in 2020. Substation automation The first best practice we recommend is
the grid, thus reducing dependency on equipment includes communications devices, to segment all networks into operational
traditional sources that may be harmful to protective relays, supervisory control and zones or areas. Networks tend to grow
the environment. data acquisition (SCADA) devices, and other incrementally, resulting in large, flat
related sensors. networks. Too often we find networks that
• Save electricity in its own operations, thus
improving the efficiency of the system. The Pike Research data shows that new have become vast, sprawling systems that are
transmission substations account for the difficult to manage or secure.
• Improve reliability by monitoring
equipment and fixing problems before they
cause an outage.
• Improve recovery time by being able to
smartly switch power around downed
areas.
• Meet peak demand without requiring
the build-out of additional traditional
generating facilities.
Figure 1. Subnets divide up devices into physical groups and make the network more manageable.
2
Be Certain with Belden
Subnets
This technique divides up devices into
physical groupings based on function or
location for ease of maintenance and
security. Each subnet has a specified range
of IP addresses and is connected to other
subnets using a Layer 3 switch or router.1 Figure 2. VLANs divide up devices into virtual groups and also help make the network more manageable.
3
Planning for PoE involves: 5. Selecting Appropriate Switches and monolithic designs present a single point of
Routers for the Environment failure in a complex system. Few systems are
• Determining all the pieces to be used so simple as to have single points of entry.
(cameras, telephones etc.). Similar to power plants, substations contain
valuable pieces of equipment typically housed For example, this is what the U.S. Department
• Identifying the power consumption (in in unconditioned control sheds inside the of Homeland Security has found:
watts) of each device. fence. While this provides some level of “In ….hundreds of vulnerability
protection against the elements, temperature assessments in the private sector,
• Totaling the power requirements of all swings can be extreme, rodents and other
PoE devices that will be wired to one PoE in no case have we ever found the
pests can invade the shed, and dirt and grime operations network, the SCADA
switch. can accumulate on the equipment. Other system or energy management
Note that most devices are “standard” PoE, stresses can include humidity, corrosion and system separated from the enterprise
requiring up to 13 watts, but some may be electromechanical noise. network. On average, we see 11 direct
classified as “PoE+”, ranging from 13 to 25.5 It is therefore important to select only connections between those networks.
watts. switches and routers that have the protection In some extreme cases, we have
against the environmental and other hazards identified up to 250 connections … .”
4. Time Synchronization (Fault Event
that exist in your substations.
Replay ) With the help of Murphy’s Law, eventually all
IEEE 1613 and IEC-61850 Part 3 describe single-point solutions are either bypassed or
As a utility operator, you need to know
the device standards that need to be met experience some sort of malfunction, leaving
when events, such as faults, occurred, what
for environmental protection. As a utility the system open to attack.
happened throughout the event, and what
operator you must ensure that network
pieces of equipment and substations were
products meet or exceed all relevant industry A more realistic strategy is based on “Defense
involved. This requires time synchronization,
standards. in Depth” (DiD ) – multiple layers of defense
also known as fault event replay.
Taking the time to select equipment with distributed throughout the control network –
Examples of the equipment that must appropriate environmental and electrical as a strategy for securing operations.
have accurate time stamping are IEDs, ratings at the beginning of a project DiD maintains an ESP firewall between the
merging units (MUs), control units, Ethernet eliminates trouble in the future. You’ll save business and control networks, but adds
switches and any other system that requires yourself costly repairs and downtime. security solutions inside the control system that
synchronization within the substation
protect the substations if the main firewall is
automation system. Factors such as the 6. Building Multiple Layers of Security
bypassed. The solutions work in parallel, with
protocols used, traffic load, communications Industrial infrastructure, especially critical one technology often overlapping with others,
media and cable distance of the network can infrastructure such as the electrical grid, is to form a significant safeguard against either
affect the timing accuracy. an increasing target for both sophisticated attack or human error.
To ensure precision, we recommend using gear cyberattacks and for activists. Protecting
substations is therefore vital. This best There are two primary options for
that supports the IEEE-1588 protocol for devices
practice looks at how to design security implementing network security technologies
on Ethernet requiring extremely high timing
measures that contribute to an overall for a substation:
accuracy, that is, to less than one microsecond.
IRIG-B is a similar, but older technology. security strategy.
• Industrial firewalls that control and
Most engineers are aware that NERC CIP monitor traffic; comparing the traffic
To implement time synchronization:
specifies the minimum requirements for passing through to a predefined security
• Determine the timing needs of your security in the power industry. Unfortunately, policy, and discarding messages that do not
application which can range from sub- NERC CIP uses an electronic security meet the policy’s requirements. Firewalls
microseconds to milliseconds. perimeter (ESP) philosophy based on hiding can be installed both at the ESP boundary
all critical assets behind a monolithic and between internal zones.
• Make sure that all switches, routers and boundary. For example, a single firewall
terminal servers in the path between • VPNs (Virtual Private Networks) are networks
could be installed on the boundary between
devices needing to be synchronized that are layered onto a more general network
all critical control assets and the business
support the timing technology being used. using specific protocols or methods to ensure
network, with the hope that it will prevent all
“private” transmission of data. VPN sessions
unauthorized access to the critical assets.
• Connect all devices to a synchronized tunnel across the transport network in an
global positioning system or master clock. Industry experience has shown that encrypted format, making them “invisible”
for all practical purposes.
4
Be Certain with Belden
A network protected using a DiD strategy protection technology, but things like
Unplanned Downtime Calculator
responds to threats, such as a traffic storm physical security and policies and procedures.
$ X product Y unit of time,
(caused by device failures) or a USB-based The techniques used should be based on per
value e.g. hour
virus, by limiting the impact to the zone doing a risk assessment for critical assets and
times
where the problem started. Alarm messages processes.
from the firewalls would pinpoint the zone V average Meant Time to Repair - MTTR
Security is covered in the Design Checkup, (in same time units as above)
and even the source of the problem.
but you may require even more assistance times
Routing Firewalls Guarding the Substation in this area. If so, choose a partner who has
W # of downtime events per year
Perimeter experience in cyber security for substations
and, in particular, securing industrial protocols. equals
To create a security perimeter for the
substation, a security control point needs $ Z
7. Adding Communications Infrastructure per year of downtime expense
to be established to restrict and monitor
traffic flowing into and out of the substation. between Master, Backup and Substations Table 1. Unplanned Downtime Calculator
Typically, this will be a dedicated firewall, but As the design of the substation begins to
in some cases a router or terminal server can come together, utilities must think about how While there have been numerous redundancy
be used. These need to be able to filter large all of the pieces will communicate with each schemes developed over the years, there
amounts of traffic and interface transparently other. How can we move the data from the are three that are particularly useful for
to IT systems using security protocols, such substation to other locations? After all, this master to substation redundancy. They
as RADIUS and TACACS+. It is critical that is what is needed to realize the promise of are Rapid Spanning Tree Protocol (RSTP),
this device is both security hardened and the smart grid. Cellular Redundancy and Parallel Redundancy
monitored for indication of attacks. Protocol (PRP) - see table 2.
Substations can communicate with the
Transparent Firewalls to Protect Core master control station and the backup RSTP uses a physical ring, but logically
Processes control station using a variety of networking disables one link to prevent messages from
technologies. These include Ethernet WAN, being forwarded on and causing message
Transparent firewalls2 are security devices
Cellular 3G or MPLS-PPP WAN. Whichever looping. If a break is detected on the
with special features for industrial use. At
technology is chosen, consider making it network, the disabled link is re-enabled and
first glance they appear on the network like a
redundant, such as adding yet another messages then flow through the network
traditional Ethernet switch, but they actually
cellular backup. Robust communication using the new path. The main advantage
inspect network messages in great detail.
keeps small issues small and ensures high of RSTP technique is that it can be used on
The “transparent” feature allows them to availability of systems. any network topology. Its main drawback
be dropped into existing systems without however is that recovery times may be as
To evaluate your utility’s need for redundancy long as 5-20ms per switch.
requiring readdressing of the station devices.
use our simple, quick equation known as the
This means that organizations can retrofit
Unplanned Downtime Calculator - see When it is not possible or practical to add
security zones into live environments
table 1, below. It can help make the case for a separate physical hardwired Ethernet line,
without a shutdown. They also allow the
investing in redundancy. cellular redundancy can be used to provide
installation of security controls within a
a means of backing up communication. The
single subnetwork; for example within a large
process bus.
The “firewall” feature provides detailed Protocol Works Best On Pros Cons
“stateful”3 inspection of all network protocols so Rapid Spanning Tree Any Ethernet WAN Can implement on any Potentially long recovery.
inappropriate traffic can be blocked. For example, Protocol (RSTP) topology or mesh 5-20ms per switch
rate limits can be set to prevent “traffic storms” (IEEE 802.1D-2004)
while deep packet inspection rules can be set
Cellular Redundancy When a second hard- Provides alternative to Potentially long recovery.
to prevent inappropriate commands from being
wired Ethernet line is not running a physical line Dependent upon
sent to IEDs or controllers. available for redundancy wireless internet
Defense in Depth is Critical Parallel Redundancy Any Ethernet WAN Zero packet loss. “0ms” Requires separate
Protocol recovery. Can be added redundancy boxes.
This overall best practice depends on using to any existing network.
a multi-layer defense model, which involves (IEC 62439-3:2012-07)
not just networking, computer and device Table 2. Three Redundancy Schemes Particularly Useful for Master to Substation Redundancy.
5
cellular link remains in a standby mode until
communications via the primary hardwired
Ethernet line is lost. Communication is then
transferred to the cellular link (figure 3). The
drawback of this approach is that recovery
times will be dependent upon establishing the
wireless internet connection.
6
Be Certain with Belden
Management Special, 72 percent of all 9. Vendor Selection We recommend using the Industrial
communication errors are introduced at the Here are some factors to consider when Networking Project Checklist provided in
physical layers, such as cables and connectors. selecting a vendor of substation networking Appendix 1. Following this will help ensure
equipment: that the project has the highest probability
b. Use Industrial Grade Cabling of success.
After all of your planning, you want to be • Using a vendor that offers everything from
sure that your substation infrastructure will cables, connectors to switches, routers, Conclusion
perform as designed in its harsh environment. and security devices eliminates the need
While the smart grid promises vast
Commercial grade cable is not designed for multiple project managers from several
improvements for the reliability, efficiency
nor intended to be used in industrial different organizations.
and economics of utilities, it will not meet
environments.
• Experienced application engineers at the the goal power producers envision without
c. The Right Jacketing for the Location vendor organization should review your a robust communications infrastructure
application. Most utility process control in place at transmission and distribution
The proper jacket material will provide the
and industrial applications don’t include substations.
needed protection against the variety of
environmental and physical challenges for enough data to come anywhere near
Belden’s Substation Communications Legacy
both copper and fiber cables. Consult with Ethernet’s capacity. However, you want to
to IEC 61850 Design Checkup helps utilities
the cable manufacturer about the jacket that be sure that you will have capacity to spare
compare their designs to industry best-
would be most appropriate for the specific and that areas of risk have been dealt with
practices and provides options for improving
installation needs. in the network design.
established communications systems.
d. Choose High Performance Cable Designs • Work with an organization that has the
Investments in good network design and
ability to provide training to employees.
Copper cables with Bonded-Twisted-Pair communications infrastructure will improve
Also, look for a company that uses tools
technology are designed for high-balance reliability and contribute to an economical
designed to operate the way controls
performance for optimal signal transmission energy delivery system.
engineers and maintenance workers both
integrity. Proper fiber cable selection of work and think.
multi-mode or single mode designs is critical
to attaining system performance. • A vendor with experience specifically
securing substations and industrial
e. Consider Electrical Noise protocols will provide the best payback on
EMI and RFI noise levels must be evaluated investments in security technologies.
to determine if shielded or unshielded
constructions need to be used. Highly balanced, • A vendor that offers network certification
bonded-pair cables in shielded or unshielded with extended warranties will greatly
configurations provide the most robust noise contribute to substation future-proofing.
immunity performance. Fiber cable provides the
10. Good Project Management - A Key to
ultimate level of noise immunity.
Success
Finally, make sure that the IEC 525 Substation Project management is the key to the
cabling installation guidelines are followed. This successful implementation of substation
will ensure that the cables are installed properly design and automation. Without managing
and will work properly for years to come. the project properly, important steps could
be overlooked causing much bigger issues in
the future.
7
References
• Energy.gov “Smart Grid”.
http://energy.gov/oe/technology-development/smart-grid
• Navigant Research “Driven by Smart Grid Integration, the Electric Utility Substation Automation Market Will Reach $4.3 Billion by 2020”, Aug 7,
2012.
http://www.navigantresearch.com/newsroom/driven-by-smart-grid-integration-the-electric-utility-substation-automation-market-will-reach-
4-3-billion-by-2020
• EIA: U.S. Energy Information Administration “Market Trends — Electricity Growth in electricity use slows but still increases by 28 percent from
2011 to 2040”.
http://www.eia.gov/forecasts/aeo/MT_electric.cfm#cap_addition
• International Energy Agency “WORLD ENERGY OUTLOOK 2013 FACTSHEET” Nov 12, 2013.
http://www.worldenergyoutlook.org/media/weowebsite/factsheets/WEO2013_Factsheets.pdf
Additional Resources
1. Obtain further Substation Communication resources, including network diagrams and other tools, at:
• www.belden.com/power-td
2. Find out about the Belden Certified Industrial Network Program, which provides expert network design services, outstanding warranties and
flexibility for the future, at:
• www.belden.com/certified-industrial-network
About Belden
Belden Inc., a global leader in high-quality, end-to-end signal transmission solu-
tions, delivers a comprehensive product portfolio designed to meet the mission-
critical network infrastructure needs of industrial, enterprise and broadcast
markets.
With innovative solutions targeted at reliable and secure transmission of rapidly
growing amounts of data, audio and video needed for today’s applications, Belden
is at the center of the global transformation to a connected world.
Founded in 1902, the company is headquartered in St. Louis and has manufacturing
capabilities in North and South America, Europe and Asia. For more information go
to www.belden.com or @BeldenInc.