On the Horns of a Dilemma

Successful Patch Management Strategies in SCADA and Industrial Control Systems Eric Byres, P.Eng. CTO VP of Engineering CTO, Tofino Security
A Belden Brand

Eric Byres, Byres VP Engineering, Engineering Tofino Security

Founder of the BCIT Critical Infrastructure Security Centre Centre, a leading academic facility for SCADA cyber-security research Canadian representative for IEC TC65/WG10 standards effort for the protection of industrial facilities from cyber attack Chair of ISA99 Security Technologies WG Chair of ISA99 Stuxnet Gap Analysis TG 2006 SANS Institute Security Leadership Award Six ISA and IEEE awards for security research Testified to the US Congress on SCADA Security

Caught on the Horns of a Dilemma

Why Patching is Critical Why Patching is Difficult

Zotob Worm Security Incident

August 18, 2005: 13 US auto plants were

shut h td down b by a simple i l worm.

Despite professionally installed firewalls between the Internet Internet, the company network, network and the control network, the Zotob worm had made its way into the control system (probability via a laptop). laptop) Once in the control system, it was able to travel from p plant to p plant in seconds.

50,000 assembly line workers ceased work. Estimated $14 million loss.

Hiding Behind the Corporate Firewall Doesn't Work

The Slammer Worm infiltrated: A nuclear plant via a contractors T1 line; A power utility SCADA system via a VPN; A petroleum control system via laptop; A paper machine HMI via dial-up modem. Corporate firewalls existed in at least three of these cases.

SCADA/ICS Incidents Since 2002

Malware accounts for 2/3 of the external

cyber b i incidents id t on control t l systems. t Appears to match IT trends.
DoS 6% System Penetration 13% Sabotage 13% Malware 68%

A Perimeter Defense is Not Enough

We cant just install a firewall in front of the whole SCADA system and forget about security security. The bugs eventually get in. So we must harden the p plant floor. We need Defense in Depth.
Crunchy on the Outside - Soft in the Middle

Defense in Depth Strategy Defense-in-Depth

By defense-in-depth strategy, we mean the

protection t ti measures composed d of f more than th one security control to protect the property. B By use of these kinds of multi multi-layer layer measures, another layer will protect the property even if one layer is destroyed, so the property is protected more firmly.
Yokogawa g Security y Standard of System TI 33Y01B30-01E

But Patching SCADA and ICS is Difficult

Patches can affect the reliability of SCADA/ICS software. software Patches may require staff with special skills to be present. Patches may not be approved by SCADA/ICS vendor. P t h may require Patches i a system t reboot. b t Patches may not be available.

What No Patch for That?

Many SCADA/ICS vendors have a poor history of releasing security patches. patches According to Sean McBride of Critical Intelligence, less than half the disclosed SCADA/ICS vulnerabilities ever have released patches.

What No Patch for That?

Many systems still used in process control or SCADA are no longer supported with security patches:
Beginning January 1, 2005, Pay-per-incident and Premier support t will ill no l longer b be available il bl f for Wi Windows d NT S Server 4.0. This includes security hotfixes. On July 13, 2010, Extended Support for Windows 2000 Server will end. Support of FactoryLink will cease December 31, 2012

11

The Typical ICS/SCADA Patch Situation

No central Patch Management process in

place l f for process or SCADA systems t Patching is up to each system owner to decide if and when to do do IS/IT department is trying to force their overall patching process onto automation systems systems
Joakim Moby, ISA Expo 2006

Solutions for Patching in Process and SCADA Systems

Patching by Criticality Patching in Waves Using Compensating Controls

Patch Management is Not an Option

A procedure for patch management shall be established documented established, documented, and followed followed. Requirement 4.3.4.3.7 ANSI/ISA-99.02.01

Patch Management is Change Management

Patch management is about managing the risk of change Richard Brown, Dow Chemical

Patch Management in Control Systems

Inventory All SCADA Devices Classify SCADA Devices By Risk

Feed Results Back on D i Devices

Create Patch/ Vulnerability Tracking g System

Execute Response Stages

Create Response Plan

Inventory SCADA and Control Devices

All process control and SCADA devices are inventoried in terms of: Device type Criticality to process Core software/firmware components Patching requirements Should include both computer and non non-computer computer devices like PLCS, RTUs and SIS.

Categorizing SCADA and Control Devices

All devices are categorized into groups that define when and how they are to be patched patched. Example: Trial Adopters receive patches as soon as available and

act as Test/Quality Assurance machines. N T No Touch h machines hi require i manual li intervention t ti and/or d/ detailed vendor consultation.

Create Patch/Vulnerability Tracking System

Develop procedure for keeping track of new patches and vulnerabilities: Level of importance to control operations. Devices affected Severity level For each category, the severity level provides a time frame during which a specific patch must be installed to minimize risk.

Create Response Plan

Create a system where patch implementation levels are preset and tied to Response Plans for a given device class.
Aggressiveness Mi i Minimum Moderate Maximum Implementation Window Q Quarterly t l By end of following week Within 48 hours Level of Testing Hi h High Best Effort Minimal

Response Plan Al h Alpha Bravo Zebra

Patch Deployment

Each patch deployment is divided into several waves. waves One wave installs the patch/patches onto a number of machines. The purpose of waves is to find issues before the patch is installed on the systems with highest complexity or regulatory impact impact.

Typical Active Active Patching Patching Cycle

1 cyc cle = 34 4 workin ng days s
1. Central Test Platforms 4 days
Rapid deployment Formal responses gathered
Critical infrastructure Sample clients & servers (0.5%)

2. Critical & Control Group

4 days

Devices of low regulatory impact Allows time for change control and scheduling Relies on success of previous waves, not on testing Responses by exception

3. Early Adopters

6 days

4. Mainstream

10 days

Mainly devices of high regulatory or safety impact Otherwise as for waves 3 & 4

5. Late Adopters
Validated servers High risk devices

10 days

Source: Joakim Moby, Astra-Zeneca, ISA Expo 2006

22

Feedback

When cycle has ended, a report is produced that can be used as a guide for the systems still not following the process (such as new systems). Feed patching results back into inventory database and patch/vulnerability database, including issues.

Exceptions to the Patch Management System

Patch Management System catches between 50% and 70% of the critical patches patches. Some critical devices/systems will not fit into this Microsoft/PC patch model: The patch may not be available or approved by the SCADA

vendor in a timely manner. Servers with old operating p g systems y ( (like NT4 or Windows 2000) cannot be patched. PLCs and RTUs may not be patchable.

Compensating Controls

If patching isnt possible, a compensating control is needed: Turn system off. Improve system isolation. Reduce system exposure to other systems.

Protection for Unpatchable Systems

Example: Industrial firewall for NT Servers restricts possible infection vectors vectors.
Internet

Office LAN

NT4 Based Server Plant Network

Control LAN

Firewall restricts communicating devices and filters bad traffic

FCS

Protection for Embedded Systems

Example: MODBUS/TCP industrial firewall sanitizes and limits MODBUS commands commands.
Internet

Office LAN

Plant Network

MODBUS/TCP FW only allows MODBUS Read commands

Control LAN

Safety System

Conclusions

Patching is a change management process

(you must ( t a have h a process). ) Patching does NOT end with the Microsoft patches. patches Compensating controls are important to catch vulnerabilities that cant can t be patched using standard methods. Patching is not a silver bullet bullet. Patching must be done together with other activities to secure SCADA systems systems.

References
Tofino Security White Papers and Application Notes Using g IEC/ISA-62443 Standards for SCADA Security y
https://www.tofinosecurity.com/blog/using-ansiisa-99-standards-scadasecurity-plus-white-paper

Shamoon Malware and SCADA Security What are the Impacts?

https://www.tofinosecurity.com/blog/shamoon-malware-and-scada-security%E2%80%93-what-are-impacts