27th Annual INCOSE International Symposium (IS 2017)

Adelaide, Australia, July 15-20, 2017

Test Strategy to detect Industrial Control Systems

common Cyber Weaknesses and Vulnerabilities

Obaid Ur Rehman Keith F. Joiner

Capability Systems Centre, UNSW Capability Systems Centre, UNSW
Canberra. Canberra.
s.obaid.rehman@gmail.com k.joiner@adfa.edu.au

Copyright 2017 by OU. Rehman and KF. Joiner. Published and used by INCOSE with permission.

Abstract. Industrial control systems (ICSs) play a vital role in the operation and monitoring
of a nations critical infrastructure. These systems consist of software and hardware and use
protocols which are not usually designed for secured transmissions, such that they become
vulnerable targets for cyber attackers. In this paper, we propose to research the efficacy of
statistically rigorous methods such as design of experiments (DOE) techniques in testing the
cyber vulnerabilities of ICSs. DOE methods in modern software packages use advanced
statistical and mathematical methods such as High Throughput Testing (HTT) combinatorial
methods to allow for multifactor, multi-response testing and analysis so as to create a
probabilistic and static model of the response of a system that helps quickly focus (i.e., screen)
on the more effective cyber threats and thus greatest vulnerabilities. These methods also help
to determine optimum defensive settings for cyber-resilience in the presence of the tested
threats. The proposed research will have applicability to defense of complex systems more
broadly than cybersecurity of ICSs, since it illustrates a structured technique for dealing with
the very high numbers of test permutations that arise when considering complex system
architectures exposed to a myriad, and forever changing, arsenal of possible malicious intent.

Introduction
The increasing threat of cyber warfare for the World and Australia was reviewed extensively
by Austin (2016) and includes significant cyber threat to civil infrastructure. Industrial control
systems (ICSs) such as supervisory control and data acquisition (SCADA) system, distributed
control system (DCS) and programmable logic controller (PLC) play a vital role in the
operation and monitoring of a nations critical infrastructure (US department of Energy, 2011).
The ICS networks are found in many civilian and military applications and are an excellent
example of the cybersecurity problems and efforts that are engaging systems engineers more
generally. The ICSs control distribution of water, electricity, oil, gas, and management of air
and road traffic. Attack on any of these activities may cause catastrophic and cascading effects
with significant potential to reduce the defensive capability of a nation. Threats against
corporate Information Technology (IT) systems may impact the financial viability of the
company, however the potential consequences of attacks on ICS/SCADA represent a threat to
safety and human life.
Industrial control systems use a combination of hardware and software components to
accomplish the control and monitoring of a system and they are usually connected to a network
(Stouffer, K. et. al., 2015). The ICSs which are controlled through SCADA are located on
several different geographical locations and connected to the SCADA server through a
network. Each location accomplishes its given task and reports to the SCADA server. At the
server, information is processed and analyzed and then used to take necessary control actions
once sent to the connected locations. Thus ICS can be considered in some instances as
representative of a system of systems where each location is responsible for its own operation
despite being part of a larger system. Although ICS uses software and IT based
communications, they are different in many ways from the IT system. One of the differences
is that the IT systems acquires security patches and updates quite regularly, whereas regular
patching and update are not possible with the ICSs due to their distribution and often limited
design capability to accept such updates.
The security of SCADA and real-time systems represents a significant challenge (Harp, D. and
Gregory-Brown, B., 2015). The communication protocols used in the system are not designed
for security. SCADA components such as remote terminal units (RTUs) & PLCs are designed
for their functionality and do not have any authentication process available in the system. Since,
SCADA is now often connected to corporate networks which in turn connect to the internet,
this has made many SCADA system an easier target for cyber-attack. Over 1 million SCADA
systems are connected to the internet and this number is increasing every day (Thales, 2013).
The problem of defending SCADA systems from cyber-threats is an insight to the challenge of
defending more complex legacy platform systems like aircraft, vehicles and ships that until
recently were relatively standalone from ICT systems (Bryant, 2016).
The SCADA system consists of several layers and attack on any layer can halt the operation of
entire system (Zhu B. et. al., 2011). For example an attack on a communication network layer
may result in gaining access to the communication protocol and sending erroneous signals to
the system. Any attack on hardware has a potential to change the configuration files on
RTUs/PLCs and therefore enable an attack on an application layer; for example, using
malicious software on the SCADA system can make the system invisible from operator view.
If a SCADA system is connected to the Internet these attacks can be performed remotely.
Attacks on SCADA systems generally originate from IT systems and reach to the SCADA
system through the communication network. In order to avoid these attacks, ICS should be
tested against possible threats and an impact analysis should be performed. Such test and
analysis must unfortunately be performed routinely so as to account for continuously evolving
cyber threats and respond with appropriate defensive software and hardware changes.
The aim of this paper is to propose structured research on the efficacy of statistically rigorous
methods such as design of experiments (DOE) techniques in testing the cyber vulnerabilities
of ICSs. Methods such as high throughput testing (HTT) combinatorial methods allow for
multifactor, multi-response testing and analysis so as to create a probabilistic and static model
of the response of a system that helps quickly screen for the more effective cyber threats and
thus greatest vulnerabilities. These methods also help to determine optimum defensive settings
for cyber-resilience in the presence of the tested threats. The proposed research will have
applicability to defense of complex systems more broadly than cybersecurity of ICSs, since it
illustrates a structured technique for dealing with the very high numbers of test permutations
that arise when considering complex system architectures exposed to a myriad, and forever
changing, arsenal of possible malicious intent.

Preparation
Security threat testing of ICS is not new but it is increasing dramatic in its effects due to a large
number of systems that are now connected to the Internet and the rapidly evolving cyber-threat.
New threats are estimated at 60-80 per day, so the best way is to design ICS to be as robust as
possible (Stirland, J. et. al., 2014) and to monitor and re-test as often as possible. Stirland et.
al. (2014) proposes seven phases to examine any cyber incident:

1. identification and preparation,

2. identifying data sources,
3. volatility assessment and contamination impact analysis,
4. examination,
5. analysis,
6. reporting and presentation, and
7. reviewing results.

The first phase is to identify and prepare and this paper focuses on the first phase. In order to
prepare for any threat the first step is to identify all the software and hardware systems present
in the ICS. For example in a SCADA network the following subsystems are usually present:
1. SCADA server/master terminal,
2. Human Machine Interface,
3. communication network/protocols,
4. configuration workstation,
5. PLCs/RTUs/IEDs (Intelligent Electronic Devices).
In order to identify the system component, a white-list of all the software, protocols, ports and
hardware which have been used in the system should be prepared (Folkerth. L., 2015). Figure
1 shows a list of IP addresses, protocols and ports that are used in an example SCADA system.
This information is obtained by capturing the network data and viewing the information using
a software package such as Wireshark1 as shown in Figure 1.
The network data provides useful information about the IP and ports during the SCADA
communication and helps to generate a white-list for all these entities. Because in SCADA
networks the subsystems interfaces use communication protocols which usually have weak
protection, this step will help test engineers to identify strategies to improve the interface
protocols. In order to identify all the processes running during the control process a memory
dump of the workstation can be analyzed using one of the memory analyzer package such as
Volatility2. Figure 2 shows a list of processes along with their process identifications (PIds)
running in a workstation.
The white-list provides a way to identify application and system components that are
authorized to be present or active during any operation. This process defines a baseline for any
threat analysis and helps to quickly initiate a mitigation strategy. In addition to the white-list,
the system should be tested against commonly known threats, so that known weaknesses in the
system can be dealt with in advance and have the countering threat methodology resident or
ready to be applied (Christensen, 2015). The MITRE3 organization provides a dictionary for

1
https://www.wireshark.org/
2
http://www.volatilityfoundation.org/
3
https://www.mitre.org/
software weaknesses called common weakness enumeration (CWE)4 which comes with the
knowledge of software weaknesses and deals with the impact on the system. Such common
taxonomy is now in widespread use by the U.S. Department of Defense (DoD) in their
cybersecurity T&E (Christensen, 2015).

Figure 1. A snapshot of SCADA network traffic using Wireshark internet protocol analyzer.

Common Weakness Enumeration

The CWE construct provides a unified and measurable set of software weaknesses and thus
vulnerabilities that can be used to better understand any threat to a system. This database
contains information on any software-related vulnerability, its impact, the likely attack vector
and any suitable mitigation strategies. The CWE database is an evolving dictionary of common
vulnerabilities and their likely impacts to systems. The impact is presented in a form of certain
outputs related to the particular vulnerabilities. Although the CWE construct deals with
software weaknesses it is also applicable to the ICS because of the extensive use of software
and communication networks therein. The CWE construct can effectively be used to design
initial screening test strategies for ICS. For each individual CWE, the information in Table 1
is provided.

4
http://cwe.mitre.org/
Also, each CWE provides the target area of the particular weakness as well as provides the
information about the following output elements related to the potential cyber-attack or cyber
vulnerabilities:
1. Weakness prevalence
2. Remediation cost
3. Attack frequency
4. Consequences
5. Ease of detection
6. Attacker awareness etc.

For example Figure 3 is a snapshot of CWE number 89 (CWE-89). Figure 3 shows the level
or value for each output elements for SQL injection weakness, where more details related to
this particular vulnerably can be obtained from the CWE website. Any attack on software
effects various aspects of the system and therefore the significance of attack is different for
different organizations. Consequently, it is necessary as part of the test preparation to devise a
mechanism to quantify the threat level based on the impact on the system under test.

Figure 2. A snapshot of the process tree viewed using Volatility.

 Table 1. CWE information database (taken from http://cwe.mitre.org/ )

Figure 3. Information database for CWE-89 (taken from http://cwe.mitre.org/ ).

Common Weakness Scoring System (CWSS)

Common Weakness Scoring System (CWSS)5 provides a mechanism to obtain a single score
value for the impact. CWSS can be subdivided into three metric groups as shown in Figure 4
below:

5
http://cwe.mitre.org/cwss/cwss_v1.0.1.html
 Figure 4. CWE metric group based on the impact on the system (this diagram is taken from
http://cwe.mitre.org/ ).

The CWSS score is calculated considering the impact of each factor that effects the metric
group. The score can be different for a different entity based on contextual preferences or
requirements. Table 2 shows the scoring level of one of the factors for the technical impact in
the base finding sub group.
Table 2 (taken from http://cwe.mitre.org/cwss/cwss_v1.0.1.html ).

All other factors have a similar scoring system and thus can be quantified (see
http://cwe.mitre.org/cwss/cwss_v1.0.1.html). Once every factor is given a score then weight is
assigned to each factor according to the contextual preferences and requirements. This
produces three sub-scores for three group metrics and finally a single score is calculated. This
process has been shown in Figure 5 below.
 Figure 5. CWSS scoring factors (taken from http://cwe.mitre.org/cwss/cwss_v1.0.1.html ).
The CWSS is a method to provide an overall score for each weakness and therefore helps to
prioritize software weaknesses in an open and flexible manner. The CWSS scoring system can
also be used to investigate the impact of each CWE on any particular application or SCADA
system where different applications, software and hardware are employed in the system. Note
that common vulnerability exposure (CVE) from MITRE and common vulnerability (CVSS)
from the National Vulnerability Database (NVD)6 can also be used to score each vulnerability
in any particular systems.

Testing Strategy
There are a large number of CWE available and many of their factors overlap and produce the
same impact on a system. However, in a particular system only a few weaknesses may be
significant for a particular cyber threat profile. If all CWE are taken into account then it may
take a long time and require significant resources for testing. To resolve this issue, a screening
process can be developed to screen the CWE which may impact a particular aspect of the
system. Considering the information available under each CWE its a challenging task to use
available information to develop a screening process. This research deals with the development
of rigorous test methodologies which help to quickly test the weaknesses in a system for a
given threat profile. The test strategy obtained through these methods will help to prioritize
mitigation strategies and thus reduce cyber incident response times. Here we propose a
statistically rigorous method using aspects of Design of Experiments (DOE) (Schmidt &
Launsby, 2008) and Design for Six Sigma (DFSS) (Reagan & Kiemele, 2008) to develop a
screening process to screen CWE according to their impact on a particular system and use them

6
https://nvd.nist.gov/cvss.cfm
to develop deeper test strategies around each of these weaknesses. Figure 6 shows a tree
diagram for CWE that shows the relationship to the process it targets.

CWE Screening
In order to setup a testing process we use CWE category information according to its impact
to the system components. As an example we take 25 CWE which are sorted under three high-
level categories (2009 top 25) as follows in Tables 3, 4 and 5.
1. Insecure interaction between components.
Table 3 (taken from http://cwe.mitre.org/)

2. Risky Resource Management.

Table 4 (taken from http://cwe.mitre.org/)

3. Porous Defences.
Table 5 (taken from http://cwe.mitre.org/)
 Figure 6. CWE categories.

For DOE analysis, information provided in CWE should be extracted in a certain way. We need
to extract input factors which can be applied to the system under test as well as define output
responses of interest. For example, CWE provides consequences of particular weaknesses as
the impact on:
1. code execution
2. data loss
3. denial of service (DoS)
4. security by-pass etc.
The denial-of-service weakness can be a difficult test metric that has been researched and
improved by Mirkovic et. al. (2009). CWE also provides some of the factors which affect the
system under threat. After a careful investigation we propose to use the following input factors
at a specified level, where the number of applications and hardware configurations in particular
are unique to each context and only two are shown here for illustration. The factors and their
chosen levels are shown in Table 6.
 Table 6. Input factors and their specified level.

Figure 7 shows the test setup for a representative system required to be tested, in particular a
representative set of cyber-security output response metrics for an ICS. The testing needs to
analyze the impact of each of the factors on the system as well as determine if there are any
interactions among factors. Such testing is inherently multifactor and multi-response. The test
can be used to focus or screen for more in-depth testing on areas of weakness, and in turn to
optimize the system performance by carefully selecting system elements and defensive strategies
to lower the CWSS score. Other tests can be designed at different abstraction levels depending
on the requirement.

Conventional test approach

In order to test the system as proposed in Figure 7, all CWEs presented above need to be tested.
This is called full-factorial test design, where every factor is paired with other factors at each
level and all combinations are tested. In order to run a full-factorial test with the three input
factors considered above (obscuration, frequency & prevalence) for each hardware and software
configuration; a conventional test approach would require 540 test runs for a three-level (33
2 2 5) and 160 test runs for a two-level design (23 2 2 5).

Figure 7 Test Model.

High Throughput Testing (HTT)
HTT is a technique used to determine all pair test coverage for combinations of many factors.
It is related to DOE and provides a screening solution to the problem of excessive test cases.
HTT uses combinatorial mathematics, which is based on the complex optimization algorithms
and heuristics, to reduce the total number of test cases to a minimum while ensuring a
predetermined coverage level (Reagan & Kiemele, 2008, pp. 127-129). The increased use of
combinatorial test planning in computer security test planning has been noted by Yenign,
Yilmaz and Ulrich (2016) from meta-research. The orthogonal all pairs test matrix can be
determined using software packages such as rdExpert, Pro-test or Praxis. Using the HTT
Orthogonal Array (OA) option of rdExpert LiteTM 7, the screening can be done (for first-order
effects) using just 18 runs as shown in Figure 8.
These 18 tests cover all-pairs test combinations and provide an efficient way to screen test the
SCADA system for the most significant areas of weakness. Also, in a real scenario there are
usually a greater number of hardware and software configurations that need to be tested than
the two by two shown. If the number of applications or the number of hardware types grows,
or both grow, up to the number of CWE levels shown (i.e. five by five), then approximately
the HTT OA test design will only grow to 25 runs whereas the conventional approach will
grow to over 3000 for 3-level and 1000 for 2-level full factorial testing. The test matrix
presented in Figure 8 provides an efficient way to test the ICS. Once the system has been tested,
the recorded value of the output can be further analyzed to find out which input factor is
significant or if there any interaction between these factors. The result will help to focus or
screen for the more detailed testing that should follow in areas of identified weakness, 8 which
in turn will help choose the most appropriate mitigation strategies for each critical weakness
and thus improve the cyber-incident response and overall cyber-resilience.

Figure 8 rdExpert output.

7
rdExpert LiteTM is copyright Phadke Associates Inc.
8
For example the Cooperative Vulnerability and Penetration Assessment (CVPA) used by the U.S. Department
of Defence (Brown et. al., 2015)
 Conclusion
Given the increasing prevalence of cyber threats, countering any cyber threat to ICS requires
improved test planning and identification of system components and cyber weaknesses. In this
paper, we propose research into the efficacy of advanced test planning and analysis methods
for screening the cyber vulnerabilities of ICS. In order to generate a white-list of processes,
communication protocols, and hardware configuration we discuss the use of available software
packages such as Wireshark and Volatility. Also, to determine system weaknesses and system
vulnerabilities in a standardized and comparable way, we propose to use the common weakness
enumeration database and scoring system methodology. We presented that publically available
information on software weaknesses such as these can be used to structure defensive and
offensive test of ICS/SCADA systems. For an efficient test strategy and due to the
multiplicative effect of possible architecture permutations, defensive postures, offensive threats
and weakness types, such testing needs to use combinatorial mathematics in the test design,
such as high throughput testing. We presented a systematic approach to illustrate how to use
such methods to screen cyber vulnerabilities and to setup an efficient test procedure for more
detailed penetration testing. Such a structured test approach should lead to better ICS cyber
resilience and cyber incident response by being less resource-intensive. Opportunities are
sought to apply this proposed research method to applied systems in Australia.9 The proposed
research will have applicability to defense of complex systems more broadly than cybersecurity
of ICSs, since it illustrates a structured technique for dealing with the very high numbers of test
permutations that arise when considering complex system architectures exposed to an evolving
arsenal of possible malicious intent.

Glossary
CVE: Common vulnerability exposure.
CVPA: Cooperative Vulnerability and Penetration Assessment.
CVSS: Common vulnerability.
CWE: Common Weakness Enumeration.
CWSS: Common Weakness Scoring System.
DOE: Design of experiments.
HTT: High throughput testing.
ICS: Industrial control system.
IP: Internet protocol.
NVD: National Vulnerability Database.
OA: Orthogonal Array.

9
The UNSW Canberra Research centres for Capability Systems and for Cyber Security are available for such
research and offer a subject known as ZEIT 8034 Advanced T&E Techniques as part of their Master programs
(https://www.unsw.adfa.edu.au/capability-systems-centre/advanced-test-and-evaluation-techniques#overlay-
context= )
PId: Process identification.
PLC: Programmable Logic Controller.
RTU: Remote terminal unit.
SCADA: Supervisory control and data acquisition.

References
Austin, G., 2016. Australia Rearmed! Future Needs for Cyber-Enabled Warfare, Discussion
Paper No 1 of the Australian Centre for Cyber Security at UNSW, Canberra,
developed from the International Conference Redefining R&D Needs for Australian
Cyber Security on 16 November 2015 & released publically on 19 January 2016.
Available at https://www.unsw.adfa.edu.au/australian-centre-for-cyber-
security/news/australia-rearmed.
Brown, C.; Christensen, P.; McNeil, J.; & Messerschmidt, L., 2015. Using the Developmental
Evaluation Framework to Right Size Cyber T&E Test Data and Infrastructure
Requirements. The ITEA Journal 36 (1): 2634.
Bryant, W. D., 2016. Defending the Virtual Walls: Active Cyber Defense of Weapon
Systems. The ITEA Journal, 37: 236-240.
Christensen, P. 2015. Introduction to Cyberspace T&E. Tutorial presentation at ITEA
International Conference, Washington DC, August 2015.
Harp, D. and Gregory-Brown, B., 2015 The State of Security in Control Systems Today, A
SANS Survey, SANS Institute, (US).
Folkerth. L., 2015, Forensic Analysis of Industrial Control System, Sans Institute (US).
Mirkovic, J.; Hussain, A.; Fahmy, S.; Reiher, P. Thomas, R. K. (2009). Accurately
Measuring Denial of Service in Simulation and Testbed Experiments. IEEE
Transactions On Dependable And Secure Computing, VOL. 6, NO. 2, APRIL-JUNE.
Pringle, P.; Steinfeld, H.; & Lardieri, P. (2016). Table Top Wargaming: Cost-effective
Planning for Cyber Security Test and Evaluation: Or, How I learned to Stop Worrying
& Love the Six-Phase Cyber Security T&E Process. The ITEA Journal, 37: 257-267.
Reagan, L. A., & Kiemele, M. J., 2008. Design for Six Sigma: The Tool Guide for
Practitioners. Colorado Springs: Air Academy Associates, LLC.
Schmidt, S. R. and Launsby, R. G., 2008 Understanding Industrial Designed Experiments,
4th Edition.
Stirland, J. et. al., 2014, Developing Cyber Forensic for SCADA Industrial Control
Systems. Proc. of the Int. Conf. on Info. Security and Cyber Forensic.
Stouffer, K. et. al., 2015 Guide to Industrial Control Systems (ICS) Security, National
Institute of Standard and Technology, report, 800-82, Rev 2 (US).
Thales, 2013, Cyber Security of SCADA Systems, White paper.
US department of Energy, 2011 Vulnerability Analysis of Energy Delivery Control
Systems, special report, Idaho National Laboratory, (US).
Yenign, H.; Yilmaz, C.; & Ulrich, A. (2016). Advances in test generation for testing
software and systems: An introduction to selected papers from ICTSS 2013,
International Journal of Software Tools for Technology Transfer, 18: 245249.
Zhu B. et. al., 2011, A taxonomy of Cyber Attacks on SCADA systems, Proc. of the
International Conference on Internet of Things.
 Biography
Dr. Obaid Ur Rehman is a Senior Research Associate and Lecturer with the Capability
Systems Centre, University of New South Wales, Canberra. He holds Bachelor degree in
industrial electronics engineering, Masters in engineering, and Doctor of Philosophy degrees
in electrical engineering. He has worked as a professional engineer and manager for 8 years in
the space industry before taking research career in 2008. He is author of a book and has more
than 30 publications in peer-reviewed conferences proceedings and journals.

Dr. Keith F. Joiner joined the Air Force in 1985 and became an aeronautical engineer, project
manager and teacher over a 30 year career before joining the University of New South Wales
in 2015 as a senior lecturer in test and evaluation. From 2010 to 2014 he was the Director-
General of Test and Evaluation for the Australian Defence Force where he was awarded a
Conspicuous Service Cross. He was responsible for Defence-wide test and evaluation policy,
ensuring all new capability submissions to Government had test plans, for conducting trials on
all proposed new capabilities (prototypes and off-the-shelf), and operational field evaluations
for new Army and joint capabilities. During his tenure in this role he testified to the Senate
Inquiry into Defence Procurement (2012) and was tasked by the Service Chiefs with
developing and implementing new reforms in test and evaluation. Dr Joiner has completed a
number of post-graduate degrees including a Masters of Science in Aerospace Systems
Engineering with distinction through Loughborough University in the United Kingdom, a PhD
in Calculus Education and a Masters of Management. In previous roles he was a design
engineer for aircraft and missiles, a project engineering manager, a chief engineer for several
aircraft types, and a base commander for an airfield and domestic infrastructure. In 2009 he did
wartime service as a plans officer in Baghdad for the Multi-National Force Iraq where he was
awarded a United States Meritorious Service Medal for his work developing drawdown plans.
He is a Certified Practicing Engineer with the Institute of Engineers Australia and a Certified
Practicing Project Director with the Australian Institute of Project Management. In all his roles
he has worked extensively with Defence scientists, sponsoring, conducting and trialing
experiments and research including recent important human factors and anthropometric work
for future submarines and combat vehicles.