Security Testing Methodology for Critical Infrastructures

SEVENTH FRAMEWORK PROGRAMME
Theme SEC-2011.2.5-1 (Cyber attacks against critical infrastructures)
D5.1 Security Testing Methodology

Contract No. FP7-SEC-285477-CRISALIS
Workpackage
Author
Version
Date of delivery
Actual Date of Delivery
Dissemination level
Responsible
WP 5 - Vulnerability Discovery
Marco Caselli, Frank Kargl
1.0
M12
M12
Public
UT
The research leading to these results has received funding from the European Communitys
Seventh Framework Programme (FP7/2007-2013) under grant agreement n285477.

Theme SEC-2011.2.5-1 (Cyber attacks against critical infrastructures)
The CRISALIS Consortium consists of:

Symantec Ltd.
Alliander
Chalmers University
ENEL Ingegneria e Innovazione
EURECOM
Security Matters BV
Siemens AG
Universiteit Twente
Project coordinator
Contact information:
Dr. Corrado Leita
2229 Route des Cretes
06560 Sophia Antipolis
France
e-mail: corrado_leita@symantec.com
Phone: +33 673 41 91 27
Ireland
Netherlands
Sweden
Italy
France
Netherlands
Germany
Netherlands
Contents
1. Introduction
6
1.1. General Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.2. Standardization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3. Testing Critical Infrastructures . . . . . . . . . . . . . . . . . . . . . . . . 10
2. Standard
2.1. OSSTMM . . . . . . . . . .
2.1.1. General information
2.1.2. Overview . . . . . .
2.2. NIST SP800-115 . . . . . .
2.2.2. Overview . . . . . .
2.3. ISSAF . . . . . . . . . . . .
2.3.1. General Information
2.3.2. Overview . . . . . .
.
.
.
.
.
.
.
.
.
3. Requirements
3.1. Approach . . . . . . . . . . .
3.2. Questionnaire . . . . . . . . .
A. General information . . . . .
B. Security requirements, design,
C. Testing and methodologies . .
D. Research . . . . . . . . . . . .
E. Follow up . . . . . . . . . . .
3.3. Further analyses . . . . . . .
4. Methodology
4.1. Overview . . . . . . . .
4.2. Workflow . . . . . . . .
4.2.1. Main Workflow .
4.2.2. Workflow phases
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
and regulations
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
12
12
12
14
19
19
22
29
29
31
.
.
.
.
.
.
.
.
37
37
38
38
40
41
46
46
46
.
.
.
.
50
50
51
51
53
4.2.3. Penetration Testing Steps . . . . . . . . . . . . . . . . . . . . . . . 57

4.3. Final Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
5. Conclusion
63
Abstract
Existing security testing methodologies and tools cannot be applied directly to critical
environments, due to a number of safety and availability requirements. One of the goal
of WP3 is to understand what assessment processes we can exploit in Industrial Control
System (ICS) environments and what are the constraints given by industrial components
and infrastructures. This document provides an overview of assessment methodologies
used in standard IT. We analyze these methodologies to investigate their applicability
in an ICS environment. We are also conducting a survey among stakeholders in order
to detail existing best practices adopted by companies to test their infrastructures and
discover vulnerabilities in their systems. From this we are proposing a new methodology
tailored to support security testing in Critical Infrastructure environments. As the
evaluation of the feedback and the methodology is requiring more work than originally
foreseen and as our on-going investigations are also bringing up new aspects not foreseen
in the original proposal (e.g., incident handling), this deliverable should be regarded only
as a first version of D5.1 that we intend to extend in a version 2 either after year 2 or
at the end of the project in order to address these additional aspects.
1. Introduction
Security testing involves various techniques which complement security engineering, i.e.,
design and implementation of secure systems. In contrast, security testing implies testing
of those systems with regards to their security properties. Security testing includes
analysis-centered activities like a code review but also practical tests to penetrate a
security system and find vulnerabilities and implement attacks. The latter is often
termed penetration testing.
Per definition, a penetration test is a simulation of an attack aimed at assessing the
security of a computer system or a network. Simulation refers to the fact that the test
is conducted by the system owner or on his request. This test has multiple phases and
allows to highlight infrastructures and protocols weaknesses. During a penetration
test, the penetration tester tries to discover and sometimes exploit known or unknown
vulnerabilities in order to obtain useful information or to access targeted hosts. At the
end, all security problems discovered are reported to the system owner together with an
assessment of their impact on the system. The analysis should also propose technical or
organizational mitigations to identified problems.
In its most simplistic form, a penetration test only involves running some automated
security scanner, like, e.g., Nessus [1]. However, it is generally agreed that serious penetration testing typically involves also manual test planing, preparation, and conduction
to be able to flexibly react to the system under test. In such a case, the penetration
tester should follow some structured methodology to ensure valid, reproducible, and well
documented results.
The development and use of such manual penetration testing methodologies is motivated by several reasons. First of all, there are numerous vulnerabilities that may be
difficult or impossible to detect with automated tools. Network or application vulnerability scanning software are nowadays highly sophisticated. However, sometimes, high
profile systems and infrastructures need a comprehensive and systematic approach to
be compromised which cannot be achieved by automated tests. This is, e.g., related to
the detection of high-risk vulnerabilities resulting from a combination of low-risk ones.
As multiple minor weaknesses are exploited in a particular sequences, a penetration test
may reveal all the possibilities identifying the possibly dangerous combinations.
More importantly than identifying vulnerabilities, determining the feasibility of a particular set of attack vectors is another important target of a penetration test. As having
1.1. General Classification

theoretically unassailable systems seems impossible in practice, companies and organizations are interested in running infrastructures that are practically secure. Therefore,
the analysis becomes a way to rank vulnerabilities both by severity and realizability.
The potential business impact is a further valuable indicator of importance for a vulnerability. Conducting practical attacks also aims at properly identifying necessary attacker effort which is a prerequisite for correct risk estimation. Assessing the magnitude
of economical and operational problems related to a successful cyber attacks is usually
the last phase of analysis of a penetration test. Based on the knowledge of business
processes such analysis gives an accurate picture of the risk related to the infrastructure
and to the work such infrastructure performs.
Finally, penetration tests are the experiments with which companies try out the ability
of network defenders to successfully detect and respond to cyber attacks. Their capacity
of reaction is measured during the test and evaluated at the end of it. It will also
provide evidence to support increased investments in security personnel and technology
or it will suggest modifications in methods and strategies are the output of such kind of
assessments.
1.1. General Classification

Penetration tests can be performed in a broad variety of ways. The following criteria
provide some orientation and classification.
The amount of information known by the fake attackers is one of the main discriminating factor of penetration tests. The simplest classification describes three different
types of tests: white-box, black-box, and gray-box.
White-box penetration testing outlines a situation in which fake attackers have full
knowledge of the system under attack. The goal of a white-box penetration test is
to simulate the presence of an expert malicious insider. In some special cases such
insider has also basic credentials for the target system. Among the information known
to the fake attacker we can have: network diagrams, domain names, IP addresses, but
also phone numbers, e-mail addresses. White-box testing of applications usually allows
intruders to have also programs source code. White-box test design techniques include
[2]:
Control flow testing
Data flow testing
Branch testing
FP7-SEC-285477-CRISALIS
1. Introduction
Path testing
Statement coverage
Decision coverage
Black-box penetration testing is the opposite of the previous one. This method examines
infrastructures and applications functionalities without knowing anything about them.
In general, the tester is aware of what the software is supposed to do but is not aware
of how it does it [3]. Thus, specific inputs (e.g network packets, shell commands, etc.)
return specific outputs without the intruder knowing how the mechanisms within the
system work. The goal of a black-box penetration test is to simulate a typical external
hacking or also cyber warfare scenario. Typical black-box tests design techniques include
[4]:
Decision table testing
All-pairs testing
State transition tables
Equivalence partitioning
Boundary value analysis
Finally, Gray-box penetration testing is a combination of white-box and black-box methods. A gray-box tester partially knows the internal structure of the system under attack.
This knowledge can include network topology, installed applications or also more in deep
information like used algorithms and data structures. Since any combination of knowledge owned by the testers falls in the domain of gray-box penetration testing this method
is the most widely used, especially in the field of web applications. Gray-box testing
techniques include [5]:
Matrix Testing
Regression testing
Pattern Testing
Orthogonal array testing
1.2. Standardization
Other ways to characterize penetration tests include the type of system access (external network, internal network, physical access), whether a test is conducted in pure
physical ways or whether social engineering attacks are also performed, or whether testing includes performing (possibly disruptive) attacks or whether such attacks should
only be identified but not performed. The specifics of a test need to be carefully defined
in the test plan.
1.2. Standardization
In this last decade the interest in penetration testing techniques has increased. Companies, organizations and governments provide numerous services on the Internet and
need to assess the security of their systems. Given the proliferation of different penetration testing methods, some organizations and standardization bodies decided to propose comprehensive and usable methodologies. While many penetration testers follow
a home-brewed self-defined testing approach, following standardized approaches has
some general benefits.
A standardized and generally accepted penetration methodology provides a certain
confidence that no important steps or aspects of a test are forgotten by accident. As tests
should be repeated after significant changes to the system under test, following the same
methodology also provides the possibility to compare multiple tests to identify whether
a system got more secure after mitigating measures have been taken. A standardized
approach may also be required in cases where security tests are required from a legal or
insurance perspective. A standardized approach and common reporting forms also allows
to compare results with results from other installations. Finally, accepted standards
simplify negotiations between a security tester and clients about the scope of such tests.
So there are a number of good reasons to look into such standardized and commonly
agreed approaches.
Over the years, many standardized methodologies have been proposed. From most
general to system-specific ones, both governments and private organizations have proposed different ways to approach penetration tests. Among the most well-known penetration testing techniques are: OSSTMM (Open Source Security Testing Methodology
Manual), NIST SP800-115, ISSAF (Information Systems Security Assessment Framework), OWASP (Open Web Application Security Project), and PTES (Penetration Testing Execution Standard). Later in this document, we are going to give a broad overview
of some of these approaches.
In addition to methodologies, different kind of systems exist to assess specific contexts
of security testing without a structured approach. Companies can decide not to follow a
1. Introduction
methodology but just perform a light assessment using different methods. In this cases
they can exploit several guidelines and toolkits developed for numerous and different
purposes. According to [6] toolkits implement and bundle in a convenient package a set
of testing techniques, usually aimed at discovering specific classes of security problems.
Guidelines instead organize the process of security testing, by collecting sets of best
practices, comprehensively listing items to be tested, and structuring any other kind of
useful advice. Finally, there are also collections of penetration testing tools such as
Backtrack Linux or metasploit that are regularly used by penetration testers.
1.3. Testing Critical Infrastructures

Despite the recent interest for penetration testing methodologies, none of the proposed
methodologies specifically considers industrial control systems and critical infrastructures despite the fact that penetration testing is also performed in such environments
on a routinely basis.
We argue that standard methodologies should not be applied there without special
consideration. These systems and the networks that interconnect them show many differences compared with standard IT networks. Since critical infrastructures control and
automate real world processes or equipments which are potentially dangerous for humans, any activity different from the main purpose including penetration testing should
be limited. So just running a full Nessus or Metasploit test in such an environment could
have disastrous consequences.
There are three main concerns that make penetration testing on critical infrastructures
challenging. As introduced in the previous paragraph, any penetration attempt can have
a consequence in the real world. What follows is an interesting example of such concerns.
In a real incident, a gas utility conducting a penetration test on its corporate network
performed this in a careless manner affecting some of the DCS systems. During the
test this DCS system was locked up by mistake and the utility was not able to send
gas through its pipelines for four hours leading to a large economic loss [7]. The strong
relation between the system under test and physical components that people depend
on is the first important difference with respect to standard IT. This implies that any
CI penetration testing methodology needs to include safeguards to protect from such
failures.
Another important difference concerns the devices used in Critical Infrastructure,
these systems usually have very limited resources compared to devices found in normal
IT environments. The longevity of network components is greater and the cycle of
replacement of devices may also be in the order of decades. This implies that devices
10
1.3. Testing Critical Infrastructures

may be out of service, no patches may be available, and generally proposing mitigation
strategies may take different approaches. This also means that critical infrastructures
can be more easily overloaded with respect to other IT systems. A penetration testing
methodology has to take care not to override the normal operation of every component
in the ICS unless identifying DoS attacks is specifically in the scope of the test.
Finally, due to the size of most of the critical infrastructures, a penetration testing has
always to check if localized attacks have effects in different parts of the network which
may be hard to assess due to indirect effects. At the end of a test the identification of
a vulnerability is followed by the calculation of the damage that it can cause. Linking
these two elements in a system with thousands of components requires the assistance
of experienced personnel both in the IT field and in the field related to the industrial
process under control. As a consequence, penetration tests cannot be conducted by
security experts in isolation.
We think that a penetration testing methodology for critical infrastructure must explicitly describe how to deal with the previous issues. Moreover, it has to identify specific
methods, tests, and tools already used in standard penetration testing techniques and
explain how to tune these in order to be suitable for industrial control systems. Finally,
such an ICS penetration testing methodology must provide where necessary new
instruments and customized methods to test IT and networking components not found
in standard ICT (e.g., Process Control Units, Remote Terminal Units, field networks,
etc.).
11
2. Standard
In this chapter, we are going to review multiple penetration testing methodologies proposed by standardization bodies and other organizations. The aim is to provide a broad
overview over the state of the art before defining specific requirements for CI penetration testing and then proposing an own CRISALIS penetration testing methodology for
Critical infrastructures that is distilled from those existing approaches.
2.1. OSSTMM
The Open Source Security Testing Methodology Manual (OSSTMM) [8] is an open standard methodology for security tests. Developed by Pete Herzog at the end of 2000 as
an ethical hacking framework, it has rapidly grown to become a comprehensive methodology to assure security at operational level. Version 3, released in 2008, encompasses
tests for every security aspect: from personnel qualification to physical security, from
control of communication to electronic systems safety. As every standard methodology,
it is designed to be consistent and repeatable. Moreover, it is openly available and thus
allows a free dissemination and free use.
Figure 2.1 shows a comparison among OSSTMM and common security tests in terms
of accuracy and thoroughness. By its nature, the methodology can be adapted to any
kind of audit operation (e.g. penetration tests, ethical hacking, security and vulnerability assessments, red/blue teaming, etc.). The primary purpose always remains the
description of a security examination path and the definition of a way to consistently
correlate test results.
The usage of the OSSTMM allows the auditor to perform a validated set of tests and
to qualify its examination as a certified OSSTMM audit.
Differently from other methodologies, mitigations to security flaws are not required
as part of an OSSTMM audit. Recommendations may well be part of the final results,
but common operational situations often imply that auditors have a limited view and
knowledge of the client environment. In these cases it is difficult to propose proper
and feasible solutions. For this reason solutions are usually avoided in final reports but
should we worked out together with the system owner after the test. This is an aspect
12
2.1. OSSTMM
Figure 2.1.: OSSTMM comparison with common security tests (from [8])
that is well in line with our findings from the previous chapter as it will also apply to
CI.
An important aspect of the OSSTMM is the compliance with general security policies.
The methodology is developed taking into account major legislation and regulations.
Furthermore, OSSTMM defines three different types of compliance and a set of rules to
deal with all of them. The three types of compliance defined in the methodology are:
Legislation: enforces the security test to be compliant with region/country legislation. The lack of this requirement could lead to criminal charges.
Regulation: enforces the security test to be performed accordingly to standard
regulation within industry sector. A failure in this task could lead to a dismissal
of the company from the group.
Policy: enforces the security test to be done according to business and organization
policies. Violation could cause a dismissal of the responsible from the company.
OSSTMM provides a list of national and international regulations already proved to

be compliant to the proposed methodology.
13
2. Standard
2.1.2. Overview
Before describing the workflows, the OSSTMM introduces several concepts to outline
what kind of security tests can be performed and what the scope of such tests can be.
Figure 2.2 presents the six different security test types defined by OSSTMM. Possible
security tests are however not limited to these cases and can be tuned to more specific
requirements.
Figure 2.2.: OSSTMM security test types map (from [8])
Blind: this type of security test describes a situation in which the attacker does not
know anything about the target. On the other side, the defender knows everything
about the audit operations and it is prepared to take countermeasures. A blind
security test is primarily used to evaluate attacker skills.
Double Blind: as before the attacker does not know anything about its target but,
this time, the target is not notified in advance about the security test. A double
blind audit is used to test both the skills of the attacker and the preparedness of
the defender.
Gray Box: in a gray box security test the attacker has limited knowledge of the
target and can better engage the defender. The defender still has the advantage
of full knowledge about the audit in progress. As in the blind security test, this
case is mostly used to evaluate attackers skills.
14
2.1. OSSTMM
Double Gray Box: in this type of security test both the attacker and the defender
have limited knowledge about their counterparts. Such situation is used to evaluate
both the contenders. With respect to double blind this test increases the level of
accuracy and pervasiveness.
Tandem: in the Tandem security test the attacker has full knowledge of the
target and the defender is fully prepared to bear the test. Such situation is used
to improve the thoroughness of the audit as the attacker has a full view of all tests
and their responses.
Reversal: as in the previous situation, the attackers knows everything about the
target while the defender knows nothing about its contender. The purpose of this
test is to evaluate every operational aspect of the defender.
Once the test type of interest is identified, it is necessary to identify the scope of
the audit. OSSTMM divides such scope into three different channels: COMSEC (communication security), PHYSSEC (physical security), SPECSEC (spectrum security). A
thorough security audit would require testing all three channels but usually the tests are
tuned on more specific needs. For this reason, the methodology makes a further partitioning, addressing these three channels into five logical sections. Channel PHYSSEC is
divided into Human and Physical, channel SPECSEC contains Wireless Communication, and channel SPECSEC is divided into Data Networks and Telecommunications.
What follows is a description of the five logical sections:
Human: concentrates on the human element and its communication processes.
Physical: focuses on physical security testing and comprises tangible elements
of security where interaction requires physical effort or an energy transmitter to
manipulate.
Wireless Communications: concentrates on security tests on wireless electronic
communications and signals.
Data Networks: comprises tests on electronic systems and data networks.
Telecommunications: focuses on digital/analog telecommunications where interaction takes place over established telephone network lines.
The full methodology is divided into seventeen modules. Every module defines a
specific target and several tasks that are needed to achieve it. Each module has an input
15
2. Standard
and an output. The input represents the information needed to perform each task while
the output is the result of the accomplished tasks. Figure 2.3 shows the methodology
flow.
OSSTMM uses the same workflow for all the channels. The seventeen modules and
the methodology flow apply in the same way to the five logical sections. However, while
the methodology remains the same, each channel will differ in the proposed tasks.
The methodology is divided into four phases: Regulatory Phase (modules 1 to
3), Definitions Phase (modules 4 to 7), Information Phase (modules 8 to 13) and
Interactive Controls Test Phase (modules 14 to 17). Each phase faces a different
level of depth in the audit. The regulatory phase focuses on understanding the audit
requirements, scope and constraints. The definitions phase improves the description of
the scope and of the inner interactions and processes. The information phase uncovers
misplaced and mismanaged information flows in the target. Finally, the interactive
controls test phase focuses on practical penetration testing. This is often the final phase
of a security test. This ensures that practical attacks do not affect less invasive tests.
In what follows we briefly describe the focus of every module.
Posture Review: it is used to define the scope and identify what tests must
be performed. This module focuses on rules, norms, regulations, legislations and
policies applicable to the target.
Logistics: it defines which limitations the audit process has. It relates to interaction constraints such as physical distance, communication speed, etc.
Active Detection Verification: it takes into account any practical restriction
imposed against performing interactive tests on the targets.
Visibility Audit: it determines available targets within the scope. This module
provides the first in depth description of the targets and how they interact with
the scope.
Access Verification: it measures the breadth and depth of interactive access
points. It verifies if such access point to the target exists and if it requires authentication.
Trust Verification: it identifies trust relationships from and between the targets.
This module verifies if targets accept interaction freely and without credentials.
Controls Verification: it measures the effectiveness of process-based attributes
like non-repudiation, confidentiality, privacy and integrity.
16
2.1. OSSTMM
Figure 2.3.: OSSTMM methodology flow (from [8])
17
2. Standard
Process Verification: it verifies the existence, effectiveness and maintenance of
security levels defined for the targets.
Configuration/Training Verification: it explores the default conditions under
which targets operate regularly to understand their intents, business justification
and reasoning.
Property Validation: it checks the use of illegal or unlicensed intellectual property or application within the targets.
Segregation Review: it checks the amount of personally identifiable information
available depending on the applied rules and legislations.
Exposure Verification: it searches for freely available information which uncovers indirected or unwanted visibility of the targets.
Competitive Intelligence Scouting: it searches for freely available information
which could harm or adversely affect the target.
Quarantine Verification: it determines the effectiveness of authentication in
terms of black and white listing quarantines.
Privileges Audit: it measures the impact of misusing credentials and privileges
and the consequences of unauthorized escalations of privileges.
Survivability Validation/ Service Continuity: it checks the resistance of the
target to excessive or adverse changes.
Alert and Log Review/End Survey: it is a review of the performed audit
activities.
The outcome is a comprehensive assessment of the tested channels. Results are presented through Risk Assessment Values (RAVs) that are a set of security metrics providing an overview on the state of the channels (e.g. measurement of the attack surface,
the amount of uncontrolled interactions with a target, etc.). To finally measure the
thoroughness of the test performed, the OSSTMM suggests to conclude the work with
a Security Test Audit Report (STAR). This step requires to record in the appropriate
form the following information:
Date and time of test
Duration of test
18
2.2. NIST SP800-115

Auditors and analyst involved
Test type
Scope of test
Test Index
Channel tested
Test Vectors
Verified test and metrics calculations of the operational protection levels, loss controls and security limitations
Knowledge of which tests have been completed, not completed, or only partially
completed, and to what extent
Any issue regarding the test and the validity of the results
Test error margins
Any processes which influence the security limitations
Any unknowns anomalies
Overall, OSSTMM provides a very comprehensive framework that addresses a large

variety of practical aspects. It is specifically useful to ensure completeness of our later
approach. However, it is also relatively generic and is not going into details of specific
systems and is not addressing the CRISALIS scenarios at all.
2.2. NIST SP800-115

The National Institute of Standard and Technology (NIST) has designed its own security
assessment methodology in the Special Publication 800-115 [9]. The purpose of this
publication was to provide guidelines for organizations on planning, conducting and
evaluating information security testing. Even if the overall goal is to propose an overview
of main key elements of technical security assessments, NIST SP800-115 provides also
practical recommendations and technical information relating to penetration tests.
19
2. Standard
NIST defines the security assessment as the process of determining how effectively
an entity being assessed meets specific security objectives. From an high-level point of
view, there are three possible ways to accomplish such target:
through Testing, by exercising the assessment objects comparing actual and expected behaviors
through Examination, by analyzing the assessments objects and understand their
functioning
through Interviewing, by conducting extensive discussions with organizations
and experts to identify possible problems
A methodology that implements one of the previous points has to contain at least
three different phases:
Planning: this phase is necessary to collect all the information needed for the assessment. This concerns elements like assets, threats, policies in use, etc. Moreover,
the assessment should have a final goal and a set of objectives to work towards.
Execution: this phase concerns the operational part of the assessment. The Execution implements operations of environment discovery and analysis, vulnerability
identification and result validation. Practical activities for this phase obviously
differ from case to case. However, the main goal is to bring to light problems at
technical and organizational level.
Post-Execution: this phase describes all the analysis aim at identifying vulnerability causes and mitigation strategies. The development of a final report is the
main output of the Post-Execution.
As already discussed, NIST SP800-115 offers an overview of security assessment key

points. However it suggests several practical techniques to implement penetration testing. According to the authors, the most commonly used pentesting techniques can be
grouped in the following categories:
Review Techniques define how to examine and analyze assets and their security
policies.
Target Identification and Analysis Techniques define how to identify assessments targets and how to describe their properties.
20
2.2. NIST SP800-115

Target Vulnerability Validation Techniques: define how to verify the presence of vulnerabilities within the targets and how to test their security levels.
The NIST methodology focuses on how these techniques can be used together, but
does not provide a list of technical activities to be used in any specific circumstance. In
addition to practical penetration testing techniques, the methodology proposes several
non-technical methods that can be used to enforce the assessment (e.g. physical security
testing).
Finally, NIST SP800-115 concentrates on how singular tests and final reports can be
compared with each others. The possibility to compare two security assessments relies
on the correct representation of the Testing Viewpoint. Every test can be performed
from different viewpoints. This implies different information owned by the auditors or
different physical location of the pentesters. Results of comparisons among different
assessments always need to consider the kind of testing. NIST SP800-115 defines four
testing types:
External Security Testing: this type of testing is conducted from outside the
organizations security perimeter. The goal of this security test is to reproduce the
view of an external attacker and to draw the attention to vulnerabilities already
visible from outside the company premises (e.g. from the Internet). External
testing always begins with discovery techniques aiming at examining organization
public information.
Internal Security Testing: differently from the previous one, this kind of testing
is performed within an organizations perimeter (e.g. internal network). The
purpose of the Internal Security Testing is to impersonate a trusted insider or also
an attacker that has penetrated external defenses. This test allows pentesters to
have some level of access to the network and, thus, to internal information. These
elements make it possible to assess internal security mechanisms.
Overt Security Testing: also known as White Hat testing, defines an Internal
or External Testing in which pentesters have a full knowledge of organizations
systems and processes. Company staff is also aware of the test and it usually tries
to limit the testing impact. This kind of test is often used as company training.
Covert Security Testing: also known as Black Hat testing, uses an adversarial
approach by not giving any knowledge about the company to the pentesters. In
this case, company IT staff is usually also not aware of the test and its response
is used for testing the technical and organizational security controls within the
company.
21
2. Standard
2.2.2. Overview
In this section we describe in detail the three groups of techniques discussed before and
the three phases into which the methodology is logically structured.
Review Techniques describe the process of passively gathering information related
to the organization and the consequent analysis operations. This information regards
both policies and equipments (e.g. systems, networks, etc.). Since these techniques are
passive, they do not interfere with company operations and systems. For this reason,
this is usually the first phase in every security assessment. Possible review techniques
are:
Documentation Review examines the technical aspects of used polices and procedures. Security flaws or weaknesses in such elements can lead to missing or
improperly implemented security controls. The results of the analysis can also be
used to better tune the following techniques.
Log Review determines if important information and operations are properly
logged and recorded. This is a necessary element to prove that all the systems are
working according to the applied policies and procedures. Log analysis can show
also misconfigured systems and malicious operations (e.g. attempted or successful
attacks).
Ruleset Review checks the collection of rules and signature-based security mechanisms used to detect malicious operations or implement organizations processes.
Routers, firewalls and intrusion detection systems are the main targets of this analysis. Router access control lists are checked to identify whether rules are no longer
needed or enforce the refusal of unknown traffic. The review refines in depth firewall rules by checking unnecessary open ports and allowed services. Moreover, it
searches for possible backdoors and malware. Finally, IDS rulesets are verified to
be up to date and tuned to organizations needs.
System Configuration Review searches for weaknesses in systems and applications configurations (e.g. programs not enforcing security or non-compliant with
security policies). This review usually concerns: unnecessary services running on
servers, improper user account and password settings, etc.
Network Sniffing implements the passive monitoring of a data network. It can
rely on communication flows, protocols decoding, and header/payload analysis.
It is used to map organization communication infrastructures and gathering information about used systems, applications and services. It can also identify unau-
22
2.2. NIST SP800-115

thorized and inappropriate activities or capture unencrypted personal information
(e.g. usernames and passwords).
File Integrity Checking provides a way to check file modifications. It is usually
implemented through checksum.
Target Identification and Analysis Techniques focus on identifying active devices through
open ports and running services and analyze them looking for vulnerabilities. NIST
SP800-115 discusses several different techniques belonging to this group:
Network Discovery concentrates on discovering active and responding host on a
network. Active techniques send various types of packets to the network waiting for
responses. This system is usually an improvement of the passive analysis discussed
in the Review Techniques as it is able to find hosts not currently involved in any
communication. During this analysis, pentesters have to face firewalls and intrusion
detection systems that usually identify many instances of scans. Active discovery
generally produce network noise and communication delays.
Network Port and Service Identification is usually performed in parallel with
the network discovery. This check involves the use of port scanners able to understand services and application running on remote hosts. Information gathered
during the process allows pentesters to exploit OS fingerprinting tools. Such applications guess operating systems working on the network and provide a description
about their versions and updates.
Vulnerability Scanning, like the previous two operations, collects information
about hosts in the network. Moreover, it also attempts to identify specific vulnerabilities on discovered hosts. Vulnerability scanners usually provide: a check on
the compliance of security policies, a list of targets for the following penetration
testing, and preliminary information on how to mitigate discovered vulnerabilities.
Wireless Scanning: as the number of wireless devices is increasing, these techniques are becoming more and more important. Wireless scanning techniques can
be further divided in: passive scanning, active scanning, bluetooth scanning, and
device location tracking. An example of passive scanning involves the analysis of
MAC addresses (e.g. vendor identification or white/black listing of devices). An
active scanning can be used instead to verify authentication mechanisms and data
encryption. Bluetooth security issues have been extensively discussed in literature [10], [11]. Bluetooth scanning is more difficult than wireless one due to the
its short range. However, a confirmation of compliance with organization security
23
2. Standard
requirements is needed to have a comprehensive security assessment. Finally, wireless networks can be used to locate organizations devices. This check enforces the
reconstruction of the companys network topology.
Target Vulnerability Validation Techniques uses the information found in the two previous phases and further explore the existence of potential vulnerabilities. The target is
usually not limited to identifying the vulnerability but also to show the security exposure
by trying to exploit it. This procedure is often risky for the organization infrastructure
and processes. For this reason is usually the last active work performed on the network.
In this way, it will not compromises other tests. NIST SP800-15 defines and implements
its penetration testing method in this phase. The targets of the penetration testing can
be numerous. Pentesters are interested in understanding how well the system tolerates
real attacks, identifying necessary countermeasures, and evaluating defenders abilities.
Moreover, this attack simulation can show what level of skills attackers need to compromise organizations systems. The proposed penetration testing is divided in four steps
as show in Figure 2.4.
Figure 2.4.: NIST SP 800-115 penetration testing steps

The Planning step prepares the necessary information and background for the attack.
The Discovery step integrates information already owned by pentesters with a new
session of network scanning. With respect to the other phases, the attackers try to
acquire information about: host names and IP addresses, employee names and other
contacts, and system information (e.g. names and shared resources). In some cases also
techniques as dumpster diving can be used to enforce the process. The Discovery
step improves also the vulnerability analysis and finalizes the list of targets. The Attack
step is the core of every penetration testing. During the attack a continuous feedback
24
2.2. NIST SP800-115

is provided back to the discovery step as succeeding to access a system gives pentesters
additional knowledge of the infrastructure under attack. Figure 2.5 shows an example
of attack.
Figure 2.5.: NIST SP 800-115 attack example (from [9])

NIST SP800-115 organizes the most common vulnerabilities exploited during the Attack step in the following categories:
Misconfiguration: misconfigured security settings (e.g. insecure default settings).
Kernel Flaws: security flaws in the core code implementing an operating system
that can endanger the whole system.
Buffer Overflows: inadequate checks on input sizes. This misbehavior can cause
arbitrary code been executed with the privileges of the running program.
Insufficient Input Validation: missing checks on the content of input strings,
files, etc. Network applications without filters can run malicious commands (e.g.
SQL injection).
25
2. Standard
Symbolic Links: as the operating system allows to change permissions related to
files, a malicious user can trick the system by creating symbolic links and modifying
permission passing through them.
File Descriptor Attacks: malformed file descriptors can grant access to the
related files.
Race Conditions: an attacker can take advantage of high privileges processes by
modifying their execution order and manipulate results.
Incorrect File and Directory Permission: incorrect permissions can cause
various lacks of information (e.g. reading and writing password files).
The last step, called the Reporting, occurs simultaneously with the other three steps
and records all the results. At the end of the test, pentesters report the identified
vulnerabilities, provide a risk rating, and propose possible mitigation techniques for
discovered weaknesses.
Security Assessment Planning provides information on creating an assessment policy,
scheduling tests, identify the most suitable approach, and addressing possible problems.
It also enforces assessments to be compliant to current regulations and legislations.
According to NIST SP800-115 this planning activity should identify:
Organizational requirements assessments have to take into account
Roles and responsibilities
Adherence to already established methodologies
Assessment frequency
Documentation requirements
In this phase organizations have to choose assessment techniques and customize them
to fit the requirements. Organizations should also carefully consider risks related to use
such techniques on their infrastructures. The selection of auditors and the identification
of needed skills is another fundamental step of the planning. Assessors experience can
determine the success or the failure of the evaluation. Organizations can choose the type
of assessment (e.g. External/Internal and Overt/Covert) and finally propose technical
tools and resources available to the assessors.
At the end, the assessment plan should answer to the following basic questions:
26
2.2. NIST SP800-115

What is the scope of the assessment?
Who is authorized to conduct the assessment?
What are the assessments logistics?
How should sensitive data be handled?
What should occur in the event of an incident?
Security Assessment Execution phase highlights key points for assessors to consider
throughout the execution phase.
Assessors have to be coordinated with various entities and stakeholders within the
organization. Appropriate organizations personnel should help and supervise the assessment for the whole duration of the activity. During operations assessors can face
different kind of problems (e.g. technical, operational, political, etc.). These can include:
Resistance: to the assessments, from entities within the organization (e.g. network administrators or end users).
Lack of Realism: that indicates the preemptive modification to system settings
in order to make the infrastructure more secure to pass the tests.
Immediate Mitigation: that describes the adjustment and the resolution onthe-fly of weakness found during the assessment. This can disturb following tests
and compromise the final report.
Time: time constraints are usually a problem as security assessments are often
an exception more than a normal operation integrated in the development and
deployment cycle.
Resources: organization can make a stand on providing and maintaining adequate
resources for security assessments.
Evolving Technology: tools used for security assessments as well as used techniques can be out-of-date.
Operational Impact: even if a security assessment should prevent any impact
on infrastructure functioning, there is always a chance of accidental problems. For
this reason, an incident response plans should be already in place during testing.
27
2. Standard
Assessors can make some preliminary analysis on the assessment during the assessment
itself. This preliminary analysis of the causes of the noticed problems becomes the input
for the last phase of the security assessment. Most common problem causes can be
organized in the following sets:
Insufficient patch management
Insufficient threat management
Lack of security baselines
Poor integration of security into the system development life cycle
Security architecture weaknesses
Inadequate incident response procedures
Inadequate training for end users and network/systems administrators
Lack of security policies or policy enforcement
Finally, the Post-Testing Activities concentrates on finalizing the analysis and proposing mitigation actions and recommendations. Security testing reports should be used by
the organization as follows:
As a reference point for corrective action
In defining mitigation activities to address identified vulnerabilities
As a benchmark for tracking an organizations progress in meeting security requirements
To assess the implementation status of system security requirements
To conduct cost/benefit analysis for improvements to system security
To enhance other life cycle activities (e.g. risk assessment)
To meet reporting requirements
Overall, NIST SP800-115 also provides a well-structured and complete approach that
goes into more technical detail than OSSTMM. From this perspective, it is interesting
to discuss whether such details are necessary and helpful and make a methodology more
applicable or whether they just inevitably restrict its use to certain use cases.
28
2.3. ISSAF
2.3. ISSAF
2.3.1. General Information
The Information Systems Security Assessment Framework (ISSAF) [12] is a peer reviewed structured framework designed by the Open Information Systems Security Group
(OISSG). The methodology defined by ISSAF covers all the aspects related to security
assessments: from an high-level perspective (e.g. business impact and organizational
models) to practical techniques (e.g. security testing of passwords, systems, network,
etc.).
The framework is divided in four main phases structured in several working packages
(named activities). The four phases are respectively: Planning, Assessment, Treatment, and Accreditation. Figure 2.6 shows ISSAF workflow.
Figure 2.6.: ISSAF framework overview (from [12])

The Planning phase concerns all the operation needed to define a project plan for the
29
2. Standard
whole assessment process. Before proceeding with the operational part, organizations
have to motivate the assessment to substantiate the underlying concerns. Moreover,
organizations have to prepare a business plan to cover costs and identifying available
personnel and resources. The main steps in this phase are:
Information Gathering
Project Chartering
Resource Identification
Budgeting
Cash Flow
Work Breakdown Structure
Project kick-off
The Assessment phase defines the approach needed to evaluate the Information Security Risks within an enterprise. This phase focuses on the risk assessment process and
addresses every component involved. It is divided into two categories: Inherent Risk
Identification and Controls Assessments. The first consists of the following activities:
Identification of Assessment entities: during this activity the organization
spots the targets of the assessment (e.g. processes, assets, facilities, etc.).
Identification of Threats and Vulnerabilities: this activity describes how to
search, list and describe organizations vulnerability.
Impact Assessment: this activity measures the impact of an exploited vulnerability to the business of the organization
Likelihood Assessment: during this activity the organization evaluates the likelihood that a vulnerability is exploited.
Control Assessment explores instead the following fields:

Evaluation of Legal and Regulatory Compliance: this activity evaluates the
position of the organization with respect to current regulations and legislations.
30
2.3. ISSAF
Evaluation of Enterprise Information Security Policy: this activity focuses
on understanding current organizations approach on implementing and maintaining its security policies.
Evaluation of Enterprise Information Security Organization and Management: this activity directly follows the previous one, as a review on how the
Information Security is organized inside the company.
Assessment of Enterprise Information Systems Security and Controls:
this activity is in charge of reviewing different security aspects of the organization.
For example: physical security, environmental security, technical controls (e.g.
network security, host security, etc.), social engineering, etc.
Evaluation of Enterprise Security Operations Management: during this
activity the organization gains an understanding of the risk of specific operations
processes. The operational areas involved in this operations are: Capacity Management, Vulnerability Management, User Management, etc.
Evaluation of Enterprise Business Continuity Management: this activity
evaluates organization capability of ensuring continuous availability of the Information Technology infrastructure.
The output of the Assessment phase is reported within the Inherent Risk and the
Residual Risk documents.
The Treatment phase implements a platform for taking decision about the aforementioned risks. Its work-flow goes through the selection of safeguards, the development of implementation plan for countermeasures, and the development of a decision
making process. In this phase the organization decides to accept, mitigate, or avoid
vulnerabilities and associated risks identified in the previous phase.
Finally, the process of Accreditation defines the steps needed to an organization to
obtain the ISSAF certification.
2.3.2. Overview
In this section we will review the Penetration Testing methodology proposed by ISSAF leaving aside all practical details regarding specific network components assessing
methods.
ISSAF penetration testing methodology evaluates organizations network and systems.
It consist of three phases and nine assessment steps. The three phases are: Planning
and Preparation; Assessment; Reporting, Clean-up and Destroy Artifacts.
31
2. Standard
The Planning and Preparation phase comprises all the assessments preliminary operations. These regards: requirements definition, actions to be performed, and expectations.
ISSAF requires the identification of a contact person from the target company and those
who will perform the tests. During the kick-off meeting both parties have to agree on the
specific engagement team, the exact dates, times of the test, escalation path and other
arrangements. The meeting has to be concluded with the signing of a formal Assessment
Agreement that will provide mutual legal protection.
The Assessment phase is the core of the penetration testing methodology. ISSAF
approach is a path that leads the attacker to nine operation steps as shown in Figure
2.7.
Information Gathering: ISSAF defines the information gathering in the broader
sense possible. It comprises both technical and non-technical methods. The former looks for DNS records and exploit tools like WHOIS. The latter implements
some social engineering methods (e.g. using search engines and mailing lists to find
information about the target). The more a pentester knows about his victim the
more he will improve his chances to successfully attack it. During this phase, the
attacker does not always need to establish a contact with the target. Important
information can be collected also from public sources (e.g. the Internet). As security assessments are generally limited in time and resources, information gathering
is important to identify points that will be most likely vulnerable and focus on
them.
Network Mapping: this phase directly follows the previous one and refines the
information gathering with a more technical approach. The target is to identify
what devices are working in the network and outline its topology. Network mapping
involves the use of fingerprinting and network analysis tools. More in details,
ISSAF specifies different classes of actions to be performed:
Find live hosts

Port and service scanning
Perimeter network mapping (e.g. router, firewalls)
Identifying critical services
Operating System fingerprinting
Identifying routes using Management Information Base (MIB)
Service fingerprinting
32
2.3. ISSAF
Figure 2.7.: ISSAF pentesting overview (from [12])
33
2. Standard
At the end of this phase, pentesters will have the possibility to confirm or dismiss
some preliminary made hypotheses regarding target systems.
Vulnerability Identification: once specific targets are selected, the attackers
will perform several activities to detect actual exploitable vulnerabilities in the
system. ISSAF lists some of the activities used to detect weak points:
Identify vulnerable services using service banners

Perform vulnerability scans
Perform false positive and false negative verification
Enumerate discovered vulnerabilities
Estimate probable impact
Identify attack paths and scenarios for exploitation
Penetration: this phase is the first actual operation against the target. The attacker will try to circumvent security measures and gain access to the system under
attack. ISSAF methodology emphasizes the following steps within the penetration:
Find proof of concept code/tools to use for the attack

Develop new tools/scripts if needed
Test proof of concept code/tools before use them to avoid problems during
the actual pentesting
Use proof of concept code against target
Verify or disprove the existence of vulnerabilities
Document findings
Gaining Access & Privileges Escalation: in this phase pentesters will confirm
the possibility to access the target. To do that the attacker has to compromise all
the intermediate systems in the path leading to the physical system under attack
(e.g. firewalls, routers, etc.). Moreover, the attacker will usually access the system
with the least privileges possible and he will try to obtain administrators rights
exploiting local vulnerabilities with root-kits.
Enumerating Further: this phase lists malicious activities the attacker can perform within the compromised system. Such activities are:
Obtain encrypted passwords for off-line cracking

Obtain passwords by using sniffing or other techniques
34
2.3. ISSAF
Sniff traffic and analyze it
Gather cookies and use them to exploit sessions and for password attacks
E-mail address gathering
Identifying new routes and networks
Mapping internal networks
Compromise Remote Users/Sites: this phase focuses on the presence of secure
communications (e.g. Virtual Private Networks) between remote users/sites and
enterprises network. In this scenario, the attackers can try to extend his control
to remote users and detached networks. In case of success, the attacker will repeat
all the previous steps in the new environment.
Maintaining Access: in this phase pentesters make sure to control the system
also in the future. As a vulnerability can always be patched by system operators
an attacker can install malicious code in the system to create covert channels and
backdoors. Such verification shows the degree of exposure that a system can have
once compromised. The Maintaining Access phase is often not performed as part
of a penetration test due to the risks involved (e.g. a real attacker capable to take
possession of the covert channel and gain access to the system).
Covering Tracks: this phase ensures attackers to be invisible to later analysis.
Usually, during a penetration testing, attackers should be as open as possible
about their operations (unless the company expressly requests otherwise). There
are several actions that pentesters can perform to hide their traces:
Hide modified files and used tools

Clear logs by checking history and edit log files
Defeat integrity checking
Defeat Anti-viruses
Implement customized Root-kits
(optional) Audit: system audits should be performed after completing a penetration test. This phase can usually bring to light further potential vulnerabilities
within the system.
The Reporting, Clean up and Destroy artifacts phase concludes the assessment. According to ISSAF a minimal reporting should consist of:
35
2. Standard
Verbal Reporting: that describes the feedback the company has to have immediately after the pentesting. The attackers will give to the company an overview
on their finding and they will focus on major critical issues.
Final Reporting: that is an in depth presentation with detailed results of the
tests. Each discovered vulnerability has to be presented with related countermeasures and a proposal on how to implement them. The report should follow a
well documented structure. ISSAF proposes a list of information needed for the
document:
Management Summary
Scope of the project
Tools that have been used
Dates & times of the actual tests on the systems
Outputs of tests performed
A list of all identified vulnerabilities with included recommendations on how
to solve problems found
A list of action points
Finally, all information that is created and/or stored during the test on target systems
should be removed. If this is not possible, such information (e.g. files and directories)
should be mentioned to system operators to proceed to their removal.
This concludes the discussion of existing penetration testing methodologies. In a later
version, we may extend this to other approaches like OWASP, PTES, or CHECK.
36
3. Requirements
3.1. Approach
Within the CRISALIS project we merge the expertise of different stakeholders. Academia,
SCADA vendors, and SCADA operators are three of the main contributors to our assessment methodology for critical infrastructure. However, to further enlarge the information
basis that our approach is built upon, we decided to create a questionnaire about testing
security infrastructure. The purposes of interviewing CI and security experts are several.
There is the need to understand what are the requirements and the expected feedbacks.
Moreover, we also need to validate the benefits of establishing a methodology tuned to
CIs and also evaluate its comprehensiveness and accuracy. Using a questionnaire seemed
the most efficient way to reach a broader community and inquire about the state of the
art of CI testing in general and to identify possible steps forward toward the definition
of the more comprehensive concept of CI vulnerability assessment. The questionnaire
is then extended by more detailed interviews with some stakeholders.
The questionnaire is divided into four sections: General information; Security requirements, design and regulations; Testing and methodologies; and Research.
The General Information section focuses on characteristics of the interviewed persons. Information on area of expertise and experience will be used to evaluate the later
responses and comments.
The security requirements, design and regulations section collects information about
deployed security mechanisms and authentication schemata. Moreover, the section investigates on what limitations there are in deploying security mechanisms in Critical
Infrastructures.
The testing and methodologies section is the core part of the questionnaire. It aims
to identify currently used security practices focusing on: kind of tests performed, test
frequency, and tools used. Furthermore, the section asks about the expectations the
experts have from a CI security methodology.
Finally, the Research section concludes the questionnaire asking about CI security
research status.
We believe that integration of CRISALIS expertise and questionnaire outputs will provide the necessary background from which to build a comprehensive security assessment
37
3. Requirements
methodology for critical infrastructure. Such a methodology is supposed to take into account already existing best-practice provided by industry and different, but relevant, IT
fields. The output has to be compliant with CI experts and operators needs and should
allow them to have a common evaluation framework to compare their infrastructures.
Section 3.2 presents the questionnaire we used to collect information. The questionnaire is also available online at Critical Infrastructure Security & Penetration Testing
[13].
3.2. Questionnaire
Critical Infrastructure Security & Penetration

Testing
Industry/Researcher Questionnaire
University of Twente
December 2012
This questionnaire is anonymous unless you want to participate in the follow up mentioned in section E. The purpose of this work is exploring the need for a standard penetration testing methodology for Critical Infrastructure. With the following questions we
will collect ideas and requirements useful to realize such methodology.
N.B. If you have never worked with Critical Infrastructure please fulfill sections B and
C with your opinion on how these systems are supposed to work and be tested. Questions
marked with * are intended only for Critical Infrastructures operators/deployers.
A. General information
1. In which area are you working?
# SCADA Vendor
# SCADA End User
# SCADA Consultant
38
A. General information
# Security Consultant
# Academia
# Other:
Comments:
2. What is your domain of interest? (Multiple selections are possible)
2 Water
2 Energy
2 Oil & Gas
2 Transportation
2 Other:
Comments:
3. Briefly describe your field of work/research and your job.
4. How much experience do you have in working with Critical Infrastructures (CIs)?
(little) 2222 (a lot)
Comments:
5. How much experience do you have in security?

Comments:
39
3. Requirements
6. How important is computer security in your domain?
Comments:
B. Security requirements, design, and regulations

7. What types of security components do you use/manifacture/deploy?
(Multiple selections are possible)
2 Firewall
2 Application White-listing
2 Intrusion Detection System
2 Intrusion Prevention System

2 Forensic Analysis Tool
2 Other:
Comments:
8. Rank these limitations to security in order of importance from 1 (High
importance) to 6 (Low importance).
Costs are a big factor limiting the amount of research and spending on security.
Time constraints. No time to investigate all potential security issues and
solutions.
Off-the-shelf security technology are not suitable for CIs and their proprietary
software.
The underlying hardware used in most CI cannot handle modern security
features.
Regulations/laws do not allow all security features.
There is no incentive to secure every aspect of CIs.
Other:
Comments:
40
C. Testing and methodologies

9. * How important is the correct functioning of security components in
your system?
(little) #### (a lot)
Comments:
10. * How is the access to the systems information and specification controlled? (Multiple selections are possible)
2 Only those who have a need-to-know can access such information
2 For some specific aspects a very small set of people has access to such information
2 Specific area of the company terrain are only accessible by authorized personel
2 Only the manufacters/suppliers have the full specification of the components
2 Open source software is used
2 Also internally, the security tests can mostly be done as black-box tests
2 Other:
Comments:

11. What kind of security tests are done? (Multiple selections are possible)
2 Stress tests and Denial-of-Service tests
2 Penetration tests
2 Security code review
2 Formale code verification

2 Other:
Comments:
12. How often are these tests performed? (Multiple selections are possible)
2 At design time
41
3. Requirements
2 During development
2 During the setup
2 Regularly during operation

2 When some changes occur
2 Other:
Comments:
13. Are security tests mainly black-box (systems specifications unknown)
or white-box (full disclosure of information to pentesters)?
# Black-box
# White-Box
# Unknown
Comments:
14. * Is it possible to test the system remotely (passing through public

networks)?
# Yes
# No
# Unknown
Comments:
15. * Is the security testing outsourced, i.e. done by externals?

# No, everything is tested in-house.
# No, generally everything is done in-house but in some cases external security
testers are included.
# Yes, but only specific elements are tested by externals.
# Yes, the whole system is tested for security by externals.

# Unknown
# Other:
Comments:
42

16. Is the testing performed within company premises?
# Yes, always.
# If outsourced, it can be performed also outside company premises.

# Unknown
# Other:
Comments:
17. For web servers security/penetration testing methodologies and tools
exist.
Are there any methods/procedures/standards used for testing CI security?
# No, I am not aware of any general procedures/standards.
# No, there are no standards but some methodologies specifically designed for
each CI exist.
# Yes, internally a set of documented procedures exist which are followed for
each CI.
# Yes, published/standardized methods are used for testing wherever possible.
# Unknown
# Other:
Comments:
18. Can you mention some methods used?
19. What are the procedures for security testing CIs? How is the security
tested? (Multiple selections are possible)
2 A security test is done on the whole live system.
43
3. Requirements
2 A security test is done when the system is offline.
2 A security test is done on a backup system.
2 Different components are tested separately
2 Unknown
2 Other:
Comments:
20. The security tests are done by... (Multiple selections are possible)
2 The team working on the component(s).
2 An internal security team.
2 An external security team.

2 Unknown
2 Other:
Comments:
21. Are there any CI-specific tools/software for the testing?
selections are possible)
(Multiple
2 No, I am not aware of any CI-specific tool.
2 Some simple tools are developed when necessary.
2 Yes, existing software tools are used for testing specific components.
2 Yes, we developed our own tools/software for testing the security of the ICS.
2 Unknown
Comments:
22. Referring to previous question, can you mention any of such tools/software?
44
23. Is there a need for a (standard) CI penetration testing methodology?

# No. There is no incentive to do thorough security testing.
# A little. The methodologies/documented procedures will not be used. We do a

flexible security testing strategy that is decided upon for each individual case.
# Yes. General guideline methodologies are good, but there is no need for more
specific methodologies.
# Yes. A CI-specific and comprehensive methodology would make security testing more organized/ repeatable/ comparable.
# Unknown
Comments:
24. This CI penetration testing methodology should... (Multiple selections

are possible)
2 include a guideline for testing the whole CI.
2 include a guideline for testing specific components.
2 include general guidelines about each testing aspect.
2 include very detailed information about each testing aspect.

2 suggest software/tools to be used.
2 state how to document the results.

2 Other:
Comments:
25. Are there any areas/aspects/topics that need special attention?
45
3. Requirements
D. Research
26. Should more research be done in the area of CI security (either at
university or at companies)?
# No, there is no incentive to do thorough security testing.
# No, there is already a lot being done (also in non-CI areas that can be applied
to CI).
# Yes, more time should be spent in studying CIs security.
# Other:
Comments:
27. Additional comments:
E. Follow up
28. Would you agree to be contacted for a more in depth discussion?
# Yes
# No
29. If yes, please provide us your email address:
Thanks for your time. In case you provided your email address, we will contact you to
organize a 1 hour phone interview.
3.3. Further analyses

At the moment of this writing (April 2013) the distribution and filling out of the questionnaire is still in progress. Due to the limited feedbacks we collected until now, it is
46

difficult to generalize the results. However we are starting to see some trends and common believes in the answers. Interviewed people belong to Academia, SCADA vendors
and utilities, consulting companies, etc. This is one of the main reasons why we propose
a two step approach to D5.1 were we provide a preliminary proposal now that is later
extended and validated based on further input. We still provide a preliminary summary
of the found requirements in this section.
As we expected, Critical Infrastructures and Security seem to be two inseparable
concepts. All the participants to the survey agreed on considering computer security
a priority in implementing and managing Critical Infrastructures. However there are
some limitations in deploying security. Most of the interviewed experts identified costs
as the biggest obstacle to properly ensure security. Costs are always a big factor limiting
the amount of research and spending on security. Almost equally important as costs are
time constraints. Several people complain about the limited time available to investigate
all potential security issues and solutions. Ranked a bit lower, Off-the-Shelf security
technologies in combination with unusual hardware used in SCADA seem issues easier
to resolve. On one hand, Off-the-Shelf tools and techniques can be customized and used
on CIs and their proprietary software. On the other hand, CIs underlying software,
even outdated, can handle modern security features. Finally, participants agreed on the
fact that laws and regulations already in place do not impede operators to deploy new
security features to their infrastructures.
It can be deduced that a penetration testing technique and, more in general, an assessment methodology for Critical Infrastructures has to focus on reducing cost and being
time efficient. Standard methodologies have to be reduced to a minimum sequence of
activities in order to save money and time. Automation of testing steps will play an
important role. Penetration testers and auditors have to cut all disposable processes
from their work and maximize the throughput. At the same time, external testers may
have a clearer time and cost budget they can plan with.
According to the results of our interviews there are no preferences in choosing a specific kind of security mechanisms with respect to the others. Firewalls, Application
White-Listing, Intrusion Detection and Prevention Systems and also Forensic Analysis
tools are widely deployed and used in Critical Infrastructures. So a penetration testing
methodology should not focus on one of them but include them all.
The same is true for the different kinds of security test. In the questionnaire we
proposed four types of tests asking which of these are already implemented in Critical
Infrastructures. Our experts confirm that Stress tests and Denial-of-Service tests, Penetration tests, Security code reviews, and Formal code verification are quite known and
used in CI field. For this reason, a security assessment can freely exploit one or more of
them depending on the purpose.
47
3. Requirements
All tests are so far mostly done during CI development and setup. According to
our results, only a small percentage of the survey participants considers a live system
suitable for a penetration test. However, situations in which the system is off-line for
maintenance seem suitable to run specific tests. The reason behind these considerations
relies on the concern that a test may affect the proper functioning of Critical Infrastructure and endanger both physical processes, people, and company business. Therefore,
security assessments must not have an impact on live systems. Every action must be
evaluated beforehand with respect to this risk. In any case, security operators have to be
ready to face problematic or dangerous situations that can be caused by a penetration
test.
Questions about strategies and constraints on penetration testing reveal that there is
no common agreement on the issue. First of all, Critical Infrastructures use both WhiteBox and Black-box penetration testing depending on the purpose of the test. Together,
the two techniques contribute to perform a comprehensive assessment that focuses both
on vulnerabilities and defense reaction capabilities. However, some experts exclude one
or the other for different reasons. Time constraints and potential dangerous impacts are
some of them. Remotely accessing the network is another big issue. Not all the operators
are available to open their systems to Wide Area Networks (WAN) or to the Internet.
On the other hand, some of them agree on passing through public networks to better
simulate real attacks. Finally, some operators agree on outsourcing security testing and
giving access to the system to third parties. These tests are usually performed within
company premises. In any case, outsourced tests cannot substitute internal testing.
However, almost all the interviewed persons agreed that an outsourced test is always
specific for a component or a subsystem but never general and comprehensive of the
whole Critical Infrastructure.
The majority of respondents is not aware of any specific penetration testing methodology for Critical Infrastructure. Some experts suggested that (non-public) customized
methodologies for specific tests exist but we can safely conclude that no comprehensive
security assessment methodology has been developed and published yet. We know from
the results that several Off-the-Shelf tools and software are used to test Critical Infrastructures. Some of the experts provide examples. Metasploit [14], Nessus [1], Achilles
[15, 16], Mu Dynamics products [17] but also standard IT fuzzers and code checking
tools are used in the CI context. In several cases, operators and vendors developed their
own tools or customized standard one for specific purposes.
At the end of the questionnaire we evaluated how a new assessment methodology
specifically tuned on Critical Infrastructure would be received within the community.
Almost all the interviewed agreed on the need for a standard CI penetration testing
methodology. Such an opinion is however divided in two different connotations. On the
48

one hand, half of the experts say that general guidelines methodologies are good, but
there is no need for more specific methodologies. On the other hand, the others ask for a
CI-specific and comprehensive methodology that would make security testing more organized/repeatable/comparable. Again, the requests about what a methodology should do
are various and different among each other. Almost all the responses want the methodology to suggest specific software/tools to be used. The experts also emphasize the need
for a comprehensive evaluation capable to provide general guidelines about each testing
aspect. Moreover, it is important to define how to document testing results in order to
have a common background useful for comparisons. Assessing specific components with
respect to a general evaluation of the infrastructure seems a priority. This is probably
due to the difficulty of analyzing results related to large and, sometimes, geographically
distributed infrastructures. It is worth noting that almost no one asks for very detailed
information about each testing aspect. This is related to the difficulties that a detailed
methodology will have to face with respect to already in place policies and procedures
inside specific Critical Infrastructures.
Thanks to the questionnaire and interview feedback, we have a first insight on what
operators and vendors want from a CI assessment methodology. The sometimes varying
answers reflects the real world situation where there is no one-size-fits-all solution. From
system to system and from company to company specific needs lead to the development
and customization several different testing methods. The idea behind this deliverable is
to collect such different information and create a general methodology based on common elements among already existing methods. At the same time, the new CRISALIS
penetration testing methodology needs to be flexible enough to be adaptable to different
scenarios and environments.
49
4. Methodology
4.1. Overview
In this section we focus on formalizing and merging all the concepts described in the
previous two sections. The output of this analysis will be a new methodology for a
security assessment methodology for Critical Infrastructures. We need to note that the
current proposal should only be seen as a draft and that the concluded survey together
with experiences with applying our approaches to testbeds available in CRISALIS will
lead to a refinement and extension of this methodology in the second version of this
document.
Our work has two different starting points. On one hand we have the inputs provided
by the questionnaire and the interviews. This is a necessary contribution for different
reasons. First of all, experts in the field (e.g. SCADA vendors, utilities, security consultants, etc. ) have provided their expertise that we are building on. Thanks to their
knowledge we were able to identify problems and concerns linked to the application domain and constraints in an industrial environment. Moreover, they allow us tailor or
methodology to their actual needs. The interviewees explained how assessments and
penetration tests are performed today in CIs and what they consider is still missing in
these evaluations. Our CRISALIS penetration testing methodology addresses these issues and provides a generalized approach capable of taking the best of specific assessment
methods already used, unifying differences and overcoming limitations.
On the other hand, we have examined a number of relevant security assessment
methodologies adopted from standard IT. OSSTMM, NIST SP800-115 and ISSAF are
well known and extensively used. However, we see some shortcomings. E.g., several of
the steps may not always be needed in the Critical Infrastructure field. Our idea of a
security assessment methodology for Critical Infrastructure is a process that adopts the
overall structure, the terminology, and certain individual elements or steps from these
standard methodologies. However, at the same time, it leaves aside details regarding IT
elements that are of secondary importance (or are completely missing) in the industrial
field. Even if Industrial Control Systems and standard IT networks are merging more and
more, assessing Critical Infrastructure must include also verifying specific components
and properties of such heterogeneous systems such as PLC. The scope can therefore be
50
4.2. Workflow
narrowed but some additional steps must also be included.
Our work maps the needs identified by the experts on standard security methodologies. We do not want to create a new standard from scratch but rethink the existing
ones. The result will be a lightwheight security assessment methodology involving a
penetration testing methodology tuned to industry components and constraints. The
use of a common schema to evaluate Critical Infrastructure will allow operators, vendors and security consultants to share information and assessments in an easier way and
will make results more meaningful. A uniform approach will allow to easily spot major
problems and threats inside Critical Infrastructure and will eventually contribute to a
general improvement of the security of such systems.
4.2. Workflow
4.2.1. Main Workflow
As introduced in the previous paragraph, we conceptually split our methodology in two
major parts:
General Assessment: it covers all the aspects of a security assessment. It involves three sub-phases: a Pre-Assessment in which a company identifies the goals
of the assessment and solves any bureaucratic and legal issue regarding the following operations; the actual Practical Assessment in which auditors and pentesters
perform practical tests on the assets; and the Post-Assessment phase in which the
company evaluates the results and decides about countermeasures and information
disclosure.
Practical Assessment: it represents the intermediate part of the General Assessment. It focuses on practical tests performed on Critical Infrastructures. The
Practical Assessment is also divided in three different sub-phases: the Preparation
in which pentesters discuss technical details with system operators and prepare
tools and strategies to be used during the test; the Testing where all the tests are
performed; and the Analysis in which the results of the tests are documented and
detailed.
Figure 4.1 shows the methodology workflow.

There are two different contact points between the two methodologys parts. The
former links the Pre-Assessment phase to the Preparation one while the latter links the
Analysis phase to the Post-Assessment one. In both the cases there is a substantial
51
4. Methodology
Figure 4.1.: Critical Infrastructure Methodology
movement of information flowing from a phase to the other. We will denote these two
activities as Information Flows one (1) and two (2).
Finally, there are two possible loops in the workflow. The first feedback is given by the
Analysis phase. At the end of the practical assessment it is possible to restart it without
pass to the Post-Assessment. This situation describes a case in which the requirements
identified in the General Assessment do not match with the actual testing. The second
loop concentrates on the Testing phase and will be described in 4.2.3.
Before discussing each working phase in details, it is worth noting that the reason to
separate the methodology in two different parts is twofold. The first reason concerns the
usability of the methodology. Such schemes allows operators to perform one of the two
parts independently. Let us imagine a situation in which the procedures described for
the Practical Assessment are not compliant with company policies or they are simply
not suitable for companys purposes. In this case, operators can exploit the General
Assessment scheme to perform all the accessory operations and substitute the Practical
Assessment with a different model in line with their expectations. Thanks to the proposed methodology they will have just to take care about the Information Flows and
modify them properly. In the same way, operators can use just the Practical Assessment
with the condition to provide the right input and to modify the output by taking into
52
4.2. Workflow
account requests provided by a different security assessment.
The second reason behind the proposed scheme regards the maintainability. Critical
Infrastructures have changed a lot since the 70s and they are likely going to change even
more in the future. Moreover, legislations and company policies also change over time.
The possibility to substitute a part of the methodology without altering the rest is a
definite advantage in terms of cost and time. In case of such modifications, operators will
have to modify the Information Flows to link the new part to the rest of the methodology.
4.2.2. Workflow phases

In what follows we describe the five phases and the two Information Flows more in
details:
Pre-Assessment: the Pre-Assessment is the starting point of our security evaluation methodology. In this phase the company opens the discussion about the
security assessment and explains its motivations and its goals. This phase should
involve at least three different actors: the Chief Security Officer (CSO) or whoever is representative for this function and may decide to approve the starting of
assessment operations; the Chief Financial Officer (CFO) or whoever is representative for this function and may decide on the budget allocated for the assessment
operations and about acceptable risks; and ICS and SCADA operators that will be
responsible to contribute to the discussion with the technical point of view. The
discussion deals with several topics. First of all, the actors have to decide about
security assessments purposes. The main goal and the motivations behind the
following operations have to be stated clearly. What follows is the identification
of company assets. The actors have to focus on one or more targets according
to the proposed objectives. Critical Infrastructures are complex systems and, for
this reason, it is unlikely to have always a comprehensive assessment. Most of the
time, specific sections or processes of the Critical Infrastructure will be checked and
analyzed. An additional and very important step is a detailed risk analysis that
clearly identifies activities endangering the safety of the system and its environment. This needs to be documented as well as an agreement to either skip those
steps or perform them with specific safety guards. During the Pre-Assessment,
involved actors have to agree on the time plan and decide on how to distribute
resources to the five phases of the methodology. Finally they have to investigate
and ensure that legal and contract obligations regarding assessment operations are
met. This last step overlaps with the Control Assessment proposed by ISSAF and
already discussed in 2.3.1.
53
4. Methodology
Information Flow 1 : at the end of the Pre-Assessment phase its participants
have to prepare a document summarizing all the results of the discussion. The
Information Flow describes the work the operators will do for translating such
outcomes in practical directives for the pentesters. For example, the assets identified in the previous phase will become systems and components to be tested. In
the same way, the main goal will be divided in several sub-goals which will be
mapped into specific tests. Such information will be the input of the Preparation
phase.
Preparation: the Preparation is the first phase of the Practical Assessment. This
phase involves two actors: ICS and SCADA operators that report the outcomes of
the Pre-Assessment phase; and the pentesters which are in charge of performing
necessary tests. The two actors can can be from the same organization if the
purposes or the constraints of the security assessment respectively allow or do not
permit the presence of third parties (e.g. external companies, trained pentesters,
etc.). Operators and pentesters have to discuss several issues. First of all, they will
decide on what kind of pentesting schema to follow. They will decide on the type
of penetration testing as proposed by OSSTMM in 2.1.2 (e.g. Blind testing, Grey
testing, etc.). Moreover, they will choose on the timing of the attack (e.g. working
hours, weekends, etc.). Finally, they will decide whether to attack the live system,
or a backup, or performing all the tests when the system is off-line. The attack
impact is one of the main concerns that has to be faced during the Preparation.
Critical Infrastructures cannot be put in danger by a penetration testing even if
the reason to perform the analysis is verifying that possibility. During this phase
operators should decide and implement specific safeguards to put in place during
tests and prepare an emergency plan with countermeasure to take in case one of
the security tests succeed in penetrating and corrupting the system. This plan
has to be ready also in case of defensive black box testing. In that situation
operators participating to the test will not be aware of the plan unless a problem
occurs. In this phase operators have to verify if all the objectives defined during
the Pre-Assessment are achievable. Otherwise they should motivate any problem
that represents an obstacle to the test. The final step of this phase is the setup of
the environment and of all the resources needed to the pentesting activity.
Testing: the Testing phase implements the security tests that have been discussed
during the Preparation. Its duration depends on different factors such as: accuracy
of the tests (e.g. comprehensive tests vs. vulnerability identification), pentester
skills, allocated budget, etc. During the entire phase pentesters have to log all
54
4.2. Workflow
the performed operations and related discoveries. Such report will be the input
of the Analysis phase. Security operators responsible for the emergency plan will
follow the operations from a different point of view. They will check the presence
of penetration side effects on the Critical Infrastructure that the pentesters are
not aware off. Such information will be integrated to the pentest log at the end
of the Testing phase. It is worth noting that the testing plan arranged in the
Preparation phase can change accordingly to the evolution of the tests. Unexpected
obstacles can interfere with the penetration test and undermine the feasibility of
some operations. Such situations have to be also detailed in the log. In case a
test shows severe vulnerabilities that can endanger the Critical Infrastructure, the
safeguards and emergency plan should be triggered and prevent real damage. The
effect of such an event is the interruption of the methodology workflow and the
immediate feedback to the Chief Security Officer. The level of severity needed
to trigger the emergency plan has to be identified in the Preparation phase. The
testing phase will be extensively discussed in section 4.2.3.
Analysis: the Analysis phase concludes the Practical Assessment. As in the
Preparation phase, pentesters and operators are the two actors involved. The
former will report about the findings of the Testing phase and will propose a
risk rating of the security to the Critical Infrastructure. The latter will critically
review the testing, contextualizing the problems and and justifying any design or
maintenance choices. Moreover, operators have to perform two important tasks:
check if the status of the system has returned to normal after the testing session;
and apply fixes to minor and easy-to-fix problems and vulnerabilities. At the end of
the Testing phase, the pentesters will have to undo permanent modifications to the
system, like removing any file or software installed inside the Critical Infrastructure
or unpluging any machine used for pentesting. During the Analysis, operators will
check the correctness of such activity and will run some integrity checks to verify
that the Infrastructure is brought back to a safe state. It is probable that problems
such as misconfigurations and other simple activities from the penetration induce
some errors into the system. Operators should fix such issues without reporting
them to the next stage of the methodology. In this sense, the Analysis phase is
also a filter identify the more severe security problems and what types of problems
a later security deployment may create.
Information Flow 2 : at the end of the Analysis phase operators and pentesters
will prepare a document that, starting from the report of the Testing phase, summarizes findings and security issues. The format of the documentation can be based
55
4. Methodology
on the the ones proposed by ISSAF in its Final Reporting 2.3.2 and partially also
on the final requirements of the OSSTMM 2.1.2. Some information needed to the
document describing the Analysis phase are:
Date and time of test
Duration of test
Auditors and analyst involved
Tools that have been used
Test type
Scope of test
Comprehensive list of performed tests
Knowledge of which tests have been completed, not completed, or only partially completed, and to what extent
Test error margins
Vulnerability discovered/exploited with recommendations on how to solve
problems found
Unknown anomalies
Operators will have to motivate and justify such documentation during the PostAssessment phase.
Post-Assessment: the Post-Assessment closes our security assessment methodology. With respect to the Pre-Assessment phase the presence of the Chief Financial Officer is no longer needed as any decision regarding new operations will
be taken afterwards. Therefore, in this phase, the discussion involves ICS and
SCADA operators and the Chief Security Officer or whoever represents this role
in the company. Operators will explain the steps performed during the Practical
Assessment. Pentester can also participate to the meeting. Their task will be
to present their findings focusing on major problems encountered. Operators will
also propose countermeasures agreed on the basis of the discussion in the Analysis
phase with the pentesters. The CSO will evaluate the proposed countermeasures
deciding on what actions to take to solve the most pressing issues. It is worth
noting that, during the Post-Assessment phase, participants will also discuss the
possible impact of vulnerability disclosures. The output of this phase is the final
result of the security assessment methodology. Such documentation is both an
high-level overview about companys vulnerabilities, possible mitigations, limitations in deploying security and relation to security legislation or policies on the
56
4.2. Workflow
one hand and on the other hand a low-level overview describing specific problems
and vulnerabilities. The conclusions of the assessment match the list proposed by
the NIST in 2.2.2 (Insufficient patch management, Lack of security baselines, Inadequate incident response procedures, etc.). The output of the Post-Assessment
methodology can be used for following security assessments (e.g. to identify solved
and still in place problems and vulnerabilities) or as a starting point for different
kinds of assessment (e.g. risk assessment introduced in CRISALIS deliverable 2.2).
Figure 4.2 shows a summary of the concepts discussed above. We will now discuss the
penetration testing in more detail.
Figure 4.2.: Critical Infrastructure Methodology Detailed Schema
4.2.3. Penetration Testing Steps

The penetration testing process is the core of the assessment methodology. During
the Testing phase, pentesters go through five operational steps: Information Gathering,
Network Mapping, Vulnerability Identification, Penetration, and Maintaining Control.
57
4. Methodology
Each step groups a set of operations necessary for the next step. We also envision
a circular representation of the testing steps as the penetration testing process is not
necessarily one-time. A successful attack and the consequent control of a part of the
system by the pentesters could open new attack vectors. In this case, pentesters can
traverse again through the five pentesting steps exploiting new targets or attack vectors.
The process ends when all the prepared attack scenarios as well as those discovered
during the activity have been tried and evaluated.
The specifications of the five penetration testing steps are based on the schema proposed by ISSAF in 2.3.2 as we consider this the most appropriate and comprehensive
approach as far as the actual penetration testing is concerned. In what follows we
describe each step in detail:
Information Gathering: the Information Gathering step focuses on systems for
collecting and analyzing information regarding the Critical Infrastructure. This
information concerns several elements such as working personnel, company policies,
used hardware/software, etc.. The Information Gathering involves both technical
and non-technical methods. Depending on the type of testing (Black Box vs.
White Box), pentesters can have some initial information at their disposal that
helps them to understand the environment they are going to attack. It is unlikely
to find useful information about the Critical Infrastructure internal functioning
on the Internet. However, the knowledge of the working staff and their skills and
backgrounds can give some ideas on what pentesters will have to face once the
attack has started. In case of a White Box penetration testing the Information
Gathering is important to quickly identify points that will be most likely vulnerable
and focus on them.
Network Mapping: the Network Mapping step directly follows the Information
Gathering and contributes to acquire more information about the Critical Infrastructure. In this case, pentesters concentrate only on hardware and software. The
main target is to identify what devices are working inside the Critical Infrastructure, identify communication patterns the network topology. Different tools can
be used during this activities (e.g., sniffers, fingerprinters, etc.). Approaches to fingerprint devices and map networks are investigated in WP4 of CRISALIS and are
presented in deliverables 4.1 and 4.2.. Such tools can be highly useful to automate
some work in this step and thus save time and costs. During this step pentesters
will perform the following operations:
Topology discovery and finding live hosts
58
4.2. Workflow
Differentiate from network components (e.g. switch, firewalls, etc. ), ICS
devices (PLC, RTU, etc.) and standard computers (servers, laptops, etc.).
Port and Service scanning
Operating System and Services Fingerprinting
Critical Services Identification
The Network Mapping step will allow pentesters to confirm or discard attack hypotheses formulated during the Preparation phase or the Information Gathering
step.
Vulnerability Identification: once one or more targets are selected, pentesters
will perform a vulnerability scanning. The most commonly used tools for vulnerability identification in Critical Infrastructure are Achilles [15] [16] and Nessus [1].
The list of the activities of this step comprises but is not limited to:
Perform vulnerability scans

Enumerate discovered vulnerabilities
Perform vulnerabilities verification (e.g. trying to eliminate false-positive)
Estimate possible impact of exploiting found vulnerabilities and verify that
attack can be performed safely
Identify suitable attack vectors for exploitation
Penetration: during the Penetration step, pentesters try to gain access and obtain control of Critical Infrastructures components. This is the most dangerous
operation of the Testing phase as the attackers try to subvert normal operations
of the system or a part of it. The Metasploit Framework [14] is one of the most
widely adopted tool to exploit software vulnerabilities. In its professional version, the framework also provides several exploits against Programmable Logic
Controllers (PLC). However, most of the times, pentesters will resort to develop
customized scripts and tools to exploit ICS-specific vulnerabilities. This is the case
of the fuzzers developed within the CRISALIS project and presented in deliverable 5.2. What follows is a list of operations performed by pentesters during the
Penetration step:
Find or develop suitable code/scripts/tools to use for the attack

Test developed code/scripts/tools before using them on the Critical Infrastructure (to avoid problem during pentesting and unwanted effect on the
whole system)
59
4. Methodology
Use code/scripts/tools against the Critical Infrastructure
Verify or disprove the existence of vulnerabilities
Maintaining Control: once in control of a machine, attackers will verify what
information can be retrieved from the machine and if they can use the compromised
computer to find new targets on the network. Pentesters will search for valuable
data like:
Passwords
Running processes
Information regarding the physical process under control
Generic contacts (e.g. emails)
If the compromised machine is at the edge of the network it may also link to
different networks. Pentesters have to evaluate this possibility and, if this is the
case, restart the pentesting process against the new target. Attackers have always
to verify the possibility to maintain access to the machine for future activities
by installing malicious code (e.g., root-kits and backdoors). This guarantees to
maintain control even if the exploited vulnerability is patched. Whether this should
be conducted needs to be decided in the planing phase. Finally, pentesters have
to cover tracks of the intrusion. Such operations are usually performed:
Hiding modified files and tools
Cleaning logs
Restoring integrity checks
At the end of this step the Testing restarts on a new target or ends up letting
pentesters proceed with the Analysis phase.
Figure 4.3 proposes a comprehensive schema of the assessment methodology plus the
chain of the five penetration testing steps.
4.3. Final Remarks

The presented CRISALIS penetration testing approach represent a first draft towards a
security assessment methodology for Critical Infrastructures. To the best of our knowledge, this work is the first attempt to formalize Critical Infrastructure assessments and
pentestings. With our proposal we tried to incorporate requests and advices given by
Critical Infrastructure experts with existing standard methodologies. The result is a
60
4.3. Final Remarks
Figure 4.3.: Critical Infrastructure Methodology plus Penetration Testing Steps
61
4. Methodology
light-weight but comprehensive list of working phases linked by specific information and
documentation exchanges that should lead pentesters through a fast, cost-saving and
effective evaluation of a system.
Our methodology differentiates among two logical parts. The first, called General
Assessment, concentrates on the administrative process that a company must follow
to organize a security assessment. The second, called Practical Assessment, regards
the technical preparation of the evaluation process and the consequent analysis. Finally,
the process shown in 4.2.3 describes the actual penetration testing. In this phase we
discussed the steps pentesters have to follow to check and evaluate a Critical Infrastructure. As described in the introduction of this chapter, the use of separated but connected
logical components improve both usability and maintainability of the methodology.
The proposed methodology is not refined in arbitrary level of detail and yet needs to
be tested in a Critical Infrastructure. This will be done once the survey and interview
results are fully available as this will also provide a detailed understanding of the level
of detail that is actually required.
In the meantime, we have conducted some initial penetration testing experiments in
one of the CRISALIS testbeds where one new vulnerability was found. These tests also
helped to shape and refine this methodology as we were able to identify importance and
feasibility of different steps. It also highlighted that there are insufficient structures to
report newly found vulnerabilities of incidents to manufacturers and users. This should
be addressed in a second version of this deliverable.
62
5. Conclusion
In this deliverable, we have started with motivating the need and benefits for a methodology for security penetration testing for Critical Infrastructures and Industrial Control
Systems. Benefits include time and cost savings for penetration testers, easier comparison of results, and better confidence in completeness and correctness of results. We
analyzed a series of existing methodologies regarding their applicability in the CRISALIS
context. Based also on the preliminary feedback from a survey and interviews we have
conducted, we identified requirements and the need to provide an adapted and revised
approach. While we make a first proposal for such an approach already in this version
of the deliverable, we suggest to provide a second version of the document by the end of
Y2, as this will allow us to finish the survey, use this to provide an updated and refined
methodology, and finally also include results from own penetration testing experiments
in our testbeds.
63
Bibliography
[1] R. Deraison, H. Meer, and C. Van Der Walt. Nessus network auditing. Syngress
Media Incorporated, 2004.
[2] Wikipedia. White-box testing Wikipedia, the free encyclopedia, 2004. [Online;
accessed 14-January-2013].
[3] R. Patton. Software testing. Sams, 2005.
[4] Wikipedia. Black-box testing Wikipedia, the free encyclopedia, 2003. [Online;
[5] Wikipedia. Gray-box testing Wikipedia, the free encyclopedia, 2006. [Online;
[6] M. Prandini and M. Ramilli. Towards a practical and effective security testing
methodology. In Computers and Communications (ISCC), 2010 IEEE Symposium
on, pages 320325. IEEE, 2010.
[7] D.P. Duggan, M. Berg, J. Dillinger, and J. Stamp. Penetration testing of industrial
control systems. Sandia National Laboratories, 2005.
[8] P Herzog. Osstmm 3the open source security testing methodology manual.
barcelona, espa
na: Isecom, 2010.
[9] Karen Scarfone, Murugiah Souppaya, Amanda Cody, and Angela Orebaugh. Nist
special publication 800-115: Technical guide to information security testing and
assessment, 2008.
[10] Creighton T Hager and Scott F MidKiff. An analysis of bluetooth security vulnerabilities. In Wireless Communications and Networking, 2003. WCNC 2003. 2003
IEEE, volume 3, pages 18251831. IEEE, 2003.
[11] Creighton T Hager and Scott F Midkiff. Demonstrating vulnerabilities in bluetooth
security. In Global Telecommunications Conference, 2003. GLOBECOM03. IEEE,
volume 3, pages 14201424. IEEE, 2003.
64
Bibliography
[12] Balwant Rathore, Mark Brunner, Miguel Dilaj, Omar Herrera, Piero Brunati,
Rama K Subramaniam, Subash Raman, and Umesh Chavan. Issaf 0.2.1 - information systems security assessment framework, 2006.
[13] Marco Caselli and Frank Kargl. Critical infrastructure security & penetration testing questionnaire. https://docs.google.com/spreadsheet/viewform?formkey=
dG15cUV5ck9HWlg1eTVuVk51ZnNYb1E6MQ/, 2012.
[14] LLC Metasploit. The metasploit framework, 2007.
[15] Achilles test platform.
http://www.wurldtech.com/product_services/
discover_analyze/achilles_test_platform/.
[16] Achilles test software.
http://www.wurldtech.com/product_services/
discover_analyze/achilles_test_software/.
[17] Mu dynamics software. http://www.mudynamics.com/products/.
65

Security Testing Methodology for Critical Infrastructures

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Security Testing Methodology for Critical Infrastructures

Hochgeladen von

Copyright:

Verfügbare Formate

SEVENTH FRAMEWORK PROGRAMME

Theme SEC-2011.2.5-1 (Cyber attacks against critical infrastructures)

D5.1 Security Testing Methodology

SEVENTH FRAMEWORK PROGRAMME

The CRISALIS Consortium consists of:

4.2.3. Penetration Testing Steps . . . . . . . . . . . . . . . . . . . . . . . 57

1.1. General Classification

1.1. General Classification

SEVENTH FRAMEWORK PROGRAMME

1.3. Testing Critical Infrastructures

SEVENTH FRAMEWORK PROGRAMME

1.3. Testing Critical Infrastructures

OSSTMM provides a list of national and international regulations already proved to

Figure 2.2.: OSSTMM security test types map (from [8])

SEVENTH FRAMEWORK PROGRAMME

SEVENTH FRAMEWORK PROGRAMME

Figure 2.3.: OSSTMM methodology flow (from [8])

SEVENTH FRAMEWORK PROGRAMME

2.2. NIST SP800-115

Overall, OSSTMM provides a very comprehensive framework that addresses a large

2.2. NIST SP800-115

As already discussed, NIST SP800-115 offers an overview of security assessment key

SEVENTH FRAMEWORK PROGRAMME

2.2. NIST SP800-115

SEVENTH FRAMEWORK PROGRAMME

2.2. NIST SP800-115

Figure 2.4.: NIST SP 800-115 penetration testing steps

SEVENTH FRAMEWORK PROGRAMME

2.2. NIST SP800-115

Figure 2.5.: NIST SP 800-115 attack example (from [9])

SEVENTH FRAMEWORK PROGRAMME

2.2. NIST SP800-115

SEVENTH FRAMEWORK PROGRAMME

Figure 2.6.: ISSAF framework overview (from [12])

Control Assessment explores instead the following fields:

SEVENTH FRAMEWORK PROGRAMME

Find live hosts

SEVENTH FRAMEWORK PROGRAMME

Figure 2.7.: ISSAF pentesting overview (from [12])

Identify vulnerable services using service banners

Find proof of concept code/tools to use for the attack

Obtain encrypted passwords for off-line cracking

SEVENTH FRAMEWORK PROGRAMME

Hide modified files and used tools

SEVENTH FRAMEWORK PROGRAMME

Critical Infrastructure Security & Penetration

# SCADA End User

SEVENTH FRAMEWORK PROGRAMME

2 Oil & Gas

5. How much experience do you have in security?

B. Security requirements, design, and regulations

2 Intrusion Detection System

2 Intrusion Prevention System

SEVENTH FRAMEWORK PROGRAMME

C. Testing and methodologies

C. Testing and methodologies

2 Security code review

2 Formale code verification

2 Regularly during operation

14. * Is it possible to test the system remotely (passing through public

15. * Is the security testing outsourced, i.e. done by externals?

# Yes, the whole system is tested for security by externals.

SEVENTH FRAMEWORK PROGRAMME