TEST METHODOLOGY

Advanced Endpoint Protection (AEP)

OCTOBER 19, 2018

v3.0
 NSS Labs NSS Labs Advanced Endpoint Protection Test Methodology v3.0

Table of Contents
1 Introduction .................................................................................................................... 3
1.1 The Need for Advanced Endpoint Protection ................................................................................. 3
1.2 Advanced Endpoint Protection Definition...................................................................................... 3
1.3 About This Test Methodology ....................................................................................................... 4
1.4 Inclusion Criteria ........................................................................................................................... 4
1.5 Deployment .................................................................................................................................. 4

2 Test Components ............................................................................................................ 5

2.1 Security Effectiveness ................................................................................................................... 5
2.1.1 Socially Engineered Malware ..................................................................................................................... 5
2.1.2 Exploits ....................................................................................................................................................... 6
2.1.3 Blended Threats ......................................................................................................................................... 6
2.1.4 Evasions ..................................................................................................................................................... 6
2.1.5 Threats Introduced via Offline Method ...................................................................................................... 6
2.1.6 Unknown Threats ....................................................................................................................................... 6
2.1.7 False Positives ............................................................................................................................................ 6
2.2 Total Cost of Ownership ................................................................................................................ 7
2.3 Threat Event Reporting ................................................................................................................. 7
2.4 Anti-Tampering Protection ............................................................................................................ 7
2.4.1 Service tampering ...................................................................................................................................... 7
2.4.2 Log Tampering ........................................................................................................................................... 8
2.4.3 DLL Hijacking.............................................................................................................................................. 8

3 Product Guidance ............................................................................................................ 9

3.1 Recommended.............................................................................................................................. 9
3.2 Security Recommended ................................................................................................................ 9
3.3 Neutral ......................................................................................................................................... 9
3.4 Caution ......................................................................................................................................... 9

Appendix: Change Log ......................................................................................................... 10

Contact information ............................................................................................................ 11

Advanced Endpoint Protection Test Methodology v3.0_101918

2
 NSS Labs NSS Labs Advanced Endpoint Protection Test Methodology v3.0

1 Introduction
1.1 The Need for Advanced Endpoint Protection
Cybercriminals are becoming ever more adept at technical and social engineering, and contemporary threat actors
are capable of carrying out sophisticated attacks that consistently breach modern network defenses. Oftentimes,
these breaches lead to end user systems being infected and subsequently used as a stage for further compromise.
Strong anti-threat protection technologies on the endpoint are the best chance enterprises have at defeating these
many incursions, but current products and techniques are generally unable to stop even the least capable of the
advanced threats that threaten the enterprise, let alone the truly determined advanced persistent threat.
In addition to the need for innovative protection technologies as a part of a thorough defense-in-depth strategy,
the superior product must offer comprehensive monitoring and threat visibility. Why blindly block an infection or
exploitation attempt when identifying metadata and incursion telemetry could mean the difference between the
threat continuing to attack until it succeeds and permanently stopping it?
Today’s enterprise requires both superior protection and robust, pervasive threat monitoring capabilities in its
security posture. This is the purview of advanced endpoint protection.

1.2 Advanced Endpoint Protection Definition

Advanced endpoint protection (AEP) refers to endpoint security technology that combines the protective
capabilities of anti-threat products with the detection, investigation, and prevention capabilities of endpoint
monitoring products.

An AEP product is one which provides automatic threat prevention and threat event reporting capabilities to every
endpoint system it protects. The following features are fundamental to an AEP product:
• Continuous monitoring of endpoint
• Detection of anomalous activity
• Prevention of anomalous activity
• Forensic detail to support basic incident response needs
o What was detected or blocked?
o What indicator(s) led to determination/conviction of block?
In order to detect different types of malicious or threatening behaviors, an AEP product may utilize multiple
approaches and technologies. Examples of such approaches include process monitoring, detecting communication
with potentially malicious hosts and lateral movement to other machines, as well as auditing the file system and
registry.
The level of detail provided within threat event reporting must provide threat forensic teams with the information
they need to investigate suspicious activity. Examples include, but are not limited to, source of infection, file
metadata, lateral infection information, pre- and post-infection data, time of retrieval, and endpoint query. Note
that presuming a certain level of responder capability for the purpose of a test conducted according to this
methodology is beyond the scope of this document.

Advanced Endpoint Protection Test Methodology v3.0_101918

3
 NSS Labs NSS Labs Advanced Endpoint Protection Test Methodology v3.0

Protection and provision of threat event forensic detail are the primary features of AEP products. While additional
functionality may be available that enhances the overall security of the endpoint system, neither protection nor
forensic monitoring may be removed without declassifying the AEP system.

1.3 About This Test Methodology

This document establishes a methodology for evaluating a security product that furnishes protection for some
number of endpoint systems on an enterprise network and that meets the criteria for classification as an AEP
product.
The output of a test conducted according to the methodology outlined in this document should reflect the
effectiveness of the product in its goal of securing an organization.
In order to thoroughly evaluate a product under test, several factors, attributes, and performance metrics are
gathered and appraised. The scope of this methodology includes:

• Security effectiveness
• Total cost of ownership (TCO)
• Secure communication
• Threat event reporting

1.4 Inclusion Criteria

NSS Labs welcomes the participation of any vendor whose product meets the minimum platform definition as
written above. No product may consist of any appliance or sub-solution, physical or virtualized, beyond the
endpoint agent/management station architecture.

1.5 Deployment
An AEP product is often deployed as an agent on the endpoint that reports to a central management apparatus.
The management apparatus normally resides on either a physical appliance, a virtualized appliance, or in the
cloud.
Products will be deployed during testing in a manner that reflects the “enterprise standard,” or “default”
protection policy.

NSS must approve and validate the configuration of products prior to their being tested. Records and configuration
backups (if available) will be taken to ensure consistency across the testing process. Any approved changes made
to default/out-of-box configurations may be noted in the relevant test reports.
NSS Labs test methodologies are continually evolving in response to feedback. If you would like to provide input,
please contact advisors@nsslabs.com. For a list of changes, please reference the Change Log in the Appendix.

Advanced Endpoint Protection Test Methodology v3.0_101918

4
 NSS Labs NSS Labs Advanced Endpoint Protection Test Methodology v3.0

2 Test Components
Several components inform the recommendation of an AEP product, including a product’s security effectiveness,
TCO, and threat event reporting. Each of these are explained below.

2.1 Security Effectiveness

In this portion of the test, a product is assessed for its ability to provide continuous protection to secured endpoint
systems. Diverse threat scenarios are played out against endpoint systems designated as victims. The product
under test must identify malicious behaviors and modifications to the endpoint machine and must work to prevent
the evaded threat from damaging the endpoint.
One of the most common threats to the enterprise is the infection of enterprise systems by malicious software.
Products are tested against threats from the following categories:

• Socially engineered malware

• Exploits
• Blended threats
• Evasions
• Offline threats
• Unknown threats
• False positives
• Any combination of the above in addition to follow-on threat actions or behaviors
Where feasible, these threats will be employed as “real world” in the test environment (i.e., using actual URLs, IPs).
Furthermore, the threats will be introduced into host systems in a “real-world” manner (i.e., the product under
test will not be disabled to get the malware on the host).
The effects of threats from each of these categories against the endpoint systems is measured. A combination of
publicly available tools and proprietary software is used. The endpoint is monitored at various stages of the attack,
as is the behavior of the threat.
In addition to detection of and protection against threats, a product should log relevant threat data about any
detection or block action (see Section 2.3 for additional information).

2.1.1 Socially Engineered Malware

Socially engineered malware can infect an endpoint using numerous attack vectors or delivery methods. Examples
include:

• Binary attachments sent via email

• Executable downloads from websites

NSS evaluates the ability of the product under test to prevent malware from maliciously compromising the victim
endpoint system by exposing a set of endpoints to a continuous stream of live malware sourced from the wild.

Advanced Endpoint Protection Test Methodology v3.0_101918

5
 NSS Labs NSS Labs Advanced Endpoint Protection Test Methodology v3.0

2.1.2 Exploits

An exploit is an attack against a computer that takes advantage of a vulnerability in some part of the system, such
as a logical flaw in a program installed on the machine. Exploits often do not require user intervention or
knowledge.
This test attempts to exploit certain vulnerabilities on a victim endpoint system. Various vulnerable applications
and system features are used, and multiple payloads are employed in conjunction with them.
If a victim is successfully exploited, additional malicious activity may be attempted. Examples of such post-
exploitation activity include data exfiltration, lateral movement, and pivoting.

2.1.3 Blended Threats

Blended threats combine multiple delivery vectors in their attempts to compromise an endpoint system. For
example, instead of relying on one malicious email or a single exploit attempt, the blended threat will leverage
exploitation of multiple vulnerabilities, spear phishing, infected peripherals, and sophisticated antivirus evasion
techniques to infect the endpoint system.

2.1.4 Evasions

There are many ways to defeat the detection and prevention capabilities of endpoint protection products. This
methodology stipulates that all features of security effectiveness testing employ some form of evasion method,
including but not limited to:

• Executable binary packing

• File compression
• In-memory execution
• Malware analysis and awareness
• Code morphism
• Web socket abuse

2.1.5 Threats Introduced via Offline Method

This test evaluates a product’s ability to detect or prevent threats when the endpoint is offline/disconnected from
the network.

2.1.6 Unknown Threats

This test evaluates a product’s ability to detect or prevent previously unknown threats, i.e., threats that should not
have been previously encountered by the product.

2.1.7 False Positives

An AEP product must be able to correctly identify and permit non-malicious activity on an endpoint system if it is
to successfully protect against attack and subsequent compromise. When a product incorrectly identifies benign
data or programs on an endpoint as threats to the system and network, this reduces the enterprise’s ability to
defend against actual threats. Any content or activity on an endpoint system that the product incorrectly perceives
as malicious is considered a false positive.

Advanced Endpoint Protection Test Methodology v3.0_101918

6
 NSS Labs NSS Labs Advanced Endpoint Protection Test Methodology v3.0

Attempts to generate false-positive reports involve the use of benign software or data on the endpoint system by a
simulated user.

File types that are commonly seen throughout the enterprise are examined in this portion of the test, including but
not limited to:

• Portable executable (PE32) files

• Document files (.docx, .pdf, etc.)
• Scripts

2.2 Total Cost of Ownership

Total cost of ownership, or TCO, refers to any data that can be used to quantify a dollar figure, which represents
the expected costs of utilizing a product for endpoint defense. The scope of these metrics includes, but may not be
limited to:

• Product purchase costs

• Product maintenance and update costs
• Installation costs
• Threat-associated costs
• Time to access key data points that support a threat conviction or malicious event
• Usability
• False positives

2.3 Threat Event Reporting

After a product has detected or blocked malicious or anomalous behavior, it should support follow-on
investigation, incident response, and/or remediation efforts. The product should convey detailed and timely threat
event metadata to its central management system. Metadata should be paired with the basic threat event data to
support the detection or block conviction. This also enables responders to act more quickly or establish incident
severity more readily. Decisions on which attributes to assess will depend on the threat types or on the techniques
used (see section 2.1) but may include information such as source and destination IPs, file hashes, file system
paths, registry information, shell commands, command-and-control or outbound connections, and convicted
blacklist IP matches. The product must also be able to correctly separate irrelevant data and system noise from a
threat’s activities.

2.4 Anti-Tampering Protection

This test assesses what protection measures are in place to prevent tampering with key capabilities or with data
that supports the AEP product itself. These protection measures are critical in instances where more advanced
variants are able to “cover their tracks,” or where there is an insider threat.

2.4.1 Service tampering

The AEP product should be capable of preventing users from disabling inspection or scanning capabilities, logging
services, or other core product functionality in order to avoid or impair investigation actions.

Advanced Endpoint Protection Test Methodology v3.0_101918

7
 NSS Labs NSS Labs Advanced Endpoint Protection Test Methodology v3.0

2.4.2 Log Tampering

The AEP product should be capable of preserving logs of detection and prevention activity on the endpoint.

2.4.3 DLL Hijacking

The AEP product should have hardened libraries and execution precedence to ensure no DLL hijacking can occur by
a malicious process.

Advanced Endpoint Protection Test Methodology v3.0_101918

8
 NSS Labs NSS Labs Advanced Endpoint Protection Test Methodology v3.0

3 Product Guidance
NSS issues summary product guidance based on the evaluation criteria that is important to information security
professionals. These criteria include:

• Security effectiveness
• Threat event reporting
• Total cost of ownership
Each product will be given one of the following guidance ratings:

3.1 Recommended
A Recommended rating from NSS indicates that a product has performed well and deserves strong consideration.
Only the top technical products earn a Recommended rating from NSS—regardless of market share, company size,
or brand recognition.

3.2 Security Recommended

A Security Recommended rating from NSS indicates that a product has exhibited exemplary security effectiveness
throughout the testing cycle. However, the product’s costs of ownership are greater than average, making
adoption of the product into an enterprise a costly proposition.

3.3 Neutral
A Neutral rating from NSS indicates that a product has performed reasonably well and should continue to be used
if it is the incumbent within an organization. Products that earn a Neutral rating from NSS deserve consideration
during the purchasing process.

3.4 Caution
A Caution rating from NSS indicates that a product has performed poorly. Organizations using one of these
products should review their security posture and other threat mitigation factors, including possible alternative
configurations and replacement. Products that earn a Caution rating from NSS should not be shortlisted or
renewed.

Advanced Endpoint Protection Test Methodology v3.0_101918

9
 NSS Labs NSS Labs Advanced Endpoint Protection Test Methodology v3.0

Appendix: Change Log

V2.9: August 17, 2018
• Removed duplication between section 1.6 and section 2
• Added section 2.3 to describe how Threat Event Reporting will be measured
• Removed the concept of Overall Prevention Rate (OPR)
• Removed secure communications test cases
• Numerous minor edits for readability
• Minor clarifications made to text in Section 1.5: Deployment
• Minor clarifications made to text in Section 2.1: Security Effectiveness, Section 2.1.5: Threats introduced via
Offline Method, and Section 2.1.7: False Positives
• Section 2.1.6 added: Unknown Threats
• Section 2.4 added: Anti-Tampering Protection

Advanced Endpoint Protection Test Methodology v3.0_101918

10
 NSS Labs NSS Labs Advanced Endpoint Protection Test Methodology v3.0

Contact information
NSS Labs, Inc.
3711 South Mopac Expressway
Building 1, Suite 400
Austin, Texas, 78746-8022
USA
info@nsslabs.com
www.nsslabs.com

This and other related documents available at: www.nsslabs.com. To receive a licensed copy or report misuse,
please contact NSS Labs.

© 2018 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, copied/scanned, stored on a retrieval
system, e-mailed or otherwise disseminated or transmitted without the express written consent of NSS Labs, Inc. (“us” or “we”).
Please read the disclaimer in this box because it contains important information that binds you. If you do not agree to these
conditions, you should not read the rest of this report but should instead return the report immediately to us. “You” or “your”
means the person who accesses this report and any entity on whose behalf he/she has obtained this report.
1. The information in this report is subject to change by us without notice, and we disclaim any obligation to update it.
2. The information in this report is believed by us to be accurate and reliable at the time of publication, but is not guaranteed. All
use of and reliance on this report are at your sole risk. We are not liable or responsible for any damages, losses, or expenses of
any nature whatsoever arising from any error or omission in this report.
3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY US. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT, ARE HEREBY DISCLAIMED AND EXCLUDED
BY US. IN NO EVENT SHALL WE BE LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, PUNITIVE, EXEMPLARY, OR INDIRECT
DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE
POSSIBILITY THEREOF.
4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software)
tested or the hardware and/or software used in testing the products. The testing does not guarantee that there are no errors or
defects in the products or that the products will meet your expectations, requirements, needs, or specifications, or that they will
operate without interruption.
5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in
this report.
6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their
respective owners.

Advanced Endpoint Protection Test Methodology v3.0_101918

11