2012 Tolly Enterprises, LLC

1 of 5 Tolly.com
#212131
October 2012
Commissioned by
Symantec Corp.
Executive Summary
As virtualization-aware endpoint security solutions continue to
evolve, more and more functionality is offloaded from a single VM to
its supporting infrastructure in the form of Virtual Appliances (VA). In
addition to considering the performance impact of this re-
architecting, administrators must also ensure that the protection
offered remains fully-functional, even in virtual environments.
Symantec Corp. commissioned Tolly to evaluate the A/V effectiveness
of its new Symantec Endpoint Protection (SEP) 12.1 within VMware
vSphere 5 virtual environments vs. competitive agentless and agent-
based solutions. Testing focused on evaluating vendor solutions
against a malware source using on-access scanning to determine a
solutions real-time effectiveness in virtual environments.
TEST HIGHLIGHTS
Symantec Endpoint Protection 12.1
Anti-virus Eectiveness in VMware vSphere 5 Virtual Environments
1
Blocked or neutralized 99% of malware threat
samples
Exhibited 0% false positive rate for non-malicious
samples
2
Symantec Endpoint Protection 12.1:
0%
20%
40%
60%
80%
100%
P
e
r
c
e
n
t

o
f

T
h
r
e
a
t
s
Blocked Neutralized Compromised
On-Access Virus Detection
As reported by Solution Consoles
Notes: Windows 7 Professional, 64-bit installation. 870 confirmed malicious samples were used to evaluate effectiveness. OfficeScan used SmartScan method. Tolly created a custom
application to launch and monitor the success or failure of solutions as the machines accessed the malicious URLs. Results reported are best of 5 runs. See test methodology section for
details.
Figure 1 Source: Tolly, September 2012
A B C D
(870 Malware Samples)
A Symantec Endpoint Protection B Trend Micro OfceScan
C Kaspersky Security for Virtualization D McAfee MOVE
Executive Summary (cont)
Tolly engineers set up a script to
automatically run each client against a set of
870 samples which were confirmed
malicious applications. In order to evaluate
the false positive rate, 50 legitimate
applications were included in a separate
corpus.
Tolly used a bank of 20 VMs per solutions,
scripted using AutoITv3 to download and
execute the samples.
McAfee and Kaspersky were tested using
their vShield-enabled configurations, while
Symantec and Trend Micro were tested on
their agent-based platforms, which had
been optimized for virtual environments.
Test Results
On-Access Virus Detection Rate
Throughout the work day, the endpoint
security solution is invoked to scan files and
other registry/RAM contents as they are
accessed.
For this test, Tolly created a custom
application to launch and monitor the
effectiveness of solutions as the cleanly-
booted machines accessed the malicious
URLs.
When a solution defended against the
threat, the threat was deleted upon
download, whereas a neutralized threat was
downloaded but forbidden to run, or
cleaned of its malicious content.
Tolly engineers observed how solutions
performed when they accessed the 870
samples of malware- whether the solutions
under test blocked or neutralized the
threats, or whether the threats were
permitted to run, and thus compromised
the system.
Symantecs detection and protection
against viruses was the highest among
solutions tested. SEP 12.1 blocked 96% of
threats, and neutralized another 3.3%, while
only compromising 0.7%, (6 out of 870). See
Table 1.
Kaspersky also performed well, blocking
92.2% of threats and neutralizing another
3.2%. Kaspersky, however, compromised the
system 5.5 times more than Symantec,
allowing 4.6% of the malicious programs to
run.
Trend Micro blocked only 59.3% of threats,
and neutralized another 0.4%, thus allowing
nearly half of the malicious samples to
compromise the system.
McAfee blocked 79.5% of threats and
neutralized another 0.2%, allowing 20.2%of
malicious samples to compromise the
system. See Figure 1.
False Positive
Though the primary function of any
endpoint security solution is to block
malicious applications, an endpoint security
solution must also be able to determine
legitimate software and allow it to run
uninterrupted. When a system blocks a
legitimate application from running, it
negatively impacts the user experience.
Many organi zati ons wi th custom
applications may face this challenge when
deploying updates or custom software
within their organizations. The ideal scenario
with any endpoint security solution is to
recieve zero false positives and a 100%
malicious detection rate, however, this is not
a realistic goal given the ever-expanding
threat of new malware.
Endpoint security solutions handle
legitimate programs differently. Some
merely issue a warning when faced with a
legitimate application that they believe to
be malicious, while others go a step further
and block the application.
Symantec Endpoint Protection 12.1 Efcacy #212131
2012 Tolly Enterprises, LLC
Page 2 of 5 Tolly.com
Symantec
Corporation
Symantec
Endpoint
Protection
12.1
Endpoint
Security for
Virtualization
Ecacy
Tested
September
2012
Defended Neutralized Compromised Total
Symantec Endpoint
Protection
835 29 6 870
Trend Micro OfficeScan 515 5 350 870
Kaspersky Security for
Virtualization
802 28 40 870
McAfee MOVE Agentless 692 2 176 870
On-Access Virus Detection Rate of Endpoint Protection Solutions
Using 870 Samples of Conrmed Malware
Table 1 Source: Tolly, September 2012
When the endpoint security solution
prohibits legitimate software from running,
a false positive is generated.
Tolly engineers used a corpus of 50
legitimate software samples from CNET to
determine which solutions under test
permitted their use.
Symantec Endpoint Protection 12.1 allowed
all legitimate samples and generated no
false positives
McAfee, Kaspersky and Trend Micro also
generated no false positives.
Test Setup
All clients were deployed from a clean
Windows 7 64-bit with 1vCPU and 2GB RAM
fully updated as of August 13, 2012. Each
solution was installed per each vendors
best practices on a template VM, with
signatures updated to August 13, 2012 (See
Table 2). Using VMware View 5, these
images were then deployed into a non-
persistent 20-VM linked clone pool each.
Tolly engineers prepared a script which
emulated a user download of a particular
file. During the setup phase, malware and
clean samples were aggregated from
multiple sources to form the test corpus.
All files (clean and malicious) were hosted
locally to provide an identical environment
between different solutions, as each
malware was delivered via the same static
sample.
Test Methodology
Multiple different scripts were running in the
environment for the duration of the test,
which were all prepared by Tolly. The goal of
the workload was to enable each client VM
to boot up from a fresh (clean) image,
snapshot its file system and running
processes, and proceed to download and
execute a sample from the file server.
The director script kept track of which
samples had already been downloaded for a
particular iteration and test corpus. Each
time a VM booted up, it checked in with this
application and was assigned a file to
download.
The workload script was created using
AutoI T, l everagi ng the I E_Create
functionality to download the samples.
Using process and file system snapshots
taken immediately after the download, the
script determines whether or not the file has
been successfully downloaded to the
system. Then, if the file exists, the script
attempts to execute it, followed by another
file and process snapshot to determine if the
file was allowed to run.
Symantec Endpoint Protection 12.1 Efcacy #212131
2012 Tolly Enterprises, LLC
Page 3 of 5 Tolly.com
Table 2
Vendor Product Components Implementation
Symantec Corp.
Endpoint Protection
12.1
Symantec Endpoint Protection Manager 12.1.1959.1959;
Symantec Shared Insight Cache 12.1.1959.1959
Endpoint client with Shared Insight
Cache for on-demand scan
optimization
Trend Micro, Inc. OceScan 10.6
OceScan 10.6.1062
VDI plug-in
Endpoint client with VDI plug-in for
on-demand scan optimization
Kaspersky Lab
Kaspersky Security
for Virtualization 1.1
Kaspersky Security Center 9.2.69
Kaspersky Security for Virtualization (ksv appliance) 1.1.0.54
Single virtual appliance. Agentless
client communicates via VMware
vShield API
McAfee, Inc. MOVE Agentless 2.5
McAfee ePolicy Orchestrator 4.6.2 (Build: 234)
[McAfee move-sva: McAfee MOVE AV Agentless 2.5.0.228
McAfee VirusScan Enterprise for Linux 1.7.0
McAfee Agent for Linux 4.6.0.2156]
Single virtual appliance. Agentless
client communicates via VMware
vShield API
Systems Under Test
Source: Tolly, September 2012
Using this data, the script determines at
what stage each of the samples was allowed
and where it was detected and defended by
the software, reporting this information to
the director.
After a full run-through of the client
workload, the client writes all its data back
to a shared file server, and performs a logoff.
On the View Composer, the last script
performs a client disk refresh as each
workload finishes, deleting the VMstate and
cloning out a fresh base disk from which to
boot for the next iteration.
Each s ampl e i t er at i on r equi r ed
approximately ten minutes, with all VMs
randomized to avoid excessive resource
consumption. The director logged over
6,000 boots per solution over the one week
duration of the test. Each sample was run
through at least four different VMs to ensure
accuracy.
Symantec Endpoint Protection 12.1 Efcacy #212131
2012 Tolly Enterprises, LLC
Page 4 of 5 Tolly.com
TOLLY A/V PERFORMANCE
TEST HIGHLIGHTS
1
Lowest single-VM disk I/O and memory demand for on-demand scan with
fast per-machine run time
Demonstrates avoidance of anti-virus stormsthrough implementation of
randomization algorithm for resource-intensive functions
2
Symantec Endpoint Protection 12.1:
See Tolly Report #212130 : Symantec Endpoint Protection 12.1
Competitive Anti-virus Performance in VMware vSphere 5 Virtual Environments
for complete test findings by scanning the QR code.
Source: Tolly Report #212130 October 2012
Figure 2 Source: Tolly, September 2012
Test Bed Diagram
Terms of Usage
This document is provided, free-of-charge, to help you understand whether a given product, technology or service merits additional
investigation for your particular needs. Any decision to purchase a product must be based on your own assessment of suitability
based on your needs. The document should never be used as a substitute for advice from a qualified IT or business professional. This
evaluation was focused on illustrating specific features and/or performance of the product(s) and was conducted under controlled,
laboratory conditions. Certain tests may have been tailored to reflect performance under ideal conditions; performance may vary
under real-world conditions. Users should run tests based on their own real-world scenarios to validate performance for their own
networks.
Reasonable efforts were made to ensure the accuracy of the data contained herein but errors and/or oversights can occur. The test/
audit documented herein may also rely on various test tools the accuracy of which is beyond our control. Furthermore, the
document relies on certain representations by the sponsor that are beyond our control to verify. Among these is that the software/
hardware tested is production or production track and is, or will be, available in equivalent or better form to commercial customers.
Accordingly, this document is provided "as is", and Tolly Enterprises, LLC (Tolly) gives no warranty, representation or undertaking,
whether express or implied, and accepts no legal responsibility, whether direct or indirect, for the accuracy, completeness, usefulness
or suitability of any information contained herein. By reviewing this document, you agree that your use of any information contained
herein is at your own risk, and you accept all risks and responsibility for losses, damages, costs and other consequences resulting
directly or indirectly from any information or material available on it. Tolly is not responsible for, and you agree to hold Tolly and its
related affiliates harmless from any loss, harm, injury or damage resulting from or arising out of your use of or reliance on any of the
information provided herein.
Tolly makes no claim as to whether any product or company described herein is suitable for investment. You should obtain your own
independent professional advice, whether legal, accounting or otherwise, before proceeding with any investment or project related
to any information, products or companies described herein. When foreign translations exist, the English document is considered
authoritative. To assure accuracy, only use documents downloaded directly from Tolly.com. No part of any document may be
reproduced, in whole or in part, without the specific written permission of Tolly. All trademarks used in the document are owned by
their respective owners. You agree not to use any trademark in or as the whole or part of your own trademarks in connection with
any activities, products or services which are not ours, or in a manner which may be confusing, misleading or deceptive or in a
manner that disparages us or our information, projects or developments.
About Tolly
The Tolly Group companies have been
delivering world-class IT services for
more than 20 years. Tolly is a leading
global provider of third-party
validation services for vendors of IT
products, components and services.
You can reach the company by email at
sales@tolly.com, or by telephone at
+1 561.391.5610.
Visit Tolly on the Internet at:
http://www.tolly.com
Interaction with Competitors
In accordance with our process for conducting comparative tests, Tolly contacted the
competing vendor, inviting them to review test methodology and their results prior to
publication. Trend Micro and McAfee did not respond to our request. Kaspersky Lab
responded to the invitation, reviewed the proposed test methodology and confirmed the
test results prior to publication.
For more information on the Tolly Fair Testing Charter, visit:
http://www.tolly.com/FTC.aspx
Symantec Endpoint Protection 12.1 Efcacy #212131
2012 Tolly Enterprises, LLC
Page 5 of 5 Tolly.com
212131-tb-12-mts-yy - 2012-Oct-26-VerJ