White Paper

Retail Best Practices

Performance Testing Retail Web
and Mobile Properties
Table of Contents

Executive Summary..................................................................................................... 3

1. eCommerce Characteristics.................................................................................. 4

2. What is an Effective Test Strategy......................................................................... 5

3. Do I Have a Successful eCommerce Site Yet?..................................................... 5

4. Testing for Success................................................................................................ 6

5. Testing Tactics...................................................................................................... 10

6. Retail Examples.................................................................................................... 12

7. Security................................................................................................................. 14

Appendix: CloudTest Performance Data Q&A......................................................... 16

About SOASTA, Inc................................................................................................... 17

SOASTA RETAIL BEST PRACTICES WHITE PAPER | 2

Retail Best Practices:
Performance Testing

Six of the top 10 US retailers trust SOASTA to execute

load and performance tests on their eCommerce sites.

Retailers competing in the high-pressure world of eCommerce have many challenges: not only do
they need to provide an interesting and compelling shopping experience; they need to make that
experience fast and reliable. Shoppers have proven to be discriminating and can easily move from
one online store to another.
A companys online presence has become as important as a physical store, and for some the online
store is their only sales channel.
Many of those responsible for the performance of retail web sites have come to realize that external
testing against the production infrastructure is the best way to get a true picture of capacity and
performance in the real world. Testing in production is the only way to ensure that online applica-
tions will perform as expected. Of course, this does not obviate the need or eliminate the benefits
of testing in a lab environment as well; and its important to have continuity between the two.
So, how can a company build confidence that their online store will provide a great shopping expe-
rience and withstand the traffic spikes caused by seasonal upswings, events and promotions? How
should a retailer measure success?

SOASTA RETAIL BEST PRACTICES WHITE PAPER | 3

 Third-party content can add functionality or information to
a site, but it also adds a layer of traffic and complexity that
you may or may not control.

1. eCommerce Characteristics
Most retailers have brick and mortar stores to maintain, staff, and ensure are run profitably, in ad-
dition to an online site to complement their physical presence. The look and feel of the online site
typically mirrors the target market of their retail stores: discount, high-end womans shop, books
and gifts, big box electronics, etc.
As a result, balancing performance with the rich content often used to capture a specific design
goal can be a struggle. Marketing departments spend millions of dollars promoting events in-
tended to drive users to the site. Web developers build compelling sites to enable a visitor to easily
navigate between departments and effectively sift through a range of size, color, style and other
product attributes. While these content-rich sites are visually attractive and more easily understood
(hopefully!), they can also cause performance issues as traffic grows.
Third-party content is quite common on eCommerce sites, including ads, cross-promotions, and
relevant consumer information. This content can add functionality or information to a site, but it also
adds a layer of traffic and complexity that you may or may not control. Understanding the impact of
this content is critical to managing the overall performance of the site. And, of course, the goal of
any eCommerce site is to sell products. Back-end and third-party systems for payment processing,
authorization, inventory checking and more can also add performance challenges.
Retailers also know that site performance directly relates to revenue. Aberdeen Group found that
50% of companies lost revenue opportunities because of poorly performing applications and 32%
experienced damage to their brand reputation. With most users more than willing to share their ex-
perience on Facebook or Twitter, the news of a poorly performing site can spread around the globe
in minutes.
Without a doubt, much is riding on the performance of an eCommerce site. A retailer can position
themselves for success by following SOASTAs proven Load and Performance Methodology, which
is described on page 5.

SOASTA RETAIL BEST PRACTICES WHITE PAPER | 4

 2. What is an Effective eCommerce Test Strategy?
Creating a successful strategy is key to a successful performance engagement. It begins with
understanding how your users interact with your eCommerce site. For example, the following
information is core to defining the test engagement strategy:
Main business processes (flows) followed by site visitors
Average time a user stays on the site
Average percentage of users that complete a purchase vs. users who browse only
Abandonment ratios and where in the process users abandon
Average and peak number of concurrent users/hour
Average and peak page views per minute/hour
Average and peak orders placed per minute/hour
Traffic pattern differences for the above metrics for specific events such as Black Friday or Cyber
Monday
Geographic regions of the site traffics source
Percentage of traffic that comes from mobile devices, the types of devices, and the differences in
user flows and the impact on the above metrics
If a CDN is used, percentage of content that is served from the CDN
This information helps to define a successful performance testing strategy and drives how your test
will be architected and executed.

3. Do I Have a Successful eCommerce Site Yet?

IF A RETAILER WANTS TO BE One of the toughest questions to answer is Is the site ready? That question becomes infinitely
IN THE TOP 25 CATCHPOINT easier when you have a solid strategy in place. eCommerce sites have specific key performance
RANK THEY NEED TO HAVE indicators (KPIs) that help a retailer measure their sites performance:
PAGE LOAD TIMES OF TWO
Orders per Minute for an eCommerce site, orders completed in a period of time is the gold
SECONDS OR LESS.
standard KPI. Orders directly translate to revenue. SOASTAs CloudTest platform can track
exactly how many orders are successfully completed each minute (or over any period of time) as
well as how many had errors.
Page Views per Hour or Minute Customers can only complete orders if they can effectively
use the site. Ensuring that the eCommerce site can serve web content as users search, compare
and interact is critical.
Sessions per Hour Sessions are individual users or systems interacting with the site. Ensuring
that those users can establish and maintain a session for the duration of their interaction is very
important.
Errors It may seem obvious, but keeping track of both the overall error rate and the type of
errors you are receiving are critical. Not all errors are created equally.
Average Response Time Understanding how long, on average, pages and page assets take to
be served is important for uncovering potential bottlenecks. Its also the basis for many custom-
ers measuring the utility of an eCommerce site. SOASTA CloudTest shows the Average Response
Time for an entire test scenario, transactions, individual pages, and (unlike other performance
applications) provides information about each individual asset that makes up a page. It even
provides the ability to split out Average Response Times by domain so its easy to see if a third-
party asset, a CDN or specific data center is impacting performance.
90th Percentile Response Time This provides an even finer level of granularity when it comes
to response time. 90th Percentile removes the slowest 10% response times. This eliminates
timeouts (which can average 120 seconds) while giving a true indication of the response for 90%
of the users.
SOASTA recommends that a retailer striving for good site performance have an Average Response
Time of less than 3 seconds with a 90th Percentile of less than 2.75 seconds. A retailer wanting
to ensure their site is among the leaders in performance needs to know how fast is fast enough.
Catchpoint Systems ranks the site performance of the top 50 US retailers each year. If we consider
their rankings for 2011, we note that if a retailer wants to be in the top 25 they need to have page
load times of 2 seconds or less on Cyber Monday one of the heaviest traffic days of the year.

SOASTA RETAIL BEST PRACTICES WHITE PAPER | 5

 This indicates that even an additional second of delay can put you at the back of the pack. It is no
accident that six of the top 10 US retailers have worked with SOASTA to ensure they have world-
class performance every single day.
Its worth noting there are different expectations for different types of pages. A user browsing the
site will expect lightning fast response times while the initiation of a video or the final step in a
checkout process is generally expected to take a bit longer. If an eCommerce site can maintain a
target level of performance while testing with 150-200% of expected peak load, a retailer can go
into any marketing event or holiday season with increased confidence.

3.1 What to Watch for When Testing

Understanding what makes an eCommerce site fast or slow helps to focus the testing effort. Many
eCommerce sites are quite complex and made up of several different components and application
tiers. Its important to understand each of these components and how they interact with each other.
Some of the common areas to focus on while testing are:
Application Issues There is no such thing as perfect code. Watch for inefficient code, synchroni-
zation issues, garbage collection, memory leaks, and application dead locks.
Database Performance This is the core of performance. Watch for locking and contention, miss-
ing indexes, inefficient queries, memory management, connection management and unmanaged
growth of data.
Configuration Settings The default settings are rarely optimal. Check for differences between
environments and understand the tuning options and best practices for the various devices in the
architecture.
Load Balancing Use hardware efficiently. Look for algorithms that are not optimized and under-
utilized features and capabilities.
EVEN AN ADDITIONAL Connectivity Communication is vital. Ensure that systems can communicate with a minimum of
SECOND OF DELAY CAN latency, that the firewall has sufficient capacity, that the system is optimized for mobile networks,
PUT YOU AT THE BACK OF
that the DNS routing is correct, and that CDN caching is optimized.
THE ECOMMERCE PACK
Bandwidth Can you be reached? Ensure that the bandwidth pipe is big enough for the traffic.
Review the content making up the pages. Rich content can mean big data and bandwidth require-
ments. Verify that the site can support different connection types/speeds including mobile devices.
Architecture Match the car to the engine. Watch for unbalanced tiers, mismatched technology
choices or a dead-end scalability path.
Third-Party Services You are only as fast as your slowest link. Ensure that analytics and tracking,
I payment systems, aggregated content, social networks or CDNs are not slowing the site down.

4. Testing for Success

Its clear that to test for success an effective eCommerce test strategy is critical. Having the right
team in place to implement the strategy will then enable the tests to be executed successfully. Test-
ing for success means understanding the value of testing in production and in the performance lab.
SOASTAs methodology emphasizes various tests in different places to find what is often a needle
in a haystack.

4.1 SOASTAs CloudTest Methodology

The SOASTA Performance Test Methodology is a software QA performance testing strategy with a
set of lightweight processes that fit into the software development lifecycle and ensure that online
applications are ready for peak usage. The methodology includes testing applications in a lab
environment inside of the firewall and processes for testing the production infrastructure used by
customers.
Cloud Testing leverages the efficiencies of the cloud to test web applications and sites. By using
the infrastructure and services provided by companies like Amazon Web Services and Rackspace,
SOASTA customers can reduce the cost of load and performance testing while increasing the ac-
curacy of representing the actual traffic to the site.
CloudTest is a distributed architecture that can also be deployed completely behind the firewall,
or in a combination of on-premise and cloud-based configurations. Based on years of experience
testing from the Cloud and leveraging existing best practices and proven performance testing
methodologies, the SOASTA methodology extends traditional approaches to address new opportu-
nities and challenges. These include:

SOASTA RETAIL BEST PRACTICES WHITE PAPER | 6

 Testing both in the lab and with live web-based applications in production

Leveraging the Cloud to test at both typical and peak traffic levels, from hundreds of users to
millions
Responding to the acceleration of development cycle times by making agile performance testing
a realistic alternative
Generating geographically dispersed load to most accurately represent real-world traffic
Generating both internal and external load and against both the lab and production environments
for the most efficient and effective results
Analyzing performance intelligence in real-time to speed problem resolution
Tests are not executed until the latter half of the methodology. This is because a successful perfor-
mance test is the result of defining a strategy that is best for an eCommerce site, and then putting
the right people and processes in place.

4.2 Strategy and Planning

SOASTAs approach to performance engineering calls for an umbrella strategy with associated
individual test plans. Test plans roll up into an overall view that ensures confidence in the ability of
key revenue generating applications to perform as expected. The result is an ongoing performance
engineering strategy throughout a sites evolution. It includes a number of test plans centered on
individual objectives, such as holiday readiness, a major architectural change, or the release of a
major version of code.
Having a well-defined strategy, with explicit test plans, provides business and engineering leaders
with a high degree of confidence in operational readiness. Using this approach gives greater insight
into an applications performance.
Using an iterative process within test plans to achieve defined goals allows for a stream of continu-
ous improvement in the applications being tested. It starts with the test definition and ends with
HAVE
HAVEA WELL-DEFINED-
A WELL-DEFINED obtaining actionable intelligence results.
STRATEGY,
STRATEGY,WITH EXPLICITLY
WITH EXPLIC-
DEFINED TEST PLANS.
ITLY DEFINED TEST PLANS .
The process of creating a test plan starts with the Define Phase. During this phase, the flows to be
tested throughout the site are defined, metrics to be monitored are established, and success crite-
ria for the tests are agreed upon.
In the Design Phase, the user scenarios are written and test parameters are set up. Things such as
the mix of users executing different parts of the application, the virtual user targets, and the ramp-
time are modeled.
The Test Phase is where the execution of tests takes place, and where data is collected for assess-
ment.
Finally, the Assess Phase, parts of which may occur during the test execution, is when the data col-
lected throughout test execution is used to provide actionable intelligence.

4.3 Production Testing

There are a variety of reasons for testing in production, but in large part its becoming more com-
mon because its now affordable and because the positive experience of SOASTAs customers has
reduced the fear associated with testing a production infrastructure. The only way to truly build
confidence in a site is to test that site. Due to the sheer number of users accessing their site at any
given time, very few companies can replicate a production environment in a staging environment or
in a performance lab. SOASTA has developed a methodology for testing in production that mini-
mizes the impact on real users while getting valuable, actionable information.
Testing in production is the best way to get a true picture of capacity and performance in the real
world and is the only way to ensure that online applications will perform as expected.
Can retail companies really test in production? The answer is a resounding yes!

SOASTA RETAIL BEST PRACTICES WHITE PAPER | 7

The SOASTA CloudTest
Methodology is a software QA
performance testing strategy
with a set of lightweight pro-
cesses that ensure that online
applications are ready for peak
usage.

SOASTAs production testing methodology helps identify the invisible walls that show up in archi-
tectures after they move out of the lab. Traditionally, testers have been limited to making extrapola-
tions over time about whether small tests on a few servers in a lab can support exponentially higher
amounts of load in production. Without proper testing, these types of assumptions always result in
unexpected barriers. There are many things that a production testing approach catches that cannot
be found with traditional test methods. These include:
Batch jobs that are not present in the lab (log rotations, backups, etc.) or the impact of other
online systems affecting performance
Load balancer performance issues such as misconfigured algorithm settings
Network configuration problems such as 100MB settings instead of 1GB on switches and routing
problems
Bandwidth constraints
Data pipe not configured as burstable to handle spikes in traffic
Latency between systems inside and outside of application bubbles
Misconfigured application servers or web servers
CDN not configured to serve up new content
Radically different performance depending on database size

4.4 Testing in the Performance Lab

Ongoing performance testing in a lab allows application engineering teams to assess performance
over time, and helps catch any show-stopping performance bugs before they reach production. In
addition, the lab provides a place to performance test code and configuration changes for perfor-
mance regression before releasing changes to production and outside of the normal build cycle.
This could include things like a quick bug fix in a page, or a seemingly minor configuration change
that could have a performance impact. Often, these kinds of changes are deployed with little to no
testing and come back later as a cause of performance issues.
Testing in the lab also gives you the ability to break the system without having to worry about the
impact on users. It is important to understand how the eCommerce site responds when taken to
failure and how it recovers. No one wants to bring a live production site to failure.

SOASTA RETAIL BEST PRACTICES WHITE PAPER | 8

Building an End-to-End Test
Plan where production testing
is an essential component.
TESTING IN PRODUCTION
IS THE BEST WAY TO GET A
TRUE PICTURE OF CAPACITY
AND PERFORMANCE IN THE
REAL WORLD AND IS THE
ONLY WAY TO ENSURE THAT
ONLINE APPLICATIONS WILL
PERFORM AS EXPECTED.
SOASTA
4.5 Types of Tests
In addition to testing for baseline performance metrics, the methodology includes failure, duration,
spike and break point testing. There are a number of different test types that might be included in
a test strategy, which are described in more detail in the Testing Tactics section on page 10.
Executing different types of tests in different environments is a key element in finding and resolving
bottlenecks in architectures that make up an eCommerce site, and are often quite complex. The
graphic below illustrates how different types of tests in the lab and in production can help pinpoint
specific issues as you work toward optimal performance.
4.6 Content Delivery Network (CDN) and Third-Party Content
When a retail company tests in production, it can also fully test the caching/offloading capabili-
ties of its CDN. This is vital to understanding the true performance of a production environment.
A primary purpose of a CDN is to reduce the amount of times content has to be requested from
the origin servers by delivering certain content from strategically placed servers within the CDN
network.
When preparing to test in production it is imperative to engage with the CDN provider early in
the process to ensure that a production test has the support of the CDN provider. SOASTA has
worked with Akamai to develop a set of best practices for tests including CDN assets.
The vast majority of test labs do not have a CDN as part of the infrastructure. Therefore, its
impossible to test how the CDN affects performance without testing in production. One SOASTA
customer, a performance engineer for a multi-billion-dollar retailer with more than 2000 store-
fronts, considers the inclusion of a CDN a vital part of production testing. He says, When we test
in production, we generally have Akamai caching enabled. This definitely influences the testing in
terms of number of requests that reach origin (main site) and the response times depending on
from where the page is served - Akamai or origin. In the lab it is most likely that we wont include
Akamai. This means we have to include the CDN in our production tests or we have no true idea of
our production performance.
Many eCommerce sites use third-party providers to enhance their overall site content. Its vital to
involve those third-party providers that might have an impact on performance when the strategy
is being formulated. On the other hand, you would not typically include domains such as Google
Analytics or Omniture metrics as part of your test. They do not want to be surprised by a test or
have it bring down their service or site with fake transactions. Involving third-party providers early
in the process helps to ensure their support. They may even want to be measured in the process.
After all, they want to have a well performing site, too.
SOASTA CloudTest gives retailers the ability to analyze the performance of individual third-party
provider content and provide that information to the third-party provider.
RETAIL BEST PRACTICES WHITE PAPER | 9
 4.7 Potential Live Customer Impact
Testing in production can be done with or without live users on the environment. The majority of
customers testing in production with CloudTest are doing so on a live environment. Its possible,
but not always feasible, to put up a maintenance or turn-away page and wait until all existing user
sessions have finished. In reality, this method is actually rarely utilized because the right tool and
methodology working together can allow testing to take place with active users on the site in all but
the most extreme cases. To use one SOASTA customers exact words, The cost of not testing is
far greater than the potential impact on live customers. You can make revenue at the same time
youre testing with the right approach.
Its also possible to segment a portion of the live environment during a low-traffic period and allow
for testing on this sub-environment. Or test against one data center while live users are all routed
to another data center. Typically a separate IP address is configured on a load balancer and serv-
ers are moved out of the live pool and placed into the test pool. Sometimes configuration changes
need to be made to other shared components. This is a more costly approach due to the need for
extra hardware and the maintenance overhead. Its also somewhat less reliable because you start
to deviate from the actual production configuration, and you cannot test to true scale. It is, how-
ever, a more realistic test than simply testing in a lab.

4.8 Three Requirements for Successful Live Testing

The first requirement for being able to do a large-scale test with live customers is having real-time
analytics in your test tool. With up-to-the-second information about site performance, you can
quickly tell if a site has started to experience poor performance or become completely unrespon-
sive.
The second requirement is a good kill switch. Pressing stop or abort in a running CloudTest will
stop the load immediately. Bandwidth, concurrent connections, threads in use, and other typical
pinch-points will all drop back to normal. With real-time analytics and the kill switch, you can stop
a test as soon as you suspect customers may be impacted. If the test tool takes a long time to kill
a test then this poses a serious risk to real customers. CloudTest is designed to accomplish a near-
immediate stop.
Finally, having good monitoring practices internally, preferably integrated with CloudTest, can pre-
vent you from ever needing to abort a test because of live user impact. Real-time visibility into met-
EXECUTING DIFFERENT
rics like CPU utilization, heap usage, garbage collection, and connection counts on load balancers
TYPES OF TESTS ACROSS
DIFFERENT ENVIRONMENTS or firewalls can help prevent those thresholds from ever being violated by routine testing.
IS KEY TO FINDING AND
RESOLVING BOTTLENECKS 5.0 Testing Tactics
IN THE ARCHITECTURES OF
ECOMMERCE SITES. Running the tests is exciting because it is the culmination of many hours of work. This is not the
time to take shortcuts! If a good test strategy has been developed then the time has come to ex-
ecute it.
SOASTAs testing methodology, as described above, has proven successful for retailers of all sizes.
The Test and Assess phases are where you see the fruits of your labor. This is when you simu-
late the expected traffic; monitor the results in real-time, and collect the data to help identify any
necessary remedial actions. Many SOASTA customers use the dashboards while the test is running,
greatly reducing the break/fix cycle time.
Testing in production allows the tests to be executed the same way the customer will interact with
the application. Its worth repeating that the SOASTA methodology is a continuous process. Retail-
ers should test early and test often. Minor changes can have major impact and is another reason
we recommend testing in both the performance lab and in production. It helps find performance
issues earlier in the testing process.

5.1 Roles and Responsibilities

It is vital to have the right people involved in the test execution. The size of your company and staff,
the scale of the site, the complexity of the testing, whether or not youve outsourced development
or deployed third-party applications within your site, the use of a managed service provider for
infrastructure and more all influence how many individuals are involved in a test.
In some cases its one or two people who do everything. In other cases, such as the complete
reconstruction of a site or testing a popular site for seasonal readiness, there will be multiple people
for many of the responsibilities, from architects to individual contributors responsible for specific
elements of the infrastructure.

SOASTA RETAIL BEST PRACTICES WHITE PAPER | 10

 There are a number of responsibilities to address as part of a testing strategy:
Coordination Coordination of testing activities with all key stakeholders. This includes applica-
tion engineering, operations, vendors and third-party connected services, as well as business
and senior leadership.
Communication Communication of test results, issues, issue resolution plans, and progress
against overall strategy to business and engineering leadership.
Strategy Definition of an overall strategy that includes establishing business processes for ap-
plications, key performance indicators, capacity plans, monitoring coverage, and individual test
plans that roll up into a performance management aspect of the software development life cycle.
Architecture Delivering best practices on creating high-performing application and infrastruc-
ture architectures for new projects or for improving existing applications.
Test Creation Turning business process definitions into executable tests, creating individual
performance test workloads from capacity plans, and maintaining the current library of test cases
on an ongoing basis.
Test Execution Executing ongoing performance tests in the performance lab and production
environment with delivery of test results.
Analysis Reviewing test results from all environments and analyzing the performance with
respect to previous tests. Responsibility for calling out any violations of established success cri-
teria, thresholds that may be violated on key performance indicators, or deviation from previous
WHEN THE RIGHT MIX OF baseline tests.
PEOPLE ARE BROUGHT TO- Diagnosis Root cause analysis of bottlenecks or performance problems that might arise. Using
GETHER, TESTS ARE MUCH
domain knowledge and specialized tools such as profilers, memory leak detectors, and monitor-
MORE PRODUCTIVE.
ing tools to pinpoint problem areas.
Tuning Tuning involves applying best practices, tuning recommendations and isolating parts of
the application or infrastructure that could be optimized to give better capacity or performance.
Measurement Responsibility for analyzing activities and their progress against the overall strat-
egy, in addition to making process improvement recommendations for optimizing all strategic or
tactical activities.
In small companies one person may take on all of these responsibilities, often assisted by SOASTA
in test strategy, creation, execution and analysis. Some or all of the roles may be outsourced, typi-
cally to take advantage of expertise, create focus on areas lacking attention and/or to lower costs.
Normally there is a blend of individuals and roles. For example, the presence of a Project Manager,
performance engineers and/or specialists would either remove the need for a tech lead, or allow
that person to fill other gaps in the process.
Below are common roles/titles and how they might map to the various responsibilities:
Project Manager Coordination, communication
Technical Lead Coordination, communication, strategy, architecture, analysis, diagnosis, tun-
ing, measurement
Architect Infrastructure architecture, app architecture
Performance Engineer Develop strategy, architecture, analysis, diagnosis, tuning, measure-
ment
Test Engineer Test creation, execution, analysis
Specialist Diagnosis, tuning
When the right mix of people are brought together, tests are much more productive. When issues
arise, having the right team available enables those issues to be quickly taken care of and decisions
to be made when actionable intelligence is provided.

5.2 Types of Tests

A question that often comes up is What kind of tests need to be run? There are several test op-
tions baseline, stress, spike, endurance and failover. For a retailer, all of them have value.
Baseline Retailers need to establish a baseline of what is acceptable performance for their
site when under average load. We recommend taking the last six months of analytics and the
busiest hour from each day and using that as your average load values for page views/hour and
orders/minute.
SOASTA RETAIL BEST PRACTICES WHITE PAPER | 11
 Stress Retailers need to run stress tests to ensure that site performance does not degrade
under heavy load for an extended period of time. It is not unusual for memory leaks or faulty gar-
bage collection to cause performance issues that are not seen until stress tests are conducted.
SOASTA recommends that stress tests be executed at 150-200% of peak expected load.
Spike A spike test is when a very large number of users ramp-up very quickly or almost si-
multaneously. For retailers spike tests are vital. Many retailers have spike events flash promo-
tions, Black Friday, Cyber Monday, Valentines Day, etc. and these events can bring a site to its
knees. Retailers have to ensure that during such events orders are not lost or users turned away.
SOASTA recommends that spike tests be run at 200% of peak expected load.
Endurance Endurance tests differ from stress tests in magnitude. While a stress test will have
a very high number of page views and orders, an endurance test will simulate slightly less load
but for a longer period of time. These tests have great value as they can uncover additional areas
where performance can degrade over a prolonged period of time. There are batch jobs, database
backups and cache refreshes that can be scheduled and run on a recurring basis. Running an
endurance test over several hours may show that these types of events effect performance.
Failover Most retailers have disaster recovery or failover servers that should spring into action
in case of a failure. If a hardware failure occurs during a period of heavy load can the backup
hardware adequately step into the void? There is no way to know unless a failover test is ex-
ecuted.
In early testing the goals are often simply to understand the performance characteristics of the site.
For those tests the ramp-up rates may be slower and more methodical. Once these baselines are
THE COST OF NOT TESTING established, and you have a good sense of what the site can handle, its critical that the tests be
IS FAR GREATER THAN THE executed as realistically as possible. This includes having realistic user levels and ramp-up rates.
POTENTIAL IMPACT ON LIVE Reviewing the analytics from previous years will help you build user ramp-up levels. It is imperative
CUSTOMERS.
that you do not cheat the test. Do not alter think times or remove critical components of the test.
Removing or altering them will only skew the results and lessen the value of the tests.

5.3 Monitoring the Test

Monitoring is an integral part of the testing process. CloudTest includes system monitors that cap-
ture real time data on the systems under test. It monitors the performance of the entire application
stack and enables all test participants to monitor how the system under test is reacting under load.
Additionally, SOASTA integrates with third-party application monitoring tools like CA Application
Performance Management, AppDynamics, AWS CloudWatch, Correlsense and New Relic. This
allows even greater insight into the application stack during the test. SOASTA also works with the
monitoring systems that retailers already have in place. Most retailers have extensive monitor-
ing covering the breadth of their architecture. When a test is executed that combines SOASTAs
monitoring with the retailers monitoring, unparalleled insight can be gained into the systems true
performance.
Understanding exactly how the infrastructure reacts under load allows for modifications to be made
during the test. For example, it is not uncommon to identify an application server that is not in rota-
tion or a thread pool that is incorrectly configured. Due to SOASTAs real-time analytics, changes
can be made on the fly during the test and the impact immediately measured.

6. Retail Examples
Are there retail companies that follow this methodology today? Yes! SOASTA works with several
retailers that test from lab to production and the results have been outstanding.

6.1 Multi-billion-dollar Retailer with 2000+ Storefronts

The company conducts load and performance tests in both a test environment and a production
environment. They test on a continual basis to see if there are performance issues that need to
be addressed very early in the development cycle. Prior to the 2011 Holiday Season, the retailer
conducted 21 tests in their production environment. This enabled them to fully test several different
configurations to achieve optimal performance.
Additionally, the company knew their systems exact capacity and where the first bottleneck could
potentially appear. That is powerful knowledge to have going into a busy retail season. Consider
how much time is lost trying to pinpoint the bottleneck and how to overcome the performance
issue. This allows a contingency plan to be put in place and ready to execute if higher than antici-
pated load is observed.

SOASTA RETAIL BEST PRACTICES WHITE PAPER | 12

We mandate that we test in production. ... This is the only way we can ac-
curately test our load balancers, switches and third-party providers, which are
generally not present in the lab.
Performance Engineer,, Multi-Billion-Dollar Retailer
AFTER PRODUCTION
TESTING, THE RETAILER
EXPERIENCED THEIR BEST
ONLINE RETAIL EVENT
EVER. OPTIMAL PERFOR-
MANCE WAS MAINTAINED
DURING PEAK PERIODS.
THEY WERE ABLE TO
EASILY HANDLE 3X THEIR
PREVIOUS HIGH VOLUME
DAY.
SOASTA
How could these tests be executed in production? Wasnt it afraid of affecting actual customers?
Naturally it was concerned about its customers. Hence, when tests were executed in production,
careful monitoring was conducted at several layers by both SOASTA and by its internal monitor-
ing systems. This served a dual purpose. First, any significant degradation in performance could
be immediately detected and the test stopped, if necessary. Second, it was a real-world test of the
systems that needed to be monitored, stopped or rebooted when Black Friday arrived. The team
tasked with ensuring the viability of the entire company infrastructure had great confidence going
into Black Friday as they had already run more than 20 realistic tests.
The companys performance engineer observed, Due to the discrepancies between the lab and the
production environment, there is no guarantee that the application that is certified to be perform-
ing well in the lab would perform the same way in production. Hence, we mandate that we test in
production, which enables us to test the application with even higher than expected load levels
since there are different hardware constraints. This is the only way we can accurately test our load
balancers, switches and third-party providers, which are generally not present in the lab.
This testing allowed the retailer to place 6th in Catchpoint Systems rank of the top 50 retailers
when they tested Document Complete times.
6.2 Dillards, a Department Store Chain with 330 Storefronts
Dillards was new to the testing in production model. Previously, they had conducted all load and
performance testing in their test environment and had experienced performance issues during criti-
cal retail events. Not only did this lead to a loss of revenue, but their reputation was also impacted.
Dillards, in true proactive fashion, took on the challenge to get to the root cause of the performance
issues. For Dillards, this meant looking at everything from the architecture and code to the hard-
ware. Significant changes were made across the board.
As might be expected, Dillards wanted to start testing in production gradually. SOASTA conducted
low-volume tests of a few hundred users hitting the site at a rate of a few thousand page-views per
hour and generated a relatively low number of orders. These low-volume tests allowed Dillards to
understand the methodology and gain a comfort level with SOASTA, and the confidence to test in
production. Once these low-volume tests proved successful, larger tests could be conducted to
reach the page view per hour and order per hour benchmarks Dillards had set as targets. These
increasingly larger tests began to reveal performance bottlenecks. Resolving these issues would
prove critical to having acceptable performance during peak traffic periods.
A few database issues were identified that werent found in the QA environment. It was discovered
that a certain database job would kick off at certain times each day. This would not impact per-
formance during average retail periods. However, when SOASTA would run a peak-level load test,
performance would noticeably degrade when this job was active. Without knowing this, its likely
that the customer experience would have been adversely affected during these periods. In addition,
some database locks were identified when the site was under high load. Had these locks not been
identified it would have proven catastrophic.
It was also observed during one of the large load tests that the connection pool limit was being
reached on all the application servers. This led to further tuning to optimize the performance of
each of the application servers.
Dillards experienced their best online retail event ever. Their environment maintained optimal per-
formance during peak periods. They were able to easily handle 3X their previous high volume day.
RETAIL BEST PRACTICES WHITE PAPER | 13
 6.3 One of the worlds largest retailers
One of the worlds largest retailers conducted extensive performance tests in production. Dozens
of tests were conducted against production sites located on three continents, including mobile ap-
plications. This company has a corporate mandate of always testing in production. In fact, over 250
personnel, including the CTO, attended the final load test before Thanksgiving, certainly an indica-
tion of how important such testing is to the entire organization. They fully understand that the only
way to find large-scale bugs is to test at scale.
During one round of testing a significant database contention issue was discovered. This issue was
only observed at very high scale thousands of orders per minute with tens of millions of page
views per hour. This would have been impossible to replicate in a lab or test environment. Finding
this issue directly led to a successful Black Friday event where their systems were able to remain
stable, even under extreme load.
Another example of how vital testing in production proved to be was illustrated when just three
weeks before Black Friday, a new shopping cart flow was released. It had passed all internal tests
and was believed ready for large-scale load. However, when SOASTA performed a load test it was
quickly apparent that under heavy load that this new shopping cart flow would not remain stable.
This discovery allowed changes to be made so it could withstand the expected influx of users.
Without this test, it is quite likely that many customers would not have been able to complete their
purchases on Black Friday.

7. Security
Of course, retail companies, like most enterprises, have specific considerations when it comes to
security, data integrity and systems management when testing their infrastructure.
THIS COMPANY HAS A Data security is one of the key concerns when it comes to performance testing. Companies do
CORPORATE MANDATE not want their corporate data outside corporate firewalls. SOASTA does not ask customers to
OF ALWAYS TESTING IN export data or their application outside their firewalls. We typically generate synthetic data or use
PRODUCTION. IN FACT, scrubbed data depending on the customers business and security requirements. CloudTest does
OVER 250 PERSONNEL, not save any response content on the CloudTest servers or in the result database. Only key http/
INCLUDING THE CTO, https metrics and statistical data are captured for reporting, analytic, and diagnostic purposes.
ATTENDED THE FINAL
LOAD TEST BEFORE Security is one of those things that no one notices until something goes wrong. Data is just sup-
THANKSGIVING, CER- posed to be secure. You rarely see a story on the news about how a company is doing a great job
TAINLY AN INDICATION of keeping their customer data secure. But you will see a story about a company letting customer
OF HOW IMPORTANT data slip into the wrong hands. With good reason, customers must be cautious about letting a
SUCH TESTING IS TO
company have their data unless they are sure it will be safe. To ensure that our customers data re-
THE ENTIRE ORGANIZA-
TION. mains safe, SOASTA has implemented a stringent security policy that meets the needs of our retail
customers.

7.1 Test Development

Data that is used for test development by a Performance Engineer (i.e. HTTP(s) GET/POST requests
and responses) is not typically retained by SOASTA.
At the customers request, this data will be deleted immediately after test development.
Test definitions (SOASTA test clips) are saved in the SOASTA repository for future testing. These are
the steps that comprise the scenario and, while they can be deleted upon request, since there is no
confidential data or other company private information they are typically retained for ongoing use.

7.2 Test Metrics

This includes the performance results of the test(s), such as response times, errors, data transfer
rates, etc., as well as infrastructure metrics for any system resources that are monitored as part
of the test. Monitored metrics may come from SOASTA monitoring agents, SSH access to system
information or third-party monitoring systems such as described above. Test result metrics for
on-demand engagements are kept for a minimum of one year, primarily to allow for comparative
reporting over time. Additionally, if SOASTA is delivering the testing as a service, results can be
exported to a CloudTest instance that is located on the customers premises.
At the customers request, this data (performance metrics and system resource monitors) will be
deleted immediately after the delivery of test reports or if the results are exported and sent to the
customer.

SOASTA RETAIL BEST PRACTICES WHITE PAPER | 14

 AWS

SOASTA IBM

CloudTest
Rack-
GoGrid space SUT/AUT

Azure
Cloud Testing with SOASTA
provides completely distrib-
Firewall
uted architecture, automated
deployment and multiple cloud
providers to effectively evaluate
the System Under Test.

7.3 Data Encryption

Passwords and usernames for the SOASTA CloudTest application and for system resource moni-
tors are encrypted. Seed data that uses any actual customer data can also be encrypted. Synthetic
seed data that is fabricated for logging into an application or web site to emulate real users is not
encrypted. All information used to setup monitors is retained in the SOASTA repository for future
testing.
At the customers request, this data, the usernames and passwords, will be deleted immediately
after testing.

7.4 Test Infrastructure

When delivered as a service, CloudTest customers have their own tenant (with associated creden-
tials) and this tenant is maintained for at least one year from the date of the last test. At customer
request, the tenant will be deleted immediately after all testing and reporting has been deemed
completed.
The actual load generators and results aggregation servers are temporary cloud-based instances
and run only during test execution. This provides an additional level of security as all instances are
discarded after each test, with performance metrics retained only in the results database.
See Appendix A for Performance Data Q & A that covers additional topics related to data security.

SOASTA RETAIL BEST PRACTICES WHITE PAPER | 15

Sample Master Data

APPENDIX A: CloudTest Performance Data Q&A

What is Performance Data?
There are many different types of performance data that are leveraged as part of performance test-
ing. Performance data can be broken up into three major categories: master data, user-generated
data and external data. Master data typically exists in the customers database(s) and is required for
conducting business (usernames, passwords, etc). User-generated data is anything that the user
inputs in editable fields on the application (new email address, new address, etc). External data is
provided upon execution of the application (i.e. confirmation numbers, session ids, etc).

What data is used during testing?

Data requirements are dependent on the application and the business processes under test. For
example, a static site might not require any performance data, just access to the site to make the
requests, while a more complex application might require all three types of performance data.

What happens to the response data received?

All customer-related response data is discarded from CloudTest servers during load testing. Only
performance data related to that response data is retained (response times, errors, etc.). In addition,
as noted above, CloudTest servers are temporary instances discarded after each test, with only
performance metrics retained in the results database.
However, all customer and external data is stored on the CloudTest server during script creation
and debugging. The data used to create scripts can be deleted after script creation is completed to
ensure no customer data is store on SOASTA servers.

How are log files or other metrics that capture information about data handled?
SOASTA does not keep log files or other metrics that capture information about customer data dur-
ing load tests. But as previously noted, we do store this type of data during script creation.

What data is sent back to the CloudTest servers?

During load testing, only key external data is kept, such as http response codes, cookies, session
IDs, etc. All data is sent back to the CloudTest servers to parse, at which point it is fully discarded
from any SOASTA servers.

SOASTA RETAIL BEST PRACTICES WHITE PAPER | 16

 What is the difference between scrubbed and synthetic data?
Scrubbed data resides in a customer database and has gone through a process so it no longer
includes any real customer data; it is, in essence, data that has been transformed and/or created
from actual customer data. Synthetic data is generated from scratch and is designed to be an ac-
curate representation of the customers production database.

How do we create synthetic data?

SOASTA works closely with your application team to ensure a rich spread of representative data is
created. Data is also created to target the business process under test and can expand as testing
expands.

What happens to the data once the testing is over?

When testing is complete, the CloudTest servers only store performance metric data relevant to the
test runs. No corporate data is retained on the CloudTest servers.

How is the test data stored in the cloud?

For tests executed by SOASTA, the test results are stored in the cloud. The results are on Linux
servers that are taken down at the conclusion of the testing event. These servers are only available
during testing sessions or when results are being analyzed. In most cases, the result data is stored
in a relational database in EC2 in an EBS (elastic block store). This data is only available to SOASTA
employees.

Can test result data be removed from the cloud?

Yes, results can be deleted and removed from the cloud at the request of the customer. When
requested, the results are not deleted until after the result report is completed. Once the results are
deleted, they are permanently deleted; there are no backups of deleted results.
Note that deleting results limits the ability to compare results from prior and future tests. The only
way to compare results once a result is deleted is by looking at the result report.

Can test result data be exported from the cloud for offsite storage?
Yes. All results are exportable in XML format. Results can be exported from the cloud and moved
to a customer-defined location for safekeeping. Depending on the length of the test and amount
of performance data related to the test, it can take up to 24-48 hours to export the data from the
cloud and move to a secure location. Optionally, the results can then be deleted from the cloud as
well.

Contact us: To learn more visit: 2012 SOASTA. All rights reserved. SOASTA,
the SOASTA logo, and SOASTA CloudTest
866-344-8766 soasta.com
are trademarks of SOASTA. All other product
444 Castro St. or email us at or company names may be trademarks and/
Mountain View, CA 94041 info@soasta.com or registered trademarks of their respective
owners.

EST-1001-1-0512

Connect with us: