Multivariate testing

Understanding multivariate testing techniques and how to apply them to

your email marketing strategies

An Experian Marketing Services white paper

Multivariate testing

Testing should be standard operating procedure

Whether it is in search, display, direct mail, email, mobile or social, one would be
hard-pressed to find a marketer today who isnt aware of the concepts of A/B or
multivariate testing (MVT) and the value they offer. With the amount of data that is
being captured, stored, processed and analyzed in todays world, combined with
the speed at which results boomerang back to the tester, we operate in a time
and place where there is no excuse not to continually optimize your marketing
campaigns via some sort of testing procedure.
Even the president tests
Testing has indeed become mainstream. Highlighting this fact is a recent article in
Businessweek, which details the use of statistical testing by the Obama reelection
campaign when trying to solicit donations from their email subscribers. The article
describes how the campaign staff would initially experiment with a handful of
different subject lines for their emails. After sending these seed emails out to a
limited number of subscribers, the team would wait for the results (amount of
money donated) to trickle in before it became clear which email subject lines
performed best. Now equipped with the knowledge of what emails would perform
significantly better than others, the team would send the winning emails out to the
rest of the millions of email subscribers on the presidents list, arguably increasing
the donations haul from email marketing by millions of dollars.
Whats being tested?
Testing ranges in complexity. A/B and A/B/C testing are the most common forms
of testing used in marketing today. These types of tests are fairly straightforward
to perform and interpret, as they focus solely on measuring the effects of changing
values from only one factor. As shown in the graph on the following page from
Experian Marketing Services December 2012 Email Market Study, where email
marketers across eight verticals were surveyed about their email marketing
initiatives, most companies today are leveraging A/B and A/B/C testing to
optimize different facets of their email campaigns (subject line, frequency, time of
day, day of week, call-to-action, etc.), but very few (11 percent) are leveraging the
more complicated, yet more powerful, tool of multivariate testing to optimize their
email marketing efforts.

Page 1 | Understanding multivariate testing techniques and how to apply them to your email marketing strategies

Multivariate testing

What type of testing do you perform on email campaigns?

100%

97%

81%
80%

60%
50%
43%

39%

40%

36%
28%
21%

20%

0%

11%

Subject
line

Creative Frequency

Time
of day

Day
of week

HTML
versus
text

11%

Call-to- Product Number

action placement
of
products

11%

Friendly Multivariate
from

Source: Experian Marketing Services Email Market Study, Acquisition and engagement tactics, December 2012

A common A/B testing mistake

A common mistake when implementing A/B testing is performing sequential A/B
tests in an effort to arrive at optimal levels for multiple factors. These experiments
often start with the standard or status quo settings of the key factors to be tested.
The levels of the factor that is believed to be the most responsible for performance
are tested first, while the other factor levels remain constant. After the responses
have been gathered and the optimal level for the first factor is determined, the
factor regarded as the second most influential is tested next, with the optimal
first level factor remaining fixed for the rest of the experiment. This process
continues to repeat itself until each factor level has been individually tested.
To better illustrate this flawed process, consider the following example with
two factors, each with just two levels (the simplest case possible). Suppose an
organization wants to determine the best image and ad copy to put in an email
with the click-to-open ratio being the metric to maximize. We will denote the
different images as I1 and I2 and the different ad copy as C1 and C2. In their first
email blast to 20,000 customers, the company decides to send half of these
customers an email with the combination (I1, C1) and the other half with the
combination of (I1, C2).

An Experian Marketing Services white paper | Page 2

Multivariate testing

The results are as follows:

Click to open percent:

(I1, C1) = 7.5 percent

(I1, C2) = 8.5 percent
Based on these results, the company believes that Copy 2 is the preferred
ad copy and fixes the next email blast to 10,000 more customers at this level so
that the next test in the sequence will only vary the image level not already tested.

Click to open percent:

(I2, C2) = 9.5 percent

Seeing these results, the company decides that (I2, C2) is the optimal combination
in terms of being able to generate the highest click-to-open ratio.

The problem is that the company may have missed out on finding the global
optimum by not testing the fourth combination of (I2, C1), which may have
yielded a click-to-open ratio even greater than 9.5 percent.
A/B tests assess one level of one factor versus the control group, but cannot
measure the interaction effect across factors.
The reason this method of sequentially testing one factor at a time fails to find
the optimal factor levels is that an interaction effect exists between factors 1
and 2. That is, the factor effects are not additive, but rather the combinations
among different factors and their levels produce an additional effect (interaction)
when used simultaneously. By not being able to capture interaction effects, this
sequential approach may miss the optimum altogether.
In situations such as these, it is more appropriate to perform a multivariate
test (factorial test to be exact), where all factors are changed together and
all combinations are accounted for. There is a wide array of different kinds of
multivariate tests available, but when the number of factors and the number of
factor levels to be tested are limited, the full factorial approach is the best option,
as it retains the most amount of information about the factors.

Page 3 | Understanding multivariate testing techniques and how to apply them to your email marketing strategies

Multivariate testing

Multivariate testing: a brief history

Multivariate testing has its roots in the statistical discipline of experimental
design. Experimental design is the process of planning a study to meet specified
objectives. Planning an experiment properly is important in order to ensure that
the right data and sufficient sample sizes are available to answer the research
question of interest, as clearly and efficiently as possible. One of the first recorded
applications of experimental design comes from the early 20th century and a
statistician named R.A. Fisher. Fisher was hired by an agricultural research center
to determine the effects that various factors (soil type, sun exposure, rainfall,
fertilizer, etc.) had on plant growth. By designing proper, randomized controlled
experiments before any seeds had been planted into the ground, Fisher was able
to harvest results that allowed him to determine the optimal levels of each factor
that would maximize plant growth and crop output. His work proved seminal,
and others soon found good use of experimental design in a wide variety of
industrial and manufacturing problems. Only recently have brands discovered
the power of experimental design (now termed multivariate testing) in optimizing
marketing campaigns.
Advantages of MVT over A/B testing
Multivariate testing has several advantages over that of A/B or A/B/C testing. In
A/B or A/B/C testing, it is easy to determine which particular creative performed
best at generating the most amount of conversions, but this type of testing cant
tell you why one creative performed better than the next. Was the increase in
conversions due to the change in background color, the call to action, the main
image, etc.? MVT allows you to measure the size of the effect that each of these
factors has in generating conversions and provides guidance on where to focus
your next round of testing efforts.

An Experian Marketing Services white paper | Page 4

Multivariate testing

The graph below is an example of what may be found through MVT testing:
Factor influence on click rates

42.0%

Layout

39.0%

Factors

Main image
10.8%

Call to action
Size
Color
0%

5.9%
2.3%
10%

20%

30%

40%

50%

Percent effect

MVT also saves the tester both time and

money, as it tests multiple factor-level
settings simultaneously. Instead of being
limited to varying the values of only one
factor, MVT, as its name implies, allows you
to test multiple factors at the same time,
A/B
thereby eliminating the need to run multiple
MVT
A/B tests. This increased efficiency gives the
tester the ability to find the optimal factorlevel settings more quickly, and the quicker
the optimal settings are found, the sooner results improve and return on
investment increases.
Lastly, as noted in the earlier example, MVT provides the decision maker with
better information as potential factor-level interaction effects can be captured. It
is most likely that the different factors that make up your email campaigns dont
work in isolation, meaning synergistic effects do exist among the different images,
colors, calls to action, sizes, etc. that make up your email messages. The phrase
The whole is greater than the sum of its parts rings true in these situations.
For example, you may have a particular image that performs OK on its own, and
a particular background color that also performs OK on its own, but when used
together, these particular settings create an email that performs amazingly well.
This type of interaction information cannot be captured via A/B or A/B/C testing
because only one factor is being analyzed at a time. By setting up tests that more

Page 5 | Understanding multivariate testing techniques and how to apply them to your email marketing strategies

Multivariate testing

accurately account for the true multidimensional nature of your data, you will be
able to gather deeper analytic insights that just arent possible to collect via A/B or
A/B/C testing.
Applications of multivariate testing
Email creative testing
There are many different dimensions of an email creative that could possibly
affect subscriber engagement: layout, main image, call to action, color, size, etc.
Depending on the number of factors you are interested in testing and the amount
of variations you have to test for each factor, the overall number of potential
creative combinations can quickly become rather daunting. For example, if you
are interested in assessing the effects of two different layouts, two different main
images, five different calls to action, two different background colors and two
different sizes, you are left with 80 different email combinations to test. However,
through the science of experimental design, there exists ways to reduce this
number of combinations to test down to a much more reasonable figure. In fact,
a properly designed multivariate test can reduce the number of combinations to
test from 80 down to as few as 16, a much more manageable situation. Another
benefit to this approach is that the tester loses very little in terms of back end,
actionable insights when reducing the number of creative combinations to be
tested in an intelligent manner. In other words, the ability to identify which factors
have a significant effect (and conversely, which factors show little to no effect) on
user engagement is still able to be captured, along with the identification of what
the optimal settings of the email factors should be in order to maximize subscriber
response/engagement.

Single
factor A/B

Fractional
factorial design

Full factorial
design
Layo

Layo

Buy now

Act now

Get more
info

Cal

l to

acti

on

Easy to manage, but

slow and narrow

X X
X
X
X
X
X
X
Cal
X
l to
a
ctio

Broadest learning, but expensive

and often infeasible

ut

X
Image

Image

ut

X
Cal

l to

acti

on

Complex to design, but most

efficient and effective

Source: Experian Marketing Services

An Experian Marketing Services white paper | Page 6

Multivariate testing

Time of day/Day of week testing

The following graphs show at an aggregate level what was historically observed as
the best days and times for an email deployment in Q4 2012. Charts with the best
days and times by major industry can be found in the appendix section of
this report.
Day of week performance
Q4 2012 all industry
Days
Monday

Volume %
17%

16.6%

2.7%

.12%

$.17

$173

Thuesday 15%
Wednesday 16%

16.8%

2.7%

.12%

$.16

$189

16.7%

2.6%

.11%

$.13

Thursday

15%

16.5%

2.5%

.11%

$.14

Friday

17%

16.4%

2.6%

Saturday

9%

Sunday

10%

17.8%
17.8%

.13%

2.9%
2.9%

0.0%
20.0% 0.0%
4.0%
Unique opens Unique clicks

.16%
.14%

$188
$195

$.16

$179

$.20
$.19

$176
$189

0.10% .20% $.10 $.20

$0 $100 $200
Trans. rate Rev. per email Average order

Time of day performance

Q4 2012 all industry
% of
% of
Volume Transactions
Range name
14%
12:00a.m.3:59a.m. 12%
22%
4:00a.m.7:59a.m. 20%
42%
8:00a.m.11:59a.m. 40%
12%
12:00p.m.3:59p.m. 15%
4:00p.m.7:59p.m.

10%

8:00p.m.11:59p.m. 2%

9%
2%

17.6%
16.2%

3.2%
2.5%

.20%
.15%

$.22
$.18

16.1%
17.6%

2.4%
2.8%

.13%
.13%

$.17
$.15

18.2%
21.7%

2.9%
4.2%

.15%
.34%

$.16

0.0% 30%
Unique
opens

$135
$164
$174
$188

$.48

$182
$246

0.0% 6.0% 0.0% .40% $0

$.60 $0
$300
Unique
Trans.
Rev. per
Average
clicks
rate
email
order

Source: Experian Marketing Services

Page 7 | Understanding multivariate testing techniques and how to apply them to your email marketing strategies

Multivariate testing

Looking at these results, we see that Saturday and Sunday had higher open,
click and transaction rates, but much lower volume. For weekdays, Monday
had the highest revenue per mail, but Friday had the better click rate. Volume also
plays a role in time of day, as 8 p.m. to midnight and midnight to 4 a.m. had higher
responses than the more popular 4 a.m. to noon times. However, as every brands
customers behave differently, it is most likely the case that the optimal day of
week and time of day for your email deployments will show a different pattern of
performance. It is important to remember that the data from the aggregate graph
is retrospective, not a controlled study. When designing a time of day/day
of week study for your subscribers, it is imperative that the various day/time
groups are defined ahead of time and in a manner that incorporates random
sampling techniques.
A properly designed MVT will give you clarity into what matters most in terms
of deployment decisions and which day/time combinations have proven
themselves to be statistically significant in their ability to provide improved
metrics for your business.
Frequency and recency testing
The question of How often to mail? cannot be answered without a
comprehensive testing plan. Trying to assess the effects of frequency or recency
on retrospective data has a much higher risk of confounding or misallocating
performance due to other subscriber attributes like engagement or tenure. To
help remove this noise from the process, MVT can be leveraged to create proper
test groups beforehand, as well as identify what sample size requirements need
to be met in order for results to be statistically significant. The use of forwardlooking multivariate test designs versus backward-looking retrospective analyses
will provide you with the assurance that the conclusions drawn from your study
are accurate and statistically meaningful determinations of the optimal delivery
settings for your subscribers.
When testing for optimal frequencies, it is also important to remember that
not all engagement segments should be exposed to the same amount of
experimentation. Your engaged audience is already demonstrating desirable levels
of performance, and you want this behavior to remain. You dont want to run the
risk of disrupting this routine with too many frequency tests. At the other end of
the spectrum, you do not have much to lose when testing against your inactive
audience; there is only upside to be gained. Take advantage of this situation,
and test against a wider range of subscribers to see if any different patterns of
frequency tend to spur added levels of engagement.

An Experian Marketing Services white paper | Page 8

Multivariate testing

Email cadence
Test
conservatively

Highly
engaged
Moderately
engaged

Test
liberally

Inactive

Sample learnings

Segment

Current frequency/
recency

Optimal frequency/
recency

Highly engaged

12x per month/

evenly staggered

15x per month/

evenly staggered

Moderately engaged

12x per month/

evenly staggered

10x per month/

evenly staggered

Inactive

12x per month/

evenly staggered

2x per month/
clustered

For more information on how to re-engage inactives, download Experian

Marketing Services reactivation white paper.

Page 9 | Understanding multivariate testing techniques and how to apply them to your email marketing strategies

Multivariate testing

Conclusion
Remember, testing all the various aspects of your email program for optimal
performance is an important tactic for any successful email marketing
campaign. The testing process is also iterative, with no prescribed end. As the
business environment and goals of your brand change alongside the needs and
wants of your customers, it is important to continually test and fine-tune user
experiences on a regular basis to ensure your marketing messages are primed
for maximum effect.

For more information on how to design and execute multivariate

testing with your email marketing programs, please visit our website
at www.experian.com/marketingservices or call us at 1 866 626 6479.

An Experian Marketing Services white paper | Page 10

Experian Marketing Services

955 American Lane
Schaumburg, IL 60173
1 866 626 6479
www.experian.com/marketingservices

2013 Experian Information Solutions, Inc. All rights reserved

Experian and the Experian marks used herein are service marks or
registered trademarks of Experian Information Solutions, Inc.
Other product and company names mentioned herein are the property
of their respective owners.
03/2013