Field Experimentation 1

© 2016, American Marketing Association

Journal of Marketing Research
PrePrint, Unedited
All rights reserved. Cannot be reprinted without the express
permission of the American Marketing Association.

Field Experimentation In Marketing Research

Ayelet Gneezy

Ayelet Gneezy

Associate Professor of Behavioral Sciences and Marketing

Rady School of Management, UC San Diego

9500 Gilman Drive #553

La Jolla, CA 92093
 Field Experimentation 2

Field Experimentation in Marketing Research

ABSTRACT

Despite increasing efforts to encourage the adoption of field experiments in marketing

research (e.g., Campbell 1969; Cialdini 1980; Li et al. 2015), the majority of researchers

continue to rely primarily on laboratory studies (Cialdini 2009). For example, of the 50 papers

published in the Journal of Marketing Research in 2013, only three (6%) were based on field

experiments. The goal of this paper is to motivate a methodological shift in marketing research,

and increase the proportion of empirical findings obtained using field experiments. We start by

making a case for field experiments, and offer a description of their defining features. We then

demonstrate the unique value field experiments can offer, and conclude with a discussion of key

considerations researchers should be mindful of when designing, planning, and running field

experiments.

Keywords: field experiments; lab experiments

 Field Experimentation 3

THE CASE FOR FIELD EXPERIMENTS

Researchers have proposed several reasons for why field experiments constitute only a

small fraction of experiments conducted by social scientists (e.g., Cialdini 2009). For example,

field investigations tend to be highly time-consuming: identifying potential partners and

obtaining their consent, in addition to planning and actual data collection can take several years.

In less successful instances, one’s efforts might prove futile for reasons beyond her control (e.g.,

a company’s legal or public relations considerations). These uncertainty and time-intensive

characteristics decrease the attractiveness of field experiments for any researcher, especially

during the early stages of one’s academic career. Second, field experiments focus on observed

behavior and limit researchers’ ability to investigate the psychological processes underlying that

behavior, and with it, the project’s theoretical contribution (e.g., Fong et al. 2015). Finally, due

to the dynamics inherent in field (vs. lab) settings, field experiments often involve a high degree

of noise and limited control over the experimental procedure, which in turn might introduce

challenges for data analyses and interpretation. At this point, one might be asking why, given the

challenges and potential downsides, running field experiments should even be considered.

To us, the answer is clear. It begins with the simple yet profound notion that behavioral

research aims to understand how individuals behave, what influences their preferences and

choices, how initial behaviors shape future behaviors, and so on. Although most published

papers discuss how the reported results apply to real-world situations, they do so by

extrapolating from research conducted in the lab, which often requires a considerable leap of

faith. By contrast, results obtained using field experiments that study behavior in the most
 Field Experimentation 4

relevant environment from an outcome perspective can be taken at face value—they tell us the

size of the effect, who is most affected and under what conditions, and the short- and long-term

implications for, for example, sales, loyalty, profitability, and voting participation.

We therefore propose that marketing research would benefit greatly from having papers

that offer converging evidence, using theoretical models, the laboratory, field experiments, and

naturally generated empirical data.

What constitutes a field experiment? In field experiments, participants are either unaware

they are taking part in a study or, if they are aware, they are engaging in activities as they would

regardless of the experiment (Charness, Gneezy and Kuhn 2013). Typically, field-experiment

participants are unaware of the fact that the researcher is manipulating factors in the environment

and measuring behaviors and outcomes. Examples include changing prices for products (e.g.,

Anderson and Simester 2003), varying expiration dates of promotional offers (e.g., Shu and

Gneezy 2010), and testing responses to different ads (e.g., Bertrand et al. 2010).

Field experiments can be instrumental in several ways. For example, an experiment

conducted in the field can be used to apply a previously established theory/framework—observe

the extent to which it applies to “real” life (VanEpps, Downs and Loewenstein 2016), identify

the effects size, and capture second-order effects, long-term effects, and so on (e.g., John et al.

2016; Putnam-Farr and Riis 2016). Alternatively, in some instances, starting with a field

experiment and, if necessary, following up with laboratory experiments to understand the process

underpinning the effect would make sense. Examples include research on voting behavior (e.g.,

Bryn, Walton, Rogers, and Dweck 2011), response to location-based promotional offers (e.g.,

Fong et al. 2015), and pay-what-you-want pricing (e.g., Gneezy et al. 2010; 2012). Finally,

researchers can use field experiments to pit several theories against one another, and observe the
 Field Experimentation 5

relative impact of each, thereby providing a more complete account of behavior within a given

domain (e.g., Sudhir, Roy, and Cherian 2016).

A Word on Field Experiments and External Validity

Researchers have argued that research conducted in naturally occurring settings offers

greater external validity (e.g., Calder, Phillips, and Tybout 1982; Winer 1999). Lynch (1999)

rejected this proposition and claims that similar to the lab, one real-world setting is unlike

another, and thus any claim about the generalizability of any type of experiment is spurious.

Instead, Lynch proposes that the key to understanding whether findings can be generalized is to

“have a theory that specifies moderator variables and boundary conditions and that specifies

what variables should not moderate the findings reported and to test for the asserted pattern of

interactions” (p. 368).

We concur with Lynch’s assertion that external validity depends on one’s theory, not

method. At the same time, we would like to note that experiments conducted in natural settings

can often uniquely tell us where and when certain phenomena are likely to occur, help refine

existing theories, and importantly, highlight their importance and likely impact (Campbell 1969;

Cialdini 1980, 2009; Lewin 1997; McGuire 1969).

FIELD EXPERIMENTS EXAMPLES

One of the most important decisions firms face is how to price their products or services.

Although the classic assumption in economic modeling is that firms indeed set profit-

maximizing prices, empirical evidence shows this assumption is often violated (e.g., Simon

1955). Ostensibly, a firm can identify the best price once it estimates the demand function—
 Field Experimentation 6

observing sales at different price levels—and subsequently chooses the profit-maximizing price.

Alternatively, one could estimate demand elasticity by employing econometric techniques using

historical price and sales data (Rosse 1970; Nevo 2001), which can then be used to derive a

demand function. Although elegant and simple, this approach hinges on the critical assumption

that the firm has been setting profit-maximizing prices, an assumption that even if true, restricts

price variation, thereby handicapping the econometric approach. Finally, historical demand is

often non-diagnostic of future demand. For example, the demand for VCR players was likely

stable at some point but is surely quite different now.

A cleaner, straightforward, and accurate approach for deriving optimal prices is to run a

field experiment—vary prices and observe demand. Notably, this approach does not rely on

assumptions and is flexible enough to detect and correct for fluctuations in demand over time.

Example 1: Price, Quality, and Demand

Gneezy, Gneezy, and Lauga (2014) studied the price-quality heuristic in an experiment in

which they varied the wine’s quality (high vs. low) and price ($10, $20, $40). Participants were

winery visitors who came for wine tasting: they received a printed page with the names and

prices of nine wines available for tasting, and chose six wines to taste. This design allowed the

authors to examine the effect of price and quality on liking, demand, revenue, and profits. The

main finding was that for the higher-quality wine, demand went up when the price increased

from $10 to $20, and went slightly down when the price was $40. By contrast, demand for the

lower-quality wine consistently decreased with price. The most profitable combination was $20

for the higher-quality wine. Using these findings, the winery priced its better Cabernet

Sauvignon at $20, resulting in an 11% profit increase.

 Field Experimentation 7

Note that if the same experiment were conducted in the lab, any conclusion regarding the

implications for demand, sales, and profitability would require some leap of faith and take the

form of informed speculation at best.

Example 2: Price and Perceived Fairness

On July 12, 2011, Netflix announced its decision to replace its membership plan, which

offered unlimited streaming and unlimited DVDs, with two distinct plans: unlimited streaming

and unlimited DVDs, priced at $7.99 each. This 60% price increase caused the company’s

market value to drop by over two thirds—$11 billion—in three months. The outrage of Netflix’s

customers was rooted in a deep sense of unfairness and disappointment, which could not be

captured by standard pricing strategies. A field experiment, however, could have uncovered the

effects of different prices on fairness perceptions, identified the pros and cons associated with

each, and allowed Netflix’s leadership to make a well-informed decision.

Anderson and Simester (2008) used a field experiment to investigate the impact of prices

on perceived fairness and sales. Collaborating with a women’s plus-size clothing catalog, the

authors mailed four versions of the catalog to potential customers, and varied the magnitude of

the price premium for the larger sizes. The results show demand decreased as the premium

increased, arguably due to perceptions of unfairness. Using actual sales data, the authors

estimated the cost of this sense of unfairness to the firm to be a 6%-8% decrease in gross profits.

KEY CONSIDERATIONS

In this section, we offer an overview of some of the key issues in designing and

implementing field experiments, and ways for addressing those issues.

 Field Experimentation 8

Context/Setting

In some cases, the nature of one’s research question necessarily dictates the experimental

context. For example, research looking at the effect of Facebook “liking” on subsequent attitudes

and behavior needs to be done on Facebook (John et al. 2016). Sometimes, however, researchers

have more flexibility in choosing the setting for their experiment, allowing them to conduct

independent field experiments (see below).

Experiment Type

Generally speaking, field experiments can be conducted with (collaborative) or without

(independent) a non-academic partner (e.g., organization/firm/seller). The decision of which way

to go is entirely the researcher’s, as long as it allows one to test predictions, proposed theories,

and so on. Note, however, that non-collaborative experiments are often limited to a one-time

interaction, meaning they do not lend themselves easily to second-order, spillover, and long-term

effects.

Independent experiments. Sometimes researchers can design and conduct an experiment

that possesses all the necessary features of a field experiment. For example, to investigate the

factors influencing behavior under pay-what-you-want pricing, researchers offered donuts

(Saccardo et al. 2016) and coffee (Jung et al. 2014) to passersby. Another example is the lost-

letter technique (Milgram, Mann, and Harter 1965), used to assess community orientation toward

political groups, human cooperation (Yezer, Goldfarb, and Poppen 1996), and other types of

social preferences. Needless to say, this freedom comes at a cost, so successfully identifying

those real-world behaviors and situations that best fit the research question is critical.
 Field Experimentation 9

Collaborative experiments. A research partner could take the form of any marketplace

actor interacting with consumers: a coffee cart manager (e.g., Jung et al. 2016), a university’s

human resource department (Grinstein and Kronrod 2016), hotel management (e.g., Baca et al.

2013; Goldstein, Cialdini, and Griskevicius 2008), government/public agencies (e.g., Yeager et

al. 2014), and so on. The general rule of thumb is that the smaller the organization, the more

quickly an understanding can be reached.

We strongly recommend developing an understanding of potential non-academic partners

before approaching them. One should know how they operate, what their challenges and needs

might be, and importantly, the potential costs and benefits of the proposed experiment for them.

Whereas highlighting potential gains is inherent to one’s persuasion efforts, we strongly

recommend proactively presenting potential caveats and discussing ways to address them.

Potential benefits. From the perspective of potential partners, the potential benefits

associated with running a field experiment with someone who knows what he or she is doing

might include increased sales and profits, and improved effectiveness of, for example,

promotions (Fong et al. 2015) and communications (Putnam-Farr and Riis 2016). Most

important, a well-planned and well-executed experiment can reveal what works (vs. not, or not

as well), under what conditions, and the associated ramifications (Bone et al. 2016). Note that in

our experience, pitching the importance of enhancing scientific knowledge is not convincing and

might be detrimental.

Potential costs. Understanding possible concerns and addressing them before

approaching a research partner is important. Common concerns include costs (e.g., time, money),

reputation and legalities (see the criticism of Facebook’s emotional-manipulation experiment by

Kramera, Guillory, and Hancock, 2014), and interference with operations.

 Field Experimentation 10

Design Tightness and Experimental Noise

As we have previously alluded to, field experiments often involve a significant loss of

control and an increased number of variables that need to be considered in the design phase, as

well as during data analyses. Consider the case of time effects: people are likely to feel they are

“richer” earlier in the month, right after receiving their paycheck, suggesting differential

preferences with respect to, for example, spending (Fong et al. 2015) and saving (Grinstein and

Kronrod 2016). Similarly, predicting the specific time of day (e.g., before or after lunch) could

substantially, and possibly systematically, influence participants’ preferences and behavior, and

should therefore be considered as well.

More generally, researchers should capture any variation in the experimental setting,

small or large (preferably in real time), determine its potential to undermine the experiment, and

take the steps necessary to address such concerns. Consider a study in which researchers offer

chocolate cupcakes using pay-what-you-want pricing to a local farmers’ market’s visitors over

four consecutive Sundays. Imagine that in week 3, outside temperatures hit 100 degrees

Fahrenheit, as opposed to 72 degrees Fahrenheit observed in prior weeks, significantly

decreasing the attractiveness of the goods, and thus consumers’ willingness to pay. The

researcher could take several steps to address this problem, such as pausing data collection and

resuming the following week, or continuing with data collection and controlling for temperature

in the analyses. The main takeaway is simple: researchers should develop and maintain a system

allowing them to capture and address (unexpected) situational factors in real time. One could

deal with some of these issues, such as time of day, by developing a detailed protocol that

defines randomization, data recording, documentation of irregularities, and so on. The more

prepared a researcher is, the better. However, even the most meticulous protocol is unlikely to
 Field Experimentation 11

account for all possible problems, which is why the presence of the researcher on the ground is

imperative, particularly during the first few sessions.

Finally, when planning a field experiment, researchers sometimes need to “give in”—

adjust and modify aspects that are otherwise considered the golden standard. For example,

(Putnam-Farr and Riis 2016) provide an account of compromises dictated by the circumstances

surrounding their Experiment 2: lack of complete randomization within each client’s pool of

employees and a slight variation in the rewards offered (and the text describing them) by each

client. Although largely undesired, hiccups of this type are inherent to the raw nature of the real

world, and should be acknowledged and addressed to the extent possible.

A FINAL NOTE

The goal of this paper is to demonstrate the importance of field experiments, and make

the case for their adoption in marketing research. The merits of field experiments are substantial,

but they often come with some costs. Our recommendation is clear and simple: researchers

interested in understanding a behavioral phenomenon—its antecedents, consequences, and

implications—should make field experiments an integral component of their everyday research-

methods toolkit. Lastly, note that the Journal of Marketing Research’s commitment to promoting

the adoption of behavioral field experimentation is aligned with what seems to be a general shift

in our field, as evidenced by editorials published by the Journal of Consumer Research

(Peracchio, Luce, and McGill 2014) and Marketing Science (Sudhir 2016).
 Field Experimentation 12

REFERENCES

Anderson, Eric and Duncan Simester (2003), “Effects of $9 Price Endings on Retail Sales:

Evidence from Field Experiments,” Quantitative Marketing and Economics, 1 (1), 93-110.

—— (2008), "Does Demand Fall When Customers Perceive That Prices Are Unfair? The Case

of Premium Pricing for Large Sizes," Marketing Science, 27 (3), 492-500.

Baca-Motes, Katherine, Amber Brown, Ayelet Gneezy, Elizabeth Keenan, and Leif D. Nelson

(2013), “Commitment and Behavior Change: Evidence from the Field,” Journal of Consumer

Research, 39 (5), 1070-1084.

Bertrand, Marianne, Dean S. Karlan, Sendhil Mullainathan, Eldar Shafir, and Jonathan Zinman

(2010), “What's Advertising Content Worth? Evidence from a Consumer Credit Marketing Field

Experiment,” The Quarterly Journal of Economics, 125 (1), 263-306.

Bone, Sterling A., Katherine N. Lemon, Clay M. Voorhees, Katie A. Liljenquist, Paul W.

Fombelle, Kristen B. DeTienne, and R. Bruce Money (2016), “Mere Measurement “Plus”: How

Solicitation of Open-Ended Positive Feedback Influences Customer Purchase Behavior,” Journal

of Marketing Research, XXX (X), XXX-YYY.

 Field Experimentation 13

Bryan, C. J., Gregory. M. Walton, Todd Rogers, and Carol S. Dweck (2011), “Motivating Voter

Turnout by Invoking the Self,” Proceedings of the National Academy of Sciences, 108 (31),

12653-12656.

Calder, Bobby, Lynn Phillips, and Alice Tybout (1982), “The Concept of External Validity,”

Journal of Consumer Research, 9 (3), 240-244.

Campbell, Donald (1969), “Reforms as Experiments,” American Psychologist, 24(4), 409-429.

Charness, Gary, Uri Gneezy, and Michael A. Kuhn (2013), "Experimental methods: Extra-

Laboratory Experiments-Extending the Reach of Experimental Economics," Journal of

Economic Behavior & Organization, 91, 93-100.

Cialdini, Robert (1980), “Full-cycle Social Psychology,” Applied Social Psychology, 1, 21-47.

—— (2009), Influence: Science and Practice. Boston: Allyn & Bacon.

Gneezy, Ayelet, Uri Gneezy, Gerhard Riener, and Leif D. Nelson (2012) “Pay-What-You-Want,

Identity, and Self-Signaling,” Proceedings of the National Academy of Sciences, 109 (19): 7236–

7240.

Gneezy, Ayelet, Uri Gneezy, and Dominique Lauga (2014), “Reference-Dependent Model of the

Price-Quality Heuristic,” Journal of Marketing Research, 51(2), 153-164.

 Field Experimentation 14

Goldstein, Noah. J., Robert Cialdini, and Vlad Griskevicius (2008), “A Room with a Viewpoint:

Using Social Norms to Motivate Environmental Conservation in Hotels,” Journal of Consumer

Research, 35 (3), 472-482.

Grinstein, Amir, and Ann Kronrod (2016), "Does Sparing the Rod Spoil the Child? How

Praising, Scolding, and an Assertive Tone Can Encourage Desired Behaviors," Journal of

Marketing Research, 53 (3), 433-441.

John, Leslie K., Emrich, Oliver, and Michael I. Norton (2016), “Does “Liking” Lead to Loving?

The Impact of Joining a Brand's Social Network on Marketing Outcomes,” Journal of Marketing

Research, XXX (X), XXX-YYY.

Jung, Minah H., Leif D. Nelson, Ayelet Gneezy, and Uri Gneezy (2014), “Paying More When

Paying for Others,” Journal of Personality and Social Psychology, 107 (3), 413-413.

Jung, M. H., Perfecto, H., and Nelson, Leif D. (2016), “Anchoring in Payment: Evaluating a

Judgmental Heuristic in Field Experimental Settings,” Journal of Marketing Research, 53(3),

354-368.

Lewin, Kurt (1997), “Problems of Research in Social Psychology,” in Resolving Social Conflicts

& Field Theory in Social Science, Washington DC: American Psychological Association, 279-

288.
 Field Experimentation 15

Li, Jimmy Q., Paat Rusmevichientong, Duncan Simester, John N. Tsitsiklis, and Spyros I.

Zoumpoulis (2015), "The value of field experiments," Management Science, 61 (7), 1722-1740.

Lynch, John Jr. (1999), “Theory and External Validity,” Journal of the Academy of Marketing

Science, 27 (3), 367-376.

McGuire, William (1969), “Theory-Oriented Research in Natural Settings: The Best of Both

Worlds for Social Psychology,” in Interdisciplinary Relationships in the Social Sciences, 21-51.

Milgram, Stanley, Leon Mann, and Susan Harter (1965), "The lost-letter Technique: A Tool of

Social Research," Public Opinion Quarterly 29 (3), 437.

Nevo, Aviv (2001), “Measuring Market Power in the Ready-to-Eat Cereal Industry,”

Econometrica, 69 (2), 307-342.

Paluck, Levy and Robert Cialdini (2014), “Field Research Methods.” in Handbook of Research

Methods in Social and Personality Psychology, Harry Reis and Charles Judd, eds. New York:

Cambridge University Press, 81-100.

Peracchio, Laura A., Mary Frances Luce, and Ann L. McGill (2014), "Building Bridges for an

Interconnected Field of Consumer Research," Journal of Consumer Research, 40 (6), v-viii.

 Field Experimentation 16

Putnam-Farr, Eleanor, and Jason Riis (2016), "Yes/No/Not Right Now”: Yes/No Response

Formats Can Increase Response Rates Even in Non-Forced-Choice Settings," Journal of

Marketing Research 53 (3), 424-432.

Rosse, James (1970), "Estimating Cost Function Parameters without Using Cost Data -

Illustrated Methodology," Econometrica, 38 (2), 256-275.

Saccardo Silvia, Anya Samek, Charis Lee, and Ayelet Gneezy (2016), “Paying Responsibly:

Framing and Payments in Consumer Elective Pricing,” Working paper.

Shu, Suzanne and Ayelet Gneezy (2010), “Procrastination of Enjoyable Experiences,” Journal of

Marketing Research, 47 (5), 933-944.

Simon, Herbert (1955), “A Behavioral Model of Rational Choice.” Quarterly Journal of

Econometrics, 69 (1), 99-118.

Sudhir, K. (2016), “Editorial—The Exploration-Exploitation Tradeoff and Efficiency in

Knowledge Production," Marketing Science, 35 (1), 1-9.

Sudhir, K., Roy Subroto, and Mathew Cherain (2016), “Do Sympathy Biases Affect Charitable

Giving? The Effects of Advertising Content," Marketing Science, forthcoming.

 Field Experimentation 17

VanEpps, Eric M., Julie S. Downs, and George Loewenstein (2016), "Advance Ordering for

Healthier Eating? Field Experiments on the Relationship Between the Meal Order–Consumption

Time Delay and Meal Content," Journal of Marketing Research 53 (3), 369-380.

Winer, Russell (1999), “Experimentation in the 21st Century: The Importance of External

Validity,” Journal of the Academy of Marketing Science, 27 (3), 349-358.

Yeager, David. S., Valerie Purdie-Vaughns, Julio Garcia, Nancy Apfel, Patti Brzustoski, Alison

Master, A., William T. Hessert, Mattew E. Williams and Geoffrey L. Cohen (2014), “Breaking

The Cycle of Mistrust: Wise Interventions to Provide Critical Feedback Across the Racial

Divide,” Journal of Experimental Psychology: General, 143 (2), 804-824.

Yezer, Anthony M., Robert S. Goldfarb, and Paul J. Poppen (1996), "Does Studying Economics

Discourage Cooperation? Watch What We Do, Not What We Say or How We Play." The

Journal of Economic Perspective, 10 (1), 177-186.