AMA Marketing Effectiveness Online Seminar Series

Lynette Rowlands American Marketing Association

A wealth of information is available for marketing professionals at

www.MarketingPower.com
The #1 marketing site on the web

Proprietary and Confidential, Maritz Inc.

Commonly Asked Questions

1. Will I be able to get copies of the slides after the event?

2.

Is this web seminar being taped so I or others can view it after the fact?

Proprietary and Confidential, Maritz Inc.

Commonly Asked Questions

1. Will I be able to get copies of the slides after the event?

Yes
2. Is this web seminar being taped so I or others can view it after the fact?

Yes

Proprietary and Confidential, Maritz Inc.

Introducing Today s Speaker

Using Importance Measurement to Drive Product Improvements

Keith Chrzan Vice President, Marketing Sciences Maritz Research

Proprietary and Confidential, Maritz Inc.

Introduction

Agenda
   

Introduction Stated Importance Methods Derived Importance Methods Summary

Proprietary and Confidential, Maritz Inc.

Introduction

Why Measure Importance?

 Products or services can be thought of as bundles of attributes (properties, features, benefits, etc.)  Marketers usually cannot afford to optimize all attributes at once, so they must prioritize  In order to prioritize, marketers must know the relative importance of the attributes, hence the need for importance measurement  May be part of customer satisfaction, image/positioning, brand choice, loyalty, concept testing or other studies

Proprietary and Confidential, Maritz Inc.

Introduction

Myers and Alpert s Clarification

 Salience attribute is easily brought to mind  Importance attribute is important  Determinance attribute is important and brands differ on it, so that it determines preference or choice

Proprietary and Confidential, Maritz Inc.

Introduction

Stated Importance
 Direct questioning methods
 e.g. Open-ends, rating, ranking, sorts, etc.

 Respondents answers directly tell us what attributes are more important than others

Proprietary and Confidential, Maritz Inc.

Introduction

Derived Importance
 Indirect questions
 Respondent describes her actual experience  Respondent evaluates that experience  We infer importance by relating descriptions to the evaluations  We apply statistical predictive models to respondent-supplied attribute and evaluative judgments and use statistical outputs as measures of importance

Proprietary and Confidential, Maritz Inc.

Stated importance

Measuring Attribute Importance

 Introduction  Stated Importance Methods
 Unconstrained methods  Constrained methods

 Derived Importance Methods  Summary

Proprietary and Confidential, Maritz Inc.

Stated importance

Objectives
 As a result of this section, you will be able to
 Describe two broad classes of stated importance measures  List seven specific kinds of stated importance measurement  Identify the strengths and weaknesses of the various kinds of stated importance measurement  Determine the best stated importance measure for your project using a decision tree

Proprietary and Confidential, Maritz Inc.

Stated importance

Measuring Attribute Importance

 Introduction  Stated Importance Methods
 Unconstrained methods
 Open-end questions  Importance ratings

 Constrained methods

 Derived Importance Methods  Summary

Proprietary and Confidential, Maritz Inc.

Stated importance

Open-End Questions
 Example: What attributes are important when choosing a <widget>?  Advantages
 Simple to ask in any survey modality (mail, phone, Web, etc.)  Question does not force response categories on respondent researcher may learn about new attributes

 Disadvantages
 Importance and recall are confounded  Open-ends may measure salience more than importance

 Recommendation: Use for exploratory research, not for importance measurement

Proprietary and Confidential, Maritz Inc.

Stated importance

Importance Ratings
 Example: When choosing a <widget> how important is
Not at all Important Fast service Wide variety Reliability Ease of use Country of origin Package color 1 1 1 1 1 1 2 2 2 2 2 2 3 3 3 3 3 3 4 4 4 4 4 4 Extremely Important 5 5 5 5 5 5

Proprietary and Confidential, Maritz Inc.

Stated importance

Importance Ratings
 Advantages
 Easy to ask in any survey modality  Respondents are familiar and comfortable answering importance ratings  Clients are familiar and comfortable receiving importance ratings

Proprietary and Confidential, Maritz Inc.

Stated importance

Importance Ratings
 Disadvantages
 Respondents tend to rate most attributes very positively this reduces ability to discriminate among them and hampers subsequent analyses  Scale use heterogeneity
 Some respondents use high part of the scale others use low part positional heterogeneity  Some respondents use a wider range of the scale than do others heteroskedasticity  These response effects hampers interpretation and multivariate analyses and harms cross-cultural comparisons

 Recommendation: Use as a last resort

Proprietary and Confidential, Maritz Inc.

Importance Ratings
 Empirical evidence against importance ratings
 Discriminant analysis with brand used as dependent variable and importance ratings as predictors seldom makes sense, and it should, if importances are meaningful  Derived importance works like this: You model some dependent variable (Y) as a linear function of some performance measures (X) and you calculate coefficients that are proxies for importance  If you do the reverse, modeling Y as a function of importance ratings, the coefficients should be measures of performance, but they usually are uncorrelated with actual measures of performance

 Importance ratings are next to worthless

Proprietary and Confidential, Maritz Inc.

Stated importance

Measuring Attribute Importance

 Introduction  Stated Importance Methods
 Unconstrained methods  Constrained methods
     Rank order Q-sort Constant sum Method of paired comparisons Maximum difference scaling

 Derived Importance Methods  Summary

Proprietary and Confidential, Maritz Inc.

Stated importance

Rank Ordering
 Example: Please rank these 11 features of <widgets>. Give the most important feature a 1, the next most important feature a 2, and so on until the least important feature has a rank of 11.  Advantages
 Works well on mail, Web or in-person surveys if attribute list is short  Response distribution is constrained so rank information is standardized for use in cross-cultural comparisons
 All respondents scores have the same mean  All respondents scores have the same variance

Proprietary and Confidential, Maritz Inc.

Stated importance

Rank Ordering
 Disadvantages
 Difficult to administer in phone surveys  Difficult to administer if attribute list is long  Non-parametric statistical tests for rank orders are relatively weak, so less discriminating than even importance ratings

 Recommendation: Do not use

Proprietary and Confidential, Maritz Inc.

Stated importance

Q-Sort
 Example: If 17 attributes are to be evaluated, respondent is instructed to sort them so that
     1 is in a pile for the most important attribute 3 are in a pile for the next most important attributes 1 is in a pile for the least important attribute 3 are in a pile for the next least important attributes 9 are implicitly sorted into the pile of middle importance

 Resulting distribution is 1:3:9:3:1

Proprietary and Confidential, Maritz Inc.

Stated importance

Q-Sort
 Advantages
 Forced distribution standardizes responses, making this a viable technique for cross-cultural comparisons  Discriminating  Works in face-to-face, mail and Web surveys

 Disadvantages
 Will not work in phone interviews  Time consuming task if the number of attributes is large

 Recommendation: May use if there are not too many attributes and data collection is not phone

Proprietary and Confidential, Maritz Inc.

Stated importance

Constant Sum Allocation

 Example: Please assign 11 points to these attributes according to how important they are to you when choosing a <widget>. You can assign some, none, or all 11 points to a given attribute, as long as the total number of points you assign is 11.
Fast service Wide variety Reliability Ease of use Package color Country of origin TOTAL _____ _____ _____ _____ _____ _____ 11

Proprietary and Confidential, Maritz Inc.

Stated importance

Constant Sum Allocation

 Advantages
 Ratio measurement of importance  Discriminating trade-off prevents all attributes from being important

 Disadvantages
 Difficult to do in telephone interviews unless attribute list is very short  Difficult to administer at all if attribute list is long  Unknown scale use heterogeneity  Unknown ability to standardize cross-cultural studies

 Recommendation: Perhaps use with short attribute lists

Proprietary and Confidential, Maritz Inc.

Stated importance

Method of Paired Comparisons

From Thurstone (1927) Updated design theory by David (1988) Updated analysis via hierarchical Bayesian analysis Ask attributes two at a time, forcing respondent to choose which is more important  Ask 1.5 times as many pairs as there are attributes
   

Proprietary and Confidential, Maritz Inc.

Stated importance

Method of Paired Comparisons

 Example:  Is ease of use or reliability more important when choosing a <widget>?  Is reliability or package color more important when choosing a <widget>?  Is variety or fast service more important when choosing a <widget>?

Proprietary and Confidential, Maritz Inc.

Stated importance

Method of Paired Comparisons

 Advantages
 Easy to administer in any survey modality  VERY discriminating  Automatically standardized for cross-cultural comparisons (no scale-use bias)  HB analysis produces individual level importances  Ideal for input to needs-based segmentation  If well balanced (all attributes occur equally often with each other) analysis is simple: importance is proportional to percentage of times an attribute is chosen when it is available

Proprietary and Confidential, Maritz Inc.

Stated importance

Method of Paired Comparisons

 Disadvantages
 A paired comparison question takes about 50% longer for respondent to answer in a phone survey, so that a 2 minute importance rating battery becomes a 3 minute paired comparison battery, all else being equal

 Recommendation: Good method, use when possible

Proprietary and Confidential, Maritz Inc.

Stated importance

Maximum Difference Scaling

 Multiple choice extension of MPC  3+ attributes per question; respondent picks most and least important  Example:
Which of the following is least important when you buy a widget and which is most important? Least [] [] [] [] Most [] [] [] []

Ease of use Country of origin Package color Reliability

Proprietary and Confidential, Maritz Inc.

Stated importance

Maximum Difference Scaling

 Advantages
      Handles more attributes with fewer questions than MPC HB analysis produces individual level importances Even MORE discriminating than MPC Automatically standardized for cross-cultural comparisons Ideal for input to needs-based segmentation If well balanced, analysis is simple: importance is the log of the number of times chosen divided by the number of times available

Proprietary and Confidential, Maritz Inc.

Stated importance

Maximum Difference Scaling

 Disadvantages
 Requires visual presentation of stimuli (i.e. paper and pencil or Web survey)  May require analysis to produce importances

 Recommendation: Good method, use when possible

Proprietary and Confidential, Maritz Inc.

Stated importance

Stated Importance Decision Tree

Proprietary and Confidential, Maritz Inc.

Derived importance

Measuring Attribute Importance

   

Introduction Stated Importance Methods Derived Importance Methods Summary

Proprietary and Confidential, Maritz Inc.

Derived importance

Objectives
 Following this section, you will be able to
 Identify four types of derived importance models, in two broad classes  Use a decision tree to settle on a derived importance model for your project  Identify the strengths and weaknesses of the various kinds of derived importance measurement

Proprietary and Confidential, Maritz Inc.

Derived importance

Overview - Derived Importance Measures

 Respondent evaluates a brand/product using
 A rating (satisfaction, performance, liking, purchase intent)  A share (market share, share of preference, an allocation)  A choice of one brand/product/therapy over others

 Respondent rates product on multiple attributes  Predictive statistical model relates attributes to the evaluation  Model yields coefficients as proxies for importance  Correlation and choice-based methods

Proprietary and Confidential, Maritz Inc.

Derived importance

Measuring Attribute Importance

 Introduction  Stated Importance Methods  Derived Importance Methods
 Correlation-based Methods
 Correlation  Regression  True Driver Analysis

 Choice-based Methods

 Summary

Proprietary and Confidential, Maritz Inc.

Variance
 Correlation-based methods (correlation, regression, True Driver Analysis) depend on variance patterns
 Importance just means shared variance with the dependent variable  If an attribute doesn t share variance with a dependent variable, it can t be important

 Attributes which do not vary, or that vary only randomly, cannot share variance with the dependent variable, and so cannot be important
 e.g. cost of entry attributes like 4 wheels on a car or airline safety

Proprietary and Confidential, Maritz Inc.

Derived importance

Correlation
 Description: Bivariate correlations of evaluation with each attribute in turn  Advantages
 Easy to do  Coefficients are unaffected by multicollinearity

 Disadvantages
 Lack of statistical control because attributes are analyzed in isolation
 Importance can be double (or triple or more) counted to the extent attributes are correlated with one another  No composition rule for coefficients, so model cannot support simulation

 Scale use heterogeneity can spuriously inflate all correlations

Proprietary and Confidential, Maritz Inc.

Derived importance

Correlation
 What correlation does

Proprietary and Confidential, Maritz Inc.

Derived importance

Multiple Regression
 Description: All attributes are used simultaneously to predict the evaluation; coefficients or standardized coefficients are importances  Advantages
 Accessible technique learned in school that clients are familiar with  Easy to do

 Disadvantage - multicollinearity (intercorrelation of independent variables) is omnipresent in survey research and has a pernicious effect on regression coefficients
 It can distort the size and even the sign of coefficients  Makes coefficients unstable

Proprietary and Confidential, Maritz Inc.

Derived importance

Multiple Regression
 What regression does

Proprietary and Confidential, Maritz Inc.

Derived importance

Multiple Regression
 Why multicollinearity is bad news for regression

Proprietary and Confidential, Maritz Inc.

Derived importance

A Middle Ground
 Kruskal came up with a way of computing importance by doing regression analysis and averaging over orderings  Theil added an information-theoretic framework for this averaging over orderings  This idea works so well to solve the shortcomings of both correlation and regression that we ve built it into a family of techniques for importance measurement that we call True Driver Analysis

Proprietary and Confidential, Maritz Inc.

Derived importance

True Driver Analysis (TDA)

 Advantages
 Immune to multicollinearity  Importances are . . .
 Additive  Ratio scaled  Intrinsically meaningful

 Disadvantages
 Complex programming

Proprietary and Confidential, Maritz Inc.

Derived importance

Measuring Attribute Importance

 Introduction  Stated Importance Methods  Derived Importance Methods
 Correlation-based Methods  Choice-based Methods
 MNL

 Summary

Proprietary and Confidential, Maritz Inc.

Logit vs Regression
 In regression, the unit of analysis is a brand and its ratings:

Obs. 1

Overall 4

Att 1 5

Att 2 3

Att 3 2

Att 4 5

Proprietary and Confidential, Maritz Inc.

Logit vs Regression
 In MNL, the unit of analysis is a choice, so we collect multiple brands ratings:

Obs. 1 1 1 1

Chosen 0 0 1 0

Att 1 5 3 4 5

Att 2 3 2 5 4

Att 3 2 2 1 4

Att 4 5 3 1 2

Proprietary and Confidential, Maritz Inc.

Derived importance

Multinomial Logit (MNL)

 Description
 Evaluation is brand preferred over others (choice or share)  Models this relative preference as a function of attributes of all competing brands  Between-brand differences drive the model (as they drive choice in the real world)  MNL coefficients are significant when differences between brand ratings relate to brand choice i.e. it measures determinance

Proprietary and Confidential, Maritz Inc.

Derived importance

Multinomial Logit (MNL)

 Advantages
 Model built from between-brand differences in attribute ratings, so scale use heterogeneity is not a problem  For the same reason, less likely to be affected by multicollinearity  Explicitly includes competitive context  Specifically answers the oft-asked question Why do customers choose one product over another (or competitors products over mine, or mine over theirs )

Proprietary and Confidential, Maritz Inc.

Derived importance

Multinomial Logit (MNL)

 Disadvantages
 Because we need to ask every respondent about several brands attributes, questionnaire can get long if there s a lot more in it

Proprietary and Confidential, Maritz Inc.

Derived Importance Decision Tree

Proprietary and Confidential, Maritz Inc.

Derived importance

Measuring Attribute Importance

   

Introduction Stated Importance Methods Derived Importance Methods Summary

Proprietary and Confidential, Maritz Inc.

Stated Vs. Derived Importance

 Fairly often, stated and derived measures give conflicting views of what is important

Proprietary and Confidential, Maritz Inc.

Back to Myers and Alpert

 One reason stated and derived methods may not agree is that they are not even measuring the same thing
 Open-end measures confound the measurement of salience and importance  Other stated methods may measure importance, though there is good reason to believe attribute importance ratings do not do so very well  MNL measures determinance, NOT just importance  Regression-based derived importance methods measure a cross-sectional version of determinance

Proprietary and Confidential, Maritz Inc.

Stated Vs. Derived Importance

 Among stated importance methods
 Simpler approaches, like importance ratings and rankings, have serious shortcomings  More complex methods (MPC, MaxDiff) overcome at least some of the shortcomings
 MPC for phone surveys  MaxDiff for mail, in-person or Web-based surveys

Proprietary and Confidential, Maritz Inc.

Stated Vs. Derived Importance

 Among derived importance methods
 Simple approaches (correlations, regression) just aren t good enough  More complex methods (TDA, MNL) fix a lot of the shortcomings
 TDA for customer satisfaction and concept testing  MNL for brand choice/brand preference

Proprietary and Confidential, Maritz Inc.

Q&A

Proprietary and Confidential, Maritz Inc.

Thanks for your time and participation today!

To replay this webcast (available September 10): go to www.MarketingPower.com/ResearchSeries For copies of todays presentation: www.maritzresearch.com/measurement or (877) 4 MARITZ To contact todays speaker: Keith Chrzan keith.chrzan@maritz.com Questions for AMA: Lynette Rowlands lrowlands@ama.org

Proprietary and Confidential, Maritz Inc.