Beruflich Dokumente
Kultur Dokumente
JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted
digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about
JSTOR, please contact support@jstor.org.
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
Qiarterly ^h^hhh
Estimating the Effect of Common Method Variance:
The Method-Method Pair Technique with an
Victoria 3010
AUSTRALIA
rajeevs@unimelb.edu.au
Philip Yetton
Australian School of Business
AUSTRALIA
p.yetton@unsw.edu.au
Jeff Crawford
Introduction ^ i
Tulsa, OK 74104
U.S.A.
jeff-crawford@utulsa.edu
Abstract
^etmar Straub was the accepting senior editor for this paper. Jonathan
Trower served as the associate editor. in leading IS journals. In contras
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
Method Variance
Multitrait-Multimethod Techniques
While the existence of CMV is widely acknowledged, tech
niques used to estimate its effect in individual research
domains have limitations. Two broad categories of techni
ques have been proposed to control for the effect of CMV.
relationship (Cote and Buckley 1987; Podsakoff et al. 2003), although few
studies report observing this effect. Doty and Glick (1998, p. 401) analyzed
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
Figure 1. Nomological Network of a Relationship Including the Effect of CMV (Adapted from Podsakoff
et al. [2003, Figure 3A, Table 5, p. 896] and Le et al. [2009, Figure 1, p. 178])
2003).
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
domains.
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
The critical insight here is that observed effect sizes vary with
Susceptibility of Method-Method
Pairs to CMV
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
respondents.
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
Actual data are obtained from historical records and other objective sources, including usage
records captured by a computer system.
Example items: Computer generated records of "time spent on the computer" or "number of
Behavioral Continuous
Items refer to specific behaviors or actions that people have carried out. Responses are
generally captured on a continuous open-ended scale.
Example items: "How many e-mail messages do you send on a typical day?"; "How many
hours did you spend last week on this system?"; "What percentage of your time did you spend
on this application?"
Behaviorally Anchored
Items refer to specific actions that people have carried out. Responses are generally captured
on scales with behavioral anchors, such as "Never-More than once a day."
Example items: "How often do you use this system?" captured on a scale with anchors
ranging from "Not at all" to "Very often."
Perceptually Anchored
Anchors)
Susceptibility to
CMV Due to
Abstractness of
Measures
CMV[MMP-I]
Very low
Very low
Very low
Very low
Very high
Low
Low
Low
Very high
High
High
High
Very high
Very high
Very high
Very high
Susceptibility to
CMV Due to Data
Susceptibility to CMV
(Respondent)
Source
Method-Method Pair
Due to Response
Format (Scales and
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
r? = + u? + e?
where
population of studies
Ui= Effect of between-studies differences on the correla
tion coefficient of Study i
ej = Within-studies error
Both e? and u? are normally and independently distributed
279). Formally,
where
number of studies)
e? = Within-studies error
Both e? and u? are normally and independently distributed
1996).
forU),
(b) Mean effect size (correlation) (perceptually an
chored method for PU with behavioral continuous
An Illustration
Sample
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
appropriate.
Appendix B.
The most frequent reason for excluding a study from the
sample was that it did not report the PU-U correlation, or a
of the mean.
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
0.36
Mean correlation1
0.06 to 0.66
0.04 to 0.68
390
153.43
Fail-safe
0.028 (0.168)
0.005 (0.071)
0.023 (0.152)
17.8%
82.2%
^he mean correlation reported here is sample-size-weighted mean of the observed correlations. The mean correlation after correcting correlations
reported in individual primary studies for reliability of PU and applying inverse variance weights (Hunter and Schmidt 2004, p. 124) is r = 0.38. The
mean value of reliability (see Appendix for details of reliability data) of perceived usefulness is used where reliability data are not available. Data
for reliability of usage are available for only 26 of the 75 data sets; hence, the reported correlations are not corrected for reliability of usage. See
variance.
the three raters (Fleiss 1971) is 0.91 (p < 0.01). The dif
Control Variables
the item level rather than at the construct level, this rating was
were obtained from each individual study and each item was
Table 4.
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
CMV[MMP-1]
Very low
Low
Medium
18
Behaviorally Anchored
High
Perceptually Anchored
Very high
Total
31
13
"T?"
Factor
Publication Type
Respondent Type
Perceived Usefulness Type
CMV[MMP-I]
Error
Sum of
Squares
2.19
9.07
14.30
187.84
121.49
df
67
Variance
Mean
Explained
Square
1.21
5.00
7.88
25.90
2.19
9.07
14.30
46.96
1.81
Significance
(p value, two
(Eta Squared)
0.65%
2.71%
4.27%
56.09%
36.28%
tail)
0.276
0.029
0.007
0.000
Findings
Table 5 reports the results of a random effects ANOVA, in
4This analysis is conducted after correcting the observed correlations for the
reliability of PU. Each data point is weighted by its inverse variance weight.
The inverse variance weight employed for each study is its sample size
adjusted for the reliability of PU (Hunter and Schmidt 2004, pp. 122-132).
The results are similar when uncorrected correlation values are employed in
this analysis, with CMV[MMP-I] explaining 54.5% of the between-studies
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
System-captured
Contrast
Significance
(p value, two-tail)*
0.16
Behavioral Continuous
(CMV[MMP-I] = Low)
Behaviorally Anchored
(CMV[MMP-I] = High)
Perceptually Anchored
(CMV[MMP-I] = Very High)
0.29
< 0.020
0.42
< 0.015
0.59
< 0.015
*The fifth category of usage identified during the rating process, mixed behavioral continuous and behaviorally anchored (CMV[MMP-I] = Medium)
is excluded from this analysis as no contrasts are hypothesized for this category. Mean correlation for this category is 0.38, which lies between
(p<0.02).
Discussion
ship in A research.
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
measures.
Limitations
The MMP technique developed and illustrated in this paper is
methods, ranging from low to high CMV[MMP]. BothBehavioral Continuous > Objective Measure
conditions are frequently satisfied.
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
CMVn
OM
CMVr
CMVp
CB
CMVR
CMVR
BA
CMVp
CMVp
CMVp
CMVp
PA
Note: CMVBB refers to the susceptibility to CMV between a Behaviorally Anchored (B) measure and another Behaviorally Anchored (B) measure;
CMVpB refers to the susceptibility to CMV between a Perceptually Anchored (P) measure and a Behaviorally Anchored (B) measure; and so on.
to CMV of the two methods in the method-method pair,estimate of the construct-level correlation (Hunter and
Appendix E presents some alternative assumptions to rank
Schmidt 2004; Le et al. 2009; Schmidt et al. 2003). This issue
order the method-method pairs in Table 7. However, the rankis beyond the scope of this paper but is addressed briefly in
CMV[MMP] partitioned into two components:for example, Cronbach 1995). Trait and method are con
CMV[MMP-I] and CMV[MMP-T]. However, we do notsidered so intertwined that a method of measuring a trait is
A related limitation is that system-captured measures con155). Researchers need to consider the definitions of con
found source-based CMV effects and temporal separationstructs and methods as they investigate the effects of method
based CMV effects. This arises from the fact that system
on cumulative findings within a research stream.
captured and other objective measures frequently aggregate
archival data over a period of time (see, for example, Straub
niques to separate the confounding effects of CMV and tranImplications for IS Research
sient error (Le et al. 2009). In the present study, we investi
gated this confounding effect by repeating the analysisThe findings of this study have specific implications for IS
ported even when these data points are excluded from the those obtained from the analysis of MTMM-based studies in
the social sciences. For example, Podsakoff et al. report, as
analysis.
Third, further work is needed on techniques to estimate the
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
The results presented here are similar. Table 6 reports that the
mean PU-U correlation, when both are measured on percept
Malhotra et al.
6See Appendix C for the results and for details of the technique employed to
estimate the construct-level correlation controlling for the effect of CMV.
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
Conclusion ?HH1HHI
Acknowledgments
References
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
740-755.
Erlbaum.
pp. 206-224.
Sharma, R., and Yetton, P. 2003. "The Contingent Effects of
Management Support and Task Interdependence on Successful
Information Systems Implementation," MIS Quarterly (27:4), pp.
533-556.
AIS ( ), pp 380-427.
Straub, D., Limayem, M., and Karahanna-Evaristo, E. 1995.
"Measuring System Usage: Implications for IS Theory Testing,"
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms
462-468.
universities.
This content downloaded from 14.139.157.20 on Fri, 19 Aug 2016 05:17:31 UTC
All use subject to http://about.jstor.org/terms