Measuring Information Technology Payoff:

A Meta-Analysis of Structural Variables in

Firm-Level Empirical Research
Rajiv Kohli Sarv Devaraj*
Department of Management, University of Notre Dame, Notre Dame, Indiana 46556
rkohli@nd.edu sdevaraj@nd.edu
P
ayoffs from information technology (IT) continue to generate interest and debate both
among academicians and practitioners. The extant literature cites inadequate sample size,
lack of process orientation, and analysis methods among the reasons some studies have shown
mixed results in establishing a relationship between IT investment and firm performance.
In this paper we examine the structural variables that affect IT payoff through a meta-
analysis of 66 firm-level empirical studies between 1990 and 2000. Employing logistic regres-
sion and discriminant analyses, we present statistical evidence of the characteristics that dis-
criminate between IT payoff studies that observed a positive effect and those that did not. In
addition, we conduct ordinary least squares (OLS) regression on a continuous measure of IT
payoff to examine the influence of structural variables on the result of IT payoff studies.
The results indicate that the sample size, data source (firm-level or secondary), and industry
in which the study is conducted influence the likelihood of the study finding greater improve-
ments on firm performance. The choice of the dependent variable(s) also appears to influence
the outcome (although we did not find support for process-oriented measurement), the type
of statistical analysis conducted, and whether the study adopted a cross-sectional or longitu-
dinal design. Finally, we present implications of the findings and recommendations for future
research.
(Meta-Analysis) Information Teehnology Payoff; Business Value IT; Firm-Level; Discriminant Anal-
ysis; Logistic Regression; Process-Orientation)
1. Introduction
Researchers and business managers consider infor-
mation technology (IT) investment as an enabler for
improved organizational efficiency and competitive-
ness. Measurable performance improvements result-
ing from IT investment can help sustain investment in
future IT initiatives. However, as demand for IT in-
vestment increases, the assumed payoff is likely to
come under scrutiny. Although the much talked about
"productivity paradox" (Ahituv and Giladi 1993,
"Both authors contributed equally to the manuscript.
Roach 1987, Strassmann 1985) has largely been put to
rest by recent studies (Brynjolfsson and Hitt 2000,
Jorgenson 2001, Jorgenson and Stiroh 2000, Kraemer
and Dedrick 2001, Oliner and Sichel 2000), not all
studies have demonstrated clear payoff from IT
investment.
Among the reasons attributed to equivocal results of
past studies are structural issues such as inadequate
measurement and analysis methodologies (Brynjolfsson
1993, Robey and Boudreau 1999) and time lags in mea-
suring payoff (Devaraj and Kohli 2000a). Suggestions
to examine IT payoffs include improving the quality
1047-7047/03/1402/0127$05.00
1526-5536 electronic ISSN
INFORMATION SYSTEMS RESEARCH, 2003 INFORMS
Vol. 14, No. 2, June 2003, pp. 127-145
KOHLI AND DEVARAJ
Measuring Information Technology Payoff
of data and the analytical rigor (Robey and Boudreau
1999), examining valuation and conversion barriers
(Chircu and Kauffman 2000, Davern and Kauffman
2000), applying improved modeling techniques (Hitt
and Brynjolfsson 1996), and examining intermediate
and context-related variables {Barua et al. 1996, Barua
and Mukhopadhyay 2000). Similar calls for improving
execution of payoff studies and to improve reliability
of results have been made by past review papers and
syntheses of IT payoff studies (Brynjolfsson and Yang
1996, Mahmood et al. 1999, Sircar et al. 1998). There-
fore, there is need for a systematic analysis to under-
stand the structural characteristics of past IT payoff
studies and how they affect their outcomes. Such find-
ings will not only help critically view results of past
studies, they will also serve as a guide for future
research.
We report a meta-analysis of firm-level IT payoff
studies to catalog the extant literature, examine vari-
ables that influence the findings, and provide sugges-
tions for conducting future studies. In conducting this
meta-analysis, we examine the structural dimensions
along which IT payoff studies differ. The analysis at-
tempts to observe a pattern, if any, which discrimi-
nates between studies that result in positive IT payoff
and those that do not. Furthermore, we investigate the
extent of payoff resulting from such structural
dimensions.
The paper is organized as follows. In 2, we briefly
review the meta-analysis studies in the IT payoff lit-
erature and present a literature-guided framework to
review the IT payoff studies. In 3, we present details
of the procedure adopted for the meta-analysis. In 4,
we present results of the statistical analysis of IT payoff
studies. Finally, 5 presents our findings of the influ-
ence of structural variables, followed by limitations,
contributions, and areas for future research.
2. Review of Literature
2.1. Review of IT Payoff Literature
Past firm-level studies of IT payoff can be viewed as
addressing three general questionsWhat is mea-
sured, hoxv is it measured, and zvhere is it measured?
(Banker et al. 1993, Berger et al. 1988, Mahmood and
Szewczak 1999). In what is measured, past studies pro-
pose that IT performance is associated with variables
that transcend traditional measures and include mea-
sures of productivity, in addition to profitability
(Mahmood and Mann 2000). Although there is not a
consistent set of performance variables, even when
measurement variables are identified, the quality and
completeness of data and the subsequent robustness of
the analysis appears to impact the outcome of the IT
payoff studies (Brynjolfsson and Yang 1996, Mahmood
et al. 1999). In other words, the data source and anal-
ysis approach have a bearing on the IT payoff result.
Study characteristics, such as duration of data col-
lection and the process of IT investment, describe how
the data are gathered. We find that some studies gather
data at one point in time (Prattipati and Mensah 1997),
while others collect three to five annual data points
(Barua et al. 1995, Dewan and Min 1997, Hitt and
Brynjolfsson 1996, Prasad and Harker 1997). The du-
ration of studies, combined with the number of firms,
determines the sample size or data points captured in
a study. The process of IT investment can be examined
to assess if appropriate IT assets and impacts were
measured (Soh and Markus 1995). This process mea-
surement view proposes that IT expenditures have to
be converted into appropriate IT assets. The appropri-
ate use of IT assets leads to IT impacts, and IT impacts
when positioned competitively, lead to impacts on or-
ganizational performance (Lucas 1993, Mooney et al.
1996, Soh and Markus 1995). Even when IT spending
is shown to improve intermediate variables of orga-
nizational productivity, such as improved communi-
cation leading to the need for reduced inventories
(Dudley and Lasserre 1989), it does not necessarily
lead to improvements in productivity (Barua et al.
1991).
On the question of lohere measurements for IT payoff
should occur, prior studies indicate that payoff has
been harder to measure in some industries than others.
Furthermore, studies that use firms as the data source
are likely to show a positive relationship between IT
investment and firm performance because of the com-
pleteness and availability of required variables (Hitt
and Brynjolfsson 1996, Sircar et al. 1998).
Although greater payoffs among firm-level studies
are generally expected (Brynjolfsson and Yang 1996,
Devaraj and Kohli 2000a, Sircar et al. 1998), our review
128 INFORMATION SYSTEMS RESEARCH/ VOI. 14, No. 2, June 2003
KOHLI AND DEVARAJ
Measuring Information Technology
of the literature finds prevalent differences in the con-
texts, characteristics, data sources, and variables em-
ployed in firm-level studies. To examine the differ-
ences in the execution of IT payoff studies, we derive
a set of structural variables to develop a framework
along which studies vary (Figure 1). The categories
with the relevant literature are cited in Table 1 and are
discussed thereafter. Appendix A provides the codes
assigned to each of the dimensions in our meta-
analysis.
The categories and subcategories in our framework
expand past work that recommends taking into ac-
count research designs and the process of conversion
of IT expenditure into benefits in examining IT payoff
(McKeen and Smith 1993). Resulting from tbe discus-
sion of categories of structural variables in the frame-
work, we present propositions to examine the influ-
ence of each structural variable on IT payoff. To assist
future IT payoff studies, we present a summary of
studies witb their dimensions data source, method,
and dependent variables selected in Appendix B.
2.2. Structural Variables and Propositions
2.2.1. Context. Industry sector studies can differ
by the type of industry of tbe IT investment and sub-
sequent payoff measurement. IT's role and intensity
are often affected by the competitive nature of the in-
dustry. Furthermore, technology applied in manufac-
turing, for instance, computer aided design and com-
puter aided manufacturing (CAD/CAM) or electronic
data interchange (EDI), can yield tangible and mea-
surable efficiency outcomes (Mukbopadhyay et al.
1995b). The measurable impact of such outcomes is sig-
nificantly different from IT investment in healthcare
Figure 1 A Framework tor the Structural Categories Influencing
IT Payott
Contexi
Study
Characteristics
Data Soiirce(s)
f
Djtii Analysis

Results of IT
Pavolf
i.
Variables
Employed
services, for instance, to detect drug interactions
through a clinical information system, which are less
tangible. The issue of measurement can also become
complicated when the industry type is a state or fed-
eral government where traditional profitability mea-
sures may not apply. Consequently, studies in differ-
ent industries, even when measured for similar IT
systems, can lead to different results (Irani et al. 1997,
Kelley 1994, Sohai et al. 2001).
PROPOSITION 1. IT payoff loill differ among the industry
sector of the firms.
2.2.2. Study Characteristics. The studies re-
viewed in this paper have varying levels of data ag-
gregation, namely, month, quarter, or year. Similarly,
the studies cover varying periods over which data are
collected. Most firm-level studies use aggregation of
data at the annual level. It can be hypothesized that
aggregation at the quarterly or monthly level may bet-
ter locate the payoff (Kohli and Devaraj 2000), as op-
posed to annual aggregation where gains in part of a
year can be offset by low-gain months in the same year.
From an analysis standpoint, more frequent data helps
in identifying appropriate time lags to detect the pay-
off from investments.
The duration for which studies capture and analyze
data also varies widely. About half of the studies in
our meta-analysis have collected data for less than a
five-year period. Finally, studies may simply vary in
the number of firms that were included in the analysis.
We created a variable sample size to capture the number
of observations as well as to account for the aggrega-
tion and duration of the study when it was longitudi-
nal or panel. A small sample size increases standard
errors and thus makes it more difficult to isolate the
effects of IT investment from random noise. Further, a
small number of data points may not be sufficient for
establishing a trend for IT payoff, especially when
there are lag effects resulting from IT investment and
measurable payoff (Mukhopadhyay et al. 1997a).
PROPOSITION 2. Studies using larger sample sizes will
show greater IT payoff.
2.2.3. Data Source. The accuracy of IT payoff
studies is contingent upon tbe quality of data utilized.
Therefore, the source of data is critical to the IT payoff
INFORMATION SYSTEMS RESEARCH/VOI. 14, No. 2, June 2003
129
KOHLI AND DEVARAj
Measuring Information Technology Payoff
Table 1 Categories, Subcategories, and Dimensions in iT Payoff Studies
Categories Subcategories and Dimensions
Context Industry Sectorf^ianufacturing, Services, Government, Nonprofit (Brynjolfsson and Yang 1996, Robey and Boudreau 1999)
Study Characteristics Sample Size
Aggregation: Month, Quarter, Year (Devaraj and Kohli 2000a)
Duration: Number of Years (Brynjolfsson 1993, Mahmood and Szewczak 1999)
Data Source Firm, Commercial Databases (Brvnjoltsson and Yang 1996, Devaraj and Kohli 2000a)
Variables Employed Dependent Classification (Robey and Boudreau 1999}
Data Analysis Statistical Analysis (Lee and Barua 1999)
Method: Cross-sectional, Longitudinal (Brynjolfsson and Hitt 1998, Devaraj and Kohli 2000a, Mahmood and Mann 1997)
Level of DetailIT Assets, IT Impact, Organizational Impact (Brynjolfsson and Hitt 1998, Soh and Markus 1995, Ward et al,
1996)
Result Positive, Negative, Neutral, Partial; Percent + ve and-ve significant variables
analysis. Data gathered from secondary sources is eas-
ier to obtain and generally objective. In addition, sec-
ondary sources can provide access to a greater number
of firms and thus improve the generalizability of con-
clusions. Such wider access to data also allows re-
searchers to replicate and verify past results by using
novel analytical approaches. On the other hand, sec-
ondary or public data sources may be limited in detail
and may not include data considered competitor-
sensitive by contributing organizations. Further, data
fields in commercial databases are predetermined and
may not match the exact needs of the researcher. Data
from the firms can overcome several of the above lim-
itations by providing greater access, additional detail,
and supplementary variables for triangulation of re-
sults. When data are accurate, they exhibit fewer errors
in variable bias. Although it may be difficult to deter-
mine the direction of the bias, in practice accurate data
tend to bias coefficients toward zero. Thus, from an
estimation perspective, it is always desirable to have
as accurate data as possible.
In addition, it can be difficult to conduct a consistent
analysis across firms without uniform data definitions.
Firm-level contexts also provide greater detail through
access to contextual variables (Brynjolfsson and Hitt
1998, Devaraj and Kohli 2000a, Harris and Katz 1991,
Morrison and Berndt 1991).
PROPOSITION 3. Studies using primary data sources will
show greater IT payoff than those using secondary data
sources.
2.2A. Dependent Variables Employed. Past
studies have employed various types of dependent
variables in examining firm performance. The most
commonly used dependent variables are financial,
such as return on investment (ROI) and return on as-
sets (ROA) (Barua et al. 1995, Byrd and Marshall 1997,
Lai and Mahapatra 1997, Mahmood and Mann 1993b,
Rai and Patnayakuni 1997, Tam 1998a) and revenue
(Lichtenberg 1995). Productivity- or output-based
measures captured as dependent variables include
management output (Prattipati and Mensah 1997),
milk production (Van Asseldonk et al. 1988), and total
mail sorted (Mukhopadhyay et al. 1997a). Some re-
searchers have used expense-based measures, such as
labor hours (Mukhopadhyay et al. 1997a), expenses
(Francalanci and Galal 1998), capacity utilization
(Barua et al. 1995), and inventory turnover
(Mukhopadhyay et al. 1995a). Given the wide distri-
bution of measures, we created a classification scheme
to capture whether the study measured productivity,
profitability, or both as dependent measures of firm
performance. The following proposition aims to as-
sess if outcomes varied depending upon the type of
dependent measure utilized.
PROPOSITION 4. Studies with profitability-based depen-
dent variables will have different IT payoff tban those that
measure productivity or both.
2.2.5. Data Analysis
2.2.5.1. Analytical Approach. The analytical ap-
proach to measure IT payoff has varied widely among
130 INFORMATION SYSTEMS RESEARCH/VOI. 14, No. 2, June 2003
KOHLI AND DEVARAJ
Measuring Information Technology Payoff
past studies. Among the approaches utilized are
regression-based statistical analyses, such as produc-
tion economics (Brynjolfsson and Hitt 1993,1996, Lich-
tenberg 1995) and stochastic production frontier (Lee
and Barua 1999, Mukhopadhyay et al. 1997a). Some
studies have utilized canonical correlation analysis
(CCA) as a second step to the conventional pair-wise
correlation analysis. CCA controls for the heterogene-
ity among firms and provides a comparison of similar
firms (Ahituv et al. 1999, Mahmood and Mann 1993a).
Structural equation modeling (SEM) has also been ap-
plied to examine the interrelationships of a set of vari-
ables because, unlike linear regression, SEM solves
for all the relationships simultaneously (Byrd and
Marshall 1997). Others have applied less rigorous
correlation-based analyses to examine payoff from in-
vestment in IT (Lubbe et al 1995, Prattipati and
Mensah 1997). There are indications that methodolog-
ical differences can lead to very different outcomes
(Barua and Lee 1997, Shu et al. 2001).
We classify the studies into two categories contain-
ing regression and production economic analysis, and
a second category that includes descriptive statistics
and correiation-based analyses. Given that most
regression-based analyses are based on an underlying
model(s), the researcher is better able to control for
sample heterogeneity. Further, such models also typi-
cally include control variables and covariates to ac-
count for the effect of contextual variables. Therefore,
we would expect to detect and capture payoff more
accurately using regression-based methods.
PROPOSITION 5A. Studies applying regrcssiofi or eco-
nomic models will have higher IT payoff than those applying
correlation-based analyses.
2.2.5.2. MethodLongitudinal vs. Cross Sec-
tional. Cross-sectional data are presumably easier to
obtain and do not require specialized time-series anal-
ysis to analyze longitudinal or panel data. It is also an
efficient and practical approach to examining the im-
pact of IT when the investment being examined is a
one-time investment with a known beginning and end-
ing. Longitudinal data analysis, although more re-
source intensive, allows the researcher to obtain a
deeper understanding of the impact of technology
along the continuum of IT investment (Lucas 1993).
Longitudinal or panel data can often improve the ac-
curacy of the results because they can control for firm-
specific effects. Furthermore, they also allow the re-
searcher to examine lag effects of the impact of
technology (Devaraj and Kohli 2000a, Peffers and Dos
Sontos 1996). In fact, the lack of consideration for lag
effects has been cited as one of the factors contributing
to the so called "productivity paradox" (Brynjolfsson
and Hitt 1996, Dewan and Min 1997, Lee and Barua
1999). Therefore, we propose that studies with longi-
tudinal or panel data may be able to better explain the
IT payoff.
PROPOSITION 5B. Studies with longitudinal designs will
have greater IT payoff than those with cross-sectional
designs.
2.2.5.3. Level of Detail. Although many re-
searchers have recognized the need for considering
organizational factors in addition to IT investment
(Brynjolfsson 1993, Byrd and Marshall 1997, Dewan
and Min 1997, Grover et al. 1998, Mahmood and Mann
1997), few have followed this advice in practice. It is
suggested that the process of IT investment leading to
payoffs should be examined in greater detail because
an investment in IT need not imply that it was the cor-
rect investment or that it was targeted appropriately.
Soh and Markus (1995) propose that the IT investment
process should be broken down further to examine if
it created the appropriate IT assets, and if such assets
lead to the appropriate IT impacts.
Although the studies reviewed in this meta-analysis
did not clearly outline whether process steps, such as
IT assets, or impacts of the IT investment were consid-
ered, we examined each study to analyze its process
orientation. We qualitatively coded the studies
through a binary classification based on whether or not
IT assets were identified. Similarly, another binary
variable was created to determine if the study mea-
sured IT impacts.
PROPOSITION 5C. Studies capturing IT assets and IT im-
pacts (process orientation) will have higher FT payoff than
those lacking process orientation.
2.2.6. Result of Study. One of the objectives of
this meta-analysis is to understand which variables
make a difference in the outcome of IT payoff studies.
INFORMATION SYSTEMS RESEARCH/VOI. 14, No. 2, June 2003 131
KOHLI AND DEVARAJ
Measuring Information Technology Payoff
Given that a significant number of studies reported
positive or partially positive results, we recoded the
result variable into a binary result"positive" and
"nonpositive." The primary reason for the binary clas-
sification was that the remaining three categories (see
Table 1)negative, neutral, and mixedhad relatively
small sample sizes. If the category sizes are not equal
to or greater than the number of independent vari-
ables, the estimation of the discriminant function and
the classification of the observations can be adversely
affected (Hair et al. 1998, p. 258). To test the robustness
of this classification scheme we conducted two checks:
(1) an analysis with a percentage-based continuous
payoff measure (described is 3) and (2) an examina-
tion of the coefficients when each outcome variable of
a study was recoded as an observation.
3. Method
Meta-analysis is rooted in the fundamental values of
scientific enterprisereplicability, quantification,
causal and correlation analysis (Bangert-Drowns 1986,
Benbasat and Lim 1993), and offers direction for future
research (Hunter and Schmidt 1990). As MIS research
advances, so does the need to conduct more meta-
analytic studies (Hwang 1996). The results of this
meta-analysis will contribute in developing metrics
and, eventually in building a theory of IT impacts on
firm performance, a vision proposed by several re-
searchers (Brynjoifsson and Yang 1996, DeLone and
McLean 1992, Keen 1980, Nault and Benbasat 1990).
3.1. Meta-Analysis
Consistent with the process suggested by Glass et al.
(1981), this meta-analysis broadly consists of the fol-
lowing steps:
(1) Development of a framework listing factors that
contribute to explaining IT payoff (discussed in 2);
(2) Selection of studies to be included in the analysis;
(3) Documentation and coding of the various char-
acteristics of studies included in the analysis;
(4) Statistical meta-analyses through regression (lo-
gistic and ordinary least squares) and discrinrtinant
analysis procedures;
(5) Documentation of findings from the two statis-
tical procedures, and directions for future research.
Our meta-analysis of the IT payoff focused on em-
pirical, firn^-level studies that measured the impact of
IT on performance. Our sample consists of 66 studies
(Appendix B) published between 1990 and 2000, un-
published dissertations since 1995, and recent working
papers.** The studies were obtained through a search
of the Social Sciences Index using the keywords IT pay-
off, information technology, firm, productivity, firm perfor-
mance, technology, and profitability. In addition, we re-
viewed the bibliographies of previously published
review papers (Barua and Mukhopadhyay 2000, Bryn-
joifsson and Yang 1996, Sircar et al. 1998) to identify
other firm-level studies. We also searched the Disser-
tation Index for doctoral dissertations on the topic. Dis-
sertations were not included if they resulted in a pub-
lished paper already included in our meta-analysis.
Dissertations are represented in Appendix B by aster-
isks. Further, we contacted more than 15 active re-
searchers in this topical area to solicit working papers.
3.2. Variable Coding
The independent variables employed in the analyses
were (a) dependent variable classification, (b) sample
size, (c) data source, (d) IT impact examination, (e) IT
asset identification, and (f) industry. These variables
were coded and employed in the analysis in the fol-
lowing manner (also, see Appendix A). Dependent
variable classification ranged from one to three de-
pending on whether the outcome variables employed
in the study were productivity-based, profitability-
based, or a combination of both. The sample size is a
function of both the number of firms examined and the
number of time periods data were analyzed for lon-
gitudinal and panel data. This was readily available
from the studies. Data source was a dummy variable
that assumed a value of "1" for primary data and "2"
for secondary data. The next two variables examined
the intermediate process variables of IT asset and IT
impact. The IT asset variable was coded "1" if an IT
asset was identified and "0" otherwise. Similarly, the
IT impact variable was coded "1" if an IT impact was
assessed and "0" otherwise. Finally, we had multiple
dummy variables to identify the type of industry sec-
tor. However, because our objective was to come up
**Some working papers have since been accepted for publication and
are listed as "Forthcoming."
132
INFORMATION SYSTEMS RESEARCH/ VOI. 14, No. 2, June 2003
KOHLI AND DEVARAJ
Measuring Information Technology Payoff
with a parsimonious set of variables that would ex-
plain the outcome of a study, our final model em-
ployed only the dummy variable for industry that dis-
played statistical significance.
To check the reliability of the coding procedure, in
addition to the authors, a graduate student was trained
to independently code the above variables by captur-
ing data from each study. For the coded variables, the
interrater reliability measure. Kappa, was calculated as
0.82. Disagreements in coding were resolved through
discussion and clarification.
3.3. Analysis
3.3.1. Logistic Regression (LR). Logistic regres-
sion (LR) is preferred when assessing the contribution
of variables because it (i) is less affected by variance
covariance inequalities across groups, (ii) is able to
handle categorical variables easily, and (iii) offers case-
wise diagnostic measures for examining residuals
(Hair et al. 1998, p. 314). LR has been applied in meta-
analytic studies to examine clinical heterogeneity
(Schulz and Altman 1996) and effectiveness of drugs
in cattle (Peters et al. 2000). Estimators from the logistic
regression in meta-analytic studies have also shown
less bias and better coverage probabilities than other
approximations (Chang et al. 2000).
3.3.2. Discriminant Analysis (DA). DA is suit-
able for understanding and explaining research prob-
lems that involve a single categorical-dependent vari-
able and several independent variables. DA is a robust
analytical technique that can accommodate mixed in-
dependent variables, as is the case in our meta-analysis
(Hair et al. 1998). Past meta-analytic studies have ap-
plied DA to examine public transport in different
countries (Nijkamp et al. 2000), clinical disciplines of
neurological deterioration and anti-oxidant profiles
(Marschoffa et al. 1997), and the influence of unpub-
lished studies included in meta-analyses (McAuleya et
al. 2000). Similar to our approach, a recent study has
verified the robustness of its results by utilizing more
than one analysis such as DA and LR (Chang et al.
2000).
3.3.3. Continuous IT Payoff Variable: OLS Re-
gression. We developed a percentage-based "contin-
uous" IT payoff variable by categorizing the study out-
come (or dependent) variables into subcategories
positive significant (PSIR)/ positive not significant (pn^g)'
negative significant (n^^J, and negative not significant
("nsig)- The formula to calculate the continuous payoff
variable for each study is: Continuous payoff variable
= K^Psig ~ I )^|^)/Total # of outcome variablesl*100.
For example, a study with all dependent variables
showing positive and significant results will be coded
as 100%, whereas a study that had a total of six depen-
dent variables (three positive significant and one neg-
ative significant) will get a score of [(3 -1)1/6 * 100) or
33.33%. Thus any score in the range from -100 to
4-100 is possible depending on the total number of
outcome variables employed in the study, positive and
negative significant.
4. Results
The frequency of studies included in the meta-analysis
indicate that Management Science and Information Sys-
tems Research were the most popular outlets for IT pay-
off studies. As indicated in Figure 2, over 40% of stud-
ies reported in this paper were published between 1995
and 1997. After peaking in 1997, firm-level studies
dropped in the late 1990s. However, IT payoff studies
appear to be on the rise in the year 2000. Figure 3 in-
dicates that both binary (positive and nonpositive) and
continuous percent payoff outcomes reported by the
studies in our meta-analysis dropped between 1994
and 1998, after which they were on the rise. This can
be viewed in the context of the period during which
the "productivity paradox" was widely discussed
(early to mid-90s) and eventually dismissed (late 90s
to present).
Figure 2
1 4 ~
3 10 ~
V> Q
Number of IT Payoff Studies and the Year of Publicafion
6
n , n . n
CSI
a> Oi
CO
O)
CO
O
Oi
o
CM
Year of Publication
INFORMATION SYSTEMS RESEARCH/VOI. 14, No. 2, June 2003 133
KOHLI AND DEVARAJ
Measuring Information Technology Payoff
Figure 3 A Plot of Studies and Payoff Oufcomes by Puiiiicafion Year Table 2
Variable
Logistic Regression Anaiysis for Positive vs. Nonpositive IT
Payoff Studies
For the purposes of the statistical analysis, two stud-
ies found to be outliers/ influential observations ac-
cording to the Belsley-Kuh-Welsch criteria (Belsley et
al. 1980) were excluded from the analysis. To identify
the structural variables that influence the results of IT
payoff studies we employed (a) LR and (b) DA for our
statistical analyses. As mentioned in 3, we also con-
ducted OLS regression to examine the extent to which
the structural variables infiuenced IT payoff study
results.
The results of LR displayed in Table 2 show that the
fit of the overall LR model (indicated by the - 2 log
likelihood ( - 2 LL) statistic) improved when compared
with the base model value (73.474). The 2LL statistic
is comparable to the overall F test in multiple regres-
sion. The log likelihood value is a measure to assess
the lack of predictive fit; therefore, a lower value in-
dicates a better model fit.
Since the Cox and Snell R^ measure cannot reach a
value of 1, a standardized R^ called Nagelkerke R~
ranging between 0 and 1 was computed as 0.505 (Hair
et al. 1998, p. 143). The choice of independent variables
was based on the objective of constructing a parsimo-
nious model to explain the IT payoff outcome. Results
of the logistic regression procedure indicate that the
sample size of studies included in the analysis was sig-
nificantly associated (at the 5% level) with the outcome
of studies. Specifically, the larger the sample size, the
more likely it was for studies to show a positive and
significant payoff.
The nature of the dependent variable was also sig-
nificantly related to the payoff at the 5% level. Studies
that employed productivity-based measures were
Std.
Error
Wald
Chj-square p-value
Dependent Classification
Sample Size
Data Source
IT Impact Examined
IT Assets Identified
Industry
- 2 Log Likelihood
Goodness of Fit
Cox and Snell R^
Nagelkerke R'
Chi-square
-1. 173
0.003
-1. 184
0.551
1.347
- 1. 099
48.266
73.474
0.378
0.505
25.207 (df = 6)
0.508
0.001
0.632
0,921
0,984
0.822
Sig 0.000
5.337
5,017
3,508
0,358
1.873
1,789
0.011
0,013
0.031
0.275
0.085
0.091
Classification Results
Actual Group
Membership
Predicted Group
Membership
Nonpositive
Positive
Correct Classification
Nonpositive
11.3%
9.44%
75.45%
Positive
15.09%
64.15%
more likely to report positive payoffs as compared to
those that used profitability-based measures.
The source of the data, primary or secondary, was
also found to be significantly related to payoff at the
5% level. All else being equal, studies that utilized a
primary data source (firm) were more likely to report
positive payoffs than those that utilized secondary
data. The identification of an intermediate process
variable, namely the IT asset, that was the focus of the
IT investment was also associated with payoff, albeit
at the 10% level. In other words, studies that reported
having identified the specific IT asset were likely to
report positive payoffs. The second process variable,
IT impacts or the identification of an intermediate im-
pact, probably at a process level) was not significantly
related to payoff. Finally, the type of industry had a
moderate impact on the payoff metric with signifi-
cance at the 10%' level. Results indicated that it is more
likely to see positive payoffs if the firm under consid-
eration was nongovernmental. We report the results of
our DA in Table 3.
134 INFORMATION SYSTEMS RESEARCH/VOI. 14, No. 2, June 2003
KOHLI AND DEVARAJ
Measuring Information Tcchnoh^
To conduct DA, (a) the sample for each group should
be greater than the number of independent variables
and (b) there should be minimum of five cases per in-
dependent variable. Our data and variables utilized
meet both the above requirements. We begin our DA
with a test of the assumption of equal covariance ma-
trix. Box's M statistic (significance 0.729) verified that
our data do not violate that assumption. The overall
canonical correlation 0.578 indicates that the model of
independent variables in our analysis explains (0.578)^
or 33.4% of the variance in the dependent variable
positive or nonpositive result. The discriminant load-
ings in Table 3 highlight the variables that significantly
discriminate between studies with positive and non-
positive results.
Table 3 Discriminant Analysis for Positive vs. Nonpositive IT Payoff
Studies
Variable
Dependent Classification
Sample Size
Source ot Data
IT Impact Examined
Industry
IT Assets Assessed
Discriminant Function
Pooled Within Group
Correlation Between
Variables and
Discriminant Function
0.694
- 0. 302
0.378
- 0. 078
0.329
- 0,078
Standardized
Canonical
Discriminant
Function
Coefficients
0.822
- 0,302
0,562
0.150
0.130
0,077
Group Centroids
Eigenvalue
Wilks' Lambda
Degrees ot Freedom
Classification Results
+ 1.187 (Positive Results)
- 0. 406 (Not Positive Results)
0,502
0.666 Sig 0,005
Actual Group
Membership
Predicted Group
Membership
Nonpositiue
Positive
Correct Classitication 84.3%
Not Positive
13.73%
3.92%
Positive
11.76%
70.58%
Generally, any variable with a pooled, within-group
correlation loading higher than 0.30 or less than ( - )
0.30 is considered substantive. Our analysis reveals
that dependent classification, duration of study, source of
data, and industry variables meet this criterion. IT assets
and IT impacts do not meet this threshold. Finally, the
model correctly classified 84% of the cases.
Results obtained using the continuous measure of
payoff (Table 4) are generally similar to the results ob-
tained from logistic regression using the binary-coded
variable. The sample size was again statistically sig-
nificantly related to the payoff measure at the 1 % level.
The larger the saniple size, the better the likelihood of
payoff. However, the nature of the dependent variable,
productivity or profitability, did not have a significant
association with the continuous metric. As before, the
variable capturing the source of the data was signifi-
cantly associated with payoff at the 1% level. Studies
that used data from primary sources show higher pay-
off than studies that used secondary data. The identi-
fication of IT assets as an intermediate step is shown
to significantly impact payoff at the 5% level. Contrary
to expectations, the negative sign for the coefficient in-
dicates that studies that reported identifying an IT as-
set fared worse than those that did not. Further dis-
cussion of this finding is presented in 5. As was the
case in logistic regression, the IT impact variable was
not significantly associated with the payoff metric. The
type of industry had a stronger relationship with the
percent payoff, with significance at the 5% level. This
qualitative inference is similar to earlier analyses in
which the nongovernment sector showed more posi-
tive payoffs than the government firms.
Table 4 OLS Regression Analysis for Continuous IT Payoff Variable
Variable
Dependent Classification
Sampie Size
Data Source
IT Impact Examined
Industry
IT Assets Identified
Model Summary
B
- 2.654
0.036
-46, 948
-17, 168
- 21,561
-24, 611
R
0.669
Std, Error
6,218
0,012
12,498
11,806
11.290
11,732
0,448
/-value
- 0, 427
2,918
-3, 757
- 1, 454
-1, 910
-2, 098
Adj, R'
0.363
p-value
0,336
0,003
0.000
0.077
0,032
0.021
INFORMATION SYSTEMS RESEARCH/VOI. 14, No. 2, June 2003 135
KOHU AND DEVARAJ
Measuring Informaficii Ti'dmology Payoff
A final check of the robustness of our classification
of the payoff outcome involved an examination of the
coefficients when each outcome variable of a study was
recoded as an observation. The results were still direc-
tionally correct after we corrected for heteroscedastic-
ity using the Huber-White procedure (for details see
Appendix C).
5. Conclusions and Future Research
5.1. Variables Influencing IT Payoff Study Results
This section discusses the implications of the findings
of our meta-analysis of the framework as presented in
Figure 1 and Table 1. Where appropriate, the impli-
cations of the findings are discussed in terms of their
relationship with the outcomes of IT payoff studies
and recommendations for future studies.
5.1.1. Influence of Context Variables. The stud-
ies in this meta-analysis consisted of the following in-
dustry sectorsmanufacturing (14%), service (28%),
government (10%), nonprofit (35%), and combined
(13%). The regression results of the continuous depen-
dent variable indicate that the industry sector influ-
ences the outcome of the studies. We find a significant
difference between outcomes of studies from varying
industry types. Further analysis indicates that studies
conducted in nonprofit and government sectors show
a greater degree of positive outcomes than in financial
and manufacturing sectors combined. A potential ex-
planation is that nonprofit and government databases
contain public information and can offer greater con-
textual detail to the researcher. Open and accessible
detailed data are critical for verification and triangu-
lation of payoff results. Therefore, Proposition 1 is
supported.
5.1.2. Influence of Study Characteristics. Both
binary and continuous analyses indicate that sample
size positively influences the outcomes. DA loadings
also confirm that sample size is a discriminating vari-
able between studies reporting positive and nonposi-
tive outcomes. Therefore, there is strong evidence that
sample size influences the results of IT payoff studies.
This is consistent with conventional wisdom that
larger sample sizes provide greater confidence in the
results. Past studies have suggested that shorter du-
ration or fewer data points may have contributed to
the equivocal results in some IT payoff studies. There-
fore, future studies can increase the confidence in the
results through a combination of granular data collec-
tion and longer duration of data. Therefore, Proposition
2 is supported.
5.1.3. Influence of Data Source. We find support
for the proposition that the source of data in firm-level
studies influences the results of IT payoff studies. The
analysis indicates that when using the binary classifi-
cation result, studies utilizing firm-level data have a
greater incidence of positive results (p<0.10). Further
univariate analysis to examine specific secondary
sources indicates that the database utilized in obtain-
ing data leads to significant differences in the result of
the study. For instance, studies that have utilized Com-
puter Intelligence (CD Infocorp or IDG database show
a higher mean value of positive results (p<0.10). An
examination of the studies in this meta-analysis indi-
cates that many recent studies that have explicated the
productivity paradox issue have also used one or both
these databases. Therefore, Proposition 3 is supported.
5.1.4. Influence of Dependent Variables Em-
ployed. By classifying the diverse dependent vari-
ables into productivity, profitability, or both, we find
evidence (p<0.01 in binary classification) that such
classification is associated with the result of past IT
payoff studies. Studies utilizing productivity-based
dependent variables appear to be positively associated
with payoff from IT investment. This suggests that
productivity-based measures may be more suited to
capture the payoff. On the other hand, profitability-
based measures can be confounded by other factors
that may influence the firm-level profitability. By con-
trast, productivity-based variables also tend to be
closer to the process and, therefore, less likely to be
confounded by external variables. Therefore, Proposi-
tion 4 is supported.
Although, data regarding the type of investment, for
example, hardware or software, were not readily avail-
able in published studies and many appear to have
invested in both, we conducted a univariate analysis
to examine any differences in outcomes. The results do
not show a significant difference between the types of
investment.
136
INFORMATION SYSTEMS RESEARCH/VOI. 14, No. 2, June 2003
KOHLI AND DEVARAJ
Measuring Information Techitohg}/ Payoff
5.1.5. Influence of Data Analysis Approach.
This data analysis approach category includes the sta-
tistical analysis (regression or model vs. correlation),
the method of data collection and analysis (cross-
sectional and longitudinal), and the process orienta-
tion (measurement of IT assets and IT impacts). Uni-
variate analysis of the statistical analysis variable
indicates no significant difference between studies util-
izing models and those that utilize correlation or other
basic analysis. It should be noted that less than 20% of
the studies in our meta-analysis applied a correlation-
based approach. Further data collection may be nec-
essary to exanriine this issue in greater detail, perhaps
at a granular level. Therefore, Proposition 5a is not
supported.
The data collection method of a study in and of itself
does not lead to a significant difference in the result.
Contrary to previous suggestions (Lee and Barua 1999,
Robey and Boudreau 1999), our findings indicate that
longitudinal studies do not appear to influence the re-
sults of the studies. However, it should be noted that
the longitudinal data variable is partly reflected in the
san:iple size. In other words, longitudinal data are
likely to have larger sample sizes and can help detect
lag effects. As discussed above, larger sample sizes
lead to positive outcomes. Therefore, Proposition 5b is
not supported.
Finally, the process-orientation variables indicate
weak support for their influence on the outcomes of IT
payoff studies. First, in the DA, neither IT assets nor
IT impacts loaded at an appropriate level (>0.30). Sec-
ond, LR found only IT assets significant at {p<0.10).
Further, contrary to expectations, IT assets were nega-
tively associated with outcomes of studies. In other
words, studies that did not measure the creation of
appropriate IT assets exhibited positive payoff. The
reason for these unexpected results could be that our
coding reflected whether the study attempted to iden-
tify IT assets, not the results of such assets or the suit-
ability of the assets. Therefore, Proposition 5c is not
supported.
Table 5 summarizes the findings of analyses. Al-
though, there are characteristics of data analyses that
were not supported, taken together, our results indi-
cate that structural variables do influence the outcomes
of IT payoff studies.
5.2. Limitations
Our meta-analysis includes mainly studies from the
information systems discipline. IT is utilized in almost
every segment of the economy, and it is likely that IT
payoff studies published containing discipline-specific
keywords in bibliographic databases limited our
search of such trade and academic publications. Sec-
ond, although we captured the process orientation
Table 5 Summary of Findings
Categories
Context
Study
Characteristics
Data Source
Variables
Empioyecj
Data Analysis
Subcategories
imjustry Sector
Sample Size
Firm, Commercial
Databases
Dependent Classitication
Statistical Anal ysi s-
Model vs. Correlation Method
Cross-sectional vs. Longitudinal
Level of DetailAssets
Level of DetailImpact
Logistic Regression
(-Hve vs. not -i-ve)
Supported"
Supported"
Supported"
Not Supported
Not Supported*
Not Suppor t ed" "
Supported'
Not Supported
Discrim. Analysis
(-i-ve vs. not -i-ve)
Supported
Supported
Supported
Supported

Not Supported
Not Supported
OLS (Continous
Variable)
Supported"
Support ed"*
Support ed"*
Not Supported

Supported**
Supported*
p<0.10; "p<0.05; "' p<0. 01; '"'univariate analysis.
INFORMATION SYSTEMS RESEARCH/ VO! . 14, No. 2, June 2003 137
KOHLI AND DEVARAJ
Measuring Information Technology Payoff
variables, i.e., whether a study identified IT assets or
assessed IT impacts, we did not assess their suitability
or accuracy. It is possible that some firms that mea-
sured the process of IT impact may not have done so
comprehensively or accurately.
5.3. Contribution and Recommendations for
Future Research
With the exception of Mahmood et al. (1999), our meta-
analysis represents one of the first attempts to empir-
ically validate IT payoff characteristics emphasized by
past research (Barua et al. 1995, Barua et al. 1996,
Brynjolfsson and Yang 1996, Brynjolfsson and Hitt
1993, Devaraj and Kohli 2000a, Grover et al. 1998,
Markus and Robey 1988). We hope that this empirical
validation of structural variables and the recommen-
dations will improve the reliability of IT payoff studies
at a time of increasing investment in IT. This meta-
analysis contributes to the firm-level IT payoff litera-
ture by examining, summarizing, and analyzing the
empirical studies in an attempt to understand those
structural characteristics that discriminate between
positive and nonpositive results. The findings indicate
that structural variables of IT payoff studies and the
manner in which they are conducted do make a dif-
ference. First, our findings suggest that, to the extent
possible, future studies should consider gathering data
from primary sources. Second, when secondary data
sources are used, databases such as IDG and Computer
Intelligence Infocorp may provide a better opportunity
for finding influences of structural variables in IT pay-
off outcomes. Third, researchers should gather larger
samples comprising of longitudinal or panel data to
assess the lag effects of IT payoff. A greater number of
firms can also increase sample sizes. Fourth,
productivity-based dependent variables are better
suited to assess payoff results than those based on prof-
itability measures alone.
While studies reported in our meta-analysis were
generally conducted in traditional IT investment, the
recent growth in electronic commerce (EC) provides
exciting opportunities for IT payoff assessment. We be-
lieve that the structural variables analyzed in this
meta-analysis will continue to apply to EC-based IT
investment, yet inherent characteristics of new tech-
nologies can impact IT payoff measurement. For in-
stance, surveys indicate that firms use EC to more dil-
igently differentiate products than to reduce costs. As
a result many firms have not accrued profits from in-
vestment EC initiatives (Belleflamme 2001). Chircu and
Kauffman (2000) present case-study evidence that
demonstrates industry and organizational barriers to
valuation and conversion of EC-related IT invest-
ments. In such cases the process-oriented structural
measurement for IT payoff can yield payoff assessment
at the process-level, before organizational payoffs sur-
face due to the impending lag effects. Gybermediaries
require extensive IT investment and are part of virtual
value chains. Future studies can measure the payoff
accruing to partners. There are indications that some
partners may not accrue the same payoff as others
(demons and Row 1993, Rao et al. 1995). A process
measurement approach can identify what and how
value is created when one partner invests and how it
results in payoff to others in the supply chain.
Future research may also examine the accuracy and
appropriateness of process measuresIT assets and IT
impactand examine how well they relate to the find-
ings of organizational impacts. In such cases IT assets
and IT impacts can also be treated as dependent vari-
ables, just as the previous studies have treated orga-
nizational impacts. In addition, an examination of eco-
nomic and statistical models and their appropriateness
to the data and hypotheses will identify differences in
IT payoff results due to analysis.
To facilitate future meta-analyses, IT payoff studies
may explicitly report sample sizes, independent and de-
pendent variables, and correlation coefficients and their
statistical significance. In addition, future studies should
explicitly report which complementary changes in busi-
ness practices, such as business process engineering,
business-to-business electroruc-commerce initiatives, and
enterprise resource planning accompanied the IT invest-
ment. Such uniformity in the conduct and analysis of
studies will isolate and identify the effectiveness of com-
plementary changes that lead to IT payoffs (Barua and
Mukhopadhyay 2000). Such transparent discussion will
also enable replication and theory development.
Acknowledgments
The authors thank Robert Bretz and the Management Department
al the University of Notre Dame for a grant to suppt)rt this research.
The paper has benefited from comments from Matt Bloom, Lorin
Hitt, Mo Adam Mahmood, Lynne Markus, and Sue Sherer. Research
support from Krishnan Gopalakrishnan, Sevugan Nagappan, and
Rick Tan is greatly appreciated.
138
INFORMATION SYSTEMS RESEARCH/ VOI. 14, No. 2, June 2003
KOHLI AND DEVARAJ
Measuring Information Technology Pai/off
Appendix A
Subcdtegories
Industry Sector
Aggregation
Duration
Data Source
Dependent Variable
Classification
Coding and Explanation
1 Manufacturing
2 Services
3 Government
4 Nonprofit
5 Combined
1 Year
2 Quarter
3 Month
Duration of Data Collection (in Years)
1 Firm
2 Secondary (Public, Commercial Databases)
I Productivity Only
2 Profitability Only
3 Both Productivity and Profitability
Analysis
Method
IT Assets
IT Impact
Result
1 Regression
2 Not Regression
1 Longitudinal
2 Cross-Sectional
0 No IT Assets Identified
1 IT Asset Identified
0 No IT Impact Identified
1 IT Impact Identified
] Positive
- 1 Negative
0 Neutral
2 Mixed or Partial
(Some Positive, Some Negative)
Appendix B. IT Payoff Studies and Their Characteristics
Studies Data Source Method
1'ay off
Coding Dependent Variables
(!) Ahituv et al . (1999)
(2) Alshilash'*(]997)
(3) Barua and Lee (1997)
(4) Barua et al. (1995)
(5) Barua et al. (2000)
(h) Bergnon and Dexter (1999)
(7) Bharadwaj et al. (1999)
(8) Bresnahan etal. (2000)
(9) Brynjoifsson and Hitt (1993)
(10) Brynjoifsson and Hitt (1995)
(11) Brynjoifsson and Hitt (1996)
(12) Brynjoifsson and Hitt (2000)
(13) Byrdand Marshall (1997)
(14) Chen* (1996)
(15)Clino*(1999)
(16) Devaraj and Kohli (2000b)''
117) Devaraj and Kohli (2000a)"
(18) Dewan and Min (1997)
(19) Francalanci and Galai (1998)
(20) Grover etal. (1998)
(21) Harris and Katz (1991)
Computerworld, Fortune
Survey of Saudi Arabia
Firms
MPIT Database
MPIT Database
Hoover's Online
Firms
Information Week 500
Cl Infocorp, Compustat,
Survey
IDG, Compustat
IDG, Compustat
IDG, Compustat
Cl Infocorp, Compustat,
IDG
IDG Computerworld
Premier 100
Computerworld Premier,
COMPACT
Kailroad firm
Firms
Firms
IDG, Compustat
LOMA, A. M. Best
Insurance Reports,
Compustat, lOK Reports of
Publicly Traded Firms
Survey of Firms
Life Office Management
Association (LOMA)
Database
Longitudinal
Cross-Sectional
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Cross-Sectional
Cross-Sectional
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Cross-Sectional
Cross-Sectional
Longitudinal
1
1
1
1
2
1
1
1
1
1
1
1
2
1
1
1
1
1
2
2
1
Output, Labor Productivity
Degree of IT Use, Organizational
Functions
Output, Labor Productivity
Capacity Ufilization, Inventory Turnover
Quality
Adoption Benefit, Process Savings
Tobin's q
Value Added to Firm
ROI
Sales, IT Capital, IT Labor
Sales, IT Capital, IT Labor
Capital, Labor, Computer elasticity
ROI, ROS, Revenue Growth
ROA, ROS, ROE
Revenue Ton per Mile/Fmployee/
Revenue, Quality
Revenue, Quality
Sales, IT Capital, IT Labor
Income per Employee, Ratio of Total
Operating Expense/Premium Income
IT Diffusion, Perceived Extent of Process
Change, Perceived Productivity
Improvement
Premium Income
INFORMATION SYSTEMS RESEARCH/ VOI. 14, No. 2, June 2003 139
KOHLI AND DEVARAJ
Measuring Information Technology Payoff
Appendix B. (cont'd.)
(22) Haynes and Thompson (2000)
(23) Hitt and Brynjoifsson (1996)
(24) Kelley (1994)
(25) Kivijarvi and Saarinen (1995)
(26) Kohli and Devaraj (2000)
(27) Koski (1999)
(28) Kudyba and Diwan (2000)
(29) Kwon and Stoneman (1995)
(30) Lee and Barua (1999)
(31) Lee and Perry (2000)
(32) Lehr and Lichtenberg (1998)
(33) Li and Ye (1999)
(34)Licbtenberg(1995)
(35) Loveman (1994)
(36) Lubbe et ai. (1995)
(37) Mahmood and Mann (1993a)
(38) Mahmood and Mann (1997)
(39) Markus and Soh (1993)
(40) Mayberry-Stewart* (1996)
(41) McKeen and Smith (1993)
(4)2 Menon and Lee (2000)
(43) Menon et al. (2000)
(44) Mistry and Johnston (2000)
(45) Mukhopadhyay et al. (1995a)
(46) Mukhopadhyay et al. (1997b)
(47) Mukbopadbyay et al. (1997a)
(48)Panthawi*a999)
(49) Papp (1999)
Banking World
IDG Surveys, Compustat 11,
Bureau of Economic
Analysis, Council of
Economic Advisors
Firms
Talouselama Magazine,
Survey
Firms
The Federation of Finnish
Electrical and Electronics
Industry (SETELI)
Cl Infocorp
Firm Surveys, Lotus
Datastream
MPIT Database
ClInfocorp
Bureau of Labor Statistics
(BLS) Federal Productivity
Measurement Program, Cl
Infocorp
Compustat;
Information Week, U.S. Dept
of Labor's U.S. Industrial
Outlook
IDG, Information Week,
Computerworld
MPIT Database
Firms Survey
Computerworld Premier
100, Compact DISCLOSURE
Database
Computerworld Premier
inn
l UU
Federal Reserve Bank
Dornfest Hospital Database
Firm
State Healthcare Database
State Healthcare Database
Fed Reserve Bank
Firm
Firm
Firm
Firms in Thailand
Fortune, Otber
Longitudinal
Longitudinal
Cross-Sectional
Cross-Sectional
Longitudinal
Longitudinal
Cross-Sectional
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
1
2
1
2
1
2
1
1
-1
1
1
1
1
-1
1
1
1
1
2
1
1
1
1
1
1
1
2
0
ATM Adoption
Sales Revenue, Labor Productivity, Total
Factor Productivity
Production Hours, Setup Hours, Machine
Hours
Revenue
Sales Revenue, Labor Productivity, Total
Productivity
Revenue
Value Added to Firm
Output, Labor Productivity
Gross State Product
Productivity, Labor Costs
Return on Assets, Return on Sales
Revenue
Output, Labor Productivity
Computerization Index, Operating
Expense Ratio
ROS, ROI, Growth in Revenues
ROS, Growth in Revenues
Profitability
IT Performance (Application and
Technology Intensity)
Revenue, CPU Time, Employees
Costs
Costs
Labor Costs, Revenue
Inventory Turnover, Costs, Production
Volume
Labor Hours, Number of Transactions,
Type of Transaction
Total Output, On-Time Output, Labor
Hours
Firm Performance
Financial Performance
140 INFORMATION SYSTEMS RESEARCH/ VOI. 14, No. 2, June 2003
KOHLI AND DEVARAJ
Measuring Information Technology Payoff
Appendix B. (conl'd.)
(50) I't;tft.>rs and Dus Sontos (1996)
151) Prasad and Marker (1997)
(52) Prattipati and Mensah (1997)
(53) Ragowsky et al. (2000)
(54) Rai and Patnayakuni (1997)
(55) Rai etal. (1996)
(56) Rao etal. (1995)
(57) Siegel (1997)
(58) Stoneman and Kwon (1996)
(59) Strassman (1990)
(60)Tam (1998a)
(61)Tam (1998b)
(62) Teo and Wong (1998)
(63) Van Asseldonk et al. (1988)
(64) Wang' (1997)
(65) Weill (1992)
(66)Xia'*(1998)
Federal Deposit Insurance
Corporation (FDIC), Federal
Reserve Board of
Governors, Conference of
State Bank Supervisors
Wharton Financial
Institutions Survey
Computerworld Premier
100
Survey
[n/ormationWeek,
Compustal
InformationWeek,
Compustat
Mail Survey, Compustat
U.S. Census Bureau
Firm Surveys Centre for
Urban and Regional
Development Studies, U.K.
MPIT Database
Asian Computer Directory
(ACD)
Asian Computer Directory
(ACD), PACP and Global
Vantage Databases, Asian
Development Bank
Survey of Managers'
Perception
NRS Royai Dutch Cattle
Syndicate (Netherlands)
Taiwan Firms
Firm Surveys, Interviews,
Site Visits
Survey
Longitudinal
Cross-Sectional
Cross-Sectional
Cross-Sectional
Cross-Sectional
Cross-Sectional
Cross-Sectional
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Longitudinal
Cross-Sectional
Cross Sectional
Cross-Sectional
Longitudinal
Cross-Sectional
1
2
1
2
2
2
0
1
1
0
2
2
2
1
1
1
1
Market Share, Performance Before Taxes
Productivity, Profitability
Management Output, Economic Value
Added (EVA)
Executive Perception
ROA, ROE
ROA, ROE
ROA, Long-Term Debt to Equity
Totai Factor Productivity
Gross Profit
Return on Management, Value Added
Operating Costs, Computer Capital,
Noncomputer Capital
ROA, ROE, ROS
Competitive Performance, Productivity,
Management Performance
Milk Fat and Protein Production, Calving
Interval
Firm Performance
Sales Growth, ROA, Labor Productivity
Effectiveness of IS Function/Business
Processes, Organizational Performance
Note. With the exception of Cline (1999), all longitudinal studies used multifirm data are also referred to as panel data.
Appendix C
We treated each outcome variable of a study as an observation, thus
creating multiple observations from each study. While this may pro-
vide us with a significantly larger sample size, it will suffer from
iionconstant variance. It is well established that the presence of het-
iTOScedasticity in the disturbance terms can lead to inefficient pa-
rameter estimates and inconsistent covariance matrix estimates
(White 1980). Therefore, we corrected the standard deviation using
the Huber-White procedure. Tbis was implemented in SAS using
Proc Mixed. The coefficients for ail the independent variables were
in the expected direction (same as the results using logistic regres-
sion). Further, the coefficients for data source, dependent classifica-
tion, and industry were statistically significant at the 1% level, and
sample size was significant at the 10% level. These results provided
further reassurance that our classification of payoff studies was rea-
sonably robust.
References
Ahituv, N., R. Giiadi. 1993, Business success and information tech-
nology: Are they really related? Proc. 7th Annual Conf. Manage-
ment IS. Tel Aviv University, Israel.
INFORMATION SYSTEMS RESEARCH/VOI. 14, No. 2, June 2003
141
KOHLI AND DEVARAJ
Measuring Information Technology Payoff
, S. Lipovetsky, A. Tisbler. 1999. The relationship between firms'
information systems policy and business performance: A mul-
tivariate analysis. M.A. Mahmood, E.j. Szewczak, eds. Mea-
suring Information Technology Payoff: Contemporary Approaches.
Idea Croup, Hershey, PA, 62-82.
Alshilash, A. 1997. A study to measure the use and effects of infor-
mation technology on organizational functions. Unpublished
D.Sc. dissertation, George Washington University, Washington,
D.C.
Bangert-Drowns, R. L. 1986. Review of developments in meta-
analytic method. Psych. Bull. 99 388-399.
Banker, Rajiv D., Robert ]. Kauffman, Mo Adam Mahmood. 1993.
Strategic Iitforuiatkm Technology Management: Perspectives on Or-
ganizationnl Gnnvtii and Competitive Advantage. Idea Group Pub.,
Harrisburg, PA.
Barua, A., B. Lee. 1997. An economic analysis of the introduction of
an electronic data interchange system, tnform. Systems Res. 8(4)
398-422.
, T. Mukhopadhyay. 200(1 information technology and business
performance: Past, present, and future. R. W. Zmud, cd. Fram-
ing the Domains of IT Research: Projecting the Future Through the
Past. Pinnafiex Educational Resources, Cincinnati, OH, 65-84.
, C. Kriebel, T. Mukhopadhyay. 1991. An economic-analysis of
strategic information technology investments, MIS Quart. 15(3)
313-331.
, , . 1995. Information technologies and business
valueAn analytic and empirical investigation. Inform. Systems
Res. 6(1)3-23.
, C. H. S. Lee, A. B. Whinston. 1996. The calculus of reengineer-
ing. Inform. Systems Res. 7(4) 409-428.
, E. Yin, A. B. Whinston. 2000. Not all dot coms are created equal:
A study of the productivity of Internet based companies. 65-
84.
Belleflamme, P. 2001. Oligopolistic competition, IT use for product
differentiation and the productivity paradox. Internat. j. Indust.
Organ. lW-2) 227-248.
Belsley, D. A., E. Kuh, R. Y. Welsch. 1980. Regression Diagiiostics: hien-
tif\/ing Influential Data and Sources of ColUnearity. John Wiley &
Sons, New York.
Benbasat, I., L. Lim. 1993. The effects of group, task, context, and
technology variables on the usefulness of group support sys-
tems. Small Croup Res. 24(4) 430-462.
Berger, P., D. Sutherland, J. Kobelious. 1988. Measuring Business
Value of Information Technologies. International Center for Infor-
mation Technologies, Washington, D.C.
Bergnon, M., A. Dexter. 1999. An assessment of financial EDI value
to corporate adopters. M, A. Mahmood, E. J. Szewczak, eds.
Measuring Information Technology Payoff: Contemporary Ap-
proaches. Idea Group, Hershey, PA, 346-364.
Bharadwaj, A. S., S. G. Bharadwaj, B, R. Konsynski. 1999. Informa-
tion technology effects on firm performance as measured by
Tobin's q. Management Sci. 45(7) 1008-1024,
Bresnahan, T. F., E. Brynjolfsson, L. Hitt. 2000. Information technol-
ogy, workplace organization, and the demand for skilled labor.
Quart. /. Econom. Forthcoming.
Brynjulfsson, E, 1993. The productivity paradox of information tech-
nology. Comm. ACM 36(12) 66-77.
, L. Hitt. 1998. Beyond the productivity paradox. Comm. ACM
41(8) 49-55.
, -^ . 1993, Is information systems spending productive? New
evidence and new results. Froc. 14th Internat. Conf. Inform. Sys-
tems, Orlando, FL, 47-64.
-, . 1995. Information technology as a factor of production:
The role of differences among firms. Econom. Innovatiou and Neii'
Tech. 3 183-199.
V . 1996. Paradox lost? Firm-level evidence on the returns to
information systems spending. Management Sci. 42(4) 541-558.
; . 2000. Beyond computation: Information technology, or-
ganizational transformation and business performance. /.
Econom. Perspectives 14(4) 23-^8.
, S. Yang. 1996. Information technology and productivity: A re-
view of literature. Adv. Comput. 43 179-214.
Byrd, T. A., T. E. Marshall. 1997. Relating information technology
investment to organizational performance: A causal model
analysis. Omega Internat. /. Management Sci. 25(1) 43-56.
Chang, B. H., S. Lipsitz, C. Waternaux. 2000. Logistic regression in
meta-analysis using aggregate data. /. Appl. Statist. 27(4) 411-
424.
Chen, W. 1996. On the relationship between information technology
investment and corporate performance: An empirical investi-
gation of the United States industries. Unpublished Ph.D. dis-
sertation, Drexel University, Philadelphia, PA.
Chircu, A.M., R.J. Kauffman. 2000. Limits to value in electronic
commerce-related IT investments. /. Management Inform. Sys-
tems 17(2) 59-80.
Clemons, E. K., M. Row. 1993. Limits to interfirm coordination
through information technology: Results of a field study in con-
sumer packaged goods distribution. /. Management Inform. Sys-
tems 10(1) 73-96,
Cline, M. 1999. The impact of information technology investment on
productivity improvements in the American railroad industry
between 1986 and 1995. Unpublished Ph.D. dissertation, Elor-
ida State University.
Davern, M. J., R. J. Kauffman. 2000. Discovering potential and real-
izing value from information technology investments. /. Man-
agement Inform. Systetns 16(4) 121-143.
DeLone, W. H., E. R. McLean. 1992. Information systems success: In
search of the independent variable. Inform. Systems Res. 3(1) 60-
95.
Devaraj, S., R. Kohli. 2000a. Information technology payoff in the
healthcare industry: A longitudinal study. /. Management In-
form. Systems 16(4) 41-67.
, . 2000b. Performance impacts of information technology:
Is actual usage the missing link? Management Sci. Forthcoming.
Dewan, S., C. K. Min. 1997. The substitution of information technol-
ogy for other factors of production: A firm level analysis. Man-
agement Sci. 43(12) 1660-1675.
Dudley, L., P. Lasserre. 1989. Information as a substitute for inven-
tories, Eur. Econom. Rev. 31 1-21.
142
INFORMATION SYSTEMS RESEARCH/VOI. 14, No. 2, June 2003
KOHLI AND DEVARAJ
Measuring Information Tcchtiology Pai/off
Francalanci, C, H. Galal. 1998. Information technology and worker
composition: Determinants of productivity in tbe life insurance
industry. MiS Qtmrt. 22(2) 227-241.
Glass, G. V., B. McCaw, M. L. Smith. 1981. Meta-nnati/sis in Social
Research. Sage, Beverly Hills, CA.
Grover, V., J. Teng, A. Segars, K. Fiedler. 1998. Tbe influence oi in-
formation technology diffusion and business process cbange on
perceived productivity: The IS executive's perspective. Inform.
Mmiagcmnil 34(3) 141-159.
Hair, Joseph F., R. Tatham, R. Anderson, W. Black. 1998. Multivariate
Dntii Annh/sis. Prentice Hall, Upper Saddle River, NJ.
Harris, Sidney E., Joseph L. Katz. 1991, Organizational performar\ce
and IT investment intensity in the insurance. Organ. Sci. 2(3)
263-295.
Haynes, M., S. Thompson. 2000. Productivity, employment and "IT
Paradox": Evidence from financial services. R. Barrel, G. Ma-
son, M. O'Mahony, eds. Productivity, Innoimtion and Economic
Performance. University Press, Cambridge, U.K., 93-116.
Hitt, L. M., E. Brynjolfsson. 1996. Productivity, business profitability,
and consumer surplus: Three different measures of information
technology value. MIS Quart. 20(2> 121-142.
Hunter, J. E., F. L. Schmidt. 1990. Mefhotls of Met a-Analysis. Sage,
Newbury Park, CA.
Hwang, M. I. 1996. The use of meta-analysis in MIS research: Prom-
ises and problems. Data Base for Adv. Inform. Systems 27(3) 35-
48.
Irani, Z., J. N, Ezingeard, R. J. Grieve. 1997. Integrating the costs of
a manufacturing IT/IS infrastructure into the investment
decision-making process. Technovaiion 17(11-12) 695-706.
Jorgenson, D. W. 2001. Information technology and the U.S. econ-
omy. Amer. Econom. Rev. 91(1) 1-32.
, K. J.Stiroh.2000. Raising the speed limit: U.S. economic growth
in the information age. Bnmkings Papers on Econom. Activity 1
125-235.
Keen, Peter G. W. 1980. MIS research: Reference disciplines and a
cumuLitive tradition. Proc. 1st Intcrnat. Conf. Inform. Systems,
Philadelphia, PA, 9-18.
Kelley, M. R. 1994. Productivity and information technologyThe
elusive connection. Management Sci. 40(11) 1406-1425.
Kivijarvi, H., T. Saarinen. 1995. Investment in information systems
and the financial performance of the firm. Inform. Management
28(2) 143-163.
Kohii, R., S. Devaraj. 2000. Contribution of decision support systems
to organizational performance: Evidence from a longitudinal
study. Decision Support Systems. Forthcoming.
Koski, H. 1999. Tbe implications of network use, production network
externalities and public networking programmes For firm's pro-
ductivity. Res. Policy 28(4) 423-^39.
Kraemer, K. L., J. Dedrick. 2001. The productivity paradox: Is it re-
solved? Is there a new one? What does it all mean for manag-
ers? CRITO, University of California (February).
Kudyba, S., R. Diwan. 2000. Productivity in the U.S. information
technologies. Proc. North Amer. Productivity IVorfcs/io;?, Schenec-
tady, NY.
Kwon, Myung Joong, Paul Stoneman. 1995. The impact of technol-
ogy adoption on firm productivity. Econom. Innovation and New
Tech. 3 219-233.
Lai, V. S., R. K. Maliapatra. 1997. Exploring the research in infor-
mation technology implementation. Inform. Management 32(4)
187-201.
Lee, B., A. Barua. 1999. An integrated assessment of productivity
and efficiency impacts of information technology investments:
Old data, new analysis and evidence. }. Productivity Analysis
12(1)21^3.
Lee, G., J. Perry. 2000. Are computers boosting productivity? A test
of paradox in state governments. University of California,
Irvine, CA.
Lehr, W., F. R. Lichtenberg. 1998. Computer use and productivity
growth in U.S. federal government agencies, 1987-92. /. indust.
Econom. 46(2) 257-279.
Li, M. F., L. R. Ye. 1999. Information technology and firm perfor-
mance: Linking with environmental, strategic and managerial
contexts. Inform. Management 35(1) 43-51.
Lichtenberg, Frank R. 1995. The output contributions of computer
equipment and personal: A firm-level analysis. Econom. Inno-
vation and New Tech. 3 201-217.
Loveman, Gary W. 1994. An Assessment of the Productivity impact of
the Information Technologies. Oxford University Press, New York.
Lubbe, S., G. Parker, A. Hoard. 1995. The profit impact of IT invest-
ment. /. Inform. Tech. 10 44-51.
Lucas, Henry C. 1993. The business value of information technology:
A historical perspective and thoughts for future research. R.
Banker, R. Kauffman, M. A. Mahmood, eds. Strategic Informa-
tion Technology Management. Idea Group, Harrisburg, PA, 359-
374.
Mahmood, M., G. Mann. 1993a. Measuring the organizational im-
pact of information technology investment: An exploratory
study. /. Management Inform. Systems 10(1) 97-122.
, . 1993b. Measuring the organizational impact of informa-
tion technology investment: An exploratory study. /. Manage-
ment Inform. Systems 10(1) 97-122.
, . 1997. How information technology investments affect or-
ganizational productivity and performance: A longitudinal
study. Proc. 1997 Inform. Resources Management Association In-
temat.Conf. 187-191.
, --. 2000. Special issue: Impacts of information technology
investment on organizational performance. /. Management In-
form. Systems 16(4) 3-10.
, Edward J. Szewczak. 1999. Measuring Information Technology
Investment Payoff: Contemporary Approaches. Idea Group Pub.,
Hershey, PA.
, -, A. Abddahllah. 1999. Information technology investment
and organizational performance: A meta-analysis. Proc. Na-
tional Decision Sci. Inst. Conf. New Orleans, LA, 669-671.
Markus, M. L., D. Robey. 1988. Information technology and
organizational-changeCausal-structure in theory and re-
search. Management Sci. 34(5) 583-598.
INFORMATION SYSTEMS RESEARCH/VOI, 14, No. 2, June 2003
143
KOHLI AND DEVARAJ
Measuring Information Technology Payoff
, Christina Soh. 1993. Banking on information technology: Con-
verting IT spending into firm performance. R. Banker, R. Kauff-
man, M. A. Mahmood, eds. Strategic Information Technology
Management. Idea Group, Harrisburg, PA, 375-404.
Marschoffa, E., R. Domingueza, E. S. de Lustiga, A. Famularia, J.
Serra. 1997. Neurological deterioration, heterogeneity of Alz-
heimer disease and antioxidant profiles: A four trials meta-
analysis. /. Neurological Sci. 150(1) S22.
Mayberry-Stewart, M. 1996. Determinants of information technology
strategic performance: An analysis of firm performance, IT per-
formance, IT investment intensity and IT governance in the
health care sector. Unpublished Ph.D. dissertation, Claremont
Graduate School, Claremont, CA.
McAuleya, Laura, Ba' Phama, Peter Tugwellb, David Moher. 2000.
Does the inclusion of grey literature influence estimates of in-
tervention effectiveness reported in meta-analyses? The Lancet
356(9237) 1228-1231.
McKeen, James D., Heather A. Smith. 1993. The relationship between
information technology use and organizational performance. R.
Banker, R. Kauffman, M. A. Mahmood, eds. Strategic Informa-
tion Technology Management, Idea Croup, Harrisburg, PA, 405-
445.
Menon, Nirup M., Byungtae Lee. 2000. Cost control and production
performance enhancement by IT investment and regulation
changes: Evidence from Washington State healthcare service
industry. Decision Support Systems. Forthcoming.
, , L. Eldenburg. 2000. Productivity of information systems
in tbe healthcare industry. Inform. Systems Res. 11(1) 83-92.
Mistry, ]., H. Johnston. 2000. Differential impacts of information
technologies on cost and revenue driver relationsbips in se-
lected banking functions. Pruc. Amer. Accounting Association,
Philadelphia, PA, 1-34.
Mooney, J. G., V. Gurbaxani, K. L. Kraemer. 1996. A process oriented
framework for assessing the business value of information tech-
nology. (Reprinted from Proc. 16th Annual Intern. Conf. Inform.
Systems, ] 995, pp. 17-27). Data Base for Adv. Inform. Systems 27(2)
68-81.
Morrison, C, E. Berndt. 1991. Assessing the productivity of infor-
mation technology equipment in U.S. manufacturing indus-
tries. Working paper. National Bureau of Economic Research.
Mukhopadhyay, T., A. Barua, C. H. Kriebel. 1995a. Information tech-
nologies and business valueAn analytic and empirical inves-
tigation. Inform. Systems Res. 6(2) U2-U2.
, S. Kekre, S. Kalathur. 1995b. Business value of information
technologyA study of electronic data interchange. MIS Quart.
19{2) 137-156.
, F. Javier Lerch, Vandana Mangal. 1997b. Assessing the impact
of information technology on labor productivity: A field study.
Decision Support Systems 19(2) 109-122.
, S. Rajiv, K. Srinivasan. 1997a. Information technology impact
on process output and quality. Management Sci. 43{12) 1645-
1659.
Nault, B., I. Benbasat. 1990. An evaluation of empirical research in
managerial support systems. Derision Support Systems 6 203-
226.
Nijkamp, P., P. Rietveld, L. Spierdijk. 2000. A meta-analytic com-
parison of determinants of public transport use: Methodology
and application. Environment and Planning B-Planning & Design
T7{b) 893-903.
Oliner, S. D., D. E. Sichel. 2000. The resurgence of growth in the late
1990s: Is information technology the story? /. Econom. Perspec-
tives 14(4) 3-22.
Panthawi, P. 1999. An investigation of it effects on various aspects
of organizational effectiveness (information technology, firm
performance, productivity). Unpublished Ph.D. dissertation,
Claremont Graduate University, Claremont, CA.
Papp, R. 1999. Business-IT alignment: Productivity paradox payoff?
Indust. Management & Data Systems 99(7-8) 367-373.
Peffers, K., B. L. Dos Sontos. 1996. Performance effects of innovative
IT applications over time. IEEE Trans. Engrg. Management 43(4)
381-392.
Peters, A. R., T. A. Martinez, A. J. C. Cook. 2000. A meta-analysis of
studies of the effect of GnRH 11-14 days after insemination on
pregnancy rates in cattle. Theriogeiwlogy 54(8) 1317-1326.
Prasad, B., P. Harker. 1997. Examining tbe contribution of informa-
tion technology toward productivity and profitability in U.S.
retail banking. Working paper 97-09, Financial Institutions
Center, The Wharton School, University of Pennsylvania, Phila-
delphia, PA.
Prattipati, S. N., M. O. Mensah. 1997. Information systems variables
and management productivity. Inform. Management 33(1) 33-43.
Ragowsky, A., M. Stern, D. A. Adams. 2000. Relating benefits from
using IS to an organization's operating characteristics: Inter-
preting results from two countries. /. Management Inform. Si/s-
tems. 16(4) 175-194.
Rai, Arun, Ravi Patnayakuni. 1997. Technology investment and busi-
ness performance. Comm. ACM 40(7) 89-97.
, , N. Patnayakuni. 1996. Refocusing where and bow IT
value is realized: An empirical investigation. Omega Internat. }.
Management Sci. 24(4) 399-412.
Rao, H. R., C. C. Pegels, A. F. Salam, K. T. Hwang, V. Seth. 1995. The
impact of EDI implementation commitment and implementa-
tion success on competitive advantage and firm performance.
Inform. Systems J. 5(3) 185-202.
Roach, S. 1987. America's technology dilemma: A profile of the in-
formation economy. Special economic study, Morgan Stanley,
New York.
Robey, D., M. C. Boudreau. 1999. Accounting for the contradictory
organizational consequences of information technology: Theo-
retical directions and methodological implications. Inform. Sys-
tems Res. 10(2) 167-185.
Schulz, Kenneth F., Douglas G. Altman. 1996. Methodological qual-
ity in randomized trials and statistical heterogeneity in meta-
analyses. Controlled Clinical Trials 17(2) 50-51.
Shu, W., B. Lee, N. Menon. 2001. Information technology productiv-
ity paradox research: A tragedy of statistical illusion? Working
paper. University of Illinois, Chicago, IL.
Siegel, D. 1997. The impact of computers on manufacturing produc-
tivity growth: A multiple-indicators, multiple-causes approach.
The Rev. Econom. Statist. 68-78.
144 INFORMATION SYSTEMS RESEARCH/VOI. 14, No. 2, June 2003
KOHLI AND DEVARAJ
Measuring Information Technology Payoff
Sircar, S., J. Turnbow, B. Bordoloi. 1998. The impact of information
technology investments on firm performance: A review of lit-
erature. Engrg. Valuation and Cost Analysis 1 171-181.
Snh, C, M. Markus. 1995. How IT creates business value: A process
theory synthesis. Proc. 16th Internat. Conf. Inform. Systems, Am-
sterdam, The Netherlands, 29-41.
Sohal, A. S., S. Moss, L. Ng. 2001. Comparing IT success in manu-
facturing and service industries, tnternat. ]. Oper. Production
Management 21(1-2) 30-45.
Stoneman, P., Myung Joong Kwon. 1996. Technology adoption and
firm profitability. The Econom. j . 106 952-962.
Strassmann, Paul A. 1985. Information Payoff: The Transformation of
Work in the Electronic Age. Free Press; Collier Macmillan, New
York, London, U.K.
. 1990. The Business Value of Computers: An Executiz>e's Guide. In-
formation Economics Press, New Canaan, CT.
Jam, K. 1998a. Analysis of firm-level computer investments: A com-
parative study of three Pacific-Rim economies. IEEE Trans.
Engrg. Management 45(3) 276-286.
-. 1998b. The impact of information technology investments on
firm performance and evaluation: Evidence from newly indus-
trialized economies. Inform. Systems Res. 9(1) 85-98.
Teo, T., P. Wong. 1998. An empirical study of the performance im-
pact of computerization in the retail industry. Omega Internat.
j . Management Sci. 26(5) 611-621.
Van Asseldonk, M., R. Huriane, A. Dijkhuizen. 1988. Effects of in-
formation technology on dairy farms in The Netherlands: An
empirical analysis of milk production records. /. Dairy Sci.
81(10) 2752-2759.
Wang, H. 1997. The value of information technology: An empirical
study of the impact of IT capabilities and IT use on economic
growth and on organizational performance in Taiwan. Univer-
sity of Hawaii.
Ward, J., P. Taylor, P. Bond. 1996. Evaluation and realisation of IS/
IT benefits: An empirical study of current practice. Eur. }. In-
form. Systems 4(4) 214-225.
Weill, Peter. 1992. The relationship between investment In infor-
mation technology and firm performance: A study of the valve
manufacturing sector. Inform. Systems Res. 3(4) 307-333.
White, H. 1980. A heteroskedasticity-consistent covariance matrix
estimator and a direct test for heteroskedasticity. Econometrica
48(4)817-838.
Xia, W. 1998. Dynamic capabilities and organizational impact of IT
infrastructure: A research framework and an empirical inves-
tigation (information technology infrastructure). Unpublished
Ph.D. dissertation. University of Pittsburgh, Pittsburgh, PA.
Tridas Mukhopadhyay, Associate Editor. This paper was received on August 15, 2000, and was with the authors 7 months for 2 revisions.
INFORMATION SYSTEMS RESEARCH/VOI. 14, No. 2, June 2003
145