Performance Variability, Ambiguity Intolerance, and Balanced Scorecard-Based

Performance Assessments

Stephen L. Liedtka De Sales University Bryan K. Church Georgia Institute of

TechnologyManash R. RayCost Technology, Inc.

ABSTRACT: This study extends prior research on general Balanced Scorecard (BSC)
evaluation tendencies (e.g., Lipe and Salterio 2000; Ittner et al. 2003; Banker et
al.2004) by documenting that patterns in BSC evaluations vary with a quality of the
evaluator. Specifically, using data from an experimental performance assessment
exercise, we find that evaluators ambiguity intolerance (Budner 1962) influences
their reaction to variation among performance measures within a BSC category.
Further, we find that increased variation within a BSC category causes ambiguity-
intolerant evaluators to give lower performance evaluation scores when the BSC
category indicates relatively strong performance, but has no significant effect when
the BSC category indicates relatively weak performance. These results are
consistent with the argument that ambiguity intolerant individuals are more likely to
discount or ignore ambiguous information when the ambiguity relates to positive
information. Our findings have significant practical implications regarding the
accuracy and consistency of BSC evaluations.

Keywords: Balanced Scorecard; ambiguity; ambiguity intolerance; performance

evaluation.

Data Availability: Contact the first author for data availability

INTRODUCTION

Academics and practitioners long have argued that traditional, financial-oriented

performance measurement systems provide an incomplete view of corporate
performance (Drucker 1954; American Institute of Certified Public Accountants
[AICPA]1994). The Balanced Scorecard (BSC) addresses this criticism by
supplementing financial performance measures with measures from customer-
related, internal business processes, and organizational learning and growth
categories (Kaplan and Norton 1992).Since its introduction, the BSC has been widely
adopted by commercial enterprises, non-profit organizations and government
entities (Kaplan and Norton 2001).

Consistent with the recommendations of Kaplan and Norton (1996, 217222), firms
commonly use the BSC to evaluate and reward managers. A survey of 60 BSC users
reveals that 70 percent base managers compensation on the BSC or some variant,
17 percent have actively considered the use of the BSC to determine compensation,
and 15 percent use the BSC to evaluate performance, but not to determine
compensation (Ittner and Larcker 1998,221). BSC proponents believe that by linking
evaluations to BSC objectives, firms can better align managerial effort with
organizational strategies and goals and thus improve overall performance (Kaplan
and Norton 1996, 217).

Academic research identifies situational factors that influence whether evaluators

fully consider each BSC performance measure and make performance evaluation
judgments consistent with their organizations global strategies and goals. Lipe and
Salterio (2000), for example, conduct an experiment in which participants used
BSCs to evaluate performance of two separate divisions of a firm. The resulting
evaluations indicate that the participants only considered performance measures
that were common to both BSCs and ignored measures that were unique to either
division. Ittner et al. (2003) employ archival data from a financial services firm and
find evidence that the firms BSC evaluators over emphasized financial category
performance and failed to consider BSC measures that are predictive of future
financial results.

These prior studies suggest that general human cognitive tendencies and other
factors can result in performance evaluations that are not in line with the strategies
and goals articulated in a BSC. This evidence indicates there may be a need to
modify or clarify BSC-based performance evaluation procedures. Several
experimental studies, for instance, focus on strategies to improve BSC-based
evaluation judgments by mitigating the common measures bias observed by Lipe
and Salterio (2002). These strategies include giving evaluators assurance of BSC
information quality and making evaluators justify their evaluations(Libby et al.
2004), supplying detailed strategy information to evaluators (Banker et al.2004),
disaggregating the BSC assessment process (Roberts et al. 2004), and providing
evaluators with supplemental training (Dilla and Steinbart 2005a).

This study extends and diversifies research on general BSC evaluation tendencies
by documenting that patterns in BSC-based performance assessments vary with a
quality of the evaluator. We examine (1) whether evaluators ambiguity
intolerance (Budner 1962) influences their reaction to variation among
performance measures within a BSC category, and (2) if that influence depends
upon whether the overall performance in that category is relatively strong or
relatively weak. Previous research provides evidence that merely placing
performance measures in a BSC category primes evaluators to expect a relationship
among those measures (Lipe and Salterio 2002). As indicated by the rationale for
employing multiple measures, the various measures within a BSC category are not
expected to produce identical signals regarding performance. To the extent that
these signals vary, psychology research predicts that there is increased ambiguity
regarding the appropriate response to the relevant judgment task (Budner 1962;
Norton 1975). Psychology research also finds that individuals differ in their
willingness to tolerate ambiguity (Budner 1962; Norton 1975). Individuals who are
uncomfortable with ambiguity may respond by discounting or ignoring ambiguous
information (Van Dijk and Zeelenberg 2003). Hence, differences in evaluators
willingness to tolerate ambiguity may lead evaluators to differ in the extent to
which they consider or ignore BSC categories with highly varied performance
results. Furthermore, the application of Prospect Theory (Kahneman and Tversky
1979) suggests that whether ambiguity within a BSC category causes evaluators
discomfort may depend on whether the relevant BSC category reports positive or
negative information. Hence, ambiguity-intolerant evaluators may react
differently to ambiguity depending upon the situation.

Investigating these issues has important implications for practice. For example, if
ambiguity-tolerant and intolerant evaluators provide differing evaluations of
managers based on the same BSCs, we contend that at least one groups
evaluations are inaccurate and potentially not in line with a companys strategic
goals. Such evaluations, in turn, can lead to suboptimal strategic choices including
improper allocation of resources to managers and poor promotion decisions.
Performance evaluation processes also can influence the decision-making of
subordinates, who may shift effort away from those activities that appear to
beignored by evaluators (McNamara and Fisch 1964; Holmstrom and Milgrom
1991).Furthermore, employees may view inconsistent evaluations as arbitrary and
unfair, which can reduce the employees incentive to provide effort (Gibbs et al.
2004) and create dissatisfaction with the BSC (Dilla and Steinbart 2005b). We
elaborate further on these practical implications in our discussion section.

Our study required 85 MBA students to complete an experimental performance

evaluation exercise. Data from the experiment allowed us to investigate three
hypotheses related to whether variability (ambiguity) among performance measures
within a BSC category is associated with evaluators discounting or ignoring the BSC
category in making their evaluations. First, we predict that variability within a BSC
category will not affect the overall performance evaluations of ambiguity-tolerant
evaluators, because such evaluators are not expected to experience the discomfort
that may result in the discounting of ambiguous information. Second, we expect
that increased variability within a BSC category in which mean performance is
relatively strong will cause ambiguity-intolerant evaluators to give lower overall
evaluation scores (compared to ambiguity-tolerant evaluators) due to discounting of
the positive, but ambiguous, information. Third, we predict that increased variability
within a BSC category in which mean performance is relatively weak will not impact
overall performance judgments made by ambiguity-intolerant evaluators (compared
to ambiguity-tolerant evaluators). This hypothesis follows from the Prospect Theory
based argument that, when an outcome appears to be negative, ambiguity should
not present a threat to any evaluators because there are no perceived gains that
could prove to be illusory. Results presented in this paper support all three
hypotheses.