Beruflich Dokumente
Kultur Dokumente
Deniz Eseryel
Syracuse University, IDD&E, 330 Huntington Hall
Syracuse, New York 13244 USA
Tel: +1 315 443 3703
Fax: +1 315 443 9218
deseryel@mailbox.syr.edu
ABSTRACT
Introduction
Evaluation is an integral part of most instructional design (ID) models. Evaluation
tools and methodologies help determine the effectiveness of instructional
interventions. Despite its importance, there is evidence that evaluations of training
programs are often inconsistent or missing (Carnevale & Schulz, 1990; Holcomb,
1993; McMahon & Carter, 1990; Rossi et al., 1979). Possible explanations for
inadequate evaluations include: insufficient budget allocated; insufficient time
allocated; lack of expertise; blind trust in training solutions; or lack of methods and
tools (see, for example, McEvoy & Buller, 1990).
Part of the explanation may be that the task of evaluation is complex in itself.
Evaluating training interventions with regard to learning, transfer, and organizational
impact involves a number of complexity factors. These complexity factors are
associated with the dynamic and ongoing interactions of the various dimensions and
attributes of organizational and training goals, trainees, training situations, and
instructional technologies.
• Goal-based evaluation
• Goal-free evaluation
• Responsive evaluation
• Systems evaluation
• Professional review
• Quasi-legal
Kirkpatrick (1959) CIPP Model (1987) IPO Model (1990) TVS Model (1994)
1. Reaction: to 1. Context: obtaining 1. Input: evaluation of 1. Situation:
gather data on information about system performance collecting pre-
participants the situation to indicators such as training data to
reactions at the end decide on trainee qualifications, ascertain current
of a training educational needs availability of levels of
program and to establish materials, performance within
program objectives appropriateness of the organization and
training, etc. defining a desirable
level of future
performance
On the other hand, systems-based models (e.g., CIPP, IPO, and TVS) seem to be
more useful in terms of thinking about the overall context and situation but they may
not provide sufficient granularity. Systems-based models may not represent the
dynamic interactions between the design and the evaluation of training. Few of these
models provide detailed descriptions of the processes involved in each steps. None
provide tools for evaluation. Furthermore, these models do not address the
collaborative process of evaluation, that is, the different roles and responsibilities
that people may play during an evaluation process.
This situation does not seem to be very different in Europe, as evident in two
European Commission projects that have recently collected data exploring evaluation
practices in Europe. The first one is the Promoting Added Value through Evaluation
(PAVE) project, which was funded under the European Commission’s Leonardo da
Vinci program in 1999 (Donoghue, 1999). The study examined a sample of
organizations (small, medium, and large), which had signaled some commitment to
training and evaluation by embarking on the UK’s Investors in People (IiP) standard
(Sadler-Smith et al., 1999). Analysis of the responses to surveys by these
organizations suggested that formative and summative evaluations were not widely
used. On the other hand, immediate and context (needs analysis) evaluations are
more widely used. In the majority of the cases, the responsibility for evaluation was
that of managers and the most frequently used methods were informal feedback and
questionnaires. The majority of respondents claimed to assess the impact on
employee performance (the ‘learning’ level). Less than one-third of the respondents
claimed to assess the impact of training on organization (the ‘results’ level).
Operational reasons for evaluating training were cited more frequently than strategic
ones. However, information derived from evaluations was used mostly for feedback
to individuals, less to revise the training process, and rarely for return on investment
decisions. Also, there were some statistically significant effects of organizational size
on evaluation practice. Small firms are constrained in the extent to which they can
evaluate their training by the internal resources of the firm. Managers are probably
responsible for all aspects of training (Sadler-Smith et al., 1999).
The second study was conducted under the Advanced Design Approaches for
Personalized Training-Interactive Tools (ADAPTIT) project. ADAPTIT is a European
project within the Information Society Technologies programme that is providing
design methods and tools to guide a training designer according to the latest
cognitive science and standardisation principles(Eseryel & Spector, 2000). In an
effort to explore the current approaches to instructional design, a series of surveys
conducted in a variety of sectors including transport, education, business, and
industry in Europe. The participants were asked about activities that take place
including the interim products produced during the evaluation process, such as a list
of revisions or an evaluation plan. In general, systematic and planned evaluation was
not found in practice nor was the distinction between formative and summative
evaluation. Formative evaluation does not seem to take place explicitly while
summative evaluation is not fully carried out. The most common activities of
evaluation seem to be the evaluation of student performance (i.e., assessment) and
there is not enough evidence that evaluation results of any type are used to revise the
training design (Eseryel et al., 2001). It is important to note here that the majority of
the participants expressed a need for evaluation software to support their practice.
Ross & Morrison (1997) suggest two categories of functions that automated
evaluation systems appear likely to incorporate. The first is automation of the
planning process via expert guidance; the second is the automation of the data
collection process.
Based on this input, an expert system can provide guidance on possible evaluation
design orientations, appropriate collection methods, data analysis techniques,
reporting formats, and dissemination strategies. Such expert guidance can be in the
form of flexible general strategies and guidelines (weak advising approach). Given
the complexities associated with the nature of evaluation, a weak advising approach
such as this is more appropriate than a strong approach that would replace the human
decision maker in the process. Indeed, weak advising systems that supplement rather
than replace human expertise have generally been more successful when complex
procedures and processes are involved (Spector et al., 1993).
Such a system may also embed automated data collection functions for increased
efficiency. Functionality of automated data collection systems may involve
intelligent test scoring of procedural and declarative knowledge, automation of
individual profile interpretations, and intelligent advice during the process of
learning (Bunderson et al., 1989). These applications can provide increased ability to
diagnose the strengths and weaknesses of the training program in producing the
desired outcomes. Especially, for the purposes of formative evaluation this means
that the training program can be dynamically and continuously improved as it is
being designed.
Conclusion
Different approaches to evaluation of training discussed herein indicate that the
activities involved in evaluation of training are complex and not always well-
structured. Since evaluation activities in training situations involve multiple goals
associated with multiple levels, evaluation should perhaps be viewed as a
collaborative activity between training designers, training managers, trainers, floor
managers, and possibly others.
There is a need for a unifying model for evaluation theory, research, and practice that
will account for the collaborative nature of and complexities involved in the
evaluation of training. None of the available models for training evaluation seem to
account for these two aspects of evaluation. Existing models fall short in
comprehensiveness and they fail to provide tools that guide organizations in their
evaluation systems and procedures. Not surprisingly, organizations are experiencing
problems with respect to developing consistent evaluation approaches. Only a small
percentage of organizations succeed in establishing a sound evaluation process that
feeds back into the training design process. Evaluation activities are limited to
reaction sheets and student testing without proper revision of training materials based
on evaluation results. Perhaps lack of experience in evaluation is one of the reasons
for not consistently evaluating. In this case, the organization may consider hiring an
external evaluator, but that will be costly and time consuming. Considering the need
for the use of internal resources and personnel in organizations, expert system
technology can be useful in providing expert support and guidance and increase the
power and efficiency of evaluation. Such expert systems can be used by external
evaluators as well.
References
• Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and
directions for future research. Personnel Review, 26(3), 201-213.
• Bassi, L. J., & van Buren, M. E. (1999). 1999 ASTD state of the industry
report. Alexandria, VA: The American Society for Training and
Development.
• Bramley, P. (1996). Evaluating training effectiveness. Maidenhead: McGraw-
Hill.
• Bernthal, P. R. (1995). Evaluation that goes the distance. Training and
Development Journal, 49(9), 41-45.
• Bunderson, C. V., Inouye, D. K., & Olsen, J. B. (1989). The four generations
of computerized educational measurement. In R. L. Linn (Ed.). Educational
measurement (3rd ed.) (pp. 367-407). New York: Macmillan.
• Bushnell, D. S. (March, 1990). Input, process, output: A model for evaluating
training. Training and Development Journal, 44(3), 41-43.
• Carnevale, A. P., & Schulz, E.R. (July, 1990). Return on investment:
Accounting for training. Training and Development Journal, 44(7), S1-S32.
• Dixon, N. M. (1996). New routes to evaluation. Training and Development,
50(5), 82-86.
• Donoghue, F. (1999). Promoting added value through evaluation of training.
Dublin: European Commission Leonardo-PAVE Project.
• Eseryel, D., Schuver-van Blanken, M., & Spector, J. M. (2001). Current
practice in designing training for complex skills: Implications for design and
evaluation of ADAPT-IT. In C. Montgomerie & J. Vitelli (Eds.),
Proceedings of ED-MEDIA 2001: World Conference on Educational
Multimedia, Hypermedia, & Telecommunications (pp. 474-479). Tampere,
Finland: Association for Advancement of Computing in Education.
• Eseryel, D., & Spector, J. M. (2000). Assessing adaptive instructional design
tools and methods in ADAPT-IT. In M. Crawford & M. Simonson (Eds.),
Annual Proceedings of Selected Research and Development Papers
Presented at the National Convention of the Association for Educational
Communications and Technology (Vol. 1) (pp. 121-129). Denver, CO:
Association for Educational Communications and Technology.
• Fitz-Enz, J. (July, 1994). Yes…you can weigh training’s value. Training,
31(7), 54-58.
• Gagné, R., & Briggs, L. J. (1974). Principles of instructional design. New
York: Holton, Rinehart & Winston.
• Goldstein, I. (1993). Training in organizations: Needs assessment,
development, & evaluation. Monterey, CA: Brooks-Cole.
• Gordon, J. (August, 1991). Measuring the “goodness” of training. Training,
28(8), 19-25.
• Gustafson, K. L, & Branch, R. B. (1997). Survey of instructional
development models (3rd ed.). Syracuse, NY: ERIC Clearinghouse on
Information and Technology.
• Hamblin, A. C. (1974). Evaluation and control of training. Maidenhead:
McGraw-Hill.
• Holcomb, J. (1993). Make training worth every penny. Del Mar, CA:
Wharton.
• Kirkpatrick, D. L. (1959). Techniques for evaluating training programs.
Journal of the American Society of Training Directors, 13, 3-26.
• Mager, R. F. (1962). Preparing objectives for programmed instruction. San
Francisco, CA: Fearon Publishers.
• McEvoy, G. M., & Buller, P. F. (August, 1990). Five uneasy pieces in the
training evaluation puzzle. Training and Development Journal, 44(8), 39-42.
• McMahon, F. A., & Carter, E. M. A. (1990). The great training robbery.
New York: The Falmer Press.
• Phillips, J. J. (1991). Handbook of training evaluation and measurement
methods. (2nd ed.). Houston, TX: Gulf.
• Phillips, J. J. (July, 1997). A rational approach to evaluating training
programs including calculating ROI. Journal of Lending and Credit Risk
Management, 79(11), 43-50.
• Rossi, P.H., Freeman, H. E., & Wright, S. R. (1979). Evaluation: A
systematic approach. Beverly Hills, CA: Sage.
• Ross, S. M., & Morrison, G. R. (1997). Measurement and evaluation
approaches in instructional design: Historical roots and current perspectives.
In R. D. Tennyson, F. Scott, N. M. Seel, & S. Dijkstra (Eds.), Instructional
design: Theory, research and models. (Vol.1) (pp.327-351). Hillsdale, NJ:
Lawrence Erlbaum.
• Sadler-Smith, E., Down, S., & Field, J. (1999). Adding value to HRD:
Evaluation, investors in people, and small firm training. Human Resource
Development, 2(4), 369-390.
• Spector, J. M., Polson, M. C., & Muraida, D. J. (1993) (Eds.). Automating
instructional design: Concepts and issues. Englewood Cliffs, NJ: Educational
Technology Publications, Inc.
• Tennyson, R. D. (1999). Instructional development and ISD4 methodology.
Performance Improvement, 38(6), 19-27.
• Warr, P., Bird, M., & Rackcam, N. (1978). Evaluation of management
training. London: Gower.