The Clearing House: A Journal of Educational Strategies,

Issues and Ideas

ISSN: 0009-8655 (Print) 1939-912X (Online) Journal homepage: http://www.tandfonline.com/loi/vtch20

Evaluating Staff Development

Dr. Donald C. Orlich

To cite this article: Dr. Donald C. Orlich (1989) Evaluating Staff Development, The
Clearing House: A Journal of Educational Strategies, Issues and Ideas, 62:8, 370-374, DOI:
10.1080/00098655.1989.10114097

To link to this article: http://dx.doi.org/10.1080/00098655.1989.10114097

Published online: 29 Jul 2010.

Submit your article to this journal

Article views: 2

View related articles

Citing articles: 1 View citing articles

Full Terms & Conditions of access and use can be found at

http://www.tandfonline.com/action/journalInformation?journalCode=vtch20

Download by: [UNSW Library] Date: 22 April 2016, At: 19:18

 Evaluating Staff Development
Four Decision-Making Models
DONALD C. ORLICH
T he goal of all inservice education, given adequate
resources, is to offer the highest quality program
possible. If staff developers are to achieve and maintain
high-quality programs, they must collect trustworthy in-
formation so that their immediate and future decisions
are based on empirical data. There are a variety of gen-
eral evaluation models from which staff developers may
select to assess their programs.
Ernest R. House (1978) identified eight major evalua-
tion models, their assumptions, and uses. These eight
Downloaded by [UNSW Library] at 19:18 22 April 2016
models are applicable to staff development and subsume
the program evaluation strategies described by Bruce W.
Tuckman (1985). The models described in Houses tax-
onomy are the following:
1. Systems Analysis
2. Behavioral Objectives
3. Decision Making
4. Goal-Free
5. Art Criticism
6. Accreditation
7. Adversary
8. Transaction
According to House, models one through four attempt to
provide objectivity or explicit measures. Models five
through eight tend to provide more subjective, or im-
plicit, knowledge about that which is being judged.
For the purpose of this paper, only four decision-mak-
ing models will be discussed, as these four tend to be the
most practical and efficient methods of evaluating staff
development programs and are the ones most often used.
One assumption this writer makes is that any evalua-
tion activity must ultimately lead to better decision mak-
ing. If an evaluation model is to be helpful, it must be
viewed as a process by which staff development programs
Dr. Orlich k a professor in the Department of Educa-
tional Adminktration and Supervision at Washington
State University.
370
are improved through rational decision making based
upon objectively collected data.
Decision-Making Models
The models selected have various proponents: Daniel
L. Stufflebeam (CIPP Model), Marvin C. Alkin (CSE
Model), Michael Scriven (formative and summative),
Matthew B. Miles (temporary systems). Each model is ex-
panded here and applied to staff development programs.
CIPP Model
Daniel L. Stufflebeam (1971) proposed the CIPP
Model as having four major components: (1) context, (2)
input, (3) process, and (4) product-leading to the acro-
nym CIPP. Each component requires specific decision-
making actions. Stufflebeam views the decision-making
process as one that modifies, adjusts, sustains, or discon-
tinues any program or any of its parts.
Context evaluation, according to Stufflebeam, is con-
ducted at the activity site so that one may gather informa-
tion concerning needs, problems, and objectives. The
context evaluation is a reality check.
One uses input evaluation to gather data on the
strengths and weaknesses of alternative strategies, each
of which could probably accomplish the programs ob-
jectives. For example, if a series of workshops were to be
conducted on the subject of preventing accidents in
sports, then one would consider what resources would be
needed to achieve the projects stated mission.
Process evaluation methods are used to determine the
various techniques, strategies, and designs by which pro-
cedures are used in a program.
Product evaluation refers to a final, overall decision-
making process in which one decides whether to (1) con-
tinue a project as is, (2) modify the project and continue
to use it, or (3) terminate the project.
To use the CIPP Model, a school district must have
personnel who can identify the specific elements of each
of these CIPP components. These personnel have to
specify exactly what they consider to be the context, in-
put, process, and product components. Second, person-
 1989, Val. 62 Staff Development
nel must be designated to collect the data for each of the
four areas. Finally, a series of standards must be devel-
oped by which the evaluation itself is judged either mean-
ingful or useless. The CIPP Model is rather complex and
requires well-trained evaluators.
The CSE Model
Marvin C. Alkin (1970) suggested the more eclectic
CSE Evaluation Model. (CSE refers to UCLAs Cen-
ter for the Study of Evaluation.) The basic principle be-
hind the CSE Model is that evaluation is an ongoing
process that helps decision makers to select among alter-
natives in a more informed way. (This element is found in
all models in some form.) Yet it should be noted that to
select among alternatives means that viable alternatives
are in fact available. Evaluation must be viewed, argues
Alkin, as a means by which directions can be changed,
programs modified or eliminated, and personnel reshuf-
fled as need be. In most cases, staff developers do not use
Downloaded by [UNSW Library] at 19:18 22 April 2016
evaluation to specify alternative directions.
Alkin identified five decision areas and their concomi-
tant evaluation requirements.
Decisions Evaluations
1. Selections of objec- Needs assessment
tives or problems
2. Programs to meet Plans
objectives
3. Program operations Implementation
4. Program improvement Progress
5. Program certification Outcomes
Alkin stresses that each of the five paired areas re-
quires the collection of information, an evaluation of
that information or data, and, finally, a decision based
on the quantifiable information. In all steps, the evalua-
tor must realize that the judgments are based on a proba-
bility of success.
When one judges a staff development program, the
first two pairs would be most critical, that is, objectives/
needs assessments and program/plans. However, the
judging of instructional components of the program
would rely chiefly on the last three pairs-operations/im-
plementations, improvement/progress, and certifica-
tion/outcome.
The CSE Model requires continuous interaction
among all elements and providers of a staff development
program. Feedback from all participants in an inservice
project is critical in the CSE Model. The feedback is used
by project directors who want to capitalize on their own
human resources.
Formative-SummativeEvaluation
The basic objectives of any evaluation system are to
determine (1) the extent to which the project objectives
are being achieved and (2) the impact of the project on
371
the participants. To accomplish these evaluation objec-
tives, staff developers may choose to use two additional
evaluation methodologies: formative and summative.
Michael Scriven (1%7) and others have suggested these
modes. Let us examine the components.
Formative evaluation is designed to provide ongoing
feedback as quickly as possible. Formative instruments
are specifically designed to monitor the activities or com-
ponents of a program as they take place, in order to de-
termine where problems are emerging. Formative evalua-
tion allows problems to be speedily identified and recti-
fied.
Only a few selected items need to be checked in the
course of a formative evaluation. These would all be based
on the stated learning objectives for the project. The im-
portant point is to collect feedback while enough time re-
mains to make corrections.
Summative evaluation is conducted as the final assess-
ment of a project (or part of a project). Summative eval-
uations may take several forms, as long as they are con-
sistent with the prescribed objectives of the program.
Summative data can be tabulated into absolute responses
and then given as a percentage for each item. Compari-
sons between participants may be made on summative
data (but not on the formative measures). Recall that for-
mative evaluations are designed to give feedback,
whereas summative evaluation is for grading. These
evaluations are placed at logical points in the project,
such as at the ends of units, learning activities, or pro-
gram elements. Most important, a single summative eval-
uation is inadequate. The summative sets are arranged in
profiles to illustrate the sum of evaluation activities.
Temporary Systems Approach
In any social system, permanent structures, such as in-
stitutions or organizations, endure beyond the lifetimes
of their members. Humans expect their institutions to ex-
ist forever. Churches, the military, colleges, govern-
ments, families, and the schools are all permanent fea-
tures of society. Thus, we live in a society marked by per-
manence and the knowledge that some things simply do
not change very much.
Matthew B. Miles (1%) made these observations and
then asked the question, How do permanent organiza-
tions change? One apparent solution was through the
establishment of temporary systems within the perma-
nent ones. Temporary systems, noted Miles, operate for
only short durations, have well-established goals, and are
expected to end after short periods of time. Workshops,
conferences, clinics, seminars, and training sessions, in
which a small number of people meet for a defined pe-
riod of time to achieve a specified set of goals or objec-
tives, operate as temporary systems.
While involved in a temporary system, participants
temporarily drop most of their usual roles and responsi-
bilities (from their permanent systems) and concentrate
 372 The Clearing House April

on a few short-term objectives. The participants know
that they will be in the temporary system for only a brief
period. During the temporary system phase of a training
project, participants are free to try out new ideas, prac-
tice a new technique without the usual penalties for mis-
takes, and work in a generally supportive and noncom-
petitive climate removed from back-home interruptions.
FIGURE 1
Project Desires
Directions: Please place an X on the line between the
paired words to indicate your desires for
this workshop.
I desire that this workshop be
1. More information - - - More practice
oriented oriented
- - - Conducted with
Downloaded by [UNSW Library] at 19:18 22 April 2016
To install a temporary system into the evaluation para-
digm of any inservice education project, staff developers
must consider five rather simple phases: (1) planning or
preparing for the project; (2) organizing for the projects
start-up; (3) operating the project; (4) closing the system,
that is, preparing the participants for their customary
roles; and ( 5 ) implementing the strategies learned in the
project.
The evaluation of staff development activities is time-
consuming and requires a commitment to using data that
are generated. However, evaluations of all the activities
of any inservice project should be carried out to ensure
that the following take place:
Staff reactions and perceptions are obtained.
Adjustments are made as needed.
Successful activities are identified for future use.
Outstanding presenters are identified and used again.
Success or failure of a project can be determined early.

2. Conducted by
lecture as the hands-on Long- or short-range profiles are compiled.
primary format experiences Participants learn that their evaluations have an impact
3. Oriented toward - - - Oriented toward on staff development.
student problems teacher problems
4. Theoretically - - - Experientially When using a temporary systems model, data must be
oriented oriented systematically collected from all participants. For exam-
5. Balanced: - - - Information-giving ple, if a workshop leader is unsure about the format and
lectures, only emphasis workshop participants want, he or she would
demonstrations,
and activities query them, using a questionnaire such as that shown in
figure 1. Decisions about the presentations could then be

FIGURE 2
Feedback (Use after full-day session)

Directions: Please place an X on the line that best describes your response.

1. Do you think your time is being used effectively?

Used very Used Used Used very

effectively effectively ineffectively ineffectively

2. Is there adequate discussion following activities?

Very Very
adequate Adequate Inadequate inadequate

3. Are the project sessions informative enough?

Very Very
informative Informative Uninformative uninformative

4. To what extent is the material presented relevant to your classroom instruction?

Very Very
relevant Relevant Irrelevant irrelevant

5. Any suggestions or comments?

 1989, Vol. 62 Staff Development 373

FIGURE 3
Perceptions

You are having a variety of experiences during this project and, of course, these experiences affect what you
learn. These experiences and your consequent learning will help the program director improve the project.
For each item below, circle the number showing how well you think the management tasks have been done by
the director and the staff.

Low 0 1 2 3 4 5 6 7 8 9 High

Frequencies
1. Project goals were not specified 0 1 2 3 4 5 6 7 8 9 Project goals were specified clearly.
clearly.

2. The climate of this project was 0 1 2 3 4 5 6 7 8 9 The climate of this project was very
poor. good.

3. The wrong people came to this 0 1 2 3 4 5 6 7 8 9 The right people came to this
project. project.

4. The overall design of this project 0 1 2 3 4 5 6 7 8 9 The overall design of this project
was ineffective was quite effective.
Downloaded by [UNSW Library] at 19:18 22 April 2016

5. This project did not get off to a 0 1 2 3 4 5 6 7 8 9 This project did get off to a very good
good start. start.

6. This project will have no influence 0 1 2 3 4 5 6 7 8 9 This project will strongly influence
on how I teach. how I teach.

7. Staff resources were poorly used 0 1 2 3 4 5 6 7 8 9 Staff resources were well used in
in this project. this project.

8. No experiential or hands-on 0 1 2 3 4 5 6 7 8 9 Experiential or hands-on activi-

learning activities were used in ties have been frequently used in this
this project. project.

9. I would definitely not recommend 0 1 2 3 4 5 6 7 8 9 I would definitely recommend that

that this project be conducted this project be repeated for others.
for others.

Note: See Orlich and Hannaford (1986) for a 14-year longitudinal study using this model. The elements of goals, climate,
design, and influence tend to be critical for success.

adjusted before the workshop begins. Feedback is an es-
sential part of this model (figure 1).
Figure 2 shows how feedback can be obtained on four
elements of a typical inservice session. A final, or sum-
mative, evaluation could be conducted using the model in
figure 3.
A school district staff developer could compile the data
obtained from the various instruments and prepare pro-
files of various inservice efforts. Decision-making evalua-
tion models are used to find out how well the project was
perceived, in order to make rational decisions in the fu-
ture. Trends that are observed can be maintained, if desir-
able, or changed if the trends are considered undesirable.
The author has been using temporary systems manage-
ment on all inservice projects that he has directed since
1972. There is no doubt that this method has improved
the operation of the projects. In 1986, Marion Hanna-
ford and the author reported on a 14-year longitudinal
study that used the temporary systems approach. Of the
traits listed in figure 3, four tended to be critical for suc-
cess: (1) clear goals, (2) positive climate, (3) overall de-
sign, and (4) influence on instruction. These findings may
not be surprising, for they represent elements critical for
successful staff development programs.
Conclusion
There are many reasons to evaluate staff development
programs and inservice education projects. Depending
on the project objectives, one could use group tests, stu-
dent achievement tests, classroom observations, attitude
scales, or anecdotal records as mechanisms for determin-
ing the success of an inservice project. When using deci-
sion-making evaluation models, staff developers can
make critical adjustments in any training program so that
every participant masters the content and, more impor-
tant, uses the skills or knowledge as was intended.
 374 The Clearing House
Bruce Joyce and Beverly Showers (1988) provided a
practical and realistic rationale for evaluating inservice
programs. They concluded that if we truly intend to in-
crease student learning through staff development pro-
grams, serious evaluation of those programs will be nec-
essary (p. 127). No question about it, systematic evalua-
tion of inservice projects is essential.
REFERENCES
Alkin, M. C. 1970. Products for improving educational evaluation.
Evaluation Comment, 3(1): 1-4.
Downloaded by [UNSW Library] at 19:18 22 April 2016
April
House, E. R. 1978. Assumptions underlying evaluation models. Educa-
tional Reseurcher, 7(3): 4 1 2 .
Joyce, B., and B. Showers. 1988. Student achievement through staff
development. New York: Longman.
Miles, M. B. 1964. On temporary systems. In Innovations in education,
edited by Matthew B. Miles. New York: Teachers College Press.
Orlich, D. C., and M. E. Hannaford. 1986. Longitudinal evaluation
strategy for staff development. Paper presented at the National Staff
Development Council Annual Conference, Atlanta, Georgia,
December 15.
Scriven, M. 1967. The methodology of evaluation. AERA Monograph
Series on Curriculum Evaluation, No. 1: 39-83.
Stufflebeam, D. L. 1971. The relevance of the CIPP evaluation model
for educational accountability. Journal of Research and Development
in Education, 5(1): 19-21.
Tuckman, B. W . 1985. Evaluating instructional program, 2d ed.
Boston: Allyn and Bacon.