Teacher Professional Development Evaluation Guide

Teacher Professional
Development
Evaluation Guide
M. Bruce Haslam
Policy Studies Associates
Washington, D.C.
January 2010
Prepared for the National Staff

Development Council
Acknowledgments
P
reparation of the original version and schools.
of this guide for use by educators Margaret Pfaff, director of
in Maryland was supported curriculum, instruction, and staff
under contracts with the Harford development, Carroll County Public
County Public Schools and the Maryland Schools, also contributed to the
State Department of Education. development of the early version of the
Subsequent modifications of the guide guide. Fortunately, she has continued
were supported under a contract with the to offer keen insights and valuable
National Staff Development Council. I practical suggestions for revisions and
am pleased to acknowledge this support. improvement.
Many people contributed to the Finally, I would like to extend a
development of this guide. I am especially very special thanks to Colleen Seremet,
pleased to thank Carol Williamson, assistant state superintendent for
superintendent, Queen Anne’s County instruction in the Maryland State
Public Schools; Jeff Maher, director Department of Education. She has been
of professional and organizational a constant and unwavering source of
development, St. Mary’s County Public support and guidance for this work — a
Schools; and Jackie Tarbert, professional critical friend in every way possible. Her
development coordinator, Harford deep understanding of what is necessary
County Public Schools, for their to ensure high-quality professional
invaluable advice and suggestions at every development for all teachers, her insights
step in the development of the initial about what is possible, and her ambitious
version of this guide. Their deep interest expectations for what should happen set
in teacher professional development and a high standard for what is attainable.
intensely pragmatic perspectives on what
is possible and necessary to improve M. Bruce Haslam
professional learning helped keep the Washington, D.C.
guide grounded in the reality of districts January 2010
2 Teacher Professional Development Evaluation Guide National Staff Development Council www.nsdc.org
Table of contents
Acknowledgments.................................................................................................................................................2
Foreword.......................................................................................................................................................................4
Introduction................................................................................................................................................................8
Five questions to inform evaluation planning................................................................................... 12
Evaluation design and data collection strategies............................................................................ 24
Data quality and data analysis...................................................................................................................... 45
Reporting................................................................................................................................................................... 55
Appendix A: Resources for evaluating teacher professional development..................... 61
Appendix B: Sample items for surveying participants’ views of professional
development.......................................................................................................................................................... 66
Afterword.................................................................................................................................................................. 70
National Staff Development Council 800-727-7288 Teacher Professional Development Evaluation Guide 3
FORE W OR D
Foreword
O
n behalf of the Maryland standards as a way of articulating
Teacher Professional a Maryland-specific, policy-
Development Advisory relevant definition of high-quality
Council and educators from across professional development; and
Maryland, I am pleased to share an • Offer recommendations for ongoing
important part of our work around improvements in professional
teacher professional development with development programs and policies
colleagues in other states who share our to ensure that they meet the new
commitment to quality professional standards.
development as an integral part of After reviewing the advisory council’s
professional practice. initial reports and sharing them with the
We began in 2003, when Dr. Maryland State Board of Education, Dr.
Nancy S. Grasmick, the Maryland State Grasmick affirmed the council’s ongoing
Superintendent of Schools, created work and refined her original charge.1
the Maryland Teacher Professional In December 2006, she asked us to look
Development Advisory Council. at state and local efforts to apply our new
Recognizing the potential of high-quality standards in creating a statewide system
professional learning for all teachers as a of high-quality professional development
key to helping all students succeed, we and to identify challenges associated with
called on our 26 members to: implementing the standards.
• Look carefully at teacher professional As part of its review of challenges
development policies and programs associated with applying the new
at the state and local levels; professional development standards,
• Set professional development the council examined the state of the
1 Beginning with its 2004 report, Helping Teachers Help All Students: The Imperative for High-
Quality Professional Development, the advisory council produced four reports. These reports, as
well as a number of other professional development products, are available at the Maryland State
Department of Education web site: www.marylandpublicschools.org/MSDE/divisions/instruction/
prof_standards.
FORE W OR D
art of evaluating teacher professional the planning framework in the Maryland

development in Maryland. The council Teacher Professional Development
reviewed evaluation activities at both our Planning Guide is a useful tool for
state Department of Education (MSDE) designing rigorous evaluations. The
and in our 24 local school districts and Council also notes that the forthcoming
offered the following assessment in its guide for evaluating teacher professional
2008 report to Dr. Grasmick and the development, which will complement the
State Board of Education: planning guide, will also be a useful tool
“Overall, the council concludes for designing rigorous evaluations.
that the increase in state and local “In urging improvements in
efforts to evaluate teacher professional evaluating teacher professional
development, combined with increased development, the council is acutely aware
awareness of the need for more rigorous that there are few good models elsewhere
evaluations, set the stage for significant in the country to guide improvements
progress in the future. The council in Maryland. At the same time,
recognizes that ongoing review of heightened demands for accountability
teachers’ perceptions of their professional and solid evidence of reasonable returns
learning experiences, both in terms of on investments make it imperative
how these experiences are organized to improve evaluations of teacher
and the perceived likelihood that they professional development.”
will result in changed in teaching and As the council noted in this report,
learning, are important. At the same a consortium of several districts and
time, the council concludes that the MSDE worked with Bruce Haslam
districts, MSDE, and other stakeholders to develop the initial version of
need to work together to improve the Maryland Teacher Professional
both the scope and quality of their Development Evaluation Guide. Our
evaluation efforts. In particular, future evaluation guide was intended to
evaluation efforts should focus on careful complement the Maryland Teacher
examination of the effect of professional Professional Development Planning
development on observable and Guide as a tool to inform planning and
measurable changes in teacher knowledge evaluations of local and state professional
and skills and classroom practice. These development programs and initiatives in
evaluations should also focus on linking Maryland.
specific changes in practice to changes in We were very pleased when the
student learning. The council notes that National Staff Development Council
FORE W OR D
(NSDC) learned of the work under way strategies that will add rigor while, at
in Maryland and commissioned revisions the same time, recognizing limits in
to the Maryland evaluation guide to capacity and resources. Finally, the guide
ensure its usefulness in other states and consistently underscores how evaluation
districts. can contribute to improving the quality
The new guide is intensely practical. of teacher professional development by
It emphasizes the idea that evaluations of informing providers and others about
professional development should reflect what is going well and where midcourse
the professional development design as corrections are needed to increase
envisioned by facilitators and designers. participation and improve outcomes
in the kinds of long-term professional
development that hold so much potential
The guide urges evaluation planners and those for strengthening practice. The results
of rigorous evaluations can also inform
responsible for conducting the evaluation to use choices about future investments in
professional development and decisions
the guide to inform evaluation design and to about continuing and perhaps expanding
it. Finally, the guide provides a useful
make choices among design options and data framework for local and state professional
development providers, facilitators, and
collection strategies that will add rigor while, at program managers to have conversations
with evaluation specialists and
the same time, recognizing limits in capacity and consultants who may be hired to assist in
the process.
resources. Based on our early experience in
Maryland, we have found the evaluation
It recognizes that evaluation planning guide to be a useful tool. At the same
is an important part of overall planning time, we are learning that time and
for professional development and should resource constraints, as well as lack of
begin early in the process. In addition, evaluation experience among potential
the guide urges evaluation planners and users, may limit the extent to which the
those responsible for conducting the guide is actually used in the field. For this
evaluation to use the guide to inform reason, I encourage state departments of
evaluation design and to make choices education and school districts to support
among design options and data collection individuals who are expected to use the
FORE W OR D
guide. Support can include seminars to explicitly reflect attention to the main
introduce the guide and its key concepts, principles emphasized in this guide.
ongoing technical assistance during Initial cycles of planning and review will
planning, and critical friend reviews of almost certainly require guidance and
evaluation plans and evaluation reports. support, as well as thoughtful feedback.
In addition to these and other kinds Our experience in Maryland, however,
of training and technical assistance, has shown a good return on these
state departments and school districts investments of people, time, and money.
can bolster professional development All of us in Maryland who have
grant programs and other initiatives participated in the early development
to support professional development of the guide are pleased to share it with
and school improvement by requiring colleagues around the country. We hope
rigorous evaluations accompanied by an that you will find the guide useful and
iterative review process. This multistep that you will share your experiences with
process includes “no-fault” reviews and us and with others in NSDC’s national
feedback in the early phases of the grant and state networks and meetings, and in
application/approval process. Balancing other forums.
requirements with no-fault feedback that
can be incorporated into requests for Colleen P. Seremet
proposals and various planning district Assistant state superintendent for instruction
and school planning processes should Maryland State Department of Education
require evaluation plans and designs that January 2010
IN T RO D UC T ION
Introduction
“Evaluating professional development enables program managers and participants to

make data-based decisions about the program. If the evaluation is done well, everyone
benefits. If done poorly, it will be a waste of resources. The most useful evaluations
result from a desire to improve both the program and its results…” (p. 140).
— Joellen Killion, Assessing Impact: Evaluating Staff Development, 2nd edition, 2008
E
ach year, thousands of teachers promote understanding in several ways.
across the country participate in a • Early or formative evaluations
range of professional development gauge teacher satisfaction with
that reflects substantial investments of professional learning and help
time and money. Yet despite widespread determine whether the professional
reliance on professional learning as a learning took place as planned,
core component of efforts to improve whether teachers mastered new
education for all children, educators have knowledge and skills, and whether
little systematic information to allow teachers applied the new knowledge
us to assess the quality of professional and skills in their classrooms.
learning or to gauge their contributions to • Formative evaluations help
professional practice and student learning. professional development
In short, there is little information on participants, facilitators, providers,
the return on the investment and little and sponsors determine whether
information to use to persuade decision the professional learning is on track
makers that the investment should be or whether midcourse changes
continued or even increased. are necessary to achieve intended
outcomes.
How evaluations can promote • Final or summative evaluations
understanding of teacher help stakeholders understand
professional development whether the professional
Rigorous, ongoing evaluations can development achieved the intended
IN T RO D UC T ION
outcomes as these outcomes are suggestions presented here apply

reflected in changes in teachers’ to a broad range of professional
professional practice and increased development. Evaluations should
student learning. be tailored to the professional
• Ongoing evaluations may yield development being evaluated.
information about changes in school Evaluation questions, data collection
organization and culture that result strategies, and reporting will vary
from teacher participation in the depending on the nature of the
professional development being activity and the evaluation’s purpose.
evaluated. Just as high-quality professional
This guide is intended to help development is a process that extends
staff in school district central offices, over time, successful evaluations
schools, state departments of education, of professional development focus
and faculty and staff in institutions on all phases of the process. In
of higher education work with other addition, the availability of resources
professional development providers and (money, people, and time) that can
consultants to plan, conduct, and report be allocated to the evaluation will
on evaluations of teacher professional influence what can be accomplished.
development. Experience suggests the Decisions about the evaluation’s
guide also can inform conversations design and scope should be realistic
about the evaluation process as district and practical.
professional development staff, • Evaluation planning should be
professional development providers, and an integral part of professional
others work with external evaluators to development planning. Those
plan evaluations. responsible for evaluation should
be on professional development
Guiding assumptions about planning teams from the beginning.
evaluating teacher professional Planning teams should think about
development evaluation as they: 1) identify
The guide rests on four assumptions the need for the professional
about planning and conducting development; 2) identify the
evaluations of teacher professional intended participants; 3) specify
development. the intended professional learning
• No single “best” approach to outcomes and related indicators and
evaluation exists, although the the expected outcomes in student
IN T RO D UC T ION
learning and related indicators; and expanded to include involvement in

4) decide what kind of professional evaluating professional development.
learning is most likely to result in the Just as they contribute to plans for
intended outcomes. Without clear professional development, teachers
decisions in each of these areas, it is can be involved in planning the
difficult to plan rigorous evaluations. evaluation by helping to sharpen
Moreover, evaluations not planned the evaluation questions, collecting
and analyzing data, and reporting
on evaluation results and making
Just as they contribute to plans for professional recommendations based on those
results. Involving teachers in these
development, teachers can be involved in and other ways complements and
extends their professional learning
planning the evaluation by helping to sharpen by increasing opportunities for
reflection, professional discourse, and
the evaluation questions, collecting and collaboration with colleagues. Being
involved also adds a dimension of
analyzing data, and reporting on evaluation teacher leadership and professional
responsibility.
results and making recommendations based on • Evaluations of teacher professional
development should be separate
those results. and distinct from teacher
performance appraisals. Evaluation
teams should establish safeguards
in advance seldom yield useful to ensure that data collection and
information and results and are reporting evaluation results do not
therefore a waste of time and money. overlap with teacher performance
• Teachers have key roles to appraisals. The safeguards should
play in evaluating professional be clearly visible to teachers and
development. A core element in others involved in the professional
NSDC’s definition of high-quality development and evaluations.
professional development is that
teachers should be actively engaged What’s in the guide?
in planning and leading their own The guide offers practical suggestions
professional learning. This role can be for planning and conducting evaluations
IN T RO D UC T ION
of teacher professional development. It as they are collected, as well as strategies

begins by posing a series of questions for data analysis. The last section offers
for planning teams to consider as they advice on preparing evaluation reports.
start working on an evaluation design The guide also includes two appendices.
and plans for conducting the evaluation. Appendix A is an annotated bibliography
The second section discusses various of evaluation resources, including data
approaches to evaluation design and collection instruments. Appendix B
options for data collection. The third includes sample items for surveys of
section discusses the importance of teacher perceptions of their professional
monitoring the quality of evaluation data learning experiences.
F i v e q u e s t i o n s to i n f o r m e va luat i o n p l a n n i n g
Five questions to inform

evaluation planning
M
any factors influence plans the checklist at the end of this section to
for evaluating teacher be sure the plan is complete.
professional development.
This section of the guide poses five 1. Should the activity be evaluated?
questions about various factors planners Because evaluations require
need to consider in designing an time and money, consider whether it
evaluation. Because the answers to these makes sense to evaluate a particular
questions will define the basic parameters professional development program or
of the evaluation, the evaluation team initiative. In some cases, the decision
should come to consensus about them will be based on whether funders, policy
before moving forward. Failing to makers, or federal and state regulations
address these questions in advance require an evaluation as a condition of
invariably results in superficial and providing support. Ideally, these kinds
incomplete data collection and analyses of professional development should be
and missed opportunities to identify evaluated, because the evaluation results
ways to make improvements and learn can provide important information
about the payoffs of teacher professional to key stakeholders, including the
development. Similarly, retrofitting an participants:
evaluation plan and data collection on • Large-scale professional development
professional development that is well (that includes large numbers of
under way, or even complete, limits data teachers, extends over relatively
collection opportunities and generally long periods of time, and/or
makes it difficult to tailor the evaluation represents significant investments of
to the professional development in professional development resources);
meaningful ways. • Professional development that is a
Answer the five questions posed here key component of state, district,
and look ahead to other elements of the or school improvement initiatives,
plan discussed in the next section. Use such as learning supported as part
of state and local strategies to satisfaction instruments and data systems

address the four education reform that can store large numbers of responses
assurances included as priorities may use satisfaction measures for an
under the American Recovery and ongoing look at satisfaction and ratings
Reinvestment Act (ARRA)2; and of usefulness to inform the planning of
• Pilot professional development future professional development. Absent
that is likely to be taken to scale other complementary data collection,
if the initial evaluation results are these results are, however, almost always
positive or suggest a clear path for cursory and of very limited use in
strengthening the learning. gauging impact and effectiveness.
Evaluations of other kinds of In most cases, the decision about
professional development are likely whether to conduct an evaluation will be
to have limited payoffs in terms of fairly easy. Nevertheless, given the cost
understanding what works and what and time necessary to conduct rigorous
doesn’t work. It makes little sense to evaluations, planning teams should think
invest in evaluating: carefully about whether to proceed. They
• One-time or short-term professional should also consider the following advice
development with few participants from Hayes Mizell (2009) as he lamented
and no prospects for scale-up; the absence of a rigorous evaluation of
• Professional development for which an important professional development
there is very limited or unclear initiative:
expectations for teacher outcomes; or “Don’t conceive and implement
• Professional development that is not a new initiative without creating, on
clearly and explicitly aligned with the front end, a sound process for
state, district, or school priorities. documenting and reporting how the
Districts and professional project unfolds and what it achieves.
development providers that have Don’t ignore the challenges of
developed standardized participant determining what data you will need
2 These assurances include: improving educator effectiveness and equitable distribution;

developing a statewide longitudinal data system; adopting rigorous standards and assessments;
and turning around low-performing schools. A recent survey of state policy makers found that
many states are including teacher professional development as part of their strategies to address
the four assurances. For more information, see An Early Look at the Economic Stimulus Package
and the Public Schools (Center on Education Policy, 2009). It is also likely that teacher professional
development will figure prominently in state applications and for ARRA Race to the Top funds and
other ARRA funds.
to collect, the availability of the data, factors are likely to influence the
who will be responsible for collecting professional development, and how are
the data at what intervals, and who will these factors likely to influence the extent
analyze and use the data to prepare a to which teachers apply new knowledge
written narrative describing the progress, and skills in their classrooms?
problems, and outcomes of the initiative If the professional development
(p. 8). 3 planning team has not already addressed
these issues, the evaluation planners
2. What are the key elements of should encourage the group to do
the professional development so. One good way to address these
that will be evaluated, and what questions and to gain consensus about
assumptions hold these elements the answers is to develop a logic model
together? to help identify key components of the
Quality evaluations are tailored professional development, the underlying
to the professional development being assumptions, the timeline, and the
evaluated. An important step in the expected outcomes. Creating a logic
planning process is to understand model is especially helpful in planning
the overall design of the professional long-term professional development that
includes several kinds of professional
learning. Creating a logic model is
Each professional development program or equally helpful in planning an evaluation.
The logic model becomes the road map
initiative requires a unique logic model. for the evaluation.
The figure on page 15 illustrates
development. What are its goals and what a professional development logic
objectives? Who are the expected model might look like. Each professional
participants? What kind of professional development program or initiative
learning will take place, and on what requires a unique logic model.
timeline? What resources are necessary to The boxes on the left side of the
ensure that the professional development logic model list the inputs necessary
takes place as planned and yields the for teacher professional development.
intended outcomes? What contextual Evaluation planners should also recognize
3 See: Making a Path, Leaving a Trail, remarks of Hayes Mizell on December 5, 2009, at the
annual meeting of 12 Under 12, a national network of seven public schools. Available at www.
thescsdc.org/120509.pdf.
Teacher Professional Development Logic Model
Timeline in months (or some other metric)
Inputs
Interim Interim
Effective needs
Professional Professional Outcomes/ Professional
Outcomes/
assessment and targeting Learning Learning Indicators/ Learning Outcomes/
Indicators/ Indicators
of participants Activities I Activities II Activities III
Benchmarks I Benchmarks II
National Staff Development Council 800-727-7288

Teacher
Adequate materials, New
perceptions Changes in
equipment, facilities to Presentations Teacher Presentations knowledge
Presentations student
ensure full participation perceptions and skills learning
New knowledge
Workshops Workshops and skills Workshops
New knowledge Change in Change in
Demonstrations and skills Demonstrations practice student
Adequate staff to ensure Demonstrations Change in
practice behavior and
full participation Study groups Change in Study groups Change in
Study groups engagement
school Change in school school
School-based organization School-based organization and School-based organization
coaching and and culture coaching and culture coaching and and culture
follow-up follow-up
Adequate funds to ensure follow-up
full participation Changes in
student learning
and other
student
outcomes
Adequate time in school
and district schedules to
ensure full participation
Formative Evaluation Summative Evaluation
Contextual factors that may affect participation and outcomes (e.g. other professional development initiatives, competing improvement priorities, change in leadership)
Teacher Professional Development Evaluation Guide 15

that identifying teacher learning and II and Outcomes/Indicators can

needs and who will participate in the include the various outcomes, indicators,
professional development will go a long and benchmarks that planners expect
way toward determining baselines against to observe and/or measure at different
which to gauge improvements in teacher times as the professional development
knowledge, skills, and practice and continues. For example, the first set
changed student outcomes. Identifying of interim outcomes could include
all of the inputs in the logic model is participants’ perceptions of the usefulness
important, but carefully identifying of the professional development, initial
teacher learning needs is especially mastery of new knowledge and skills,
important. Too often, professional and changes in school organization
development planners explicitly or to accommodate later classroom
implicitly assume that all teachers have applications of new knowledge and skills.
essentially the same learning needs. As the professional learning progresses,
Professional development that rests on the second set of interim outcomes could
this assumption tends to reflect a one- include more extensive mastery and
size-fits-all model, which often means application of new knowledge and skills
that the professional development doesn’t and an early look at whether changes
actually fit anyone’s needs very well. in student learning are occurring as
The three boxes labeled Professional expected. The last set of boxes on the
Learning Activities I, II, and III right side of the model depicts what
indicate that professional learning may changes are expected in professional
be ongoing and extend over a number practice as a result of the activity and
of months or even several years. The what improvements in student learning
activities included in each of these are expected to result from the changes in
boxes may differ as the professional teacher practice.
development unfolds, or some may As a general rule, begin by thinking
be repeated several times. Note that about the final outcomes and related
the logic models for some professional indicators depicted by the boxes on
development programs or initiatives may the right side of the logic model,
require only one of the learning activity especially when planning longer-term
boxes. Other initiatives may require professional development. Put simply,
several boxes for an adequate description. ask two questions: What are we trying
The boxes labeled Interim to accomplish? How will we know
Outcomes/Indicators/Benchmarks I when we have accomplished it? After
answering these questions, figure out evaluations will not be able to control for
what interim outcomes and indicators these factors.
will provide evidence that the activity is For these reasons, professional
on track to accomplish the longer-term development planners and evaluators
outcomes. Note that only after specifying should concentrate on outcomes for
the outcomes can planners reasonably teachers and proximal learning outcomes
determine the kinds of professional for students, with the latter reflected
learning necessary to achieve the in indicators such as student work
intended outcomes. samples and results on locally developed
Several criteria should guide the benchmark assessments and/or end-of-
selection of both the interim and final course assessments. When the prompts
outcomes and indicators. First, criteria
should be reasonable and not overly
Although the ultimate goal of teacher
ambitious. In addition, although the
ultimate goal of teacher professional
professional development is to improve student
development is to improve student
learning, the more immediate goal (as
learning, the more immediate goal (as reflected
reflected in the outcomes and indicators)
is improved teacher knowledge, skills,
in the outcomes and indicators) is improved
and practice.
Several fundamental reasons make
teacher knowledge, skills, and practice.
it difficult to empirically establish
clear causal relationships between
teacher participation in professional that generate work samples and local
development and changes in student assessments are tightly aligned with
learning, especially as these changes are state assessments or other standardized
measured by standardized assessments. assessments, student learning outcomes
First, only the most ambitious, long- observed on these indicators can also
term professional development aims to be predictors of improved results on
improve teacher knowledge and skills standardized assessments, or at least
across the entire scope of content covered portions of them.
by these assessments. Second, many The timeline across the top of the
factors affect student learning in all of the logic model not only describes the
areas covered by the assessments besides professional development schedule,
their performance as test takers. Most it also helps determine the evaluation
schedule. The broken lines across the development explicitly reflects these
bottom of the logic model suggest that assumptions. And, as already noted,
results from early or formative stages of developing a logic model creates a road
the evaluation can be used to modify the map for an evaluation.
professional development design. The
results also can be used to report early 3. Who is likely to be interested
outcomes to interested stakeholders. in the evaluation, and what do
The final or summative phase of the they want to know about the
evaluation is completed after the learning professional development?
has taken place at a point when expected Consider potential audiences and
outcomes and related indicators might be their interests.
observed. • Teachers will want to see their
The box at the bottom of the logic reflections and feedback on the
model hypothesizes that there are professional development and its
many contextual factors that influence benefits, as well as those of their
professional development and its colleagues, captured in the evaluation
outcomes, including outcomes related results.
to the application of new professional • District professional development staff,
knowledge and skills and to changes especially professional development
in student learning. Planners may not coordinators, school-based professional
be able to identify all of the relevant development staff, supervisors, and
contextual factors or predict their curriculum coordinators, will want
influence. Nevertheless, trying to identify to know whether the professional
them may help avoid problems as the learning took place as planned,
professional development unfolds. including whether they attracted
The importance of developing and the intended participants. They will
coming to consensus on a logic model also want to know what participants
to guide professional development and thought about the professional
the evaluation is difficult to overstate. development as a possible indicator
Developing a logic model compels of whether the participants will
planners to critically examine their apply the new knowledge and skills
assumptions about how various kinds and what midcourse changes may
of professional learning will contribute be required. Later, they will want
to changes in practice and student to know whether the professional
learning and ensure that the professional development achieved the intended
outcomes and perhaps whether in wanting to know whether

the evaluation suggests payoffs in the professional development
continuing the activity or taking it achieved the intended changes in
to scale. They will also want to know teachers’ knowledge and skills and,
what lessons can inform plans for consequently, led to positive changes
future learning. in student learning. They might also
• Principals and other school leaders want to know how these results stack
will want to know whether the up against the results from other
professional development produced professional development.
the intended changes in teachers’ District leaders and policy
knowledge and skills and whether makers are also likely to want
and when these changes are likely to to know about cost, especially if
result in positive changes in student the evaluation findings suggest
outcomes. Like district professional continuing or expanding the activity.
development staff, principals will Examining professional development
also want to know about the lessons spending is outside the scope of the
they can learn for planning and evaluations described in this guide.
facilitating future professional Evaluators can, however, examine
development in their schools. the budget prepared as part of the
• Providers, including college and planning process. The evaluation
university faculty, consultants, might also ask whether funds were
and vendors, will want to know spent according to the plan and, if
what participants thought of not, how the spending varied from
the professional development the plan and why.
and whether the professional • Parent and community groups will
development achieved the intended want to know whether and how
results. the professional development
• Funders and program managers will contributed to changes in instruction
want to know the evaluation results, and student learning outcomes,
either to satisfy their own reporting especially when the professional
requirements or to inform decisions development required a substantial
about additional or follow-up investment of district resources and/
funding. or required teachers to be out of their
• District leaders, including school board classrooms and away from students
members, will join other stakeholders for extended periods.
Learn what key stakeholders would stakeholders pose. Pointing out to

like to know or need to know about the stakeholders that resource constraints will
professional development and when they make it difficult to answer at least some
would like to have or need to have the of their questions could be a strategic
information; however, the evaluation lever to garner additional resources. A
may not address all stakeholder commitment to address key stakeholder
questions. Planners may need to reach a questions about the professional
compromise about the evaluation’s focus development’s effectiveness and impact
may be a way to ensure that necessary
Recognize and acknowledge that evaluating resources for the evaluation are available.
Ideally, the professional development
professional development is an unfamiliar task plan will include a description of the
evaluation and the estimated budget,
for many educators. Some capacity building including in-kind costs such as staff time.
and training will almost certainly be necessary. 5. Who will work on the evaluation?
Knowing who will work on the
Identify staff willing and able to be involved in evaluation is a critical part of the resource
issue. Decide who will be responsible for
this part of the work. overseeing or managing the evaluation
and who else will work on the evaluation
(e.g. develop data collection instruments,
and scope if information needs exceed collect and analyze data, prepare
what is possible given the resources and reports). Also think about whether
staff available for the evaluation. and how participants (teachers, in this
case) and other school and district
4. What resources are available to staff might be involved. (Options for
support the evaluation? teacher involvement in evaluations of
Because resources to support professional development are discussed in
evaluations are often limited, know in more detail later in the guide.)
advance what support is available and Evaluation work should begin soon
plan accordingly. Recognize that resource after the professional development gets
constraints may make it difficult, if under way, so have staff ready to work
not impossible, to address all of the on the evaluation as early as possible.
information needs and questions various Additional staff may be needed as the
process proceeds. If additional staff are these costs, keep in mind that working
not available, adjust the evaluation plan with an external evaluator can have
accordingly. several advantages. An external evaluator
Recognize and acknowledge that can add objectivity and credibility.
evaluating professional development is An external evaluator also can make
an unfamiliar task for many educators. the evaluation process more efficient,
Some capacity building and training will reducing overall cost while significantly
almost certainly be necessary. Identify improving technical quality.
staff willing and able to be involved in Working with an evaluator means
this part of the work. that evaluation planners and professional
When determining staff
requirements for the evaluation, also
decide whether to seek help from an An external evaluator can add objectivity and
evaluator or evaluation consultant. These
individuals can contribute to discussions credibility as well as make the evaluation process
about professional development
outcomes and indicators and can suggest more efficient.
approaches for collecting and analyzing
data. Skilled evaluators and data analysts
can help with more complicated design development planners become more like
and analytic tasks, especially choosing users or consumers of the evaluation
samples of participants to include in rather than those actually doing the
data collection, linking participation in evaluation. They are the evaluator’s
professional development to changes in clients. Being a good client means
student learning outcomes over time, working hard to be sure that evaluators
analyzing large amounts of quantitative have the information that they need
data, and experimental and quasi- and being as clear as possible about the
experimental evaluation designs. evaluation’s purpose. Conversely, being a
Good sources of evaluation expertise good evaluator means listening carefully
are district research and accountability to clients and asking questions.
office staff and college and university
faculty, as well as evaluation firms and Summary
contractors. Ideally, planning an evaluation of
Cost can be a concern when hiring a professional development program
an external evaluator. When considering or initiative will get under way as the
activity itself is being planned. Evaluation the assumptions that hold these elements
planning, like professional development together conceptually. A logic model or
planning, is an iterative process, with some other schematic representation can
the evaluation plan evolving as the help describe the elements of the activity
professional development plan evolves. and provide the evaluation framework.
The five questions posed in this section Anticipating possible audiences,
address practical considerations for including their needs and interests, helps
planning a successful evaluation. sharpen the evaluation’s scope and focus.
Answering these questions helps to define Knowing what evaluation resources are
what should be done in an evaluation necessary and available guides choices
and what’s possible. The first step is to about the evaluation’s scope and will
determine whether an activity should almost certainly set some constraints on
be evaluated. Does the basic plan for what can be accomplished.
the activity hold some promise for the Finally, a viable evaluation plan will
activity resulting in important outcomes? clearly identify staff who will work on
Will a solid evaluation yield results that the evaluation, their roles, and the kinds
will help inform subsequent phases of the of training and support they will need to
activity and plans for new professional complete their assigned tasks.
learning? The following checklist can serve as
The second step is to identify the a reminder about the key components of
activity’s key components (e.g. needed a good evaluation plan and help planners
inputs, professional learning, and keep track of which components are in
anticipated outcomes and indicators) and place and which are not.
Checklist for teacher professional development evaluation plans

The plan includes a logic model clearly indicating: Did the professional development achieve the intended
❑❑ Key inputs necessary for the professional outcomes specified in the plans for the activity and as
development to succeed. reflected by measurable and/or observable indicators?
❑❑ Professional learning necessary to achieve ❑❑ The plan explicitly describes appropriate
intended interim outcomes and intended final data sources (e.g. reports from systematic
outcomes for teachers. observations and/or similar activities, local
❑❑ Measurable and/or observable professional and state assessment results and student work
learning outcomes for teachers and, as samples for students of teachers participating
appropriate, learning outcomes for students. in the professional development) for each of
❑❑ A timeline for completing key professional the indicators, appropriate data collection
learning and attaining interim and final outcomes procedures, and a reasonable timeline for
for teachers and, as appropriate, students. addressing this question.
The evaluation plan clearly describes how the Organization and staffing
evaluation will address each of the following The plan clearly indicates who will be responsible for:
questions about the professional development ❑❑ Developing appropriate data collection
activity. instruments (if necessary).
Did the activity take place as planned? ❑❑ Collecting data to address each evaluation
❑❑ The activity included the intended participants. question.
❑❑ All of the participants engaged in all of the ❑❑ Data analysis.
professional learning. ❑❑ Reporting.
❑❑ All of the professional learning took place as
planned. The plan includes an overall time for evaluation and
❑❑ All of the necessary materials, personnel, and specifies reasonable completion dates for:
equipment were available. ❑❑ Selecting and/or developing appropriate data
❑❑ The plan explicitly describes data collection collection instruments.
activities, including specifying appropriate ❑❑ Each of the data collection activities included in
instruments and a reasonable timeline for data the plan.
collection, for addressing each of the parts of this ❑❑ Data analysis.
question. ❑❑ Reporting.
What were the participants’ perceptions of the relevance The plan includes budget estimates, specifying direct
and usefulness of the professional development? costs and in-kind contributions for:
❑❑ The plan explicitly describes data collection ❑❑ Salaries/stipends for the evaluators.
activities, including appropriate instruments and a ❑❑ Supplies and materials.
reasonable timeline, for addressing this question. ❑❑ Communications.
E va luat i o n d e s i g n a n d d ata co l l e c t i o n s t r at e g i e s
Evaluation design and data

collection strategies
E
ffective evaluations are explicitly take place as planned?
tailored to the professional • What were teachers’ perceptions of
development being evaluated. the professional development?
As professional development planning • Did the professional development
progresses, planners identify questions achieve the intended outcomes?
that the evaluation will answer and In deciding whether and how to
how best to answer them. This section address each of these questions, carefully
of the guide begins with suggestions review the professional development
for how evaluations can address three plan and the logic model to determine
basic questions about professional what the evaluation should focus on
development programs and initiatives. and what kinds of data to collect. If the
Next, the guide discusses two design professional development plan and the
options that can add rigor to an logic model are unclear or inconsistent
evaluation: focusing on a sample of about the inputs and resource needs,
participants versus all of the participants professional learning, and/or expected
and adding a comparative dimension. outcomes and indicators, clear up
This section also explains how to prepare any ambiguities with the professional
for data collection, and it ends with a development planning team. A
discussion of options and opportunities conversation with the professional
for involving teachers in collecting data development planning team may, in turn,
on professional development outcomes lead to modifications in the evaluation
and indicators. plan, including a clearer definition of the
expected outcomes.
Evaluation questions
Comprehensive evaluations of Looking at professional
teacher professional development should development implementation4
focus on three basic questions: Examining implementation of the
• Did the professional development planned professional development,
including identifying any problems of frequency and duration and

and impediments to implementing included the content specified in the
the professional development as professional development plan and/
planned, can inform decisions about or logic model;
future professional development as • The extent to which all of the key
well as decisions about necessary players (e.g. presenters, facilitators,
modifications or midcourse corrections school-based professional
to the professional development being development staff, principals) carried
evaluated. Specifically, in looking at out their responsibilities as specified
implementation of the professional in the professional development plan
development, evaluators should collect and/or logic model;
data on: • The extent to which contextual
• Individual teacher participation factors (e.g. changes in school
to determine whether teachers or district leadership, changes
who were targeted to participate in school or district priorities,
actually did participate and whether changes in resources, changes in
they participated in all of the key teacher assignments, changes in
professional learning specified in the student characteristics) influenced
professional development plan and/ implementation, including teacher
or logic model; participation.
• The availability of the supplies, Collecting data on the
materials, and equipment necessary to implementation of professional
improve or implement new professional development is relatively straightforward
practices, as specified in the but does require careful record keeping
professional development plan and/ that, in turn, requires reliable procedures
or logic model; to collect and store the data necessary to
• Whether the professional learning track implementation. These data might
occurred at the intended levels include sign-in sheets to track teacher
4 Readers should note that this guide uses the term “implementation” in two ways. First,
implementation, as in implementation of the professional development, refers to the extent to
which the professional development took place, or was implemented, as planned. Second, the
term also refers to the application of new knowledge, skills, and practices that may be the result
or outcome of participation in the professional development. Examining implementation of the
professional development and the early outcomes is sometimes referred to as formative evaluation
or process evaluation. Gauging participants’ reactions to and perspectives on the evaluation, which
is discussed later in the guide, is often included in formative evaluations.
participation in centralized activities and in funds also could result in shortages of

activity logs maintained by school-based instructional materials and equipment
professional development staff to track needed to implement new practices
follow-up support for individual teachers and programs. If such changes occur,
and groups of teachers. Using electronic document the changes and learn as much
files and online systems to maintain as possible about how they affect the
activity logs and similar records is very implementation and outcomes of the
efficient and inexpensive, especially with professional development.
well-designed reporting forms. Once What are the consequences of not
these record-keeping systems are in place, carefully examining implementation of
it is easy to update the information in the professional development? If there
them. are no data — or are very limited data
Although almost certainly not — on implementation, evaluators will
a central focus of data collection on be unable to determine whether the
implementation, changes in school professional development took place as
and district contexts are likely to affect planned and what factors contributed to
whether the professional development or impeded implementation. Evaluators
took place as planned. For example, then will be at a loss to pinpoint
a change in a district’s priorities for problems that might be addressed
instructional improvement could through midcourse corrections in
significantly affect teacher participation, the professional development design
especially if the change is somehow and plans. Lack of examination can
incongruent with the focus and goals be especially serious in evaluations of
of the professional development long-term, multiphase professional
being evaluated. For example, a development. A second serious concern
new reading program is introduced is that without data on implementation,
that calls for different instructional evaluators will be hard-pressed to
strategies than those being emphasized attribute any observed changes in teacher
in the professional development or knowledge, skills, and professional
a new mathematics curriculum is practice to professional development.
not compatible with the professional This problem is particularly severe when
development content. Similarly, there are gaps in the data on teacher
unanticipated reductions in funding participation in comprehensive, long-
for substitute teachers could lower term professional development.
teacher participation rates. Reductions
Examining participant perceptions current assignments/responsibilities,

of the professional development with special attention to teachers’
Examining what participants thought perceptions of its usefulness in
of the professional development can working with their own students;
yield an early indicator of whether and • Perceptions of the extent to which
how participants are likely to apply new professional development met
knowledge and skills in their professional individual professional learning
practice. Participant perceptions can needs;
also yield useful information on whether • Ratings of how well the content
the professional development was or focus of the professional
implemented as planned and can help development aligned with district or
pinpoint components of the professional school improvement priorities, plans,
development that went well and those and goals;
that may require modification. • Perceptions of the kinds of support
The easiest and most efficient way to and encouragement teachers received
collect data on participants’ perceptions to actively engage in the professional
of professional development is through development;
one or more surveys. (Appendix B • Perceptions of the kinds of support
includes sample items that evaluators can and encouragement they received to
tailor for surveys of teacher perceptions apply new knowledge and skills in
of their professional development their classrooms;
experiences.) Depending on the nature • Ratings of the likelihood of applying
of the activity and the reasons for new knowledge and skills in the
examining participant perceptions, classroom; and
surveys can address some or all of these • Overall ratings of the usefulness
topics: of the professional development
• Participants’ understanding of compared with that of other
the purpose of the professional professional development.
development5; If the survey is about a session
• Ratings of the usefulness or or series of sessions with multiple
relevance of key components of components and/or that extend over a
the professional development to relatively long period of time, the survey
5 Surveys often ask participants to rate the clarity of the purpose of the activity. Although
these ratings are important elements in overall ratings of quality, they do not address the issue of
whether participants actually understand what the purposes are.
should include items about participation paper-and-pencil instruments. These

in each component of the activity to help surveys can be administered quickly
determine overall participation patterns and efficiently through district e-mail
and whether they appear to be related systems or district web sites.6 District
to attaining the intended outcomes. information technology staff can provide
Alternatively, an evaluation could include the best advice on how to administer
several surveys, with each one focusing online surveys and how to meet local
on a key component of the professional requirements related to online security
development (e.g. initial workshops and and personal privacy.
presentations, school-based follow-up, Despite the advantages of surveying
opportunities for reflection and practice). participants at a reasonable interval
Whenever feasible, surveys should be after participating in professional
administered three to six weeks after the development, it may be difficult to get
professional development is completed, teachers to respond to surveys after they
instead of the standard end-of-the- return to their classrooms. One way
session-on-the-way-to-the-parking-lot to increase the number of responses
approach to survey administration used is to provide a small reward, such as a
in many local evaluations. Administering gift card, for completing the survey. A
the survey some weeks after the event second option is to make receiving credit
gives participants time to reflect on for participating in the professional
the professional development and how development contingent on completing
they benefited from it and makes it the survey. Over the longer term, if
possible to ask questions about follow-up teachers see their perceptions reflected
professional development and support, in reports or other communications
which are key elements in high-quality or if they see the survey results leading
professional development. to positive changes in professional
Also consider administering surveys development, they may be more willing
online. Online surveys are inexpensive, to complete these surveys.
and available software packages make As an alternative to or to
developing, administering, analyzing, complement a survey, an evaluation
and reporting survey information can also include focus group interviews
much easier than using traditional to collect data on teacher perceptions
6 Survey Monkey is an example of an easy-to-use survey software package. The basic survey
package is free, and more comprehensive and versatile versions are available. For more information
on this software and how to use it, visit www.surveymonkey.com.
of their professional development Evaluations that combine

experiences. As with the more traditional examination of teacher perceptions
surveys, the focus groups should be of all components of the professional
conducted at a reasonable interval after development (especially follow-
the professional development has ended. up and school-based activities), a
Note that these focus groups can also comprehensive look at implementation
be included as part of ongoing data of the professional development, and
collection that follows major components professional development outcomes are
of the professional development. much more useful in understanding what
The primary advantage of focus happened and how professional learning
groups is that they can yield rich paid off for teachers and their students.
data about teachers’ perceptions of
professional development and can Looking at outcomes and indicators
complement the survey data. The Evaluations of teacher professional
primary disadvantages of focus groups development collect data on outcomes
are that they are more time-consuming for teachers, including their mastery of
than surveys, both in terms of data new knowledge and skills, changes in
collection and data analysis, and they professional practice, and changes in
typically include relatively small samples student learning outcomes, behavior, and
of participants. engagement in school that are associated
A final caution about focusing on with the teacher outcomes. In addition,
participant perceptions: Evaluations that evaluations may examine changes in
rely solely on teacher perceptions of some
components of professional development,
As a general rule, plan to include strategies
typically the training components that
take place at the beginning of a longer
for collecting data on all of the outcomes and
series of professional learning sessions,
are of limited use because they provide
related indicators included in the plan for the
incomplete information. Many of these
surveys are superficial and often trivialize
professional development being evaluated.
teacher perspectives by limiting the focus
of questions to unimportant or irrelevant
topics. Put somewhat differently, smiley school organization and culture that
faces are not good metrics for examining may be linked to the professional
professional perspectives. development.
As a general rule, plan to include whether the first outcome (participants’

strategies for collecting data on all of mastery of curriculum content) was
the outcomes and related indicators achieved could include participant scores
included in the plan for the professional on a written test. The data collected
development being evaluated. Consider to determine whether teachers can
the examples in the box below. apply appropriate pedagogical practices
These examples suggest progressively could come from direct observations
more extensive professional development. of instruction, from written responses
They also suggest increasingly complex to prompts calling for descriptions of
evaluation efforts and clearly illustrate instructional activities that would help
the need for multiple data collection students master various parts of the
strategies. More importantly, they relevant content area, or from teacher
illustrate the relationship between logs that include self-reports on their
choices about indicators included in use of various instructional strategies.
the plan for the activity and decisions Data on whether teachers are applying
about the kinds of data that will need appropriate assessment strategies could
to be collected to determine whether come from reviewing teacher-developed
the outcomes were achieved. Using the classroom assessments and assignments.
example, the data collected to determine Data on changes in student learning
Choosing outcomes and indicators

If one of the expected outcomes of the professional development to be evaluated is having
participants master the content of one of the subject areas included in a local or state curriculum
framework, an indicator might be scoring at a specified level of proficiency on a written test on the content
area after completing the professional development.
In a more ambitious professional development activity, one outcome could be mastering the content
of one of the subject areas in the curriculum framework, with a second outcome being understanding of
and ability to apply appropriate pedagogical strategies necessary to help students master the content. The
indicator for the first outcome could be passing the written test on the content. An indicator for the second
outcome could be a demonstration of the appropriate instructional practices in a simulated or actual
classroom setting.
A third set of outcomes and indicators could focus on understanding and applying appropriate
classroom assessments to gauge student mastery of content in one of the content areas of the curriculum,
and a fourth set could focus on changes in student learning associated with changes in instruction.
could come from a review of samples of of having resource teacher support in

student work prepared in response to other areas. This new perspective could,
teacher-developed prompts related to the in turn, lead to increased demand for
relevant curriculum content, a review school-based professional development
of student scores on relevant sections provided and facilitated by skilled
of local benchmark assessments, or, less resource teachers in other content
frequently, a review of student scores on areas. Alternatively, poorly planned and
state assessments. incompletely implemented school-based
Along with being sensitive to professional development in one subject
changes in local context that affect area could seriously undermine teachers’
implementation of the planned interest and willingness to participate in
professional development, keep in similar professional development in other
mind how changes in context affect the subject areas. Both of these outcomes are
outcomes. Changes in school or district important, and evaluators should make
priorities could greatly influence teacher every effort to learn how and why they
application and use of new knowledge occurred.
and skills. For example, decisions to
introduce a new reading program or a Focusing on a sample of
new math program, mentioned earlier as participants versus all participants
factors influencing the implementation For professional development that
of the professional development, also includes large numbers of participants,
could mean that teachers are no longer participants in a large number of schools,
expected to use instructional strategies and/or multiple instances of professional
and curricular materials that were the learning that extend over time, collect
focus of the professional development data from a sample or samples of
being evaluated. participants for at least some parts of the
Also be sensitive to unanticipated evaluation. The advantage of looking at a
outcomes that may be attributed to the sample of participants over time is that it
professional development. For example, becomes more feasible to gather detailed
professional development that relies on or in-depth information, especially about
math resource teachers to help math things like changes in classroom practice.
teachers implement new content and Focusing on a sample or samples
instructional practices could result in of participants has several challenges.
other teachers and school administrators The first is the sampling task itself. One
understanding the potential benefits option is to select a random sample. A
second option is to draw a sample that to looking at professional development

is representative of all the participants. outcomes is to compare changes in
Drawing a representative sample requires participants’ (the treatment group)
having information about individual knowledge and skills with changes in
characteristics that might somehow affect the knowledge and skills of teachers who
participation patterns, application of did not participate in the professional
new knowledge and skills, or both. For development or who participated in
example, in a professional development other professional development (the
activity focused on understanding and comparison group). Alternatively, the
using instructional practices to foster evaluation could compare the learning
reading comprehension, participants outcomes for students of teachers in the
might be divided between teachers treatment group with those of students of
who are reading specialists and those teachers in the comparison group. When
who are not, or teachers with extensive done well, both kinds of comparisons
experience and those who are new to help gauge professional development’s
the profession. In this example, the impact on teacher knowledge and skills
participant sample for the evaluation or classroom practice, and on student
should reflect the overall distribution of learning.
teacher characteristics. Adding a comparative dimension
A second challenge is to an evaluation also poses substantial
communicating the sampling procedure. challenges and may add significantly
How and why were participants selected to the cost. In most local evaluations,
for the evaluation? Despite evaluators’ adding a comparative dimension to the
intentions to the contrary, identifying evaluation will require the assistance of
samples might suggest that some a skilled evaluator who can help with
participants are being singled out for sampling issues, as well as complex data
special attention or scrutiny. Address analysis. At a minimum, this design
these concerns from the outset, and make option requires identifying comparison
every effort to allay any fears to ensure groups of teachers who share many,
that teachers will be willing participants if not all, of the characteristics of the
in the evaluation. participating teachers. For example,
the comparison group should include
Adding a comparative dimension to teachers who teach at the same grade
the evaluation level and in the same content areas as the
A potentially powerful approach participating teachers. In addition, the
comparison group should teach students random assignment of teachers and/or

with similar characteristics and work in students to treatment and comparison
schools similar to participants’ schools. groups to teachers, parents, and others in
Identifying comparison groups the local education community.
matched to treatment groups on certain
variables helps to focus on the treatment Preparing for data collection
— professional development — as the In every evaluation of teacher
cause of the observed outcomes and to professional development, preparation is
eliminate other explanations of causality.7 essential for the effort’s success. Preparing
An even more powerful evaluation for data collection involves:
design is randomly assigning teachers • Selecting the appropriate
to treatment and comparison groups instrument(s);
and comparing the two groups on the • Preparing staff who will collect the
outcome variables at the end of the data; and
professional development or at the point • Gaining access to people and data.
at which planners expect to be able to In some evaluations, completing
observe the intended outcomes. Random these preparations will be fairly simple.
assignment designs are the gold standard In others, preparations will be more
in studies of program effectiveness and complicated. Some preparations are
impact. inexpensive, while others are expensive
For most districts and many and time-consuming. In some
evaluations, both design options have evaluations, costs trump other factors
practical disadvantages. They can be in deciding how to complete these
costly and time-consuming. Second, preparations. Recognize and be prepared
they can require large samples of teachers to explain the trade-offs being made.
and students. In some cases, not enough Selecting appropriate instruments.
teachers may be available to form either (Resources included in Appendix A
a treatment or comparison.8 A third provide extensive information about
challenge may be having to justify the instruments that can be used to evaluate
7 Carefully identifying professional learning needs and strategically targeting participants as the
professional development is being planned help define which variables can inform identification of
appropriate comparison groups.
8 One solution to this problem is to identify comparison groups from other districts. While this
strategy can add rigor to the evaluation design, it also adds cost and may not be feasible for other
reasons.
teacher professional development.) In perspectives on their professional

addition to participant surveys discussed development experiences. These
earlier, evaluations of teacher professional instruments may require only minor
development can employ: modifications to be appropriate for
• Written tests to assess participants’ the evaluation being planned. Some
mastery of professional knowledge curriculum materials include assessment
(including knowledge of curriculum instruments to measure content mastery,
content) and skills; and locally developed observation
• Protocols for observations and protocols and guides may be useful to
interviews to collect data about gauge how well new practices are being
participants’ perspectives on implemented. Finally, a number of
the professional learning; their instruments measure teacher attitudes.9
implementation of professional Adopting and/or tailoring existing
development; and their application instruments can be relatively inexpensive
of new knowledge and skills in their and require very little time. In some
classrooms and other settings; cases, the developer may provide
• Personal logs and journals to record guidance for administering and using
accounts of the application of new these instruments. In addition, results
knowledge and skills and reflections from other evaluations using these
on these experiences. instruments may be available to use for a
Some evaluations collect data on comparative analysis.
students. Most evaluations that focus on Several criteria can help guide
student outcomes associated with teacher selection and/or development of
participation in professional development instruments.
rely on student work samples, results • Instruments should be designed to
on state and local assessments, and minimize the burden on respondents
administrative records of student and users. Observation protocols
attendance, behavior, and discipline. should not require more than 30
A practical first step in selecting minutes per observation, and less
instruments is to see if appropriate if possible. Surveys should require
instruments already exist. For example, 15 to 20 minutes at most and less
some districts have developed generic if possible. Overly long surveys and
survey instruments to gauge teacher interviews tend to yield bad data and
9 Visit http://people.ehe.ohio-state.edu/ahoy/research/instruments/ for links to a number of
survey instruments and detailed information about their use and psychometric properties.
may result in respondents’ resistance instructional materials and

or unwillingness to participate in equipment;
other data collection. ◦◦ The particulars of how a teacher
• The content or substantive focus of uses prompts to assign student
the instruments should be aligned work (e.g. time spent discussing
with the content of the professional the assignment, opportunities
development. Observation protocol for students to collaborate and
prompts should call for observations collect information).
and data that explicitly reflect In contrast, protocols that
key elements of the content of do not rely on well-defined
professional development (e.g. rubrics or that call for anecdotal
frequency of teacher use of specific reports or subjective judgments
questioning strategies, teacher use
of appropriate curriculum materials,
teacher use of appropriate strategies As a general rule, observation protocols should
to gauge student learning).
• As a general rule, observation call on observers to describe classroom activities.
protocols should call on observers to
describe classroom activities. These These protocols should not call on observers
protocols should not call on observers
to judge or rate the activities (e.g. rate to judge or rate the activities (e.g. rate teacher
teacher proficiency on a scale of 1 to
5). Protocols that collect descriptive proficiency on a scale of 1 to 5).
data could, for example, call on
observers to count or report:
◦◦ The number of times a teacher yield data that are difficult
uses various questioning to analyze or not very useful.
strategies; Protocols that call for observers’
◦◦ The number of students to judgments or ratings require that
whom the teacher directs the observers receive extensive
questions and feedback; training to ensure consistency
◦◦ Teacher use of academic within and across the judgments
language versus less formal made during the observations.
language; • Survey instruments and other
◦◦ The presence and use of certain self-report forms should, with few
exceptions, include close-ended items.10 sure they are suitable for the evaluation
Responses to these items are easier being planned. For example, protocols,
and less expensive to analyze, and guides, and checklists developed for
they ensure a degree of consistency classroom observations, walk-throughs,
among respondents. Responses and teacher performance appraisals
to open-ended questions require may be useful to gauge the impact of
careful coding and are therefore professional development on teacher
more difficult and more expensive to performance. However, be sure these
analyze. instruments are designed to collect
A note of caution about adopting data on the specific indicators included
locally developed instruments, especially in the evaluation. In other words, the
instruments designed for a purpose other instrument should meet reasonable
than evaluating teacher professional standards for validity.
development: Instruments that are For evaluations that focus on
generic or that focus on indicators other student outcomes to gauge the long-term
than those included in the evaluation impact of professional development,
will not yield data that will help gauge be sure that assessment tools, including
the effectiveness of the professional locally developed tests and assignments
development. Selecting or developing or prompts used to generate student
appropriate instruments depends largely work samples, explicitly align with
on clearly specifying what the instrument the knowledge and skills teachers
is intended to help measure. What are are expected to demonstrate as a
the key indicators that new practices are result of participating in professional
being implemented? How often should development.
the indicators be present and under When using student work samples,
what circumstances? Ask these and other evaluation planners and professional
questions. Those planning the professional development planners should think
development should provide the answers. carefully about how the assignments
Carefully review instruments to be and prompts will be administered and
10 Close-ended items call for respondents to select from among two or more response options.
These items may ask respondents to pick from several statements one that best describes their
perspectives on a particular issue (e.g. the usefulness of a particular professional learning activity,
the extent to which they have had opportunities to apply new knowledge and skills). Close-ended
items also may ask respondents to indicate the extent to which they agree or disagree with certain
statements or to respond yes or no. Open-ended questions ask respondents to write a few words or
sentences.
Using student work samples in evaluating teacher professional

development: A cautionary example
T
he professional development of the two teachers that took place
planners and evaluation after the professional development
planners agree that student indicated that both had mastered the
essays on the causes of the Korean new content and related instructional
War will be one source of data about strategies, the evaluation team initially
student outcomes that result from concluded that the students of Teacher
teacher participation in a long- B were more able than the students
term professional development of Teacher A. However, there was no
designed to help improve instruction other obvious evidence of differences
in 20th-century American history. in students’ ability; the evaluation
Teacher A and Teacher B have team was puzzled by the result and
recently completed three months decided to look a second time at the
of professional development, and observation data. They discovered that
the evaluation team will review their although instruction was quite similar
students’ essays to gauge whether the in the two classrooms, Teacher A had
teachers successfully implemented the written the prompt on the board at
instructional strategies and curriculum the beginning of a class period and,
content that were the focus of the after responding to a few questions,
professional development. had given the students two days to
In reviewing the completed complete their essays. Teacher B had
essays, the evaluation team observed spent most of a class period discussing
that the essays of the students of the assignment and responding to a
Teacher A were much shorter and variety of student questions. Next, she
less well-developed than those of organized students into small groups
the students in Teacher B’s class. The to brainstorm possible explanations
team also observed that the essays of for the causes of the Korean War. After
Teacher A’s students reflected limited that, she encouraged them to go to
understanding of the economic and the school library to collect additional
political factors that preceded the information before completing their
conflict. Because the two teachers gave essays. The result of these different
their students the same assignment, approaches to the same assignment
and because classroom observations was essays of very different quality.
whether and how the evaluation will in the evaluation, and with whom
monitor the process. Differences in and how they are expected to share
how assignments are designed and/ information.
or how prompts are used may result • A detailed review of what expected
in considerable differences in student changes in practice look like. This
work. These differences could, in turn, review should familiarize data
lead to differences in judgments about collectors with the details of new
instructional practices. Consider the instructional strategies and other
example in the box on p. 37. outcomes that are the focus of
Preparing the evaluation team. No the professional development.
matter what evaluation design or data For example, if the professional
collection instruments are used, a critical development is designed to help
step in conducting good evaluations is to teachers develop new questioning
thoroughly prepare those individuals who strategies and apply the strategies
will be responsible for collecting data. in their classrooms, preparation
Ideally, this preparation should include: for data collection ideally should
• A thorough orientation to the include opportunities to see concrete
evaluation plan, with special attention examples of these strategies in
to data collection tasks, responsibilities, practice.
and the amount of time required. • A detailed review of the data collection
This is particularly important when instruments and how to use them.
data collection includes interviews, Explain to data collectors the
observations, and other direct purpose of using the instrument or
contact with participants. These instruments and expectations for
forms of data collection can be how data are to be recorded and/
time-consuming, so data collectors or reported in preparation for data
need to understand the amount of analysis. Provide clear guidelines for
time that they will need to devote the frequency and duration of use
to the task, especially if district staff of observation protocols, interview
are responsible for collecting data. guides, and similar instruments. For
Review confidentiality issues that example, if an observation protocol
may arise in the course of collecting will be used to measure instructional
data collection. Data collectors need changes or application of new
to understand that it is imperative to instructional strategies and content,
protect the identity of participants observers need to know how many
times they are expected to observe how evaluation results will be

each teacher, appropriate intervals communicated and to whom.
between observations, and how they Teachers may be concerned about
are to record and report data. the extent to which others will view
• Opportunities to practice using the instruments as an evaluation
data collection instruments, such as of the teachers and their work.
observation protocols and interview Reassure teachers that the evaluation
guides. Having data collectors is assessing the professional
practice using the instruments development and not teacher
ensures that they understand knowledge and skills and/or their
their responsibilities and that performance in the classroom. At
data collection is consistent the same time, make it clear that
and thorough. Practice in using teacher knowledge and skills and
observation protocols also may afford performance are included among
opportunities to test the instruments’ the expected outcomes and are,
reliability and to make refinements. therefore, the focus of the data
Practice might include, for example, collection activities.
role-playing in which members of • Confidentiality. Assure teachers
the evaluation team use the interview and others who participate in the
protocols with other team members evaluation that all data collected
or team members completing from and about them will be
observation protocols as they maintained in strict confidence
watch videos vignettes of classroom and that they will not be identified
instruction. by name in any reports or other
Securing participation in the communications about the
evaluation. Adhering to three simple evaluation. As appropriate, explain
principles can increase the likelihood the procedures for maintaining
that teachers and others will be willing to confidentiality. For example, the
participate. The principles are: evaluation team will probably store
• Transparency. Be clear about the data in secure files accessible only
the evaluation’s purposes, what to team members and may assign
participation will entail in teachers and others who participate
terms of time and other possible in the evaluation an identification
commitments, what data will number so that names and other
be collected and how, and identifying information can be
eliminated from data files. the kinds of data that are necessary,
• Voluntary participation in the procedures for protecting student privacy
evaluation. As a general rule, teachers and confidentiality, and how the data
and others who may be involved will be used are important. In addition to
in the professional development obtaining parental consent, work closely
as participants or in other ways with district staff who maintain district
should be invited but not required to data systems. Indeed, it may be necessary
participate in the evaluation. They to call on them to extract data from
also should be permitted to opt out student files.
of the evaluation at any point in the
process. To ensure that participants Special considerations in using
understand what is being asked classroom observations
of them, prepare a short, written Classroom observations, when done
description of the evaluation that, at well, are an excellent source of data on
a minimum, explains the evaluation’s professional development outcomes
overall purpose, the amount of time related to teachers’ understanding and
required to participate, and how application of new knowledge and skills.
the results will be reported and to The success of these data collection
whom. activities depends, in large part, on the
Gaining access to students availability of good instruments and the
and student data. If an evaluation work of observers who are well-prepared
focuses on changes in student learning for the task. Observers should:
or in other student outcomes (e.g. • Conduct multiple observations of each
attendance, truancy, disciplinary referrals, teacher over several weeks or perhaps
suspensions, expulsions) or on student even a few months;
perceptions of changes in teacher • Complete data collection reports as
performance, it may be necessary to soon as possible but no later than 48
seek parental consent to gain access to hours after the observations;
student records, to interview students, or • Avoid scheduling observations on the
to ask them to complete a survey. Many days before or after school holidays or
districts have well-defined procedures on “special days,” which may have
for gaining access to students and shortened or alternative schedules
student data. Pay close attention to these or otherwise represent atypical
procedures. Written communications instructional periods;
explaining the evaluation’s purposes, • Immediately report any problems,
including concerns from teachers role in an evaluation of professional

who are observed, to the person development from their performance
leading the evaluation and/or appraisal role. In addition, teachers
managing the observations. who are included in the evaluation may
Also think about following up not understand or appreciate the dual
classroom observations with short roles and may be uncomfortable during
interviews to collect additional data. observations related to the evaluation.
These interviews can elicit teachers’ (They may be uncomfortable being
views of what happened, how things observed as part of a formal performance
worked, and why. These interviews review, but that is another matter.)
can help observers examine teachers’ One option for involving principals
reasons for using various content and and assistant principals in collecting data
instructional strategies as a way of is to have them interview and observe
learning about the connections between teachers in other schools. Outside
teacher knowledge and the application of observations have several potential
specific instructional strategies. Schedule advantages. Using professionals from
interviews to avoid additional disruption outside the school lends an important
of classroom activities, but as soon as element of objectivity and credibility to
possible after the observations and no the evaluation and helps to distinguish
later than the end of the day on which it from performance reviews. Second,
the observations were conducted. this strategy affords the data collectors
An important issue in planning opportunities to learn what is going on
the observational components of an in schools other than those in which they
evaluation is to decide what role, if any, normally work. These data collection
district staff will play in data collection. activities, especially when they are
School-based professional accompanied by collaborative debriefings
development staff, supervisors, principals, with all of the observers, can provide
and assistant principals typically spend a valuable professional development for the
lot of time in both formal and informal data collectors.
classroom observations. Therefore, it may In addition to deciding whether
make sense to recruit them to collect data to recruit building administrators
for the evaluation. Recognize, however, and district staff as data collectors,
that it will be difficult for principals consider using information collected
and others responsible for teacher for other purposes (e.g. performance
performance appraisals to separate their reviews, monitoring implementation
of a new practice) in an evaluation collection process and the purpose of

of professional development. Taking the professional development means
advantage of information that has that the data collected for these other
already been collected can be tempting, purposes will be of limited use for the
and it is possible that the professional evaluation. Such misalignment could
development planning team used some of easily generate inappropriate results
these data to determine the need for the and lead to incorrect conclusions about
professional development and to identify the professional development. Third,
the teachers who should participate. review the quality of the data collection
At least three factors will influence process and the data. Inadequate data
decisions about using these data. First, collection and incomplete or weak data
are of no use to the evaluation. (The
same considerations apply to using
Involving teachers in collecting and later analyzing these data for determining the need for
the professional development and who
data will extend and enrich their professional should participate.)
learning. Involving teachers in evaluating their Involving teachers in data collection

In addition to taking active roles
professional development establishes their in planning and leading professional
development, teachers also can play
ownership of the effort as well as the results. key roles in evaluating professional
development, especially professional
determine whether district personnel/ development that is designed by school
human resource policies and negotiated improvement teams as part of school
agreements prohibit using data from improvement plans and that takes
performance reviews for any other place in school as part of regular school
purpose or if they require that teachers activities. Involving teachers in collecting
give their permission for other uses and later analyzing data will extend
of this data. Second, carefully review and enrich their professional learning.
the alignment of the data collection In addition, involving teachers in
process (e.g. formal observations, evaluating their professional development
walk-throughs) with the content and establishes their ownership of the effort
purpose of the professional development. as well as the results.
Misalignment between the data One way to involve teachers in data
collection is through observing peers’ the evaluators. These tasks require

classroom practices. Peer observations committing time during the
generate a lot of information on teaching regular school day and arranging
practices and help school faculties to cover the classes of teachers
develop and use a common language conducting the observations. Ideally,
about instruction and how to improve observations can take place during
it.11 In addition, teachers can help observers’ preparation time or other
develop observation protocols and the noninstructional periods. Principals
preparations for using them. Teachers and teachers will need to work
preparing for peer observations should together to find ways of covering
practice using the protocols and need classes to minimize disruptions in
clear guidance and explanations about instruction.
how to record and share data. • Orienting them to the task, especially
Although there are benefits to strategies for collecting and
teachers observing peers to collect data, analyzing data and discussing the
several challenges need to be addressed. data (including observational data
These include: and student work samples) with
• Establishing an atmosphere of trust in colleagues. The specifics of this
which observations and comments orientation are similar to preparing
can be freely shared. Principals and other data collectors, as explained
other school leaders need to establish earlier in this section.
this atmosphere among teachers, to
help them recognize the payoffs of Summary
the observations, to promote frank Most evaluations of teacher
discussions of the results of improved professional development will focus
practice, and to determine the focus on whether the program or initiative
of future professional learning. took place as planned, participants’
• Setting aside adequate time in perceptions of the professional learning
the regular school schedule for as part of the overall plan, and whether
conducting observations, recording the professional development achieved
and reviewing data, and discussing its intended results in terms of changes
observations with colleagues and in teacher knowledge and skills, changes
11 For a more extensive discussion of how to plan and use peer observations, visit the Annenberg
Institute’s “Tools for School Improvement Planning” at www.annenberginstitute.org/tools/index2.
php.
in professional practices, and, in some perceptions. If these are the only

cases, changes in student outcomes that source of data, the evaluation will be of
are attributable to changes in teacher limited use. However, when combined
practice. with other kinds of data, they can add
Determining whether the considerable rigor to an evaluation.
professional development occurred Examining professional development
as planned is essentially a process of outcomes and indicators can be a
documenting the extent to which the challenge because good evaluation
necessary resources were available, requires carefully selecting instruments,
whether those teachers who were carefully preparing to use them, and
expected to participate did so, and addressing the knotty problem of
whether all of the planned professional attributing causality to professional
learning took place as planned. When learning. Evaluation planners may want
professional development does not take to consider evaluation designs that focus
place as planned, a good evaluation will on samples of participants and include
attempt to document why. comparison groups. Both options can
Teacher perceptions of the add rigor to the evaluations. At the same
quality and usefulness of professional time, these options are costly and may
development are often early indicators be beyond the reach of most evaluations.
of whether teachers will benefit from the These sophisticated design options
learning and apply what they learned in may help determine causality, but a
their schools and classrooms. Teacher more practical strategy may be to talk
perceptions also may help pinpoint with teachers and ask them to explain
substantive problems in the professional why they do what they do. Carefully
development design. Well-crafted surveys analyzing what they say can help
and other data collection instruments explain how professional development
can provide valuable data about teacher contributes to changes in practice.
D ata q ua l i t y a n d d ata a n a ly s i s
Data quality and data analysis
D
ata analysis begins early with data collection is under way is essential
ongoing monitoring of data to the evaluation’s success. Here are some
quality and continues with tips.
the application of appropriate analytic Monitor response rates on surveys
procedures. Just as data collection reflects and other quantitative data collection.
decisions about outcomes and indicators When administering surveys, pretest/
that will be of interest in the evaluation, posttest instruments to measure changes
data analysis anticipates the reporting in participant knowledge, or similar
task and sets the stage for presenting instruments, ensure that participants
evaluation findings. This section provides are completing and returning the
suggestions for monitoring data quality instruments. Ideally, establish a
and general approaches to data analysis. procedure for tracking individual survey
The guide does not, however, discuss responses that will make it possible
specific statistical procedures that to follow up with nonrespondents. If
evaluators can use to analyze evaluation surveys are administered online, send
data. e-mail reminders to nonrespondents. If
individual responses cannot be tracked,
Monitoring data quality send e-mail reminders to all participants.
A key to successful evaluation is If the survey permits identifying specific
having solid data. In addition to selecting groups of participants (e.g. elementary
appropriate instruments and preparing school reading teachers, elementary
the evaluation team to use them, school special education teachers), send
evaluators need to monitor the quality e-mail reminders to groups with low
of data as the data are being collected. response rates.
Although there are exceptions, gathering The higher the survey response rates,
additional data after completing the the better. Time and resource constraints
planned data collection is difficult. permitting, aim for response rates of at
Therefore, monitoring data quality while least 80% for all participants or 80%
for each identifiable group. Analyzing detail. For example, if the professional
data or drawing meaningful conclusions development begins with five days
is more difficult if response rates drop of workshop sessions, have a sign-
below 80%. in sheet for each workshop session.
Monitor record-keeping systems Having separate sign-in sheets permits
that track individual participation tracking overall participation as well as
and implementation of the activities participation patterns. Enter data from
specified in the professional the sign-in sheets in the evaluation
development plan. To be able to database as soon as possible after
report on participation patterns and participants sign in.
implementation of key components of When professional development
the professional development, record- includes multiple sessions over several
keeping systems need to be developed weeks or months, organize the record-
and maintained. As professional keeping system to track individual
development increases in size, scope, teacher participation in all of the
and duration, record keeping becomes professional development. Later, these
both more important and more data can help profile participation
complicated. Nevertheless, these data patterns. If the variations are significant,
are critical in order to be able to present compare and contrast participation
detailed findings about participation patterns to determine if they may relate
and implementation. A problem in to teacher outcomes. In the shorter term,
many evaluation reports is that they interim evaluation reports may pinpoint
describe what was supposed to happen gaps in participation, making it possible
and assert that teachers participated as for professional development providers
expected. Both the description and the and facilitators to identify ways to
assertion may be valid, but neither can be improve participation as the professional
supported without solid evidence. development continues.
Sign-in sheets are good sources of Work with providers and facilitators
data on participation in large-group to track any follow-up professional
professional development. Work with development, such as observations,
professional development providers and feedback, collaboration with other
facilitators to ensure that participants’ teachers, and assistance from school-
attendance at these sessions is recorded. based professional development staff.
Ensure that the sign-in sheets collect One approach is to ask school-based
information at an appropriate level of professional development staff to
maintain simple activity logs to record that are not included in the protocols,
information about their interactions and, if so, how can the protocols be
with participants in school or classroom- modified to capture these things? What
based follow up. Data from the logs findings are emerging? Should data
can easily be combined with data from collection instruments and plans be
sign-in sheets to generate more complete modified to examine these findings in
participation records. more depth?
Monitor the accuracy and
completeness of entries in observation Data analysis
protocols and similar data collection Data analysis can be very simple or
instruments. If data collection includes complex, or, more likely, somewhere in-
structured or unstructured observations, between. In almost all evaluations, it is a
interviews with teachers and others, or good idea to start with the basics. More
similar activities, monitor the processes complex analyses, including sophisticated
to ensure that the data collection statistical analysis, occur later if they are
activities are taking place as planned. warranted and feasible given the quality
Also check completed reporting forms and quantity of the evaluation data and
to ensure that data are being recorded the resources available.
correctly. As with the other data quality Learn about the data by starting
control efforts suggested here, tracking with the basics. A real benefit of
these efforts can help ensure that there monitoring data quality is that the
are no gaps in the data and that the process helps familiarize evaluators with
data are high quality. Data collection the data and what the data suggest about
and data quality problems are easier to the professional development. Indeed, an
resolve if they are identified early while important first step in data analysis is to
data are being collected, but are almost get to know the data and to see what they
impossible to resolve after the fact. appear to suggest even before conducting
When an evaluation includes several a more careful and systematic analysis.
cycles of observations and interviews, In many evaluations of teacher
convene the data collectors after each professional development, much of the
cycle to review the data collection data analysis involves counting. A starter
activities. Invite data collectors to discuss list of questions can be answered by
any problem they experience in using counting various data:
observation and interview protocols. Are • How many teachers participated in
they seeing or hearing important things all of the learning opportunities,
and how many teachers participated of participants. For example, if data

in only a few of the learning collection for an evaluation of large
opportunities? professional development activity
• How many participants gave the included participant surveys and the
professional development high marks surveys collected information about
for usefulness and relevance? teacher characteristics such as years of
• How many participants reported experience and particulars of current
that they received adequate help teaching assignments, analyses could
and encouragement in applying compare and contrast the perceptions
new knowledge and skills in their of various groups of teachers (e.g. new
classrooms? or inexperienced teachers and more
• How many participants viewed the experienced teachers, lower elementary
professional development as a good grade teachers and upper elementary
start versus providing them with all grade teachers, reading teachers and
the information they needed to apply content specialists). An example in the
new knowledge and skills? box on p. 49 illustrates a more complex
• How many participants scored at or comparison and the kinds of insights it
above proficient on a written test of can yield.
content-area knowledge administered Quantify the qualitative data.
one month after the professional Qualitative data, including data collected
development? Are consistent gaps in through observations, interviews, self-
content knowledge evident? reports, and student work samples, yield
• Based on completed observation rich descriptions of professional learning,
protocols, how many participants classroom instruction, and student
could be rated as proficient in learning outcomes. These descriptions
applying new instructional strategies are especially useful when they include
in their classroom? Do observational some sort of quantification to illustrate
data suggest gaps in application? the extent to which they are typical
Depending on the professional or representative of all participants’
development design, the kinds of data experiences. Thus, observation protocols
available, and the quality of the data that yield detailed descriptions of
systems developed for the evaluation, instructional practices (i.e. the number of
it may also be possible and useful to students engaged in class discussions, the
compare and contrast the perceptions number of students to whom teachers
and experiences of various groups direct particular kinds of questions, the
A second look at evaluation data
T
he activity being evaluated focused on extensive coaching appeared to be related to
helping participants understand and apply greater implementation of new instructional
instructional strategies to increase reading practices. Subsequently, when evaluators reviewed
fluency. The first phase of professional development activity logs maintained by the reading resource
included a concentrated series of workshops on these teachers, they found considerable variation in
strategies, and the second phase included ongoing amounts and kinds of school-based follow-up to
observations and coaching from school-based support implementation. Some teachers received
reading resource teachers. The evaluation design help several times a week, while others received
called for documenting participation in phases, using help once a week or even less. When evaluators
a written test to assess teachers’ understanding of re-examined observation protocol data used to
instructional strategies to develop reading fluency measure the extent to which teachers applied
and assessing teachers’ application of the strategies new practices, they found that teachers who had
through a series of structured observations four received more follow-up help were better able to
to eight weeks after the teachers completed their use new instructional practices than were teachers
professional learning, including the school-based who had received less help.
supports. Looking at the observation records a second
Initial evaluation results indicated that time, the evaluators also found school-level
teachers participated in all learning opportunities differences in the amount of coaching support that
and that most gave the professional development teachers received. Teachers in some schools received
high marks for potential usefulness. Most considerably more support than did teachers in
participants also agreed that the strategies other schools. Curious, the evaluators returned to the
would work well with their students. In addition, schools to talk with principals and reading resource
their written test scores clearly suggested that teachers about the arrangements for working with
a majority of participants understood the new individual teachers. These conversations revealed
instructional strategies. important differences in how principals and reading
To the evaluators’ surprise, however, the initial resource teachers worked together and in the
review of the observational data revealed rather amount of support and encouragement principals
large differences in classroom implementation provided. This finding led evaluators to conclude that
of the new instructional practices. Because they stronger working relationships and higher levels of
were able to link the data on coaching support encouragement and support were clearly related
with the data on classroom instruction, evaluators to the reading resource teachers providing more
were able to examine the extent to which more classroom support.
availability and arrangement of classroom new instructional practices, can be useful

supplies and materials, etc.) also make in analyzing changes in teacher practice.
it possible to report on how many Once again, counting is important. How
teachers are actually implementing new many teachers reported applying a new
practices. Reporting observers’ judgments instructional strategy and provided a
about the extent to which teachers are concrete example? How many teachers
implementing new practices makes it reported changes in student outcomes
incumbent on the evaluators to explain that resulted from teachers applying a
the extent to which the observations may new strategy and provided a concrete
reflect observer bias. example of change in student work or
When an evaluation collects assessment results?
more open-ended data, such as data
from semistructured observations or Explaining cause and effect in
interviews, the analytic task is to review evaluations of teacher professional
write-ups from data collection activities development
for key words and key themes and to Perhaps the greatest challenge
count their frequency. Because of the in evaluating teacher professional
costs associated with preparing observers development is determining patterns
and interviewers to carry out these data of causality. The bad news is that
collection activities, as well as the costs the evaluation designs necessary to
associated with data analyses, think empirically determine causality are
carefully about pursuing this design beyond the scope of most state and
option. One solution to the problem of local evaluation efforts. The good news
analyzing large amounts of qualitative is that there are options for looking at
data, especially data from structured causality in evaluations, including several
and semistructured observations and discussed in this guide.
interviews, is to use software programs Look carefully at the data
designed specifically for this purpose.12 professional development planners used
The advantage of using software is that it to assess the need for the professional
helps organize and analyze large amounts development. What do these data
of qualitative data. suggest about teacher knowledge, skills,
Anecdotal data, including self- and practice prior to the professional
reports of implementation and use of development? For planning and design
12 Visit www.qsrinternational.com for more information about software available for use in
qualitative data analysis. Some versions of the software are available at no cost.
purposes, these data can be considered for the reasons for the changes.
as baseline or pre-intervention data. One way to examine causality in
In general, the goal of professional the kinds of evaluations discussed in this
development is for teachers to move guide is to ask teachers directly about
from the baseline to higher levels of whether and how their professional
knowledge, skills, and performance, and development experiences contributed
the purpose of evaluation is to determine to changes in knowledge, skills, and
whether these changes did occur and, if professional practice. For example, after
so, whether they are attributable, at least a series of classroom observations that
in part, to teacher participation in the yield clear and consistent evidence of
professional development. appropriate application of instructional
Next, following the suggestions strategies that were the focus of the
offered earlier, determine whether professional development being
the professional development was evaluated, interview teachers to ask
implemented or occurred as planned. where and/or how they learned about
If it did take place as planned, look these strategies and what motivated them
for evidence that the intended changes to try the strategies in their classrooms.13
in teacher knowledge, skills, and Follow-up questions should probe
professional practice (as specified in for concrete examples of factors that
the outcomes and indicators included influenced teacher use of new strategies.
in the original plan) occurred. If these Follow-up questioning should also
changes did occur and if they occurred explore the relative salience of various
after participation in the professional factors in teacher decisions. It is possible,
development, the evaluation should even likely, that teacher responses will
focus on explaining the reasons for suggest an array of factors contributing
those changes. If the evaluation finds to these changes. Carefully analyze these
substantial variation between the original responses to help evaluators understand
plan and the professional development whether and how participation in
that occurred and the evaluation also professional development caused
finds changes in participant knowledge, changes in teacher knowledge, skills, and
skills, and/or professional practice, look performance.
13 It is generally not a good idea to rely on surveys or other self-report forms to ask teachers
about the factors that influenced their use of new strategies unless direct follow-up permits
probing their initial responses for concrete examples. Open-ended survey questions are especially
difficult to analyze and are not good sources of information.
Extending the causal links to student professional development to changes in

outcomes related to teacher participation student learning.
in professional development is even more In the end, findings and conclusions
difficult. This difficulty also underscores of the professional development
the importance of examining indicators evaluation are almost certain to be
of student learning that are closer somewhat tentative and speculative. The
(proximal indicators) to the focus of the challenge is to present data clearly and
professional development than those to provide a compelling case for linking
that are more distant (distal indicators). participation in professional development
Examples of proximal indicators are to changes in teaching and learning.
student work samples produced in The case is most compelling when it
response to prompts given by teachers recognizes the complexity of the causal
and scores on locally developed chain and acknowledges that a variety of
assessments. Examples of distal indicators factors contribute to changes in teaching
and learning. The case is least compelling
when it simply asserts that teacher
Teacher involvement in data analysis is, by itself, participation in professional development
resulted in improved teaching and
a valuable professional learning opportunity. learning.
are scores on state assessments. Involving teachers in data analysis

The advantages of using proximal Just as they can be involved in
indicators are that proximal indicators are collecting data, teachers can play a role in
observable and measurable sooner than are analyzing data. Teachers can, for example,
distal indicators, and proximal indicators share peer observation results and examine
typically reflect a narrower range of what they learned about implementation
student learning. A related advantage of new instructional strategies. Teachers
is that because they are observable also can participate in reviewing student
sooner — closer to the period of teacher work samples that serve as indicators of
participation in professional development the implementation of new instructional
— they are less likely to be influenced by strategies.14 Teacher involvement in data
other variables. Nevertheless, be careful analysis is, by itself, a valuable professional
about linking teacher participation in learning opportunity.
14 For more ideas about involving teachers in the analysis of student work and using protocols in
this process, visit www.lasw.org/protocols.html.
Strategies for involving teachers in student work samples. In this approach,

data analysis are similar to those related the teachers could develop the protocol
to involving them in data collection, for reviewing the work samples and
which were discussed in the previous discuss their observations. In both cases,
section. the completed protocols and detailed
In some evaluations, evaluators records of the conversations would
may facilitate teachers’ work on analytic provide important data while also serving
tasks. For example, if the analytic task as a potentially powerful approach to
entails reviewing a collection of student data analysis.
work samples to look for evidence
that teachers who participated in the Summary
professional development are applying Data analysis is an iterative
new instructional practices in their process that should begin soon after
classrooms, evaluators could call on data have been collected. Early tasks
teachers, including participants, to assist are monitoring data quality, which,
in the analysis. The evaluator, perhaps depending on the evaluation design, can
in collaboration with teachers, would include monitoring survey response rates
develop a protocol or rubric to guide the and following up with nonresponders,
analysis of student work samples. Prior making sure that record-keeping systems
to distributing the work samples, the are functioning properly and that data
evaluator would train teachers in using are being entered according to plans, and
the instrument to ensure a degree of ensuring that observations, interviews,
consistency in the reviews. Each teacher and other data collection are proceeding
would receive a set of work samples on schedule.
and be asked to use the protocol to Evaluators should begin formal
complete the review. The work samples analysis of data by familiarizing
would not identify students or teachers. themselves with the basics, including
After a reasonable amount of time for comparing and contrasting the
review, the evaluator would facilitate a perceptions and experiences of various
discussion organized around the protocol groups of teachers. Also examine
and what the teacher-analysts saw when data on professional development
they looked at the student work samples. outcomes and indicators. The challenge
Alternatively, a group of teachers might is to carefully document outcomes,
decide to collaboratively evaluate their typically described in terms of changes
own professional learning by examining in teacher knowledge, skills, and/or
practice. Some evaluations go a step typically beyond the scope and resources
further and try to gauge the impact of available for most evaluations of teacher
these changes on student learning and professional development, including
other student outcomes. The central almost all local evaluations.
challenge in this phase of the analysis This guide recommends a more
is to examine causality. What evidence direct approach to getting at causality. In
is available to suggest how and to what the context of collecting data on various
extent participation in professional teacher outcomes, evaluators can ask
development contributed to changes teachers to explain certain behaviors,
in teacher outcomes? Sophisticated particularly the reasons behind them.
evaluation designs, including carefully This questioning must be done carefully,
chosen comparison groups and complex and it should probe for concrete
analytic procedures, can address this examples as well as evidence of other
issue. The important payoffs of these factors that may have contributed to
approaches notwithstanding, they are changes in practice.
Reporting
Reporting
T
he last step in any evaluation overlap. For example, all of these
is preparing the evaluation groups and others who may read the
report. As with the evaluation report will want to know whether the
itself, there is no single formula for the professional development achieved the
report. Reports should, however, adhere results specified in the original design.
to some basic principles to help readers Audiences are likely to vary in their
understand the professional learning interest in whether the professional
that took place, the extent to which the development unfolded as planned,
professional development contributed although principals, participants, and
to changes in teacher knowledge and staff who worked on the professional
skills, and whether and how teachers development are likely to be interested
applied the new knowledge and skills in in problems that may have arisen as the
their classrooms. More ambitious reports professional development unfolded.
examine the links between these changes Almost all readers will welcome
and changes in student outcomes. In short, nontechnical reports. Here are
general, evaluation reports should: ways report writers can help readers:
• Include an executive summary and
1. Anticipate readers’ information clear, concise summaries at the end
needs and interests. of each major section of a report.
Planning the evaluation involved The executive summary is often of
anticipating who would be interested primary interest for the majority
in the results. Potential audiences of readers. The executive summary
include funders, state and local policy should concisely present the main
makers, district leaders and central points readers should take away.
office staff, building administrators, • Use advance organizers to provide a
and, of course, the participants. map of the report and its key points.
Each group is likely to have different Advance organizers guide readers
interests, although there may be some through the report by pointing out
Reporting
what is coming and what has been misinterpretation. Few things

discussed. undermine an evaluation report’s
• Define important terms (and possibly credibility more than errors in
include a glossary). Explaining or descriptions of the professional
describing instructional practices development and errors in reporting
that may be the focus of the findings. Review drafts with great
professional development helps care and, whenever possible,
readers understand the activity and ask several people who did not
interpret evaluation results. Similarly, contribute to the report to review
explaining what the professional and comment on the draft.
learning (e.g. teacher study groups, • Create clear and simple charts and
peer observations, action research) exhibits to present data and illustrate
entails also helps readers understand important findings. Including charts
and interpret evaluation findings and and exhibits is an efficient way to
conclusions. communicate lots of information
• Provide concrete examples to amplify or explain key concepts. However,
descriptions and observations. One charts and exhibits that are
of the best ways to add richness unnecessarily complex or that
and depth to an evaluation report require a lot of study can be
is to include anecdotes and relevant distractions and don’t add value to
quotes to highlight important points. the report.
Choose examples carefully and • Include technical appendices to
present them in context to explain provide additional detail on the
what they actually represent. A evaluation design, data collection,
quote from a teacher that reflects and data analyses for interested
many participants’ views may be readers. Most readers will not be
worth sharing. Another teacher’s interested in extensive technical
comment that reflects a minority detail about data collection and data
view also may be important. Both analysis, so it makes sense to include
examples communicate important this information in appendices.
information, but this information • Use the active voice to communicate
is most useful when presented in clearly and concisely. Reporting in
context. the active voice increases readability
• Carefully check the facts to and adds precision to descriptions
avoid misunderstanding and and explanations.
Reporting
2. Explain the evaluation’s purpose interested readers to draw their

and address each of the evaluation own conclusions about key findings
questions. and results.
Introduce the logic model (or other Describe the kinds of analyses
conceptual framework) that guided conducted and, as appropriate, why these
the professional development and procedures were selected. It may make
informed the evaluation. Alternatively, sense to discuss the analyses in greater
discuss the key assumptions that guided detail in a technical appendix to the
the professional development and the evaluation report.
evaluation.
After discussing the purpose(s) of 5. Present all of the findings and
the evaluation, list evaluation questions results — both the positive and the
and explain why they are important. The negative.
answer(s) to each question should include Discuss all of the key findings
as much detail as necessary to help readers about each evaluation question, whether
understand the evaluation results, but the findings are positive or negative.
these discussions should also include a Evaluation reports that present only
clear, concise summary of the answers. positive results and/or gloss over or
neglect negative findings have little or
3. Describe the approach to the no credibility among readers and add
evaluation. little value to subsequent conversations
Explain what data were collected, about professional development plans,
how the data were collected, from whom practices, policies, and funding. Most
they were collected, and the timeline. readers understand that professional
Include copies of key data collection development — especially professional
instruments, or include the instruments development that involves large numbers
in technical appendices. Finally, present of participants and that extend over
information about response rates, the long periods of time — rarely take place
extent to which complete data are exactly as planned. Similarly, professional
available for all participants included development does not always achieve
in the evaluation, and any problems its intended outcomes or achieves
encountered. them partially or only among some
participants.
4. Explain data analysis procedures Presenting evaluation results
in sufficient detail to permit objectively and in a straightforward
Reporting
manner helps readers understand what The first and third examples convey
happened, what results were achieved, judgments about the findings and may
and why. Report also on any problems reflect the evaluator’s bias. The second
that affect the evaluation. For example, example simply reports the finding.
evaluation reports should be clear about Both the first and third examples call for
problems such as low survey response an explanation of what criteria, either
rates that make it difficult to interpret implicit or explicit, led the evaluator to
survey results in meaningful ways. make such a judgment. Why is 25% of
Similarly, if classroom observations or teachers either “scant” or “impressive?”
teacher interviews included as part of the The second example does not require an
evaluation design were not conducted explanation, although the evaluator may
as planned, discuss these problems and choose to provide one in a summary or
provide a clear explanation of how data concluding section of the report.
collection problems or gaps in the data In general, when an evaluation
affected analysis and the confidence that report presents judgments about the
readers should have in the evaluation quality of professional development
findings. or about its impact or effectiveness,
While it is important to report the report should also be clear about
negative findings and problems that the criteria or standards used to make
may have affected evaluation results, it the judgment. For example, if the
is equally important that reports not evaluation report concludes that the
overemphasize the negative findings. professional development did not
To illustrate how a few words change achieve the intended results, the report
the “meaning” of an evaluation finding, should remind the reader what the
consider the examples of a description of expected outcomes were. Further, if
survey results in the box below. the report attempts to explain why the
How a few words can redefine the evaluation findings

• A scant 25% of teachers who responded to the survey reported that they found the activity useful
compared with other professional development that was available to them.
• Twenty-five percent of teachers who responded to the survey reported that they found the activity
useful compared with other professional development that was available to them.
• An impressive 25% of teachers who responded to the survey reported that they found the activity
useful compared to other professional development that was available to them.
Reporting
professional development did not achieve 7. A final consideration: One

the intended results, the report should evaluation report or several
be very clear about the extent to which reports?
the proffered explanations are based on Comprehensive, long-term
findings from the evaluation, especially professional development activities and
findings about implementation and initiatives probably suggest the need
participation, or the extent to which they for several reports. If the professional
are based on speculation. development includes several cycles
of learning spread out over four to six
6. Draw on key findings and months or longer, and if there are clearly
conclusions to present actionable defined interim benchmarks specified
recommendations. in the overall plan, prepare several
As students and observers of reports. For example, if the professional
teacher professional development, development extends over two school
especially when they are involved years and teachers are expected to begin
from the earliest stages of planning, using new instructional practices during
evaluators are in a good position to the first year, an interim report prepared
make recommendations. Just as a over the summer between the first and
good logic model provides a road second year of professional development
map for an evaluation, thoughtful could examine the first-year professional
recommendations provide a road map learning and initial implementation
for improving and expanding teacher of new practices. A final report, which
professional learning opportunities. continues to track the professional
Useful recommendations anticipate development in the second year and
steps readers are likely to be able to take examines ongoing changes in instruction,
and offer concrete suggestions for taking could be completed in the summer after
them. Provide a brief rationale for each the second year.
recommendation, drawing on specific Preparing multiple evaluation reports
findings and conclusions. If it makes has several advantages. First, interim
sense for the recommended action steps reports can help providers and managers
to occur in a particular order, explain determine whether the professional
the order. If challenges are likely with development is taking place as planned
the recommended action steps, point and whether they appear to be achieving
them out and provide suggestions for intended outcomes. Findings in these
overcoming them. reports can also inform midcourse
Reporting
corrections that increase the likelihood was evaluated, the approach to the
of achieving overall outcomes. Interim evaluation, and evaluation findings and
reports make it possible to provide conclusions. Most evaluation reports
information about results — albeit also include recommendations that
incomplete information — to policy outline specific action steps related
makers and funders sooner. Depending to the activity being evaluated and/
on the circumstances, these results or related to future professional
may help leverage additional resources development. Effective evaluation
for the professional development. To reports are objective and candid.
be sure, interim reports that indicate They describe what happened and, as
that the professional development is appropriate, what did not happen. When
not proceeding smoothly or that it is the professional development was not
not producing expected results could implemented according to plans or did
undermine support for the professional not achieve the intended outcomes, the
development. At the same time, evaluation report describes factors that
knowing what does not work or does not may have contributed to incomplete
contribute to improved practice is almost implementation or that impeded
as important as knowing what does work. successful attainment of the outcomes.
In a period of increasing
Summary accountability for quality professional
Well-written evaluation reports, development and for measurable and
including interim reports and final observable outcomes, weak evaluations
reports, can inform decisions about and inadequate reports do not add
whether to continue, expand, and/ value to the important conversations
or modify a professional development about next steps, and they can derail
program or initiative. Evaluation findings the conversations. In contrast, solid
and recommendations also can inform evaluations and articulate evaluation
decisions about future initiatives, reports can influence both policy and
including decisions about the design and practice in positive and constructive
the resources necessary to ensure success. ways. The evaluation process can, in
Effective evaluation reports powerful ways, extend professional
anticipate readers’ interests and address learning and reflection for teachers and
these interests with clear descriptions others engaged in the work.
of the professional development that
App e n d i x A : R e s o u r c e s f o r E va luat i n g t e a c h e r p r o f e s s i o n a l d e v e lo p m e n t
Appendix A
Resources for evaluating teacher professional development
The resources listed here provide conceptual models, guidance on developing

instruments and conducting evaluations, and samples of data collection instruments
(especially instruments for collecting various kinds of qualitative data). These resources
should be considered a starter set; they by no means represent a comprehensive sample.
In addition, as evaluation planners review the examples of instruments and data
collection activities, they should recall the importance of tailoring the instruments and
data collection activities to meet the needs of the evaluations that they are planning.
Administration on Children, teacher professional development have

Youth, and Families. (no date). The effects on teaching and learning? Analysis
program manager’s guide to evaluation. of evaluation findings from programs for
Washington, DC: U.S. Department of mathematics and science teachers in 14
Health and Human Services. Available at states. Washington, DC: Council of Chief
www.acf.hhs.gov/programs/opre/other_ State School Officers. Available at www.
resrch/pm_guide_eval/reports/pmguide/ ccsso.org/content/pdfs/cross-state_study_
pmguide_toc.html. rpt_final.pdf.
This guide and a companion guide, This report examines findings from
The Head Start Bureau Evaluation a number of evaluations of professional
Handbook, help program staff plan and development for teachers across
conduct local evaluations. Both guides the country and provides extensive
include tools to use in local program information about evaluation designs.
evaluations and explain how programs Council of Chief State School
can use evaluation results to improve Officers & National Science
program design and services to children Foundation. (2008, April). Bridging
and their families. the gap between research & practice in
Blank, R.K., de las Alas, N., & professional development: Selected research
Smith, C. (2008, February). Does and resources. Washington, DC: Authors.
App e n d i x A : R e s o u r c e s f o r e va luat i n g t e a c h e r p r o f e s s i o n a l d e v e lo p m e n t
A companion to the review of S., & Hughes, G. (2002, June). The

evaluation results, this report includes 2002 user-friendly handbook for project
extensive lists of resources that are evaluation. Washington, DC: The
available for evaluating professional National Science Foundation. Available
development, as well as information at www.nsf.gov/pubs/2002/nsf02057/
about locating these resources. nsf02057.pdf.
Dirr, P.J. (no date). Classroom This handbook explains various
observation protocols: Potential tools for approaches to program evaluation in
measuring the impact of technology in the easy-to-understand language.
classroom (Policy and Planning Series Guskey, T. (2000). Evaluating
#104). Alexandria, VA: Appalachian professional development. Thousand Oaks,
Technology in Education Consortium CA: Corwin Press.
with support from the CNA Prepared for the National Staff
Corporation. Available at www.eed.state. Development Council, this book
ak.us/edtech/pdf/ATEC-PP104Tools. describes five levels of professional
pdf. development evaluations, explains how
This report explains how to use to conduct evaluations at each level,
various observation protocols to and describes the relative benefits of
collect information on the impact of the results. The book also contains an
interventions in classrooms. The report exhaustive list of references to other
also contains information about a resources for professional development.
number of well-designed protocols that Henry, M.A., Murray, K.S., &
can be used for this purpose, including Phillips, K.A. (2007, November
their use in pre-/post-evaluation designs. 30). Meeting the challenge of STEM
Elmore, R. (2007, April). classroom observation in evaluating teacher
Professional networks and school development projects: A comparison of
improvement. The School Administrator, two widely used instruments. St. Louis,
4(64). Available at www.aasa. MO: M.A. Henry Consulting, LLC.
org/SchoolAdministratorArticle. Available at http://sirc.mspnet.org/index.
aspx?id=6812. cfm/14976.
This article describes how This report discusses the challenges
observations that yield descriptions associated with using classroom
rather than judgments are important in observations in evaluating teacher
understanding classroom practice. professional development and offers
Frechtling, J., Frierson, H., Hood, concrete suggestions for overcoming
the challenges. It also reviews two King, J.A., Morris, L.L., & Fitz-
observational tools, including their Gibbon, C.T. (1987). How to assess
reliability and validity. program implementation. Newbury Park,
Killion, J. (2008). Assessing impact: CA: Sage Publications.
Evaluating staff development (2nd ed.). Part of the nine-book Program
Thousand Oaks, CA: Corwin Press & Evaluation Kit, this small volume
NSDC. provides detailed guidance on how to
This guide is an excellent practical plan and conduct evaluations of program
resource for planning and conducting implementation. Discussions of data
evaluations of teacher professional collection and data analysis are especially
development. It explains a variety of helpful, and a number of sample data
approaches to evaluations and includes collection instruments are included that
a number of tools that are useful to can be adapted for evaluations of teacher
both practitioners and evaluators. An professional development.
accompanying CD provides specific Kirkpatrick, D. (1994). Evaluating
suggestions for how to introduce the training programs. San Francisco: Berrett-
guide to various users. Koehler Publishers.
Killion, J. & Harrison, C. (2006). One of the classics on program
Taking the lead: New roles for teachers evaluation, this book introduces the
and school-based coaches. Oxford, OH: notion of various levels of evaluation of
NSDC. training programs and explains the uses
This book, which focuses on the of each level.
multifaceted role of school-based Maryland State Department
professional developers, includes detailed of Education. (2008, November).
suggestions for evaluating their impact in Maryland teacher professional development
schools and classrooms. planning guide. Baltimore, MD: Author.
Killion, J. & Roy, P. (2009). Available at www.marylandpublicschools.
Becoming a learning school. Oxford, OH: org.
NSDC. The guide describes the elements of
From setting the stage to engaging an effective plan for teacher professional
the community in understanding the development and presents a six-step
purpose of collaborative professional planning process.
learning teams, this volume covers what McDonald, J.P., Mohr, N., Dichter,
leaders need to know to implement more A., & McDonald, E.C. (2007). The
effective professional learning. power of protocols: An educator’s guide
App e n d i x A : R e s o u r c e s f o r e va luat i n g t e a c h e r p r o f e s s i o n a l d e v e lo p m e n t
to better practice. New York: Teachers Newbury Park, CA: Sage Publications.
College Press. Another in the Program Evaluation
The authors explain how carefully Kit, this book explains how qualitative
crafted protocols can be used to structure methods can and should be used in
discourse about practice, to collect, evaluations. In addition to guidance
organize, and analyze data, and to on designing qualitative evaluations, it
examine student work. offers extensive guidance on conducting
National Staff Development observations and interviews and
Council. (2001). NSDC’s standards for analyzing qualitative data.
staff development. Oxford, OH: Author. Puma, M. & Raphael, J. (2001).
Twelve standards define what is Evaluating standards-based professional
necessary if staff development is to development for teachers: A handbook
impact student achievement. Each for practitioners. Washington, DC: The
standard is accompanied by a two-page Urban Institute. Available at www.urban.
discussion that includes a rationale, case org/publications/410432.html.
study, discussion questions, references, This detailed guide provides step-
and next steps. Includes an assessment by-step directions for evaluating what
instrument, annotated bibliography, and the authors call standards-based teacher
suggestions for use. professional development. The guide also
Ontario Ministry of Education. develops cases of two district evaluations
(2007, September). Teacher moderation: to illustrate different approaches to
Collaborative assessment of student evaluation.
work (The Literacy and Numeracy Roy, P. (2007). User’s guide:
Secretariat). Toronto, Ontario: Author. Innovation configurations for NSDC’s
Available at www.edu.gov.on.ca/eng/ standards for staff development. Oxford,
literacynumeracy/inspire/research/ OH: NSDC.
Teacher_Moderation.pdf. This user’s guide explains what
This publication describes how Innovation Configurations are and
teachers can work together to assess how to use them in planning staff
student work as part of larger efforts to development that meets NSDC’s
understand and improve instruction. A standards. The guide also explains
90-minute video illustrates various parts how developing and using Innovation
of the process. Configurations can help gauge
Patton, M.Q. (1987). How to progress in implementing high-quality
use qualitative methods in evaluation. professional development.
Singer, N.R. & Scollay, D. (2007). nwp_file/11082/beyondstrategies.pdf?x-

Increasing student achievement in writing r=pcfile_d.
through teacher inquiry: An evaluation of Drawing on research funded by the
professional development impact. Available Local Sites Initiative of the National
at www.nwp.org/cs/public/download/ Writing Project, this article reports
nwp_file/10561/Gateway_LSRI_ on a small-scale case study comparing
Cohort_II.pdf?x-r=pcfile_d. teaching practices of a teacher who
This report is a good example of participated in an inquiry-focused
a rigorous local evaluation of teacher professional development activity with
professional development in the area of one who did not. In addition to finding
writing instruction. positive results attributable to the
Taum, A.K.H. & Brandon, P.R. professional development, the study and
(2006, April 9). The iterative process of the report are good examples of teacher
developing an inquiry science classroom involvement in professional development
observation protocol. Paper presented at research and evaluation.
the annual meeting of the American W.K. Kellogg Foundation. (1998,
Educational Research Association, San January). W.K. Kellogg Foundation
Francisco. Available at www.hawaii.edu/ evaluation handbook. Battle Creek, MI:
crdg/programs/documents/evaluation/ Author. Available at www.wkkf.org/
ISOCS_AERA_2006.pdf. knowledge-center/Resources-Page.aspx.
This essay offers clear guidance Long a leader in the field of program
and examples for developing classroom evaluation, the W.K. Kellogg Foundation
observation protocols. published this guide to help program
Whitney, A., Blau, S., Bright, A., leaders and staff plan and conduct
Cabe, R., Dewar, T., Levin, J., Macias, rigorous evaluations. The guide also
R., & Rogers, P. (2006, April). Beyond includes case studies of how programs
strategies: Teacher practice, writing process, have used evaluation results to enhance
and the influence of inquiry. Available program quality and outcomes.
at www.nwp.org/cs/public/download/
App e n d i x B : S a m p l e i t e m s f o r s u r v e y i n g pa r t i c i pa n t s’ v i e w s o f p r o f e s s i o n a l d e v e lo p m e n t
Appendix B
Sample items for surveying participants’ views of professional
development
Planners should conduct their own searches for instruments and data
collection strategies that may be appropriate for their evaluations. Use these and
other search terms to locate helpful documents on the Internet.
• Looking at student work.
• Assessing/evaluating program impact.
• Measuring/assessing teacher knowledge (add a content area to the search
terms).
• Evaluating training (and/or training programs).
• Program evaluation/impact evaluation.
• Logic model/theory of change.
The following items can be adapted and used in surveys of participants’ views
of their professional learning.
1. Understanding the purpose of the professional development.

Sample item: Which of the following statements best describes the primary purpose
of (insert the name of the professional development)? (Select one.)
The purpose of the professional development was:
A. To communicate new ideas for me to consider using in my classroom.
B. To provide an opportunity for me to learn from other teachers.
C. To help me understand (insert content of professional development).
D. To help me apply/implement (insert content of professional development or other
descriptor) in my classroom.
E. Not clear.
F. Other (specify).
Note: In using this or a similar item, be sure that one response option includes the
intended purpose of the professional development.
2. Ratings of the usefulness of key components of the professional

development.
Sample item: Which of the following statements best describes the usefulness of
(insert the name of the professional development or a specific component)? (Select one.)
A. It was a good start.
B. It was a good start, but I have a lot of questions.
C. It was a good start, and I look forward to using the new ideas in my classroom.
D. It provided everything I need to use the new ideas in my classroom.
E. I don’t think that these ideas will work very well in my classroom.
F. It’s too soon to tell.
3. Perceptions of the extent to which the professional development met

participants’ needs.
Sample item: Indicate the extent to which (insert the name of the professional
development) met your professional learning needs. (Select one.)
A. It addressed my professional learning needs completely.
B. It addressed some of my professional learning needs.
C. It did not address my professional learning needs.
D. This professional development did not help much because I was already familiar
with this topic.
4. Ratings of the alignment of the content of the professional development

with improvement priorities.
Sample item: To what extent was (insert the name of the professional development)
aligned with (school/district) (goals/priorities) for improving instruction? (Select one.)
A. The professional development was very closely aligned with (goals/priorities) for
instructional improvement.
B. The professional development was somewhat aligned with (goals/priorities) for
C. The professional development was not aligned with (goals/priorities) for
D. The professional development was inconsistent with (goals/priorities) for
E. I don’t know.
5. Perceptions of support and encouragement to participate in the

professional development.
Sample item: Which of the following statements best describes the support that
you received from your principal (or other school leader or school-based professional
development staff) to participate in (insert the name of the professional development)?
(Select one.)
A. The principal strongly encouraged me to participate.
B. The principal encouraged me to participate.
C. The principal tried to discourage me from participating.
D. I did not discuss the professional development with the principal prior to
participating.
6. Perceptions of support and encouragement to apply new knowledge and

skills.
Sample item: Which of the following statements best describes the support that
you received from your principal to apply what you learned in (insert the name of the
professional development) in your classroom? (Select one.)
A. The principal has encouraged me to apply what I learned in my classroom.
B. The principal has encouraged me to apply what I learned in my classroom and
has offered to help.
C. The principal has not encouraged me to apply what I learned in my classroom.
D. I have not discussed what I learned with the principal.
Note: Additional items and responses can focus on encouragement from other
school leaders, school-based professional development staff, and other teachers.
7. Ratings of the likelihood of applying new knowledge and skills in the

classroom.
Sample item: Which of the following statement best describes the likelihood that
you will apply what you learned in (insert the name of the professional development) in
your classroom? (Select one.)
A. I have already (practiced/applied) (skill/practice) in my classroom.
B. I have already (practiced/applied) (skill/practice) in my classroom, and it seemed
to work well.
C. I have already (practiced/applied) (skill/practice) in my classroom, but it was not
appropriate for my students.
D. I look forward to (practicing/applying) (skill/practice) in my classroom in the next
few weeks.
E. I look forward to (practicing/applying) (skill/practice) in my classroom sometime
later this year.
F. I would like to (practice/apply) (skill/practice), but I don’t have the materials that I
need.
G. I don’t think that these things will work with my students.
8. Overall ratings of the usefulness of the professional development

compared with other professional development.
Sample item: Which of the following statements best describes how (insert the name
of the professional development) compares with other professional development in which
you have participated during the past six months (or other period)? (Select one.)
A. This professional development was more useful than other professional
development that I have participated in.
B. This professional development was about the same as other professional
C. This professional development was less useful than other professional
D. I don’t have an opinion.
E. I don’t have an opinion because I haven’t participated in any other professional
development in the last six months.
Afterword
Afterword
E
ffective professional learning NSDC is grateful to M. Bruce Haslam,
requires evaluation for Policy Studies Associates, and Colleen
improvement and accountability. Seremet, assistant state superintendent
Evaluation provides information to for instruction in the Maryland State
strengthen professional learning and Department of Education, and the
provide evidence for accountability. The school district leaders in Maryland
National Staff Development Council who contributed to the development of
emphasizes the importance of evaluation this practical resource guide to help in
in its definition for professional learning the effective evaluation of professional
and its standards. development. In particular, NSDC
NSDC’s definition of professional appreciates their spirit of collaboration
development states: “(v) regularly assesses and willingness to share this resource
the effectiveness of the professional guide with other professional learning
development in achieving identified leaders.
learning goals, improving teaching, As schools and districts strive to
and assisting all students in meeting increase student achievement through
challenging state academic achievement building educators’ capacity, evaluation
standards.” becomes an essential element of the
The Evaluation standard of NSDC’s improvement process. As professional
Standards for Staff Development states, learning moves closer to the classroom,
“Staff development that improves the it is increasingly important for school
learning of all students uses multiple leaders, school improvement teams, and
sources of information to guide teacher leaders to apply the strategies
improvements and demonstrate its outlined in this guide to evaluate
impact” (NSDC, 2001). professional learning’s effect on teaching
Because educators’ experience and and student learning. Central office staff
competence with evaluation is limited, members also are significant partners and
they often are less willing to engage in it. resources to schools in their evaluation
70 Teacher Professional Development Evaluation Guide NSDC www.nsdc.org

Afterword
work. Engaging in evaluation heightens results with professional development

the effectiveness of professional learning leaders. Evaluation elevates the value
and its impact on teaching and student of professional learning and helps to
learning. expand the field and enrich practice so all
This guide and the many resources teachers and students benefit.
identified within it assist educators
in conducting formal and informal Joellen Killion
evaluations of professional learning. Deputy executive director
After conducting evaluations, educators National Staff Development Council
are encouraged to share the evaluation
NSDC 800-727-7288 Teacher Professional Development Evaluation Guide 71

Teacher Professional Development Evaluation Guide

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Teacher Professional Development Evaluation Guide

Hochgeladen von

Copyright:

Verfügbare Formate

Teacher Professional

Prepared for the National Staff

Five questions to inform evaluation planning................................................................................... 12

Evaluation design and data collection strategies............................................................................ 24

Data quality and data analysis...................................................................................................................... 45

Appendix A: Resources for evaluating teacher professional development..................... 61

Appendix B: Sample items for surveying participants’ views of professional

art of evaluating teacher professional the planning framework in the Maryland

“Evaluating professional development enables program managers and participants to

outcomes as these outcomes are suggestions presented here apply

learning and related indicators; and expanded to include involvement in

of teacher professional development. It as they are collected, as well as strategies

Five questions to inform

of state and local strategies to satisfaction instruments and data systems

2 These assurances include: improving educator effectiveness and equitable distribution;

Timeline in months (or some other metric)

National Staff Development Council 800-727-7288

Formative Evaluation Summative Evaluation

Teacher Professional Development Evaluation Guide 15

that identifying teacher learning and II and Outcomes/Indicators can

outcomes and perhaps whether in wanting to know whether

Learn what key stakeholders would stakeholders pose. Pointing out to

Checklist for teacher professional development evaluation plans

Evaluation design and data

including identifying any problems of frequency and duration and

participation in centralized activities and in funds also could result in shortages of

Examining participant perceptions current assignments/responsibilities,

should include items about participation paper-and-pencil instruments. These

of their professional development Evaluations that combine

As a general rule, plan to include whether the first outcome (participants’

Choosing outcomes and indicators

could come from a review of samples of of having resource teacher support in

second option is to draw a sample that to looking at professional development

comparison group should teach students random assignment of teachers and/or

teacher professional development.) In perspectives on their professional

may result in respondents’ resistance instructional materials and

Using student work samples in evaluating teacher professional

times they are expected to observe how evaluation results will be

including concerns from teachers role in an evaluation of professional

of a new practice) in an evaluation collection process and the purpose of

learning. Involving teachers in evaluating their Involving teachers in data collection

collection is through observing peers’ the evaluators. These tasks require

in professional practices, and, in some perceptions. If these are the only

Data quality and data analysis

and how many teachers participated of participants. For example, if data

A second look at evaluation data

availability and arrangement of classroom new instructional practices, can be useful

Extending the causal links to student professional development to changes in

are scores on state assessments. Involving teachers in data analysis

Strategies for involving teachers in student work samples. In this approach,

what is coming and what has been misinterpretation. Few things

2. Explain the evaluation’s purpose interested readers to draw their

How a few words can redefine the evaluation findings

professional development did not achieve 7. A final consideration: One

The resources listed here provide conceptual models, guidance on developing

Administration on Children, teacher professional development have

A companion to the review of S., & Hughes, G. (2002, June). The

Singer, N.R. & Scollay, D. (2007). nwp_file/11082/beyondstrategies.pdf?x-

1. Understanding the purpose of the professional development.

2. Ratings of the usefulness of key components of the professional

3. Perceptions of the extent to which the professional development met

4. Ratings of the alignment of the content of the professional development

5. Perceptions of support and encouragement to participate in the