ME T

project

Feedback
for Better
Teaching
Nine Principles for Using Measures
of Effective Teaching

Partners include representatives of the following institutions and organizations: American Institutes for Research. Educational Testing Service. the Hillsborough County Public Schools. Teachscape.org.ABOUT THIS document: This brief highlights a set of guiding principles from the Bill & Melinda Gates Foundation to inform the design and implementation of high-quality teacher support and evaluation systems based on three years’ of work by the Measures of Effective Teaching (MET) project. Empirical Education. University of Chicago. University of Southern California. its partners. the Denver Public Schools. teachers. Stanford University.metproject. University of Michigan. Cambridge Education. Harvard University. National Math and Science Initiative. University of Virginia. The Danielson Group. RAND. and other leading school systems and organizations. The approximately 3. the Memphis Public Schools. Funding is provided by the Bill & Melinda Gates Foundation. University of Washington. Dartmouth College.000 MET project teachers who volunteered to open up their classrooms for this work are from the following districts: The Charlotte- Mecklenburg Schools. New Teacher Center. and Westat. ABOUT THE MET PROJECT: The MET project is a research partnership of academics. the Dallas Independent Schools. University of Texas. More in-depth discussion of the MET project’s analyses to date may be found in the project’s research reports and non-technical briefs at www. Rutgers University. the New York City Schools. January 2013 . and education organizations committed to investigating better ways to identify and develop effective teaching. National Board for Professional Teaching Standards. and the Pittsburgh Public Schools.

evaluation measures and practices could tions.Creating the Conditions for Success Teachers want to succeed. schools. but they typically lack the conditions for success. They’re on their own to adjust practice to better serve students. What guidance they get often is plagued by vague teaching standards. expressed little faith that traditional tricts. research institu. as disconnected from what they valued Nine Principles for Using Measures of Effective Teaching 1 . unions. its measures improve teacher support. and testing systems that measure only some of the outcomes that educators value for students. The partners in the Measures of Nearly all of the teachers on the MET Effective Teaching (MET) project—a project’s advisory panel similarly group of thoughtful individuals in dis. and technical organizations—told provide usable information to guide us from the outset that current evalu. more effective teaching. They decried ation systems were not being used to evaluation as perfunctory. Teachers generally work in isolation. Success itself remains ill defined. overly numerous and often trivial learning objectives.

Improvement align effort. But good information is hard to produce. fall into of how information can be distorted. and Invest in Improvement. stand and then close the gap between menting feedback and evaluation systems tive. Ensure High-Quality Data. and collaboration ways that traditional evaluation systems with districts. the measures has been the primary goal of measures of teaching effectiveness could right measurement processes. Well-designed evaluation systems It will require care and attention for teaching without good information will continually improve over time. Identifying and validating better and ultimately supported. teacher evaluation measures to serve about actual teaching practice. strong the MET project and a core concern of be valid and reliable. we offer nine that trustworthy measures could inform and the actual teaching occurring in guiding principles based on three years’ improvements in teaching practice in classrooms. as shown When given the right type of attention. the claim that It requires the right measures. and an awareness the districts with which we work. three overarching imperatives. These advisors nevertheless agreed their expectations for effective teaching that support teachers. have not. Measuring for measures can help set expectations and in Figure 1: Measure Effective Teaching. about teaching and learning. To help states test measures of effective teaching so Figure 1 A Framework for Improvement-Focused Teacher Evaluation Systems MEASURE EFFECTIVE TEACHING  Set expectations  Use multiple measures  Balance weights INVEST IN IMPROVEMENT ENSURE HIGH-QUALITY DATA  Make meaningful distinctions  Monitor validity  Prioritize support and feedback  Ensure reliability  Use data for decisions at all levels  Assure accuracy 2 Feedback for Better Teaching . of study. These principles. communications. explained on the following pages. observation. and districts navigate the work of imple- observation practices as highly subjec. Our prior reports tested. Note the cyclical presenta- It is very hard to support effective tion. and its that school systems can clearly under. The both professional development and MET project has sought to build and accountability purposes.

and they have included multiple measures. knowl. likely to be neglected. In each case. they have upheld high standards for data quality. we defined effective measured each facet of effective teaching as sensitivity to students’ teaching. they have emphasized the importance of investing in improvement. and target support. When com- service of student success. It was important that we MET project. are able to provide feedback at all levels of the system—school leadership. They see feedback as the path to better teaching. An unmeasured facet is academic and social needs. Measure Effective or developed measures to reflect Teaching all key aspects of its definition of effective teaching: student surveys ■■ Set Expectations. and behaviors that enable better observation instruments to assess student learning. edge of subject-matter content and pedagogy. content systems is for stakeholders to agree tests to assess teachers’ knowledge on the teacher knowledge. The first step to assess the supportiveness of the in designing teacher evaluation instructional environment. skills. and the ability to put that ■■ Balance Weights. Guiding Principles for Improvement-Focused Teacher Evaluation Systems Our district partners are beginning to build and implement systems for teacher feedback and evaluation. What counts most knowledge into practice. coaching support. and entire system by providing a shared student assessments to measure language to talk about teaching. of their subject and how to teach it. all in the gets the most attention. the MET project sought to indicate meaningful differences continues on page 6 Nine Principles for Using Measures of Effective Teaching 3 . and even central office administration—to align efforts in support of more effective teaching and learning. In the students. while focused on teaching. professional development. 50 percent of the weight to student tifaceted nature of effective teaching. The choice allocate between 33 percent and of measures should reflect the mul. They understand that the measures. achievement measures are sufficient For instance. This benefits the teachers’ classroom practice. set the learning gains of a teacher’s priorities. we have found that approaches that ■■ Use Multiple Measures. bining measures into a single index.

and district on classroom observations.0 4. Ms.0 ➍ Managing student ➌ behavior Difference between actual and predicted 2.0 4. Displayed are results for a MET project teacher (the name is fictional).0 achievement on 2010 state math test Creating an environment of respect & rapport 0 Engaging students in learning Managing classroom -2.0 4. A 6th grade | Valley View Middle School | XYZ School District ➊ Multiple Measures Bar (achievement gains. student perception surveys.0 -2. A’s . The teacher can see her overall results and where her results sit within the systemwide distribution for each measure and individual teaching competency.0 2. contextualized information on their practice for use in professional development.25 0.0 Score on FFT Scale 4 Feedback for Better Teaching . student survey) Equally Weighted Composite State Math Test — Achievement gains Classroom Observation — FFT Student Survey — Tripod Achievement Gains Classroom Observations Middle School Math Scores Score on Danielson Framework for Teaching (FFT) ➋ -0.0 3. Diagnosing Practice with Multiple Measures These pages use MET project data to illustrate how multiple measures can provide teachers with rich.0 0 2.0 3.. her school.5 -0.5 1.0 instruction Student achievement on 2009 state math test Using questioning & discussion techniques Students in Ms.0 procedures Establishing a culture for learning -4.0 Using assessment in -4.25 0 0. Communicating with Classroom School District Actual = Predicted students Achievement 1.. and student achievement gains. 40th observation.0 2.

and Captivate represents the school average. after adjusting for English language learner and free and reduced-price lunch status. The orange box represents the middle 50 percent of all teachers. represents the district average. with red representing Composite percentile: 40th low performance.0 2. teachers at the very high end tend to do well on all of the measures. ➋ & ➌ Box Plots The box plots at level ➋ depict scores for each measure. Distance from the line represents the gap between predicted and actual performance. Scores Composite score: 228 of 500 for the MMC and its individual measures are color-coded to performance standards for each measure.0 The dark blue dot represents The line within the box is the the teacher. Care ➍ Achievement Gains Scatterplot Challenge The scatterplot shows the gap between actual and predicted performance for all district 6th grade students on last year’s state math assessment. Points above the line Consolidate represent higher-than-predicted performance for students with similar characteristics.0 4. on the left to the Score on Tripod Survey 5th percentile and on the right to the 95th percentile. yellow representing average performance. Scores beyond these lines are considered outliers. MMC scores determine placement on the bar from the lowest MMC score on the left to the highest MMC score on the right. Points below the line represent lower- Control than-predicted performance. The center (dashed) line represents Clarify actual performance equal to predicted performance. Above predicted Score on Tripod Scale performance is credited as positive and below predicted performance is debited as negative. The top row is the MMC and the rows below represent the achievement gains for the state math assessment and the average scores for the Framework for Teaching classroom observations and the Tripod student survey. Each column represents a single teacher. indicating a high level of agreement among them. and the opposite is true for those at the very low end.0 student’s performance against predictions. Student Surveys Legend Lines extend from each side of the box.0 3.0 3. Predicted performance is the average performance for students with Confer similar prior scores.0 4. Nine Principles for Using Measures of Effective Teaching 5 .0 5.0 2. ➊ Multiple Measures Bar This bar contains a score for every teacher on each measure within the Multiple Measures Composite (MMC).0 5. A teacher’s value-added score is calculated by averaging each of his or her 1. The box plots at level ➌ depict scores for each component within the student survey and the teacher observation measures. and green representing high performance. In other words. The light blue dot median (middle) teacher. 1. Note that the colors Tier: Satisfactory generally match across the four measures near each end of the bar.

the skill. Because two observ- and effort away from improvement sures are needed. not pass our validity test and was above 0. but higher ■■ Assure Accuracy. Teachers who Measurement of teaching should competencies within an observation demonstrate skills and score high on reflect the quality of teachers’ instrument. two additional full lessons. focus their limited time and atten. School systems the classrooms of teachers with full-lesson observations can increase should do the same. student survey results. continued from page 3 School systems needn’t go to such student survey measures and tests. then new mea- tently wrong. assurance of Overweighting any single measure exercise. and requires rigorous training on how to Ensure High-Quality Data this undermines trust in the system. gains. yet it achieves the equal-sized groups. Accuracy of observations indicates measurement error. higher on a measure. Moreover. ers agree. better the reliability of observation ratings. ■■ Make Meaningful Distinctions. observer. lessons. If group of students. Reliability without observations. Validation is not a one-time for survey questions. we learned that fied with the teacher the names of the found that students learn better in short observations to supplement students in the class. then it ably measure teacher practice. Low reliability correct. If teachers begin to score confidentiality. MET project data classrooms of students to teach. ■■ Ensure Reliability.65 when a principal observes sometimes less than 1 percent. Chief scores and survey responses means is not the best place for teachers to among these is the need to observe crediting them to the right teacher. In addition. This does not reflect ite measure. For ries. and told very few. Assuring accuracy of student test to better student outcomes. lengths. or the instrument as intended before they than teachers with lower scores. balanced teachers’ performance on each mea. administrators observe three partial reality. Indeed. The above scenario is that separates teachers into four erable effort to randomly assign more efficient. differentiate performance across all ■■ Monitor Validity. and prior School systems can use a variety of Invest in Improvement success raising student test scores combinations of observers and les. The MET project are allowed to rate teachers’ practice. and narrow a focus on one measure. does not lead has learned much about how to reli. the consistency weights avoid the risks posed by too sure with their student achievement of the data collection process. better observation scores. same reliability as when a principal suggest that teachers’ effectiveness ers to determine if measures could observes two full lessons and a peer is unlikely to be distributed equally identify effective teachers regardless or another administrator observes among several performance catego- of student assignment—and they did. (whether student achievement. (adjusted for students’ different sons observed to improve reliability. it does not mean they are on the other measures. more than one lesson and include Whenever the MET project collected tion. therefore omitted from our compos- one full lesson and peers or other they were not. of the questions. we found this to be far 6 Feedback for Better Teaching . We have tested the validity of more than one observer for each student data from a classroom it veri- all measures in the MET project and teacher. Many traditional evaluation sys- starting points). but they should compare reliability is a function of the content among teachers. as measured. One measure of For example. but neither does a system The MET project invested consid. It also requires assess- a measure should experience more practice and not the idiosyncrasies ment of observers’ abilities to apply success in helping students learn of a particular lesson. or surveys) invites scores are no longer associated with accuracy amounts to being consis- manipulation and detracts attention desired outcomes. we found that school tems told almost all teachers they content knowledge for teaching did systems could achieve reliability were satisfactory.

support teachers’ improvement bunched at the center of the distribu. This would suggest a large selves teach was one of their most Hillsborough County to focus its middle category of effectiveness with valuable professional development professional development support on two smaller ones at each end. where 50 percent of the teach. among teachers in this vast middle. This led three.4 points of each waste of effort to use measures of fessional development and whether other (on a four-point scale) using teaching only for high-stakes deci. alone.2 participated in the MET project video observation measures indicate percent of teachers scored above a study told us that seeing them. Measures of effective teaching Nine Principles for Using Measures of Effective Teaching 7 . feedback. Rather experiences.from the case. MET project teachers’ ■■ Prioritize Support and Feedback. Only 7. it’s a systems know where to target pro- ers scored within 0. room for improvement—rather than The responsibility for improving efforts would be better spent work. classroom management skills that teaching shouldn’t rest with teachers ing to improve their practice. many of the teachers who to areas of teaching that classroom ers scored below a two. A number of Charlotte Danielson’s Framework for sions. need improvement most. Sound measures help school tion.) Public Schools—have shifted one-quarter point different from the we didn’t study the effectiveness of professional development resources average. Multiple measures provide our partner districts—including the Teaching.5 percent of teach. the supports work. Teachers at the 25th and rich information to help teachers Denver and Hillsborough County 75th percentiles scored less than improve their practice. rigorous instructional techniques— than trying to make fine distinctions for which teachers showed the most ■■ Use Data for Decisions at All Levels. enable school systems to better classroom observation scores were While some teachers’ low perfor. most teachers had clearly mastered. mance will require administrative needs. and only 4. action on behalf of students. Although (Fla.

states and districts should commit to measurement but hold lightly to the specific measures as the field continues to gain new knowledge. Understanding how teachers are performing is an important first step. The Next Phase of Work States and districts have learned a great deal in the last few years about how to create better teacher development and evaluation systems. But there’s still much to learn as these systems are implemented and improved over time and aligned to new expectations for students. 8 Feedback for Better Teaching . As they move forward. But the real work lies ahead: understanding how to use that data to help all teachers improve their practice and the outcomes for America’s young people. One of the most exciting prospects is aligning teacher development and evaluation systems to the Common Core State Standards.

it seeks to ensure that all people— especially those with the fewest resources— have access to the opportunities they need to succeed in school and life.S. the Bill & Melinda Gates Foundation works to help all people lead healthy. Based in Seattle.gatesfoundation. Gates Sr.Bill & Melinda Gates Foundation Guided by the belief that every life has equal value. Washington.org.. All Rights Reserved. In the United States. Bill & Melinda Gates Foundation is a registered trademark in the United States and other countries. For more information on the U. under the direction of Bill and Melinda Gates and Warren Buffett. Program. please visit www. which works primarily to improve high school and postsecondary education. the foundation is led by CEO Jeff Raikes and Co-chair William H. productive lives. it focuses on improving people’s health and giving them the chance to lift themselves out of hunger and extreme poverty. ©2013 Bill & Melinda Gates Foundation. . In developing countries.

gatesfoundation.org .www.

Master your semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master your semester with Scribd & The New York Times

Cancel anytime.