MET_Feedback-for-Better-Teaching_Principles-Paper.pdf | Survey Methodology | Teachers



for Better
Nine Principles for Using Measures
of Effective Teaching

Partners include representatives of the following institutions and organizations: American Institutes for Research. University of Chicago.metproject. the New York City Schools. More in-depth discussion of the MET project’s analyses to date may be found in the project’s research reports and non-technical briefs at Educational Testing Service. its partners. Dartmouth College. the Denver Public Schools. Empirical Education. teachers. Stanford University. and the Pittsburgh Public Schools. and Westat. and education organizations committed to investigating better ways to identify and develop effective teaching. RAND. University of Washington. University of Texas. Teachscape.000 MET project teachers who volunteered to open up their classrooms for this work are from the following districts: The Charlotte- Mecklenburg Schools.ABOUT THIS document: This brief highlights a set of guiding principles from the Bill & Melinda Gates Foundation to inform the design and implementation of high-quality teacher support and evaluation systems based on three years’ of work by the Measures of Effective Teaching (MET) project. Funding is provided by the Bill & Melinda Gates Foundation. The Danielson Group. University of Virginia. Cambridge Education. the Hillsborough County Public Schools. University of Michigan. the Memphis Public Schools. Harvard University. ABOUT THE MET PROJECT: The MET project is a research partnership of academics. New Teacher Center. University of Southern California. The approximately 3. National Math and Science Initiative. National Board for Professional Teaching Standards. Rutgers University. and other leading school systems and organizations. January 2013 . the Dallas Independent Schools.

Creating the Conditions for Success Teachers want to succeed. more effective teaching. overly numerous and often trivial learning objectives. They decried ation systems were not being used to evaluation as perfunctory. evaluation measures and practices could tions. and testing systems that measure only some of the outcomes that educators value for students. expressed little faith that traditional tricts. research institu. as disconnected from what they valued Nine Principles for Using Measures of Effective Teaching 1 . and technical organizations—told provide usable information to guide us from the outset that current evalu. They’re on their own to adjust practice to better serve students. What guidance they get often is plagued by vague teaching standards. schools. The partners in the Measures of Nearly all of the teachers on the MET Effective Teaching (MET) project—a project’s advisory panel similarly group of thoughtful individuals in dis. unions. Teachers generally work in isolation. its measures improve teacher support. Success itself remains ill defined. but they typically lack the conditions for success.

Improvement align effort. These principles. Identifying and validating better and ultimately supported. fall into of how information can be distorted. Note the cyclical presenta- It is very hard to support effective tion. we offer nine that trustworthy measures could inform and the actual teaching occurring in guiding principles based on three years’ improvements in teaching practice in classrooms. of study. three overarching imperatives. explained on the following pages. the claim that It requires the right measures. the measures has been the primary goal of measures of teaching effectiveness could right measurement processes. as shown When given the right type of attention. and an awareness the districts with which we work. The both professional development and MET project has sought to build and accountability purposes. strong the MET project and a core concern of be valid and reliable. have not. Measuring for measures can help set expectations and in Figure 1: Measure Effective Teaching. These advisors nevertheless agreed their expectations for effective teaching that support teachers. and its that school systems can clearly under. Ensure High-Quality Data. communications. stand and then close the gap between menting feedback and evaluation systems tive. and collaboration ways that traditional evaluation systems with districts. Our prior reports tested. observation. and districts navigate the work of imple- observation practices as highly subjec. Well-designed evaluation systems It will require care and attention for teaching without good information will continually improve over time. But good information is hard to produce. and Invest in Improvement. teacher evaluation measures to serve about actual teaching practice. To help states test measures of effective teaching so Figure 1 A Framework for Improvement-Focused Teacher Evaluation Systems MEASURE EFFECTIVE TEACHING  Set expectations  Use multiple measures  Balance weights INVEST IN IMPROVEMENT ENSURE HIGH-QUALITY DATA  Make meaningful distinctions  Monitor validity  Prioritize support and feedback  Ensure reliability  Use data for decisions at all levels  Assure accuracy 2 Feedback for Better Teaching . about teaching and learning.

while focused on teaching. are able to provide feedback at all levels of the system—school leadership. of their subject and how to teach it. content systems is for stakeholders to agree tests to assess teachers’ knowledge on the teacher knowledge. coaching support. An unmeasured facet is academic and social needs. Guiding Principles for Improvement-Focused Teacher Evaluation Systems Our district partners are beginning to build and implement systems for teacher feedback and evaluation. all in the gets the most attention. They see feedback as the path to better teaching. and they have included multiple measures. they have upheld high standards for data quality. and the ability to put that ■■ Balance Weights. achievement measures are sufficient For instance. the MET project sought to indicate meaningful differences continues on page 6 Nine Principles for Using Measures of Effective Teaching 3 . In each case. edge of subject-matter content and pedagogy. skills. The first step to assess the supportiveness of the in designing teacher evaluation instructional environment. In the students. It was important that we MET project. professional development. bining measures into a single index. and target support. knowl. What counts most knowledge into practice. we have found that approaches that ■■ Use Multiple Measures. likely to be neglected. 50 percent of the weight to student tifaceted nature of effective teaching. we defined effective measured each facet of effective teaching as sensitivity to students’ teaching. set the learning gains of a teacher’s priorities. and behaviors that enable better observation instruments to assess student learning. The choice allocate between 33 percent and of measures should reflect the mul. and entire system by providing a shared student assessments to measure language to talk about teaching. They understand that the measures. they have emphasized the importance of investing in improvement. This benefits the teachers’ classroom practice. When com- service of student success. Measure Effective or developed measures to reflect Teaching all key aspects of its definition of effective teaching: student surveys ■■ Set Expectations. and even central office administration—to align efforts in support of more effective teaching and learning.

A’s .5 1.0 ➍ Managing student ➌ behavior Difference between actual and predicted 2.25 0 0.0 4.0 instruction Student achievement on 2009 state math test Using questioning & discussion techniques Students in Ms.25 0. The teacher can see her overall results and where her results sit within the systemwide distribution for each measure and individual teaching competency. Diagnosing Practice with Multiple Measures These pages use MET project data to illustrate how multiple measures can provide teachers with rich.0 4.. and student achievement gains.0 2.0 3. Displayed are results for a MET project teacher (the name is fictional). Communicating with Classroom School District Actual = Predicted students Achievement 1.0 0 2. her school.0 -2. student survey) Equally Weighted Composite State Math Test — Achievement gains Classroom Observation — FFT Student Survey — Tripod Achievement Gains Classroom Observations Middle School Math Scores Score on Danielson Framework for Teaching (FFT) ➋ -0.0 achievement on 2010 state math test Creating an environment of respect & rapport 0 Engaging students in learning Managing classroom -2..5 -0. student perception surveys. 40th observation. A 6th grade | Valley View Middle School | XYZ School District ➊ Multiple Measures Bar (achievement gains. and district on classroom observations. contextualized information on their practice for use in professional development.0 procedures Establishing a culture for learning -4. Ms.0 Using assessment in -4.0 3.0 2.0 4.0 Score on FFT Scale 4 Feedback for Better Teaching .

0 3. Above predicted Score on Tripod Scale performance is credited as positive and below predicted performance is debited as negative. Predicted performance is the average performance for students with Confer similar prior scores.0 2.0 5. with red representing Composite percentile: 40th low performance. Care ➍ Achievement Gains Scatterplot Challenge The scatterplot shows the gap between actual and predicted performance for all district 6th grade students on last year’s state math assessment. on the left to the Score on Tripod Survey 5th percentile and on the right to the 95th percentile.0 5. Points above the line Consolidate represent higher-than-predicted performance for students with similar characteristics. The top row is the MMC and the rows below represent the achievement gains for the state math assessment and the average scores for the Framework for Teaching classroom observations and the Tripod student survey. MMC scores determine placement on the bar from the lowest MMC score on the left to the highest MMC score on the right.0 4. teachers at the very high end tend to do well on all of the measures. Each column represents a single teacher. ➋ & ➌ Box Plots The box plots at level ➋ depict scores for each measure. and green representing high performance. after adjusting for English language learner and free and reduced-price lunch status. Distance from the line represents the gap between predicted and actual performance. Scores Composite score: 228 of 500 for the MMC and its individual measures are color-coded to performance standards for each measure. A teacher’s value-added score is calculated by averaging each of his or her 1. yellow representing average performance.0 student’s performance against predictions.0 2. and the opposite is true for those at the very low end. 1. In other words. The box plots at level ➌ depict scores for each component within the student survey and the teacher observation measures. Scores beyond these lines are considered outliers. The center (dashed) line represents Clarify actual performance equal to predicted performance.0 3.0 4. Nine Principles for Using Measures of Effective Teaching 5 . Student Surveys Legend Lines extend from each side of the box. The light blue dot median (middle) teacher. Points below the line represent lower- Control than-predicted performance. ➊ Multiple Measures Bar This bar contains a score for every teacher on each measure within the Multiple Measures Composite (MMC). The orange box represents the middle 50 percent of all teachers. and Captivate represents the school average.0 The dark blue dot represents The line within the box is the the teacher. indicating a high level of agreement among them. represents the district average. Note that the colors Tier: Satisfactory generally match across the four measures near each end of the bar.

lessons. does not lead has learned much about how to reli. we found that school tems told almost all teachers they content knowledge for teaching did systems could achieve reliability were satisfactory. not pass our validity test and was above 0. student survey results. MET project data classrooms of students to teach. Teachers who Measurement of teaching should competencies within an observation demonstrate skills and score high on reflect the quality of teachers’ instrument. therefore omitted from our compos- one full lesson and peers or other they were not. Validation is not a one-time for survey questions. or the instrument as intended before they than teachers with lower scores. observer. better the reliability of observation ratings. Chief scores and survey responses means is not the best place for teachers to among these is the need to observe crediting them to the right teacher. ■■ Make Meaningful Distinctions. and narrow a focus on one measure. as measured. (adjusted for students’ different sons observed to improve reliability.65 when a principal observes sometimes less than 1 percent. This does not reflect ite measure. Assuring accuracy of student test to better student outcomes. better observation scores. We have tested the validity of more than one observer for each student data from a classroom it veri- all measures in the MET project and teacher. Moreover. Reliability without observations. ers agree. assurance of Overweighting any single measure exercise. of the questions. or surveys) invites scores are no longer associated with accuracy amounts to being consis- manipulation and detracts attention desired outcomes. differentiate performance across all ■■ Monitor Validity. gains. but neither does a system The MET project invested consid. If group of students. yet it achieves the equal-sized groups. then new mea- tently wrong. it does not mean they are on the other measures. For ries. Because two observ- and effort away from improvement sures are needed. In addition. If teachers begin to score confidentiality. more than one lesson and include Whenever the MET project collected tion. It also requires assess- a measure should experience more practice and not the idiosyncrasies ment of observers’ abilities to apply success in helping students learn of a particular lesson. Low reliability correct. focus their limited time and atten. but higher ■■ Assure Accuracy. we found this to be far 6 Feedback for Better Teaching . administrators observe three partial reality. School systems the classrooms of teachers with full-lesson observations can increase should do the same. The above scenario is that separates teachers into four erable effort to randomly assign more efficient. the skill. the consistency weights avoid the risks posed by too sure with their student achievement of the data collection process. but they should compare reliability is a function of the content among teachers. then it ably measure teacher practice. Accuracy of observations indicates measurement error. two additional full lessons. higher on a measure. Indeed. ■■ Ensure Reliability. and told very few. we learned that fied with the teacher the names of the found that students learn better in short observations to supplement students in the class. and requires rigorous training on how to Ensure High-Quality Data this undermines trust in the system. same reliability as when a principal suggest that teachers’ effectiveness ers to determine if measures could observes two full lessons and a peer is unlikely to be distributed equally identify effective teachers regardless or another administrator observes among several performance catego- of student assignment—and they did. (whether student achievement. The MET project are allowed to rate teachers’ practice. continued from page 3 School systems needn’t go to such student survey measures and tests. Many traditional evaluation sys- starting points). lengths. balanced teachers’ performance on each mea. One measure of For example. and prior School systems can use a variety of Invest in Improvement success raising student test scores combinations of observers and les.

room for improvement—rather than The responsibility for improving efforts would be better spent work. Rather experiences. the supports work.2 participated in the MET project video observation measures indicate percent of teachers scored above a study told us that seeing them. MET project teachers’ ■■ Prioritize Support and Feedback. many of the teachers who to areas of teaching that classroom ers scored below a two. mance will require administrative needs. Measures of effective teaching Nine Principles for Using Measures of Effective Teaching 7 .from the case. Although (Fla. rigorous instructional techniques— than trying to make fine distinctions for which teachers showed the most ■■ Use Data for Decisions at All Levels. among teachers in this vast middle. This led three. This would suggest a large selves teach was one of their most Hillsborough County to focus its middle category of effectiveness with valuable professional development professional development support on two smaller ones at each end.5 percent of teach. A number of Charlotte Danielson’s Framework for sions. where 50 percent of the teach. it’s a systems know where to target pro- ers scored within 0. Only 7.) Public Schools—have shifted one-quarter point different from the we didn’t study the effectiveness of professional development resources average.4 points of each waste of effort to use measures of fessional development and whether other (on a four-point scale) using teaching only for high-stakes deci. Teachers at the 25th and rich information to help teachers Denver and Hillsborough County 75th percentiles scored less than improve their practice. Multiple measures provide our partner districts—including the Teaching. most teachers had clearly mastered. enable school systems to better classroom observation scores were While some teachers’ low perfor. support teachers’ improvement bunched at the center of the distribu. action on behalf of students. alone. classroom management skills that teaching shouldn’t rest with teachers ing to improve their practice. Sound measures help school tion. feedback. and only 4. need improvement most.

states and districts should commit to measurement but hold lightly to the specific measures as the field continues to gain new knowledge. But there’s still much to learn as these systems are implemented and improved over time and aligned to new expectations for students. The Next Phase of Work States and districts have learned a great deal in the last few years about how to create better teacher development and evaluation systems. One of the most exciting prospects is aligning teacher development and evaluation systems to the Common Core State Standards. But the real work lies ahead: understanding how to use that data to help all teachers improve their practice and the outcomes for America’s young people. 8 Feedback for Better Teaching . Understanding how teachers are performing is an important first step. As they move forward.

the Bill & Melinda Gates Foundation works to help all people lead healthy. please visit www. In developing countries. In the United States. it seeks to ensure that all people— especially those with the fewest resources— have access to the opportunities they need to succeed in school and life. Bill & Melinda Gates Foundation is a registered trademark in the United States and other productive lives. Washington.S.gatesfoundation. All Rights Reserved.Bill & Melinda Gates Foundation Guided by the belief that every life has equal value. . Based in Seattle. Gates Sr. which works primarily to improve high school and postsecondary education. Program. it focuses on improving people’s health and giving them the chance to lift themselves out of hunger and extreme poverty. under the direction of Bill and Melinda Gates and Warren Buffett. For more information on the U. the foundation is led by CEO Jeff Raikes and Co-chair William H.. ©2013 Bill & Melinda Gates Foundation. .

Sign up to vote on this title
UsefulNot useful

Master Your Semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master Your Semester with a Special Offer from Scribd & The New York Times

Cancel anytime.