1

Richards Middle School Reading Institute Program Evaluation Georgia Southern University Spring 2010 Betsy Beach Scott Price Casey Manning

2 Table of Contents I. II. III. Focus of the Evaluation IV. V. VI. VII. Appendices pg 16 Recommendations and Conclusions pg 14 Presentation of Results pg 11 Overview of Evaluation Plan and Procedures pg 9 pg 8 Introduction pg 6 Executive Summary pg 3

Executive Summary
The Reading Institute focuses on moving the students who did not meet on the Reading CRCT to the meets category. Although Richards Middle School made Adequate Yearly Progress last year, the school did not make AYP in years prior. Meeting AYP is crucial for Richards Middle this year as it will ensure that the school does not receive N12 status. In years past, Richards Middle has missed AYP by very few percentage points; every single child that is currently enrolled in Reading Institute has a direct impact on Richard Middles ability to make or not make AYP. Does the Richards Middle Reading Institute effectively increase student fluency, comprehension, and vocabulary? Using weekly fluency tests, coach connected post tests, and domain specific vocabulary acquisition and other standardized tests administration should be able to monitor the effectiveness of the Reading Institute. With written reports to administrators and oral discussions about results, interpretations about the program and possible changes can be made to ensure positive results. What are the overall perceptions of the Richards Middle Reading Institute and is there sufficient support for the program? The program has been well funded and supported by school administration. Teachers were not forced to participate in the program; all feel strongly enough about the program to give up their planning periods and attend a great deal of professional development. The parents and students are also in consensus. Students choose to attend the Reading Institute and parents must agree to have their child participate. The students choose to give up a more exciting connections class to attend the Reading Institute.

4 What are the primary benefits of the program experienced by the students? The students will gain confidence in their reading ability and pass the Reading and ELA CRCT. Hopefully, setting them up for future success in the classroom and continued growth in their reading ability. To what extent has the program been successful in meeting AYP for Richards Middle? Every child that is currently enrolled in Reading Institute has a direct impact on Richard Middles ability to make or not make AYP. Since the program is still in progress, and the results of the CRCT are not yet available, the outcome has not been determined. Evaluation Results Fluency Data Baseline Fluency Reading 96 Final Fluency Reading 139.76 Comprehension Data Coach Connected Coach Pretest Score Connected Post Test Score 60 77 Percentage Increase On or Above Grade Level Before (Y/N) 2 Y, 28 N On or Above Grade Level After (Y/N) 24 Y, 6 N Percentage Increase 46.71 On or Above Grade Level (Y/N) 26 Y, 4 N

Average

Average

36.81

5 Recommendations Hold a roundtable discussion with instructors to determine their needs. Offer a variety of professional development courses for instructors. Target the lowest students in each area, creating specific learning activities for those students. Utilize flexible grouping to continuously move students based on weekly fluency scores. Create and implement a mid-institute test to monitor comprehension.

Introduction
The Richards Middle Reading Institute program is designed to increase student reading fluency, comprehension and vocabulary. The program has approximately 70 students enrolled and focuses on moving the students form the did not meet status on the Reading CRCT to the meets category. This will help Richards Middle to meet AYP for this year. Purpose The evaluation was designed to determine if the Middle Reading Institute is effectively increasing student reading fluency, comprehension and vocabulary. The principle goal of this report is to enlighten on the following questions: Does the Richards Middle Institute effectively increase student fluency? Does the Richards Middle Institute effectively increase student reading comprehension? Does the Richards Middle Institute effectively increase student vocabulary?

Setting up a model created at the beginning of the year, the evaluation will conclude with results of either success or failure of the program. Audience The evaluation will involve the stakeholders; school administrators, program recipients, and program coordinators. The administration directly will oversee the RMRI and make the final

7 decision on the future of the program. The teachers along with the administration, are the creators of the program, and will determine the criteria and procedures. The students and parents will be directly affected by the outcome of the evaluation. Limitations The program relies upon students actively participating and applying themselves. Students that choose to drop out of the program may not see improved reading skills or improvement on their CRCT scores. Parents and teachers that do not invest the necessary time and devotion to the program can also adversely affect the overall results of the program. Overview The following sections are included in this report: Focus of the evaluation- which includes a description of the Richards Middle Reading Institute and the questions used for the study Brief overview of evaluation plan and procedures on how the data was collected and the methods, criteria used to interpret data Presentation on evaluation result Conclusions and recommendations

Focus of the Evaluation

The Richards Middle Reading Institution was implemented in January 2010 to help students improve their reading skills and to ensure that Richards Middle School makes Adequate Yearly Progress this year. The program consists of students that missed the meets category on the CRCT by no more than 15 points. Progress of the students will be judged by periodic testing. Parents, teachers and school administration are all in support of this program. In order to inform on the following questions, students enrolled in the Reading Institute program were given fluency tests, standardized tests and post tests. Does the Richards Middle Reading Institute effectively increase student fluency? Does the Richard Middle Reading Institute effectives increase student reading comprehension? Does the Richards Middle Reading Institute effectively increase student vocabulary?

The data was then collected from these test results along with descriptive statistics to determine the effectiveness of the program. For additional details refer to the Appendix.

Overview of Evaluation Plan and Procedures

Question 1: Does the Richards Middle Reading Institute effectively increase student fluency? A. Information Required a. CRCT reading scores b. Weekly fluency tests B. Information Source a. Students who are in the Richards Middle School Reading Institute (program recipients) C. Information Collection Arrangements a. Information collected by RMRI instructors b. Information collected weekly since January 5th c. Information collected in a one on one situation D. Analysis Procedures a. Descriptive statistics E. Criteria a. Average fluency score for 7th grade: 130-170 b. Level 2 CRCT: 800 F. Reporting of Information a. To whom: school administrators, program recipients, and program coordinators b. Context: Help answer the question: How effective is the RMRI? c. How: Written report to administratiors with oral discussion to discuss results, their interpretations, and possible changes d. When: May Question 2: Does the Richards Middle Reading Institute effectively increase student reading comprehension? A. Information Required a. CRCT reading scores b. Coach Connected Pre and Post tests

10 B. Information Source a. Students who are in the Richards Middle School Reading Institute (program recipients) C. Information Collection Arrangements a. Information collected by RMRI instructors b. Pretest collected January 5th c. Posttest collected April 15th D. Analysis Procedures a. Descriptive statistics E. Criteria a. A score of 75 or above (Level 2) for those who scored a level 1 in 6th grade. A score of 85 or above (level 3) for those who scored a level 2 in 6th grade. b. Level 2 CRCT: 800 F. Reporting of Information a. To whom: school administrators, program recipients, and program coordinators b. Context: Help answer the question: How effective is the RMRI? c. How: Written report to administrators with oral discussion to discuss results, their interpretations, and possible changes d. When: May

11

Presentation of Evaluation Results

At the beginning of the Reading Institute, students took a fluency test and a Coach Connected online computer test to serve as baseline data. Throughout the Institute, students took one fluency test per week, and students worked through the Coach Connected prescribed programs based on their pretest. At the end of the Reading Institute, students took a Coach Connected post test. Question 1: Does the Richards Middle Reading Institute effectively increase student fluency? Baseline Fluency Reading 96 Final Fluency Reading 139.76 Percentage Increase 46.71 On or Above Grade Level (Y/N) 26 Y, 4 N

Average

The seventh grade fluency range is between 130 and 160. The average fluency score for all students at the beginning of the Reading Institute was 96. The average fluency score for all students at the end of the Reading Institute was 139.76 The average percentage increase was 46.71%. At the beginning of the Reading Institute, 0 students had a baseline fluency score on grade level. At the end of the Reading Institute, 26 students had a fluency score on grade level.

Question 2: Does the Richards Middle Reading Institute effectively increase student reading comprehension? Coach Connected Coach Percentage On or Above On or Above Pretest Score Connected Post Increase Grade Level Grade Level Test Score Before (Y/N) After (Y/N) Average 60 77 36.81 2 Y, 28 N 24 Y, 6 N

12 Stakeholder Perceptions Student Participants At the beginning of the program, students (along with teachers) filled out goal sheets. Students made goals for fluency scores, Coach Connected scores, and grades in each class. Each week, students filled out Fluency Checklists at the end of their fluency tests. At the end of the program, students filled out comment sheets. At the end of the program, 96% of student participants believe that what they learned in this program will be valuable for [their] future. At the end of the program, 96% of student participants believe that the instructors cared about the progress of their students. At the end of the program, 100% of student participants were satisfied with the learning experiences in this program. Instructor Participants At the end of the program, instructors filled out evaluation surveys and left comments. A 75% is considered on grade level for the Coach Connected test. The average Coach Connected Pretest score was a 60. The average Coach Connected Posttest score was a 77. The average percentage increase was 36.81%. Before the Reading Institute began, 2 students scored on grade level, and 28 students score below grade level. After the Reading Institute, 24 students scored on grade level, and 6 students scored below grade level.

13 At the end of the program, 100% of instructors believe that they had adequate time to work with [their] students. At the end of the program, 96% of instructors believe that their students made progress throughout the course of this program At the end of the program, 100% of instructors believe that this program should be continued and/or expanded into next year. 50 % of instructors believe that the facilities were of good quality. 64% of instructors believe that the professional development provided [them] with the information [they] needed to adequately teach reading comprehension and fluency.

14

Recommendations and Conclusions

Based upon the data collected from students and instructors, this evaluation study concludes: Program Strengths At the end of the program, 87% of students were reading on grade level. Student fluency increased an average of 46% throughout the course of the program. At the end of the program, 80% of students were on grade level in the area of reading comprehension. Student comprehension increased an average of 37% throughout the course of the program. 100% of students were satisfied with their experience in the program. 96% of instructors believed their students made progress throughout the course of the program. Program Limitations At the end of the program, 4 students were still below grade level in the area of fluency. At the end of the program, 6 students were still below grade level in the area of reading comprehension. Half of the instructors believed the facilities were not of good quality.

15 34% of the instructors believed their professional development did not aid in their ability to teach reading comprehension and fluency. Recommendations Hold a roundtable discussion with instructors to determine their needs. Offer a variety of professional development courses for instructors. Target the lowest students in each area, creating specific learning activities for those students. Utilize flexible grouping to continuously move students based on weekly fluency scores. Create and implement a mid-institute test to monitor comprehension. Based on the aforementioned data, it is the recommendation of the evaluation study that the positive impact of the Reading Institute on its participants fluency levels and reading comprehension scores justifies its continuation into the future. And overwhelming majority of the students have progressed to reading on grade level from the beginning to the current date of the Reading Institute. In order to improve this program and ensure that all targeted students are reading on grade level, it is our recommendation that the administrators of the program hold detailed conversations with instructors to determine their professional needs and needs for resources. The evaluation study also recommends that the instructors institute flexible grouping to move the students on a weekly basis, determined by their fluency scores. Although there are weekly fluency tests, there are no weekly assessments for monitoring comprehension. It is the recommendation of the study that the Reading Institute creates a comprehension tests to monitor comprehension more frequently throughout the course of the program.

16

Appendices
I. Student fluency data Student Number 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Average Baseline Fluency Reading 80 94 115 104 89 96 92 104 109 76 97 92 80 90 95 115 104 113 102 86 89 94 91 99 92 85 86 94 97 120 96 Final Fluency Reading 130 132 169 137 138 145 125 132 141 128 142 129 140 134 149 153 138 143 157 140 136 136 123 148 139 138 133 139 149 150 139.76 Increase Percentage Increase 62.50 Y 40.43 46.96 31.73 55.06 51.04 35.87 26.92 29.36 68.42 46.39 40.22 75.00 48.89 56.84 33.04 32.69 26.55 53.92 62.79 52.81 44.68 35.16 49.49 51.09 62.35 54.65 47.87 53.61 25.00 46.71 Y Y Y Y Y N Y Y N Y N Y Y Y Y Y Y Y Y Y Y N Y Y Y Y Y Y Y 26 Y, 4 N On or Above Grade Level (Y/N)

50 38 54 33 49 49 33 28 32 52 45 37 60 44 54 38 34 30 55 54 47 42 32 49 47 53 47 45 52 30 43.76

17 II. Student Comprehension Data Student Number Coach Connected Pretest Score Coach Connected Post Test Score Increase Percentage Increase On or Above Grade Level Before (Y/N) Y Y N N Y Y Y Y Y Y Y Y Y Y Y Y N N Y Y Y N Y Y Y N Y Y Y Y 24 Y, 6 N N N N N Y N Y N N N N N N N N N N N N N N N N N N N N N N On or Above Grade Level After (Y/N)

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Average

30 36 52 57 49 74 28 74 68 64 59 63 65 69 73 42 62 64 68 71 67 49 59 54 58 64 68 62 67 73 60

76 79 67 69 84 74 78 74 84 89 76 84 81 79 74 78 63 69 75 79 75 69 75 79 80 73 81 78 82 76 77

46 43 15 12 35 0 50 0 16 25 17 21 16 10 1 36 1 5 7 8 8 20 16 25 22 9 13 16 15 3 17

153.33 N 119.44 28.85 21.05 71.43 0.00 178.57 0.00 23.53 39.06 28.81 33.33 24.62 14.49 1.37 85.71 1.61 7.81 10.29 11.27 11.94 40.82 27.12 46.30 37.93 14.06 19.12 25.81 22.39 4.11

36.81 2 Y, 28 N

18 III. Fluency Checklist Fluency Checklist

Did I take my time and read like Im having a conversation?

Yes

No

Did I pay attention to punctuation and pause at the correct places?

Did I use appropriate expression?

Did I read smoothly and clearly?

Did I do my best to say each word correctly?

Did I self-correct any mistakes?

19 IV. Student Comment Survey End of Program Evaluation/Comments

1. The instructors were available for consultation and advice when I needed to speak with them. 2. The instructors in the program inspired me to do my best. 3. The instructors in the program gave me helpful feedback on my work. 4. The instructors in the program had thorough knowledge of the content of the courses they taught. 5. The instructors were enthusiastic about the program. 6. The instructors cared about the progress of their students. 7. What I have learned in this program will be valuable for my future. 8. Overall I was satisfied with the quality of my learning experiences at this institution.

Yes

No

20 V. Instructor Survey End of Program Evaluation/Comments

1. All facilities (computers, classrooms, resources) were of good quality. 2. Adequate facilities (computers, classrooms, resources) were available to use. 3. I had adequate time to work with my students during the Institute. 4. Administration was willing to work with instructors to obtain resources, facilities and appropriate time. 5. My professional development provided me with the information I needed to adequately teach reading comprehension and fluency. 6. I believe that my students made progress throughout the course of this program. 7. I believe that this program should be continued and/or expanded into next year. 8. Overall I was satisfied with facilities, resources, time, and the presentation of the program as a whole.

Yes

No