Formative Evaluation Report of TenMarks Math

Submitted to SSSD Math Faculty

By Matthew Long
12/6/2011

Abstract TenMarks Math is a program touted for its ability to provide personalized math learning that adapts to each students needs. Students work on teacher assigned concepts while having the ability to view help options such as hints and videos related to the problem they are completing. Student performance is reported in real-time which allows the classroom teacher to assign review concepts for struggling learners while providing enrichment or advanced concepts to students who are successful. TenMarks Math was used four times during a two week evaluation period. Students took attitude surveys both prior to and after completing use of the program. The survey questions pertained to math and the program in general. Students also filled out data sheets each day they used the program and program staff kept track of program implementation through written observations. TenMarks Math was determined to have a positive overall effect on students attitudes towards math. Students enjoyed using the program and recommended its use several times a week in class as a supplement to in-class instruction. This may stem from the students just enjoying being able to use the laptops in math class. If a teacher does follow the recommendation students attitudes towards program use should be monitored for burnout. Due to the multiple help options available within the program, TenMarks Math offers a solid alternative to homework problems from the textbook. Homework problems serve as a source of frustration for the majority of students and the options provided by the program may help to ease some of that frustration. Program staff deemed the program easy to use for teachers and helpful to differentiate instruction for the whole classroom. This evaluation used the free version of TenMarks Math so it is recommended for others to try out the premium version to determine if the benefits are worth a monetary investment by the district. Concerns related to the depth of student performance information collected by the program were discussed, but could be addressed by future updates from the company or are possibly already addressed in the premium version of the program. Another recommendation for future program use includes studying the effect of the program on student learning. This could be used in conjunction with 4Sight Testing to identify students in need of remediation/intervention. There are also several programs such as Khan Academy and Manga High that are similar to TenMarks Math that should be examined. Evaluations of those programs in a similar manner to this would provide beneficial comparison data. This would allow the SSSD math faculty to make an informed decision about the best choice for a differentiated math instruction tool to use with our students.

Introduction A common request administrators make to faculty is for the teachers to implement differentiated instruction (DI) in their classrooms. Teachers tend to regard DI as a noble yet impractical instructional theory whose implementation requires exorbitant amounts of time and additional stress for the teacher. TenMarks Math presents itself as a vehicle for DI that helps math teachers avoid the struggles associated with carrying out DI. The web-based math program provides teachers with real-time insight into student performance that allows the teacher to target each individual students needs with engaging, standards-based math exercises. The purpose of this evaluation was threefold. First, TenMarks Math does not provide a proposed best use for the program in terms of method of instruction; e.g. instead of direct instruction for new concepts, as a supplement to direct instruction, etc. This evaluation sought to determine which use TenMarks Math is best suited for. The evaluation also looked at whether the use of the program had an effect on students attitudes towards mathematics and math homework. Finally, program implementation was examined to provide recommendations for future use. This document is a formative evaluation report of the first implementation of the TenMarks Math program at Shanksville-Stonycreek School District. This report contains a description of the program, a description of the evaluation procedures, a presentation of evaluation results, and recommendations for the SSSD Math faculty. Description of TenMarks Math TenMarks Math was developed to be used with any mathematics student in 2nd grade through high school. The program is designed for use with entire classes as it can accommodate the wide variety of needs typically found in heterogeneous ability settings. The program does not provide a specified length for its use as usage can vary depending on both teacher and student needs. The program can be used to practice only a single concept or as practice for the entire curriculum. Based on observations doing during the evaluation, on average a student completes 10 problems in 15-20 minutes if they are working on an unfamiliar concept. This time includes the student using the help options that the program provides. TenMarks Math is best suited for a 1:1 computer setting so that each student can work on problems geared towards their own needs. Because it is a web-based program, an internet connection is needed. Paper and pencil are not required but are highly recommended so that students can work out the practice questions on paper instead of doing all the computations mentally. The objectives of the program are related to trying to resolve common criticisms of the mathematics curriculum and style of instruction. The objectives examined in this evaluation are To alleviate student frustration with concepts theyre having trouble with by providing help options. To change students attitudes towards mathematics in a positive manner. To provide a differentiated instruction vehicle that is teacher-friendly and useful.

TenMarks Math offers two different versions, a free version that was used for the evaluation and a Premium version that includes several features that will be discussed later. TenMarks Math begins with teacher registration. Afterwards, teachers create accounts for their students and assign concepts for their students to work on. The teacher can choose to assign either a single concept or multiple concepts to whichever students they wish. The students see these assignments when they login to the program. All of the practice problems are multiple choice and each problem contains directions. Some problems also involve graphics such as tables or diagrams. Each problem is interactive in the sense that students can view hints and watch video lessons pertaining to the concept they are practicing. This interaction is intended to foster student engagement and lessen student frustration. After each problem is submitted, TenMarks Math provides instant feedback as well as a link to view how to correctly answer the problem if the students answer was incorrect. A student may switch to practice a different concept at any time and while exiting a concept the program displays a report on their work. After the completion of several concepts, students can unlock badges, certificates, and games which provide another source of engagement and motivation. TenMarks Math keeps real-time performance records of each students work. This allows teachers to view reports on both a class and student level. This capability helps teachers identify each students needs and take appropriate action. For example if a student is struggling with adding unlike fractions, the teacher could assign that student practice problems involving finding the least common multiple. Other students who demonstrate proficiency with adding unlike fractions might be given practice problems relating the concept to a new situation such as adding mixed numbers. The curriculum is differentiated in this manner. Evaluation Method The participants in the evaluation consisted of 14 6th grade students in a mixed ability Math class. The group was comprised of 5 boys and 9 girls. All of the subjects were Caucasian and ranged in age from 11-12 years old. None of the students had an Individualized Education Plan. The students were selected due to the wide range of ability levels as determined by their first marking period grades. The only program staff member was Mr. Long who also served as the evaluator. This resulted in the evaluation lending itself to a transaction goal-based model. Mr. Long is currently in his third year of teaching Math at SSSD and is also currently in the Master of Educational Technology program at Boise State University. The program was used in two different manners during the evaluation period. The original intent for the evaluation included looking at the effectiveness of TenMarks Math on student learning as measured by a change in 4Sight Test scores. Thus, the first time the students used the program they worked on concepts related to problems they had missed on the 4Sight Test. Prior to using the program the students did take a pre-usage survey on their attitudes towards mathematics. Two days were originally scheduled for working on these concepts however on the second day the program website was down for maintenance. Mr. Long was also not there to supervise use of the program that day so the students worked independently under the supervision of a substitute

teacher on the day that the program was available for use. Each day, the students were responsible for getting out their laptops and logging into their TenMarks Math account before the period began. The students then had a one week break from using the program. The evaluations purpose shifted to examining program implementation and the best uses for the program. Students used the program for three consecutive days with each day involving a different purpose for use. The first day involved students identifying concepts they had struggled with, by looking at questions missed on previous quizzes and tests. They then found the corresponding concept within the programs curriculum and worked on correcting their flawed understanding. Some assistance from the program staff was needed to help students find the concepts within the program. The second day consisted of working on concepts that the students had received instruction on the previous week. These concepts were assigned by the teacher with the purpose of reinforcing understanding. The third day returned to the initial usage purpose as students were assigned a concept they had no previous instruction on. This was to determine whether students enjoyed learning a concept with TenMarks Math as the only means of instruction. The program was used a total of 3.5 class periods over the two week evaluation period. The program was used for the entire 40 minute class period on two days and roughly 30 minutes of the period on both the initial and final day of use. The other 10 minutes of those periods was spent taking the pre-usage and post-usage surveys. As there is no recommended protocol from TenMarks Math regarding the frequency and length of use, the program was used on consecutive days for the majority of the class time to enable students to focus on evaluating the program. Interventions that took place during the program included help provided by the program staff to students who encountered issues with their laptops in the form of error messages or other hardware/software issues not related to TenMarks Math. Also, after the first day of working with fractions on the program, the / symbol was explained to represent a fraction bar as many students had questions about it. Student Attitude Students completed two surveys via Google Forms regarding their attitudes towards math and TenMarks Math. One survey was taken prior to using TenMarks Math and the other survey was taken directly following the completion of program use. Both surveys were completed anonymously. The survey taken prior to use contained 8 Likert-scale items. The survey taken after usage contained 15 questions including 6 Likert-scale items, 6 fixed alternative items, one of which branched from a fixed alternative item, and 3 open-ended items with one of those branching from a fixed alternative item. Program staff also used obtrusive open-ended observation analysis to collect data. Program Use Students were given user data sheets each day to fill out as they worked on the program. These sheets contained areas to fill out about the concept the student was working on, the number of

times different help options were used, and any type of problems they encountered with the questions, program, or laptops. Performance records from the TenMarks Math website were reviewed after completion of the program. Also, program staff kept observation notes daily. Results Student Attitude- Alleviating Student Frustration Responses to the survey questions dealing with frustration due to homework are shown below in Table 1. 84% of the students have experienced frustration at some time while working on homework. 77% of the students ask others for help when they dont understand a question on the homework. These numbers indicate that there is a role for a program that helps students on concepts theyre struggling with. After using TenMarks Math, 64% of students concluded that they thought they would be less frustrated if TenMarks was assigned for homework instead of problems from their textbook. Program staff noted that when the students used TenMarks to work on concepts that they had no previous instruction on, several complained after they struggled to understand the concept. Table 1. Survey results related to student frustration Pre-Usage Survey Results 1- Never 5- Every time have How often do you get frustrated 15% 8% 38% 38% 0% working on math homework? 1- Strongly 5- Strongly 2 3 4 disagree agree If I dont understand a question on math homework, I ask someone for 0% 8% 15% 46% 31% help. 1- Strongly 5- Strongly 2 3 4 agree disagree Homework would be less frustrating if I could get hints or clues on questions I 0% 38% 38% 15% 8% was having trouble with. Post-Usage Survey Results Slightly Slightly Much Much less No less more more frustrated change frustrated frustrated frustrated If you were assigned TenMarks as homework instead of the textbook, how 36% 14% 14% 36% 0% do you think it would change your frustration with math homework? Note: 1 student was absent on the day of the pre-usage survey and did not complete it.

Student Attitude- Feelings on Math The responses to the survey questions dealing with students feelings about math are summarized in Table 2. Overall the results show students generally have neutral or positive feelings towards math and math class. 53% of students feel that they pick up mathematical concepts easily compared to 39% who report having a tougher time. The use of TenMarks Math served only to boost students feelings towards math. 50% of students reported liking math more after use of the program while the other 50% reported no change in their enjoyment of math. Table 2. Survey results related to students attitudes towards math Pre-Usage Survey Results 1- I hate it! 2 3 How do you feel about math? 8% 8% 46% 1- Least favorite How do you feel about math class? 0% 8% 62% 1- Strongly 2 3 disagree I am able to learn math easily. 8% 31% 8% Post-Usage Survey Results Like math less now No change Has using TenMarks changed your attitude towards Math at 0% 50% all? Useful Differentiated Instruction Tool The comments and notes of the program staff are summarized in this section relating to their feelings on the usefulness of TenMarks Math as an instrument for direct instruction. The staff described the creation of teacher and student accounts to get the program up and running as easy and straightforward. Concepts were listed according to Common Core standards which made the assigning of concepts to work on by the teacher very important. Otherwise, students had a hard time finding the concept and time was wasted. This issue came up when students were not assigned concepts and instead were asked to find the concept that matched up with problems they had missed on old quizzes and tests. The staff also noted that the performance records of the students were easy to use but did not have as much depth as they would have liked. They noted about wanting the program to show how many of each help option each student had used as well as an average time per question for each concept.

4 31%

23% 4 38%

5- I love it! 8% 5- Favorite class 8% 5- Strongly agree 15%

Like math more now 50%

Program Use This section summarizes results related to the students use and feelings about TenMarks Math. Table 3 summarizes the data sheets that were kept each day by the students. Hints were the most commonly used help option no matter the purpose for program use. Videos were used most frequently on new concepts and concepts learned previously. Students most frequently listed problems encountered related to questions that did not seem to fit with the concept they were working on and help options that didnt provide enough help to correctly solve a problem. Table 3. User data sheet results Helping review old concepts # of hints viewed 56 # of videos viewed 10 Laptops # of problems students encountered, sorted by source of problem 13

Helping practice Teaching new current concepts concepts 58 23 4 13 Confusing Questions/ Graphics Help not useful 5 16

Totals 137 27 TenMarks servers 8

Table 4 shows the results from the post-usage survey regarding the students thoughts on TenMarks Math. 57% felt they were engaged while using TenMarks and 71% recommended using the program a couple times a week during class. 64% of students believe the best use for TenMarks is as a supplement to current instruction. Complaints about the program dealt mainly with unhelpful hints and videos, although those same help options tied as being the most helpful. 86% of students have the internet at home which makes use of the program as an alternative to textbook homework problems a possibility.

Table 4. Post-usage survey results related to TenMarks Math Yes No Do you have the internet at 86% 14% home? 1- Not 2 helpful at all When working on concepts that you had not done well on previously, how helpful was 7% 7% TenMarks in helping you learn? 1- Bored 2 3 whole time Did you get bored using 0% 29% 14% TenMarks? 1- Not at all Was TenMarks confusing to use? If you thought TenMarks was confusing, what was it that was confusing? 21% 2 50% 3 21%

3 43% 4 43% 4 7%

4 43%

5- Extremely helpful 0%

5- Never got bored 14% 5- Extremely confusing 0%

Videos not helpful/hard to understand (3) Hints not helpful/hard to understand (3) Few times a month 7% 1x a week 7% Couple times a week 71% Every day 7% Teaching new concepts 7%

Never If it was up to you, how often would we use TenMarks Math in class? 7%

Helping review old concepts Which do you think is the best use for TenMarks? What are your favorite thing(s) about TenMarks? What would you change about TenMarks? 29%

Helping practice current concepts 64%

Help options (7) Easy questions (4) Games (2) Nothing (6) More helpful and well-explained hints (4) Clearer questions (3) More games (2) View solution after Video help Hints missing a problem 43% 0% 43%

None were helpful 14%

Which help option did you find most helpful?

Discussion This formative evaluation was done to determine if TenMarks Math met teacher and student needs as well as examining if the procedures for program use and the program itself were working as expected. Results showed that frustration while working on math homework is common for most students. Using TenMarks Math in a classroom setting kept students engaged and had only positive effects on their feelings towards math. The vast majority of students have the internet at home so using the program as an alternative to homework problems from the book is a possibility. The recommendation from the students was that the best use for TenMarks Math is as a supplement to in-class instruction for concepts theyre currently learning about. The program itself can quickly be setup and is an easy to use differentiated instruction option for teachers. The program is not perfect however as several hints and videos are not helpful and the depth of information available to teachers for analyzing student performance could be enhanced. TenMarks Math has shown it could serve as a useful tool for both math teachers and students. Almost two-thirds of the students believed that using TenMarks Math to work on concepts theyre learning would cause them less frustration then the current use of their textbook for homework problems. This belief most likely stems from the availability of help that TenMarks has built into the program for users. TenMarks Math did not cause any of the students to dislike math more and actually had half of the students reporting an increased enjoyment for the subject. The source of complaints from the students was surprising given the range of help options that TenMarks provides. None of the students reported that they used the view solution help option that while the most in-depth and time consuming, could possibly be the most useful as students could see exactly where they went wrong in solving the problem. This could possibly be due to several reasons. The option to view the solution is only available after the student has completed the problem so it might be thought of as extra work compared to help. The option does not have a button around it to delineate it as help but is rather a link beside the go on to next question button. Also, the option was not listed on the user data sheet so some students may not have recognized it as a help option and avoided it because they didnt understand its purpose. TenMarks Math looks to be a useful program to differentiate instruction while also providing support to help learners who are struggling. It is best suited for use as either an in-class review tool or as an alternative to homework problems from the textbook for concepts students are currently learning. If tablet devices are purchased for the students as has been alluded to in the newspaper and last faculty survey, this would take care of the issue of students not being able to use TenMarks Math at home. It is also recommended that teachers interested in using the program contact TenMarks to get details about the TenMarks Math premium version. The version enables enhanced abilities to customize the assigning of concepts including the integration of standardized testing results into student records. The premium version is not free though and the company prices the program per student. There is a possibility of discounted pricing if the teacher is willing to conduct a study on the effectiveness of the program and provide the results to the company. Several other programs

exist which are similar to TenMarks Math including Khan Academy and Manga High. Evaluations should also be done with these programs to determine the best fit for future education within the district. Project Cost PERSONNEL Matthew Long: 6 days x $500/day TOTAL PERSONNEL OPERATING Supplies Miscellaneous supplies TOTAL OPERATING TOTAL PROJECT BUDGET $3,000 $3,000

$20 $20 $3,020

Appendix A: Survey conducted prior to usage of TenMarks Math The survey can be found at http://sssdmath.wikispaces.com/survey

Appendix B: TenMarks Math User Data Sheet Name _____________________________________________ Number of Hints Viewed (Use tally marks): Number of Video Concepts Viewed (Use tally marks): Problems [Didnt understand question, couldnt log-in, videos wouldnt work, page never loaded, etc.]: ___________________________________________________________ ______________________________________________________________________ ______________________________________________________________________ Date _______

Concept Working On: ___________________________________________________

Appendix C: Survey conducted after usage of TenMarks Math The survey can be found at http://sssdmath.wikispaces.com/postusesurvey