This article was downloaded by: [Universiti Teknologi Malaysia] On: 01 September 2013, At: 21:55 Publisher: Routledge

Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Learning, Media and Technology

Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/cjem20

How effective is feedback in ComputerAided Assessments?

Mundeep Gill & Martin Greenhow
a a b

Learning and Teaching Development Unit, Brunel University, Middlesex, UK

b

Department of Mathematical Sciences, Brunel University, Middlesex, UK Published online: 17 Sep 2008.

To cite this article: Mundeep Gill & Martin Greenhow (2008) How effective is feedback in ComputerAided Assessments?, Learning, Media and Technology, 33:3, 207-220, DOI: 10.1080/17439880802324145 To link to this article: http://dx.doi.org/10.1080/17439880802324145

PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the Content) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/termsand-conditions

Learning, Media and Technology Vol. 33, No. 3, September 2008, 207220

How effective is feedback in Computer-Aided Assessments?

Mundeep Gilla* and Martin Greenhowb
Learning and Teaching Development Unit, Brunel University, Middlesex, UK; bDepartment of Mathematical Sciences, Brunel University, Middlesex, UK
a

Downloaded by [Universiti Teknologi Malaysia] at 21:55 01 September 2013

(Received 30 November 2007; final version received 2 May 2008 )

Taylor and Francis Ltd CJEM_A_332581.sgm Learning, 10.1080/17439880802324145 1743-9884 Original Taylor 2008 0 3 33 Mundeep.Gill@brunel.ac.uk MundeepGill 000002008 &Article Francis Media (print)/1743-9892 and Technology (online)

Computer-Aided Assessments (CAAs) have been used increasingly at Brunel University for over 10 years to test students mathematical abilities. Recently, we have focussed on providing very rich feedback to the students; given the work involved in designing and coding such feedback, it is important to study the impact of the interaction between students and feedback. To make feedback more focussed, examination scripts have been analysed to identify common student errors and misconceptions. These have then been used to code distracters in multiple-choice and responsive numerical input-type questions. Since random parameters are used in all questions developed, distracters have to be coded as algebraic or algorithmic mal-rules. This paper reports on the methodology used to identify students errors and misconceptions and how the evidence collected was used to code the distracters. The paper also provides hard evidence that real learning has taken place while students have interacted with the CAAs. Statistical analyses of exam performance over eight years indicate that students are able to improve performance in subsequent formative and summative assessments provided that they have truly engaged with the CAA, especially by spending time studying the feedback provided. Keywords: feedback; computer-aided assessments; formative assessments; mal-rules; evaluation

Introduction Computer-Aided Assessments (CAAs) have been widely used at Brunel University over the last 10 years for both summative and formative purposes. The need to provide feedback to students has become more pressing with the ever-increasing numbers of increasingly diverse students entering Higher Education (HE) which makes the process of providing students with useful formative assessment much more difficult. Resource constraints have also led to infrequent assignments being integrated into a course (Gibbs and Simpson 2004; Yorke 2003). The combination of these two factors has resulted in a reduction in the quantity, quality and timeliness of feedback (Gibbs and Simpson 2004; Brown, Bull, and Pendlebury 1997). Timeliness of feedback is a key factor for any formative assessment mechanism and should be provided whilst students are still engaged with the assessment task. Hence, the demand for CAA has increased at Brunel University, both within and beyond the mathematics department. The advantages of using CAA to design and deliver assessments have been widely discussed in the literature and an extensive overview is provided by Bull and McKenna (2004). The feedback that is provided via CAA needs to be in a form such that students are able to learn from mistakes and correct errors or
*Corresponding author. Email: mundeep.gill@brunel.ac.uk
ISSN 1743-9884 print/ISSN 1743-9892 online 2008 Taylor & Francis DOI: 10.1080/17439880802324145 http://www.informaworld.com

208

M. Gill and M. Greenhow

misconceptions (Brown and Glasner 1999; Gibbs and Simpson 2004; Ramsden 2005). Selfevidently, it is important to alert students to their mistakes so that they can improve in areas of weaknesses and avoid repeating the same mistakes. At present, lack of feedback is a recurring student concern, as measured, for example, in the National Student Satisfaction survey (NSS 2007). Formative assessment is not only about providing students with feedback, but it is also seen as a mechanism to drive and support students in their learning (Sadler 1989). Engaging with formative assessments enables a process whereby students can recognise, evaluate and react to their learning, which may involve them reflecting on the task or receiving feedback on their learning (Bell and Cowie 2001). Students therefore have a central role in formative assessments and must be active participants, using the feedback provided to modify their concepts or cognitive map of the topic, or to close skill gaps (Taras 2002). They must therefore identify their strengths and weaknesses before they can make progress with their learning (Harlen and James 1997). These benefits will only occur if they actually engage with the intended activity. Whilst such engagement is often driven by marks, at least initially, students will need feedback that connects with their existing knowledge and encourages them, thus promoting learning. If the quality of the feedback provided is of a high standard and the overall assessment task is related to current work and students abilities (i.e. topics covered in the course at the right level), then students should be able to see and capitalise on the benefits of engaging with feedback. This paper provides evidence that this is precisely what happens and quantifies the efficacy of the CAA on students subsequent exams marks, which provide a measure of student learning. Although the advantages of CAA have been widely discussed in the literature, the promised productivity benefits have been slow to appear (Conole and Warburton 2005). Many case studies have reported the effectiveness of CAA, e.g. Pitcher, Goldfinch, and Beevers (2002), Wood and Burrow (2002) and Pattinson (2004). In each case study, the aim of testing the CAA is different, for example, Pitcher, Goldfinch, and Beevers (2002) reported on the effectiveness of incorporating assessments into the computer-aided learning package Mathwise and one of the issues investigated was how useful students found the assessments. In a similar way, Wood and Burrow (2002) used questionnaires to obtain students views on the assessment system TRIADS. Pattinson (2004) goes further than just collecting qualitative data, by incorporating statistical analysis in the evaluation of the CAA she used in her module. Pattinson (2004) compared results students achieved in the online end-ofmodule examination against the number of times students accessed the CAA for formative purposes before the (summative) examination. The results indicated that there was a highly significant positive correlation between the number of times a student accessed the CAA and the examination mark of the summative assessment. Although the results are positive, Pattinson also reported that a proportion of students (38%) did not make use of the CAA for formative purposes. Providing directive feedback In formative assessment, students work is marked not only to relay and justify the grade students have achieved, but also to aid students with additional comments (i.e. feedback) to improve their knowledge, understanding and their problem-solving strategies (Brown, Bull, and Pendlebury 1997). For the objective questions of CAA, the first two are more-or-less automatic and we here concentrate on the third item. Our questions not only tell students their mark, but, when they have answered incorrectly, we test their input against that resulting

Downloaded by [Universiti Teknologi Malaysia] at 21:55 01 September 2013

Learning, Media and Technology

209

Downloaded by [Universiti Teknologi Malaysia] at 21:55 01 September 2013

from a coded mal-rule so that students are also told not just that they are wrong, but quite precisely where they went wrong. A fully worked solution is also provided which students can then use to identify and compare their solution strategy with the correct one. In being told of the exact error made and being provided with a fully worked solution, it was conjectured that students would be able to use this feedback to improve upon their underlying skill set, particularly with respect to the tested skill(s), and would then quickly identify the advantages of engaging with the CAA. Consequently, students will invest time in doing the tests, expecting some return on this investment, such as useful feedback. Therefore, it is important to spend time and attention on providing students with detailed and constructive feedback. To make distracters more reliable and realistic, and hence the feedback more targeted, students examination scripts were analysed to identify the types of mistakes that students actually make and common misconceptions that they have in the area of mechanics. In total 341 end-of-module examination scripts were analysed across two levels (level zero and one) and across two departments in different institutes (the Physics Department at the University of Reading and the department of Mathematical Sciences at Brunel University). Mistakes students made were categorised into error types and a taxonomy of error types was produced before and during the analysis (see Table 1). The classification has been kept as general as possible so that the same taxonomy can be used for other areas of mathematics, and possibly in other areas of science and technology, although it may be necessary to add further classes. In addition to identifying the most common types of mistakes, the exam script analysis was used to code distracters in the CAA questions. Since all CAA questions developed at Brunel University make use of random parameters, the distracters have to be coded algorithmically as mal-rules (Gill and Greenhow 2005). For this reason, before the common error types that students make could be coded, each one had to be checked for reliability. This means that identified errors have to be checked so that when coded as mal-rules (i.e.
Table 1. Error type Assumption Calculation Copying errors Definition Formulas Incorrect values used Knowledge Methodology Modelling Procedural Taxonomy of errors and their classication. Classification Students assume certain things that are not true, for example, in projectile questions, that vertical velocity is equal to initial velocity Method correct but calculation errors are made Copying values incorrectly Not knowing the definition of terms given in question text, e.g. magnitude Incorrectly stating/recalling formulas Using incorrect values in method, for example, when substituting values into formulas Knowledge students are lacking that would enable them to answer questions Students attempt to use an incorrect method to answer a question Unable to model a particular situation/arrangement, i.e. unable to identify all forces or the correct forces acting on the particle The method student attempts to use is correct but he/she can only do initial/certain stages of the method. They stop halfway through when they do not know the stages that follow or when they are unable to interpret initial results Reading the question text incorrectly and confusing the value of variables Basic definitions of cosine, sine and tan incorrect. This is most apparent in questions where students are required to resolve forces

Reading Trigonometry errors

210

M. Gill and M. Greenhow

Downloaded by [Universiti Teknologi Malaysia] at 21:55 01 September 2013

when random parameters are used) each one returns a unique realistic numerical value (or expression) and none returns the same value as the correct answer or any other distracter. Once checked for reliability, these identified errors are coded as distracters in multiplechoice and responsive numerical input type questions. If students make an error that corresponds to a coded distracter, they are told of the likely error they may have made and hence shown in the feedback where they went wrong. Simply telling students whether their answer is correct or incorrect and being provided with a worked solution is generally not enough (Mevarech 1983). Students need to be told where they went wrong or be given some guidance on how to improve so that they can develop their knowledge and competency in the skill that is being tested. Therefore, it is vital to have realistic and reliable distracters in the questions so that the feedback could be tailored to those ends. Figure 1a illustrates a question that was coded using evidence obtained from the analysis of students examination scripts. All distracters were coded algebraically from the mal-rules shown in Table 2, where the numbering convention comes from Gill (2007). If students make an error that does not correspond to a coded distracter, then they are simply supplied with the fully worked solution and given some general feedback. The fully
Figure 2. 1a. Graph 1b. Question Screen showing shot coded showing the as results a responsive feedback obtained students numerical on analysing receive inputif students type, theywhere make examination the an error distracters that scripts. corresponds are based on toevidence a coded distracter. collected from the analysis of students examination scripts.

Table 2. Evidence-based (EB) expressions that are returned for each error, when coded as a malrule (F1, F2 and F3 correspond to forces, LOD, LAB and LBC correspond to lengths, see Figure 1). Option/mal-rule Correct option EB55(b) EB60, EB67(b) EB62(a) EB62(b) EB62(c) Q F1LOD + F2 LAB + F3 LAB ( LAB + LBC ) F3 LAB ( LAB + LBC ) F1LOD F2 LAB F3 LAB ( LAB + LBC ) F1LOD F2 LAB F3 LAB ( LAB + LBC ) Classification of error Modelling Methodology Methodology Methodology Methodology

F2 LAB + F3 LAB F1LOD ( F2 + F3 )

( LAB + LBC )LOD F1LOD + F3 LAB ( LAB + LBC ) F2 LAB F3 LAB LBC LOD LBC F3 LAB ( F1 F2 ) LOD F1LOD + LAB ( F2 + F3 )

EB62(d) EB67(c) EB67(d)

Methodology Methodology Methodology

( LAB + LBC )

EB62(e)

F2 LAB + F3 LAB F1LOD ( LAB + LBC )

Methodology

Note: The mal-rules are categorised according to the taxonomy of Table 1.

Learning, Media and Technology

211

Downloaded by [Universiti Teknologi Malaysia] at 21:55 01 September 2013

Figure 1a. Question coded as a responsive numerical input type, where the distracters are based on evidence collected from the analysis of students examination scripts.

212

M. Gill and M. Greenhow

Downloaded by [Universiti Teknologi Malaysia] at 21:55 01 September 2013

Figure 1b. Screen shot showing feedback students receive if they make an error that corresponds to a coded distracter.

worked solution is provided with the expectation that students will be able to use this to self-identify their error(s). Initially, it was a thought that students might be put-off by such lengthy feedback and simply ignore it. Therefore, it was important to identify whether or not students actually engage with it and if they do, can the effect of this engagement be identified?

Evaluation of questions Evidence at school level shows that formative assessment can lead to significant gains (Black and Wiliam 1998; Bell and Cowie 2001). Here we seek to identify whether CAA is a valuable tool for providing feedback to HE students.

Learning, Media and Technology

213

Methodology To test the developed CAA questions, and more importantly, to identify the effectiveness of providing students with detailed feedback, computer lab sessions were incorporated into an undergraduate level one mechanics module for two academic years, 2004/5 and 2005/6. The sessions were an hour long with one of the authors present at every session to help students if they had any problems with either the technology or with the subject matter. The lab sessions were optional and marks students achieved from the CAA did not count towards their final module grade. In each session, a new assessment was made available to students, although access to previous assessments was maintained. This was done so that there was a structured approach to the assessments, linking in with topics that were being covered in the lectures at the same time. Each assessment consisted of, on average, five questions so that students would have enough time to complete at least one assessment per lab session. Students could reattempt an assessment as many times as they wanted to, whether within the dedicated lab sessions or in their own time. Answer files were saved for all attempts students made at the assessments and questionnaires were distributed in the last session to collect feedback from the students. A log of student activities recorded how students were using the assessments, i.e. were they attempting a question or simply moving on to the next one, and how they were interacting with the questions, e.g. working independently or discussing the questions with their peers. Initial research into the effectiveness of the feedback linked in with the findings from Formative Assessment in Science Teaching (FAST) project (FAST 2007). The main focus of that project was to investigate, using a questionnaire, how students perceptions of their assessment experience could be enhanced by the feedback they receive from their assessments. Results obtained from the lab sessions during the year 2004/5 were used to form our case study for this wide-ranging FAST project. Whilst recording such perceptions is a worthwhile activity, it is certainly not synonymous with the determination of whether real and lasting learning has actually taken place (here assumed to be measured by subsequent exam performance). We report on two of the evaluation methods that were used to identify whether students actually engaged with the CAA questions and to identify whether students could use this feedback to develop and enhance skills required for the subject area.

Downloaded by [Universiti Teknologi Malaysia] at 21:55 01 September 2013

Indicators To identify the level of engagement between the students and the feedback, a number of indicators were used. An indicator can be any characteristic that is prominently used within the questions and feedback. It was of interest to identify whether students had engaged with the feedback at such a level that they were able to use properly the indicators in their written work. For mechanics, four pervasive indicators were used: diagrams, presentation of solution, units and vectors. Diagrams were used within question text and feedback to explain the situation/solution more fully. All worked solutions were presented in a similar step-by-step manner, stating any formulas used and assumptions being made. Units were used throughout and vectors were displayed correctly (as bold). In order to identify whether students had engaged with the CAA questions, and more importantly the feedback, the use of these indicators in students examination scripts was investigated. Apart from the obvious logistical advantages, examinations usually record students activity at the peak of their knowledge and engagement and consequently, this is when making use of

214

M. Gill and M. Greenhow

Downloaded by [Universiti Teknologi Malaysia] at 21:55 01 September 2013

these four indicators is most revealing. Scripts were analysed from previous academic years (1999/20002003/4), when students did not have access to such a resource, and from the two years when lab sessions were incorporated into the module (2004/52005/6). Data was also available for the year after the lab sessions took place (2006/7), where a new lecturer had taken over the module and lab sessions were not integrated into the course, although students were made aware of the resource. Since the uptake of the CAA in the academic year 2006/7 was not significant, it can be assumed that not much interaction took place between students and the CAA. Figure 2 shows the results obtained on analysing students examination scripts. Figure 2 shows that the usage of diagrams increased significantly in students written work in the two years when lab sessions were incorporated into the module. On average, 86% of students made use of additional diagrams in their final module examination. On engaging with the CAA, students clearly recognised the advantages of drawing good diagrams to understand and answer the question better. The positive effect the CAA had on students in terms of making use of diagrams was further highlighted by the (somewhat negative) result from the academic year 2006/7 where there was a decrease of 48% in the number of students making use of additional diagrams in their written work. The structured way in which students presented their solutions in the end-of-module examination was consistent over the two years when lab sessions took place. Again this consistency may have been due to the engagement between students and the CAA. In the academic year 2006/7, there was a decrease of 7% in the number of students presenting their solution in a step-by-step manner. The percentage of students stating vectors correctly also increased during the two years when the lab sessions were introduced to the module. Students had engaged with the CAA at such a level they observed that vectors were indicated as bold and hence emulated this in their written work. The influence the CAA had on students in terms of indicating vectors correctly was again highlighted by the lack of students who did so in the academic year

Figure 2.

Graph showing the results obtained on analysing students examination scripts.

Learning, Media and Technology

215

Downloaded by [Universiti Teknologi Malaysia] at 21:55 01 September 2013

2006/7, where only 22% of students indicated vectors correctly, a decrease of 31% from the previous year. Although the feedback had a positive effect on students in terms of three of the four indicators, the CAA played a detrimental role in terms of students stating units in their written work. Since no question style was coded such that students were required to input the units to their corresponding calculated answer, the number of students stating units in their written work decreased in the two years when lab sessions were incorporated into the module. Students seemed to have disregarded the units and clearly did not recognise the importance of units as part of the final answer. This result shows the need for a question type where students are required to input the units as well as their calculated numerical values, thereby forcing students to be more aware of them. It is interesting to note that the feedback provided via the Mechanics CAA encouraged students to use diagrams, present solutions clearly and state vectors correctly. Without actually being asked to use diagrams (or the other two indicators), students were able to learn from the feedback that diagrams were helpful, presenting solutions in a structured way is practical and vectors need to be indicated correctly. Possibly, the feedback did not focus on units as much as the other three indicators. Data collected from the academic year 2006/7 (the year that lab sessions were not integrated into the module but students were made aware of the resource) illustrates the importance of incorporating resources such as CAA into modules (i.e. timetabled slots) rather than just making them available as additional resources. The lack of students making use of the Mechanics CAA in this particular academic year illustrates that students do not go out their way to make use of extra resources. In academic years 2004/5 and 2005/6, students had dedicated timetabled slots to access the CAA resource and spend time interacting with the material. This was the only difference between the two sets of cohorts since marks achieved from the CAA were not allocated towards students module assessment, i.e. students still attended lab sessions and engaged with the Mechanics CAA knowing full well that no marks would count towards their final grade. Statistical tests To identify whether students had engaged with the feedback at such a level that they were able to demonstrate the skills and knowledge required for the mechanics exams a few weeks after the CAA lab sessions, statistical tests were conducted using exam marks from the 1999/2000 to 2005/6 academic years. The set of data from 1999/2000 to 2003/4 provides a control since students did not have access to any CAA resource in any topics of mechanics, in contrast to 2004/52005/6 when they did. Figure 3 illustrates the data that was used and the different tests that were computed. The mechanics module comprises eight topics, but students were only given access to CAA material that tested four, namely beams, static equilibrium, structures and vectors. Consequently, marks students achieved for these four topics can be compared to the marks awarded in the non-CAA topics, namely centre of mass, Hookes Law, kinematics and projectiles. A factor that may influence the results is that there may be differing levels of difficulty between CAA and non-CAA topics. To quantify this, the marks students achieved in the same questions for all eight topics from previous academic years were used i.e. from academic years 1999/2000 to 2003/4. A two-tailed Students t-test was used to compare the marks students achieved for the two sets of topics in the academic years when no CAA was provided (1999/20002003/4). The group statistics for the two groups of topics were:
Figure 3. Grid showing the different sets of data available and the tests to be computed in order to make the relevant comparisons. Tests 1a and 2a identify any influencing factors that may exist for level of difficulty in topics and academic ability of students, respectively.

216

M. Gill and M. Greenhow

Downloaded by [Universiti Teknologi Malaysia] at 21:55 01 September 2013

Figure 3. Grid showing the different sets of data available and the tests to be computed in order to make the relevant comparisons. Tests 1a and 2a identify any influencing factors that may exist for level of difficulty in topics and academic ability of students, respectively.

N CAA topics Non-CAA topics 321 434

Mean mark (from a total of 25) 14.14 13.33

Standard deviation 8.447 8.624

The result obtained from the two-tailed unpaired Students t-test was:
t ( 753) = 1.292, p = 0.197

This result indicates that there is insufficient evidence to suggest that the marks achieved between the two sets of topics differ. This shows that there is no material difference in the level of difficulty between the two sets of topics and therefore the influence of topic choice is not significant. The marks students achieved in academic years 2004/5 and 2005/6, for the questions testing skills from the four CAA topics, were combined, as were the marks students achieved in the non-CAA topics. These two averages were compared to identify whether, overall, students performed better in the topics for which they had access to CAA in comparison to those questions for which no CAA material was given. The group statistics for the two sets of topics in the academic years when CAA was available were: N CAA topics Non-CAA topics 106 116 Mean mark (from a total of 25) 15.33 13.65 Standard deviation 8.216 8.492

The result obtained from the one-tailed Students t-test was:

t ( 220 ) = 1.499, p = 0.0675

Learning, Media and Technology

217

Downloaded by [Universiti Teknologi Malaysia] at 21:55 01 September 2013

This result indicates that there is insufficient evidence to suggest that students performed better in the topics for which they had access to CAA in comparison to the non-tested topics, at the p-value of 0.05. However, there is evidence at the p-level of 0.1 to support this claim. However, accepting at this p-value means that there is a possibility (i.e. a 1 in 10 chance) that there is no real difference between the two groups performance. Overall, this result may imply that students may not have engaged enough with the CAA to develop and enhance skills to the extent that a significant difference could be seen in students exam performance. Since questions set in the final examination tested the same skills over a number of years, marks were also compared across academic years. The questions testing the different skills across the academic years did not change significantly and therefore similar questions were compared. A factor that may influence the results is that different student cohorts may differ in academic ability. To identify whether there was a difference in academic ability between those students from previous academic years and those from the two years that attended lab sessions, marks students obtained from questions that tested skills from the non-CAA topics are analysed. The group statistics for the two cohorts of students for nonCAA topics were as follows: Cohort 20042006 19992004 N 116 434 Mean mark (from a total of 25) 13.65 13.33 Standard deviation 8.492 8.624

The result obtained from the two-tailed unpaired Students t-test was:
t (548) = 0.355, p = 0.722

This result implies that there is no evidence to suggest that there is a difference, academically, between the two cohorts of students. Therefore, any improvement in marks found in the comparison of data across academic cohorts is not due to the fact that students differ in academic ability. Marks students achieved for the questions that tested skills from the four CAA topics were compared across the two sets of year groups (1999/20002003/4 and 2004/52005/6). Each topic was compared separately so that a clearer inference could be made on whether or not students engaged successfully with the feedback. The one-tailed Students t-test was computed for each of the four comparisons, results of which are shown in Table 3. From these tests, it was found that there was statistically significant evidence to suggest that students performed better for the two topics static equilibrium and structures in the academic years 2004/5 and 2005/6, in comparison to students from previous academic years. However, there was no evidence to suggest that students performed better in academic years 2004/5 and 2005/6 for beams and vectors. This implies that students were able to engage with the feedback that was provided for the questions testing skills in static equilibrium and structures and were able to develop their skills for these two topics. However, this was not true for the topics beams and vectors. The level of feedback provided was comparable for all questions, as was the organisation of the feedback and so the quality of the feedback did not alter in any material way between the different topics. Observations made during the lab sessions, which were recorded in a log at the end of each session, revealed that students spent a great deal of time engaging with the feedback from the questions testing skills on structures, which may explain why students were able

218

M. Gill and M. Greenhow

Table 3. Results obtained from the one-tailed Students t-test and the group statistics for comparison of marks for the topics for which students in academic years 2004/52005/6 had access to CAA, in comparison to students from academic years 1999/20002003/4 who did not have such access. Result from one-tailed Students t-test Topic Beams Cohort N Mean Standard mark deviation 11.64 12.66 15.27 11.05 14.37 10.83 6.75 6.49 7.865 8.066 10.119 8.456 7.01 7.924 2.553 2.481 Result t(67) = 0.495 p-value obtained p = 0.311 Inference Not significant p < 0.05 p < 0.05 Not significant

Downloaded by [Universiti Teknologi Malaysia] at 21:55 01 September 2013

20042006 22 19992004 47 Static equilibrium 20042006 22 19992004 95 Structures 20042006 30 19992004 53 Vectors 20042006 32 19992004 126

t(115) = 2.031 p = 0.0225 t(81) = 2.034 p = 0.0225 t(156) = 0.522 p = 0.301

Note: Questions on Beams, Static Equilibrium and Structures were marked out of 25 and questions on Vectors were marked out of 9.

to use the CAA to develop and enhance skills for this topic rather than for beams or vectors. This was also indicated in the answer files which revealed that only 51% of students were able to complete the assessment on structures in the hour given, in comparison to 86% of students being able to complete the assessments on static equilibrium in the same time. Moreover, 145 attempts were made at questions on static equilibrium in comparison to only 14 attempts made at questions on beams. The reviewers of this paper both asked why the CAA marks are not substantially higher than those from previous years. For the two student-engaged topics, the mean mark for static equilibrium increased from 44% to 61% and for structures from 43% to 57%. These increases are not small but we remark that this study was embedded in an already successful and well-run course, so improving the marks yet further is likely to be more difficult than improving marks on a course with known problems. Secondly, we believe that evidencebased data may show increases that are generally more modest than those you might expect based solely on students perceptions of how much they are learning. However, this will require further study. Results collected from answer files from the academic year 2006/7 have been used purely for subjective inferences. Although students had full access to the CAA, lab sessions were not incorporated into the module like the previous two years, this meant that students decided their own formative assessment needs and accessed the CAA only when and if they decided to. From the 21 students registered on the module, only 9 students accessed the CAA, 19% of whom made attempts at four or more assessments (from a total of seven that were available). In total, 55 attempts were made at the different assessments, from which 75% were made during the month of January 2007, the exam month. This implies that students either waited until the last possible moment before accessing the assessments, when they were perhaps desperate for help or they used it as a revision tool. On comparing the average result achieved by students who accessed the CAA against the class average for the end-of-module examination, it was found that these students achieved 10% higher in the examination. This may simply mean that keen students will make use of a range of resources, including CAA, and plan for their own formative assessment needs, whereas less keen students need more support or motivation to do so. By having

Learning, Media and Technology

219

weekly lab sessions, all students are encouraged to make continuous use of the formative assessment via CAA and will be able to make use of such resources. Conclusion One of the key aims for providing students with CAA for the mechanics module was to identify whether students were able to engage with the assessment package and with the extensive feedback that was provided. The level of this engagement is important so that a value judgement can be made on the overall worth of incorporating such a resource into a module. The term worth here implies a consideration of costs and benefits. From a question setters point of view, costs include the time and energy required to code and develop such rich feedback and from a students side, the amount of time and effort they need to invest. The evaluation of the CAA has shown that, in the right setting, students do engage with formative assessment activities, even when no marks are allocated. We think that this is due to both the quality of the CAA and the fact that lab sessions were scheduled in students timetables, thereby providing a structured and supportive environment. On successful engagement with the feedback, students are able to map aspects of the feedback to their written work, such as use of diagrams, presentation of solutions and notation of vectors correctly, i.e. they have developed organisation and presentation skills. The statistical analysis indicates that students were also able to develop their mathematical skills for two of the four CAA topics. The reasons why students developed skills in just two of the four topics seem to be due to differences in the level of engagement between the students and questions and, more simply, the number of attempts students made at the questions for each topic. The collected evidence indicates that CAA is an effective tool to provide formative feedback to students, provided much development work is carried out to ensure that distracters are reliable, and which, in turn, focuses the feedback on likely student errors. Students engage with such high-quality feedback and benefits appear to go further than simply short-term recall. Much work has been done at Brunel University to provide extensive and useful feedback to students via CAA in mathematical areas. Consequently, students use the feedback as a learning resource. We believe the same will apply throughout science and technology subjects. Similarly, the methods used to evaluate CAA as a formative feedback package should be applicable to other subject areas. Notes on contributors
Mundeep Gill currently works in the Learning and Teaching Development Unit at Brunel University as a Learning Adviser, supporting students with Maths, Stats and Numeracy. Her principal research interests lie in computer-aided assessment and the effect it has on students performances, emphasising successful engagement with the assessment. Martin Greenhow is a Senior Lecturer in mathematics with interests in mathematics education, computer-aided assessment and study skills.

Downloaded by [Universiti Teknologi Malaysia] at 21:55 01 September 2013

References
Bell, B., and B. Cowie. 2001. The characteristics of formative assessment in science education. Formative Assessment in Science Education 85, no. 5: 53653.

220

M. Gill and M. Greenhow

Downloaded by [Universiti Teknologi Malaysia] at 21:55 01 September 2013

Black, P., and D. Wiliam. 1998. Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice 5, no. 1: 768. Brown, G., J. Bull, and M. Pendlebury. 1997. Assessing student learning in higher education. London: Routledge. Brown, S., and A. Glasner, eds. 1999. Assessment matters in higher education: Choosing and using diverse approaches. Buckingham: Open University Press. Bull, J., and C. McKenna. 2004. Blueprint for computer-assisted assessment. London: RoutledgeFalmer. Conole, G., and B. Warburton. 2005. A review of computer-aided assessment. ALT-J, Research in Learning Technology 13, no. 1: 1731. FAST. 2007. Formative assessment in science teaching. http://www.open.ac.uk/fast/ (accessed November 2007). Gibbs, G., and C. Simpson. 2004. Conditions under which assessment supports students learning. Learning and Teaching in Higher Education 1: 331. Gill, M. 2007. Development and evaluation of CAA as a formative assessment tool for universitylevel mechanics. Unpublished PhD thesis, Brunel University. Gill, M., and M. Greenhow. 2005. Learning and teaching via online maths tests. Proceedings of the Science Learning and Teaching Conference, June 2728, Warwick University, Warwick. Harlen, W., and M. James. 1997. Assessment and learning: Differences and relationships between formative and summative assessment. Assessment in Education: Principles, Policy & Practice 4, no. 3: 36579. Mevarech, M. 1983. A deep structure model of students statistical misconceptions. Educational Studies in Mathematics 14, no. 4: 41529. NSS. 2007. The National Student Survey. http://www.thestudentsurvey.com/ (accessed November 2007). Pattinson, S. 2004. The use of CAA for formative and summative assessments Student views and outcomes. Proceedings of the 8th International CAA Conference, July 67, in Loughborough University, Loughborough. Pitcher, N., J. Goldfinch, and C. Beevers. 2002. Aspects of computer-based assessment in mathematics. Active Learning in Higher Education 3, no. 2: 15976. Ramsden, P. 2005. Learning to teach in higher education. 2nd ed. London: RoutledgeFalmer. Sadler, D.R. 1989. Formative assessment and the design of instructional systems. Instructional Science 18: 11944. Taras, M. 2002. Using assessment for learning and learning from assessment. Assessment & Evaluation in Higher Education 27, no. 6: 50110. Wood, J., and M. Burrow. 2002. Formative assessment in engineering using TRIADS software. Proceedings of the 6th International CAA Conference, July 910, in Loughborough University, Loughborough. Yorke, M. 2003. Formative assessment in higher education: Moves towards theory and the enhancement of pedagogic practice. Higher Education 45, no. 4: 477501.