A Web Based Module Evaluation System

Interim Innovation Project report to the Learning & Teaching Committee

Nadine Foster (Dept of Physiotherapy Studies), Nikki Rochford (Dept of Physiotherapy Studies), Barry
Smalley (School of Chemistry & Physics), Stephen Bostock (Staff Development & Training).

Aim of Project

To develop a web-based system to facilitate the rapid analysis of module evaluations. The module
evaluation system will be based on one currently in use by the Physics Department.

Background and Purpose

The Department of Physiotherapy Studies has demonstrated a keen determination to improve the
already high standard of quality assurance mechanisms used within the department. Whilst the
department feels its present mechanism of module evaluation is effective, it’s reliance on paper
questionnaires and the resultant time required in the analysis and production of useful data, renders this
process rather intensive in terms of staff time and of questionable cost-effectiveness. The relevance of
these issues in a quality assurance programme is recognised and the aim of this project was to improve
the efficiency and cost-effectiveness of the module evaluation process.

We further proposed to develop this into a portable system using the Physiotherapy Undergraduate
course as a pilot for a more general system. This evaluation system will also facilitate the use of
information technology by students and will respond to the needs for impartial collection of student
views without unnecessary use of staff time.

The Department of Physiotherapy Studies further recognised that the paper module evaluation forms
were not fully student-centred. It was felt that they did not consistently investigate students’ attitudes
towards their own responsibility for the learning process. It was therefore intended that the modular
evaluation form developed for use on the web would investigate the extent to which the student
actively engages in the learning process.

We also recognised the need to be able to compare the students’ learning experiences across modules.
It was expected, therefore, that the web-based module evaluation system would facilitate comparison in
the student experience across modules. The development of this web-based module evaluation system,
within the Physiotherapy Department would act as a pilot to evaluate the transferability of the system
to other departments within the University. This will fit well with the aims of the Quality Assurance
initiatives to enable comparison across departments and Faculties.

It was further expected that this project would contribute to the embedding of IT within the BSc (Hons)
Physiotherapy programme, as well as the promotion of the transferable skills of life-long and active
learning, via student reflection.

Objectives

o To develop a standard module evaluation form for the BSc Physiotherapy Studies (Hons)
programme, which would facilitate comparability across modules
o To further develop & enhance the supporting software to analyse and present the evaluation
data producing meaningful web-based summary data.
o To develop a user-friendly tool which does not require a high degree of computer literacy to
use
o To pilot the developed evaluation questionnaire forms within the Physiotherapy programme
o To investigate how feasible it might be to use this web-based system of module evaluation
across the wider University.

08/09/10
Method

In order to meet the above objectives, the following procedure was undertaken:
1. Current module evaluation tools used within academic environments were reviewed for their
appropriateness and pertinence, additionally focus groups and staff discussion were utilised to
identify the main areas considered to be important in a module evaluation.
2. From these a draft questionnaire was then developed.
3. The draft questionnaire was which was reviewed by all staff and representatives of the student
body for relevance and appropriateness and further developed on the feedback.
4. The developed module evaluation form was then adapted to a web based system, which was
then piloted within all first semester physiotherapy modules.

Attached to this report are examples of the form and the outputs.

Results of Pilot

1. Comparability between web based and paper evaluations.

To ascertain if the computer results were comparable to the results gained from a paper evaluation the
student cohorts were randomly assigned to two groups, the first undertook the computer evaluation and
the second completed the same evaluation form but on paper.

The following table gives a comparison of the results obtained from these two sources.

Modal score Modal score

for web version for paper version
Module 1 4 4
Module 2 4 4
Module 3 4 4
Module 4 4 4
Module 5 4 3

Average number of students giving a low score (1,2) for a question,

out of about 38 students (and range of values for any one question)
for web version for paper version
Module 1 2.7 (range 0-22) 6.7 (range 0-31)
Module 2 7.1 (range 0-25) 5.5 (range 0-22)
Module 3 2.5 (range 0-20) 3.5 (range 0-18)
Module 4 8.5 (range 0-22) 9.7 (range 3-17)
Module 5 5.1 (range 0-14) 7.7 (range 1-15)

It is clear that the method of delivery did not systematically affect the results obtained from the module
evaluation.

2. Staff Evaluation

A questionnaire was completed by all module leaders to determine their views of the new system:

Staff preference
All 5 staff preferred the new evaluation form
2 staff preferred computer system to the paper based one
3 staff stated no preference between the two systems.
Benefits
All staff stated that the main benefit of the new system was the amount of time saved.
The ease of access to results was also considered to be a benefit.

08/09/10
 3. Student Evaluation

Students also were asked to complete a questionnaire concerning their attitudes towards the new
evaluation system. The following graphs illustrate the results of this audit.

Q 1 P re fe r p a p e r o r w e b w e b p ap er
Q2.Was form design helpful? web paper

7 0%
6 0%
5 0%
4 0%
60%
3 0%
2 0% 40%
1 0%
0% 20%
paper eit h e r web
0%
yes maybe no

D id the m ediu m m ake a diffe rence? web paper Q 2 .3 N e ed a P C la b s es s io n ? (w e b u s e rs )

100% 70%
60%
80%
50%
60% 40%

40% 30%
20%
20%
10%
0% 0%
yes m ay be no y es m ay be no

Overall, students were happy with the new system. A larger number of students who had completed
paper evaluations stated a preference for paper, but they had had no experience of the web based
system. Those who used the web form, on balance, preferred it. They were positive about the design of
the form. Various improvements were suggested, many of which had already been considered. A large
majority of students felt the medium made no difference to their answers, and those that thought it did
were not consistent in the effect they thought it would have. A majority thought it would be necessary
to book a PC lab to complete the web forms were the computer system adopted.

Further developments

o The standard web form is now being used for all physiotherapy modules, Years 1, 2 and 3.
o A programme evaluation questionnaire had additionally been developed and was piloted at the
end of May 2002; the results are pending.
o The web report now provides summary data for inclusion in the course report. The usefulness
of this will be determined after summer.
o The transferability of the system to other departments is now to be determined.

Conclusion

It is expected that, with minor amendments, this method of module evaluation will be embedded within
the plans for the new BSc (Hons) Physiotherapy course due to commence October 2002. This is
expected to result in a more cost-effective and efficient method of evaluation. Added benefits for
Physiotherapy Studies include the increasing use of IT by students and the improvement of information
quality gathered as part of module evaluation. The system design will explore long-term issues such as
ensuring portability. It is intended that the transferability of this tool to other departments will have
been explored and recommendations made for campus-wide availability.

08/09/10