MEDICAL EDUCATION
EVALUATION PARADIGMS IN MEDICAL EDUCATION
Zarrin Seema Siddiqui
A BSTRACT
Medical education is regularly challenged with new and innovative ideas in the field of curricula, teaching learning processes and
assessment. Evaluation of these approaches and techniques provide vital information regarding their subsequent use and
application to enhance the quality of learning experiences for students. Therefore, it is essentially important to choose an
evaluation approach/model that provide meaningful and valid information to concerned stakeholders. Here two major paradigm
of evaluation i.e. scientific and naturalistic are discussed emphasizing on their use, strengths and limitations. It is concluded that
no single paradigm is superior to other and it is finally left to the evaluator for making the ultimate choice depending on the
purpose and questions that need to be answered through evaluation.
KEY WORDS: Evaluation. Medical education. Educational intervention.
INTRODUCTION
Medical education worldwide has undergone important
changes and shown notable advancement in recent years.1
Various interventions have been introduced and a need has
been recognised to apply evidence-based approaches on the
impact of these interventions.2 Evaluation thus becomes an
integral part of these interventions. It implies that there is a
methodology that allows to look at the results of what can be
done in an effective way to influence the actions going
forward.3 Surprisingly, evaluation is one particular aspect of
educational cycle which we, as human being, are constantly
performing in one way or other, it may be a television
programme, a sport event, a new departmental store or just
our colleagues to determine their worth, merit or significance.
In the context of education, however, evaluation is used to
determine the effectiveness of programs and ascertain that the
objective/outcomes have been achieved. This provides
information to program staff and stakeholders in order to
identify changes that can further improve program
effectiveness.
Educational experts have suggested many evaluation models
to achieve the above - mentioned aims each having their own
limitations and strengths. 4 In this article, two major
paradigms of evaluation, used in medical education i.e.
scientific and naturalistic, are discussed with reference to their
key emphasis, methodology and analysis. The strengths and
limitations associated with each paradigm are also identified.
SCIENTIFIC PARADIGM
There has been great emphasis on scientific research methods
in initial stages when efforts were being made to introduce
program evaluation as a separate discipline. The basic idea
Department of Medical Education, University of Western Australia, WA 6009, Perth,
Australia.
Correspondence: Dr. Zarrin Seema Siddiqui, Lecturer, Medical Education, Faculty of
Medicine and Dentistry, University of Western Australia, MBDP: M515, 1st Floor ‘N’
Block, SCGH, Nedlands WA 6009, Perth, Australia. E - m a i l :
zsiddiqui@meddent.uwa.edu.au
Received July 21, 2005; accepted: February 17, 2006.
JCPSP 2006, Vol. 16 (4): 291-293
behind scientific method is planned assessment of program
effects by means of scientific measurements. It is, therefore,
referred to a model that is grounded in measuring learning
attainment. The behavioural changes are measured against
predefined learning outcomes and provide objective
information to assist in further development. The paradigm is,
therefore, focussed on the predetermined objectives and the
final outcomes rather than intervening in the learning
process.5
METHODOLOGY
There are two general categories.
1. TRUE EXPERIMENTAL DESIGN : The subjects are randomly
assigned to program and comparison groups.
Randomisation is the vital ingredient of this process to
ensure that groups are totally comparable and the
observed differences in the outcomes are not the results of
extraneous factors or pre-existing differences. For example
a study was conducted to determine impact of an
educational intervention on the students’ learning of
clinical research methodology where participants were
divided into control and experimental groups to compare
the outcomes.6 One of the limitations noted by author was
the exchange of learning material among students in two
groups, which may have influence on results.
2. Q UASI EXPERIMENTAL DESIGN: If researcher feels that
randomisation is not possible or practical, quasi
experimental design is recommended which may be of the
following types:
(a). NON -EQUIVALENT GROUP, POST-TEST ONLY: A study is
conducted to compare the knowledge scores of
medical students in problem-based learning (PBL)
and traditional curriculum on public health topics.
The results show that PBL students were significantly
more successful in the knowledge test used.7 Here
only the outcome measure is used for comparison.
This, however, does not rule out if one group was
already better than the other before experiment or
there might be other influential factors.
291
Zarrin Seema Siddiqui
(b). NON-EQUIVALENT GROUP , PRE-TEST – POST-TEST: Here
the pre- and post-test design partially eliminates the
limitation earlier discussed. Still there might be
problems resulting from students in control group
being exposed to the experimental condition.
(c). P OST-TEST ONLY, CONTROL GROUP: When a large
number of students or teachers are involved, it is
practically impossible and time-consuming to do pre
and post-test, therefore, this design is used.
(d). T IME SERIES DESIGN: Several measurements are
undertaken from both the groups prior to and after
the experimental treatment. This provides a more
reliable evaluation, although, earlier problems may
still occur if groups differ at the onset of evaluation.
(e). A SSESSMENT OF TREATMENT GROUP ONLY: Another
form of evaluation where only treatment group is
considered yet, without the information that might
have incurred in the absence of intervention/
experiment, it would be hard to know whether the
program has actually any impact.
STRENGTHS
Experimental designs are especially useful in addressing
evaluation questions about the effectiveness and impact of
programs.8 With the use of comparative data as context for
interpreting findings, experimental designs increase our
confidence that observed outcomes are the result of a given
program or innovation instead of a function of extraneous
variable or events. For example, experimental designs may
help to answer questions such as;
• Would adopting a new integrated educational program
improve student performance?
• Is problem-based learning having a positive impact on
student achievement and faculty satisfaction?
• How is the professional development program influencing
teacher’s collegiality and classroom practice?
LIMITATIONS
The main limitations associated with this paradigm is that
objectives are sometimes difficult to predict, or may change as
the course proceeds and unintended learning may be more
important than the expected outcome of a learning program. 5
The use of randomised controlled trial is specially limited in
medical education for a number of reasons, which involve lack
of appropriate resources and funding.9,10 Other limitations are
to precisely define the parameters of the experiment and
control of variables which can literally take place only in
laboratories. A genuine control is impossible. Practical
difficulties in separating groups often result in contamination
of designs.1 1 Withholding intervention to control group has
also been subjected to ethical issues. Sample reliability and
lack of control group as mentioned earlier may also affect the
results of evaluation. Finally, even when the purpose of
evaluation is to assess the impact of a program, logistical and
feasibility issues constrain the whole framework.
292
NATURALISTIC PARADIGM
This model is derived from ethnographic methodologies
developed by anthropologists. The rationale behind this
model was that no other model really captures the context of
the program that involves students, their families, teaching
staff and other surrounding elements of the community.
Naturalistic model provides detailed information about
individual, groups or institution as they occur naturally.
When compared to experimental model, the naturalistic
model considers as well as values the positions of multiple
audiences. It does not solely rely on numerical data and
focuses on program activity rather than intent. For example in
1996-97, a structured training course was organised by a
British University for Hungarian Medical Teachers. To assess
the relevance of this course to the needs of the participants, the
evaluator adapted a naturalistic approach using in-depth
interviews. These interviews were triangulated using
observation and documentary analysis. Since the aim of
evaluation was to construct the meaning of the value of MSc
experience for foreign participants, this seems to be the most
appropriate model.1 2
METHODOLOGY
The naturalistic model mainly relies on qualitative data and
analysis. Four important forms of qualitative analysis,
commonly adapted by the naturalistic model13, are
phenomenological analysis, content analysis, analytic
induction and constant comparative analysis.
Attempts are made by the evaluator to look for information
that can be identified across multiple data sources and
methods. The categories or themes are identified and
relationship among categories is established. Finally, more
evidence to support categories and relationship is collected.
Dornan et al. used phenomenological analysis in their study to
evaluate how clinicians perceive their role in problem-based
medical education and how closely those perceptions match
the curriculum they are teaching. 1 4 Yet in another study, the
role of autopsy in the modern undergraduate curriculum has
been investigated, using content analysis within a theoretical
sample.15
In addition to qualitative data in narrative form, quantitative
data may also be included in analysis but, nevertheless, the
model is highly reliant on the expertise of the evaluator to
interpret the data, determine the significance of results and
draw conclusions.
STRENGTHS
Naturalistic methods have greater validity as they encourage
multiple data types and resources. This yields rich and timely
information about the important aspects of the program
implementation, interaction between various stakeholders,
problems encountered by the program staff etc. This also
decreases the possibility of missing intended effects as well as
offers some degree of flexibility to the evaluator, which
differentiates it from experimental model.
JCPSP 2006, Vol. 16 (4): 291-293
Evaluation paradigms in medical education
LIMITATIONS
There is higher degree of reliance on subjectivity and
reliability of human observers thus their personal
observations and intuitive analysis can lead to biased opinion.
Similarly, data collection and analysis also becomes laborious
task and may potentially be very expensive.
DISCUSSION
These two major paradigms have been in debate for a long
period. One group favours experimental methods for program
evaluation on the basis that qualitative methods often provide
misleading information which may be detrimental for the
decision-making process. The other group argues that
qualitative methods best serve the purpose of evaluation and
reject any merit of experimental or quasi-experimental
methods. They see qualitative model as highly responsive and
emphasized that the entire evaluation can be performed
through naturalistic methods of information collection.16,17
An interesting debate between Robert F. Buroch and Milbrey
W. Mc Laughlin has been reported in literature.18 The occasion
was the annual meeting of the Evaluation Network and the
Evaluation Research Society in United States. The issue for
discussion was the recommendation to the Congress and the
Department of Education regarding mandatory use of
experimental methods, where appropriate, by the federal
government. Boruch argued that field tests are essentially
required for an unbiased estimate effects. He further stressed
that the quality of an evaluation will enhance provided
experimental methods are implemented at federal or state
levels. On the contrary, McLaughlin pointed out the rigidity of
experimental methods that make an evaluator ask wrong
questions, use wrong measures and thus resulting in failure to
provide valid information for policy and practice. It is clearly
evident from the debate that individuals within each
paradigm (naturalistic or scientific) when evaluate the other
paradigm may overlook the strengths of that model while
both paradigms have their own pros and cons. It is finally left
for the evaluators to make the choice considering a series of
questions (Table I) based on Stufflebeam's four essential
attributes for a sound and fair programme evaluation.1 9
Alternatively an evaluation matrix may be developed where
the set of research questions is tabulated against a selection of
possible analysis tools.20 This enables evaluators to select the
most appropriate and feasible method.
Table I: Essential questions for evaluation.
Utility Will evaluation model be useful in serving the information
needs of the intended users?
Feasibility Will the evaluation model follow practical and feasible means
for collecting evaluative information?
Propriety How the evaluation model will be conducted in an ethical
manner, with due regard for the welfare of those involved in
the evaluation, as well as those affected by its results?
Accuracy Will the evaluation model convey technically adequate
information about the features that determined the worth or
merit of the programme being evaluated?
CONCLUSION
There is no single model that is comprehensive enough to
conduct an effective evaluation. It is worthwhile to try a
l l l l l
JCPSP 2006, Vol. 16 (4): 291-293
combination of methods when situation allows depending
upon the purpose for evaluation. However, if the key concern
is program’s impact on participant outcomes or if multiple
programs are being considered with regards to their
effectiveness for decision-making, experimental designs seem
appropriate. Otherwise, the evaluation should be enriched
with detailed information to enable timely and responsive
feedback for the stakeholders.
REFERENCES
1. Yolsal N, Bulut A, Karabey S, Ortayli N, Bahadir G, Aydin Z.
Development of training of trainers programmes and evaluation of their
effectiveness in Istanbul, Turkey. Med Teach 2003. 25: 319-24.
2. Reed D, Price EG, Windish DM, Wright SM, Gozu A, Hsu EB, et al.
Challenges in systemic reviews of educational intervention studies. Ann
Intern Med 2005; 142: 1080-9.
3. Wolfensohn J. Evaluation and poverty reduction conference. Washington
DC: The World Bank, 1999.
4. Stufflebeam DL. Evaluation models: new directions for evaluation, San
Francisco: Jossey - Bass, 2001.
5. Brigley S, Littlejohns P, Young Y, McEwan J. Continuing medical
education: the question of evaluation. Med Edu 1997; 31: 67-71.
6. Pelayo - Alvarez M, Albert - Rox X, Gil - Latore F, Gutierrez - Sigler D.
Feasibility analysis of a personalised training plan for learning research
methodology. Med Educ 2000. 34: 139-45.
7. Gurpinar E, Musal B, Aksakoglu G, Ucku R. Comparison of knowledge
scores of medical students in problem-based learning and traditional
curriculum on public health topics. BMC Med Educ 2005. 5: 7.
8. Barry G, Herman J. True and quasi-experimental designs. Practical
assessment, Res Evaluation, 1997; 5(14).
9. Smith M. The whole is greater: combining qualitative and quantitative
approaches in evaluation studies, in naturalistic evaluation. New
directions for program evaluation. San Francisco: Jossey-Bass, 1986.
10. Wilkes M, Bligh J. Evaluating educational interventions. BMJ 1999;
318: 1269-72.
11. Kember D. To control or not to control: the question of whether
experimental designs are appropriate for evaluating teaching innovations
in higher education. Assess Eval Higher Educ 2003; 28: 89-101.
12. Bolden K, Willoughby SA, Claridge MT, Lewis AP. Training Hungarian
primary healthcare teachers: the relevance of a UK postgraduate course
for health educators. Med Educ 2000; 34: 61-5.
13. Payne D. Designing educational project and program evaluations: a
practical overview based on research and experience. Boston: Kluwer
Academic Publishers, 1994.
14. Dornan T, Scherpbier A, King N, Boshuizen H. Clinical teachers and
problem-based learning: a phenomenological study. Med Educ 2005.
39: 163-70.
15. Burton J. The autopsy in modern undergraduate medical education: a
qualitative study of uses and curriculum considerations. Med Educ
2003. 37: 1073-81.
16. Parlett M, Hamilton D. Evaluation and illumination: a new approach to
the study of innovative programs, in beyond the numbers game. Berkley,
CA: McCutchan, 1978.
17. Guba E, Lincoln Y. Effective evaluation: improving the usefulness of
evaluation results through responsive and naturalistic approaches. San
Francisco: Jossey- Bass, 1981.
18. Davis B. Boruch and Mclaughlin debate. Eval News 1982; 3: 11-20.
19. Curran V, Christopher J, Lemire F, Collins A, Barrett B. Application of
a responsive evaluation approach in medical education. Med Educ
2003. 37: 256-66.
20. Reeves T. Evaluation matrix. available http://mime1.marc.gatech.
edu/MM_Tools/EM.html accessed online 29th September 2005.
O
l l l l l
293