Editorial
The metric of medical education
Why a series on measurement?
Over the last decade significant ad-
vances have been made in the assess-
ment of clinical competence of
healthcare professionals.13 Historical-
ly, one major stimulus to these
advances was the publication, over
20 years ago, of a paper by Harden
and Gleeson4 describing how the long
case was an inefficient method of
assessment and could usefully be
supplemented by the objective struc-
tured clinical examination (OSCE).
Subsequently, simulated patients were
shown to be a feasible method of
controlling many testing situations.5
Expertise was developing in both the
USA and in Europe on using new
numerical techniques, such as general-
izability theory and Rasch scaling, to
improve the precision of measurement
of human capacities.6 The First
Cambridge Conference, in 1984,
catalysed the synthesis of traditional
(judgement oriented) and psycho-
metric testing procedures, one porten-
tous example being the embryonic idea
behind key features testing.7 In sum,
these events opened the way to a
renewed interest in the psychometrics
of clinical competence.
This work was enthusiastically
developed in the USA, the Netherlands,
Canada and Australia. However, a
recent survey8 and observations made
by the General Medical Council during
its round of visits to medical schools in
the late 1990s9 highlighted continuing
deficiencies in the technical aspects of
examinations in the UK. Conversations
with colleagues worldwide suggest that,
although there are centres of excellence
and much good practice, the situation
in many countries is not unlike that in
the UK; some medical schools, boards
and postgraduate institutions have
Correspondence: Brian Jolly, Centre of
Medical and Health Sciences Education,
PO Box 15, Monash University, Victoria
3800, Australia.
E-mail: brian.jolly@med.monash.edu.au
798
forged ahead with developments in
assessment techniques while others
still lag far behind. Inconsistency or
incoherence is common. Recently, even
the British Medical Journal has shown
bipolar tendencies, with two editorials
on the long case, giving contrasting
perspectives, with neither quoting or
acknowledging the other.10,11 There is
clearly a problem.
What now?
There are reasons for this dissonance.
One is philosophical. There is a sizeable
cohort of medical educators who have
been nurtured in paradigms at odds
with the technical rational or statistical
ones that drive much of psychological
measurement.1214 Adult learning prin-
ciples say a lot about negotiated goals
and objectives, but very little about
absolute professional standards.15 In an
analogous sense, medicines apparent
general reluctance to deal with
evidence-based medicine seems to be
as much due to the professions unease
with statistical reasoning as it is to
difficulties in dealing with the
application of clinical guidelines to the
patient in the chair, and the apparent
disenfranchisement this represents.16
Clinicians attempting to understand
the process of clinical decision making
have stressed observation, experience,
inference and judgement,17 and have
only recently been concerned with
data. Another reason may simply be
ignorance on the part of those charged
with devising and administering
examinations.
Examples of the difficulty examina-
tion boards have in reconciling conflict-
ing paradigms can be seen in the
reluctance to give up negative marking
or correction for guessing of MCQ
questions, and in bizarre and opaque
procedures such as close marking. The
latter arose from the difficulty of com-
bining scores from different tests with
widely variant origins or standard devi-
ations, and from the misguided belief
that attempting to make fewer fine
distinctions resolved the issue rather

Blackwell Science Ltd ME D I C AL ED U C AT I ON 2002;36:798799
than simply hiding, or worse still, com-
pounding it. Another factor is the place
of assessment in the educational frame-
work often seen as a bolt-on after-
thought rather than an intrinsic part of
the curriculum.
Some of these problems are insoluble
or very context-specific; selecting 5%
of house officers for advanced training
in neurosurgery is a different assessment
problem, requiring different strategies
and solutions, from revalidating an
entire profession. However all deserve
to be tackled at least by informed
decision making, if not by precise
algorithms. After all, assessment is
probably the area of medical educa-
tion that has the most robust evidence
base.
What are we going to do?
Starting in this issue, the Metric of
medical education series will attempt to
enable educators to better understand
the basic concepts of assessment, and
thus to offer an enhanced rationale for
attempting measurement of clinical
competence. In addition, because of
the close relationship between assess-
ment and the development of appro-
priate outcome measures for
intervention studies, the series will also
include a how to guide on designing an
educational experiment. In this latter
role, the Metric series will link with
another forthcoming series on research
methods. Metric will comprise short,
commissioned articles for the medical
teacher, researcher and curriculum
designer, written by clinicians and
assessment specialists working together.
The emphasis is on presenting the basic
framework of assessment and specific
psychometric or assessment techniques.
The first 3 articles will cover theory of
assessment, generalizability theory and
study design. Thereafter the planned
sequence is to cover techniques appro-
priate to the measurement of compe-
tence, followed by those recently
developed or enhanced for looking at
performance. This will be followed by
articles on standards.
Editorial B Jolly
Reader feedback
We are asking for reader feedback on
the Medical Education website (http://
www.mededuc.com) during and after
the Metric series: we would like
your comments on issues raised, any
controversies, and those things you
always wanted to know but were
afraid to ask. If there is a demand
for topics that we have not previously
covered, then we will try to include
them. In addition, if you think you
have a topic that you could usefully
contribute to the series please email us
on brian. jolly@med.monash.edu.au
or j.a.spencer@newcastle.ac.uk.
Brian Jolly
Monash University, Australia
John Spencer
Newcastle, UK
References
1 Newble DI, Jolly BC, Wakeford RE.
The Certification and Recertification
of Doctors: Issues in the Assessment
of Clinical Competence. Cambridge:
Cambridge University Press; 1994.
2 Norman IJ, Watson R, Murrells T,
Calman L, Redfern S. The validity

Blackwell Science Ltd ME D I C A L ED U C A T I ON 2002;36:798799
and reliability of methods to assess the
competence to practise of pre-regis-
tration nursing and midwifery stu-
dents. Int J Nursing Studies
2002;39:13345.
3 Wass V, Van der Vleuten C, Shatzer J,
Jones R. Assessment of clinical com-
petence. Lancet 2001;357:9459.
4 Harden R, McG, Gleeson F. Assess-
ment of clinical competence using an
objective structured clinical examina-
tion (OSCE). Med Educ 1979;13:41
54.
5 Stillman PL, Swanson DB. Ensuring
the clinical competence of medical
school graduates through standardized
patients. Arch Intern Med
1987;147:104952.
6 Wood R. Assessment and Testing.
Cambridge: University of Cambridge
Local Examinations Syndicate; 1991.
7 Wakeford, RE, ed. Directions in Clin-
icalAssessmentCambridge:Cambridge
University Medical School; 1985.
8 Fowell S, Bligh J. Assessment of
undergraduate medical education in
the UK. Time to ditch motherhood
and apple pie. Med Educ 2001;
35:10067.
799
9 General Medical Council. http://
www.gmc-uk.org/med_ed/meded_f
rameset.htm (accessed 13/05/02).
10 Norcini JJ. The death of the long case?
BMJ 2002;324:4089.
11 Norman G. The long case versus
objective structured clinical examina-
tions. BMJ 2002;324:7489.
12 Schon DA. The Reflective practitio-
ner: how professionals think in action.
Aldershot, UK: Arena; 1995.
13 Coles C, Fish D, eds. Developing
Professional Judgment in Health Care.
Oxford: Butterworth-Heinemann;
1998.
14 Eraut MR. Developing Professional
Knowledge and Competence. Lon-
don: Falmer Press; 1994.
15 Beckett D, Hager P. Life, Work and
Learning: Practice in Postmodernity.
London: Routledge; 2002.
16 Wilson T, Holt T, Greenhalgh T.
Complexity science: complexity and
clinical care. BMJ 2001;323:6858.
17 Elstein AS, Schwarz A. Evidence base
of clinical diagnosis: Clinical problem
solving and diagnostic decision mak-
ing: selective review of the cognitive
literature. BMJ 2002;324:72932.