You are on page 1of 4

Anaesthesia, 1998, 53, 1105–1108

................................................................................................................................................................................................................................................

S P E C I A L C O M M U N I C AT I O N
Multiple-choice examinations: adopting an evidence-based
approach to exam technique

E. J. Hammond,1 A. K. McIndoe,2 A. J. Sansome3 and P. M. Spargo3


1 Specialist Registrar and 3 Consultant, Shackleton Department of Anaesthesia, Southampton University Hospital NHS
Trust, Southampton SO16 6YD, UK
2 Lecturer in Anaesthesia, Sir Humphry Davy Department of Anaesthesia, Bristol Royal Infirmary, Bristol BS2 8HW, UK

Summary
Negatively marked multiple-choice questions (MCQs) are part of the assessment process in both
the Primary and Final examinations for the fellowship of the Royal College of Anaesthetists. It is
said that candidates who guess will lose marks in the MCQ paper. We studied candidates attending
a pre-examination revision course and have shown that an evaluation of examination technique is
an important part of an individual’s preparation. All candidates benefited substantially from backing
their educated guesses while only 3 out of 27 lost marks from backing their wild guesses. Failure to
appreciate the relationship between knowledge and technique may significantly affect a candidate’s
performance in the examination.

Keywords Education; examination technique.

......................................................................................
Correspondence to: Dr E. J. Hammond, Department of Anaesthesia,
Poole General Hospital, Longfleet Road, Poole BH15 2JB, UK
Accepted: 20 April 1998

Multiple-choice examinations (MCQs) are used as part of independently. It is said that ‘negative marking’ is used to
the assessment process in many postgraduate medical discourage guessing.
examinations. They are easier to mark than essays or A recent article in the Newsletter of the Royal College of
short answer questions and the process may be automated Anaesthetists summarised the results of the first sitting of the
using optical mark reader technology. This facilitates the new Primary examination [1]. Candidates performed con-
computerised analysis of the raw data and allows the siderably worse in the MCQ paper with only 30.5% passing
examining body to compare the performance of either as opposed to between 50 and 60% for the other parts of
the group or an individual with that of past candidates by the examination. This reduced the overall pass rate for the
the use of discriminator questions. MCQs have formed an Primary to 37%. A subsequent article suggested that two
integral part of the examination for the fellowship of the mechanisms account for the majority of failures [2]. They
Royal College of Anaesthetists (FRCA) for many years. are lack of preparation/clinical experience and technical
The structure of MCQ papers should eliminate the poten- reasons which include exam technique. It was suggested
tial for examiner bias that may occur in the assessment of that ‘at every MCQ examination there are candidates who
essays or oral examinations. The MCQ responses are clearly do not understand the concept of negative mark-
subjected to ‘negative marking’. Within this system a ing. They have initially answered sufficient questions to
correct answer gains a mark, a wrong answer loses a mark pass and then clearly guess at the rest, get them wrong and
and absence of an answer results in no mark. Whilst there therefore fail.’ The advice to candidates has been to
is a penalty for a wrong answer there is no penalty for approach negatively marked MCQs with caution with
leaving it blank or selecting the ‘don’t know’ option. the solution to this form of ‘technical’ failure being simply
Each stem question must therefore be regarded as having ‘do not guess’. Is this the right advice?
five parts or leaves (labelled A–E) which are answered This paper examines whether objective assessment of

Q 1998 Blackwell Science Ltd 1105


E. J. Hammond et al. • Multiple-choice examinations Anaesthesia, 1998, 53, 1105–1108
................................................................................................................................................................................................................................................

exam technique for negatively marked MCQs may sig- Table 2 How the candidates performed in the exam.
nificantly alter the chance of success in this form of
examination. No. of question Percentage of total
leaves answered number of leaves

Methods Correct 3197 78.9%


Incorrect 741 18.3%
The Wessex Intensive Revision course for the Primary Don’t Know 112 2.8%
FRCA examination incorporates mock mini MCQ exami-
nations using a structure similar to the Primary examina-
tion. For the purpose of this study all candidates attending
imported into a custom-built data analysis package along
three revision courses immediately preceding the next
with candidate identification information. This package
sitting of the examination were asked to participate in a
produced results sheets for individuals along with a
mock MCQ paper. The questions were drawn from a
collective report of both individual and group perfor-
bank accumulated over a number of years from the
mance. A feedback session the following day enabled
immediate recollections of candidates who had sat the
candidates to compare their knowledge base and tech-
College examinations. Thirty questions were selected and
nique anonymously against their peer group. Individuals
the content/accuracy verified to reflect a cross-section of
were given objective advice based on their performance
the current primary syllabus. Candidates sat the examination
and accuracy.
using a desktop computer. We designed and programmed
an interactive computer-based MCQ examination to allow
candidates to attempt the exam on screen. No previous Results
computer experience was required. Candidates were pre-
Thirty candidates attended the three Wessex Intensive
sented with questions consisting of a stem and five leaves
revision courses in 1997. Three candidates failed to
labelled A–E. Candidates could navigate forwards and
assign any confidence information to their responses,
backwards through the questions using the mouse or
while 27 candidates completed the 30-question multiple-
jump to a specific question. Questions could be completed
choice examination as instructed. These candidates had
in any order.
received a mean of 15.5 months training in anaesthesia
In order to address the issue of exam technique, two
(range 11–48 months). All were sitting the FRCA exami-
types of answer box were provided for each question – the
nation at the next sitting and three had failed in previous
traditional mutually exclusive ‘True’, ‘False’ and ‘Don’t
attempts.
know’ and a new concept which we describe as the
Analysis of the raw MCQ results showed the mean (SD)
confidence option. This allowed candidates to assign a
score to be 60.5 (8.5)%. Overall 78.9% of the leaves
level of certainty to each of their answers. For the purpose
were answered correctly, 18.3% incorrectly and 2.8% left
of the study three levels of certainty or confidence were
unanswered (Table 2).
available to candidates in a pop-up menu (Table 1).
Candidates assigned the ‘Positive’ confidence option
Candidates were given basic instruction on how to use
to 2011 (49.7%), ‘Educated guess’ to 1566 (38.7%), ‘Wild
the system. The MCQ paper was attempted under strict
guess’ to 357 (8.8%) and ‘Don’t Know’ to 112 of the
examination conditions. For each True or False response,
question leaves. Confidence option information was
candidates were asked to assign a level of certainty for
not entered for four out of the possible 4050 question
that response from the confidence option. Candidates
leaves answered.
were encouraged to attempt all questions and to ensure
For questions assigned the ‘Positive’ option, candidates
they assigned a level of certainty to each answer as
were correct 89.2% of the time, whilst 75% of ‘educated’
described above.
and 65% of ‘wild’ guesses were correct (Table 3).
Upon completion of the exam, the data from the
candidates’ responses were collated electronically and
Table 3 How candidates fare when they guess.

Table 1 Confidence options for the MCQ exam. Level of No. of questions
confidence answered Correct Wrong % Correct
Positive Candidate is sure of the answer
Educated Guess Candidate knows something about the subject but Positive 2011 1794 217 89.2%
is not absolutely sure of the answer Educated Guess 1566 1165 401 74.4%
Wild Guess Candidate is taking a random guess Wild Guess 357 234 123 65.5%

1106 Q 1998 Blackwell Science Ltd


Anaesthesia, 1998, 53, 1105–1108 E. J. Hammond et al. • Multiple-choice examinations
................................................................................................................................................................................................................................................

Figure 1 Cumulative average score (%).

The contribution of marks attributable to ‘educated’ should be to attempt only those questions of which they
guesses raised the cumulative average overall examination are certain. However, the College marking system only
score for the group from 38.9% to 57.8%, with ‘wild’ identifies whether a candidate has answered the question.
guesses further raising the score to 60.5% (Fig. 1). It does not and cannot collect information on whether the
The results from individuals showed that all candidates candidate is sure of the answers. Is this advice therefore
improved their score in the paper if they backed their based on fact or fallacy? How should candidates prepare for
educated guesses. This improvement ranged from 3 to 61 the exam?
marks (2–40%). Candidates sitting the mock examination in this study
For leaves assigned the wild guess option, three candi- used an electronic format allowing them to identify and
dates lost one, two and three marks, respectively. Four tag each answer with a level of certainty. This facilitated
candidates neither benefited nor lost marks by guessing the objective assessment of technique and analysed the
wildly, two made no wild guesses and 18 candidates effects of guessing.
improved their score with wild guesses contributing It is clear from this study that candidates only answer
from 1 to 23 marks (0.6–15%). correctly 90% of the 50% of the MCQs of which they
If we assume a pass mark of 50% for this paper, only one are certain of the answer. Candidates may therefore lull
of 27 candidates would have achieved the pass mark had themselves into a false sense of security if they adopt the
they answered only those questions of which they were approach of answering only those questions of which they
sure. The inclusion of educated guesses raises this to 24 out are sure. Indeed a candidate who felt confident of the
of the 27. Wild guesses further increased the number answers to 75% of the paper could, if 10% of the responses
reaching 50% to 25. Raising the pass mark to 60% were incorrect, only expect an overall score of 60%, a score
would mean that educated guesses push 11 candidates that falls perilously close to the pass mark.
over the 60% mark and wild guesses further increase this to Educated guesses are correct < 75% of the time imply-
14 out of the 27 candidates. ing that, for every 10 question leaves answered, candidates
can expect to gain about five marks. For wild guesses,
where 60% are correct the net gain is approximately two
Discussion
marks (Fig. 2).
The MCQ examination remains a significant hurdle for From a statistical viewpoint, in a negatively marked
candidates in the primary examination and the use of a exam where the penalty is the same as the reward, the effect
negative marking system is a major concern. The advice of guessing is to produce a null result rather than a negative
given to candidates has been the subject of much debate; result. Negative marking is employed to discourage guessing
some tutors advise candidates to attempt only those ques- by nullifying the overall results of guesses rather than by
tions they are sure of whilst others advise an attempt at producing a negative score. This concept is not clearly
almost every question. Most candidates believe that negative understood by candidates. The probability of getting a
marking is employed to discourage guessing by penalising question correct with a random guess is of course 50%.
incorrect answers, thus reducing their final mark. Indeed Irrespective of the number of questions or the bias of
the advice from the RCA is clear; candidates fail the MCQ the answers, random guesses will result in an overall score
by guessing and the correct approach to this examination that trends towards zero. This study suggests that even wild

Q 1998 Blackwell Science Ltd 1107


E. J. Hammond et al. • Multiple-choice examinations Anaesthesia, 1998, 53, 1105–1108
................................................................................................................................................................................................................................................

Figure 2 Net gain in marks per 10


questions answered at each Level of
Confidence.

guesses are probably inspired. Clearly the knowledge base combination of knowledge and technique can act syner-
and the propensity to take risk when deciding whether to gistically to improve a candidate’s final score. It is the
attempt a question varies considerably between individuals. acquisition of knowledge and insight that may skew the
The objective assessment of the effect of guesses in this random effect of guessing from a null score towards a
study suggests that the advice ‘do not guess’ is incorrect. positive score. Computer-based learning tools, such as the
Indeed, candidates could feasibly fail by not answering one used in this study, facilitate the process of assessing
enough questions rather than losing marks by guessing. technique. Since a candidate’s final score is a function of
The relationship between knowledge and confidence both the number of questions answered and accuracy
(level of certainty) varies between individuals and may for each level of certainty it is essential that candidates
even vary within an individual over time. In addition to evaluate their own performances in order to determine
this the distinction between a wild guess, an educated the strategy which will best reward their knowledge
guess and a positive answer is blurred and may vary and technique. Failure to do so may significantly affect
considerably between candidates. There may actually be their results.
no such entity as the wild guess. It is likely that a candidate Finally, it is worth emphasising that only proper pre-
will have some experience of the subject of a question, paration will prevent poor performance in the MCQ.
however limited their knowledge. The advantage of using
the confidence option in this study is that it enables
Acknowledgments
individuals to assess their own performance based on
their assignment of confidence levels which are unique This paper was presented in part at the Group of Anaes-
to the individual. thetists in Training Scientific Meeting in Nottingham,
In conclusion, the MCQ examination is not a com- April 1998, where it was awarded the President’s Medal.
pletely transparent examination tool and is supplemented
by short answer and viva components in most postgraduate
References
examinations. However, since it is relatively cheap, easy to
standardise and rapidly generates an objective score, it is 1 Hull CJ. The new primary examination. Royal College of
here to stay. As a consequence candidates may find that an Anaesthetists Newsletter 1997; January, supplement; 32: 6–7.
objective assessment of their technique pays dividends. 2 Strunin L. The FRCA – Why do people fail? Royal College
There is no substitute for basic knowledge, but the of Anaesthetists Newsletter 1997; March, supplement; 33: 5–6.

1108 Q 1998 Blackwell Science Ltd