Eur J Vasc Endovasc Surg (2017) 54, 654e658

TOPICAL EDUCATIONAL REVIEW

How to Write a High Quality Multiple Choice Question (MCQ): A Guide for
Clinicians
a,* b
P.A. Coughlin , C.R. Featherstone
a
Department of Vascular Surgery, Addenbrookes Hospital, Cambridge, UK
b
The Royal College of Surgeons of Edinburgh, Edinburgh, UK

Despite the variety of assessment tools available, multiple choice questions (MCQs) still play an integral part in
examinations at both a national and speciality board level. MCQs have a number of methodological advantages
yet their strength is related to the quality of the question posed. Specifically, there has been a move towards the
MCQ testing a taxonomically higher order concept of integration-interpretation and problem solving. This paper
focuses on question development and the potential pitfalls to avoid.
Ó 2017 European Society for Vascular Surgery. Published by Elsevier Ltd. All rights reserved.
Article history: Received 15 May 2017, Accepted 17 July 2017, Available online 1 September 2017
Keywords: Multiple choice question, Question writing

INTRODUCTION highlight questions for review before scores are finalised.

Assessment plays an important role in encouraging learning
and in evaluating whether learning goals have been
reached. In addition, assessment outcomes can help iden-
tify areas of strengths and weaknesses within candidates
and training programmes, to guide future development. Key
to these beneficial effects are the validity and reliability of
the assessment: does it measure what we want to measure,
and does it provide reproducible and generalisable results?
The diversity of assessment tools available to assessors
and examination bodies seems ever increasing, as tech-
nology and research enable and validate new tools that can
mimic with different levels of authenticity the “real world”
tasks candidates will be confronted with in the workplace.
Yet, the humble multiple choice question (MCQ) has
remained a favoured tool in a number of national and
specialty board examinations, and has recently been used
as an educational tool within the European Journal of
Vascular and Endovascular Surgery. As such, a critical
appraisal of how to formulate a MCQ question is both
timely and warranted.
There are a number of advantages to the use of MCQs as
a method of assessment. They allow for good blueprint
coverage, with a 3 hour examination paper typically con-
taining 180 questions. They allow for automated computer
marking, making the marking of large numbers of candi-
dates a lot more feasible than most other assessment tools,
and removing the room for inter-examiner variability in
marking. The results can be subjected to rigorous psycho-
metric analysis to identify odd patterns of results and
* Corresponding author.
E-mail address: patrick.coughlin@addenbrookes.nhs.uk (P.A. Coughlin).
1078-5884/Ó 2017 European Society for Vascular Surgery. Published by
Elsevier Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.ejvs.2017.07.012
With multiple questions typically being asked on each area
of the blueprint, they can be used to provide feedback on
how candidates performed in different areas. These ad-
vantages confer a good degree of reliability to this assess-
ment tool. But what of its validity?
Traditionally, MCQs have been seen as a useful tool for
testing factual knowledge, with higher order cognitive
processes, such as interpretation, application of knowledge
and problem solving, being left to short answer questions,
essays, and oral examinations. Fig. 1A shows a more tradi-
tional MCQ testing purely factual knowledge. This question
is purely a recitation of factual knowledge and requires no
higher order thinking. Each of the answers are either true or
false.
This is an efficient way of testing whether candidates
know the genetic mutation responsible for the syndrome.
However, depending on the purpose of the assessment and
its place within the curriculum, this may or may not be what
we want to know about the candidate. It tells us nothing
about whether candidates could diagnose patients with this
syndrome based on their presentation, about whether the
candidates know about the management of patients with
this syndrome, or about what advice they would provide to
their patients on the basis of this information.
CURRENT APPROACHES TO MCQ WRITING
Consequently, more recent guidelines on writing multiple
choice questions within the field of medical education1,2
have emphasised ways in which this question format can
be used to test skills that are required by trainees in the
workplace: data interpretation, synthesis, problem solving,
and decision making. An appropriately constructed MCQ
can extend our questioning beyond simple recall of
knowledge and comprehension towards the taxonomically
How to Write a High Quality Multiple Choice Question 655

(a) An example of a poor MCQ.

Which of the following genetic mutations is responsible for Loeys-Dietz

syndrome

(a) Fibrillin-2 mutation

(b) FOXC2 gene mutation
(c) Prothrombin G20210A mutation
(d) TGF-b1 or 2 receptor mutation
(e) TNF-a receptor mutation

(b) An example of a good quality single best answer (SBA) MCQ.

STEM: An 84 year old women presents with a fall, landing on her left arm. The
hand is cool, pale and there is no palpable pulses at the wrist or in the
antecubital fossa. Furthermore, she is unable to extend the hand at the wrist
joint yet has maintained sensation in the palm of the hand and in the ingers.
Imaging shows a displaced mid humeral fracture and a short mid brachial artery
occlusion.

LEAD IN: Which nerve is likely to have been damaged with the associated fall?

A. The axillary nerve

B. The median nerve
C. The musculocutaneous nerve
D. The radial nerve
E. The ulnar nerve

The answer is D, which is the single best answer. The rest of the answers act as
distractors.

(c): An example of a good quality extended matching question (EMQ).

Theme: Lower Limb Ulceration

Options:
A. Deep venous insuficiency
B. Diabetes
C. Lower limb arterial disease
D. Lymphoedema
E. Necrobiosis Lipoidica
F. Neuropathy
G. Non Hodgkins Lymphoma
H. Squamous Cell Carcinoma
I. Vasculitis

Figure 1. Examples of Multiple Choice Questions (MCSs).

656 P.A. Coughlin and C.R. Featherstone

Lead In: For each of the clinical scenarios below, select the single most
appropriate diagnosis from the list of options above.

Stems:
1. An 82 year old man with a history of ischaemic heart disease and a
previous carotid endarterectomy had now developed a painful shallow
ulcer on the dorsal aspect of his foot. (Answer C)
2. A 78 year old woman previously underwent a right total hip replacement
10 years ago since which she has had permanent right leg swelling. Over
the last 7 months she has developed an extensive area of super icial
ulceration over the medial aspect of her calf and ankle. (Answer A)
3. A 93 year old lady with a history of right leg DVT 25 years ago has had a
non healing static ulcer around the right medial malleolus for the last ten
years. It has been resistant to treatment and has maintained a size of
between 1 and 2cm in diameter. Over the last three months it has rapidly
increased in size becoming circumferential and is becoming increasingly
painful. (answer H).

Figure 1. (continued).

higher order concept of integration-interpretation and
problem solving.3 This approach to question writing can
turn the cost effective MCQ into a powerful simulation tool,
increasing both the validity and the educational impact of
the assessments.
In driving education and assessing candidates at the level
of consultant, the aim of an MCQ should be to assess clinical
judgement and problem solving abilities in complex situa-
tions. We want the candidate to be able to assimilate the
pieces of information given within a question and to use their
clinical judgement to determine the most correct answer. The
two most common types of multiple choice questions in
postgraduate medical examinations are single best answer
questions (SBAs) and the extended matching questions
(EMQs). Both of these question types allow assessment of
the candidates’ ability to select the most appropriate option
available, given a clinical situation they would be likely to
encounter in the workplace. The best starting point for
writing a good MCQ is therefore a situation recently
encountered in the workplace in which clinical judgement
was required in determining the appropriate next step.
FORMAT OF MOST COMMONLY USED MCQS
The anatomy of a SBA consists of a stem and a lead-in
question both of which should pose a clear question that
can be answered without looking at the options (Fig. 1B).
Then a list of options should be given, which includes dis-
tractors along with the single best answer. The distractors all
need to be plausible and lie on a continuum of least correct
to most correct. In the example in Fig. 1B, the stem is the
clinical scenario, the lead-in is the specific question being
asked, and there are a series of distractors (all nerves) that
could theoretically be damaged, yet the fact that the
patient has a wrist drop means that the candidate has to
use higher order thinking to work out that the radial artery
is a risk with this type of injury and that the clinical findings
confirm the injury.
The anatomy of an EMQ is somewhat different. EMQs
tend to focus a number of questions/clinical scenarios
around a particular theme. The layout of an EMQ starts with
the identification of the theme, then gives a number of
potential options or answers. There is then a lead-in ques-
tion followed by (usually) a number of different stems. For
each stem, the candidates choose the most appropriate or
likely option. In the example in Fig. 1C, the theme is lower
limb ulceration, the options are listed from A to E and the
lead-in poses a clear question. The stems, in this case a
couple of clinical scenarios, are then given.
THE PITFALLS OF MCQ WRITING
Although perceived to be relatively straightforward, the
writing of a high quality MCQ question often requires a
number of iterations. In addition to the difficulty of
providing enough plausible distractors, there are a number
of pitfalls that must be avoided, either because they give
away the answers to “testwise” candidates, or because they
make the question more complicated than it needs to be
through “irrelevant difficulty.”
“Testwise” candidates typically have good examination
technique, and can sometimes identify the correct answer
to a question purely on test taking skills alone and not
because of underlying knowledge or the ability to apply that
knowledge to real situations. Question writing flaws that
give away the answers to “testwise” candidates are there-
fore a threat to the validity of the assessment. Pitfalls which
play into the hands of “testwise” candidates include
How to Write a High Quality Multiple Choice Question
grammatical cues, logical cues, the use of absolute terms,
longer correct answers, word repeats, and the concept of
convergence strategy. These pitfalls, which are defined and
illustrated in Case and Swanson’s Item Writing Manual1 are
summarised below.
As the question writer naturally pays more attention to
the correct answer rather than the accompanying dis-
tractors, grammatical errors are more likely to occur within
the distractors. As such, care needs to be taken that all
options (best answer and distractors alike) follow on
grammatically and logically from the main stem. On a
similar theme, as the question writer pays more attention
to the correct answer, they are more likely to include
additional instructional material and caveats. Given this,
correct answers may be longer, more specific and more
complete than the other options and therefore easier to
spot as being the right answer. As a basic rule therefore,
answers should be of roughly the same length and should
be as short as possible with most of the detail placed within
the question stem.
If a subset of the options are along a similar theme or
mention a particular word, this can form a logical cue,
highlighting to the candidate that the correct answer is
likely to be within this subset. The use of absolute terms
(e.g. the words “always” or “never”) within a subset of
options typically allows the testwise candidate to eliminate
these options as they are less likely to be true than an
option that has a less absolute statement. Furthermore,
word repeats, namely the use of a word of phrase in both
the stem and in the correct answer will tend to highlight the
correct answer and should be avoided.
Convergence strategy can be used when the correct
answer includes the most elements in common with the
other options. This concept is more subtle than the other
flaws in question writing that have already been discussed
and tends to originate from the question writer starting
with the correct answer and then next writing a number of
allied distractors. As such, the correct answer will have
common elements with the distractors. A very simple
example of this would be if the available options were (a)
aspirin and clopidogrel, (b) aspirin and dipyridamole, (c)
aspirin and ticagrolor, (d) clopidogrel and statin, (e) clopi-
dogrel and a fibrate. A testwise candidate would look at the
answers and by a simple count would see that aspirin and
clopidogrel both appeared three times, more than any of
the other options, and as such the answer would be (a).
While this is an overt simplification it can be seen how such
a situation could arise if the question writer used differing
variations of the correct answer, for a subset of different
steps in a treatment plan, as distractors. Similar to this is
the posing of two questions within a single MCQ, which
often leads to stems including such answers as “both un-
true” or “both true.” Again, “testwise” candidates are more
likely to spot question trends and as such will score highly
on such questions.
“Irrelevant difficulty”, as its name suggests, is used to
describe a question, which is made difficult for reasons that
are unrelated to the trait that is the focus of the
657
assessment. It is therefore also a threat to the validity of the
assessment. Irrelevant difficulty can arise from negatively
phrased questions, the use of “all/none of the above,” long
and overly complicated questions or the use of vague and
imprecise terms.
Negatively phrased questions should be avoided for two
reasons. The first reason is the difficulty caused by requiring
candidates to switch from looking for the best option to
looking for the worst option. The second and arguably more
important reason is that these questions do not mimic
typical reasoning in the workplace: the normal approach is
to look for the best, not the worst outcome for patients.
“All/none of the above” can either be easily discarded or
causes confusion in the candidates, who may wonder
whether an option other than the ones provided by the
question writer should be the correct answer. As candidates
typically have a minute in which to read and answer each
question, long and overly complicated options are in danger
of testing reading speed rather than clinical judgement.
Finally, vague and imprecise terms mean that the candi-
dates have to guess the question writers’ personal defini-
tion of such terms. Indeed, a question is enhanced when
the options include more accurate information including
more specific numerical values wherever possible. When
using numerical answers, these should be standardised e
listed in numeric order and using a similar format wherever
possible. Overall, the best MCQs tend to have straightfor-
ward, short, and easy to read options.
Having explored the theory behind asking “higher order”
MCQs, and the common pitfalls of question writing, what
should question writers aim for, practically speaking, in
producing a good MCQ?
The validity and educational impact of an assessment will
be enhanced if its questions focus on concepts that are
important and relevant either because of the frequency
with which they appear in the workplace or because of the
critical need to know the answer. Having said this, question
writers should bear in mind that not all clinical or scientific
problems are amenable to testing by MCQ. Other assess-
ment tools may provide a more valid measurement,
particularly where the “right answer” is one which is under
discussion in the field, or where the candidate’s approach,
reasoning and/or justification is more important than the
answer they arrive at.
Having identified a learning objective that can be
assessed using an MCQ, question writers should start from
a scenario in which that learning objective would be used.
For example, if candidates need to know about how drug A
interacts with drug B, rather than asking “how does drug A
interact with drug B?,” start with a scenario in which
a patient, whose presentation would normally lead to
prescribing drug B, is already taking drug A. Clinical sce-
narios such as these are often the optimal way of delivering
information within the stem in a way that mimics how in-
formation is presented in the workplace. Such information
will include the presenting problem followed by (where
relevant) a focused clinical history, physical findings, and
diagnostics/imaging. If the question is more related to
658
basic science, a more relevant laboratory vignette may be
required.
The stem must be clear and focused, and should capture
with sufficient authenticity the information that the candi-
date would typically use in the workplace to determine the
best course of action. The lead-in should allow the candi-
dates to answer the question without looking at the op-
tions, to allow them to come up with an answer themselves
before checking it against the list of provided options. The
options must be standardised and consistent with all an-
swers coming from a similar category (e.g. all diagnoses or
all procedures). They also must be plausible and of similar
length and orders in a logical order (often alphabetical).
CONCLUSION
A good MCQ is one that mimics the workplace or a situation
in which the candidate must apply their knowledge, which
elicits the cognitive processes that we want to measure,
and which rewards candidates for demonstrating relevant
skills. It is with these attributes that good MCQs allow us to
P.A. Coughlin and C.R. Featherstone
encourage relevant learning, evaluate whether learning
goals have been reached, and identify areas for develop-
ment in training programmes and in trainees.
CONFLICT OF INTEREST
None.
FUNDING
None.
REFERENCES
1 Case S, Swanson D. Item writing manual. 3rd ed. NBME; 2002
[available online].
2 Naeem N, van der Vleuten C, Alfaris EA. Faculty development on
item writing substantially improves item quality. Adv Health Sci
Educ Theory Pract 2012;17:369e76.
3 Maguire T, Skakun EN, Triska OH. Student thought processes
evoked by multiple choice and constructed response item. In:
The Seventh Ottawa Conference on Medical Education and
Assessment. Maastricht; 1996. p. 253.