The purpose of standard setting

Standard setting is a process designed to ensure that an examination is fair and equitable, and
that the same standard is applied across different examination diets. As such, it is a way of
ensuring consistency and fairness when defining the pass mark of a particular examination.

Some examination material is inherently more difficult than others, particularly as it could be
based on less commonly seen conditions. For example a question on precocious puberty may
be more difficult as most trainees will base their answer largely on theory, whereas a question
on the management of pre-eclampsia may be easier as most trainees will have clinical
experience, as well as their reading to draw upon.

Standard setting takes into account the complexity of a task so that the passing standard is
determined by actual candidate performance, rather than determining in advance a set failure
rate for the assessment. Therefore the pass mark and pass rate can fluctuate from exam sitting
to exam sitting because there is no fixed level or quota.

Standard Setting Methods used at the RCOG

Part 1 MRCOG

Ebels Method: This method accounts for the difficulty of each question as well as its
relevance to ST3 practice. Ebels focuses on determining the proportion of borderline
candidates who would respond correctly to each exam question. The borderline candidate is a
hypothetical examination candidate who is of reasonable intelligence, possesses an average
amount of clinical knowledge and who has done reasonable preparation: a candidate who
should be just at the correct standard to pass. For the Ebels method, questions are classified
as easy, medium or hard and their relevance is classified as essential, important or acceptable
which creates nine separate categories. The standard setting panel then decide what
percentage of questions they believe a borderline candidate would answer correctly from each
category.

Part 2 MRCOG

Angoff Method: This method involves a cohort of subject matter experts (O&G Consultants
in NHS practice) evaluating each question and providing an estimate as to how likely the
borderline candidate would know the answer. These estimates are averaged and added to a
standard error of measurement to determine the final pass mark. The standard error of
measurement is a measure of precision of an assessment. It indicates the amount of variability
in a test administered to a group and serves a function of both the standard deviation of
observed scores and the reliability of the test.

Linear Equating: Linear equating is a means of ensuring comparable standards across

subsequent examination diets. This method compares the current results with those of
previous diets and compensates for the relative difficulty of the papers by measuring and
comparing candidate performance on a small number of anchored (repeated) and un-
anchored questions.

Part 3 MRCOG
RCOG Procedure: This method is based on taking the number of examiner judgements in a
particular examination and adding this to an additional factor as well as a standard error of
measurement (as described above). Once an exam score has been set, this is used as the
standard for assessing a competent candidate.

People involved with the standard setting exercise

Standard setters at the RCOG are all in current NHS practice in obstetrics and gynaecology
who have been recruited on the basis of their clinical expertise and their experience in
delivering medical education. The panel reviews the questions testing knowledge of O&G
practice in the UK, bearing in mind the standard a competent trainee should achieve by the
end of their core training. They have undergone specific training in standard setting and all
have a special interest in medical education and examinations.

Regulating the standards of RCOG exams

The GMC are responsible for regulating Royal College exams and all associated assessment
tools and methods. They agree the examination standards and also approve any changes to
exam methods, standard setting and the curriculum. The use of standard setting at the RCOG
bears no relationship to the percentage of candidates who will succeed in the exam, or to any
other external factors. The RCOG works hard to use standard setting is to improve the
fairness and validity of the exam process and to set levels of competence for exam success.
The GMC is sent an annual report about College pass rates of individual membership exams.