Critical appraisal criteria

Appraising the quality of cited studies

The aim of Clinical Evidence is to summarise evidence on medical interventions from high
quality systematic reviews and large well-designed randomised controlled trials. Evidence on
prognosis or baseline risk may also come from such studies or from well-designed cohort
studies.

There are numerous checklists for assessing study quality, some of which are summarised
below. No study - or checklist - is perfect, and it is not possible to lay down hard and fast
criteria for inclusion.

For practical purposes it is best to think of three categories of study:

 Methodology sound; INCLUDE

 Methodology suboptimal; INCLUDE if necessary but cite reservations
 Methodology unsound; DO NOT INCLUDE

Quality criteria for included studies

The following criteria are taken primarily from Clinical Epidemiology: a basic science for
clinical medicine, Second Edition. Sackett DL, Haynes RB, Guyatt GH, Tugwell P.

Quality criteria for systematic studies

 Questions and methods clearly stated.

 Comprehensive search methods described.
 Explicit methods used to determine which studies were included in the review.
 Methodological quality of primary studies was assessed.
 Selection and assessment of primary studies reproducible and free from bias.
 Differences in individual study results adequately explained.
 Results of primary studies combined appropriately.
 Reviewers' conclusions supported by data cited.

Quality criteria for randomised controlled trials

 Were the setting and study patients clearly described?

 Was assignment randomised and similarity between groups documented?
 Was allocation to study groups adequately concealed from patients and investigators,
including blind assessment of outcome?
 Were all clinically relevant outcomes reported?
  Were > 80% of patients who entered the study accounted for at its conclusion?
 Were they analysed in the groups to which they were randomised (intention to treat)?
 Were both statistical and clinical significance considered?

Quality for cohort studies on prognosis of baseline risk

 Was an inception cohort assembled?

 Recruitment setting, diagnostic criteria, disease severity, co-morbidity and
demographic details should be documented
 Was the referral pattern described?
 Referral or diagnostics access bias avoided?
 Was an adequate follow up rate achieved?
 > 80% patients entered accounted for in results and clinical status known?
 Were objective outcome criteria developed and used?
 Was outcome assessment blind?
 Was adjustment for extraneous prognostic factors carried out?
 Quality criteria for evidence on harm

The rules of evidence on harm are the same as the rules of evidence on the beneficial effects
of treatment. The best evidence on the harmfulness or otherwise of treatments comes from
large randomised controlled trials (or reviews thereof). However group sizes have to be large
and follow up prolonged for rare side effects to be detected and evidence on harm from RCTs
may not be available. Bias due to non-comparability of groups is more likely in cohort studies
and more likely still in case control studies. Case series or case reports are the weakest forms
of evidence, though associations in case reports have often been subsequently confirmed

When considering evidence on harm

 Was the study the strongest that could have been performed under the circumstances?
 Were study groups sufficiently comparable in respects other than exposure?
 Was determination of exposure free from bias?
 Was the determination of outcomes (in cohort studies) or the distinction between
cases and controls (in case control studies) free from bias?
 Were both clinical and statistical significance considered in reporting the strength of
the association?
 Is the association consistent in different studies?
 Is the temporal sequence of exposure and outcome in the right direction?
 Is there a dose response gradient?
  Does the association make sense?

Updated: 14 June 2006