See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/221889814

Critical Appraisal Process: Step-by-Step

Article  in  Southern medical journal · March 2012

DOI: 10.1097/SMJ.0b013e31824a711f · Source: PubMed

CITATIONS READS

5 2,132

3 authors:

Donna Timm Daniel Edward Banks

Louisiana State University Health Sciences Center Shreveport Orlando VA Medical Center
20 PUBLICATIONS   108 CITATIONS    109 PUBLICATIONS   1,614 CITATIONS   

SEE PROFILE SEE PROFILE

Jerry Wayne Mclarty

Louisiana State University Health Sciences Center Shreveport
143 PUBLICATIONS   4,192 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Cancer Care Delivery View project

on-going interest in asthma therapy View project

All content following this page was uploaded by Daniel Edward Banks on 30 January 2019.

The user has requested enhancement of the downloaded file.

 Review Article
Critical Appraisal Process: Step-by-Step
Donna F. Timm, MLS, Daniel E. Banks, MD, MS,
Abstract: We present information describing how to search to iden-
tify those reports that provide insight into the answer to the query.
We have presented a reasonable approach to searching, with our end-
point being the identification of published articles which appear to
answer our queries. The decision as to whether these articles are ap-
plicable to the patient under discussion is determined by our clinical
knowledge and the specifics of the patient’s medical concerns. This
process is recognized as critical analysis. Our structure for optimal
searching includes use of the PICO model, formulating a focused
clinical question, and defining key search terms. Using these princi-
ples, we have addressed an example important controversy in the
practice of clinical medicine; in other words, the effectiveness of
screening for prostate cancer and whether it alters the natural history
of this illness.
Key Words: computerized searching, evidence-based medicine
O ne of the six competencies that has been accepted as a part
of postgraduate medical education is practice-based learn-
ing and improvement. Practically speaking, our students are ex-
pected to learn the skills necessary to allow them to keep up with
the changes in medicine and to practice state-of-the-art medicine
as their careers progress. In this article, a step-by-step overview
of the critical appraisal process is examined, including an ex-
planation of a multidisciplinary approach and demonstration
of practical steps in obtaining and appraising clinical studies.
We then review an example of a controversial clinical question
investigated using this approach. The factors to be considered
in critical appraisal are as follows:
1. Study design:
What is the research question?
How were patients identified and recruited?
Who was included? Who was excluded?
Randomization methodology: How was equality of groups ensured?
Was the sample size adequate to answer the research question?
From the Department of Library Science and Medicine, Louisiana State
University Health Sciences Center, Shreveport, and the Brooke Army
Medical Center, San Antonio, TX.
Reprint requests to Daniel E. Banks, MD, MS, Department of Medicine,
Brooke Army Medical Center, 3851 Roger Brooke Dr, San Antonio, TX
78234-6200. Email: dbanks3@LSUHSC.edu
The authors have no financial relationships to disclose and no conflicts of
interest to report.
Accepted January 10, 2012.
Copyright * 2012 by The Southern Medical Association
0038-4348/0Y2000/105-144
DOI: 10.1097/SMJ.0b013e31824a711f
144
Copyright © 2012 The Southern Medical Association. Unauthorized reproduction of this article is prohibited.
and Jerry McLarty, PhD
2. Analysis methodology:
How was the analysis done? Was it appropriate?
Was subgroup analysis done?
Was secondary analysis done?
Did the analysis adjust for multiple testing?
3. Interpretation of results:
Did the study answer the research question?
Are the conclusions supported by the data? Is the argument
convincing?
What questions did the study not address?
What new questions does the study raise?
Is there possible investigator/author bias?
Who financed the study, are there potential conflicts of interest?
How did the authors interact with the study financiers?
The same principles apply whether a study under re-
view is a small clinical trial, a meta-analysis of a series of stud-
ies attempting to answer a singular question, or a large-scale
population-based study.
We recognize that the optimal education of our students
in evidence-based medicine (EBM) ideally requires a team. It
involves the physician who is competent at critical appraisal
and can focus on developing the correct question, and who
possesses the clinical judgment to determine whether the con-
clusion reached can be reasonably applied to the patient; a
librarian who can aid the physician in searching for references
that answer the query; and a statistician who has the skills to
explain the specifics of the tests.
Key Points
& Although the clinician may be competent and able to search
individually, the librarian with skills in structuring the search
and the statistician with a fuller understanding of the tests used
to attain the results can add immeasurably to the process.
& The critical appraisal process is complex. The cornerstones
of evaluating the content of an article begin with questions
about study design, methods used in the analysis, and inter-
pretation of the results.
& Searching for the ‘‘best’’ article requires a step-by-step approach.
It begins with the patient, intervention, comparison(s), and
outcome(s) model, in which the main thrust of the search is
defined by the medical subject headings (MeSH) and major
(Majr) topics (as assigned by the indexers at the National
Library of Medicine). Both are used to decrease the number of
articles retrieved. Information gained from this initial screen-
ing allows the searcher to formulate a clear clinical question
using key search terms with appropriate limits.
* 2012 Southern Medical Association

Roles of the Physician, Librarian,
and Statistician
Physicians have long been aware of the need for devel-
oping skills in searching the literature. Older doctors recall the
monthly issues of the Index Medicus supplements that updated
the list of recent medical publications and the bulky bound vol-
umes of the supplements that were assembled at the end of each
year. Searching for publications addressing a specific clinical
question was often laborious, requiring a review of a series of
monthly and yearly volumes of the Index Medicus. Computeriza-
tion of this process by the National Library of Medicine brought
new challenges. Although many physicians knew the question
they wanted answered, they were now required to become com-
puter literate and to learn the techniques of computer searching.
Enter the medical librarian: initially a support for physi-
cians, but increasingly a partner in the EBM process.1 The role
of the medical librarian has changed dramatically during the last
decade. Once, they managed the written collection, but now
they are required to demonstrate competence at computerized
searching of the literature and in teaching care providers this
evidence-based approach to searching and critical appraisal.
Clinicians have become competent in the clinical process, and
their goal is to make a difference in patient care.2 Librarians
work with physicians to help formulate a clearly structured clini-
cal question. The more precise the question, the easier it is to
identify key terms to be included in a search strategy to retrieve
relevant results. Questions are developed based on the PICO
model: patient, intervention, comparison(s), and outcome(s).3
Once the clinical question is formulated, key search terms are
identified and a search strategy developed. The clinician can
select from a broad number of databases or resources to search,
depending upon the type of information needed. For example,
for more general background questions, students may search for
information in online textbooks or in databases such as UpTo-
Date or DynaMed. For more specific, or foreground, questions,
searching MEDLINE or using the PubMed or Ovid interfaces
may be the best choice. The ‘‘best’’ available evidence may be in
the form of a meta-analysis, systematic review, practice guide-
line, randomized controlled trial, or observational research, such
as cohort studies, case-control studies, and case reports.4
The next step is to evaluate each study for validity and
applicability to patient care. The librarian’s role in the critical
appraisal process is to point students toward tools that help
them evaluate each study type. When the critical appraisal pro-
cess is complete, the clinician must decide whether there is
sufficient convincing evidence to support decisions that change
patient care. Statisticians can aid the physician in understand-
ing the meaning of statistical tests, often providing insight into
the reasonableness of the information as it may apply to the
specific clinical problem and even to his or her patient and his
or her set of unique comorbidities.
The goal, of course, is to make EBM ‘‘doable in every-
day practice.’’5 The very definition of EBM includes clini-
cal knowledge. Physicians base their decisions not only on
Southern Medical Journal & Volume 105, Number 3, March 2012
Copyright © 2012 The Southern Medical Association. Unauthorized reproduction of this article is prohibited.
Review Article
expert opinion but also on the evidence that matters most to
patientsVthat which has the possibility to offer additional treat-
ment options, the possibility of a better outcome, and a chance
to improve their quality of life. In the EBM process, the medi-
cal literature is no substitute for clinical judgment; it is supple-
mental to or supportive of the physician’s clinical judgment.
Evaluating the Quality of Studies
There are numerous critical appraisal worksheets that can
be used to evaluate the quality of various study types. The work-
sheets are freely available from various sources. Worksheets that
we have found particularly helpful originated from the Duke
University Medical Center Library.6 The process can become
somewhat complex as a series of worksheets are available to
assess critical appraisal for therapy, diagnosis, prognosis,
harm, systematic reviews, practice guidelines, differential di-
agnoses, economic analysis, and qualitative study; however,
each worksheet can assist the user in evaluating the validity
of a study based upon criteria unique to each study type.
In addition to the critical appraisal worksheets, there are
scales for rating the quality of primary studies before conduct-
ing a meta-analysis. The Jadad scale7,8 assesses the methods
used for randomization, double blinding, and providing follow-
up of dropouts and those who withdraw from the intervention
group. Each study is scored on the basis of a point system from
1 to 5, with a score of e2 indicating a study of low quality
and Q3 indicating a study of high quality. The Newcastle-
Ottawa scale9 rates various criteria for observational studies.
This scale uses a ‘‘star’’system, in which each study is evaluated
on the criteria of study group selection, group comparability, and
ascertainment of the exposure or outcome of case-control or
cohort studies. Also taken into consideration are the overall
study design, the content, and how easily the study can be
incorporated into a meta-analysis. Stars are assigned to rate
each criterion within each category. The drawback of this scale
is that a threshold score has not been defined to distinguish
between higher- and lower-quality studies. In most instances,
we use the critical appraisal worksheets when we are interested
in ‘‘formally’’ evaluating the studies. The Jadad and Newcastle-
Ottawa scales typically are more useful to researchers who are
writing meta-analyses or undertaking systematic reviews.
Example: The Role of Screening
in the Detection of Prostatic Cancer
Screening for early detection of prostate cancer may be
one of the most contentious topics in primary care medicine.
The crux of the disagreement is the costYbenefit analysis: pro-
state cancer often can be detected at a curable stage with the mea-
surement of prostate-specific antigen (PSA) and digital rectal
examination (DRE). Some studies have found a screening-
related decreased mortality from prostate cancer, but screening
had no apparent effect on all-cause mortality. Large-scale ran-
domized trials have studied the long-term risks and benefits of
prostate cancer screening. A clinician may formulate a search
145
Timm et al & Critical Appraisal Process: Step-by-Step
strategy by completing the PICO model (Table) with a focused
clinical question, key search terms, and MEDLINE searches.
Focused Clinical Question
On the basis of the information shown in the Table, the
searcher may pose the following clinical question: In men
50 years or older, is prostate cancer screening effective for
the early detection of prostate cancer, resulting in a decline in
prostate cancer mortality?
Key Search Terms
The key terms on which the searcher should focus are
prostate cancer, screening, diagnosis, and prevention. The
student should then try those terms in the medical subject
headings (MeSH) database to determine to which medical
subject headings those terms will map.
Sample MEDLINE Searches
Based on this clearly focused question, the following are
two possible search strategies, using MeSH and searching
MEDLINE through the PubMed interface: MeSH example 1:
‘‘Prostatic Neoplasms’’ [Majr] Limits: Humans, Practice Guide-
line, and MeSH example 2: ‘‘Prostatic Neoplasms/Prevention and
Control ’’[MeSH Terms] Limits: Humans, Meta-Analysis. It is
important for the clinician to remember to apply limits to the
search for more targeted results. Because this particular search
involves a question of etiology/harm, the ideal study design is
the randomized controlled trial. The meta-analysis limit is a
good one for this search because the meta-analysis combines
the results of a number of randomized controlled trials on the
topic, analyzing them as if one large study had been conducted.
Another good limit is ‘‘practice guideline’ because that is a
statement by a body of experts on how a condition should be
approached or managed. In this instance, the student should con-
duct searches using both of these limits. When taking an evidence-
based approach to searching the literature, the clinician must
explore a variety of options. This could be viewed as a ques-
Table. PICO model for a 50-year-old man interested
in screening for prostate cancer
PICO MeSH terms
Population/ 50-year-old man who Prostatic neoplasms
problem would like to be screened
for prostate cancer
Intervention Screening Prevention and control
as a subheading
Comparison None No search terms
intervention (if any) needed
Outcome(s) Early detection; decline in No search terms
mortality needed
MeSH indicates medical subject headings; PICO, patient, intervention, com-
parison(s), and outcome(s).
146
Copyright © 2012 The Southern Medical Association. Unauthorized reproduction of this article is prohibited.
tion of diagnosis/screening, in which case, the search may look
like MeSH example 3: ‘‘Prostatic Neoplasms/Diagnosis’’ [Majr]
Limits: Humans, Practice Guideline.
Critical Appraisal of the Retrieved Articles
Andriole GL, Crawford ED, Grubb RL 3rd, et al. Mortal-
ity results from a randomized prostate-cancer screening trial.
N Engl J Med 2009;360:1310Y1319
This study, a part of the Prostate, Lung, Colorectal and Ovar-
ian (PLCO) Cancer Screening Trial sponsored by the National
Cancer Institute, recruited 76,693 men between 55 and 74 years
old (currently the American Cancer Society recommends that
men between 50 and 74 years old be offered a prostate screening
examination). Screening in the PLCO study differed in two sig-
nificant ways from European Randomized Study on Screening
for Prostate Cancer (ERSPC) screening. Both PSA and DRE and
was performed annually. Indication for biopsy was a PSA level
Q4.0 ng/mL or a DRE suspicious for cancer.
Schroder FH, Hugosson J, Roobol MJ, et al. Screening
and prostate-cancer mortality in a randomized European study.
N Engl J Med 2009;360:1320Y1328
The primary question to be answered in the ERSPC trial
was simple: Does PSA-based screening reduce mortality from
prostate cancer? A secondary question asked was whether pro-
state screening affects overall mortality. The ERSPC is actually
a collection of trials from seven countries in Europe with dif-
ferent eligibility criteria, randomization schemes, and plans for
screening and follow-up. This was a randomized clinical trial:
182,000 men between 55 and 69 years old were recruited. Screen-
ing included PSA testing at 4-year intervals without a DRE.
Indications for biopsy differed considerably between countries,
ranging from a PSA 93.0 ng/mL for most countries to PSA
910.0 ng/mL in Belgium.
Randomization is a key issue in clinical trials. The pur-
pose is to eliminate selection bias, in other words, to make
the treated and comparison groups as similar as possible ex-
cept for treatment. It is common to sort patients by important
factors before randomization. This is called stratification. In
addition, within strata, a technique called blocking may be
used to ensure an equal balance of treatment assignments
throughout the recruitment phase of the study. The ERSPC
study stratified on country before randomization but otherwise
did not use stratification or blocking. The PLCO study used
stratification by center and age and randomized by blocks. An
interesting twist on randomization was used in three of the
ERSPC countries. In these three countries, randomization was
done before obtaining informed consent, counter to what is usu-
ally done. This and other factors resulted in slightly uneven
balance of screened/controlled patients in Finland. Also, it was
found that compliance was better in those men randomized
after consent. Both studies were designed with adequate statis-
tical power to detect meaningful differences in prostate cancer
death rates, with adjustment for anticipated compliance rates.
* 2012 Southern Medical Association

Analysis in both studies was based on ‘‘intent to screen’’V
the analysis was performed as if all of the patients complied
with the protocol to which they were randomly assigned. This
is analogous to the ‘‘intent to treat’’ principle for treatment
trials. This valid EBM principle may be important to the dif-
ferences between the two studies. Analytical techniques were
typical for time-to-event data and both studies used similar
methods. Both studies had multiple interim analyses, but only
the ERSPC manuscript reported results with and without ad-
justment for multiple testing.
In the PLCO study, the data and safety monitoring board
(DSMB) stopped the study after only 7 years of complete
follow-up (some patients had as long as 10 years). No statis-
tical futility measures had been reached, but the DSMB
reported a continuing lack of a significant difference in the
death rates between screened and unscreened groups and there
was information suggesting harm from screening. The manu-
script did not present any data about harms attributable to
treatment, overdiagnosis, or overtreatment.
Apparently the DSMB concerns about harm from screen-
ing did not come from the PLCO study but from other sources.
None of the referenced studies of harm from screening were as-
sociated with the PLCO study. The ERSPC study was not stopped
early and reached a median follow-up time of 8.8 years in the
screening group and 9 years in the control group. PSA-based
screening reduced the rate of death from prostate cancer by
20%. More than 17,000 prostate biopsies were performed in
the 73,000 men who participated. No information was provided
for the number of such procedures in the control population, but
it is likely that many more biopsies were done in the experimen-
tal group. Although more diagnoses were made, there is a risk
associated with biopsies and overdiagnosis from screening.10
Attempting to answer the clinical question posed earlier
in the article is not simple. It is striking that the two large ran-
domized clinical trials came to different conclusions concern-
ing the primary research question. The PLCO study concluded
that prostate cancer mortality was not reduced by screening.
The ERSPC study concluded that there was a significant re-
duction in prostate cancer mortality from screening. Reasons
for the disparate conclusions are complex, but it is likely that
issues of patient selection and protocol adherence were among
the most influential.
In the end, the prostate cancer mortality rate has been de-
clining in the United States since the mid-1990s (rates have
fallen approximately 4 %/year since 1992). This date coincides
with the initiation of widescale PSA screening, implying that
the use of this test has played a role in this decline.
The American Cancer Society guidelines now recommend
that men 50 or older with at least a 10-year life expectancy
should have an opportunity to make an informed decision about
prostate screening after receiving information about the risks,
benefits, and uncertainties.11 Even so, with these two reports, it
remains uncertain whether the decline in mortality is the result
of increased screening or other factors.12
Southern Medical Journal & Volume 105, Number 3, March 2012
Copyright © 2012 The Southern Medical Association. Unauthorized reproduction of this article is prohibited.
Review Article
Summary
It is our perspective that team teaching of computerized
searching of the medical literature may be the most successful
approach, yet we fully realize that learners must become able
to independently find and critically appraise studies because
they will not always have access to librarians and statisticians
for assistance in the practice of medicine.13Y16 When teach-
ing EBM concepts, it is important to begin with the necessary
critical appraisal tools that will enable students to evaluate any
study type without the guidance of a professor, librarian, or
statistician. The tools are as follows:
Assess the patient
& Take the patient’s history, perform the physical examination,
and so forth. Check your assessment with the attending physician
before proceeding to the next step.
Ask a clearly formulated clinical question
& Based on your assessment of the patient’s condition, use the PICO
model (problem, intervention, comparison intervention, outcome)
to write a clearly focused question. Use a tool such as PubMed for
Handhelds to guide you through the PICO search.
Acquire the best available evidence
& Determine which databases include the study types that will help
you answer the clinical questionVa point-of-care tool such as
UpToDate or DynaMed or the medical literature (Cochrane Data-
base of Systematic Reviews, PubMed) may be sufficient; however,
further research into the patient’s health issues may be needed.
Appraise the evidence
& Choose the critical appraisal form appropriate for the study type,
and use it to guide you through the critical appraisal process. Forms
are freely available through the Duke University Medical Center
Library’s Web site (http://www.mclibrary.duke.edu/subject/ebm/
searching/ebmresources.pdf).
Apply the evidence
& Use your clinical judgment to decide whether the best available
evidence can be applied to the patient or population in your
clinical setting.
This review has focused on the process. Not every database
is applicable to all situations, although the process is the same
across databases and disciplines. For example, if a database has
a controlled vocabulary, then it is generally best to use it for
more targeted results. In this context, whether one is searching
for information on a clinical medical question or a legal question,
the process is the same. In the legal world, the client’s situa-
tion would be assessed, a clearly focused legal question would
be asked, the best case law would be acquired using a com-
puterized search in Westlaw or Lexis, evaluate the precedents
set by the case law found, and apply it to the client’s situation.
Parallels are easily visible in the practice of clinical medicine.
Obviously, one must choose the appropriate database for the
question asked.
It takes considerable skill in searching to identify the most
relevant articles. It then takes an accomplished physician to use
his or her clinical, epidemiologic, and statistical skills to fully
understand the available data. We recognize that the clinician
often can answer many questions regarding his or her patient
using a computerized search of the literature, but this process
147
 Timm et al & Critical Appraisal Process: Step-by-Step
sometimes can be strengthened by consulting with the librarian
and the statistician.
References
1. Schwing LJ, Coldsmith EE. Librarians as hidden gems in a clinical team.
Med Ref Serv Q 2005;24:29Y39.
2. Klem WL, Weiss PM. Evidence-based resources and the role of librar-
ians in developing evidence-based practice curricula. J Prof Nurs 2005;
21:380Y387.
3. Richardson SW, Wilson MC, Nishikawa J, et al. The well-built clinical
question: a key to evidence-based decisions. ACP J Club 1995;123:A12Y13.
4. Duke University Medical Center Library. Evidence-based medicine
resources. http://www.mclibrary.duke.edu/training/ebm. Accessed September
12, 2011.
5. White B. Making evidence-based medicine doable in everyday practice.
Fam Pract Manag 2004;11:51Y58.
6. Duke University Medical Center Library. Critical appraisal worksheets.
http://guides.mclibrary.duke.edu/content.php?pid=274373&sid=2262324.
Accessed September 12, 2011.
7. Moher D, Jadad AR, Nichol G, et al. Assessing the quality of random-
ized controlled trials: an annotated bibliography of scales and checklists.
Control Clin Trials 1995;16:62Y73.
8. Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports
148
View publication stats Copyright © 2012 The Southern Medical Association. Unauthorized reproduction of this article is prohibited.
of randomized clinical trials: is blinding necessary? Control Clin Trials
1996;17:1Y12.
9. Wells GA, Shea B, O’Connell D, et al. The NewcastleYOttawa Scale
(NOS) for assessing the quality of nonrandomised studies in meta-analyses.
Paper presented at the 3rd Symposium on Systematic Reviews: Beyond the
Basics. Oxford, UK, Centre for Statistics in Medicine, 2000.
10. Barry MJ. Screening for prostate cancerVthe controversy that refuses to
die. N Engl J Med 2009;360:1351Y1354.
11. Wolf AM, Wender RC, Etzioni RB, et al. American Cancer guidelines for
the early detection of prostate cancer: update 2010. CA Cancer J Clin
2010;60:70Y98.
12. Ries LAG, Melbert D, Krapcho M, et al. SEER cancer statistics review,
1975Y2005. Bethesda, MD: National Cancer Institute. http://seer.cancer.gov/
csr/1975_2008. Accessed February 10, 2011.
13. Gruppen LD, Rana GK, Arndt TS. A controlled comparison study of
the efficacy of training medical students in evidence-based medicine lit-
erature searching skills. Acad Med 2005;80:940Y944.
14. McKibbon KA, Haynes RB, Dilks CJ, et al. How good are clinical
MEDLINE searches? A comparative study of clinical end-user and librar-
ian searches. Comput Biomed Res 1990;23:583Y593.
15. van Dijk N, Hooft L, Wieringa-de Waard M. What are the barriers to resi-
dents’ practicing evidence-based medicine? A systematic review. Acad Med
2010;85:1163Y1170.
16. Dinkevich E, Markinson A, Ahsan S, et al. Effect of a brief interven-
tion on evidence-based medicine skills of pediatric residents. BMC Med
Educ 2006;6:1.
* 2012 Southern Medical Association