Journal of Clinical Epidemiology 74 (2016) 237e240
LETTERS TO THE EDITOR
Subjectivity is a strength: a comment on ‘‘an
algorithm was developed to assign GRADE
levels of evidence to comparisons within
systematic reviews’’
Pollock et al. [1] recently published on their use of an
algorithm to facilitate GRADEing of evidence in system-
atic reviews. This approach is problematic. Although the
Grading of Recommendations Assessment, Development
and Evaluation (GRADE) process helps to identify aspects
of a body of evidence which may increase or decrease our
confidence in the evidence, it does not by itself inform us of
the confidence one should have in the evidence. Confidence
is a judgment and is subjective and sensitive to the needs of
the individual evaluating the body of evidence. Thus, the
rating of the overall body of evidence may differ depending
on who is evaluating the body of evidence and for what pur-
pose. What is important is that judgment and the reasoning
behind it be transparently reported. The subjectivity
involved in GRADEing of evidence is an essential feature
of the GRADE system as it allows flexibility and reflects
that different stakeholders have different values. The devel-
opment of algorithms to facilitate GRADEing simplifies an
inherently complex process and does not do justice to the
complexity of the judgments involved. For example, they
are using sample size to presumably assess precision
when evaluation of the confidence interval and judgment
of whether it includes appreciable harm or benefit would
be a more appropriate way to assess precision. Although
their goal is laudable, oversimplification is not the answer.
End users of the evidence, whether they be clinicians, pa-
tients, or guideline panel members, need to understand
the confidence they can place in a body of evidence before
applying it or making recommendations, but the precepts of
evidence-based medicine apply. Evidence alone is not
enough.
Michael R. Gionfriddo
Knowledge and Evaluation Research Unit
Mayo Clinic
200 1st Street SW
Rochester, MN 55905, USA
Mayo Graduate School
Mayo Clinic
200 1st Street SW, Rochester, MN 55905, USA
DOI of original article: http://dx.doi.org/10.1016/j.jclinepi.2015.08.013.
Conflict of interest: None.
0895-4356/Ó 2016 Elsevier Inc. All rights reserved.
*Corresponding author. Tel.: 1-507-266-1754; fax: 1-507-284-
0399.
E-mail address: gionfriddo.michael@mayo.edu
Reference
[1] Pollock A, Farmer SE, Brady MC, Langhorne P, Mead GE,
Mehrholz J, et al. An algorithm was developed to assign GRADE
levels of evidence to comparisons within systematic reviews.
J Clin Epidemiol 2016;70:106e10.
http://dx.doi.org/10.1016/j.jclinepi.2015.11.019
Rating the quality of evidence is by necessity
a matter of judgment
To the Editor:
We read with interest the article by Pollock et al. [1]
about simplifying grades of recommendations, assessment,
development and evaluation (GRADE) and making it more
objective. Although this is a laudable goal, we have major
concerns about the proposed algorithm and the subsequent
misguided conclusions.
First, it needs to be understood that primary contribution
of GRADE is the transparency of approach, which allows
readers to understand the rationale for the judgments
made, even if they disagreed with the perspective taken.
That being said, GRADE has shown to lead to reproducible
results when applied by independent assessors with suffi-
cient competency in GRADE [2]. Second, the relative
importance of the various domains of quality of evidence
varies depending on the situation. For example, lack of
blinding may be important for certain outcomes (e.g.,
pain or quality of life) and less important for other out-
comes (e.g., all-cause mortality). Similarly, interpreting
the I2 as a measure of inconsistency requires careful,
content-specific judgment. For example, setting an arbitrary
cut-off for not rating down for an I2 value of less than 75%,
as the authors suggest, is neither methodologically sound
nor appropriate in all situations. Third, it is problematic
and misleading to include the quality of a systematic review
DOI of original article: http://dx.doi.org/10.1016/j.jclinepi.2015.11.019.
Conflict of interest: The authors disclose that they are members of
GRADE and the US GRADE Network. They have no financial conflicts
of interest.