S P E C I A L

F E A T U R E

E d i t o r i a l

The Endocrine Society Clinical Practice Guidelines:

A Self-Assessment
Robert A. Vigersky, Shalender Bhasin, and Kathryn A. Martin
Walter Reed National Military Medical Center (R.A.V.), Bethesda, Maryland 20889; Brigham and
Womens Hospital (S.B.), Boston, Massachusetts 02115; and Massachusetts General Hospital (K.A.M.),
Boston, Massachusetts 02114

n 2005, after many years of discussion about whether

or not The Endocrine Society (TES) should produce
Clinical Practice Guidelines (CPGs), TES Council approved a carefully designed program to produce 2 to 4
high-quality CPGs each year. TES has published 20
CPGs to date (1), and several more are expected to be
published in the next 2 years. Since its inception, the
CPG program has been unique in many ways. First, a
highly regulated pathway on the route to publication
(Figure 1) permits the selection, review, and vetting of
CPGs at multiple levels, including a final peer and editorial review at the Journal of Clinical Endocrinology
and Metabolism (JCEM). Second, the guidelines have
been selected and produced without support from pharmaceutical or device-manufacturing companies. Third,
all CPG expert panels have included at least 1 international member. Fourth, all CPGs have been translated
into patient versions. Fifth, the CPGs are routinely reviewed and updated every 3 years. Finally, TES adopted
the Grades of Recommendation Assessment, Development and Evaluation (GRADE) approach for evaluating
the evidence and producing recommendations (2).
The GRADE Working Group began as an international effort to resolve confusion about how to best and
most clearly rate evidence and express recommendations. Many governmental organizations and professional societies use the GRADE approach, including the
Agency for Healthcare Research and Quality, the American College of Physicians, and UpToDate in the United
States; the National Institute for Clinical Excellence and
the British Medical Journal in the United Kingdom; and
the World Health Organization (3). Although there are

multiple systems of grading evidence, the GRADE

method is the only one that rates the strength of the
recommendation (Strong or Weak) based on a balance
of benefits vs harms, the confidence in the magnitude of
estimated effect on an outcome, and the weighing of
patient values and preferences. Although this method
allows for Strong recommendations to be made with
low (L) or very low (VL) levels of evidence and vice
versa, this should not be a frequent occurrence. TES
engaged the Knowledge and Evaluation Research
(KER) Unit at the Mayo Clinic to provide methodological assistance to each of the CPG expert panels, whose
expertise generally is in clinical medicine and not in
methodology or epidemiology.
In this issue of JCEM, members of the KER Unit have
undertaken a unique exerciseassessing the appropriateness of 357 recommendations contained in all 17
CPGs that were published by TES between 2006 and
2011 (4). Their major finding was that 206 of the 357
recommendations were Strong, of which 121 were
based on L/VL levels of evidence. Of the 121, 53 were
in categories such as sensible alternatives do not exist
or recommend additional research. KER Unit members then retrospectively applied a recently published
schema by the GRADE Working Group of paradigmatic
situations where Strong recommendations with L/VL
evidence might be appropriate (5). Using this new taxonomy of paradigmatic situations to determine the appropriateness of the remaining 68 Strong recommendations that were based on L/VL levels of evidence, they
found that it was appropriately applied to 35 recommendations and inappropriately applied to 33 (6 16).

ISSN Print 0021-972X ISSN Online 1945-7197

Printed in U.S.A.
Copyright 2013 by The Endocrine Society
Received May 20, 2013. Accepted May 22, 2013.

Abbreviations: CPG, Clinical Practice Guideline; GRADE, Grades of Recommendation

Assessment, Development and Evaluation; KER, Knowledge and Evaluation Research; L,
low; VL, very low.

For article see page 3246

3174

jcem.endojournals.org

J Clin Endocrinol Metab, August 2013, 98(8):3174 3177

doi: 10.1210/jc.2013-2300

doi: 10.1210/jc.2013-2300

jcem.endojournals.org

3175

Cushings Syndrome; 3 of 27 from

Primary Prevention of Cardiovascular Disease and Type 2 Diabetes in
2. Sub-commi
ee reviews conict-of interest
statements of panel and approves/disapproves
Patients at Metabolic Risk; 4 of 12
members
from Case Detection, Diagnosis,
3. Endocrine Society Council approves topic and
panel
and Treatment of Patients with Primary Aldosteronism; 0 of 17 from
Evaluation and Treatment of Hir4. Panel meets to review evidence and begin draing the
sutism in Premenopausal Women;
guideline . Expert panel formulates up to 2 metaanalyc quesons.
and 1 of 38 from Prevention and
5. KER Unit methodologist and panel review results of
Treatment of Pediatric Obesity).
meta-analysis and incorporate them into dra guideline.
The diversity in the frequency
6. Methodologist assists panel chair with wording of
recommendaon and grading of evidence.
of unjustified recommendations
may be due to the fact that certain
expert panels had more assistance
from the methodologist than others,
that for some CPGs it is inherently
7. Concurrent review of guideline by Clinical Guidelines
Sub-commi
ee, Clinical Aairs Core Commi
ee, Society
more difficult to develop clear recmember (on line), co-sponsoring organizaons (if
ommendations, or that the evaluaapplicable)
tion by the GRADE Working
8. Taskforce reviews, responds, and revises dra
Groups new schema was unequally applied. It is likely that
there are elements of all 3 explanations underlying the findings of
9. Council Reviews and Approves
Brito et al (4).
10. Manuscript submi
ed to JCEM for peer-review
Second, does the analysis by Brito et
11. Panel responds to peer-review and nal manuscript
al (4) diminish the validity or the clinis published
ical importance of those 33 Strong recommendations based on L/VL levels
Figure 1. The Endocrine Societys process of Clinical Practice Guideline Development.
of evidence? At the very worst, these
recommendations represent expert
opinion that, in the absence of highSupplemental Table 1 (published on The Endocrine Soquality evidence, can provide useful and in some instances
cietys Journals Online web site at http://jcem.endojournals.
the only feasible form of guidance for making difficult clinorg) shows those deemed to be inappropriate (personal
ical decisions by cutting through the cacophony of conflictcommunication from Dr Juan Pablo Brito).
ing, weak, or nonexistent data. Many of the 33 recommenA number of important questions arise as a result of this
dations based on L/VL levels of evidence either addressed
self-assessment. First, how well is the CPG program of TES
safety concerns or were related to the interpretation or
doing? It would appear that the CPG program is doing
extremely well overall. Brito et al (4) assert that 27% (33 application of laboratory tests. In addition, it should be
of the 121 Strong recommendations with L/VL evidence) noted that few laboratory tests have undergone extensive
of the recommendations were unjustified. However, it testing in large randomized trials. Historically, there has
may be more informative to look at these 33 recommen- been a paucity of population-based, rigorously derived
dations more globally using either 357 (the total number reference ranges for most hormones and analytes. Alof recommendations) or 306 (357 53 sensible alter- though TES has recently initiated efforts to generate and
natives or required additional research recommenda- harmonize population-based reference ranges for several
tions) as the base. Thus, the percentage of unjustified hormones, it remains the case that the thresholds for derecommendations is 9 or 11%, respectively. Another ap- fining hormone deficiency or hormone excess for most
proach is to assess the number of unjustified recom- endocrine disorders are necessarily based on either a L/VL
mendations in each guideline. For example, in 2008, there level of evidence or expert opinion.
Third, do systematic reviews help expert panels make
were 5 guidelines published with a total of 106 recommendations, of which 12% were unjustified according the correct recommendations? Brito et al (4) express
to the new GRADE schema (5 of 22 in The Diagnosis of concern that strong recommendations are made with
1. Clinical Guideline Sub-commi
ee selects topic
and expert panel chair.

3176

Vigersky et al

Clinical Practice Guidelines: A Self-Assessment

evidence yielding limited confidence in the estimates of

effect about the available options. It might be expected
that systematic reviews would ameliorate this legitimate concern. However, Brito et al (4) found that systematic reviews were used in 14 of 125 (12%) Strong
recommendations that were made with L/VL quality of
evidence. Weak recommendations were made with
L/VL quality of evidence to a similar degree (21 of 122,
or 19%). Thus, it is unclear whether using more systematic reviews, which are often strongly considered in
evaluating the evidence, would be helpful in avoiding
inappropriate recommendations.
Fourth, how does TES compare with other organizations? It would appear that TES compares quite favorably with other organizations in the quality and validity of CPGs. For example, it is reassuring that the rate
at which CPGs of TES apply Strong recommendations
with L/VL evidence is similar to the American Association of Blood Banks and the Society for Vascular Surgery, which made Strong recommendations with L/VL
evidence 64 and 65% of the time, respectively, using the
GRADE system, although their appropriateness was
not addressed (17, 18). Indeed, this may be viewed as an
inherent flaw of the GRADE approach or in its appropriate application. Comparing guideline development
of TES to that of other endocrinology organizations is
difficult because of the differences in methodologies
used. No other endocrinology professional society uses
the GRADE approach, and the criteria for development
and approval of guidelines of other societies are less
rigorous and/or less transparent (19, 20).
Fifth, how can TES improve the quality of its guidelines? Quality improvement is a goal of any organization. Indeed, the KER Units effort is an important and
useful attempt to assess and improve on their own performance as well as to determine how the GRADE system
can best be used to make clinically valid recommendations. TES whole-heartedly embraces the evolving nature of the GRADE guideline development process and
is committed to closely integrating methodological advice into every guideline. By recognizing paradigmatic
situations that better define when Strong recommendations with L/VL evidence are appropriate, TES should
be able to continue to be in the vanguard of producing
the highest quality guidelines for our members and their
patients. Evidence-based practice involves complex decision-making that is based not only on the available
evidence but also on patient characteristics and preferences, values of the patient and physician, and everchanging uncertainties. It is certain that high-quality
evidence will not be available in all clinical situations,
and in situations where high-quality evidence is not

J Clin Endocrinol Metab, August 2013, 98(8):3174 3177

available, expert opinion and careful synthesis of lowquality evidence will continue to appropriately guide
clinical practice.

Acknowledgments
Address all correspondence and requests for reprints to:
Robert A. Vigersky, MD, Walter Reed National Military
Medical Center, Endocrinology and Diabetes Service, 8901
Wisconsin Avenue, Bethesda, Maryland 20889. E-mail:
robert.vigersky@us.army.mil.
Disclosure Summary: R.A.V., S.B., and K.A.M. are the first,
second, and current chairs of the Clinical Practice Guideline Subcommittee, respectively. The opinions expressed in this paper
reflect the personal views of the authors and not the official views
of the United States Army or the Department of Defense.

References
1. The Endocrine Society. Current Clinical Practice Guidelines Web
site. http://www.endocrine.org/endocrine-press/clinical-practiceguidelines. Accessed June 9, 2013.
2. Atkins D, Best D, Briss PA, et al; GRADE Working Group. Grading
quality of evidence and strength of recommendations. BMJ. 2004;
328:1490.
3. GRADE Working Group. Organizations that have endorsed or that
are using GRADE Web site. http://www.gradeworkinggroup.org/
society/index.htm. Accessed May 14, 2013.
4. Brito JP, Domecq JP, Murad MH, Guyatt GH, Montori VM. The
Endocrine Society Guidelines: when the confidence cart goes before
the evidence horse? J Clin Endocrinol Metab 2013;98:3246 3252.
5. Andrews JC, Schunemann JH, Oxman AD, et al. GRADE guidelines
15: going from evidence to recommendation determinants of a
recommendations direction and strength. J Clin Epidemiol. 2013;
66(7):726 735.
6. Melmed S, Casanueva FF, Hoffman AR, et al. Diagnosis and treatment of hyperprolactinemia: an Endocrine Society clinical practice
guideline. J Clin Endocrinol Metab. 2011;96:273288.
7. Abalovich M, Amino N, Barbour LA, et al. Management of thyroid
dysfunction during pregnancy and postpartum: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab. 2007;92:
S1S47.
8. Bhasin S, Cunningham GR, Hayes FJ, et al. Testosterone therapy in
men with androgen deficiency syndromes: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab. 2010;95:2536
2559.
9. Funder JW, Carey RM, Fardella C, et al. Case detection, diagnosis,
and treatment of patients with primary aldosteronism: an Endocrine
Society clinical practice guideline. J Clin Endocrinol Metab. 2008;
93:3266 3281.
10. Speiser PW, Azziz R, Baskin LS, et al. Congenital adrenal hyperplasia due to steroid 21-hydroxylase deficiency: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab. 2010;95:
4133 4160.
11. Nieman LK, Biller BM, Findling JW, et al. The diagnosis of Cushings syndrome: an Endocrine Society clinical practice guideline.
J Clin Endocrinol Metab. 2008;93:1526 1540.
12. Hembree WC, Cohen-Kettenis P, Delemarre-van de Waal HA, et al.
Endocrine treatment of transsexual persons: an Endocrine Society
clinical practice guideline. J Clin Endocrinol Metab. 2009;94:3132
3154.

doi: 10.1210/jc.2013-2300

13. Freda PU, Beckers AM, Katznelson L, et al. Pituitary incidentaloma:

an Endocrine Society clinical practice guideline. J Clin Endocrinol
Metab. 2011;96:894 904.
14. Rosenzweig JL, Ferrannini E, Grundy SM, et al. Primary prevention
of cardiovascular disease and type 2 diabetes in patients at metabolic
risk: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab. 2008;93:36713689.
15. August GP, Caprio S, Fennoy I, et al. Prevention and treatment of
pediatric obesity: an endocrine society clinical practice guideline
based on expert opinion. J Clin Endocrinol Metab. 2008;93:4576
4599.
16. Molitch ME, Clemmons DR, Malozowski S, et al. Evaluation and
treatment of adult growth hormone deficiency: an Endocrine Society

jcem.endojournals.org

17.

18.

19.

20.

3177

clinical practice guideline. J Clin Endocrinol Metab. 2006;91:1621

1634.
Djulbegovic B, Trikalinos TA, Roback J, Chen R, Guyatt G. Impact
of quality of evidence on the strength of recommendations: an empirical study. BMC Health Serv Res. 2009;9:120.
Murad MH, Montori VM, Sidawy AN, et al. Guideline methodology of the Society for Vascular Surgery including the experience with
the GRADE framework. J Vasc Surg. 2011;53:13751380.
Bennett WL, Odelola OA, Wilson LM, et al. Evaluation of guideline
recommendations on oral medication for type 2 diabetes mellitus. A
systematic review. Ann Intern Med. 2012;156:2736.
Vigersky RA. A review and critical analysis of professional societies
guidelines for pharmacologic management of type 2 diabetes mellitus. Curr Diab Rep. 2012;12:246 254.

Download The Endocrine Societys multi-journal, full-text app

to stay up-to-date on all your mobile devices.
Available at Apple App Store https://itunes.apple.com/us/app/endo-pubs/id438308412?mt=8
Available at Android Market https://play.google.com/store/apps/details?id=com.sheridan.tes