ORIGINAL ARTICLE
The Levels of Evidence in Pediatric Orthopaedic
Journals: Where Are We Now?
Megan S. Cashin, MD, FRCSC,* Simon P. Kelley, MBChB, FRCS (Tr and Orth),*
Jeffery R. Douziech, MD,w Renjit A. Varghese, MBBS, MS (Ortho), MHSc (Epi),*
Quinn P. Hamilton, BHK,* and Kishore Mulpuri, MBBS, MS (Ortho), MHSc (Epi)*wz
Background: In recent years, it has become common to publish a
level of evidence grading for orthopaedic journal publications.
Our primary research question is: is there an improvement in
levels of evidence of articles published in pediatric orthopaedic
journals over time? In addition, what is the current status of
levels of evidence in pediatric orthopaedic journals?
Methods: All articles in Journal of Pediatric Orthopaedics-A
(JPO-A) and Journal of Pediatric Orthopaedics-B (JPO-B) for
2001, 2002, 2007, and 2008 and those in Journal of Children’s
Orthopaedics (JCO) for 2007 and 2008, were collected by an
independent reviewer. Of the 1039 articles identified, animal,
cadaveric and basic science studies, expert opinion and review
articles were excluded. Seven hundred fifty remaining articles
were blinded and randomized with respect to journal, title,
publication date, author, and institution. According to the
currently accepted grading system, study type and level of
evidence was assigned to each article. Interobserver and
intraobserver reliability were investigated. Statistical analysis
was carried out using SPSS software.
Results: There were no statistically significant differences in
study type or levels of evidence in articles published before and
after 2003. Of articles published during 2007/2008, 3.0% were
graded as level I, 5.0% as level II, 24.1% as level III, and 58.0%
as level IV. Analysis of the separate journals for all 4 years
revealed that JPO-A published 2.6% (13 of 503) level I studies,
whereas JPO-B published 4.3% (7 of 163) and JCO published
1.2% (1 of 84). The intraobserver reliability was high for study
type (k, 0.842) and substantial for level of evidence (k,
0.613).The interobserver reliability for study type and level of
evidence was high (k 0.921 and 0.860, respectively).
Conclusions: Since the introduction of levels of evidence to
orthopaedic journals in 2003, there has been minimal change in
the quality of evidence in pediatric orthopaedic publications. We
note a modest increase in level III articles and a corresponding
From the *Department of Orthopaedics; wThe Office of Pediatric
Surgical Evaluation and Innovation, British Columbia Children’s
Hospital; and zDepartment of Orthopaedics, University of British
Columbia, Vancouver, British Columbia, Canada.
No external source of funding was used in the preparation of this article.
The authors declare no conflict of interest.
Reprints: Kishore Mulpuri, MBBS, MS(Ortho), MHSc(Epi), Division of
Orthopaedic Surgery, British Columbia Children’s Hospital, A208—
4480 Oak Street, Vancouver, British Columbia Canada, V6H 3V4.
E-mail: kmulpuri@cw.bc.ca.
Copyright r 2011 by Lippincott Williams & Wilkins
J Pediatr Orthop ! Volume 31, Number 6, September 2011
decrease in level IV articles. Articles can be reliably graded by
nonepidemiologically trained individuals.
Level of Evidence: Not applicable.
(J Pediatr Orthop 2011;31:721–725)
n the 1990s, Sackett et al1 introduced the concept of
I “evidence-based medicine.” It was described as “the
conscientious, explicit, and judicious use of the current
best evidence in making decisions about the care of
individual patients.” The trend towards evidence-based
medical literature has resulted in a demand for high
quality publications. In recent years, several steps have
been made to incorporate this concept into the orthopae-
dic research community. In January of 2003, the Journal of
Bone and Joint Surgery, American Volume, (JBJS-Am)
initiated the publication of a level of evidence grading for
each scientific article.2 This marked the first of the
orthopaedic journals to publish levels of evidence.3 Similar
strides have been taken by the pediatric orthopaedic
community to apply this grading system to presentations
at the Paediatric Orthopaedic Society of North America
annual meeting. As of 2007, the level of evidence of each
presented abstract was published in the abstracts book.
The JPO-A soon followed suit, by requiring the inclusion
of a level of evidence grade for each submitted article.
Sackett’s classification system employs a hierarchy
of level of evidence from I to V, with I being the most
rigorously designed studies (ie, randomized controlled
trials).4,5 The grading is further qualified by classifying
the type of study as therapeutic, prognostic, diagnostic, or
economic/decision analysis. The addition of a modifica-
tion of this grading system to orthopaedic periodicals
allows the reader to reliably enhance the critical appraisal
of publications.2,6,7 As well, it allows the journal editorial
board and readers to follow trends in the level of evidence
within its contents.
The increasing use of levels of evidence among these
journals may influence the quality of research. This effect
has not yet been studied within the pediatric literature.
With this in mind, our primary research question is: is
there an improvement in the levels of evidence of articles
published in pediatric orthopaedic journals over time? A
secondary research question is: what is the current status
of levels of evidence in pediatric orthopaedic journals?
www.pedorthopaedics.com | 721
Cashin et al
METHODS
An independent reviewer collected all articles
published in the Journal of Paediatric Orthopaedics-A
(JPO-A) and the Journal of Paediatric Orthopaedics-B
(JPO-B) during 2001, 2002, 2007, and 2008 and those in
the Journal of Children’s Orthopaedics (JCO) for 2007
and 2008. Since the first publication of level of evidence
occurred in 2003, this was chosen as the time point at
which to compare articles.2
Of the 1039 articles, animal and cadaveric studies,
basic science studies, case report, expert opinion, editor-
ials, and review articles were excluded from the study. The
750 remaining articles were blinded with respect to
journal, title, publication date, author, institution of
origin, and, if present, level of evidence designation. The
abstract, introduction, and methods section of each article
were copied into a document to blind the format of the
journal of publication. Each document was assigned a
random numerical code, as determined by a computer
randomization program. Two identical study binders
were compiled of the randomly assorted articles. The
reviewer that blinded and randomized the articles was not
involved in rating the study type per levels of evidence.
Each of the remaining 750 articles was independently
reviewed by 2 nonepidemiologically trained pediatric
orthopaedic fellows. According to the currently accepted
grading system on the JPO website, a study type (thera-
peutic, prognostic, diagnostic, or economic/decision analy-
sis) and level of evidence (I, II, III, or IV) was assigned to
each article (Fig. 1).2 A fifth category was added for articles
that were unclassifiable by this system. The reviewers were
blinded to each other’s ratings. They did not receive training
in or discuss the application and utilization of the
classification system before rating the articles.
The intraobserver and interobserver reliability were
investigated. To test intraobserver reliability, 100 articles
were randomly selected, by a computer randomization
program, from the originally graded 750 articles. The first
author then rated these articles a second time 1 week after
the initial grading of levels of evidence. Interobserver
reliability was measured between the first author’s initial
review and the second author’s review of 100 randomly
selected articles. Disagreement among the 2 authors was
resolved by consensus opinion with the senior author, a
pediatric orthopaedic surgeon with a graduate degree
(Master’s) in epidemiology.
The frequency of each study type and level of
evidence among all analyzed articles was determined.
Individual analysis of articles published in each of the 3
journals (JPO-A, JPO-B, and JCO) was also completed.
A comparison between articles published before and after
2003 was done. Intraobserver and interobserver reliability
was measured using k values. Statistical analysis was
carried out using SPSS software (version 17).
RESULTS
Of the 1039 articles published in JPO-A, JPO-B,
and JCO in 2001, 2002, 2007, and 2008, 750 (72.2%) were
722 | www.pedorthopaedics.com
J Pediatr Orthop ! Volume 31, Number 6, September 2011
included in this study. The reasons for exclusion are
detailed in Table 1. Of the excluded articles 48.1% (139 of
289) were case reports. The majority of studies that met
the inclusion criteria were published in JPO-A (503;
67.1%). As expected, the fewest were in JCO (84; 11.2%),
as its first publication, in March of 2007, was the most
recent of all 3 journals reviewed. The remaining 21.7%
(163) were published in JPO-B.
The included articles were divided into 2 groups.
The pre-2003 group was compiled of those published in
2001 and 2002. The post-2003 group was compiled of
those published in 2007 and 2008. Three hundred ten
(41.3%) articles were in the pre-2003 group and 440
(58.7%) were in the post-2003 group.
Of the articles analyzed over all 4 years, the majority
were therapeutic (363; 48.4%) and most were level IV
evidence (448; 59.7%) (Table 2, Fig. 2). Twenty-one
articles (2.8%) were graded as level I, 44 (5.9%) as level
II, and 162 (21.6%) as level III. Seventy-five (10.0%) were
unclassifiable by the current system.
There were no statistically significant differences in
study type or levels of evidence in articles published
before and after 2003 (Table 2, Table 3). Of the 310
papers in the pre-2003 group, 2.6% (8) were graded as
level I, 7.1% (22) as level II, 18.1% (56) as level III, and
62.3% (193) were level IV. Thirty-one (10.0%) were
unclassifiable. Of the 440 included articles in the post-
2003 group, 3.0% (13) were graded as level I, 5.0% (22) as
level II, 24.1% (106) as level III, and 58.0% (255) as level
IV. Forty-four (10.0%) were unclassifiable.
Analysis of the separate journals for all 4 years
revealed that JPO-A published 2.6% (13 of 503) level I
studies, whereas JPO-B published 4.3% (7 of 163) and
JCO published 1.2% (1 of 84) (Fig. 2). There was no
significant increase in the number of level I studies over
time in JPO-A (Fig. 3). However, there was a decrease in
level IV studies with concomitant increase in level III
studies. In JPO-B, there was an increase in level I and
level III articles, but a decrease in level 2 studies
was noted.
The k values for intraobserver reliability were
moderate for study type (0.842) and substantial for level
of evidence (0.613). The interobserver reliability between
the 2 independent reviewers for study type and levels of
evidence were high, with k 0.921 and 0.860, respectively.
DISCUSSION
Since the introduction of levels of evidence to high
impact orthopaedic journals in 2003, there has been
increased interest in assessment of the levels in subspeci-
alty publications.2 This is the first study to examine the
quality of pediatric orthopaedic research, with particular
attention to the trend in level of evidence over time. This
study demonstrated a low number of level I and II studies
in several high impact pediatric orthopaedic journals. A
lack of improvement in the number of these higher level
studies was noted, when comparing articles published
before and after 2003. There is, however, a trend over
r 2011 Lippincott Williams & Wilkins
J Pediatr Orthop ! Volume 31, Number 6, September 2011 The Levels of Evidence in Pediatric Orthopaedic Journals

FIGURE 1. Levels of evidence for primary research question. This figure is reprinted with permission from JBJS American.2

time toward an increase in the number of level III studies
with an apparent corresponding decrease in level IV
studies.
Surgical research has been plagued by the inherent
difficulty in the implementation of high quality rando-
mized controlled trials (level I evidence).8,9 One of the
most obvious limitations is the inability to blind surgeons
in the majority of surgical trials. More specific to the
pediatric subspecialty are the obstacles encountered
during enrollment of children as study participants.10–12
Our finding of only 2.8% level I studies is consistent with
previous reviews of the general orthopaedic literature that
specifically studied the quality of randomized controlled
studies.8,13 Distinguishing between level I and II studies
may be quite difficult due to poor reporting of the true
methodology.13 This may explain the lower number of
level I articles in our study compared with several other
studies at 11.3%7 and 16.1%,14 which did not include an
appraisal of the true quality of level I studies.
The vast majority of reviewed articles were ther-
apeutic level IV studies. This is a consistent finding
among other high impact general and subspecialty

TABLE 1. Pediatric Orthopaedic Journal Publications Excluded

From Study
TABLE 2. Type of Study in Pediatric Orthopaedic Journal
Category Pre-2003 Pre-2003
Publications Study
Case report 55 (12.8%) 84 (13.8%) Type of Study Pre-2003 Post-2003
Review article 20 (4.7%) 14 (2.3%)
Basic science 19 (4.4%) 26 (4.3%) Therapeutic 142 (45.8%) 221 (50.2%)
Animal study 11 (2.6%) 11 (1.8%) Prognostic 108 (34.8%) 149 (33.9%)
Expert opinion 8 (1.9%) 26 (4.3%) Diagnostic 29 (9.4%) 24 (5.5%)
Cadaveric study 6 (1.4%) 6 (1.0%) Economic/decision analysis 0 (0%) 2 (0.5%)
Anatomic study 1 (0.2%) 2 (0.3%) Unclassifiable 31 (10.0%) 44 (10.0%)
Total 120 (27.9%) 169 (27.8%) Total 310 440

r 2011 Lippincott Williams & Wilkins www.pedorthopaedics.com | 723

Cashin et al J Pediatr Orthop ! Volume 31, Number 6, September 2011

300 the 2 independent reviewers for study type and levels of

250
evidence was almost perfect with k values of 0.921 and
0.860, respectively. This indicates that nonepidemiologi-
cally trained reviewers can reliably employ this grading
Number of Articles

200
150
100
50
7 1 8 4
0
I II III IV
Level of Evidence
FIGURE 2. Level of evidence in each pediatric orthopaedic
journal.
orthopaedic journals.6,7,14,15 One may tend to dismiss this
as “low-level” research, but it is important to realize that
level IV evidence is of significant scientific value.
Although, these case series are uncontrolled, observa-
tional studies with inherent limitations in ability to make
a causal conclusion, they provide a valuable contribution
to the orthopaedic literature. A well-designed case series
may be an important part of the hypothesis-generating
process for development of a higher level study.16 More
specific to pediatric orthopaedic is the relatively low
volume of certain pathologic entities, which makes it
nearly impossible to design a higher level study.
In this study, a modest increase in the number of
level III articles and an apparent corresponding decrease
in level IV articles were noted before and after 2003. This
can be explained by the addition of a control group to a
level IV study, allowing for an increase to level III status.
Many of the level IV studies reviewed in this study would
have been amenable to the addition of a control group,
thus raising their level to III or even II.
This study assessed the intraobserver and inter-
observer reliability of the currently accepted level of
evidence grading system for orthopaedic literature. This
was done using k values as described by Landis and
Koch.17 This system states that k values from 0 to 0.2
indicate slight agreement; 0.21 to 0.40, fair agreement;
0.41 to 0.60, moderate agreement; 0.61 to 0.80, sub-
stantial agreement and 0.81 to 1.0, almost perfect
agreement. According to this system, our results for
intraobserver reliability indicate an almost perfect level of
agreement for study type (0.842) and substantial for level
JPO-A system.
JPO-B
There are several limitations to this study. First,
JCO
animal and cadaveric studies, basic science studies,
editorials and expert opinion, and case reports were
excluded from this study. By the nature of their design,
6 there articles are not amenable to grading as per the
Unclassifiable
current system. It is important to recognize that many of
these studies were high quality. Therefore, the contribu-
tion of these articles is minimized or masked by our
methodology. Second, the introduction of levels of
evidence to orthopaedic journals occurred in 2003, which
is only 5 years before the initiation of this study. A greater
time frame may need to elapse for this change to influence
the number of level I and II publications. Third, the
definitions of “type of study” and “level of evidence” were
not described for surgical or, more specifically, orthopae-
dic research. The grading system has been modified to
facilitate its use by the orthopaedic scientific community.2
This may again mask the quality of publications. Finally,
the current grading system does not address the quality of
studies within each graded level. For example, the
contribution of a well-designed level IV is obscured by a
more poorly designed higher level study.
As the desire for higher quality evidence-based
orthopaedics continues, a working knowledge of the
classification system will allow one to improve the level of
a specific study. Figure 4 illustrates the steps involved in
increasing the level of evidence of a therapeutic study. For
example, the addition of a control or comparison group
to a retrospective case series will increase the grading
from level IV to III, thus creating a case-control or
retrospective cohort study. By collecting data prospec-
tively, a higher level study can again be developed. This
would be a prospective cohort study, designated as a level
II study. To further improve levels of evidence, an
understanding of proper scientific methodology is neces-
sary. As discussed above, surgical randomized controlled
trials are challenging to design and implement. The
methodological details are critical to design a level I
graded randomized controlled trial. These include a
160

of evidence (0.613). The interobserver reliability between 140

120
Number of Articles

JPO-A Pre-2003
100 JPO-A Post-2003
TABLE 3. Level of Evidence Study in Pediatric Orthopaedic 80
JPO-B Pre-2003
JPO-B Post-2003
Journal Publications 60 JCO Pre-2003
JCO Post-2003
Level of Evidence Pre-2003 Post-2003 40

I 8 (2.6%) 13 (3.0%) 20
6725 53 65 6
II 22 (7.1%) 22 (5.0%) 01 04 0 0 0
0
III 56 (18.1%) 106 (24.1%) I II III IV Unclassifiable
IV 193 (62.3%) 255 (58.0%) Level of Evidence
Unclassifiable 31 (10.0%) 44 (10.0%)
Total 310 440 FIGURE 3. Level of evidence in each pediatric orthopaedic
journal pre-2003 (2001/2002) and post-2003 (2007/2008).

724 | www.pedorthopaedics.com r 2011 Lippincott Williams & Wilkins

J Pediatr Orthop ! Volume 31, Number 6, September 2011
Level I
(Randomized Controlled Trial [RCT])
Improve the quality of RCT (eg. >80%
follow-up, proper randomization,
sample size determination)
Level II
(Prospective cohort, Poor-quality RCT)
Collect data prospectively
Level III
(Case-control, Retrospective cohort)
Add control or comparison group
Level IV
(Retrospective case series)
FIGURE 4. Methods for increasing the level of evidence for
therapeutic studies.
pre-study sample size determination, proper randomiza-
tion technique, double blinding, and >80% patient
follow-up. Similar steps can be taken to improve the
quality of prognostic, diagnostic, and economic/decision
analyses type studies (Fig. 1).
In conclusion, this study is the first to investigate the
levels of evidence in pediatric orthopaedic journals. As
the orthopaedic community strives to embrace the
concept of evidence-based medicine, it is critical to evalu-
ate the evolving levels of evidence of pediatric orthopae-
dic research. An accurate assessment of the true quality of
publications is challenging. Precision in the reporting
of study methodology and a working knowledge of the
level of evidence grading system, enhances both the
development and also the surveillance of research.18
“Absence of evidence is not evidence of absence.”19
ACKNOWLEDGEMENTS
The authors thank Dr. Ruth Milner for her statistical
support and Bronwyn Slobogean for her editorial assistance.
r 2011 Lippincott Williams & Wilkins
The Levels of Evidence in Pediatric Orthopaedic Journals
REFERENCES
1. Sackett DL, Rosenberg WMC, Gray JAM, et al. Evidence-based
medicine: what it is and what it isn’t. Br Med J. 1996;312:71–72.
2. Wright JG, Swiontkowski MF, Heckman JD. Introducing levels of
evidence to the journal. J Bone Joint Surg Am. 2003;85-A:1–3.
3. Reider B. Read early and often. Am J Sports Med. 2005;33:21–22.
4. Sackett DL. Rules of evidence and clinical recommendations on the
use of antithrombotic agents. Chest. 1986;89-2(suppl):2S–3S.
5. Phillips B, Ball C, Sackett D, et al. [Oxford Centre for Evidence-
Based Medicine Website]. November 1998. Updated by Jeremy
Howick March 2009. Available at: http://www.cebm.net/?o = 1025.
Accessed December 22, 2009.
6. Bhandari M, Swiontkowski MF, Einhorn TA, et al. Interobserver
agreement in the application of levels of evidence to scientific papers
in the American volume of the Journal of Bone and Joint Surgery.
J Bone Joint Surg Am. 2004;86-A-87:1717–1720.
7. Obremskey WT, Pappas N, Attallah-Wasif E, et al. Level of
evidence in orthopaedic journals. J Bone Joint Surg Am. 2005;87-
12:2632–2638.
8. Bhandari M, Richards RR, Sprague S, et al. The quality of reporting
of randomized trials in the Journal of Bone and Joint Surgery from
1988 to 2000. J Bone Joint Surg Am. 2002;84-A:388–396.
9. Dulai SK, Slobogean BL, Beauchamp RD, et al. A quality
assessment of randomized clinical trials in pediatric orthopaedics.
J Pediatr Orthop. 2007;27-5:573–581.
10. Moss RL, Henry MCW, Dimmitt RA, et al. The role of prospective
randomized clinical trial in pediatric surgery:state of the art?
J Pediatr Surg. 2001;6:1182–1364.
11. Thakur A, Wang EC, Chiu TT, et al. Methodology standards
associated with quality reporting in clinical studies in pediatric
surgery journals. J Pediatr Surg. 2001;36:1160–1164.
12. Curry JI, Reeves B, Stringer MD. Randomized controlled trials in
pediatric surgery: could we do better? J Pediatr Surg. 2003;38:
556–559.
13. Poolman RW, Struijs PAA, Krips R, et al. Does a “level I evidence”
rating imply high quality of reporting in orthopaedic randomized
trials? BMC Med Res Method. 2006;6:44–51.
14. Wupperman R, Davis R, Obremskey WT. Level of evidence in spine
compared to other orthopedic journals. Spine. 2007;32:388–393.
15. Hanzlik S, Mahabir RC, Baynosa RC, et al. Levels of evidence in
research published in The Journal of Bone and Joint Surgery
(American Volume) over the last thirty years. J Bone Joint Surg Am.
2009;91-2:425–428.
16. Brighton B, Bhandari M, Tornetta P, et al. Part 1. Methodological
issues in the design of orthopaedic studies. Hierarchy of evidence:
from case reports to randomized controlled trials. Clin Orthop Rel
Res. 2003;413:19–24.
17. Landis JR, Koch GG. The measurement of observer agreement for
categorical data. Biometrics. 1977;33-1:159–174.
18. Simera I, Altman DG. Writing a research article that is “fit for
purpose”: EQUATOR network and reporting guidelines. Evid Based
Med. 2009;14-5:132–134.
19. Altman DG, Bland JM. Absence of evidence is not evidence of
absence. Aust Vet J. 1996;74:311.
www.pedorthopaedics.com | 725