[ evidence in practice ]

STEVEN J. KAMPER, PhD1

Reliability and Validity:

Linking Evidence to Practice
J Orthop Sports Phys Ther 2019;49(4):286-287. doi:10.2519/jospt.2019.0702

T
he previous Evidence in Practice article introduced the idea we might ask a patient to answer the
of the “construct,” or what you are interested in measuring, 24 questions of the Roland-Morris Dis-
for example, pain, disability, or strength. As there are often ability Questionnaire (the measure) and
score his or her level of back pain–related
numerous measures available for any given construct, how
disability (construct) by adding up the
do you choose which to use? Of the number of considerations that number of “yes” responses. The patient’s
go into this, none are more important than reliability and validity. score out of 24 is valid to the extent that
Downloaded from www.jospt.org by Springfield College on 04/01/19. For personal use only.

It is no overstatement to say that if a
measure is not both sufficiently reliable
and valid, then it is not fit for purpose.
Reliability
Formally, reliability is the extent to
which a measurement is free from er-
ror. In practice, a reliable measure is one
that gives you the same answer when
you measure the same construct several
times. Consider the example of measur-
J Orthop Sports Phys Ther 2019.49:286-287.
able measures do not provide useful in-
formation; a measure that is not reliable
cannot be valid.
There are several different types of
reliability, each uniquely relevant to situ-
ations in which the measures might be
used. Intrarater reliability refers to the
situation where the same rater takes the
measure on one patient on several occa-
sions, and reliability is the extent to which
the scores from the successive measure-
the questions really reflect the construct
of disability and to the extent that having
difficulty with more of the items reflects
greater disability.
There are several different types of
validity relevant to clinical measures,
the most commonly assessed being con-
struct validity. When researchers assess
the construct validity of a measure, they
are ideally able to compare their mea-
sure to a “gold standard.” For example,

ing height (the construct) with a tape
measure (the measure). You might mea-
sure a person’s height in millimeters 3
times with the tape measure; the extent
to which the number of millimeters is the
same on each occasion is the reliability of
the measure.
The implications of unreliable mea-
sures are serious. If an unreliable diag-
nostic test (measure) was applied to a
patient several times, then the same pa-
tient might be diagnosed as both having
and not having the condition on differ-
ent occasions or by different people. If an
unreliable measure of symptom severity
was collected from a patient before and
after an intervention, then it would be
impossible to tell whether that symptom
improved, stayed the same, or got worse.
Essentially, data collected from unreli-
ments are the same. Interrater reliability
is relevant when multiple raters use the
same measure on a single person, and re-
liability is the extent to which scores from
the different raters are the same.
Validity
Validity is the extent to which the score
on a measure truly reflects the construct
it is supposed to measure. This is rela-
tively straightforward when it comes to
things like height or strength, but waters
quickly become murky when we consider
unobservable or “latent” constructs such
as pain, quality of life, or disability. For
these sorts of constructs, we collect in-
direct measures, such as self-reported
experiences and behaviors, or recall of
beliefs and emotions, and assume that
these reflect the construct. For example,
arthroscopic visualization of the anterior
cruciate ligament is considered a gold
standard of anterior cruciate ligament
rupture, so a study might compare results
from Lachman’s test to the findings from
arthroscopy to assess the validity of Lach-
man’s test.
Unfortunately, there are no gold stan-
dards for many constructs in which we
are interested (eg, latent constructs such
as disability and pain). In these cases,
construct validity is tested against a “ref-
erence standard,” which is a sort of im-
perfect gold standard. When there is no
gold standard, the best way to test valid-
ity is via hypothesis testing. This involves
setting out a series of hypotheses before
collecting the data. These hypotheses are
theorized relationships between a score
on the measure and other characteris-

School of Public Health, University of Sydney, Camperdown, Australia; Centre for Pain, Health and Lifestyle, Australia. t Copyright ©2019 Journal of Orthopaedic & Sports Physical
1

Therapy®

286 | april 2019 | volume 49 | number 4 | journal of orthopaedic & sports physical therapy

 tics, for example, that the score will be
strongly correlated with scores on anoth-
er measure of the same construct and less
strongly correlated with scores on a dif-
ferent but related construct. The extent
to which the data are consistent with the
predetermined hypotheses will be evi-
dence supporting validity of the measure.
Statistics
Testing reliability and validity generally
involves assessing agreement between 2
scores, either scores on the same measure
collected twice (reliability) or scores on
different measures (validity). The statis-
tics used to describe agreement depend
on whether the measures are dichoto-
mous (eg, kappa, sensitivity/specificity)
or continuous (eg, intraclass correlation
Downloaded from www.jospt.org by Springfield College on 04/01/19. For personal use only.
coefficients, correlations, limits of agree-
ment, R2).
Conclusion
There are a couple of important general
points to note about reliability and valid-
ity. First, both are on a spectrum, so mea-
sures are not “unreliable” or “reliable,” but
more or less reliable and more or less valid.
Of course, this makes choosing measures
more difficult; we need to make a subjec-
tive judgment as to whether a measure is
“reliable enough” and “valid enough” in a
particular situation. General guidelines
exist to help interpret reliability and va-
lidity statistics, but these guidelines do not
and should not replace clinical judgment.
The second point is that no measure sits
on the very end of the spectrum; there is
no perfectly reliable and perfectly valid
measure. Even in the case of measuring
height with a tape measure, successive
measures are likely to differ by a few mil-
limeters here and there. Finally, there
are practical concerns when it comes to
choosing a measure, including how long
it takes to administer, whether the patient
can comprehend text or instructions, and
how data will be stored and used.
Measurement is an entire field of re-
search by itself. Although the general
concepts are quite straightforward, you
do not have to scratch too far below the
surface before things become complicat-
ed. When reading research, you should
look for information that reassures you
that the measures used are sufficiently re-
liable and valid. The take-home message:

be very cautious about using, or trying to

“Essentially, data collected from unreliable interpret, information from a measure if
you have no information about its reli-
measures do not provide useful information.” ability and validity. t
J Orthop Sports Phys Ther 2019.49:286-287.

BROWSE Collections of Articles on JOSPT’s Website

JOSPTs website (www.jospt.org) offers readers the opportunity to browse
published articles by Previous Issues with accompanying volume and issue
numbers, date of publication, and page range; the table of contents of the
Upcoming Issue; a list of available accepted Ahead of Print articles; and
a listing of Categories and their associated article collections by type
of article (Research Report, Case Report, etc).

Features further curates 3 primary JOSPT article collections:

Musculoskeletal Imaging, Clinical Practice Guidelines, and Perspectives
for Patients, and provides a directory of Special Reports published
by JOSPT.

journal of orthopaedic & sports physical therapy | volume 49 | number 4 | april 2019 | 287