Methodological Notes
Cerebrovasc Dis 2013;36:267–272
DOI: 10.1159/000353863
Diagnostic Accuracy Measures
Paolo Eusebi
Department of Epidemiology, Regional Health Authority of Umbria, Perugia, Italy
Key Words
Diagnostic accuracy · Sensitivity · Specificity · Predictive
values · Likelihood ratios · Diagnostic odds ratio · Receiver
operating characteristic · Area under the receiver operating
characteristic curve
Abstract
Background: An increasing number of diagnostic tests and
biomarkers have been validated during the last decades, and
this will still be a prominent field of research in the future
because of the need for personalized medicine. Strict evalu-
ation is needed whenever we aim at validating any potential
diagnostic tool, and the first requirement a new testing pro-
cedure must fulfill is diagnostic accuracy. Summary: Diag-
nostic accuracy measures tell us about the ability of a test to
discriminate between and/or predict disease and health.
This discriminative and predictive potential can be quanti-
fied by measures of diagnostic accuracy such as sensitivity
and specificity, predictive values, likelihood ratios, area un-
der the receiver operating characteristic curve, overall accu-
racy and diagnostic odds ratio. Some measures are useful for
discriminative purposes, while others serve as a predictive
tool. Measures of diagnostic accuracy vary in the way they
depend on the prevalence, spectrum and definition of the
disease. In general, measures of diagnostic accuracy are ex-
tremely sensitive to the design of the study. Studies not
meeting strict methodological standards usually over- or un-
© 2013 S. Karger AG, Basel
1015–9770/13/0364–0267$38.00/0
E-Mail karger@karger.com
www.karger.com/ced
Received: June 18, 2013
Accepted: June 19, 2013
Published online: October 16, 2013
derestimate the indicators of test performance and limit the
applicability of the results of the study. Key Messages: The
testing procedure should be verified on a reasonable popu-
lation, including people with mild and severe disease, thus
providing a comparable spectrum. Sensitivities and speci-
ficities are not predictive measures. Predictive values de-
pend on disease prevalence, and their conclusions can be
transposed to other settings only for studies which are based
on a suitable population (e.g. screening studies). Likelihood
ratios should be an optimal choice for reporting diagnostic
accuracy. Diagnostic accuracy measures must be reported
with their confidence intervals. We always have to report
paired measures (sensitivity and specificity, predictive val-
ues or likelihood ratios) for clinically meaningful thresholds.
How much discriminative or predictive power we need de-
pends on the clinical diagnostic pathway and on misclassifi-
cation (false positives/negatives) costs. © 2013 S. Karger AG, Basel
Introduction
An increasing number of diagnostic tests and bio-
markers [1] have become available during the last de-
cades, and the need for personalized medicine will
strengthen the impact of this phenomenon in the future.
Consequently, we need a careful evaluation of any poten-
tial new testing procedure in order to limit the potential-
194.243.112.67 - 3/11/2014 9:21:51 AM
Paolo Eusebi, PhD
Department of Epidemiology, Regional Health Authority of Umbria
Via M. Angelonii, 61
Downloaded by:
IT–06124 Perugia (Italy)
E-Mail paoloeusebi @ gmail.com
ly negative consequences on both health and medical care
expenditures [2].
Evaluating the diagnostic accuracy of any diagnostic
procedure or test is not a trivial task. In general it is about
answering several questions. Will the test be used in a
clinical or screening setting? In which part of the clinical
pathway will it be placed? Has the test the ability to dis-
criminate between health and disease? How well does the
test do that job? How much discriminative ability do we
need for our clinical purposes?
In the following pages we will try and answer these
questions. We will give an overview of diagnostic accu-
racy measures, accompanied by their definitions includ-
ing their purpose, advantages and weak points. We will
implement some of these measures by discussing a real
example from the medical literature performing an evalu-
ation of the use of velocity criteria applied to transcranial
Doppler (TCD) signals in the detection of stenosis of the
middle cerebral artery [3]. Then we will end the paper
with a list of take-home messages.
Diagnostic Accuracy Measures
Overview
The discriminative ability of a test can be quantified by
several measures of diagnostic accuracy:
– sensitivity and specificity;
– positive and negative predictive values (PPV, NPV);
– positive and negative likelihood ratios (LR+, LR–);
– the area under the receiver operating characteristic
(ROC) curve (AUC);
– the diagnostic odds ratio (DOR);
– the overall diagnostic accuracy.
While these measures are often reported interchange-
ably in the literature, they have specific features and fit
specific research questions. These measures are related to
two main categories of issues:
– classification of people between those who are and
those who are not diseased (discrimination);
– estimation of the posttest probability of a disease (pre-
diction).
While discrimination purposes are mainly of concern
in health policy decisions, predictive measures are most
useful for predicting the probability of a disease in an in-
dividual once the test result is known. Thus, these mea-
sures of diagnostic accuracy cannot be used interchange-
ably. Some measures largely depend on disease preva-
lence, and all of them are sensitive to the spectrum of the
disease in the population studied [4]. It is therefore of
268 Cerebrovasc Dis 2013;36:267–272
DOI: 10.1159/000353863
Table 1. 2 × 2 table reporting cross-classification of subjects by in-
dex and reference test result
Reference test
subjects with subjects with- total
the disease out the disease
Index test
Positive TP FP TP + FP
Negative FN TN FN + TN
Total TP + FN FP + TN Total
great importance to know how to interpret them, as well
as when and under what circumstances to use them.
When we conduct a test, we have a cutoff value indi-
cating whether an individual can be classified as positive
(above/below the cutoff) or negative (below/above the
cutoff), and a gold standard (or reference method) which
will tell us whether the same individual is ill or healthy.
Therefore, the cutoff divides the population of examined
subjects with and without disease into 4 subgroups, which
can be displayed in a 2 × 2 table:
– true positive (TP) = subjects with the disease with the
value of a parameter of interest above/below the cutoff;
– false positive (FP) = subjects without the disease with
the value of a parameter of interest above/below the
cutoff;
– true negative (TN) = subjects without the disease with
the value of a parameter of interest below/above the
cutoff;
– false negative (FN) = subjects with the disease with the
value of a parameter of interest below/above the cutoff
(table 1).
Sensitivity and Specificity
Sensitivity [5] is generally expressed in percentage and
defines the proportion of TP subjects with the disease in
a total group of subjects with the disease: TP/(TP + FN).
Sensitivity estimates the probability of getting a positive
test result in subjects with the disease. Hence, it relates to
the ability of a test to recognize the ill. Specificity [5], on
the other hand, is defined as the proportion of subjects
without the disease with a negative test result in a total
group of subjects without the disease: TN/(TN + FP). In
other words, specificity estimates the probability of get-
ting a negative test result in a healthy subject. Therefore,
it relates to the ability of a diagnostic procedure to recog-
nize the healthy.
194.243.112.67 - 3/11/2014 9:21:51 AM
Eusebi
Downloaded by:
 Both sensitivity and specificity are not dependent on
disease prevalence, meaning that results from one study
could easily be transferred to some other setting with a
different prevalence of the disease in a population.
Nonetheless, sensitivity and specificity may largely de-
pend on the spectrum of the disease. In fact, both sensi-
tivity and specificity benefit from evaluating patients
with more severe disease. Sensitivity and specificity are
good indices of a test’s discriminative ability; however,
in clinical practice a more common line of reasoning is
to know how good the test is at predicting illness or
health: How confident can we be about the disease status
of a subject if the test has yielded a positive result? What
is the probability that the person is healthy if the test is
negative? These questions need predictive values to ad-
dress them.
Predictive Values
PPV define the probability of being ill for a subject
with a positive result. Therefore, a PPV represents a pro-
portion of patients with a positive test result in a total
group of subjects with a positive result: TP/(TP + FP) [6].
NPV describe the probability of not having a disease for
a subject with a negative test result. An NPV is defined as
a proportion of subjects without the disease and with a
negative test result in a total group of subjects with a neg-
ative test result: TN/(TN + FN) [6].
Unlike sensitivity and specificity, both PPV and NPV
depend on disease prevalence in an evaluated population.
Therefore, predictive values from one study should not be
transferred to some other setting with a different preva-
lence of the disease in the population. PPV increase while
NPV decrease with an increase in the prevalence of a dis-
ease in a population. Both PPV and NPV increase when
we evaluate patients with more severe disease.
Likelihood Ratios
LR are useful measures of diagnostic accuracy, but
they are often disregarded, even if they have several par-
ticularly powerful properties that make them very useful
from a clinical perspective. They are defined as the ratio
of the probability of an expected test result in subjects
with the disease to the probability in the subjects without
the disease [7].
LR+ tells us how many times more likely a positive test
result occurs in subjects with the disease than in those
without the disease. The farther LR+ is from 1, the stron-
ger the evidence for the presence of the disease. If LR+ is
equal to 1, the test is not able to distinguish the ill from
the healthy. For a test with only 2 outcomes, LR+ can sim-
Diagnostic Accuracy Measures
ply be calculated according to the following formula:
LR+ = sensitivity/(1 – specificity).
LR– represents the ratio of the probability that a nega-
tive result will occur in subjects with the disease to the
probability that the same result will occur in subjects
without the disease. Therefore, LR– tells us how much
less likely the negative test result is to occur in a subject
with the disease than in a healthy subject. LR– is usually
less than 1, because it is less likely that a negative test re-
sult occurs in subjects with than in subjects without the
disease. For a test with only 2 outcomes, LR– is calculated
according to the following formula: LR– = (1 – sensitivi-
ty)/specificity.
The lower the LR–, the stronger the evidence for ab-
sence of the disease. Since both specificity and sensitivity
are used to calculate the LR, it is clear that neither LR+
nor LR– depend on disease prevalence. Like all the other
measures, LR depend on the disease spectrum of the pop-
ulation under study. LR are directly linked to posttest
probabilities. Here, we concentrate on the posttest prob-
ability of having the disease. The pretest probability is the
probability of having the disease before the test is done. If
no other clinical characteristics are available of the sub-
ject, we can assume the disease prevalence as the pretest
probability. If we have a pretest probability p, we calculate
the pretest odds as a1 = p/(1 – p) and the posttest odds as
a2 = a1 × LR; then, by reverse calculation, the posttest
probability P = a2/(1 + a2).
ROC Curve
It is clear that diagnostic measures depend on the cut-
off used. There is a pair of diagnostic sensitivity and spec-
ificity values for every single cutoff. To construct an ROC
curve, we plot these pairs of values on an ROC space with
1 – specificity on the x-axis and sensitivity on the y-axis
(fig. 1) [8]. The shape of an ROC curve and the AUC helps
us to estimate how high the discriminative power of a test
is. The closer the curve is located to the upper-left corner
and the larger the AUC, the better the test is at discrimi-
nating between disease and health. The AUC can have
any value between 0 and 1 and is a good indicator of the
goodness of the test. A perfect diagnostic test has an AUC
of 1.0, whereas a nondiscriminating test has an area of 0.5
(as if one were tossing a coin).
AUC is a global measure of diagnostic accuracy. It
does not tell us anything about individual parameters,
such as sensitivity and specificity, which refer to specific
cutoffs. A higher AUC indicates higher specificity and
sensitivity along all the available cutoffs. Out of 2 tests
with identical or similar AUC, one can have significantly
194.243.112.67 - 3/11/2014 9:21:51 AM
Cerebrovasc Dis 2013;36:267–272 269
DOI: 10.1159/000353863
Downloaded by:

1.0
0.8
0.6
Sensitivity
0.4
0.2
0 the same sensitivity of the test, the DOR increases with
0 0.2 0.4 0.6 0.8
1 – specificity
Fig. 1. ROC curve. From bottom-left to upper-right corner, we ob-
serve increasing sensitivity and decreasing specificity at lowering
thresholds.
higher sensitivity, whereas the other can have significant-
ly higher specificity. For comparing the AUC of two ROC
curves, we use statistical tests which evaluate the statisti-
cal significance of an estimated difference, with a previ-
ously defined level of statistical significance.
Overall Diagnostic Accuracy
Another global measure is the so-called ‘diagnostic ac-
curacy’, expressed as the proportion of correctly classified
subjects (TP + TN) among all subjects (TP + TN + FP +
FN). Diagnostic accuracy is affected by disease preva-
lence. With the same sensitivity and specificity, the diag-
nostic accuracy of a particular test increases as the disease
prevalence decreases.
Diagnostic Odds Ratio
DOR is also a global measure of diagnostic accuracy,
used for general estimation of the discriminative power
of diagnostic procedures and also for comparison of di-
agnostic accuracies between 2 or more diagnostic tests.
The DOR of a test is the ratio of the odds of positivity in
subjects with the disease to the odds in subjects without
the disease [9]. It is calculated according to the following
formula: DOR = LR+/LR– = (TP/FN)/(FP/TN).
The DOR depends significantly on the sensitivity and
specificity of a test. A test with high specificity and sensi-
270 Cerebrovasc Dis 2013;36:267–272
DOI: 10.1159/000353863
Table 2. Cross-classification of lesions by TCD (MV cutoff >90
cm/s) and angiography (reference test)
Angiography
positive negative total
TCD
Positive 9 7 16
Negative 3 80 83
Total 12 87 99
tivity with low rates of FP and FN has a high DOR. With
the increase in the test’s specificity. The DOR does not
1.0
depend on disease prevalence; however, it depends on the
criteria used to define the disease and its spectrum of
pathological conditions of the population examined.
Example
With the help of a study published by Rorick et al. [10],
we will now apply the diagnostic measures discussed in
the previous section. The purpose of the study was to
evaluate the use of mean velocity (MV) criteria applied to
TCD signals in the detection of stenosis of the middle ce-
rebral artery. The reference test was an angiographic ex-
amination. When the positiveness of TCD was fixed at an
MV cutoff of >90 cm/s, results as can be seen in table 2
were observed (the 2 × 2 table was reconstructed by using
information available from the study). We can calculate
the diagnostic accuracy measures, which are reported in
table 3.
If a test aims at diagnosing a subject, then it is crucial
to determine the posttest probability of the disease if the
test is positive. If we assume that the pretest probability
equals the prevalence of the disease (p = 12/99 = 0.12), the
pretest odds are a1 = 0.12/(1 – 0.12) = 0.14 and the post-
test odds a2 = a1 × LR+ = 0.14 × 9.32 = 1.29, then the post-
test probability of the disease is P = a2/(1 + a2) = 1.29/
(1 + 1.29) = 0.56.
Not surprisingly, the posttest probability equals the
PPV. The reason is that we assumed as pretest probabil-
ity the disease prevalence of the study. Let us suppose
that the pretest probability is higher (0.25). Then, if we re-
peat our calculations, the pretest odds are a1 = 0.25/(1 –
0.25) = 0.33 and the posttest odds a2 = a1 × LR+ = 0.33 ×
9.32 = 3.11, and the posttest probability of the disease is
P = a2/(1 + a2) = 3.11/(1 + 3.11) = 0.76.
194.243.112.67 - 3/11/2014 9:21:51 AM
Eusebi
Downloaded by:
 Table 3. Diagnostic accuracy measures

Measure Formula Calculation Estimate 95% CI

Sensitivity TP/(TP + FN) 9/12 0.75 0.46 – 0.93

Specificity TN/(TN + FP) 80/87 0.92 0.88 – 0.94
PPV TP/(TP + FP) 9/16 0.56 0.35 – 0.70
NPV TN/(TN + FN) 80/83 0.96 0.92 – 0.99
LR+ sensitivity/(1 – specificity) 0.75/(1 – 0.92) 9.32 3.86 – 16.55
LR– (1 – sensitivity)/specificity (1 – 0.75)/0.92 0.27 0.08 – 0.61
Accuracy (TP + TN)/(TP + FP + TN + FN) (9 + 80)/(9 + 7 + 80 + 3) 0.90 0.83 – 0.94
DOR LR+/LR– 9.32/0.27 34.29 6.33 – 214.66

The study also reported results when using a cutoff of

>80 cm/s. Sensitivity increased to 83%, while specificity 1.0
decreased to 87%. We can plot the results in the ROC
space (fig. 2). The lower threshold (MV cutoff >80 cm/s) 0.9
was positioned top right of the higher threshold (MV cut-
MV >80 cm/s
off >90 cm/s). If we were able to have the cutoff along a 0.8
continuum, we would see an ROC curve. The study was Sensitivity
MV >90 cm/s
then reanalyzed in a review of Navarro et al. [11], which 0.7
demonstrated a good TCD performance against angiog-
raphy. 0.6

0.5
Key Messages
0.4
Population 0 0.1 0.2 0.3 0.4 0.5 0.6
The testing procedure should be verified on a reason- 1 – specificity
able population; thus it needs to include those with mild
and severe disease, aiming at providing a comparable
spectrum. Disease prevalence affects predictive values, Fig. 2. Visual display of test results evaluated at an MV cutoff of
but the disease spectrum has an impact on all diagnostic >80 cm/s and an MV cutoff of >90 cm/s in an ROC space.
accuracy measures.

Sensitivity and Specificity, PPV and NPV or LR?

For clinicians and health professionals, the key issue is
to know how a diagnostic procedure can predict a disease.
Sensitivity and specificity are not predictive measures,
they only describe how a disease predicts particular test
results. Predictive values inform us about the probability
of having the disease with a positive test result (PPV), or
the probability of being healthy with a negative test result.
Unfortunately, these probabilities largely depend on dis-
ease prevalence and their meaning can rarely be trans-
ferred beyond the study (except when the study is based
on a suitably random sample, e.g. population screening
studies). While often disregarded, LR should be an opti-
mal choice in reporting diagnostic accuracy, because they
share with sensitivity and specificity the feature of being
independent of disease prevalence, but they can be used
to calculate the probability of a disease while adapting for
varying prior probabilities.
Multiple Thresholds: Paired Measures or ROC
Diagnostic accuracy can be presented at a specific
threshold by using paired results, such as sensitivity and
specificity, or, alternatively, predictive values or LR. Oth-
er methods summarize accuracy across all the different
test thresholds available. In general, it is better to present
both summary and paired results, because a good dis-
criminative power of a test can occur only by chance with
194.243.112.67 - 3/11/2014 9:21:51 AM

Diagnostic Accuracy Measures Cerebrovasc Dis 2013;36:267–272 271

DOI: 10.1159/000353863
Downloaded by:
a particular threshold. In that case, a graphical presenta-
tion can be highly informative, in particular an ROC plot.
At the same time, concentrating only on the AUC when
comparing 2 tests can be misleading because this measure
averages in a single piece of information both clinically
meaningful and irrelevant thresholds. Thus, we always
have to report paired measures for clinically relevant
thresholds.
How Much Discriminative or Predictive Power?
In general, answering this question is dependent on
the stage in which it will be placed in the clinical diagnos-
tic pathway, and on the misclassification cost. For in-
stance, if I need a triage and the cost of FP is of no impor-
tance, I need to focus on sensitivity, PPV or LR+.

Variability
As always, it is crucial to report variability/uncertainty
measures for diagnostic accuracy results (95% CI).

References
1 Whiteley W, Wardlaw J, Dennis M, Lowe G,
Rumley A, Sattar N, Welsh P, Green A, An-
drews M, Graham C, Sandercock P: Blood
biomarkers for the diagnosis of acute cerebro-
vascular diseases: a prospective cohort study.
Cerebrovasc Dis 2011;32:141–147.
2 Koffijberg H, van Zaane B, Moons KGM:
From accuracy to patient outcome and cost-
effectiveness evaluations of diagnostic tests
and biomarkers: an exemplary modelling
study. BMC Med Res Methodol 2013;13:12.
3 Tamura A, Yamamoto Y, Nagakane Y,
Takezawa H, Koizumi T, Makita N, Makino
M: The relationship between neurological
worsening and lesion patterns in patients with
acute middle cerebral artery stenosis. Cere-
brovasc Dis 2013;35:268–275.
4 Montori VM, Wyer P, Newman TB, Keitz S,
Guyatt G, Evidence-Based Medicine Teach-
ing Tips Working Group: Tips for learners of
evidence-based medicine. 5. The effect of
spectrum of disease on the performance of di-
agnostic tests. CMAJ 2005;173:385–390.
5 Altman DG, Bland JM: Diagnostic tests. 1.
Sensitivity and specificity. BMJ 1994; 308:
1552.
6 Altman DG, Bland JM: Diagnostic tests. 2.
Predictive values. BMJ 1994;309:102.
7 Deeks JJ, Altman DG: Diagnostic tests. 4.
Likelihood ratios. BMJ 2004;329:168.
8 Zou KH, O’Malley AJ, Mauri LL: Receiver-
operating characteristic analysis for evaluat-
ing diagnostic tests and predictive models.
Circulation 2007;115:654–657.
9 Glas AS, Lijmer JG, Prins MH, Bonsel GJ,
Bossuyt PM: The diagnostic odds ratio: a sin-
gle indicator of test performance. J Clin Epi-
demiol 2003;56:1129–1135.
10 Rorick MB, Nichols FT, Adams RJ: Transcra-
nial Doppler correlation with angiography in
detection of intracranial stenosis. Stroke
1994;25:1931–1934.
11 Navarro JC, Lao AY, Sharma VK, Tsivgoulis
G, Alexandrov AV: The accuracy of transcra-
nial Doppler in the diagnosis of middle cere-
bral artery stenosis. Cerebrovasc Dis 2007;23:
325–330.

194.243.112.67 - 3/11/2014 9:21:51 AM

272 Cerebrovasc Dis 2013;36:267–272 Eusebi

DOI: 10.1159/000353863
Downloaded by: