⏐ FIELD ACTION REPORT ⏐
System for Rapid Assessment of Pneumonia and
Influenza-Related Mortality—Ohio, 2009–2010
| Loren E. Rodgers, PhD, John Paulson, MS, Brian Fowler, MPH, and Rosemary Duffy, DDS, MPH
THE INCIDENCE AND
Rapid mortality surveillance is critical for state emergency pre- severity of seasonal influenza
paredness. To enhance timeliness during the 2009–2010 influ- varies and can be difficult to
enza A H1N1 pandemic, the Ohio Department of Health activated predict.1 Timely mortality sur-
a drop-down menu within Ohio’s Electronic Death Registration veillance is needed to monitor
System for reporting of pneumonia- or influenza-related deaths
severity, allocate resources,
approximately 5 days postmortem. We used International Clas-
and establish control measures.
sification of Diseases—Tenth Revision (ICD-10) codes, available
2–3 months postmortem as the standard, and assessed their During the 2009 influenza
agreement with drop-down-menu codes for pneumonia- or influenza- A(H1N1)2009 pdm09 pan-
related deaths. Among 56 660 Ohio deaths during September demic, the Centers for Disease
2009–March 2010, agreement was 97.9% for pneumonia Control and Prevention (CDC)
(κ = 0.85) and 99.9% for influenza (κ = 0.79). Sensitivity was requested that all 50 states, the
80.2% for pneumonia and 73.9% for influenza. Drop-down menu District of Columbia, New York
coding enhanced timeliness while maintaining high agreement City, and 6 US territories provide
with ICD-10 codes. (Am J Public Health. 2015;105:236–239. weekly estimates of influenza-
doi:10.2105/AJPH.2014.302231) related hospitalizations or deaths
by using either a laboratory-
confirmed or syndromic approach,
with the latter including pneu-
monia or influenza cases.2 This
request was designed to enable
KEY FINDINGS rapid public health responses by
Q Among 56 660 decedents, agreement between drop-down menu codes monitoring the pandemic’s geo-
graphic and temporal attributes
and ICD-10 was 97.9% for pneumonia cases and 99.9% for influenza
while accommodating varying
(κ = 0.85 and 0.79, respectively; P ≤ .001 for both).
capabilities of state health
Q Using ICD-10 codes as a reference, drop-down menu code sensitivity departments and other reporting
was 80.2% for pneumonia and 73.9% for influenza. agencies. Pneumonia cases were
Q Drop-down-menu coding was timely, reporting causes of death ≤ 5 days included to increase the sensi-
postmortem, compared with ICD-10 codes, which are available tivity of influenza surveillance;
although influenza accounts for
approximately 2–3 months postmortem.
approximately 8.5% of pneumo-
Q Drop-down menus required limited resources; implementation required nia cases, noninfluenza causes
16 person-hours for EDRS modification, 2 days for training materials of pneumonia are believed to be
development, and 30 minutes of online training for each of 123 state relatively stable, with changes
registrars. DDM required no additional resources beyond those required in pneumonia incidence typi-
for standard EDRS operation, except time devoted to data input and cally resulting from influenza.1
extraction. Ohio was among 17 reporting
236 | Field Action Report | Peer Reviewed | Rodgers et al. American Journal of Public Health | February 2015, Vol 105, No. 2
agencies that opted to report syn-
dromically defined pneumonia
and influenza-related deaths.
SURVEILLANCE
IMPLEMENTATION
Ohio fatalities are reported by
vital statistics registrars through
the Electronic Death Registra-
tion System (EDRS)3 and include
free-text descriptions written
by certifying physicians that
are subsequently codified by
the National Center for Health
Statistics according to the Inter-
national Classification of Diseases,
Tenth Revision (ICD-10).4 This
process, considered the bench-
mark for mortality classifications,
codifies by using automated
keyword recognition combined
with ad-hoc manual review by
nosologists, and returns up to 20
ICD-10–codified causes of death
per decedent approximately
2–3 months postmortem.5 To
enhance timeliness, a drop-down
menu was implemented in the
EDRS to flag pneumonia- or
influenza-related deaths (Figure
1), reported to the Ohio Depart-
ment of Health 5 or more days
postmortem. Implementation
required 16 person-hours for
EDRS modification, 2 days
for training materials develop-
ment, and 30 minutes of online
 ⏐ FIELD ACTION REPORT ⏐

training for each of 123 state

registrars. With the exception
of the minimal additional time
for drop-down-menu data entry
and analysis, maintenance and
operation of drop-down menus
required no additional resources
beyond those required for stan-
dard EDRS operation.
Within EDRS, registrars were
instructed to use the drop-down
menus to select pneumonia for
immediate or underlying causes
FIGURE 1—The Ohio Electronic Death System user interface. The drop-down menu used by vital
of death that included the terms
statistics registrars to identify pneumonia- or influenza-related deaths is circled. Options included
“pneumonia (without indication
influenza, pneumonia, or neither (mutually exclusive). This menu was activated during September 14,
of influenza),” “pneumonia due
2009–April 4, 2010, and can be rapidly modified to monitor other mortality causes.
to Haemophilus influenza,” or
“pneumonia due to parainfluenza
virus.” Registrars were instructed
not to select “pneumonia” for
TABLE 1–Comparison of Drop-Down Menu Selections with ICD-10 Codified Causes of Death for pneumonitis, aspirational pneu-
Pneumonia-Related Deaths Reported Through the Electronic Death Registration System: Ohio, monia, or pneumococcal menin-
2009–2010 gitis. Registrars were instructed
to select “influenza” if causes of
ICD-10 death included any influenza
With Pneumonia Without Pneumonia Total virus, including the terms “flu,”
(n = 4721) (n = 51 939) (n = 56 660) “influenza,” and “H1N1” or if
both “pneumonia” and “influ-
Drop-down menu—
identified with pneumonia 3788 256 4044 enza” were indicated. We as-
Drop-down menu— sessed agreement of drop-down-
identified without pneumonia 933 51 683 52 616 menu codes with ICD-10 codes
for identifying pneumonia and in-
Note. ICD-10 = International Classification of Diseases–Tenth Revision.4
fluenza-related deaths during the
2009 influenza A(H1N1)2009
pdm09 pandemic.6

DATA SOURCE AND CASE

DEFINITIONS
TABLE 2–Comparison of Drop-Down Menu Selections with ICD-10 Codified Causes of Death for
Influenza-Related Deaths Reported Through the Electronic Death Registration System: Ohio, We queried EDRS on April 1,
2009–2010 2011, to ascertain pneumonia- or
influenza-related ICD-10 codes
ICD-10
for immediate or underlying
With Influenza Without Influenza Total causes of death and drop-down-
(n = 119) (n = 56 541) (n = 56 660) menu code selections for all
Drop-down menu— deaths occurring in Ohio during
identified with influenza 88 16 104 September 21, 2009, to March
Drop-down menu— 31, 2010. Deaths from pneu-
identified without influenza 31 56 525 56 556
monia were defined as records
Note. ICD-10 = International Classification of Diseases–Tenth Revision.4 with ICD-10 codes J12–J18.9
(i.e., pneumonia-specific codes);
cases of influenza were those with

February 2015, Vol 105, No. 2 | American Journal of Public Health Rodgers et al. | Peer Reviewed | Field Action Report | 237
 ⏐ FIELD ACTION REPORT ⏐

TABLE 3–Characteristics of the Electronic Death Registration System Drop-Down Menu Coding for
Pneumonia- or Influenza-Related Deaths: Ohio, 2009–2010
Agreement, %
κ (95% confidence interval)
Sensitivity, %
Specificity, %
Positive predicative value, %
Negative predictive value, %
Note. International Classification of Diseases–Tenth Revision4 codified causes of death functioned as references for comparisons. aAsymptotic
standard error of k for pneumonia = 0.004; P < .001 (2-sided exact test for significance of H0: k = 0).
b
Asymptotic standard error of k of influenza = 0.030; P < .001 (2-sided exact test for significance of H0: κ = 0).
ICD-10 codes J09–J11.8 (i.e.,
influenza-specific codes).
EVALUATION
We used SAS version 9.3
(SAS Institute, Inc., Cary, North
Carolina) and Microsoft Excel
2010 (Microsoft Corp., Red-
mond, Washington) to calculate
agreement, κ statistics, sensitivity,
specificity, and predictive value
of drop-down-menu codes for
pneumonia and influenza-related
deaths occurring in Ohio during
September 21, 2009, to March
31, 2010. ICD-10 codes served
as the standard for comparisons.
We calculated percentage agree-
ment as the number of death
reports that were identically
categorized by both drop-down-
menu codes and ICD-10 codes
(e.g., number of deaths identified
by both systems as having influ-
enza plus the number identified
by both systems as not having
influenza) divided by the total
number of deaths captured by
EDRS. P values for κ statistics
Pneumonia (n = 4721) Influenza (n = 119)
97.9 99.9
0.85 (0.84, 0.86)a 0.79 (0.73, 0.85)b
80.2 73.9
99.5 > 99.9
93.7 84.6
98.2 > 99.9
Among 56 660 deaths re- κ values correct for agreement
ported through EDRS, registrars expected agreement due to
used drop-down menus to select chance, and categorizations have
pneumonia in 4044 (7.1%) and been suggested to describe the
influenza in 104 (0.2%) cases, strength of agreement7; for drop-
respectively; 4721 cases (8.3%) down-menu coding, pneumonia
had pneumonia ICD-10 codes, κ = 0.79 indicates substantial
and 119 (0.2%) had influenza agreement with ICD-10 codes,
ICD-10 codes (Tables 1 and 2). and influenza κ = 0.85 indicates
By using ICD-10 codes, agree- almost perfect agreement. Sen-
ment was 97.9% (κ = 0.85l; sitivity and positive predictive
P < .001) for pneumonia and value were modest, which should
99.9% (κ = 0.79; P < .001) be taken into account when
for influenza (Table 3). Drop- interpreting drop-down-menu
down-menu coding was 80.2% coding data. In certain instances,
sensitive for pneumonia and it might be possible to adjust
73.9% sensitive for influenza. for these phenomena if rates of
Drop-down-menu coding posi- under-ascertainment are predict-
tive predictive value was 93.7% able, although more preferable
for pneumonia and 84.6% for solutions include improving
influenza. sensitivity through additional
registrar training designed to
INTERPRETATION AND enhance recognition of causes of
NEXT STEPS death that met case definitions.
Direct physician electronic entry
Drop-down-menu coding of cause of death into the system
was created to rapidly detect might be explored to further en-
specific causes of death, and hance surveillance.
we determined that the system Integration of drop-down-
maintained high agreement with menu coding into EDRS lever-
of pneumonia and influenza
mortalities to CDC, local health
departments, and the public.
Drop-down-menu coding demon-
strates the value of incorporating
an easily modified selection field
within a fatality reporting system,
and additional applications can
include guiding the deployment
of resources in response to a
health-related event or identify-
ing cases for further study during
acute mass-fatality scenarios.
About the Authors
At the time of the study, Loren Rodgers
and Rosemary Duffy were with the Centers
for Disease Control and Prevention, as-
signed to the Ohio Department of Health,
Columbus. John Paulson and Brian Fowler
were with the Ohio Department of Health,
Columbus.
Correspondence should be sent Loren
Rodgers, PhD, Centers for Disease Control
and Prevention, 1600 Clifton Rd. NE,
Mailstop A-19, Atlanta, GA 30333
(e-mail: lrodgers@cdc.gov). Reprints can
be ordered at http://www.ajph.org by click-
ing on the “Reprints” link.
This article was accepted July 23,
2014.
Note. The findings and conclusions
in this report are those of the authors and
do not necessarily represent the official
position of the Centers for Disease Control
and Prevention or the Ohio Department
of Health.
Contributors
L. Rodgers conducted the analysis,
drafted, and revised the article; R. Duffy
conceived the study and provided
epidemiologic expertise; and J. Paulson
and B. Fowler implemented the system
and provided analytic guidance. All au-
thors reviewed drafts and approved the
final version of this manuscript.
Acknowledgments
We thank Mark Kassouf for help-
ful discussions of drop-down-menu
implementation, Barbara Mattson for
analytic support, and Edward Weiss for
epidemiologic guidance and manuscript
revisions.

were calculated using the 2-sided
exact test for significance of H0:
κ = 0.
Comparison of Drop-Down-
Menu Codes With ICD-10 Codes
ICD-10 codes. Relatively infre-
quent occurrence of cases among
all deaths likely contributed to
high agreement through agree-
ment due to random chance.
aged existing resources and re-
quired limited time or money
beyond preintervention require-
ments. Drop-down menus enabled
rapid and accurate reporting
Human Participant Protection
This study was determined by CDC
to be public health practice and was
exempted from internal review board
review.

238 | Field Action Report | Peer Reviewed | Rodgers et al. American Journal of Public Health | February 2015, Vol 105, No. 2

References
1. Centers for Disease Control and Pre-
vention. Estimates of deaths associated
with seasonal influenza—United States,
1976–2007. MMWR Morb Mortal
Wkly Rep. 2010;59(33):1057–1062.
2. Jhung MA, Davidson H, McIntyre
A, et al. Preliminary results of 2009
pandemic influenza surveillance in
the United States using the Aggregate
Hospitalization and Death Reporting
February 2015, Vol 105, No. 2 | American Journal of Public Health
⏐ FIELD ACTION REPORT ⏐
Activity. Influenza Other Respir Viruses.
2011;5(5):321–327.
3. Ohio Department of Health Elec-
tronic Death Registration System.
Available at: http://vitalsupport.odh.
ohio.gov/GD/Templates/Pages/ODH/
ODHDetail.aspx?page=2&TopicRelati
onID=393. Accessed on December 6,
2013.
4. World Health Organization. Inter-
national Classification of Diseases–10th
Revision. Available at: http://apps.who.
int/classifications/icd10/browse/2010/
en#. Accessed December 11, 2014.
5. Glenn DE. National Center for
Health Statistics (US). Proceedings of
the International Collaborative Effort on
Automating Mortality Statistics, Volume
I. Hyattsville, MD: US Department of
Health and Human Services, Centers
for Disease Control and Prevention, Na-
tional Center for Health Statistics; 1999.
Rodgers et al. | Peer Reviewed | Field Action Report | 239
6. Centers for Disease Control and Pre-
vention. Updated guidelines for evaluat-
ing public health surveillance systems:
recommendations from the Guidelines
Working Group. MMWR Recomm Rep.
2001;50(RR-13):1–35; quiz CE1–7.
7. Landis JR, Koch GG. The mea-
surement of observer agreement
for categorical data. Biometrics.
1977;33(1):159–174.
Copyright of American Journal of Public Health is the property of American Public Health
Association and its content may not be copied or emailed to multiple sites or posted to a
listserv without the copyright holder's express written permission. However, users may print,
download, or email articles for individual use.