International Journal of Infectious Diseases 77 (2018) 29–33

Contents lists available at ScienceDirect

International Journal of Infectious Diseases

journal homepage: www.elsevier.com/locate/ijid

Evaluating the frequency of operational research conducted

during the 2014–2016 West Africa Ebola epidemic
Christopher Hurtadoa,b , Diane Meyera,* , Michael Snydera , Jennifer B. Nuzzoa
a
The Johns Hopkins Center for Health Security, Department of Environmental Health and Engineering, Johns Hopkins University Bloomberg School of Public
Health, 621 East Pratt Street, Suite 210, Baltimore, MD 21022, USA
b
University of Central Florida College of Medicine, FL, USA

A R T I C L E I N F O A B S T R A C T

Article history: Introduction: The West Africa Ebola epidemic of 2014–2016 was the largest Ebola outbreak on record, and
Received 30 August 2018 thousands of individuals were involved in the response, including local and national governments, as
Received in revised form 26 September 2018 well as numerous philanthropic and other non-governmental organizations. A number of after-action
Accepted 29 September 2018
reports and other reviews of the global response to the epidemic routinely pointed out key challenges,
Corresponding Editor: Eskild Petersen, Aar-
hus, Denmark
including gaps in operational research.
Methods: To determine the extent to which operational research studies were conducted during the
2014–2016 West Africa Ebola epidemic, a quantitative analysis of the literature published during and
Keywords:
Operational research
immediately after the epidemic was conducted. The goal was to identify the proportion of all Ebola-
Ebola related publications released regarding the epidemic that addressed operational aspects of the response.
outbreak It was also sought to describe, at a general level, the sorts of studies that were published during the
infectious disease epidemic, with the goal of increasing understanding of whether additional efforts are needed to
encourage the conduct and dissemination of operational studies during future public health crises.
Results: Among the 3681 publications on Ebola published between the World Health Organization
announcement of the Ebola outbreak in March 2014 and the end of 2017, 109 (3%) were determined to be
operational research publications. Among these, 64 (58%) were published after the World Health
Organization initially declared the outbreak over on January 14, 2016, reflecting the time delay of sharing
operational lessons with the broader preparedness and response community.
Discussion: Improved sharing of firsthand, operational knowledge from practitioners who respond to
outbreaks is critical for improving preparedness activities and informing the development of sound,
effective policies that support ongoing and future preparedness efforts. Based on the results from this
review, we propose several policy and programmatic innovations that could facilitate knowledge sharing
during future outbreaks.
© 2018 The Authors. Published by Elsevier Ltd on behalf of International Society for Infectious Diseases.
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-
nc-nd/4.0/).

Introduction
The West Africa Ebola epidemic of 2014–2016 was the largest
recorded outbreak of Ebola virus disease (EVD) since it was first
discovered in 1976 (CDC, 2018). Thousands of individuals were
involved in the response, including local and national govern-
ments, as well as numerous philanthropic and other non-
governmental organizations. For example, the United States
Centers for Disease Control and Prevention (CDC) deployed nearly
2000 staff to countries affected by the epidemic to help with
* Corresponding author.
E-mail address: dmeyer10@jhmi.edu (D. Meyer).
response efforts (Bell, 2016). With the vast number of governmen-
tal and non-governmental agencies contributing to the Ebola
response, there were likely many opportunities for responders on
the ground to capture operational data that could have informed
efforts to slow or stop the spread of the current outbreak, as well as
inform preparedness for future outbreaks.
A study by Ballabeni and Boggio demonstrated a marked
increase in the number of Ebola-related publications in the
biomedical literature during 2014 (Ballabeni and Boggio, 2015);
however, it is unclear to what extent these publications
contributed to the response itself. Rather, there is evidence that
suggests that numerous barriers hindered the conduct of
operational research—that is, analyses meant to “identify inter-
ventions, strategies, or tools that can enhance the quality,

https://doi.org/10.1016/j.ijid.2018.09.027
1201-9712/© 2018 The Authors. Published by Elsevier Ltd on behalf of International Society for Infectious Diseases. This is an open access article under the CC BY-NC-ND
license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
30 C. Hurtado et al. / International Journal of Infectious Diseases 77 (2018) 29–33
effectiveness or coverage” of epidemic control efforts (e.g., disease
surveillance, infection control, non-pharmaceutical interven-
tions)—during the response (Fitzpatrick et al., 2017). For example,
an analysis published by Médecins Sans Frontières (MSF), an
organization that played a major role in treating EVD patients
during the epidemic, specifically noted the following challenges
facing publication efforts: collecting standardized data across the
response organization; finding journals that would “review,
accept, and publish the results in an appropriate timeframe”;
and resolving “competing work priorities [that] resulted in delays
to some publications because authors were slow to finalize
manuscripts” (Fitzpatrick et al., 2017).
A number of after-action reports and other reviews of the global
response to the epidemic also identified gaps in operational
research as a key challenge during the response (World Health
Organization, 2015a; Moon et al., 2015; National Academy of
Medicine, 2016; United Nations General Assembly, 2016). Coltart
et al., in their review of the 2014–2016 West Africa Ebola response,
concluded that there continues to be a “lack of data to inform a
robust, evidence-based approach to control Ebola outbreaks”
(Coltart et al., 2017). The authors noted that gaps in knowledge are,
in part, due to insufficient prospectively collected data on outbreak
response upon which to base a rigorous evaluation of disease
control measures.
Operational research has been defined as “the search for
knowledge on interventions, strategies, or tools that can enhance
the quality, effectiveness, or coverage of programmes in which the
research is being done” (Zachariah et al., 2009). Although this
definition is intended for ongoing public health programs, it offers
a good guide to thinking about the goals of conducting operational
research during outbreaks and epidemics. In particular, the
emphasis on improving the quality and effectiveness of inter-
ventions is consistent with the recommendations that arose in
assessments of the response to the Ebola epidemic.
To determine the extent to which operational research studies
were conducted during the Ebola epidemic, we conducted a
quantitative analysis of literature published during and immedi-
ately after the epidemic. The goal was to identify the proportion of
all Ebola-related publications released regarding the 2014–2016
West Africa Ebola epidemic that addressed operational aspects of
the response. It was also sought to describe, at a general level, the
sorts of studies that were published during the epidemic, with the
goal of increasing understanding of whether additional efforts are
needed to encourage the conduct and dissemination of operational
studies during future public health crises.
Methods
The advanced search function of the National Center for
Biotechnology Information (NCBI) database was used to find
articles that relayed operational lessons learned from the 2014–
2016 West Africa Ebola epidemic. The initial search (search 1)
attempted to identify all articles indexed in the NCBI/PubMed
online database that contained the word “Ebola” in the title. The
articles were filtered for English language only and included those
published between March 23, 2014 (the date on which the World
Health Organization (WHO) publicly announced the outbreak
(WHO, 2014)) and December 31, 2017. For the second search
(search 2), articles were identified that contained “Ebola” in the
title, as well as at least one of several operational key terms:
“Lesson”, “Response”, “Operational”, or “Field”, to identify those
articles that focused specifically on the operational aspects of the
response. Both searches were initially conducted using the
PubMed Advanced Search Builder categories “All Fields”, “Title/
Abstract”, and “MeSH Major Topic”; however, many of the articles
retrieved through these filters were unrelated to lessons learned
from the West Africa Ebola epidemic. For this reason, both
searches were limited to only articles retrieved through the
“Title” search filter. Duplicate titles were manually removed from
the search results.
For the purposes of this study, an operational research
publication was defined as a publication that related to the
challenges faced and strategies and resources needed for disease
detection and control, including deployment and use of medical
countermeasures, such as diagnostics; establishing disease sur-
veillance systems for event detection and management; risk
communication and engagement with the affected community;
and strategies for limiting disease spread. The following articles
were excluded: those reporting clinical trials or laboratory-based
experiments (e.g. animal models of infection); basic epidemiologi-
cal analyses or epidemiology of cases (e.g., case counts that did not
also describe strategies undertaken to control disease spread); and
clinical evaluation of treatments or case management protocols. In
addition, articles that generally described broad lessons learned
that did not appear to be authored by an organization involved in
the response to the epidemic were also excluded.
In classifying articles as either operational research or not, two
researchers were assigned to screen each article identified through
search 2, and both researchers read the abstract to determine
whether the article qualified as an operational research publica-
tion under this definition. In the event that the classification was
not clear from the abstract or an abstract was not provided, the
researchers read the entire article. One researcher made initial
classifications for all of the identified articles. The articles were
then divided evenly among the remaining four researchers, and
each made their own classifications, blinded to the first
researcher’s determination. Incongruent classifications were ana-
lyzed by a third researcher to determine a final designation. In
addition to classifying the article as operational or not, the date of
publication was also noted and recorded. If both an electronic and
print date was provided, the earlier date was recorded.
Results
Search 1 identified a total of 3681 publications registered in the
NCBI/PubMed database that contained “Ebola” in the title. Among
Ebola articles published during the study period, 349 (9%) also
contained one or more of the operational key terms “Lesson”,
“Response”, “Operational”, or “Field” (Search 2) (Figure 1). Of these
articles, 109 (3% of the total number of Ebola publications) were
deemed to be operational publications after review by researchers.
Figure 1. Percentage of Ebola operational publications published between March
23, 2014 and December 31, 2017.
 C. Hurtado et al. / International Journal of Infectious Diseases 77 (2018) 29–33
Figure 2. Percentage of Ebola operational articles published during and after the
2014–2016 West Africa Ebola outbreak.
Of the total operational publications, 64 (59%) were published after
the WHO first officially declared the epidemic to be contained on
January 14, 2016 (Kupferschmidt, 2016) (Figure 2).
Discussion
This analysis appears to represent the first effort to identify
operational research conducted during and after the 2014–2016
West Africa Ebola outbreak. Although over 3500 publications that
had Ebola in the title and were written during and after the 2014–
2016 West Africa Ebola outbreak were identified, the vast majority
of these publications could not be classified as operational
according to the definition adopted. Rather, the majority of papers
published during and immediately following the outbreak
pertained to high-level assessments of the overall response efforts,
disease pathogenesis and epidemiology, or laboratory-based
studies or clinical trials aimed at countermeasure and diagnostics
development. Although many of these papers contained data that
could ultimately contribute to an enhanced understanding of the
management of EVD, additional work would be needed to translate
these studies into knowledge, practices, or strategies for contain-
ing the disease in an outbreak setting. Additionally, operational
research studies tended to be published later in the study period—
the majority of articles were published after the epidemic was
declared over, a finding consistent with the analysis conducted by
MSF.
The findings of this study should in no way discredit the
important role that other research studies (e.g., clinical, epidemi-
ological) play in improving overall preparedness. An enhanced
understanding of disease pathogenesis and epidemiology is
essential to understanding how to control disease spread and
improve patient outcomes. Similarly, clinical studies are also
needed to develop medical countermeasures, such as vaccines and
diagnostics, which will be essential for rapid containment and
prevention of future outbreaks of Ebola (The Prioritized Research
Agenda published by the WHO Ebola Science Committee supports
the need for additional basic science and clinical studies (World
Health Organization, 2015b); for example, additional studies are
needed to inform the development of therapies to treat Ebola and
reliable rapid diagnostics that can be used in outbreak settings).
31
Nevertheless, it is believed that the present analysis provides
quantitative evidence to support the need for additional peer-
reviewed studies that can inform the development of public health
response operations to improve the detection and containment of
future Ebola outbreaks and other similar outbreaks. Our conclusion
that additional operational research studies are needed to inform
the response to future Ebola outbreaks is consistent with the
findings of the WHO Ebola Science Committee (World Health
Organization, 2015b). Based on this analysis and others, it is
believed that more data are needed to inform public health efforts
to implement non-pharmaceutical interventions, limit person-to-
person transmission, protect healthcare workers caring for Ebola
patients, and develop and implement risk communication strate-
gies for communities affected by these kinds of outbreaks. A
number of these challenges arose again during the latest Ebola
outbreak in the Democratic Republic of the Congo that began in
May 2018, as did additional challenges of dispensing and
evaluating a new Ebola vaccine (Kalenga, 2018). Taken together,
this suggests that a number of questions about how best to respond
to significant outbreaks persist.
There may be many reasons why more operational research
studies did not appear in the peer-reviewed literature during the
study period. Competing demands for limited resources, such as
performing activities or conducting epidemiological studies that
are critical to a successful response, may prevent emergency
responders from conducting operational research in the midst of a
crisis (Fitzpatrick et al., 2017). Additionally, an absence of rapid and
agile funding mechanisms may serve as a barrier for academic
investigators or other teams with operational research skills to
design and implement studies during public health emergencies
(FMPHPCE et al., 2015). Finally, there is evidence that operational
research may be underutilized in public health even in non-crisis
conditions. One review found that, even among programs that
permit allocating as much as 10% of their budget to be used for
operational research (such as the Global Fund to Fight AIDS, TB, and
Malaria), some report actually utilizing less than 3% for this
purpose (Royston, 2011).
This study was subject to certain limitations. First, there was no
way to determine what would constitute an appropriate propor-
tion of operational publications, so it was not possible to assess
whether the proportion of operational research studies identified
in this study was sufficient. Secondly, this analysis does not take
into account that operational lessons learned from outbreak
responses are often published outside of the peer-reviewed
literature (e.g., in after-action reports published by responder
organizations). This limitation was recognized at the onset of the
analysis, but we chose to look only at the scientific literature for
several reasons. First, it was felt that it was important to
understand how frequently operational research studies that are
subject to the rigors of peer review are published during and after
an outbreak like Ebola. Second, information that is published after
an outbreak response (e.g., in after-action reports) may not be
shared publicly or may focus on lessons specific to the organization
or location where the response occurred rather than address
broadly applicable lessons about outbreak response operations.
Additionally, after-action reports may be ‘sanitized’ for political
purposes in order to provide more favorable external perception of
response activities and organizations.
This analysis provides additional evidence to support the need
for dedicated efforts to track the operational challenges and best
practices needed to improve future preparedness and response
efforts. As noted by the National Academy of Medicine, there
continues to be a need for data to improve preparedness and
response to public health emergencies (FMPHPCE et al., 2015). The
absence of dedicated mechanisms and resources to conduct
operational research during public health emergencies can mean
32 C. Hurtado et al. / International Journal of Infectious Diseases 77 (2018) 29–33
that valuable operational data are often lost and key preparedness
and response questions remain unanswered.
Although this analysis focused on one significant event, the
potential policy implications are likely to apply to outbreaks of
other diseases and in other settings. Since the Ebola epidemic of
2014–2016, a number of high-profile outbreaks, including of Zika
virus, Nipah virus, plague, and Lassa fever, have occurred and have
demonstrated continuing gaps in knowledge and resources for
controlling such events. Knowledge gained from previous out-
breaks may have helped response efforts during these outbreaks,
and dedicated efforts to learn from these events may have helped
to improve the response to future events.
The results of this analysis suggest that dedicated efforts are
needed to support the rapid conduct and timely publication of
operational research during outbreaks and epidemics. If those with
direct operational responsibilities for responding to outbreaks are
too busy to design, implement, write-up, and publish research
studies during an outbreak, partnerships with academic or other
outside organizations may be helpful to ensure that such studies
are completed. However, pre-event planning will be necessary to
enable outside partners to gain access to information and data
needed to conduct operational research without hindering
response operations. This will likely require flexible and timely
funding mechanisms, rapid but thorough ethical review processes,
and preexisting partnerships between researchers and practi-
tioners with operational responsibilities during outbreaks. Au-
thorship and data access and use agreements will also have to be
negotiated pre-event. In addition, the creation of a research agenda
that specifically identifies and prioritizes operational research
questions may help to support the development and implementa-
tion of research efforts that are best matched to the informational
needs of practitioners.
National governments and philanthropic donors that have an
interest in supporting the development of outbreak response
capacities should consider creating dedicated funding opportuni-
ties and guidelines for researchers and practitioners to conduct
operational research during outbreaks and epidemics in a variety of
settings. Some models currently exist, including the Rapid Research
Fund for Ebola Virus Disease Outbreaks, which is sponsored by the
Government of Canada and supports collaborations between
Canadian and African research teams (IDRC, 2018). In the USA,
the National Science Foundation has also provided ad-hoc funding
to support research during certain emergencies, for example
Hurricane Harvey (NSF, 2017). While such efforts target the need
to conduct rapid research during emergencies, the existence of
these opportunities is not fully sufficient to address the dearth of
operational research. Additionally, standing sources of support are
needed to enable participation in operational research by experts
and practitioners from across the globe. Ideally, dedicated mecha-
nisms and resources would also exist that allow prospective
researchers representing a range of expertise from a diversity of
settings (e.g., including low- and middle-income countries) with
specific research goals in mind to develop hypotheses and design
operational studies in advance of an outbreak, which can then be
tested once an appropriate outbreak is identified.
Finally, although this analysis suggests that additional efforts
are needed to increase the frequency with which operational
research is conducted during outbreaks, the findings of such
studies need to be translated into practice. Though operational
research is necessary to improve the evidence base for outbreak
preparedness and response, evidence is not, in itself, an endpoint.
While the creation of funding mechanisms, the establishment of
partnerships and agreements between researchers and practi-
tioners, and the development of data use and publication
agreements facilitate the conduct of additional operational
research studies, published research should be disseminated
and their recommendations communicated to practitioners to
enable them to take timely and appropriate action. Practitioners
will also need resources to develop and adapt outbreak prepared-
ness and response plans to ensure alignment with the best
available evidence.
Acknowledgements
The authors would like to acknowledge Matthew P. Shearer, a
fellow member of the Johns Hopkins Center for Health Security
Outbreak Observatory, and the Open Philanthropy Project, which
funds the Outbreak Observatory.
Role of funding source
This research was funded by the Open Philanthropy Project
(https://www.openphilanthropy.org/). The funders had no role in
the study design, data collection and analysis, decision to publish,
or preparation of the manuscript.
Ethics committee approval
This study did not involve human subjects.
Conflict of interest
The authors have no conflicts of interest to declare.
Author contributions
JBN conceived and oversaw the study. CH completed the initial
literature search, and JBN, DM, and MS helped evaluate each article
for relevancy. CH and JBN composed the manuscript, and all four
authors contributed to its revision. MPS (acknowledged) helped to
review articles.
References
Ballabeni A, Boggio A. Publications in PubMed on Ebola and the 2014 outbreak.
F1000Research 2015;4:68.
Bell BP. Overview, control strategies, and lessons learned in the CDC response to the
2014–2016 Ebola epidemic. MMWR Suppl 2016;65: [Cited 2018 August 17].
Available from: https://www.cdc.gov/mmwr/volumes/65/su/su6503a2.htm.
Ebola virus disease distribution map: cases of Ebola virus disease in Africa Since
1976 | 2014–2016 outbreak West Africa | History | Ebola (Ebola virus disease).
CDC; 2018 [Cited 2018 August 17]. Available from: https://www.cdc.gov/vhf/
ebola/history/distribution-map.html.
Coltart CEM, Lindsey B, Ghinai I, Johnson AM, Heymann DL. The Ebola outbreak,
2013–2016: old lessons for new epidemics. Philos Trans R Soc Lond B Biol Sci
2017;372(May (1721)).
Fitzpatrick G, Decroo T, Draguez B, Crestani R, Ronsse A, Van den Bergh R, et al. Operational
research during the Ebola emergency. Emerg Infect Dis 2017;23(7):1057–62.
Forum on Medical and Public Health Preparedness for Catastrophic Events, Board on
Health Sciences Policy, Institute of Medicine. Enabling rapid and sustainable
public health research during disasters: summary of a joint workshop by the
Institute of Medicine and the U.S. Department of Health and Human Services.
Washington (DC): National Academies Press (US); 2015 (The national
academies collection: reports funded by National Institutes of Health). Available
from: http://www.ncbi.nlm.nih.gov/books/NBK253866/.
Rapid research fund for Ebola virus disease outbreaks. IDRC — International
Development Research Centre; 2018 [cited 2018 September 25]. Available from:
https://www.idrc.ca/en/initiative/rapid-research-fund-ebola-virus-disease-
outbreaks.
Kalenga OI. We’ve halted the spread of deadly Ebola in Congo — so what went right?
| Oly Ilunga Kalenga. The Guardian; 2018 July 25. [Cited 2018 August 17];
Available from: https://www.theguardian.com/global-development/2018/jul/
25/halted-spread-deadly-ebola-drc-oly-ilunga-kalenga.
Kupferschmidt. WHO declares Ebola outbreak over. Science | AAAS; 2016 January
14K, 2016, Am 9:30. [Cited 2018 August 17]. Available from: http://www.
sciencemag.org/news/2016/01/who-declares-ebola-outbreak-over.
Moon S, Sridhar D, Pate MA, Jha AK, Clinton C, Delaunay S, et al. Will Ebola change
the game? Ten essential reforms before the next pandemic. The report of the
Harvard-LSHTM Independent Panel on the Global Response to Ebola. Lancet
Lond Engl 2015;386(November (10009)):2204–21.
 C. Hurtado et al. / International Journal of Infectious Diseases 77 (2018) 29–33
NSF awards $5.3 million in 59 grants to study effects of recent hurricanes. NSF —
National Science Foundation; 2017 [Cited 2018 September 25]. Available from:
https://www.idrc.ca/en/initiative/rapid-research-fund-ebola-virus-disease-
outbreaks.
National Academy of Medicine. The neglected dimension of global security: a
framework to counter infectious disease crises. 2016 Available from: https://
nam.edu/initiatives/global-health-risk-framework/.
Royston G. Meeting global health challenges through operational research and
management science. Bull World Health Organ 2011;89(September (9)):683–8.
United Nations General Assembly. Protecting humanity from future health crises:
report of the high-level panel on the global response to health crises. 2016
February. Available from: http://www.un.org/en/global-health-crises-task-force/.
33
WHO. Origins of the Ebola epidemic. WHO; 2014 [Cited 2018 August 17]. Available
from: http://www.who.int/entity/csr/disease/ebola/one-year-report/virus-ori-
gin/en/index.html.
World Health Organization. Report of the Ebola interim assessment panel. 2015
Available from: http://www.who.int/csr/resources/publications/ebola/report-
by-panel.pdf?ua=1.
World Health Organization. WHO Ebola science committee prioritized research
agenda. 2015 Available from: http://www.who.int/medicines/ebola-treatment/
WHO_ebola-science-committee-research-agenda.pdf?ua=1.
Zachariah R, Harries AD, Ishikawa N, Rieder HL, Bissell K, Laserson K, et al.
Operational research in low-income countries: what, why, and how?. Lancet
Infect Dis 2009;9(November (11)):711–7.