J. of Cardiovasc. Trans. Res.
(2017) 10:313–321
DOI 10.1007/s12265-017-9752-2
ORIGINAL ARTICLE
Text Mining of the Electronic Health Record: An Information
Extraction Approach for Automated Identification
and Subphenotyping of HFpEF Patients for Clinical Trials
Siddhartha R. Jonnalagadda 1,2 & Abhishek K. Adupa 1 & Ravi P. Garg 1 &
Jessica Corona-Cox 3 & Sanjiv J. Shah 3
Received: 18 January 2017 / Accepted: 16 May 2017 / Published online: 5 June 2017
# Springer Science+Business Media New York 2017
Abstract Precision medicine requires clinical trials that
are able to efficiently enroll subtypes of patients in whom
targeted therapies can be tested. To reduce the large
amount of time spent screening, identifying, and
recruiting patients with specific subtypes of heteroge-
neous clinical syndromes (such as heart failure with pre-
served ejection fraction [HFpEF]), we need prescreening
systems that are able to automate data extraction and
decision-making tasks. However, a major obstacle is the
vast amount of unstructured free-form text in medical re-
cords. Here we describe an information extraction-based
approach that automatically converts unstructured text in-
to structured data, which is cross-referenced against eligi-
bility criteria using a rule-based system to determine
which patients qualify for a major HFpEF clinical trial
(PARAGON). We show that we can achieve a sensitivity
and positive predictive value of 0.95 and 0.86, respective-
ly. Our open-source algorithm could be used to efficiently
identify and subphenotype patients with HFpEF and other
disorders.
Keywords Information extraction . Natural language
processing . Clinical trials . Precision medicine
* Siddhartha R. Jonnalagadda
sidjreddy@gmail.com
1 entire healthcare system. We hypothesized that it would be
Division of Health and Biomedical Informatics, Department of
Preventive Medicine, Northwestern University Feinberg School of
Medicine, Chicago, IL 60611, USA
2
Present address: Microsoft Corporation, 555 110th Ave NE,
Bellevue, WA 98004, USA
3
Division of Cardiology, Department of Medicine, Northwestern
University Feinberg School of Medicine, Chicago, IL 60611, USA
Introduction
The creation and acceptance of electronic health records
(EHRs) has ignited widespread interest in the use of clinical
data for secondary purposes and research [1]. One such appli-
cation that can greatly benefit from an EHR-based approach is
clinical trial screening and recruitment. In general, screening
for clinical trial recruitment is currently done manually.
Clinicians and study coordinators go through each of the eli-
gibility criteria, determine data elements relevant to the clini-
cal trial, extract the data elements from structured and unstruc-
tured EHR of each patient, and match the data elements with
the eligibility criteria to decide whether the patient qualifies
for the trial. Not only is this process slow, it is also prone to
errors. It typically takes approximately 15–20 min for a study
coordinator to examine each patient’s data, and this process
can take even longer in the context of complex clinical syn-
dromes such as heart failure with preserved ejection fraction
(HFpEF). In addition, as we advance towards precision med-
icine, finding specific subtypes of patients will be increasingly
important, making manual screening of patient records even
more time-consuming and arduous.
Because of the subjectivity involved in human decision-
making, domain knowledge, which patients are considered
for initial search, and other factors [2], there is always a pos-
sibility of type 1 and type 2 errors in the prescreening process
and biases in the overall recruitment. Furthermore, clinicians
and study coordinators typically rely on patients identified in
their own specialty clinics or in certain defined patient care
settings, thereby missing out on the advantage of screening an
possible to create a highly sensitive automated process
for prescreening with sufficient specificity to save study
coordinator time overall, and potentially reduce recruit-
ment bias because it would be possible to consider pa-
tients from a larger pool (i.e., the entire EHR). Thus, an
algorithm that can prescreen eligible patients efficiently
314
could provide a proficient and robust approach to clin-
ical trial recruitment.
Therefore, we sought to develop a high recall (sensitivity)
prescreening algorithm for recruiting patients into a multicen-
ter, randomized, double-blind, parallel-group, active-
controlled study to evaluate the efficacy and safety of
LCZ696 (sacubitril/valsartan) compared to valsartan, on mor-
bidity and mortality in HFpEF (Prospective comparison of
ARni with Angiotensin receptor blocker Global Outcomes
in heart failure with preserved ejectioN fraction,
PARAGON). Our approach involves the development of in-
formation extraction modules that use a combination of tech-
niques, including natural language processing (NLP), which
can be reused not only for other EHRs but also for other trials
using similar data elements.
Methods
Patient Records and Eligibility Criteria—Data
Description
The patient records used in this study came from the Epic
EHR used by Northwestern Medicine (Northwestern
Memorial Hospital and Northwestern University, Chicago,
IL). The initial cohort of patients we considered for our exper-
iments was very broad to ensure we were not missing any
patients that could be included—patients that were document-
ed to have HF with the ICD-9-CM Diagnosis Code 428.0 and
had an echocardiogram within the past year. We randomly
selected n = 198 of these patients for the development dataset
and n = 3002 patients for the validation dataset.
Each patient’s data consists of five types of reports: en-
counters, problem list, echocardiography reports, lab reports,
and current medication list. Encounters contain two types of
files: encounter diagnosis name and encounter progress notes.
The characteristics of the patient records for both datasets are
summarized in Table 1. There are a total of 40 eligibility
criteria—7 for inclusion and 33 for exclusion—for the
PARAGON clinical trial [3].
Table 1 Characteristics of the development and validation patient
datasets
Development set Validation set
Total number of patients 198 3002
Encounters 54,173 393,482
Echocardiography reports 96,281 883,385
Lab reports 52,393 371,879
Current medication entries 4490 41,947
Problem lists 3521 33,089
J. of Cardiovasc. Trans. Res. (2017) 10:313–321
Algorithm
The information from the patient data is extracted as part of
separate modules (Fig. 1). These modules are designed to
extract the data elements relevant to PARAGON but are reus-
able individually for other clinical trials. After extraction, a
rule-based system matches the eligibility criteria and excludes
patients who (1) do not satisfy all of the inclusion criteria, or
(2) meet one or more of the exclusion criteria. Figure 2 de-
scribes the system’s architecture in finer detail. We broadly
categorize the modules as (1) structured data normalizer, (2)
unstructured data extractor, and (3) unstructured data
classifier.
A structured data normalizer is used for the extraction of
data elements whose values are already present in the struc-
tured form. This module is further divided into two
submodules. In submodule 1, we extract the values for age,
body mass index, hemoglobin, glomerular filtration rate, and
blood pressure from structured fields. In submodule 2, we
extract medication data, including types of medications (e.g.,
angiotensin-converting enzyme inhibitors, beta-blockers,
dihydropyridine, and non-dihydropyridine calcium channel
blockers). The reports are in structured form with mapping
of the medication to the patient. This submodule requires ex-
ternal information resources, which we provide as databases
to our system.
An unstructured data extractor is used for the extraction of
values of data elements present in unstructured text. This mod-
ule also accepts input and provides output just as the previous
module but uses a complex set of regular expressions to ex-
tract the exact value. For example, we needed to extract the
left ventricular ejection fraction (LVEF) value as part of the
prescreening process for identifying HFpEF patients as part of
the PARAGON clinical trial. Here we used a set of regular
expressions to extract sentences where the LVEF value may
be present and then another set of regular expressions to ex-
tract the definite values as shown in Table 2. Regular expres-
sions 1 through 4 extract the sentences that can contain LVEF
values. Then, the sentences are parsed through regular expres-
sions 5 and 6. Regular expression 5 extracts the values present
in range format (e.g., B40–45%^ or B40% to 45%^). Regular
expression 6 extracts the freely available values: for example,
B40%.^
Next, we used an unstructured data classifier to classify
whether certain data elements are present or absent in relation
to the context of the patient. In this module, we extract all of
the instances of a given data element (diagnosis, medication,
treatment, or tests) and its synonyms in the input report(s). For
this, the module first checks for synonyms of the input term
using UMLS Metathesaurus [4], builds automatically a set of
regular expressions, and then applies them to the input report
text to extract all the instances. For example, to extract HF-
related terms the module compiles a list of synonyms: Bheart
J. of Cardiovasc. Trans. Res. (2017) 10:313–321 315

Fig. 1 Overview of the clinical Paent Before Paent Aer

trial recruitment system
architecture. We analyzed Demographics
Demographics, Lab
different heart failure-related Lab Values
Values, etc.
patient medical reports and
derived pattern-based information (Structured Data) Informaon Pathology Report
Informaon Eligibility
extraction modules that provide Extracon OUTPUT
Pathology Reports, Criteria
output of structured data to Modules Echo Report
compare against eligibility criteria Echo Report,
Informaon
for clinical trial recruitment Encounter Notes,
Encounter Notes
Discharge
Informaon
Summaries, etc.
(Unstructured Data) (All Structured
Form)

f a i l u r e , ^ BH F, ^ Bd i a s t o l i c d y s f u n c t i o n , ^ a n d Testing and Validation of the Algorithm

Bcardiomyopathy.^ Next, a set of regular expressions are au-
tomatically generated (Table 3) and used to extract all the
instances of HF-related terms. For PARAGON, the other data
elements processed in this category are Bangioedema^,
Bpancreatitis^, Bvalvular heart disease,^ etc. We adapt existing
rule-based systems to make sure the data elements are not in
their negated form using a rule-based negation detection algo-
rithm, the data elements refer to the current status (as opposed
to historical condition or a hypothesis for conducting a test),
and the data elements correspond to the patient (as opposed to
a family member [i.e., as listed in the Bfamily history^ section
of clinical notes]) [5, 6]. Our open-source algorithm is avail-
able at https://github.com/sidkgp/IECTM.
We first evaluated our methods iteratively in patients previ-
ously manually screened for the PARAGON trial at
Northwestern University. The initial iterative algorithm devel-
opment used these cases of patients found to be eligible for
PARAGON (and enrolled in the study) and those who were
found to be ineligible.
Next, we performed formal testing using the development
set of 198 patients identified from the Northwestern Medicine
Epic EHR. An experienced clinical research study coordinator
(J.C.-C.) read each patient record, extracted data elements of
relevance to PARAGON, and matched against the
PARAGON eligibility criteria. Patients deemed potentially

Fig. 2 Patient identification Paent Records IC: Age ≥ 55 y, EC: BMI > 40 kg/m2
algorithm for the PARAGON
Data
trial. Each patient’s data is parsed
IC: Number of HF terms ≥1
through three types of extraction
modules. The modules extract the Encounter Age IC: LVEF value ≥45%
appropriate information and Progress Notes BMI
create a patient profile. This EC: Number of angioedema or
Hb
profile is then checked against the Encounter pancreas or bilateral renal artery
Structured data GFR
clinical trial eligibility criteria to Diagnosis Names stenosis terms ≥1
normalizer BP
check whether or not the patient
qualifies for the study. PARAGON Problem List EC: Number of “transplant” or “ICD”
Structured data
Prospective comparison of ARni extractor LVEF
terms ≥1
Lab Report
with Angiotensin receptor blocker
Unstructured EC: Sentences with “malignant” term
Global Outcomes in heart failure Echo Report
and absence of “basal” and
with preserved ejectioN fraction; data extractor a. Heart failure
BMI body mass index; Hb Medicaon List “prostrate” terms ≥1
b. Angioedema
hemoglobin; GFR glomerular c. Pancreas EC: Hb < 10 g/dl, GFR < 30 ml/min
filtration rate; BP blood pressure; d. Organ transplant
LVEF left ventricular ejection EC: SBP value > 150 mm Hg and
e. ICD
fraction; ICD implantable number of anhypertensive drugs > 3
f. Malignant cancer
cardioverter-defibrillator; IC
inclusion criteria; EC exclusion g. Valvular heart
criteria disease EC: Number of “constricve
h. Constricve pericardis” or “genec hypertrophic
pericardis, cardiomyopathy” or “infiltrave
hypertrophic cardiomyopathy” terms ≥ 1
cardiomyopathy,
infiltrave
cardiomyopathy
316 J. of Cardiovasc. Trans. Res. (2017) 10:313–321

Table 2 Regular expressions for

extracting left ventricular ejection
fraction values from clinical notes
and reports
Step Regular expression
1 (left ventricular ejection fraction|lvef|lv ejection fraction|left ventricle ejection fraction|ejection fraction|
ef |ejection fraction)[^_%\\.]*?([\\d-\\.]+)\\s*'?%
2 (left ventricular systolic function|left ventricular function|systolic function of the left ventricle|lv systolic
function|left ventricular ejection fraction|ejection fraction|left ventricle)(normal|normal global|low
normal|well preserved|severely reduced|moderately decreased|moderately depressed|severely
decreased|severe|markedly decreased|markedly reduced|severely globally reduced|mildly
decreased|mildly depressed|severely depressed)
3 (normal|normal global|low normal|well preserved|severely reduced|moderately decreased|moderately
depressed|severely decreased|severe|markedly decreased|markedly reduced|severely globally
reduced|mildly decreased|mildly depressed|severely depressed)
4 .*(moderate|marked|severe) (lv systolic dysfunction|left ventricular dysfunction|left ventricular systolic
dysfunction).*
5 ((\\d+\\s*(\\-|to)\\s*\\d+)|(\\d*\\.\\d*\\s*(\\-|to)\\s*\\d*\\.\\d*)|(\\d*\\.\\d+)|(\\d+))(?=(\\s*(\\%)))
6 \\d+(\\.\\d+)?

eligible for the PARAGON trial by the study coordinator were
then reviewed in further detail by a cardiologist with expertise
in HFpEF (S.J.S.). The automated prescreening algorithm was
tested on the same 198 patients (blinded to the manual screen-
ing results). The manual screening was then compared to the
automated algorithm. The time needed to complete the
prescreening task was documented for both methods; in addi-
tion, a 2 × 2 contingency was created comparing both manual
and automated prescreening, and sensitivity and specificity
were calculated.
Once the development of our automated information ex-
traction system was complete, we focused our attention on its
validation. We ran our information extraction approach on
EHR-derived reports from the validation dataset (n = 3002).
The study coordinator then performed manual review of the
medical records for all patients deemed eligible for the
PARAGON trial by the automated information extraction sys-
tem. Results were again compared between the two methods.
Results
Development Dataset
For the 198 patient records included in the development
dataset, our experienced research coordinator took 2 weeks
(∼80 h) to generate the gold standard data, which then still
had to be verified by the clinical cardiologist. Our automated
information extraction (prescreening) system to successfully
parse and extract the required information from different data
reports took less than 2 min for the entire 198-patient dataset.
Table 4 shows the large number of exclusions based on
exclusion criteria for the PARAGON trial. Due to the fact that
PARAGON has a large number of stringent eligibility criteria,
the typical number of patients that qualify for the trial is small.
This is a challenge faced by many clinical trials for HFpEF
and other heterogeneous clinical syndromes, and a problem
that will only become increasingly prominent as we seek to
study narrower subtypes of patients in the era of precision
medicine. For these reasons, it is important for an automated
prescreening system to give more importance to retrieving
nearly all the qualifying patients; in other words, the recall
of the system should be close to 100%. We therefore tuned
our system in order to achieve a high recall (i.e., high sensi-
tivity) so as not to have too many false negatives (which
would result in missing potentially eligible patients). Table 5
shows the 2 × 2 contingency table for the automated
prescreening algorithm compared to the gold standard (man-
ual prescreening). Sensitivity (recall) of the algorithm was
95% and specificity was 96%. The precision (positive predic-
tive value) was 86% (F score of 90%).
Validation Dataset

Table 3 Regular expressions to extract heart failure-related terms After the development of our PARAGON HFpEF clinical trial
prescreening algorithm was complete, we tested it on the val-
Regular expression
idation dataset (n = 3002). The algorithm took ∼20 min to go
[^\w]+(h|H)eart\s+(f|F)ailure[^\w]+ through all 3002 medical records, and it identified n = 113
[^\w]+(d|D)iastolic\s+(d|D)ysfunction[^\w]+ (3.7%) patients who were potentially eligible for the
[^\w]+(c|C)ardiomyopathy[^\w]+ PARAGON trial (see Table 4 for exclusions). Our clinical trial
[^\w]+HF[^\w]+ study coordinator went through these records (and verified
results with the clinical cardiologist) and found that 46 of
J. of Cardiovasc. Trans. Res. (2017) 10:313–321 317

Table 4 Numbers of patients excluded at each step for the development or are beyond the scope of any system to check due to lack of
and validation patient datasets
data (e.g., type of cancer or cancer is malignant or benign
Exclusion criteria Report type Number of patients when the details are not present) or other extraneous factors
excluded, n (%)a that are difficult to conceptualize by text mining of clinic notes
(e.g., patient non-compliance with medications).
Development Validation
dataset dataset
(n = 198) (n = 3002) Enhancing Efficiency by Rapidly Excluding Large
Numbers of Patients
Age + BMI Encounter 22 (11) 1071 (36)
report
Heart failure-related term Encounter 3 (2) 1597 (53)
Table 4 lists the number of patients in both the development
report/- and validation datasets based on the PARAGON trial eligibil-
problem list ity criteria. The information extraction module played a major
LVEF Echo report 90 (45) 672 (22) role in screening out large proportions of patients without
Angioedema, Encounter 44 (22) 218 (7) human involvement. For example, module 2, which extracts
pancreatitis, or bilateral report
LVEF values, excluded 90 patients from the 198-patient de-
renal artery stenosis
Organ transplant or ICD Encounter 50 (25) 806 (27) velopment dataset and 672 patients from the 3002-patient val-
report/- idation dataset. This would not have been captured by any
problem list methods that aim to prioritize patients using information re-
Malignancy Encounter 49 (25) 600 (20)
report/-
trieval approaches without first extracting the values of the
problem list relevant data elements from unstructured reports.
Hemoglobin + GFR Lab value 42 (21) 507 (17)
report
Blood pressure and Encounter 6 (3) 522 (17)
anti-hypertensive report
Discussion
medications
Constrictive pericarditis, Echo report 23 (12) 314 (10) Here we have presented the development and validation of an
genetic hypertrophic information extraction system based on text mining tech-
cardiomyopathy, or
infiltrative
niques, including NLP, that efficiently identified patients
cardiomyopathy who were eligible for PARAGON, a major HFpEF clinical
trial. Given the stringent inclusion/exclusion criteria for
BMI body mass index, LVEF left ventricular ejection fraction, ICD im- PARAGON, the information extraction approach we have cre-
plantable cardioverter-defibrillator, GFR glomerular filtration rate, Echo
ated has applications to precision medicine initiatives given
echocardiography
a the need to identify specific subtypes of patients with hetero-
Patients could be excluded due to multiple criteria (thus, the percentages
listed in each column do not add up to 100%) geneous medical disorders. Importantly, we show the clear
increase in efficiency with automated screening of the EHR
for complex clinical trials. We showed in our development
the patients qualified for the clinical trial. However, 67 of the
dataset that screening time was reduced from 80 h to 2 min.
patients did not qualify for the trial. In most cases, these in-
Extrapolating manual prescreening time requirements to the
stances were not because of errors in the prescreening system
validation dataset, we see that manual prescreening of ∼3000
but rather due to certain other criteria that were either not
medical records would have taken several months. Instead,
included in the algorithm (e.g., certain specific allergies to
our automated system only took 20 min to perform the
medication, pregnancy, patient who did not live in the area)
prescreening task. Nevertheless, from these results and obser-
vations, we also understand that the system can only be used
Table 5 Development dataset 2 × 2 contingency table (compared to
for prescreening, and further validation by the clinical trial
manual prescreening gold standard)
study coordinator or clinical investigator is still required.
Prescreening gold We achieved high recall with reasonable precision on our
standard (manual) development dataset and were able to replicate the perfor-
Patients Patients
mance on a larger dataset. As with any automated system,
included excluded there are certain limitations to our proposed architecture,
which can be broadly categorized into (1) data processing
Classification outcome-based screen- Patients 38 6 and (2) data handling issues. We briefly describe some of
ing algorithm (automated) included
these issues. The precision of the system suffers from the
Patients 2 152
excluded complexity of text data. In some cases, our unstructured data
extractor module was unable to extract terms correctly. For
318
example, the module fails to identify certain HF- or ICD-
related terms. This is due to the large number of synonyms
and spelling mistakes for the relevant data elements.
In addition, there are some cases where a patient has spe-
cific allergies or may show a certain adverse reaction to a
medication, both of which are difficult to extract from unstruc-
tured notes because they are not always reported in a standard
format within the EHR. There are also cases where the patient
has moved out of the hospital’s geographic area and therefore
cannot provide consent for the clinical trial. These are details
that are likely too patient-specific for automated extraction
and can only be checked manually.
In some cases, the LVEF value (which is an important
factor for inclusion in HF clinical trials) is present in the form
of a range or qualitative description. This created a problem
while checking for eligibility according to the criterion given.
For example, in our clinical trial, we have set the lower limit of
LVEF at 45% based on the inclusion criteria. This creates a
problem when the value is contained within the range extract-
ed (40% to 45% or 30% to 50%, for example). An initial
approach of taking the average value and comparing it with
the LVEF threshold was deemed inappropriate given the need
to exclude all patients with LVEF <45% at baseline.
Therefore, we subsequently modified our algorithm to include
these patients but with a warning regarding their LVEF value.
This then served as an indication to the study coordinators to
recheck the echocardiogram report (and review the echocar-
diographic images with the clinical investigator) in order to
make further decisions about the patient’s eligibility for the
clinical trial.
There are also some cases in which clinicians are just
screening the patient for a particular diagnosis but the patient
may not actually have the disease, such as a Bmalignancy of
organ system^ check of exclusion criteria. To handle this, we
do not exclude those patients if we find the Bscreening^ term
in the sentences extracted for eligibility check. For the terms
Bcancer^ and Bmalignancy,^ we cannot exclude all patients
with these terms because the patient may have a cancer that
is currently in remission or may have only had a presumptive
diagnosis of cancer that was either never verified or found to
be inaccurate. To mitigate these issues, we currently just dis-
play a warning in these cases, as we did for LVEF. The coor-
dinator can then perform further checks and decide the classi-
fication. In other exclusion criteria where we have to check the
B-type natriuretic peptide and glomerular filtration rate
values, we face the issues of non-availability and potential
outliers in the data. For such cases too, we currently report
them as a warning to coordinators for further checking.
We also had to deal with data handling issues in some
cases. For example, in criteria where we have to perform a
check for recent hemoglobin values, we found that the value
may also be present in reports other than just blood reports. To
mitigate this issue, we check for hemoglobin values in all
J. of Cardiovasc. Trans. Res. (2017) 10:313–321
reports and then extract the most recent one. Similarly, there
were also cases where Bend-date^ of medication and
Bdepartment-name^ for encounter reports were missing or
misplaced. We handled such cases following discussions with
the data warehouse coordinator. To summarize, we can deduce
that the patient data records are noisy due to various reasons
and a preprocessing module is required to handle these issues.
An additional limitation of our study to consider is the time
saved using an automated approach. When comparing the
automated algorithm to the time required for manual review,
it is important to factor in the time to collect the EHR data and
create/tune the screening modules for the study, the time re-
quired to modify modules for a new study. These metrics are
difficult to quantify and can vary greatly from study to study
and from institution to institution, but future studies would
benefit from recording these data to get a sense of the true
time saved using an automated approach (i.e., the time the
study coordinator would not have to spend reviewing records
for the patients that the prescreening tool rules out, minus the
programmer time required to repurpose/retune the text mining
tools for the clinical trial in which it has been implemented).
Clinical trials are the gold standard by which investigators
can determine whether or not a particular treatment is effective
for a disease or clinical syndrome. However, a study of clinical
trial enrollment reported that 86% of all clinical trials are de-
layed in patient recruitment from 1 to 6 months, and 13% are
delayed by longer than 6 months [2]. In the TOPCAT trial
(spironolactone vs. placebo in HFpEF) [7], which required
270 enrolling sites in 6 countries, the average enrollment rate
was a dismal 2.3 patients per site per year worldwide, and
even worse in the USA (1.4 patients per site per year),
highlighting significant problems with HFpEF patient identi-
fication and enrollment [8]. A major cause of delay in HF
clinical trials is the inability to efficiently screen for and iden-
tify eligible patients. An automated system is therefore urgent-
ly needed to accelerate the process of prescreening patients for
clinical trials.
The surge of the use of EHRs in the USA has created
abundant opportunities for clinical and translational research.
As Friedman et al. noted, the extensive use of clinical data
provides great potential to transform our healthcare system
into a BSelf-learning Health System^ [9, 10]. In addition to
its primary purpose of providing improved clinical practice,
the use of EHRs offers means for the identification of partic-
ipants who satisfy predefined criteria. This can be used for a
variety of applications, including clinical trial recruitment,
outcome prediction, survival analysis, and other retrospective
studies [11–14].
EHRs contain patient data in both structured and unstruc-
tured formats. The structured data generally encompass a pa-
tient’s demographic data, physical characteristics (e.g., body
mass index [BMI], blood pressure), laboratory data, and diag-
noses. Structured data are not only the best representation of
J. of Cardiovasc. Trans. Res. (2017) 10:313–321
Fig. 3 Summary of techniques
for patient cohort identification Paern Matching
(Regular Expressions)
Input
Paent
Record Language Modeling
Based Methods
knowledge but also easier to process. However, there is a vast
amount of medical knowledge that is locked in the unstruc-
tured format. The unstructured data are typically text clinical
narratives present in progress notes, imaging reports (e.g.,
echocardiographic reports), and discharge summaries, for ex-
ample. Thus, a module that can automatically and efficiently
extract information from unstructured clinical text and convert
it into a structured form is needed.
The syndromic nature of HF presents unique challenges for
the identification of patients from EHR data for research [15].
HFpEF is particularly challenging to identify during
prescreening activities. The presence of large amounts of un-
structured data in patient medical reports aggravates the chal-
lenges. Previous studies have shown that clinicians often pre-
fer free text entry to coded options, in order to fully explain the
health conditions of each patient [16–18]. It has also been
noted that unstructured data are essential because of the infor-
mation they contain [19]; therefore, unstructured data are like-
ly to persist in the future. There is an immediate need for an
automated data extraction system to transform unstructured
clinical reports into a structured form, which is much easier
to process and handle [20–22].
There has been considerable research in identifying patient
cohorts from EHRs [23]. These approaches can be categorized
into three general types (Fig. 3): (1) rule-based approaches
[24–28], (2) machine learning-based approaches [29–32],
and (3) information retrieval-based approaches [33–36]. All
these approaches use either pattern matching (regular expres-
sions) or language modeling-based methods [37–40] to ex-
tract features for their system to work on. Rule-based systems
are stringent and binary (either yes or no) in nature. On the
other hand, machine learning- and information retrieval-based
methods provide output as probability or a score. Machine
learning techniques, however, require a large amount of train-
ing data to give accurate results.
Our proposed system is different from these approaches in
various ways. A majority of the reported systems aim to iden-
tify whether a patient shows a certain phenotype. Therefore,
the number of criteria required is less than that which is nec-
essary for clinical trial screening. For example, a majority of
the systems only use a variation of disease names, medications
used, or treatments taken as their eligibility criteria [25, 27].
Ours is a more diverse application. Our goal is to check wheth-
er a particular patient qualifies for a certain clinical trial.
319
Rule-Based Techniques
Machine Learning–Based
Features
Techniques
Informaon Retrieval–Based
Techniques
Clinical trials usually have a large number of eligibility criteria
that need to be checked. Therefore, a large amount of infor-
mation related to the eligibility criteria needs to be extracted.
Our study goal is similar to that of the plethora of ap-
proaches proposed in computer-based clinical trial recruitment
systems [41]. However, a majority of these approaches either
lack EHRs as data source or are not equipped to handle un-
structured data. We, on the other hand, obtain patient data
from EHRs and handle unstructured data through information
extraction methods, as opposed to the Bbag of terms^ or Bbag
of concepts^ suggested in other methods [42]. The main con-
tributions of our study are to (1) show that automated recruit-
ment systems can only serve as prescreening tools and to (2)
develop and validate a clinical trial screening system based on
information extracted from EHRs. Here, we demonstrate how
our system processes a set of eligibility criteria, extracts infor-
mation from patient records automatically into a structured
format, and finally prescreens the patients who could qualify
for the trial by matching the structured patient document with
the eligibility criteria.
Conclusions
Using an information extraction approach to the EHR, we
show here that automated identification of HFpEF patients
who are appropriate candidates for a clinical trial is feasible
and time-efficient. Our approach demonstrated excellent ac-
curacy, and it reduced the time needed for prescreening pa-
tients from a few weeks to a few minutes. As we move to-
wards a Bprecision medicine^ approach for complex, hetero-
geneous disorders such as HFpEF, we will need to find novel
ways to identify appropriate patients to augment clinical trial
enrollment of specific disease subgroups, a task that when
done manually is quite time-consuming and tedious. Given
the ability of our approach to identify subtypes of HFpEF
patients who meet specific inclusion/exclusion criteria, it
could be very useful for the targeting of specific therapeutics
to specific subtypes of HFpEF patients in future clinical trials
without overburdening study staff. Finally, an approach such
as the one we demonstrate here may decrease costs and expe-
dite clinical trials, and could enhance the reproducibility of
trials across institutions and populations.
320
Compliance with Ethical Standards
Funding Sources This work was funded by the National Library of
Medicine: R00LM011389 and R01LM011416 (to S.R.J.), and an
investigator-initiated study grant from Novartis. S.J.S. is also supported
by grants from the National Institutes of Health (R01 HL107577 and R01
HL127028). The authors acknowledge Prasanth Nannapaneni for his
valuable ideas on extracting information from the electronic health
record.
Conflicts of Interest Siddhartha R. Jonnalagadda is currently an em-
ployee of Microsoft Corporation.
Abhishek K. Adupa declares that he has no conflict of interest.
Ravi P. Garg declares that he has no conflict of interest.
Jessica Corona-Cox declares that she has no conflict of interest.
Sanjiv J. Shah reports receiving consulting fees from Novartis.
Ethical Approval All procedures performed in studies involving hu-
man participants were in accordance with the ethical standards of the
institutional and/or national research committee and with the 1964
Helsinki Declaration and its later amendments or comparable ethical
standards.
Informed Consent Informed consent was waived for this study by the
Northwestern University Institutional Review Board because the study
only involved retrospective chart review.
References
1. Jensen, P. B., Jensen, L. J., & Brunak, S. (2012). Mining electronic
health records: towards better research applications and clinical
care. Nature Reviews. Genetics, 13(6), 395–405.
2. Sullivan, J.. (2004). Subject Recruitment and Retention: Barrier to
Success. http://www.appliedclinicaltrialsonline.com/subject-
recruitment-and-retention-barriers-success. Accessed 27 July 2015.
3. PARAGON Inclusion/Exclusion Criteria (2015). https://
sjonnalagadda.files.wordpress.com/2015/08/paragon_ie-criteria_
10-01-2014.pdf. Accessed 10th August 2015.
4. Bodenreider, O. (2004). The unified medical language system
(UMLS): integrating biomedical terminology. Nucleic Acids
Research, 32(Database issue), D267–D270.
5. Harkema, H., Dowling, J. N., Thornblade, T., & Chapman, W. W.
(2009). ConText: an algorithm for determining negation,
experiencer, and temporal status from clinical reports. Journal of
Biomedical Informatics, 42(5), 839–851.
6. Mitchell, K. J., Becich, M. J., Berman, J. J., Chapman, W. W.,
Gilbertson, J., Gupta, D., et al. (2004). Implementation and evalu-
ation of a negation tagger in a pipeline-based system for informa-
tion extract from pathology reports. Studies in Health Technology
and Informatics, 107(Pt 1), 663–667.
7. Shah, S. J., Heitner, J. F., Sweitzer, N. K., Anand, I. S., Kim, H. Y.,
Harty, B., et al. (2013). Baseline characteristics of patients in the
treatment of preserved cardiac function heart failure with an aldo-
sterone antagonist trial. Circulation. Heart Failure, 6(2), 184–192.
8. Shah, S. J., Cogswell, R., Ryan, J. J., & Sharma, K. (2016). How to
develop and implement a specialized heart failure with preserved
ejection fraction clinical program. Current Cardiology Reports,
18(12), 122.
9. Friedman, C. P., Wong, A. K., & Blumenthal, D. (2010). Achieving
a nationwide learning health system. Science Translational
Medicine, 2(57), 57cm29–57cm29.
J. of Cardiovasc. Trans. Res. (2017) 10:313–321
10. Friedman, C., & Rigby, M. (2013). Conceptualising and creating a
global learning health system. International Journal of Medical
Informatics, 82(4), e63–e71.
11. Ma, X.-J., Wang, Z., Ryan, P. D., Isakoff, S. J., Barmettler, A.,
Fuller, A., et al. (2004). A two-gene expression ratio predicts clin-
ical outcome in breast cancer patients treated with tamoxifen.
Cancer Cell, 5(6), 607–616.
12. Strom, B. L., Schinnar, R., Jones, J., Bilker, W. B., Weiner, M. G.,
Hennessy, S., et al. (2011). Detecting pregnancy use of non-
hormonal category X medications in electronic medical
records. Journal of the American Medical Informatics
Association, 18(Suppl 1), i81–i86.
13. Mathias, J. S., Gossett, D., & Baker, D. W. (2012). Use of electronic
health record data to evaluate overuse of cervical cancer screening.
Journal of the American Medical Informatics Association, 19(e1),
e96–e101.
14. De Pauw, R., Kregel, J., De Blaiser, C., Van Akeleyen, J., Logghe,
T., Danneels, L., et al. (2015). Identifying prognostic factors
predicting outcome in patients with chronic neck pain after multi-
modal treatment: a retrospective study. Manual Therapy, 20(4),
592–597.
15. Onofrei, M., Hunt, J., Siemienczuk, J., Touchette, D. R., &
Middleton, B. (2004). A first step towards translating evidence into
practice: heart failure in a community practice-based research net-
work. Informatics in Primary Care, 12(3), 139–145.
16. Johnson, S. B., Bakken, S., Dine, D., Hyun, S., Mendonça, E.,
Morrison, F., et al. (2008). An electronic health record based on
structured narrative. Journal of the American Medical Informatics
Association, 15(1), 54–64.
17. Zhou, L., Mahoney, L. M., Shakurova, A., Goss, F., Chang, F. Y.,
Bates, D. W., et al. (2012). How many medication orders are en-
tered through free-text in EHRs?—a study on hypoglycemic agents.
American Medical Informatics Association Annual Symposium
Proceedings, 2012, 1079–1088.
18. Zheng, K., Hanauer, D. A., Padman, R., Johnson, M. P., Hussain, A.
A., Ye, W., et al. (2011). Handling anticipated exceptions in clinical
care: investigating clinician use of ‘exit strategies’ in an electronic
health records system. Journal of the American Medical
Informatics Association, 18(6), 883–889.
19. Raghavan, P., Chen, J. L., Fosler-Lussier, E., & Lai, A. M. (2014).
How essential are unstructured clinical narratives and information
fusion to clinical trial recruitment? AMIA Jt Summits Transl Sci
Proc, 2014, 218–223.
20. Stanfill, M. H., Williams, M., Fenton, S. H., Jenders, R. A., &
Hersh, W. R. (2010). A systematic literature review of automated
clinical coding and classification systems. Journal of the American
Medical Informatics Association, 17(6), 646–651.
21. Jha, A. K. (2011). The promise of electronic records: around the
corner or down the road? JAMA, 306(8), 880–881.
22. Friedman, C., Rindflesch, T. C., & Corn, M. (2013). Natural lan-
guage processing: State of the art and prospects for significant
progress, a workshop sponsored by the National Library of
Medicine. Journal of Biomedical Informatics, 46(5), 765–773.
23. Shivade, C., Raghavan, P., Fosler-Lussier, E., Embi, P. J., Elhadad,
N., Johnson, S. B., et al. (2014). A review of approaches to identi-
fying patient phenotype cohorts using electronic health
records. Journal of the American Medical Informatics
Association, 21(2), 221–230.
24. Nguyen, A. N., Lawley, M. J., Hansen, D. P., Bowman, R. V.,
Clarke, B. E., Duhig, E. E., et al. (2010). Symbolic rule-based
classification of lung cancer stages from free-text pathology
reports. 17(4), 440–445.
25. Mia Schmiedeskamp, P. P., Spencer Harpe, P. P. M. P. H., Ronald
Polk, P., Michael Oinonen, P. M. P. H., & Amy Pakyz, P. M. S.
(2009). Use of international classification of diseases, ninth revi-
sion, clinical modification codes and medication use data to identify
J. of Cardiovasc. Trans. Res. (2017) 10:313–321
nosocomial Clostridium difficile infection. Infection Control and
Hospital Epidemiology, 30(11), 1070–1076.
26. Penberthy, L., Brown, R., Puma, F., & Dahman, B. (2010).
Automated matching software for clinical trials eligibility: measur-
ing efficiency and flexibility. Contemporary Clinical Trials, 31(3),
207–217.
27. Kho, A. N., Hayes, M. G., Rasmussen-Torvik, L., Pacheco, J. A.,
Thompson, W. K., Armstrong, L. L., et al. (2012). Use of diverse
electronic medical record systems to identify genetic risk for type 2
diabetes within a genome-wide association study. Journal of the
American Medical Informatics Association, 19(2), 212–218.
28. Klompas, M., Haney, G., Church, D., Lazarus, R., Hou, X., & Platt,
R. (2008). Automated identification of acute hepatitis B using elec-
tronic medical record data to facilitate public health surveillance.
PloS One, 3(7), e2626.
29. Mani, S., Chen, Y., Arlinghaus, L. R., Li, X., Chakravarthy, A. B.,
Bhave, S. R., et al. (2011). Early prediction of the response of breast
tumors to neoadjuvant chemotherapy using quantitative MRI and
machine learning. American Medical Informatics Association
Annual Symposium Proceedings, 2011, 868–877.
30. Van den Bulcke, T., Vanden Broucke, P., Van Hoof, V.,
Wouters, K., Vanden Broucke, S., Smits, G., et al. (2011).
Data mining methods for classification of Medium-Chain
Acyl-CoA dehydrogenase deficiency (MCADD) using non-
derivatized tandem MS neonatal screening data. Journal of
Biomedical Informatics, 44(2), 319–325.
31. Zhao, D., & Weng, C. (2011). Combining PubMed knowledge and
EHR data to develop a weighted bayesian network for pancreatic
cancer prediction. Journal of Biomedical Informatics, 44(5), 859–
868.
32. Kawaler, E., Cobian, A., Peissig, P., Cross, D., Yale, S., & Craven,
M. (2012). Learning to predict post-hospitalization VTE risk from
EHR data. American Medical Informatics Association Annual
Symposium Proceedings, 2012, 436–445.
33. Lowe, H. J., Ferris, T. A., Hernandez, P. M., & Weber, S. C. (2009).
STRIDE—an integrated standards-based translational research
321
informatics platform. American Medical Informatics Association
Annual Symposium Proceedings, 2009, 391–395.
34. Gregg, W., Jirjis, J., Lorenzi, N. M., & Giuse, D. (2003).
StarTracker: an integrated, web-based clinical search engine.
AMIA Annual Symposium Proceedings, 855.
35. Hanauer, D. A., Mei, Q., Law, J., Khanna, R., & Zheng, K. (2015).
Supporting information retrieval from electronic health records: a
report of University of Michigan’s nine-year experience in devel-
oping and using the Electronic Medical Record Search Engine
(EMERSE). Journal of Biomedical Informatics, 55, 290–300.
36. Zalis, M., & Harris, M. (2010). Advanced search of the electronic
medical record: augmenting safety and efficiency in radiology.
Journal of the American College of Radiology, 7(8), 625–633.
37. Lehman, L. W., Saeed, M., Long, W., Lee, J., & Mark, R. (2012).
Risk stratification of ICU patients using topic models inferred from
unstructured progress notes. American Medical Informatics
Association Annual Symposium Proceedings, 2012, 505–511.
38. Carroll, R. J., Eyler, A. E., & Denny, J. C. (2011). Naive electronic
health record phenotype identification for rheumatoid arthritis.
American Medical Informatics Association Annual Symposium
Proceedings, 2011, 189–196.
39. Liao, K. P., Cai, T., Gainer, V., Goryachev, S., Zeng-treitler, Q.,
Raychaudhuri, S., et al. (2010). Electronic medical records for dis-
covery research in rheumatoid arthritis. Arthritis Care and
Research, 62(8), 1120–1127.
40. Bejan, C. A., Xia, F., Vanderwende, L., Wurfel, M. M., & Yetisgen-
Yildiz, M. (2012). Pneumonia identification using statistical feature
selection. Journal of the American Medical Informatics
Association, 19(5), 817–823.
41. Kopcke, F., & Prokosch, H. U. (2014). Employing computers for
the recruitment into clinical trials: a comprehensive systematic re-
view. Journal of Medical Internet Research, 16(7), e161.
42. Ni, Y., Kennebeck, S., Dexheimer, J. W., McAneney, C. M., Tang,
H., Lingren, T., et al. (2015). Automated clinical trial eligibility
prescreening: increasing the efficiency of patient identification for
clinical trials in the emergency department. Journal of the American
Medical Informatics Association, 22(1), 166–178.