Information Extraction from Multimodal ECG Documents

Fei Wang Tanveer Syeda-Mahmood David Beymer

IBM Almaden Research Center
650 Harry Road, San Jose, CA 95120
{wangfe,stf,beymer}@us.ibm.com

Abstract

With the rise of tools for clinical decision support, there

is an increased need for automatic processing of electro-
cardiograms (ECG) documents. In fact, many systems have
already been developed to perform signal processing tasks
such as 12-lead off-line ECG analysis and real-time pa-
tient monitoring. All these applications require an accu-
rate detection of the heart rate of the ECG. In this paper,
we present the idea that the image form of ECG is actu-
ally a better medium to detect periodicity in ECG. When
the ECG trace is scanned or rendered in videos, the peaks
of the waveform (R-wave) is often traced thicker due to pixel
dithering. We exploit the pixel thickness information, for the
first time, as a reliable feature for determining periodicity.
Results are presented on a database of 16,613 12-channel
ECG waveforms, which demonstrate robustness and accu-
racy of our image-based period detection method on these
ECGs of various cardiovascular diseases. 94.5% of brady-
cardia and tachycardia patient records are correctly identi-
fied using our estimated heart period as the disease criteria.
1 Introduction
An electrocardiogram (ECG) is an important and com-
monly used diagnostic aid in cardiovascular disease diag-
nosis. An ECG is an electrical recording of the heart that
depicts the cardiac cycle. It is routinely used as a first course
of choice in diagnosing many cardiovascular diseases. Of-
ten, 12 electrodes are used to record the electrical activity
of the heart from different leads. A normal ECG waveform
(in lead II) has a characteristic shape indicated in Fig. 1b.
Many disturbances in the heart function show as character-
istic variations in the sinus rhythm waveform of Fig.1b, and
can be served as important cues to diagnose the disease.
Physicians routinely make diagnosis by a simple visual ex-
amination of these ECG waveforms.
With the rise of tools for clinical decision support, there
(a) (b)
Figure 1. Illustration of the Electrocardiogram
(a) The heart cycle. (b) A normal ECG.
has been an increased need for automatic processing of the
electrocardiogram (ECG). In fact, many systems have al-
ready been developed to perform signal processing tasks
such as 12-lead off-line ECG analysis, and real-time patient
monitoring. All these applications require an accurate de-
tection of the heart rate so that the disease related features
within a heart cycle can be extracted. Furthermore, heart
rate estimation is a prerequisite step to identify arrhythmia
diseases such as bradycardia and tachycardia. Much of the
prior work has focused on determining the heart rate from
digital ECG time series signals. There are a large number
of ECGs still in paper form. Their digital records are cre-
ated as scanned images as shown in Fig. 2b. Not much prior
art exists in the determination of period from ECG images
obtained from scanner paper ECGs. Although most hospi-
tals now have digital ECG recorders, much of the legacy
ECG data is still in paper form. A sample paper form ECG
is shown in Fig. 2b. Unlocking these ECG records printed
on paper and exposing them to digital analysis would be
useful. It provides additional data for current ECG analysis
techniques, as well as historical data to be used for compara-
tive studies. Digital ECG recordings are sampled very finely
(1000 samples/sec) and contain a large amount of noisy data
with baseline wandering problems, whereas the paper print-
outs are actually cleaner in appearance. Further, all 12 lead
information is displayed compactly in a standardized (lead

(a) (b)
(c) (d)
Figure 2. Illustration of the Electrocardiogram
from different sources (a) Digital ECG (b)
Paper ECG. (c) ECG embedded in echocar-
diogram video (d). ECG extracted from the
echocardiogram frame from (c)
order is fixed) 3 row by 4 column format.
The goal of this paper is to present a computational al-
gorithm as well as medical decision support tool for finding
the heart period from ECG images. Specifically, we use the
knowledge that when ECG trace is scanned or rendered in
videos, the peaks of the waveform (R-wave) is often traced
thicker due to pixel dithering. We exploit the pixel thickness
information, for the first time, as a reliable feature for de-
termining periodicity. Existing period estimation algorithm,
such as estimation of R peaks as largest positive peaks [13]
often leads to detection error when R peaks are lower than
the P or T peaks. While our method is relatively robust
to these cases since our features latched on pixel dithering
is more consistent among corresponding R peaks. Further-
more, using our estimated heart period as the disease cri-
teria, 94.5% of bradycardia and tachycardia patient records
are correctly identified on a database of 16,613 ECG wave-
forms.
This paper makes several novel contributions. To our
knowledge, this is the first period estimation work on im-
age ECG waveforms from scanned ECG printouts or ECGs
embedded in echo videos. It is also the first practical ap-
plication of image-based techniques which utilize the pixel
dithering to identify ECG period. Our method can also be
used for disease diagnosis validation in healthcare decision
support applications, where we can use our technique to
identify arrythmia ECG records from patient with brady-
cardia or tachycardia diseases. Our algorithm can be espe-
cially useful for scanned paper ECGs, ECGs embedded in
echocardiogram videos, as well as digital ECGs, once it is
converted into image form.
While there is considerable prior art in ECG period de-
tection from digital time series, there has not been much
work on detecting periodicity from ECG images directly
(rather than detecting periodicity after conversion to a time
series). We will briefly review some related methods in lit-
erature next.
1.1 Previous methods
Most of the existing approaches that deal with paper
ECGs actually trace the signals to form a digital time se-
ries so that time series correlation methods can be used to
detect periodicity. There are considerable algorithms avail-
able for single ECG period detection, which have been ex-
tensively analyzed in the past using a variety of classical
methods including spectral analysis, time-frequency analy-
sis, wavelets, and machine learning methods (see survey at
(Koler et al. [4] and Tompkins [9]). One of the most popular
approaches is based on detection of the R peak or QRS peak
of the ECG waveform (Tompkins [9]). In Zimmerman et al.
[13], the R wave is detected as the largest positive peaks in
a 3 second sample window, after the ECG signal is first fil-
tered to remove the wandering zero-volt baseline. Similar
approaches were also used in Reisman et al. [6] except dif-
ferent kinds of bandpass filters were employed. For exam-
ple, those filters include LPC filter [4], two-pole recursive
filter [9], and bandpass filter in Reisman et al. [6]. These
approaches are all based on an assumption that R peaks are
the largest positive peaks in a ECG signal, which is not al-
ways true for ECG recordings. A modified feature-based
auto-correlation function is utilized in Syeda-Mahmood et
al. [8] and Wrublewski et al. [11], whereas the peaks in
the autocorrelation function correspond to the various pe-
riodicity patterns found in the signal. The most common
inter-peak duration is representative of a heart beat dura-
tion. This approach is however sensitive to the noise, and
it tends to produce a segment that is less than a heart cycle
due to multiple peaks within a heart beat.
The power spectrum of the ECG waveform can provide
useful information about the QRS complex. These spec-
trums can be generated using either Wavelet transforma-
tion (Saxena et al. [7] and Ghaffari et al. [3]) or Fourier
transformation (Tompkins [9]). The peaks of the frequency
spectrum obtained corresponds to the peak energy of the
QRS complex. For an overview of these methods, please
refer to Chapter 12 of Tompkins’ book [9]. These methods
typically require that the frequency (or the scale in wavelet
transform) of the QRS complex is known beforehand, so
that QRS candidate can be searched in a defined vicinity for
higher peaks.
More recently, machine learning methods have been an
increased trend towards heart period estimation [1]. For ex-
ample, in Mehta et al. [5], the Support Vector Machine
(SVM) was used as a classifier for detection of QRS com-
plexes in ECG; Vijaya et al. [10] propose to use Neural
Networks to estimate ECG periods. A critical difference be-
tween these learning based approaches is that these methods
require training of the ECG data, while we do not have any
such requirement.
Another source of ECGs is the synchronization ECG
used in echocardiograms which shows as a waveform em-
bedded in images as shown in Fig. 2c. The problem of
periodicity estimation is frequently encountered in lots of
spatio-temporal analysis of echo videos (Ebadollahi et al.
[2]). While it is difficult to estimate the heart cycle directly
from the depicted heart region in the video, it is relatively
easy to estimate heart rate from the synchronizing ECG.
Thus, methods are needed that can estimate the periodicity
from embedded ECGs in images.
2 Model
In this section, we describe our algorithm in detail. Our
method can be divided into two major steps: first we ex-
tract ECG envelopes from images, and then estimate the
period based on these image-based features. Once the ECG
is in the image form, either scanned or extracted from the
echocardiographic videos, the next important step is to pre-
process the data which will be useful in estimating the pe-
riod. Our approach of period detection from ECG image
is based on a key observation that the difference of the up-
per envelope and lower envelope of the ECG can signify the
local maximum & minimum of the original signal. The up-
per envelope and (shown in red) and lower envelope of the
ECG (shown in blue) are plotted in Fig. 4b. Note that the
gap between the two envelopes increase significantly when
the signal reaches its peak or valley. This phenomenon is
exploited in our algorithm to detect the heart rate, which
however cannot be used if we treat ECG as signals since
all points would have equal tracing width. We now first
discuss the extraction of ECG waveforms from the scanned
ECG images.
2.1 ECG envelope extraction
We extract the ECG waveforms as curves in the respec-
tive image segments where each lead position in the image
is segmented. Due to noise in recordings as well as the sty-
lus speed in the ECG recorder, there are often gaps which
cause problems in curve extraction. Note that these record-
ings are actually time series or functions of lead-voltage vs
time. Thus, a general purpose algorithm such as curve fol-
lowing or skeletonization, may not enforce the constraint
that a single y−value occur for each x position in image
coordinates. In places where the axis bifurcates or turns
back on itself, a post-processing step would be required for
choosing between multiple values of y for a particular value
of x.
Since the ECG curve may be multiple pixels thick at the
scanned resolution, our curve tracing algorithm follows the
upper and lower edges of the curve. Fig. 3 depicts these
curves, yu (x) and yl (x). In our tracing algorithm, we con-
tinue the curve trace from x − 1, thus extending yu (x − 1)
and yl (x − 1). We define two search ranges centered around
these values:
Ru = [yu (x − 1) − W, yu (x − 1) + W ] (1)
Rl = [yl (x − 1) − W, yl (x − 1) + W ] (2)
where W is a search window. In practice, W needs to be
set large enough to handle the voltage spike at an R wave.
Next, we define
yu (x) = min y such that T (x, y) = 0 (3)
y∈Ru
yl (x) = max y such that T (x, y) = 0 (4)
y∈Rl
which keeps yu and yl tracing the upper and lower en-
velopes of the ECG curve. After tracing the envelopes, the
average
y(x) = (yu + yl )/2 (5)
is used as the traced value y(x).
To provide robustness to noise and missing curve frag-
ments, we use morphological operators and a grouping al-
gorithm across the gaps. A morphological open (erode + di-
late) helps fill in holes inside the curve, and a morphological
close (dilate + erode) eliminates noise pixels near the curve.
To close small gaps, curve tracing is started using a num-
ber of seed points along the expected ECG curve location.
Gaps are closed between pairs of consecutive fragments if
the gap is small enough (4 pixels = 10 msec). For gaps that
are larger than the gap threshold, we have found that most
larger breaks in the curve occur at the R wave, where the
signal spikes up and down. At the R wave, however, the
curve is nearly vertical, so the sampling in equations (1)-(2)
is nearly tangent to the curve. Thus, we can handle a large
fraction of vertical dropout, as shown in Fig. 3. Fig. 4(b)
illustrates the upper and lower envelope of the ECG curve.
ydif f = yu −yl is the difference between the two envelopes,
which is shown in Fig. 4e. It is evident from this plot that
the peak of the difference signal is quite consistent among
all R peaks, and therefore can serve as a good feature to
identify the R peak locations.
2.2 ECG period detection
Differentiation forms the basis of many QRS detection
algorithms [12, 11]. The differentiator, in effect, acts as a

Figure 3. Illustration of curve tracing across
gaps in ECG images. Upper and lower
bounding curves of the ECG, yu and yl , are
shown in green and red. The R wave spikes
show a number of gaps in the original curve
that are correctly bridged by our tracing tech-
nique.
Figure 4. Illustration of various waveforms
from different stages of period detection al-
gorithm.
high-pass filter, which characterizes the steep slope of the
QRS complex of the ECG signal. Specifically, the first or-
der derivative of the ECG signal amplifies the higher fre-
quencies characteristic of the QRS complex while attenuat-
ing the lower frequencies of the P and T waves. For these
reasons, we take the product of the ECG derivative with the
envelope difference signal to filter out the high frequency
noise of the envelope difference signal. Differentiator fil-
ters have several different forms [4], we choose to approx-
imate derivative of the input ECG signal by the backward
difference.
Once the ECG envelopes are extracted from the image,
the final product consists of three waveforms, which is ex-
pressed as
−−→ −−−−−→ −−−→
S(x) = y(x) ∗ ydif f (x) ∗ y 0 (x) (6)
where → −
y is the normalized ECG waveform, in which the
wandering baseline is removed; −
y− −→
dif f is the normalized en-
→
−
velope difference (shown in Fig. 4e), and y 0 is the normal-
ized first order ECG derivative. The normalized product
waveform is shown in Fig. 4f, in which we noticed that
peaks of the product is corresponding to the R wave of the
ECG signal. The period of the ECG can therefore be esti-
mated as the interval between these peaks.
3 EXPERIMENTAL RESULTS
We now present experimental results on the application
of our algorithm to both ECGs synthesized from the digital
waveforms and those ECGs extracted from echocardiogram
videos. First, to demonstrate the robustness of our algo-
rithm, we apply our algorithm to an ECG from an atrial
fibrillation patient, in which the ECG waveform is quite
noisy and QRS complex are unpronounced. Regular meth-
ods such as a derivative-based method all failed to recover
the period, which is evident from the derivative waveform
in Fig. 5b. The product waveform S(x) produced from our
algorithm is shown in Fig. 5c. It is evident that our method
is able to pick up the R peak from the noisy waveform, even
when the noise level is high. Our method exhibits stronger
resistance to noise than the derivative-based and other filter-
based approaches.
Next, we conduct our experiments on a large collection
of digital ECG datasets, which we obtained from a large
hospital network. The database contains 16,613 12-channel
ECG waveforms. 13043 ECGs come with the ground truth
heart rate among the whole data collection. Using our
method, we are able to achieve 88% accuracy rate, where
the error between ground truth and the estimation is un-
der 5%. In comparison, cross-correlation is only able to
achieve a 72% accuracy rate. A closer examination of the
those cases in which the relative error from the ground truth
is higher than 5% reveals that the majority of those ECGs
are either i) arrhythmia cases where each channel gives a
distinct period estimate that confuses the algorithms, ii)
the ground truth for the ECGs are inaccurate. We believe
the accuracy of our algorithm can be further improved if
those cases are excluded. Additionally, of the 16613 ECGs,
we were able to additionally find a period for 2641 more
cases for which these patients had no documented periods.
We found 2538 cases of bradycardia, and 1541 cases of
tachycardia among those matching with the ground truth,
which represents 94.5% of the bradycardia and tachycardia
patients when compared with the ground truth disease la-
bels. Sample bradycardia and tachycardia ECGs that are
retrieved by our algorithm is shown in Fig. 6.
Finally, we apply our algorithm to estimate the heart cy-
cle from ECGs extracted from a large collection echocardio-
gram videos, which contains 1178 echocardiogram video
sequences. For each of these video clips, we first extract
the region of interest which contain ECG signal, and feed
it to our period detection algorithm. After we estimate the
period for all these videos, we randomly pick 100 videos
 Figure 5. Robustness in the presence of large noise. a) ECG image; b) first order derivative of the
ECG waveform; c) product waveform from our method.
(a) Bradycardia (b) Tachycardia
Figure 6. Sample bradycardia and tachycar-
dia ECGs retrieved by our algorithm.
with ground truth periods, we are able to achieve 84% ac-
curacy rate, where the error between ground truth and the
estimation is under 5%. We believe the main cause of er-
ror is from the variation of heart beat within single channel
ECG waveform embedded in the videos.
4 Conclusions
In this paper, we present a novel computational algo-
rithm as well as medical decision support tool for finding
the heart period from multimodal electrocardiogram (ECG)
documents. We exploit the pixel thickness information, for
the first time, as a reliable feature for determining period-
icity. Existing period estimation algorithm, such as estima-
tion of R peaks as largest positive peaks often leads to de-
tection error when R peaks are lower than the P or T peaks.
While our method is relatively robust to these cases since
our features latched on pixel dithering is more consistent
among corresponding R peaks. Furthermore, we are able
to correctly identify 94.5% of bradycardia and tachycardia
patient records (on a database of 16,613 ECG waveforms)
using our estimated heart period as the disease criteria. Fu-
ture work will focus on building a mathematical model for
the dithering effect so that our algorithm can be generalized
to documents of other digital signals.
References
[1] H. Bunke and A. Sanfeliu, editors. Syntactic and Structural
Pattern Recognition Theory and Applications. World Scien-
tific, 1990.
[2] S. Ebadollahi, S.-F. Chang, and H. Wu. Automatic view
recognition in echocardiogram videos using parts-based rep-
resentation. In CVPR, pages 2–9, 2004.
[3] A. Ghaffari, H. Golbayani, and M. Ghasemi. A new math-
ematical based qrs detector using continuous wavelet trans-
form. Comput. Electr. Eng., 34(2):81–91, 2008.
[4] B.-U. Koler, C. Hennig, and R. Orglmeister. The principles
of software QRS detection. IEEE Engineering in Medicine
and Biology Magazine, 21:42–57, 2002.
[5] J. Mehta and N.S.Lingayat. Comparative study of QRS
detection in single lead and 12-lead ecg based on entropy
and combined entropy criteria using support vector machine.
JATIT, 3:8–18, 2007.
[6] S. Reisman and W. Tapp. Variable threshold r wave detector:
Use in automated ecg processing. Physiology & Behavior,
35:815–818, 1985.
[7] S. C. Saxena, V. Kumar, and S. Hamde. Feature extraction
from ECG signals using wavelet transform for disease diag-
nostics. IJSS, 33:1073– 1085, 2002.
[8] T. Syeda-Mahmood, D. Beymer, and F. Wang. Shape-based
matching of ECG recordings. IEEE International Confer-
ence on Engineering in Medicine and Biology, pages 2012–
2018, 2007.
[9] W. J. Tompkins, editor. Biomedical digital signal process-
ing: C-language examples and laboratory experiments for
the IBM PC. Prentice-Hall, Inc., Upper Saddle River, NJ,
USA, 1993.
[10] G. Vijaya, V. Kumar, and H. Verma. ANN-based QRS-
complex analysis of ECG. J Med Eng Technol., pages 160–
167, 1998.
[11] T. A. Wrublewski, Y. Sun, and J. Beyer. Real-time early de-
tection of r waves of ECG signals. Engineering in Medicine
and Biology Society, pages 38 – 39, 1989.
[12] Y.-C. Yeh and W.-J. Wang. QRS complexes detection for
ECG signal: The difference operation method. Comput.
Methods Prog. Biomed., 91(3):245–254, 2008.
[13] T. Zimmerman and T. Syeda-Mahmood. Automatic detec-
tion of heart disease from twelve channel electrocardiogram
waveforms. In Computers in Cardiology, pages 809–812,
2007.