Journal of Biomedical Informatics 40 (2007) 236251

www.elsevier.com/locate/yjbin

Development, implementation, and a cognitive evaluation

of a denitional question answering system for physicians
Hong Yu a,*, Minsuk Lee a, David Kaufman b, John Ely c,
Jerome A. Oshero d, George Hripcsak b, James Cimino b
a
Department of Health Sciences, University of Wisconsin-Milwaukee, Enderis Hall 939, 2400 E. Hartford Avenue, P.O. Box 413,
Milwaukee, WI 53211, USA
b
Department of Biomedical Informatics, Columbia University, USA
c
Department of Family Medicine, University of Iowa College of Medicine, 200 Hawkins Drive, 01291-D PFP, Iowa City, IA 52242, USA
d
Thomson Healthcare, 4819 Emperor Blvd, Suite 125, Durham, NC, USA

Received 17 November 2005

Available online 12 March 2007

Abstract

The published medical literature and online medical resources are important sources to help physicians make patient treatment deci-
sions. Traditional sources used for information retrieval (e.g., PubMed) often return a list of documents in response to a users query.
Frequently the number of returned documents from large knowledge repositories is large and makes information seeking practical only
after hours and not in the clinical setting. This study developed novel algorithms, and designed, implemented, and evaluated a medical
denitional question answering system (MedQA). MedQA automatically analyzed a large number of electronic documents to generate
short and coherent answers in response to denitional questions (i.e., questions with the format of What is X?). Our preliminary cog-
nitive evaluation shows that MedQA out-performed three other online information systems (Google, OneLook, and PubMed) in two
important eciency criteria; namely, time spent and number of actions taken for a physician to identify a denition. It is our contention
that question answering systems that aggregate pertinent information scattered across dierent documents have the potential to address
clinical information needs within a timeframe necessary to meet the demands of clinicians.
Published by Elsevier Inc.

Keywords: Question answering; Information retrieval; Question analysis; Text summarization; Machine-learning; Evaluation

1. Introduction [2,8,9]. Information retrieval systems (e.g., PubMed) return

Physicians often have questions about the care of their
patients. The published medical literature and online med-
ical resources are important sources to answer physicians
questions [14], and as a result, to enhance the quality of
patient care [57]. Although there are a number of anno-
tated medical knowledge databases including UpToDate
and Thomson Micromedex that are available to physicians
to use, studies found that physicians often need to consult
primary literature for the latest information in patient care
*
Corresponding author. Fax: +1 414 229 5100.
E-mail address: HongYu@uwm.edu (H. Yu).
lists of retrieved documents in response to user queries.
Frequently, the number of retrieved documents is large.
For example, querying PubMed about the drug celecoxib
results in more than one thousand articles. Physicians usu-
ally have limited time to browse the retrieved information.
Studies indicate that physicians spend on average two min-
utes or less seeking an answer to a question, and that if a
search takes longer, it is likely to be abandoned [1,1012].
An evaluation study showed that it takes an average of
more than 30 min for a healthcare provider to search for
answer from the MEDLINE search engine, which means
information seeking is practical only after hours and
not in the clinical setting [13].

1532-0464/$ - see front matter Published by Elsevier Inc.

doi:10.1016/j.jbi.2007.03.002
 H. Yu et al. / Journal of Biomedical Informatics 40 (2007) 236251
Question answering can automatically analyze a large
number of articles and generates short text, ideally, less
than a few seconds, to answer questions posed by physi-
cians. Such a technique may provide a practical alternative
that enables physicians to eciently seek information at
point of patient care. This paper reports the research devel-
opment, implementation, and a cognitive evaluation of a
medical denitional question answering system (MedQA).
Although it is our long-term goal to enable MedQA to
answer all types of medical questions, we started with def-
initional question type because it tends to be more clear-cut
and constrained in the medical domain; this contrasts with
many other types of clinical questions that typically have
large variations in what reasonable answers might be.
2. Background
Although the notion of computer-based question
answering has been around since 1970s (e.g., [14]), the
actual research development is still a relatively young eld.
Question answering has been driven by the text retrieval
conference (TREC), which supports research within the
information retrieval community by providing the infra-
structure necessary for large-scale evaluation of text retrie-
val methodologies. TREC introduced question answering
(QA) track in 1999. TREC (2004) reported the best ques-
tion answering system to perform with 77% accuracy for
answering factoid questions (e.g., How many calories
are there in a Big Mac?) and 62.2% F-score for answering
list questions (e.g., List the names of chewing gums) [15].
In addition to factoid questions, TREC has since 2003 pro-
vided evaluation to scenario questions (e.g., denitional
question What is x?) that require long and complex
answers. Research development in scenario question
answering (e.g., [16,17]) has been supported by the
Advanced Research and Development Activity (ARDA)s
Advanced Question and Answering for Intelligence
(AQUAINT) program since 2001.
However, fewer research groups are working on medical
domain-specic question answering. Zweigenbaum [18,19]
provided an in depth analysis of the feasibility of question
answering in the biomedical domain. Rinaldi and col-
leagues [20] adapted an open-domain question answering
system to answer genomic questions (e.g., where was
spontaneous apoptosis observed?) with the focus on iden-
tifying term relations based on a linguistic-rich full-parser.
Niu and colleagues [21] and Delbecque and colleagues [22]
incorporated semantic information for term relation identi-
cation. Specically, they mapped terms in a sentence to
the UMLS semantic classes (e.g., Disease or Syndrome)
and then combined the semantic classes with surface cues
or shallow parsing to capture term relations. For example,
in the sentence the combination of aspirin plus streptoki-
nase signicantly increased mortality at 3 months the
word plus refers to the combination of two or more med-
ications [21] Xuang et al. [23] manually evaluated whether
medical questions can be formulated by problem/popula-
237
Fig. 1. A subset of clinical questions collected by Ely and his associates
[1]. The left bar represents generic question proportions. For example,
What, How, Do, Can account for 2231 (or 48%), 697 (or 15%),
320 (or 7%), and 187 (or 4%) of all clinical questions. Question examples
are on the right side of the bar.
tion, intervention, comparison, and outcome (PICO), the
criteria recommended by the practice of evidence-based
medicine. Yu [24] proposed a framework to answer biolog-
ical questions with images. None of systems described
above, however, reported a fully implemented question
answering system that generates answers in response to
users questions from a large text collection such as millions
of MEDLINE records and other World Wide Web collec-
tion as we are reporting in this study.
Empirical studies characterizing physicians information
needs and questions can illuminate dierent facets of the
question answering problem. Ely, DAlessandro and col-
leagues collected thousands of medical questions1 at clinic
setting [1,2527]. Fig. 1 lists a subset of these medical
questions.
Taxonomies were built to semantically organize medical
questions into types. Ely and colleagues [1] studied the 1396
medical questions they collected in one study to map to a
set of 69 question types (e.g., What is the cause of symp-
tom X?, and What is the dose of drug X?) and 63 med-
ical topics (e.g., drug or cardiology). Additionally, they
created an evidence taxonomy that incorporated ve
hierarchical categories [28]. The top categories were Clini-
cal or Non-clinical; the Clinical questions were further
divided into General versus Specic;General questions were
divided into Evidence and No-evidence; and Evidence ques-
tions were divided into Intervention versus No-intervention.
We developed supervised machine-learning approaches
to automatically classify a medical question into a question
type specied by the evidence taxonomy [28]. Using a
total of 200 annotated questions, our leave-one-out tenfold
cross-validation performance showed over 80% accuracy
for capturing questions that are answerable with evidence
1
The question collection is freely accessible at http://clinques.nlm.nih.
gov/.
238 H. Yu et al. / Journal of Biomedical Informatics 40 (2007) 236251
[29,30]. Because a specic answer strategy can be developed
for a specic question type, this paper reports research
development in Document Retrieval, Answer Extraction,
Summarization, and Answer Formulation and Presentation
for answering one specic question type; namely, deni-
tional questions.
To assess the ecacy of this system, we employ an
empirical study informed by a cognitive engineering
approach to the study of human computer-interaction
[31,32]. This is an interdisciplinary and task-centered
approach to the development of principles, methods, and
tools, to guide the analysis and design of computer-based
systems [33]. The goal is to understand how dierent sys-
tems mediate task performance. Although outcome mea-
sures such as accuracy, eciency, and ease of use are
important in this analysis, the focus is also on the cognitive
and behavioral processes involved in employing the system
to execute a task. In this context, we are interested in how
MedQA would aect question answering relative to a sys-
tem such as Google or PubMed. Cognitive methods of
evaluation have been used to study a range of informa-
tion-seeking tasks [3437]. Recently, Elhadad and col-
leagues [38] employed a video-analytic cognitive approach
to study the cognitive consequences of physicians use of
an automated summarization system that generates a sum-
mary that is tailored to patient characteristics. The ndings
indicate that physicians using the tailored summarizer were
better able to access relevant information with greater e-
ciency and accuracy relative to a generic summarizer.
3. Methods
Fig. 2 shows the MedQA overall architecture. In the fol-
lowing section, we will rst describe the text collection from
which answers are extracted. We will then describe the
development of Document Retrieval, Answer Extraction,
Summarization, Answer Formulation, and Presentation.
Finally, we will describe the cognitive evaluation of Med-
QA in which we compared MedQA with three other
Fig. 2. MedQA system architecture. MedQA takes in a question posed by
either a physician or a biomedical researcher. Question Classication
automatically classies the posed question as a question type for which a
specic answer strategy has been developed. Noun phrases are generated
as query terms. Document Retrieval applies the query terms to retrieve
documents from either World Wide Web documents or locally indexed
MEDLINE collection. Answer Extraction automatically identies the
sentences that provide answers to questions. Text Summarization con-
denses the text by removing the redundant sentences. Answer Formulation
generates a coherent summary. The summary is then presented to the user
who posed the question.
state-of-the-art information retrieval systems; namely,
Google, OneLook, and PubMed.
3.1. Text collection
While the heterogeneous World Wide Web collection
has shown to be comprehensive for open-domain question
answering, research has found that the information from
Web documents is not sucient for answering highly
domain-specic biomedical questions [3946]; on the other
hand, studies indicate that the biomedical literature is a
qualied resource that can answer questions posed by phy-
sicians [4,11,13,18]. MedQA is currently built upon both
the World Wide Web collection and the 15 million MED-
LINE records.
3.2. Document retrieval
Document retrieval returns a list of relevant documents
in response to a users query. We applied Google as the
search engine to retrieve the Web documents. We applied
the indexing tool Lucene2 to index documents from our
local text collections (i.e., MEDLINE abstracts). A recent
evaluation [47] shows that Lucene out-performed another
information retrieval tool Indri [48]. Since noun phrases
in general provide important content for each question,
we applied the shallow parser LTCHUNK [49] to extract
noun phrases and then applied the noun phrases as query
terms to retrieve documents. We evaluated LTCHUNK
to perform with 79.2% recall and 84.6% precision for cor-
rectly identifying noun phrases in medical questions based
on a randomly selected 100 medical questions [29].
3.3. Answer extraction
Answer extraction identies from the retrieved docu-
ments relevant sentences that answer the questions. We
developed multiple strategies to identify relevant sentences.
MedQA rst classies sentences into specic types.
Biomedical articles that report original research normally
follow the rhetorical structure known as IMRAD (Intro-
duction, Methods, Results, and Discussion) [5053]. Within
each section, there is a well-structured rhetorical substruc-
ture. For example, the introduction of a scientic paper
usually begins with general statements about the
signicance of the topic and its history in the eld [51]. A
previous user study found that physicians prefer the Results
sections to others for determining the relevance of an arti-
cle [54]. We found that denitional sentences were more
likely to be in the sections of Introduction and Background.
We have followed the previous approaches [55,56] to apply
supervised machine-learning approaches (e.g., nave bayes)
to identify dierent sections (e.g., Introduction, Back-
ground, Methods, Results, and Conclusion) [57]. The train-
2
http://lucene.apache.org.
 H. Yu et al. / Journal of Biomedical Informatics 40 (2007) 236251
ing set was generated automatically from the abstracts in
which sections were identied by the authors of the
abstracts. The trained classier was then used to automat-
ically predict the classes of MEDLINE sentences. This pro-
vided a total of 1,004,053 sentences for training, including,
86,971 Introduction, 70,850 Background, 248,630 Meth-
ods, 371,419 Results, 167,252 Conclusions, and 58,930
Others. Employing leave-one-out tenfold cross validation
with bag-of-words as features, the classier achieved
78.6% accuracy for identifying a sentence into one of the
ve sections.
In addition, MedQA further categorized sentences based
on linguistic and syntactic features. For example, in scien-
tic writing, it is customary to use the past tense when
reporting original work and present tense when describing
established knowledge [50,51]. Biomedical literature
reports not only experimental results, but also hypotheses,
tentative conclusions, hedges, and speculations. MedQA
applied cue phrases (e.g., suggest, potential, likely, may,
and at least) identied by [58], which was reported to out-
perform machine-learning approaches, to separate facts
from speculations. Factual sentences were selected for cap-
turing denitions.
The denitional sentences were identied by lexico-syn-
tactic patterns. For example, qureyterm, Formative Verb
(e.g., is and are, Noun Phrase) can be used to iden- The RlogF measure [64] balances a patterns conditional
tify a denitional sentence such as vulvar vestibulitis syn-
drome (VVS) is a common form of dyspareunia in
premenopausal women to answer a question such as
What is vulvar vestibulitis syndrome? In contrast with
other state-of-the-art denitional question answering sys-
tems [5962] that mostly captured lexico-syntactic patterns
manually, our system automatically learned lexico-syntac-
tic patterns from a large collection of training set that
was created automatically.
We automatically learned the lexico-syntactic patterns
from a large set of Google denitions. Specically, we
applied all of the terms that are included in the Unied
Medical Language System (UMLS 2005AA) as candidate
denitional terms, and crawled the Web to search for
denitions. We built our crawler on the Google:Denition
service. Google:Denition provides denitions that seem
to mostly come from web glossaries. We found that
36,535 terms (from the total of 1 million) to have deni-
tions specied by the Google. We therefore downloaded
a total of 191,406 denitions; the average number of deni-
tions for each denitional term was 5.2.
With this set of denitions, we then automatically iden-
tied lexico-syntactic patterns that comprise the deni-
tions. We applied an information extraction system
Autoslog-TS [63,64] to automatically learn the lexico-syn-
tactic patterns. In the following, we will describe the Auto-
Slog-TS and how we applied it for lexico-syntactic pattern
learning.
AutoSlog-TS is an information extraction system that
automatically identies extraction patterns for noun
phrases by learning from two sets of unannotated texts.
239
In our application, one collection of text incorporates
relevant or denitional sentences, and another collection
of text is irrelevant or background because it incorporates
sentences that are randomly selected from the MEDLINE
collection.
AutoSlog-TS rst performs part-of-speech tagging and
shallow parsing, and then generates every possible lexico-
syntactic pattern within a clause to extract every noun
phrase in both collections of relevant and irrelevant texts.
It then computes statistics based on how often each pattern
appears in the relevant texts versus the irrelevant texts and
produces a ranked list of extraction patterns coupled with
statistics indicating how strongly each pattern is associated
with relevant and irrelevant texts. The extraction patterns
are ranked based on how often each pattern appears in
the relevant texts versus the irrelevant texts. For each
extraction pattern, AutoSlog-TS computes two frequency
counts: totalfreq is the number of times that the pattern
appears anywhere in the corpus, and relfreq is the number
of times that the pattern appears in the relevant texts. The
conditional probability estimates the likelihood that the
pattern is relevant:
 
relevant relfreq
P
pattern totalfreq
probability estimate with the log of its frequency:
 
relevant
R log F pattern log2 relfreq  P
pattern
The R log F measurement were evaluated in a number of
information extraction tasks to be robust [65]. We there-
fore used R log F measure to rank the lexico-syntactic pat-
terns we generated by Autoslog-TS. We then implemented
the patterns into MedQA to capture denitional sentences.
3.4. Summarization and answer formulation
Summarization condenses a stream of text into a shorter
version while preserving its information content [66]. Since
we obtain sentences from multiple documents, frequently,
many of the extracted sentences are redundant. Summari-
zation attempts to remove the redundant sentences and
present to the users a short coherent summary that maxi-
mizes the coverage of information content. Summarization
has been an active research eld in natural language pro-
cessing (NLP). Open-domain summarization techniques
consider word distribution [67], which is based on the intu-
ition that the most frequent words represent the most
important concepts of the text; linguistic information [68],
which includes meta-linguistic markers (e.g., in conclu-
sion); location [69], which relies on the intuition that the
headings and sentences at the beginning and end of the text
may contain important information; rhetoric, which refers
to the analysis of discourse structure for identifying the
main claims of texts [70]; cohesion [66,71], which captures
semantically related terms, co-reference, ellipsis and
240 H. Yu et al. / Journal of Biomedical Informatics 40 (2007) 236251
conjunctions; and popularity [72], which is based on the
intuition that the most popular content across multiple
documents is the most important; as well as techniques
for improving summarization coherence [73] Information
extraction [74,75] were explored for summarizing medical,
domain-specic texts; however, the systems neither were
generalizable to the entire biomedical domain, nor output-
ed summaries.
Since the focus of summarization is to remove redun-
dant sentences, we cluster similar sentences into a group
based on the assumption that redundant sentences share
certain lexical similarities. MedQA applies information
retrieval techniques to summarize sentences. It clusters sen-
tences based on TF*IDF weighted cosine similarity [76]
and selects the most representative sentence from each clus-
ter. MedQA clusters sentences with hierarchical clustering
algorithms, which are widely used for other tasks such as
topical document clustering [77], and generate summaries
using sentences from dierent clusters. Specically, Med-
QA implement two hierarchical clustering algorithms,
namely, group-wise average and single-pass clustering. In
the following, we will describe both algorithms:
Group-wise average starts with the entire set of N sen-
tences to be summarized. It identies pair-wise sentence
similarity based on TF*IDF word features. It then merges
the two sentences with the highest similarity into one clus-
ter. It then re-evaluates pairs of sentences/clusters; two
clusters are mergeable if the average similarity across all
pairs of sentences within the two clusters is equal to or
greater than a predened threshold. The computational
complexity of group-wise average is 12  N  N  1 N 3 .
Single-pass clustering starts with one sentence randomly
selected from the entire N set of sentences to be summarized.
When adding in the second sentence, it calculates the similar-
ity between the two sentences and clusters the two sentences
if the similarity is above a predened threshold. It continues
to add in additional sentences. When an add-in sentence are
compared with a cluster containing several sentences, the
similarity is the average similarity of the add-in with all of
the sentences in the cluster. The computation complexity
of single-pass clustering is 12  N  N  1.
Single-pass clustering makes a decision for each sentence
as soon as the sentence is rst judged, and therefore the
process of clustering the entire sentence set is faster than
with group-wise average. On the other hand, group-wise
average outperforms single-pass clustering in topical docu-
ment clustering [78]. We balances the advantages of both
algorithms by applying single-pass clustering when we have
a large number of sentences (>150) and group-wise average
when we have a smaller number of sentences.
MedQA applies the centroid-based summarization tech-
nique [79] to select the most representative sentence and
then to generate a coherent summary. The method rst
selects from each cluster one sentence that has the highest
similarity to the rest of the sentences within the cluster.
Then the selected sentences are ordered based on the simi-
larity to the rest of selected sentences.
3.5. Incorporating Web denitions
We found that the Web is a rich resource for denitions.
On the other hand, the denitions from the Web could be
of mixed quality. For example, many online denitions
were not related to the biomedical domain. For example,
heart was dened as both . . .one of the most successful
female fronted bands in the annals of hard rock and a
hollow, muscular organ that pumps blood through the
blood vessels by repeated, rhythmic contractions; the lat-
ter is useful for MedQA, while the former is non-relevant.
In order to lter out non-relevant denitions and sepa-
rate them from biomedical domain-specic denitions,
we develop a component to identify denitions in the
World Wide Web documents and to incorporate useful
Web denitions as a part of answer output. We implement
Google: Denition to capture denitions from the World
Wide Web. We also capture other online dictionaries
including Dorlands Illustrated Medical Dictionaries. For
each Web denition, we then measure the similarity (i.e.,
TF*IDF) between the denition and the retrieved MED-
LINE abstracts. We select a Web denition if it has the
highest similarity score and yet the score is above an ad
hoc threshold we had dened.
3.6. Display
MedQA presents a user with both Web denitions and
MEDLINE sentences. Since we found that the Web deni-
tions are in general clearly stated and general-population
targeted, we decided to display with an order that Web def-
initions appeared rst and then followed by MEDLINE
denitions. We chose to select the top ve MEDLINE sen-
tences in a summary. Since the clustering step frequently
generated a large number of clusters, displaying only the
summary to the user meant that MedQA would potentially
discard the rest of clusters that might be additionally
important to the biologists or the physicians. MedQA dis-
played both the summary and other relevant sentences
to the user. Fig. 3 shows an output generated by our system
in response to the question What is vestibulitis? MedQA
links each sentence to the original document resources,
either on the Web, or the locally indexed MEDLINE
collection.
3.7. Software environment
MedQA is implemented with Perl as the core platform
and is running on a Macintosh PowerPC (dual 2 GHz
CPU, 2 GB of physical memory, Mac OS X server
10.4.2). We chose Perl in this prototype phase because it
oers advantages of having fast debugging capability,
availability of numerous modules which provide ecient
implementations for many well-known data structures
and algorithms, and easy integration with multiple compo-
nents via inter-process communication capability. The dis-
tributions of time spent among dierent components were
 H. Yu et al. / Journal of Biomedical Informatics 40 (2007) 236251 241

Fig. 3. MedQAs output of the question What is vestibulitis? The output displays an online denition that comes from Dorlands Illustrated Medical
Dictionaries, a summary that incorporates denitional sentences that are extracted from dierent PubMed records, and other relevant sentences that
incorporate other relevant sentences. The parenthetical expression incorporates the last name of the rst author and the year of the publication (e.g.,
Sackett, 2001); the expression links to the PubMed records from which the preceding sentences are extracted.

6 s for Document Retrieval, 6 s for Answer Extraction, and
6 s for Text Summarization. The full-parsing were done o-
line.
3.8. Cognitive evaluation
We design a randomized controlled cognitive evaluation
in order to assess the ecacy, accuracy, and perceived ease
of use. MedQA is compared to three other systems;
namely, Google, OneLook, and PubMed, all of which rep-
resent the state-of-the-art web-based information retrieval
systems. OneLook is a portal for numerous online dictio-
naries including several medical editions (e.g., Dorlands).
The study was approved by the Columbia University Insti-
tutional Review Board.
3.9. Question selection
The denitional questions were selected from a total of
138 denition questions out of 4,653 questions3 posed by
physicians at various clinical settings [1,2527]. The 138
denitional questions are listed in Appendix A. We
observed that the denitional questions in general fell into
several categories including Disease or Syndrome, Drug,
Anatomy and Physiology, and Diagnostic Guideline. In
order to maximize the evaluation coverage, we attempted
to select questions that covered most of these categories.
3
The question collection is freely accessible at http://clinques.
nlm.nih.gov/.
After preliminary examination, we found that many
questions did not yield answers from two or more systems
to be evaluated. For example, the question what is
proshield? did not yield a meaningful answer from three
systems (MedQA, OneLook, and PubMed). The objective
was to compare dierent systems, and unanswerable
questions present a problem for the analyzes because they
render such comparisons impossible. On the other hand,
if we screened the questions with the four systems, it may
introduce bias and a selective exclusion process. We
therefore employed an independent information retrieval
system, BrainBoost,4 which is a web-based question
answering engine that accepts natural language queries.
BrainBoost was presented with questions randomly
selected from the categories of denitional questions, and
the rst 12 questions that returned an answer were included
in the study. The question selection task was performed by
an unbiased assistant who was not privy to the reasons for
doing the search. The 12 selected questions are included in
Appendix A in bold font.
3.10. Subjects and procedure
Four physicians (one male and three female, age 3060)
with experience using information systems volunteered to
participate in the study. Each physician was presented with
all of the 12 questions selected for inclusion. For each ques-
tion, the subjects were asked to evaluate two systems in
succession. The order of the questions and the systems
4
http://www.brainboost.com/.
242 H. Yu et al. / Journal of Biomedical Informatics 40 (2007) 236251

Table 1
Actions used to answer questions
Enter Query: Entering a search term in the search text box provided by the system
Find Document: An action that involves >10 s of time spent examining the retrieved list of documents (e.g., web documents or PubMed abstracts)
Select Linkout: An action that involves another linkout (i.e., clicking on embedded link) from the selected document
Query Modication: A user modies the existing query or user-interface (for example, change to Scholar.google)
Read Document: An action that involves the subject spending >10 s to read the selected document
Scroll-Down Document: Scroll down the document for searching for an answer
Search Text Box: A subject initiates the Find function to identify the related text
Select Document: The subject selects and opens a document to examine whether the answer appears in the document
Select Text as Answer: A subject selects the text as answer to the question

was counterbalanced. Each subject posed six questions to
each of the four systems in pairs of two. The four subjects
therefore posed a total of 96 questions (12 4 2).
After consenting to participate in the study, participants
were given written instructions on how to perform the task.
They were presented with each question on a cue card and
asked to nd the text that best answered the question.
The card also indicated the two systems to be used and their
sequence. Once the text was located, they were asked to
copy and paste it into a Word document. They were free
to continue to search and paste text into the document until
they were satised that they found the best answer possible.
There was a time limit of 5 min for each question/system
event. Participants were asked to think-aloud during the
entire process. We chose 5 min as a cuto because a previ-
ous study found that internet users successfully found
health information to answer questions in an average of
5 min [42]. After completing each question evaluation com-
paring the two systems, they were asked to respond to the
following two Likert items: (1) rate the quality of answer
and (2) the ease of use of the system. We employed a ve
point rating scale 15 in which 1 represents the poorest
and 5 represents the best. We applied Morae usability
software system to record the screen activities and audio
record a subjects comments for the entire session. Morae
provides a video of all screen activity and logs a wide range
of events and system interactions including mouse clicks,
text entries, web-page changes, and windows dialogue
events. It also provides the analyst with the capability to
timestamp, code, and categorizes a range of video events.
3.11. Analysis
On the basis of a cognitive task analysis [32,35], we iden-
tied goals and actions common to all systems. Table 1
shows a list of actions we dened. We also noted system
responses (e.g., what was displayed after executing a
search), analyzed comments thematically and measured
the response times. The protocols were coded two analysts
(H.Y. and D.K.). The total coding time for four subjects
was about 30 h.
3.12. Evaluation results
In the following section, we present results of the cogni-
tive evaluation. The rst part of this section illustrates the
processes of question-answering. We also show the coding
process used to characterize participants actions. The sec-
ond part of this section focuses on a quantitative compar-
ison of the four systems. We include both objective
measures such as actions and response latency, and subjec-
tive measures, namely, participants ratings of the quality
of answers as well as their ease of use.
3.13. Illustration
The following two coding excerpts illustrate the process
of question-answering on two pairs of systems, OneLook
and MedQA and PubMed and MedQA. The excerpts are
representative of task performance. The subject was an
experienced physician with a master in informatics and
was well-versed in performing medical information seeking
tasks.
3.14. Excerpt 1OneLook and MedQA
The subject had completed four questions and was a lit-
tle more than 30 min into the session. The current question
answered was What is Bests Disease. The systems used
to nd the answer are OneLook and MedQA, respectively.
The entire segment was 5:12 min, of which 3:27 was used to
search the OneLook system and 1:36 to search MedQA
(Fig. 4).
3.14.1. Excerpt from a subjects protocol
31:23 ACTION (ENTER QUERY-ONELOOK): Best
Disease
31:30 System Response:4 dictionaries
including 2 general and 2 medical (see Fig. 4)
31:32 (User) COMMENT: So there are two
dictionary matches. . . for medicine and two
dictionary matches in general
31:51 ACTION SELECT DOCUMENT
COMMENT: Wikipedia is available and I would
just take a guess that Im going to get the
best information from the Wikipedia. . . and
Im 100% wrong.
31:54 SYSTEM RESPONSE Web Page Changes
Wiki
COMMENT: The only thing I get from the
Wikipedia outlink is a one line definition
of Best disease
 H. Yu et al. / Journal of Biomedical Informatics 40 (2007) 236251
Fig. 4. A screen shot of OneLooks return of the search term Bests Disease.
32:10 ACTION SELECT TEXT (pastes text into
Word document)
and Ill put in the system
32:42 COMMENT: So I am going to go back to
OneLook and have a look at Dorlands Medical
Dictionary
32:10 ACTION SELECT DOCUMENT (clicks on
link) Dorlands Medical Dictionary and see
if I can get a better definition, looking at
a specific area
32:49 SYSTEM RESPONSE Web Page Changes
Dorlands Medical Dictionary
COMMENT: And I get a . . . worse definition. . .
Possibly, I better go back and check. I could
have a different definition
33:05 ACTION SELECT TEXT (pastes text into
Word document)
Lets see I get a definition of congenital
macular degeneration. I think thats the
same I got in Wikipedia
33:25 COMMENT Ah I get different information,
it does give you the idea that it is an
inherited congenital disease and that it
affects vision. It does not specifically
talk about congenital macular
degeneration, youd have to outlink on
congenital macular degeneration to get
more information. about that illness
33:39 ACTION LINKOUT (clicks on congenital
macular degeneration)
33:39 SYSTEM RESPONSE Web Page Changes
Dorlands Medical Dictionary (MerckSource
resource library)
COMMENT: and there is. . . again minimal
information based on the outlink
33:59 ACTION SELECT TEXT (pastes text into
Word document)
243
32:10 ACTION SELECT DOCUMENT (Online Medical
Dictionary)
COMMENT: Do I will give it one more try and see
if I can get better information. I will try
the online medical dictionary
34:19 SYSTEM RESPONSE Web Page Changes
Online Dictionary Search
34:24 ACTION Mouses over links (does not
select one)
COMMENT: which is totally unhelpful
The participant selected the Wikipedia from the four
choices and expresses dissatisfaction with the sparse
response. He decided to continue to look for the answer
in Dorlands Medical Dictionary and much to his sur-
prise, little detail was provided. A third try with the
online medical dictionary yielded the statement autoso-
mal retinal degeneration in the rst several years of life
with several of the terms hyperlinked (e.g., to dene
retinal). The information was similar to that oered
by the other two sources and he stated that it was
totally unhelpful.
34:59 ACTION Go to MedQACOMMENT: OK, So we
are going to try MedQA
34:59 SYSTEM RESPONSE Web Page Changes
MedQA
COMMENT: I am waiting .. it is searching
Medline, it is searching the Web, done,
done. . . identifying filtering that return
35:32 SYSTEM RESPONSE Web Page Changes
MedQA (returns answer) Results and
Conclusion (reads from display) Summary
from MedLine and Wikipedia
5:33 Reviews results
5:55 COMMENT: A much. . . much much better ah
return
244 H. Yu et al. / Journal of Biomedical Informatics 40 (2007) 236251
5:59 ACTION SELECT TEXT (pastes text into
Word document)
35:59 COMMENT: With all of the annotations I
would have had to have searched manually
or pulled together and its all
coordinated to give me a complete picture
of what the disease is
So ok
So I would say that OneLook, the quality of
the answers I would say, I would just say
was poor for what I would need and the ease
of use of the system that was also poor, it
required me to make multiple guesses at
which resources would provide me with ah. . .
comprehensive definition and that could
help me as a physician
36:35 COMMENT: MedQA The quality of the
answer was excellent and the ease of use
was absolutely. I mean. . . perfect just
plain simple, I did not have to do
anything extra and I got exactly what
I needed
Excerpt 2PubMed and MedQA The subject had
completed five questions and was a little
more than 40 min into the session. The
question in this excerpt was What is
vestibulitis? The systems used to find
the answer were PubMed and MedQA,
respectively. The entire segment lasted
6 min, of which 4:25 is used to search
PubMed and 1:11 to search MedQA
44:23 ACTION (Enter Query-PubMed):
vestibulitis
44:34 SYSTEM RESPONSE: 251 MEDLINE records
returned
44:51 (User) COMMENT: OK, I definitely got
some answers that do not apply at all. . . I
have no idea why the first set of returns
are coming back with psychological
problems, but maybe not true, as a
physician just makes assumption of that
ENT would be returned, but if I am
gynecologist, that probably is what I am
looking for. Vulvar vestibulitis, I have
no idea what it is. I guess I will go find
out because I do not know
45:22 ACTION SELECT DOCUMENT
45:23 ACTION SELECT FULL-TEXT OUT-LINK
45:24 SYSTEM RESPONSE: Out-link failed
45:25 ACTION SELECT FULL-TEXT OUT-LINK
45:26 SYSTEM RESPONSE: Out-link failed
45:33 ACTION FIND DOCUMENT
COMMENT: No. . . I cannot find any definitions
46:11 ACTION (Query Modification,
vestibulitis) COMMENT: Try
vestibulitis only
46:14 SYSTEM RESPONSE: 251 MEDLINE records
returned
46:17 ACTION SELECT DOCUMENT COMMENT: Just
try this one, surgical treatment of vulvar
vestibulitis, this seems to be a good
definition
46:29 ACTION SELECT FULL-TEXT 46:39 SYSTEM
RESPONSE: Out-link failed
46:40 ACTION SELECT LINKOUT (of the full-
text article)
46:41 SYSTEM RESPONSE: Out-link failed
COMMENT: It does not seem to have any
outlink, it is only the abstract. The
abstract does not give any
characteristics of what syndrome is
47:10 ACTION SELECT TEXT AS ANSWER
47:49 ACTION FIND DOCUMENT
47:57 ACTION SELECT DOCUMENT
48:00 ACTION SELECT FULL-TEXT (PDF FILE)
48:02 ACTION READ DOCUMENT
COMMENT: seems to get pain syndromes
48:48 ACTION SELECT TEXT AS ANSWER
COMMENT: OK, I am going to leave PubMed
49:12 ACTION (ENTER QUERY-MEDQA): What is
vestibulitis?
COMMENT: MedQA uses MEDLINE, probably
will return the same information,
hopefully, it will get other information
as well.
49:52 SYSTEM RESPONSE: shown in Fig. 3
COMMENT: OK, MedQA pulls back exact the same
information, nothing else
50:23 ACTION SELECT TEXT AS ANSWER
GENERAL COMMENT: I would say that PubMed
again all the information was there but
was not held in a useful fashion and I need
to search all and I have to filter
myself. . .and quality of answer was OK and
ease of use is poor because I need to go
through everything. MedQA quality of
answer is excellent and ease of use is
excellent, I do not need to do anything
MedQA returned an integrated summary drawing on
Medline abstracts and text from the Web. Only a single
act of search was required and the information provided
was much more comprehensive. The participant com-
mented on the dierence in the quality of answer and in
the ease of use. The examples were not meant to demon-
strate the superiority of MedQA to the other systems, but
to characterize the dierence in the question answering
process and in the ways the systems support interaction.
For example, subjects would iteratively search PubMed
and OneLook until they found a satisfactory answer. As
a consequence, they would examine multiple documents
 H. Yu et al. / Journal of Biomedical Informatics 40 (2007) 236251 245

Table 2
A summary of comments of dierent systems (D for disadvantages and A for advantages)
Google (D) retrieves back a lot of links (to the question What is cubital tunnel syndrome?). Most of links seem to relate to individual cases of the
diseases, not necessarily denitions
Google (D): One needs to search and evaluate the denitions in Google
Google (A) retrieves both patient (Google) and physician-centric (scholar. Google) information
Google (A): Scholar.Google is much faster because it is the second link, while in PubMed the evaluator has to search through a lot of other articles
MedQA (D) needs to type in What is versus a direct query
MedQA (D) takes a considerable longer time to respond than other systems
MedQA (A) returns all the context otherwise the evaluator has to search manually. It is only one step and gets exactly needed
MedQA (A) gives answer (to the question What is Popper?) that OneLook did not, which is that the drug is injectable, which is important to know
for a physician
OneLook (D) pulls all links. It lets the user to guess which link contains a comprehensive answer. Sometimes, the links are broken. It is a matter of luck
to get to the right links
OneLook (D) answer quality is poor. It has a terrible user-interface. It shows two ugly photos
OneLook (A) denition has more content than PubMed.
PubMed (D) is not a good resource for denitions
PubMed (D) is not useful. It takes forever to nd information

(necessitating nd link and Linkout actions), only a few of We observed that none of the subjects used Google:Deni-
which were relevant. The subjects typically searched for tion as the service to identify denitions; instead, they
full-text articles as the Linkout actions. The iterative nature applied the query terms in Google or Scholar.Google.
of the search was also evidenced by the number of actions MedQA achieved the second highest ratings in both mea-
pertaining to query modication, searching the text box sures. OneLook received the lowest ratings for quality of
and document selection. answer and PubMed was rated the worst in terms of ease
Table 2 lists a summary of the comments made by sub- of use. All subjects gave the poorest score when there
jects throughout the evaluation. Our results show that was no answer found.
Google received more favorable comments than com- While the processing time to obtain an answer was
plaints. Both MedQA and OneLook received some good almost instant for Google, OneLook, and PubMed, the
comments and some complaints. PubMed was generally average time spent for MedQA to obtain an answer to
criticized and was not given any favorable comments. the 10 answerable questions was 15.8 7.1 s. Surprisingly,
MedQA was the fastest system in average for a subject to
3.15. Quantitative evaluation obtain the denition. For measuring the average time
spent, we excluded the cases in which MedQA and One-
The results show that the subjects did not nd answers Look returned no answer.
to one question in Google (dawns syndrome), three The subjects, on average, spent more time searching
questions in OneLook (epididymis appendix, hell pain PubMed than any of the other systems. In fact, the average
syndrome, and ottawa knee rules), three questions in
MedQA (epididymis appendix, Ottawa knee rules,
and paregoric), and two questions in PubMed (epidid- Table 4
ymis appendix and paregoric). Both MedQA and One- Average frequency of actions by category for each question
Look acknowledged no results found and returned no Google MedQA OneLook PubMed
answers if such an event occurs, while PubMed and Google Find link 0.4 0.1 0.1 1.2
returned a list of documents although subjects could not Linkout 0.2 0.1 1.2 1.5
identify the denitions from the documents within the Read document 0 0.3 0 0.8
5 min time limit. Query modication 0.2 0.2 0.6 0.6
Table 3 presents descriptive statistics of the subjective Scroll-down document 0.3 0.04 0.4 0.2
Search text box 0.2 0.2 0.1 1.4
and objective measures based on all 12 evaluated questions. Select document 1.7 0.5 2.4 3.1
In general, Google was the preferred system as reected
Select text as answer 1.4 0.7 1.6 1.3
both in the quality of the answer and ease of use ratings.

Table 3
Average score and (standard deviation) of quality of answer and ease of use and average time spent (in second) and action taken
Google MedQA OneLook PubMed
Time spent 69.6 (6.9) 59.1 (57.7) 83.1 (63.6) 182.2 (85.8)
Number of actions 4.4 (3.0) 2.1 (2.0) 6.5 (7.7) 10.3 (5.7)
Quality of answer 4.90 (0.15) 2.92 (0.24) 2.77 (0.08) 2.92 (0.88)
Ease of use 4.75 (0.29) 4.0 (0.24) 3.9 (0.32) 2.36 (0.88)
246 H. Yu et al. / Journal of Biomedical Informatics 40 (2007) 236251
PubMed search required more than three times the amount
of time required to search MedQA. This is at least partly
due to the complexity of the interaction. This is borne
out by the fact that participants needed more than 10
actions in using PubMed to answer the question, whereas
they only required two actions on average when they used
MedQA. PubMed provides a range of aordances (e.g.,
limits, MeSH) that supports iterative searching. Although
this is a powerful tool, it also increases complexity of the
task and user cognitive load. MedQA oers the simplest
mode of interaction because it eliminates several of the
steps (e.g., upload documents, search text, and selectively
access relevant information in document) involved in
searching for information. The results of the commercial
search engines, Google and OneLook, fell in between Med-
QA and PubMed. However, as evidenced by the high stan-
dard deviations, there was signicant variability between
questions.
The frequency of actions by category is reported in
Table 4. The pattern of actions employed by participants
reects the nature of interactions supported by each sys-
tem. For example, subjects would iteratively search Pub-
Med till they found a satisfactory answer. As a
consequence, they would examine multiple documents
(necessitating nd link and linkout actions), only a few of
which were relevant to dening the query term. The sub-
jects typically searched for full-text articles as the linkout
actions. The iterative nature of the search was also evi-
denced by the number of actions pertaining to query mod-
ication, searching the text box, and document selection.
4. Discussion and future work
This paper reports the development and evaluation of a
medical question answering systemMedQA. MedQA
identied noun phrases in a question and applies them as
the query terms to retrieve relevant documents. MedQA
further identied denitions based on linguistic features,
removed redundant sentences with robust statistical hierar-
chical clustering, and generated a coherent summary with a
centroid-based summarization method [79].
The contributions of this study are threefold. First, we
automatically identied a set of lexico-syntactic patterns
for capturing denitions, while most of the previous work
in denitional question answering identied the patterns
manually [5962]. Second, MedQA built upon four
advanced techniques; namely, question analysis, informa-
tion retrieval, answer extraction, and summarization to
generate a coherent answer to denitional questions; None
of the previous domain-specic question answering systems
reported the integration of the summarization techniques.
Third, we evaluated MedQA and three other state-of-the-
art information systems using a cognitive model.
Our evaluation results show that MedQA out-per-
formed OneLook and PubMed in most of the evaluation
criteria. All subjects gave Google almost perfect subjective
scores in quality of answer. The results demonstrated that
Google may be a reliable resource for physicians in identi-
fying denitions. Our results contrast with most of the pre-
vious reports that show the Internet information content
were of poor quality [44,8088], although there are dier-
ences between our study and the others. We focused on a
more general type of questions; namely, denitional ques-
tions, while the other works examined more specic medi-
cal questions (e.g., What further testing should be ordered
for an asymptomatic Thyroid Nodule solitary thyroid nod-
ule, with normal TFTs? in [89]). Second, physicians would
evaluate Google high if they found answers from the sites
even if some of the sites did not provide answers to the
questions. Other studies, in contrast, evaluated whether
answers were present in all Web sites or selected Web sites.
For example, one study [90] concluded that Google hits
were of a poor quality because only one link out of ve
led to a website with relevant information. Last, in our
study, the quality of answer was judged by the nal collec-
tion of answers that may be copied-and-pasted from multi-
ple Web pages. Other studies evaluated the completeness of
each single Web page to answer a specic question. Such
evaluation will certainly lead to a much poorer rating of
the Internet because one evaluation study [35] concluded
that information were scattered in the Internet: most of
the Web pages incorporate information either in depth or
in breath, and few Web sites incorporate information both
in depth and in breath.
Although the evaluation results show that Google was
preferred system in both quality of answer and ease of
use, the results show that MedQA out-performed Google
in time spent and number of actions, two important criteria
in eciency for obtaining an answer. Our results show that
although it took less than a second for Google to retrieve a
list of relevant documents based on a query keyword and it
took an average of 16 seconds for MedQA to generate a
summary, the average time spent for a subject to identify
a denition was 59.1 57.7 s for MedQA, which was faster
than 69.6 6.9 s for Google. A recent study [35] evaluated
high quality and authoritative medical websites and found
that information was scattered across the web: while most
of web pages had only few facts related to a specic topic,
relatively few web pages that had many facts. A user may
have to visit several sites before nding all of the pertinent
information. Our study conrmed that a subject needed to
visit multiple web pages in order to obtain the information
he/she needed because denitions were scatter across the
web pages in a Google search. The ability of MedQA to
integrate disparate pieces of text makes MedQA superior
than other search engines.
Throughout MedQA development, we identied a num-
ber of important research areas. Our current system imple-
mented the shallow syntactic chunker LT CHUNK to
capture noun phrases as query terms for answer extraction.
However, we found that LT CHUNK made many mistakes.
For example, LT CHUNK fails to identify eating disorder
in the question What is eating disorder?. The facts that LT
CHUNK was trained on general English text, and not med-
 H. Yu et al. / Journal of Biomedical Informatics 40 (2007) 236251
ical, domain-specic text, and that LT CHUNK was mostly
trained on regular sentences, not questions have greatly
undermined the capacity of LT CHUNK to eciently cap-
ture noun phrases of medical questions. A comprehensive
biomedical question answering system needs a robust and
accurate parser that is specically developed in the biomed-
ical domain. Such a parser will also be useful for capturing
lexico-syntactic patterns in answer extraction.
Another important area of a successful question answer-
ing system is to nd ways to capture users intentions to
determine the scope of answers. For example, when a phy-
sician asks What is the dawn phenomenon? he wants to
know not only the denition of this term, but also how to
diagnose it and manage it. Essentially, a denition question
(i.e., What is X?) requires answers beyond denitions
(e.g., what causes X? and How to treat X?). One can
obtain the users intentions by working directly with the
target users (e.g., physicians or biologists) throughout
MedQA development.
Summarization is an important research area. Summari-
zation incorporates three areas; namely, removing redun-
dant sentences, identifying important information, and
generating a coherent summary. Currently we implemented
a simple summarization that is based on statistical cluster-
ing. Future work one may combine statistical approaches
with both linguistic and semantic approaches such as the
one proposed by [75].
Finally, semantic information plays an important role
for both answer extraction and summarization. For exam-
ple, the following are denitions for heart and heart
attack, in which we have mapped terms to the UMLS
semantic types in [Superscript]. We link multi-word terms
(e.g., myocardial infarction) with underscore _ (e.g., answer; both advances showed the promise for MedQA to
myocardial_infarction).
Heart[Body Part, Organ, or Organ Component]: The hollow[Spatial
Concept]
muscular[Spatial Concept] organ[Body Part, Organ,
or Organ Component, Tissue] [Spatial Concept]
located behind[Spatial
Concept] [Body Part, Organ, or Organ Component]
the_sternum and
between the lungs[Body Part, Organ, or Organ Component].
Heart attack[Disease or Syndrome]: also called myocar-
dial_infarction[Disease or Syndrome]; damage[Functional Concept]
to the heart_muscle[Tissue] due to insucient blood
supply[Organ or Tissue Function] for an extended[Spatial Concept]
time_period[Temporal Concept].
We found that given a denitional term (DT) with a
semantic type SDT, terms that appear in the denition were
statistically correlated with related semantic types. For
example, when the semantic type of a denitional term
SDT is [Body Part, Organ, or Organ Component], the denition
tends to incorporate terms with the same semantic type
[Body Part, Organ, or Organ Component]
or related (e.g.,
[Spatial Concept] [Body Part, Organ, or Organ Component]
, and
Additionally, the lexico-syntactic patterns of denitional
sentences depend on the semantic types of denitional term
SDT. When SDT is [Body Part, Organ, or Organ Component], its
lexico-syntactic patterns includes . . .located. . .; and when
SDT is [Disease or Syndrome], the patterns include . . .due
247
to. . . and . . . caused by. . . We have empirically evalu-
ated the relations between semantics and lexico-syntactic
patterns [91]. Future work one may combine the semantics
with the lexico-syntactic patterns to eciently identify sen-
tences for answer extraction.
5. Conclusions
This study reports research development, implementa-
tion, and a cognitive evaluation of a biomedical question
answering system (MedQA). MedQA generates short-para-
graph-level texts to answer physicians and other biomedi-
cal researchers ad hoc questions. The contributions of this
work include:
(1) Automatic generation of lexico-syntactic patterns for
identifying denitions.
(2) The integration of document retrieval, answer extrac-
tion, and summarization into a working system that
generated a short paragraph-level answer to a deni-
tional question.
(3) A cognitive evaluation that compared MedQA with
three other state-of-the-art online information sys-
tems; namely, Google, OneLook, and PubMed.
Our results show that MedQA in general out-performed
OneLook and PubMed in the following four criteria: qual-
ity of answer, ease of use, time spent, and actions taken for
obtaining an answer. Although the evaluation results show
that Google was preferred system in quality of answer and
ease of use, the results showed that MedQA out-performed
Google in its time-spent and actions taken for obtaining an
be useful in clinical settings.
It is important to point out the limitations of this work.
This is a small scale involving four physicians who evalu-
ated 12 medical questions. These physicians may not be
representative of the broader population. In addition, the
small sample size precludes the ability to measure the statis-
tical dierences among search engines. Future work needs
to increase both the number of subjects and the number
of the questions to be evaluated. Although MedQA is a
work in progress, we can provisionally conclude that such
a system has the potential to facilitate the process of infor-
mation seeking in demanding clinical contexts.
Appendix A
Denitional questions that we found from over four
thousands of clinical questions collected by Ely, DAlessan-
, dro and colleagues [1,2527]. The 12 selected questions for
). evaluation are in bold.
1. What is cerebral palsy?
2. What is gembrozil?
3. What is D-dimer?
4. What is the marsh score?
248 H. Yu et al. / Journal of Biomedical Informatics 40 (2007) 236251

5. What is hemoglobin A0? 60. What is euthyroid sick syndrome?

6. What is TCA (tetracaine, cocaine, alcohol)? 61. What is Williams Syndrome?
7. What is Vagisil? 62. What is senile tremor?
8. What is Tamiu (oseltamivir)? 63. What is NARES (nonallergic rhinitis with eosino-
9. What is midodrine? philia syndrome)?
10. What is Ambien (zolpidem)? 64. What is WalkerWarburg Syndrome?
11. What is Terazol (terconazole)? 65. What is Carnetts sign?
12. What is Maltsupex? 66. What is the oxygen dissociation curve?
13. What is DDAVP (1-desamino-8-D-arginine vaso- 67. What is the pivot-shift test?
pressin)? 68. What is Oslers sign?
14. What is Resaid? 69. What is nephrocalcinosis?
15. What is droperidol (Inapsine)? 70. What is Vanderwoude syndrome?
16. What is an appendix epididymis? 71. What is dysbrinogenemia?
17. What is henox premer? 72. What is WalkerWarburg syndrome?
18. What is Lotrel? 73. What is the antibiotic dose?
19. What is Hytrin? 74. What is bronchiolitis?
20. What is Cozaar? 75. What is occipital neuralgia?
21. What is ceftazidime? 76. What is the cubital tunnel syndrome?
22. What is Zolmitriptan (Zomig)? 77. What is Klippel Feil Syndrome?
23. What is octreotide (sandostatin) and somatostatin? 78. What is dyskinesia?
24. What is Proshield? 79. What is Sandifer syndrome?
25. What is risperidone (Risperdal)? 80. What is heel pain syndrome?
26. What is Genora? 81. What is a blue dome cyst?
27. What is mepron (Atovaquone)? 82. What is central pontine myelinosis?
28. What is Zoloft (sertraline)? 83. What is Kenny syndrome?
29. What is uvoxamine (Luvox)? 84. What is dysdiadokokinesis?
30. What is Lunelle? 85. What is Wegeners granulomatosis?
31. What is amantadine dosing? 86. What is melanosis coli?
32. What is cyclandelate? 87. What is lipoprotein A?
33. What is clonazepam (Klonopin)? 88. What is FISH (uorescence in situ hybridization)?
34. What is westsoy formula? 89. What is schizoaective disorder?
35. What is Zyprexa? 90. What is Charcot Marie Tooth Disease?
36. What is Mexitil (mexiletine)? 91. What is the adenosine-thallium exercise tolerance test?
37. What is paregoric? 92. What is Ogilvies syndrome?
38. What is Legatrin? 93. What is prealbumin?
39. What is nimodipine (Nimotop)? 94. What is serum sickness?
40. What is glatiramer (Copaxone)? 95. What is Kussmaul breathing?
41. What is propafenone? 96. What is PraderWilli syndrome?
42. What is Ultravate cream (Halobetasol)? 97. What is an Adies pupil?
43. What is cilostazol (Pletal) (for intermittent 98. What is Noonan syndrome?
claudication)? 99. What is fetor hepaticus?
44. What is sotalol? 100. What is dissociative disorder?
45. What is Norvasc (amlodipine)? 101. What is the Jendrassik maneuver?
46. What is Uristat? 102. What is a tethered spinal cord?
47. What is nabumetone? 103. What is the postcholecystectomy syndrome?
48. What is Zofran (ondansetron)? 104. What is herpes gladiatorum?
49. What is terbinane (Lamisil)? 105. What is Fragile X syndrome?
50. What is Cetirizine (Reactin)? 106. What is a high ankle sprain?
51. What is Serzone (nefazodone)? 107. What is PeutzJeghers Syndrome?
52. What is Sansert? 108. What is an eccrine spiradenoma?
53. What is Urised? 109. What is Fanconis Syndrome?
54. What is nedocromil sodium (Tilade)? 110. What is Stills disease?
55. What is Ultram (tramadol)? 111. What is the dawn phenomenon?
56. What is the Poland anomaly? 112. What is BOOP (bronchiolitis obliterans and organiz-
57. What is vestibulitis? ing pneumonia)?
58. What is hemolytic uremic syndrome? 113. What is Bests disease?
59. What is a Lisfranc fracture? 114. What is Rovsings sign?
 H. Yu et al. / Journal of Biomedical Informatics 40 (2007) 236251
115. What is a sliding hiatus hernia?
116. What is biliary gastritis (bile gastritis)?
117. What is a Hepatolite scan (questionably the same as a
PAPIDA scan)?
118. What is seronegative spondyloarthropathy?
119. What is Stickler Syndrome?
120. What is the Delphi technique?
121. What is Xalatan eye drops?
122. What is Betamol eye drops?
123. What is the binomial distribution?
124. What is a Roux-en-Y hepatoenterostomy?
125. What is the bivariate normal distribution?
126. What is Sperlings sign (Sperlings maneuver)?
127. What is an incidence rate ratio?
128. What are Alomide ophthalmic drops?
129. What is the urological surgery technique?
130. What are Acular ophthalmic drops? (good for allergic
conjunctivitis?)
131. What are poppers?
132. What are Hawkins and Neer impingement signs (for
shoulder pain, rotator cu injury)?
133. What are the Ottawa knee rules?
134. What are endomysial antibodies?
135. What are preeclampsia labs (laboratory studies)?
136. What are Lewy bodies?
137. What are the heat injury syndromes?
138. What are pineal brain tumors?
References
[1] Ely JW, Oshero JA, Ebell MH, Bergus GR, Levy BT, Chambliss
ML, et al. Analysis of questions asked by family doctors regarding
patient care. Br Med J 1999;319(7206):35861.
[2] Straus S, Sackett D. Applying evidence to the individual patient. Ann
Oncol 1999;10(1):2932.
[3] Cimino JJ, Li J, Graham M, Currie LM, Allen M, Bakken S, et al.
Use of online resources while using a clinical information system.
AMIA Annu Symp Proc 2003:1759.
[4] Alper BS, White DS, Ge B. Physicians answer more clinical
questions and change clinical decisions more often with synthesized
evidence: a randomized trial in primary care. Ann Fam Med
2005;3(6):50713.
[5] Westbrook JI, Gosling AS, Coiera E. Do clinicians use online
evidence to support patient care? A study of 55,000 clinicians. J Am
Med Inform Assoc 2004;11(2):11320.
[6] Westbrook JI, Coiera EW, Gosling AS. Do online information
retrieval systems help experienced clinicians answer clinical ques-
tions? J Am Med Inform Assoc 2005;12(3):31521.
[7] Gosling AS, Westbrook JI. Allied health professionals use of online
evidence: a survey of 790 sta working in the Australian public
hospital system. Int J Med Inform 2004;73(4):391401.
[8] Sackett D, Straus S, Richardson W, Rosenberg W, Haynes R.
Evidence-Based Medicine: How to practice and teach EBM. Edin-
burgh: Harcourt Publishers Limited; 2000.
[9] Schilling LM, Steiner JF, Lundahl K, Anderson RJ. Residents
patient-specic clinical questions: opportunities for evidence-based
learning. Acad Med 2005;80(1):516.
[10] Alper B, Stevermer J, White D, Ewigman B. Answering family
physicians clinical questions using electronic medical databases. J
Fam Pract 2001;50(11):9605.
249
[11] Jacquemart P, Zweigenbaum P. Towards a medical question-
answering system: a feasibility study. Stud Health Technol Inform
2003;95:4638.
[12] Takeshita H, Davis D, Straus S. Clinical evidence at the point of care
in acute medicine: a handheld usability case study. Proceedings of the
human factors and ergonomics society 46th annual meeting, 2002, p.
140913.
[13] Hersh WR, Crabtree MK, Hickam DH, Sacherek L, Friedman CP,
Tidmarsh P, et al. Factors associated with success in searching
MEDLINE and applying evidence to answer clinical questions. J Am
Med Inform Assoc 2002;9(3):28393.
[14] Waltz D. An english language question answering system for a large
relational database. Commun ACM 1978;21(7):52639.
[15] Voorhees E. Overview of the TREC 2004 question answering track.
NIST Special Publication: SP 500-261, 2004.
[16] Yu H, Hatzivassiloglou V. Towards answering opinion questions:
separating facts from opinions and identifying the polarity of opinion
sentences. Proceedings of empirical methods in natural language
processing (EMNLP), 2003.
[17] Bethard S, Yu H, Thornton A, Hatzivassiloglou V, Jurafsky D.
Semantic analysis of propositional opinions. AAAI 2004 spring
symposium on exploring attitude and aect in text: theories and
applications, 2004.
[18] Zweigenbaum P. Question answering in biomedicine. EACL work-
shop on natural language processing for question answering, Buda-
pest, 2003, p. 14.
[19] Zweigenbaum P. Question-answering for biomedicine: methods and
state of the art. MIE 2005 Workshop, 2005.
[20] Rinaldi F, Dowdall J, Schneider G, Persidis A. Answering questions
in the genomics domain. ACL 2004 workshop on question answering
in restricted domain, 2004.
[21] Niu Y, Hirst G. Analysis of semantic classes in medical text for
question answering. ACL 2004 workshop on question answering in
restricted domains, 2004.
[22] Delbecque T, Jacquemart P, Zweigenbaum P. Indexing UMLS
semantic types for medical question-answering. In: Engelbrecht R
et al, editors. Connecting medical informatics and bio-informatics
ENMI 2005, 2005.
[23] Xuang X, Lin J, Demner-Fushman D. Evaluation of PICO as a
knowledge representation for clinical questions. Annual symposium
of the American medical informatics association, 2006.
[24] Yu H. Towards answering biological questions with experimental
evidence: automatically identifying text that summarize image con-
tent in full-text articles. American medical informatics association,
2006.
[25] Ely JW, Oshero JA, Ferguson KJ, Chambliss ML, Vinson DC,
Moore JL. Lifelong self-directed learning using a computer database
of clinical questions. J Fam Pract 1997;45(5):3828.
[26] Ely JW, Oshero JA, Chambliss ML, Ebell MH, Rosenbaum ME.
Answering physicians clinical questions: obstacles and potential
solutions. J Am Med Inform Assoc 2005;12(2):21724.
[27] DAlessandro DM, Kreiter CD, Peterson MW. An evaluation of
information-seeking behaviors of general pediatricians. Pediatrics
2004;113(1 Pt 1):649.
[28] Ely JW, Oshero JA, Gorman PN, Ebell MH, Chambliss ML, Pifer
EA, et al. A taxonomy of generic clinical questions: classication
study. Br Med J 2000;321(7258):42932.
[29] Yu H, Sable C. Being Erlang Shen: identifying Answerable questions.
Proceedings of the nineteenth international joint conference on
articial intelligence on knowledge and reasoning for answering
questions, 2005.
[30] Yu H, Sable C, Zhu H. Classifying medical questions based on an
evidence taxonomy. Proceedings of the AAAI 2005 workshop on
question answering in restricted domains, 2005.
[31] Kushniruk A, Patel V. Cognitive and usability engineering methods
for the evaluation of clinical information systems. J Biomed Inform
2004;37(1):5676.
250 H. Yu et al. / Journal of Biomedical Informatics 40 (2007) 236251
[32] Kaufman D, Patel R, Hilliman C, Morrin P, Pevzner J, Weinstock R,
et al. Usability in the real world: assessing medical information
technologies in patients homes. J Biomed Inform 2003;36:4560.
[33] Norman D. Cognitive engineering. User Centered System
Design: Lawrence Erlbaum Associates; 1986. p. 3161.
[34] Marchionini G. Information seeking in electronic environ-
ments. New York: Cambridge University Press; 1995.
[35] Bhavnani S. Why is it dicult to nd comprehensive information?
Implications of information scatter for search and design: research
articles. J Am Soc Inf Sci Technol 2005;56(9):9891003.
[36] Sutclie A, Ennis M. Towards a cognitive theory of information
retrieval. Interacting with Computers 1998;10(3):32151.
[37] Seol Y, Kaufman D, Mendonca E, Cimino J, Johnson S. Scenario-
based assessment of physicians information needs. MedInfo, 2004.
[38] Elhadad N, McKeown K, Kaufman D, Jordan D. Facilitating
physicians access to information via tailored text summarization.
American medical informatics annual symposium, Washington, DC,
2005.
[39] Allen E, Burke J, Welch M. How reliable is science information on
the web? Nature 1999;402:722.
[40] Culver J, Gerr F, Fumkin H. Medical information in the internet: a
study of an electronic bulletin board. J Gen Intern Med
1997;12(8):46671.
[41] Davison K. The quality of dietary information on the world wide
web. Clin Perform Qual Health Care 1997;5:646.
[42] Eysenbach G, Kohler C. How do consumers search for and appraise
health information on the world wide web? Qualitative study using
focus groups, usability tests, and in-depth interviews. Br Med J
2002;324(7337):5737.
[43] Sysenbach G, Powell J, Kuss O, Sa E. Empirical studies assessing the
quality of health information for consumers on the world wide web:
A systematic review. J Am Med Assoc 2002;287(20):2691700.
[44] Griths K, Christensen H. Quality of web based information on
treatment of depression: cross sectional survey. Br Med J
2000;321:15115.
[45] Impicciatore P, Pandolni C, Casella N, Bonati M. Reliability of
health information for the public on the world wide web. Br Med J
1997;314:18759.
[46] Thelwall M. Extracting macroscopic information from web links. J
Am Soc Inf Sci Technol 2001;52(13):115768.
[47] Lin J. User simulations for evaluating answers to question series.
Inform Process Manag 2007.
[48] Metzler D, Croft W. Combining the language model and inference
network approaches to retrieval. Inform Process Manag
2004;40(5):73550.
[49] Mikheev A. 1996. Available from http://wwwltgedacuk/software/
chunk/.
[50] Day R. How to write and publish a scientic paper. Cam-
bridge: Cambridge University Press; 1998.
[51] Gabbay I, Sctcliee R. A qualitative comparison of scientic and
journalistic texts from the perspective of extracting denitions.
ACL 2004 workshop question answering in retricted domains,
2004.
[52] Mizuta Y, Collier N. Zone identication in biology articles as a basis
for information extraction. Natural language processing in biomed-
icine and its applications, Post-COLING Workshop, 2004.
[53] Mizuta Y, Collier N. An annotation scheme for a rhetorical analysis
of biology articles. Proceedings of the fourth international conference
on language resources and evaluation (LREC2004), 2004.
[54] Elhadad N, Kan M, Klavans J, McKeown K. Customization in a
unied framework for summarizing medical literature. J Artif Intell
Med 2004.
[55] Lin J, Karakos D, Demner-Fushman D, Khudanpur S. Generative
content models for structural analysis of medical abstracts. HLT-
NAACL BioNLP, New York City, 2006.
[56] Mcknight L, Srinivasan P. Categorization of sentence types in
medical abstracts. American medical informatics association 2003,
Washington, DC, 2003.
[57] Zhu H, Yu H. A comparative study for automatic zone identication
in MEDLINE (A poster). BioLINK SIG: linking literature, Infor-
mation and knowledge for biology, 2005.
[58] Light M, Qiu X, Srinivasan P. The language of bioscience: facts,
speculations, and statements in between. HLT-NAACL 2004 work-
shop: BioLink 2004, Linking biological literature, Ontologies and
databases, 2004.
[59] Klavans J, Muresan S. Evaluation of the DEFINDER system for
fully automatic glossary construction. Proceedings of American
medical informatics association symposium, 2001.
[60] Xu A, Weischedel R. Trec 2003 QA at BBN: answering denitional
questions. TREC, 2003.
[61] Blair-Goldensohn S, McKeown K, Schlaikjer A. Answering deni-
tional questions: a hybrid approach. New directions in question
answering, 2004.
[62] Cui H, Kan M, Cua T. Generic soft pattern models for denitional
question answering. The 28th annual international ACM SIGIR
2005, Salvado, Brazil, 2005.
[63] Rilo E, Philips W. An introduction to the Sundance and AutoSlog
systems, University of Utah School of Computing, 2004.
[64] Rilo E. Automatically generating extraction patterns from untagged
text. AAAI, 1996.
[65] Yu H, Agichtein E. Extracting synonymous gene and protein terms
from biological literature. Bioinformatics 2003;19(Suppl. 1):i3409.
[66] Barzilay R, Elhadad M. Using lexical chains for text summarization.
ACL intelligent scalable text summarization workshop 1997, Madrid,
1997.
[67] Luhn H. The automatic creation of literature abstracts. IBM J Res
Dev 1958;2(2).
[68] Edmunson H. New methods in automatic extracting. J ACM
1969;16(2):26485.
[69] Hovy E, Lin C. Automated text summarization in summarist. ACL/
EACL workshop on intelligent scalable text summarization 1997,
Madrid, 1997.
[70] Marcu D. The rhetorical parsing, summarization, and generation of
natural language texts. University of Toronto, 1997.
[71] Halliday M, Hasan R. Cohesion in English. Longman; 1976.
[72] Barzilay R, Elhadad M, McKeown K. Information fusion in the
context of multi-document summarization. The 37th association for
computational linguistics 1999, Maryland, 1999.
[73] Barzilay R, Elhadad M, McKeown K. Inferring strategies for
sentence ordering in multidocument summerization. JAIR, 2002.
[74] McKeown K, Chang S, Cimino J, Feiner S, Gravano L, et al.
PERSIVAL, a system for personalized search and summarization
over multimedia healthcare information. Proceedings of the 1st
ACM/IEEE-CS joint conference on digital libraries 2001, 2001.
[75] Fiszman M, Rindesch T, Kilicoglu H. Abstraction summarization
for managing the biomedical research literature. HLT-NAACL 2004:
computational lexical semantic workshop, 2004.
[76] Salton G. A vector space model for information retrieval. CACM
1975;18(11):61320.
[77] Lee M, Wang W, Yu H. Exploring supervised and unsupervised
approaches to detect topics in biomedical text. BMC Bioinformatics
2006;7:140.
[78] Hatzivassiloglou V, Gravano L, Maganti A. An investigation of
linguistic features and clustering algorithms for topical document
clustering. SIGIR, 2000.
[79] Radev D, Jing H, Budzikowska M. Centroid-based summarization of
multiple documents: sentence extraction, utility-based evaluation,
and user studies. ANLP/NAACL workshop on summarization, 2000.
[80] Purcell G. The quality of health information on the internet. Br Med
J 2002;324(7337):5578.
[81] Jadad AR, Gagliardi A. Rating health information on the Internet:
navigating to knowledge or to Babel? JAMA 1998;279(8):6114.
[82] Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling,
and assuring the quality of medical information on the Internet:
Caveant lector et vieworLet the reader and viewer beware. JAMA
1997;277(15):12445.
 H. Yu et al. / Journal of Biomedical Informatics 40 (2007) 236251
[83] Glennie E, Kirby A. The career of radiography: information on the
web. J Diagn Radiogr Imaging 2006;6:2533.
[84] Childs S. Judging the quality of internet-based health information.
Perform Meas Metrics 2005;6(2):8096.
[85] Cline RJ, Haynes KM. Consumer health information seeking on the
Internet: the state of the art. Health Educ Res 2001;16(6):67192.
[86] Benigeri M, Pluye P. Shortcomings of health information on the
Internet. Health Promot Int 2003;18(4):3816.
[87] Wyatt J. Commentary: measuring quality and impact of the world
wide web. Br Med J 1997;314:1879.
251
[88] McClung HJ, Murray RD, Heitlinger LA. The Internet as a source
for current patient information. Pediatrics 1998;101(6):E2.
[89] Berkowitz L. Review and evaluation of internet-based clinical
reference tools for physicians: uptodate, 2002.
[90] Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder
MS, et al. Health information on the Internet: accessibility, quality,
and readability in English and Spanish. JAMA 2001;285(20):261221.
[91] Yu H, Wei Y. The Semantics of a deniendum constrains both the
lexical semantics and the lexicosyntactic patterns in the deniens.
HLT-NAACL BioNLP 2006, New York, 2006.