Trust in Health Websites: A Review of an Emerging Field
Laurian Vega Enid Montague, Ph.D.
Computer Science Industrial and Systems Engineering
2202 Kraft Drive 3107 Mechanical Engineering Building
Virginia Tech, Blacksburg, VA University of Wisconsin-Madison, WI
Laurian@vt.edu EMontague@wisc.edu
ABSTRACT
As people increasingly turn to health websites for the purposes of
self-diagnosis and healthier living, we have an obligation to
evaluate the factors that might affect a given user's assessment and
their willingness to use such sites. Constructs such as quality,
trust, and credibility need to be defined within this space in order
for us to truly understand how and why people use health
websites. In an effort to better understand these constructs we
conducted a comprehensive analysis of all peer-reviewed
empirical studies on trust in health websites -- this paper is the
result. Work on this topic was provided from eleven fields
including HCI, Informatics, Medicine, and Decision Making. Our
findings show that authors often value different facets of trust,
report different outcomes, and rarely cite each other. Without a
coherence of terms and values, the task of presenting and
understanding how users trust health information on the web will
be intractable.
Categories and Subject Descriptors
K.4.3 [Computers and Society]: Organizational Impacts –
Computer supported collaborative work; H.1.2 [Information
Systems]: Human Factors.
General Terms
Reliability, Human Factors, and Theory.
Keywords
Health, Trust, Credibility, Review, Empirical Evaluation, Human
Factors, Human-Computer Interaction, Websites
1. Introduction
Trust, or the lack there of, is a foundational part of any
relationship. In particular it is critical to relationships in the realm
of healthcare. For instance, the patient trusts that their team of
healthcare professionals is going to manage their wellness
appropriately. Health care professionals additionally trust each
other to work cooperatively and to effectively communicate
pertinent patient information. The increase in use of the internet
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
IHI’10, November 11–12, 2010, Arlington, Virginia, USA.
Copyright 2010 ACM 978-1-4503-0030-8/10/11...$10.00.
Tom DeHart
Computer Science
322 BSB, 227 Penn Street
Rutgers University, Camden, NJ
tdehart@vt.edu
has, however, impacted the traditional health care relationships.
The Pew Internet Foundation recently reported that 61% of
Americans are currently using the internet for health information,
and 20% use the internet to look up information about medicine
and experimental treatments [15]. This rise in use of health
websites indicates a need to understand how users evaluate the
credibility of online information in order to better understand the
impact it will have on the patient’s trust in different parts of the
health system.
While there has been prior work modeling user trust of various
technologies [18, 30], research has demonstrated that trust in one
system or artifact does not necessarily translate to another system
or artifact [43]. For example, while there may be valid models of
trust that workers place in factory machinery, those same models
of user trust may not apply to eCommerce websites. User trust
models should then be created and evaluated specific to the
domain of application in order to understand the nuances of each
particular system. Likewise, this practice should be used even
when attempting to understand different models of trust in online
activities (i.e. looking up medical information v. purchasing a
book). For this reason, current models of how a patient trusts their
health care provider cannot be merged with current models of how
users trust eCommerce websites. Instead, there exists a need to
understand what antecedents induce or affect trust, while also
understanding the facets and components that make up the
construct of a user’s trust in health websites.
In this paper we evaluate the body of empirical research on user
trust in e-health websites. This research is comprised of 49 papers
collected in Summer 2009 stemming from a multitude of different
backgrounds such as HCI, Medicine, and Informatics, and
Decision Making. The data and analysis used in this study was
created in conjunction with a corresponding study that examined
differences in trust measurements and varying construct
definitions as a function of time. It has been submitted as a
separate journal publication. For this paper we reviewed 49
publications in the realm of trust with one particular question in
mind -- one that is of particular important to the IHI audience:
how does research from different disciplines impact the
comprehensive understanding of trust in health websites? Our
goal is to provide an example evaluation method for
understanding emerging multidisciplinary constructs, like trust
and credibility, for web communities. This framework will
provide a method to analyze the contributions and parlance of
research on social constructs. To meet this goal we 1) used
content analysis to organize constructs presented in the
manuscripts 2) conducted a social network analysis, and 3) linked
facets to empirical findings from each discipline. We address
three key concerns: that the construct of trust is overly fragmented
by domain; that researchers on trust in health websites lack a clear
definition and what definitions they have are conflated; and, that
there is distinct lack of collaboration between authors. We
conclude with a discussion about the future of research on trust
within the larger socio-technical system and an analysis of the
user’s assessment of a website’s credibility.
2. Background & Related Work
Prior work has demonstrated an increase in the use of health
technology [37, 61, 71]. A major factor in this increase is the push
to move to a paperless medical work environment. And, in doing
so, there is an assumption that there will be an increase in the
availability of credible health information online [29]. However,
the user’s employment of health websites can vary from user-
generated content, such as Wikipedia, to more authoritative
websites, such as WebMD or IHI.gov. Understanding the impact
that these web technologies have on the patient’s perceptions of
quality of care and on the patient-provider relationship can impact
the design, implementation, and integration of health websites into
the larger health care system and the adoption of electronic
medical records.
A health website, also known as a health infomediary, is a
company that supplies health information: “an online provider that
offers information, advice, guidance, and assessment for wellness
and health-related issues, including referrals and wellness-related
evaluations” [74]. Sillence et al. have also identified different
types of web technology that can be labeled as health websites
[60]. These include web portal sites, support groups, charity sites,
government sites, pharmaceutical sites, sales sites, personal sites,
medical databases, media sites, and clinician sites. We have
adopted Sillence et al.’s definition of an e-health website as the
focus for our study.
Research on user mental models of trust in technology has been
reviewed from multiple relevant areas [18, 44, 46]. For example,
the work on user trust in automation examined how the design of
technology impacts trust in the automation technology [46]. While
this work demonstrates how users evaluate automated
technologies, it does not take into account the particularities of
trust in a website and the impact that trust has on related
relationships. In particular, factors such as ‘consistency’ that are
paramount to user trust in automation do not necessarily translate
into the realm of trust in health websites where there may be
dynamic content. Additionally, in the related work of trust in
website, few models account for interpersonal relationships that
are impacted by constructs like trust or similar constructs like
credibility, authority, and expertise within relationships [52] such
as the doctor-patient relationship.
A second group of related research explores user trust in web-
based technologies, such as eCommerce websites [32, 38, 41].
Research in this area examines the relationship between user trust
and the resulting behavior (e.g., purchasing, supplying personal
information) [18]. While this research is similar, it does not
account for the impact that the resulting behavior then has on
external relationships – such as the patient-provider relationship.
We focus on this relationship in particular given that of people
who access health information online, 50% of them report that
they then talk to their doctor about the found information [72].
Additionally, the trust in the health provider has been linked to
important quality outcomes such as adherence, satisfaction, and
improved health outcomes [1]. Models of trust provided by related
disciplines, such as eCommerce, are valuable for highlighting
domains that might be correlated, but cannot account for
intrinsically different relationships and factors that will impact
trust in health websites. It is for this reason that research trust in
health websites is critical in a field that has an increasing impact
on daily health related interactions.
2.1 Trust and Credibility
Conceptualization of the construct of trust customarily will focus
on defining trust’s facets and antecedents. Facets of trust are the
constructs that it may encompass or make up user trust. Examples
taken from the body of reviewed literature include 'well-
intentioned' [67], 'truthfulness' [52], and 'integrity' [74].
Antecedents of trust, on the other hand, are the external factors
that impact or cause variation in a user’s trust. Examples include
'relevance', 'authority', or 'disposition' [30].
In the relevant literature, the constructs of credibility and trust are
linked through their facets and antecedents. In forty-nine papers,
only thirteen included definitions of trust. Of these thirteen
papers, five papers defined trust as a facet of the construct of
credibility [14, 23, 52, 67, 72]. Heavily citing the work of
Hovland et al. [24], trust was defined as the well-intentioned,
unbiased, and truthful nature of the source of the information.
Along with expertise, which is defined as the quality of the source
of information, trust is used to understand the credibility of a
source of information.
This conceptualization of the relationship of trust and credibility,
however, is only one variation. In our conception of trust in health
websites we argue that credibility is a part of trust. In a socio-
technical system multiple humans work with technologies and are
influenced by aspects external to the system as well [49].
Interactions with and surrounding health websites are more than
individual patients working with an interface: they are portions of
a complex health care work system. Patients have primary
relationships with their care providers and those that provide care
for them. Trust is therefore constructed at different levels of that
system; patients can have trust relationships with care providers
[1, 68], health institutions [21], and health technologies [44]. A
person’s trust in a health website is formed on the system
surrounding the website and the health system that the patient is a
part of. For example if a patient finds incorrect information on an
electronic medical record, is their trust diminished in the website
that hosts the record, the company that provides the record
transferring, or the health institution or care provider? Credibility
would be placed in the company that designs the health record
system, but it is trust that will influence the patient to either use
the system or like systems again and perhaps other aspects of the
system. Given this understanding, we present our work that
reviews user evaluations of health websites.
3. Method
Three methods of analysis were used in order to understand how
trust might impact the design and implementation of health
websites and, more importantly, how it may affect the use of such
sites. After narrowing down the body of work to 49 papers, we
conducted a word frequency analysis, a social network analysis,
and, finally, a unique analysis of empirical outcomes by
discipline; all of which are explained below. While methods like
 Figure 1. The pie graphs depict the use of the factors 'quality' and 'behavior in reference to the research areas that use them. For
example, 30% of the uses of the word 'quality' came from the area of Medicine.
these may lack statistical significance in such small data sets, the
results can still prove insightful as is shown below.
3.1 Data Sources
A comprehensive search of thirteen databases (e.g.,
ScienceDirect) was completed in the Summer 2009 using search
terms such as *trust*, website*, webpage*, online*, internet*,
web site*, web page*, health*, and medic*. The initial search
produced over 2,000 articles, which were then evaluated
individually by two reviewers based on pre-established criteria of
being peer-reviewed, empirical, and including trust as a variable.
Disagreements were discussed and then agreed upon for the final
set. Supplementary computer searches were conducted from
articles that appeared in the reference sections of the set of articles
and identified as related; 49 relevant articles were identified and
examined by the authors for the purposes of this paper. Included
papers represented both experimental and theoretical peer-
reviewed conference proceedings and journal papers. Excluded
from the set were publications unrelated to the topic, those not
peer-reviewed, and those yet to be published.
3.2 Analysis Method
We employed a meta-analysis framework to examine the impact
of a research discipline on the concept of trust. This method is
appropriate given that a review of trust in health is novel and this
method has been used for similar work (see [12] for example).
Three strategies were used to analyze the papers for the impact of
research on trust by discipline: (1) content analysis to examine the
frequency of words used to discuss trust research, (2) social
network analysis to examine how researchers from different areas
cited each other, and (3) an analysis of empirically found research
outcomes by research area.
3.2.1 Content Analysis of Word Frequency
To understand how researchers from different backgrounds would
discuss and evaluate the construct of trust we analyzed the content
of each paper for the frequency of terms (see [10] for an
example). A list of three hundred and ninety-six terms was
compiled from a thesaurus and from articles [28, 45] that
employed empirical methods to derive terms related to user trust
and technology. These terms were then combined to form 108
meta-constructs (e.g., ‘difficult’ and ‘difficulty’ were combined).
Stop words such as 'available', 'care', 'independent', 'positive',
'positively', 'control', and 'certain’ were omitted as these terms
were often used outside of the context of explaining trust (e.g.,
“Independent sample t-tests were employed…”). Scripts were
then used to count the terms across all 49 papers, with each term
being counted a maximum of one time per paper. Our goal with
this analysis was to gain a richer understanding of what the
research community believes to be related to trust in health
websites. To do this we had to go beyond provided definitions,
facets, and antecedence, and examine the words that people used
through discussing the construct of trust. It was our hypothesis
that research from different domains would utilize different
constructs related to trust and with different frequencies.
3.2.2 Social Network Analysis
Social network analysis maps and measures relationships between
entities; in this study, social network analysis was used to analyze
ideas about trust in health websites. Our social network analysis
explored articles and looked for citation patterns between authors
to explore which entities were citing other. The outcome was used
to generate hypotheses about the flow of ideas and shared
conceptualizations of the definition of trust. The nodes in our
network are the authors of the papers while the links show
relationships or flows between the nodes. Social Network analysis
provides both a visual and a mathematical analysis of
relationships. More importantly social network analysis can help
visualize shared and disparate ideas in a discipline.
Several analyses were conducted using NodeXL social network
analysis software:
1. Degree: The number of vertices that the node is connected to.
In this study, Degree is the number of author articles that the
paper is connected to.
2. Betweenness centrality: The extent that a node acts as a
connection to other nodes. In our review this is operationalized to
mean that a paper may function as an interconnection between
different areas of trust research.
3. Closeness centrality: The measure of the average shortest
distance from each node. A lower centrality score indicates a
more important or central position in the network.
4. Eigenvector centrality: an adjusted measure that reports a
combination of how many connections a vertex has with the
importance (degree) of the connections.
The areas of research were color coded by clusters in Figure 2.
Clusters are groups of papers that are interconnected. They are
determined with a cluster coefficient, which measures how
connected a vertex’s neighbors are to one another (i.e. creating
"cliques" of how connected authors and articles are to one another
based on citation patterns) [64]. Papers that did not cite any other
papers in the body of reviewed work were not included in this
analysis.
Table 1. This table depicts the interconnected papers in a
social network analysis with the associated degree,
betweenness, closeness, and eigenvector centralities.
Eigenvector
Betweennes
3.2.3 Analysis of Outcomes by Discipline
To evaluate the outcomes from empirical research on trust in
health websites a comprehensive list of all outcomes was created.
These outcomes were then organized by what trust can and cannot
affect, and factors that affect user trust in health websites. These
outcomes were then coded according to which realm of research
the paper stemmed from (e.g., the Medical field; a list is available
in Figure 1). Papers could contain multiple findings that would
span multiple categories. Two members of the research team
negotiated and reached agreement on the outcome categorizations.
Our goal with this analysis was to determine the full spectrum of
empirical research outcomes that were found to affect trust in
health websites. The hypothesis was that the overall research
spectrum was fragmented by the area that the paper stemmed
from.
Specifically, outcomes were combined to create more than 60
factors that were then combined to create five categories. The five
categories were: Usability, Content of the Webpage (e.g., the
presence of advertisements and the presence of real life

examples), Information Factors (e.g., quality, adequacy, and

Centrality

Centrality

Centrality
Closeness

personalization of the information), Contextual Factors (e.g.,

Degree
Vertex

access location and consistency with current beliefs), and User

Demographics (e.g., gender, health, and age).
These categories were then further decomposed into sets of
Sillence et al. (2004) 6 0.962 3.063 0.054
empirically determined factors. For example, Gender was an
Dutta-Bergman (2003) 4 0.494 2.875 0.45 empirically derived factor that was within the category of
Rains (2007) 4 0.323 2.938 0.445 Demographics. To assess whether categories and factors were
correlated with particular areas, the outcomes were then coded by
Lemire et al. (2008) 4 1 2.5 0.416 discipline.
Rains &Karmikel (2009) 4 0.105 3.438 0.395
3.3 Data
Sillence, et al. (2005) 2 0 3.938 0.026
All papers included in our review were published between 2001
Bates et al. (2006) 2 0.226 3.688 0.159 and 2009. 2001 is the earliest date that papers on this topic were
Sillence et al. (2007a) 2 0 3.938 0.026 published. The papers were organized into background categories.
These categories were based on the journal or conference in which
Walther et al. (2004) 2 0.005 3.688 0.267 each paper was published (e.g., a paper published at an
Hong (2006) 2 0.005 3.688 0.267 eCommerce conference would be put into the eCommerce area).
The areas are as follows; communication (n=3) [25, 35, 51];
Huntington et al. (2004) 2 0.947 2.563 0.153
decision making (n=2) [53, 65]; human-computer interaction,
Sillence et al. (2007c) 2 0.068 3.188 0.258 computer science, and the internet –called Computing (n=9) [11,
Fisher et al. (2008) 2 0.902 2.75 0.066 13, 52, 58, 59, 60, 61, 62, 67]; health and medical informatics and
health – called Health Informatics (n=11) [3, 9, 16, 20, 27, 33, 37,
Khoo et al. (2008) 2 0.015 1 0 54, 72]; health services and healthcare (n=3) [6, 8, 17];
Menon et al. (2002) 1 0 4.625 0.051 information and library science (n=3) [23, 26, 39]; media and
society (n=1) [14]; medicine (n=14) [2, 3, 5, 7, 19, 22, 31, 34, 42,
Glenton et al. (2006) 1 0 4 0.017 47, 48, 56, 66, 71]; pharmacology (n=1) [69]; and, social sciences,
Newnham et al. (2006) 1 0 1 0 psychology, and sociology – called Social Sciences (n=3) [4, 50,
Newnham et al. (2005) 1 0 1 0 63]. Rosenbaum et al. paper is counted twice because it is
published in BMC Medical Informatics and Decision Making,
Sillenceet al. (2006) 1 0 4 0.017 which spans two areas.
Sillence et al. (2007b) 1 0 4 0.017
3.4 Content Analysis of Word Frequency
Zahedi& Song (2008) 1 0 1 0 The top ten words used in discussing the research on trust in
Song &Zahedi (2007) 1 0 1 0 health websites across all reviewed papers were: quality (47),
understanding (44), reliability (43), communication (41),
Halkias et al. (2008) 1 0 1 0
experience (40), knowledge (39), accuracy (39), credibility (38),
Hesse et al. (2005) 1 0 1 0 concern (38), and behavior (38). The top fifty most frequently
Wainstein et al. (2006) 1 0 1.5 0 used words were found in 37% of all the papers.
Semere et al. (2003) 1 0 1.5 0
To analyze the parlance of trust by field of study, we charted the continues.
frequency of terms that were present in each discipline. With three
areas encompassing 69% of the total group of papers, these 3.6 Analysis of Outcomes by Area
groups had a large impact on the frequency of use. For example, To analyze the outcomes of trust research in health websites, the
the construct of ‘quality’ was used by every paper in every group outcomes were sorted into sets. The first set of empirical
except for the groups of Computing and Health Services & outcomes evaluated how trust in one source would then affect
Healthcare. ‘Quality’ is used 70% of the time in the papers with trust in another. Within these outcomes, the type of health website
the largest base of work. (Figure 1 depicts the factors of quality being assessed varied by the area of research. For example,
and behavior by area and percentage of use.) In contrast to research from the area of communication focused on how trust in
‘quality’, the construct of ‘behavior’ is still used 56% of the time different sources, like the internet, physicians, and entertainment
by the three areas with the largest base of papers. This is a 14% would affect how users then trusted internet information [35, 51].
decrease in use by the three largest groups. Alternatively, research from the area of decision making focused
on how trust in the internet as a whole would enable or disable
A pattern in use and weight can be detected. The areas of people to look up more information about their medical condition
Medicine, Health Informatics, and Computing for most constructs [53, 65]. Another example is from the area of Medicine. Medical
make up the largest portion of their use. This is to be expected research focused on how trust in medical related online
given that they have the largest base of papers. However, as the information, such as prescription drugs information, online health
amount of use decreases among the three largest groups, the records, or general medical websites, all affected trust in other
remaining areas use of those words remains relatively high. This socio-technical systems such as all online prescription
can be seen in use the construct of ‘behavior’ with 31% of the information, the national healthcare service, and the intention to
papers making up 44% of the usage. Other examples include the use health websites, respectively [2, 19, 42].
term ‘negatively,’ which is used in all Communication and
Decision Making papers; whereas the term ‘negatively’ is only Twelve of the reviewed papers mentioned that the internet has an
used 44% of the time in the computing research and 29% of the effect on doctor-patient relationships. Twelve more cited this
time in the research on Medicine. relationship as an important factor for why trust in health websites
should be studied. Twenty-five papers did not cite this
3.5 Social Network Analysis
Twenty-six papers cited at least one other paper
within the body of reviewed literature. Twenty-
three, or 47% of, papers did not include citations
to other work in the body of reviewed literature
instead opting to cite only within their own fields
of research. Social network analysis provided
insight in to the connections between articles and
authors. See Table 1 for values and Figure 2 for
a visual depiction.
1. Degree: The paper published by Sillence et al.
in 2004 [58] had six vertices from the body of
reviewed papers, which was the highest number.
Followed by Dutta-Bergman [9], Lemire et al.
[37], Rains [51], and Rains and Karmikel [52].
All had four degrees.
2. Betweenness centrality: Several papers did not
act as a connection to other nodes, while the
paper by Lemire et al. [37] reported the highest
betweenness centrality score.
3. Closeness centrality: This measurement
conveyed the average shortest distance from
each node; lower centrality scores indicate more
important positions in the network. The lower
centrality measures appear towards the bottom of
Table 1, indicating that there exists a disconnect
between the papers that have more important
positions and the papers that are highly cited, Figure 2. This network is a representation of the social network analysis of all
which are the top of the table. empirical papers on trust in health websites. Boxes represent papers and edges
4. Eigenvecter centrality: This measurement is an represent citations. Colors and location indicate clusters and connectedness. Note
adjusted measure that reports a combination of that colors used in this figure are not related to those used in Figure 1. Colors are
how many connections a vertex has with the used in this figure to clarify different cliques (e.g., Bates [3], Menon [42], and
importance (degree) of the connections. After Dutta-Bergman [9] form a clique as these three papers cite each other).
adjustments with the eigenvector, the same trend
relationship in their analyses.
The second set of empirical outcomes was organized into factors
and categories of outcomes that affected trust in health websites
(Table 2). Given that papers could have outcomes spanning
multiple categories, the breakdown by the number of papers found
in each category is as follows: Usability=12, Content of the
Webpage=18, Informational Factors=19, Contextual Factors=16,
and User Demographics=13. Several correlations were found
between the area of research and the types of outcomes. In the
realm of Decision Making, both papers coded under this area
corresponded to the Contextual Factors of ‘Experience & Family’
and ‘Reputation’. In the areas of Health Informatics, 45% of the
papers focused on the Content of the Information, and
Informational Factors. In Computing, 77% of the outcomes were
related to usability, with 66% presenting outcomes that related to
the user’s navigation of a webpage. The outcomes from the papers
within Information and Library Science were diverse having
outcomes to all five categories. This finding is similar within the
papers from the area of Social Science. Social Sciences presented
outcomes from all of the areas except contextual factors. Within
the area of Medicine, many of the papers did not present outcomes
that could be coded with our framework. However, 33% of their
papers presented outcomes relating to contextual factors.
4. Discussion
Research on trust stems from a variety of disciplines: psychology
[55], sociology [57], and business [40], to cite a few. While each
of these fields may not discuss the impact of user trust on health
websites, their definitions of user trust can still prove useful. The
empirical literature on trust and health websites encompasses 49
papers from ten areas. The first analysis of this body of research
analyzed the content of each paper for the frequency of used
terms. This research demonstrated that the three largest areas of
research, Medicine, Health Informatics, and Computing had the
largest impact on the frequency of term usage. This is to be
expected since they had the largest number of papers to pull from.
This analysis also found that as the frequency of terms being used
decreased, the percentage of usage became more equalized by
areas. While the domains of Medicine, Health Informatics, and
Computing had the largest proportion of papers, it is important for
research to determine the ‘quality of care,’ can overwhelmingly
impact the facets of trust studied by the research community.
While it may seem remiss to focus so heavily on the separate
communities that research trust, the interrelation of trust research
and their disciplines proves to be notably significant. For
example, Fruhling's paper [16], one in the realm of health
informatics, investigates the "User Interface usability factors" of
eHealth technology while Flanagin's paper [14], in the area of
Media & Society, instead examines end-user "perceptions of
sponsor credibility." Different disciplines ask different questions
thus making the analysis of the community of trust an important
topic to discuss.
Our second analysis of the body of research examined the social
network of how the papers were interrelated. From this analysis
we derived four measurements of the number of interconnections
between papers, whether certain papers served as between papers
in a network and how interconnected the papers were. These
measurements facilitated an analysis of whether work on trust in
health websites is building off of each other. Perhaps the largest
and simplest finding from this analysis is how little the research in
this area is citing each other. Recall that only half of the dataset
was citing each other. While the 23 papers that were not cited and
did not cite any other work might be used to build research in
their respective areas, they are not contributing to the research
discussion on trust and health websites. Trust in health websites is
still but an emerging field; however, even with only 49 relevant
papers in this field, the research is still highly fragmented.
The work of Sillence et al. [58] had the highest number of
connections, six papers, and the second highest betweenness
centrality score. However, what should be noted is that the work
of Sillence et al. is a body of six papers. This does not diminish
the body of work that Sillence et al. have produced, but explains
the large voice that these papers make up when considering the
larger body of work.
Table 2. Percentage of papers by area reporting outcomes by
category.

Demographics
Content of the

Informational
the other voices to not become marginalized as work in this area

Contextual
becomes more prevalent. The construct of trust in health websites
Webpage
Usability

Factors
Factors

is multi-dimensional thus the verbiage used to discuss it should be

as diverse.
To explain further, the analysis of terms used is important when
considering the antecedents and facets of trust in health websites; Medicine 0 0 13% 33% 20%
the choice of some terms over others to define and describe the Health Informatics 18% 45% 45% 18% 36%
construct of trust, its facets, and antecedents, are equally as Computing 77% 66% 66% 18% 36%
important as the focus of research being on trust and health. Our
research indicates that the terms being used by the areas with the Communication 33% 0 0 0 33%
highest number of publications is not an inclusive parlance. Health Services & 0 0 33% 33% 33%
Therefore, researchers should consider terms being used across all Healthcare
areas when defining and using the construct of trust to understand
Information & 33% 66% 66% 66% 33%
the user’s assessment of web credibility.
Library Science
An equally important finding from this analysis is the fact that Social Sciences 33% 33% 33% 0 33%
there was not a cohesive voice in the discussion of user trust in
Decision Making 50% 100% 100% 100% 0
health websites. With different research areas focusing on
different terms the discussion can become fragmented by area – Media & Society 0 100% 0 100% 0
indicating that research from different areas cannot build off of Pharmacology 0 0 0 0 0
each other and further the overall research discussion. For
example, the term ‘quality,’ which is used frequently for medical

The paper with the highest betweenness score was Lemire et al.
[37]. However, many papers did not function as a node connecting
two pieces of work. For example, in Figure 2, the work of Song
and Zahedi [65, 74] stands apart from the rest of the work. The
lack of papers functioning to interconnect to each other is a
further indicator of the fragmentation present in the work on trust
in health websites.
Closeness centrality and eigenvector centrality scores show that
the more important papers also have lower degrees. This could be
because of disconnected groupings such as Zahedi [65, 74] and
Newman [47, 48]. Future work in this area should explore citation
patterns and groupings with social networking analysis in a larger
group of articles. It is important to understand the connections
between theoretical papers and empirical findings and for the
discipline to adopt a shared or progressive definition of trust.
In our third form of analysis, we examined the empirical
outcomes by the areas that the papers stemmed from. To
accomplish this analysis we divided the outcomes into two
collections. The first collection demonstrated how trust in one
would affect trust in another. For example, one set of outcomes
demonstrated that trust in general media would affect trust in
health websites. Research on the transitive nature of trust
demonstrated that the areas of research would impact what
trustees were to be studied. Given that the spectrum of health
websites is only growing as technology grows, understanding the
transitivity of trust in this medium might be intractable at this
time. For example, there has been little research on studying the
impacts on trust of examining online health information on a
smart phone versus a home PC. With smart phones becoming one
of the primary methods of people accessing online information
across the world, it seems like this would be a fruitful avenue of
research. However, it was not one that was readily visible five
years ago because of the growth of this emerging technology. Our
analysis demonstrates that while outcomes in on this topic will be
fragmented by the area of research, this is perhaps a constructive
discourse.
It is in this body of research that insight into how trust in health
websites can impact the doctor-patient relationship.
Approximately 25% of the papers examined this relationship and
found that patient trust in health websites was affecting or was
affected by the doctor-patient relationship. (See the work of [19,
35, 50].) Or, contradictorily, the work of [48] found that this
relationship was not impacted. What is of value to take away from
this review is that the research community on trust in health
websites finds this relationship critical enough to attempt to study
its impact. The fact that there is still disagreement on how and
why it is affected is a clear indicator of a need for future work.
The second collection of outcomes was organized into categories
and factors that impacted a user’s trust in a health website. The
outcomes of the papers that made up categories and factors were
then coded by area. The span of the number of papers by category
indicates that there is not one category of outcomes that
outnumbers the others. This further shows that research in the five
categories of Usability, Content of a Webpage, Informational
Factors, Contextual Factors, and User Demographics might be a
useful framework for examining trust in health websites. When
examining the outcomes by area there were some, like Social
Sciences and Library and Information Science, which did report
findings that spanned many of the categories. This indicates that
while the parlance may not be the same between all the areas, the
outcomes that are being derived may be reproducible and valid.
This is a strong and important finding in such a new area of
research. However, there were some areas that reported outcomes
within particular categories of research. For example, Computing
was highly focused on the usability of the health webpage. While
the strength of a particular research area is its focus, it is
important for research to be able to build off of the findings of
others. This means that the body of research from an area should
examine how their focus could voice an opinion about the
findings from different areas.
Unsurprisingly, prior reviews of trust in other domains find that
the use and definition of trust is fragmented [18, 36]. Even within
the larger research area of trust in medical/health informatics,
definitions are in conflict citing trust as being a behavior and a
perception [70] or as an interpersonal interdependence [73]. As
fields of study mature, however, and the context of how a
particular construct, like trust, is being used allows the research
community congregate towards a shared definition. For instance,
the work of Grabner-Krauter and Kaluscha serves as a valuable
example in examining the related constructs in the study of trust
and ecommerce [18]. They found that for trust in ecommerce, the
‘trust’ construct is multi-layered and reflects institutional
phenomena as well as personal forms of trust. They argue, as we
do as well, that understanding how the construct of ‘trust’ is being
used within a particular domain is paramount to understanding the
outcomes and what is really being studied. It is for this reason,
that understanding what research areas are contributing to the
study of trust and health websites, and how they are defining their
constructs is particularly important.
5. Conclusion
In this paper we have presented a review of all empirical literature
on trust and health websites. We have demonstrated how the
constructs of trust and credibility are related, and how the
construct of trust is varied by domain. We have addressed how the
current research has affected our collective understanding of trust.
Our findings could indicate that the definition of trust is
fragmented across the various fields, and patterns of citation
between authors in this research area.
By prioritizing our analysis of the literature on the areas that the
research came from, our results suggest that the research
community is divided. We showed that different fields of study
use different language and factors to discuss the construct of trust.
The social network analysis demonstrated that researchers from
different fields are not citing each other furthering the evidence of
a fragmented field. This fragmentation may impede efforts to
define how research from different backgrounds might impact the
understanding of trust in health websites. Last, our analysis of the
outcomes of empirical trust research demonstrated that
researchers are finding contradictory and non-reflective outcomes.
All of these findings indicate that there is a need for a common
definition and model of user trust in health websites. While trust
in health websites may be different from, say, trust in
eCommerce, it is important to work towards a unification of this
definition in both fields. When implementing and deploying
content, such as a health infomediary, the understanding of trust
over all related domains is paramount.
Future work in the area of trust in health websites should build off
of the 49 papers in this review. Additionally, researchers should
evaluate and consider how the constructs of trust and credibility
are specific to the domain of health, instead of trying to
amalgamate research from related fields. Work could also be done
to apply the methods used in this paper to the research area of
trust in health/medical informatics as a broader field. Last,
research is emerging that examines trust as part of security, which
is a growing area of research. Understanding how the field of
security might be affecting the use and definition of trust will be
valuable for future work.
6. Acknowledgements
This publication was supported by grant 1UL1RR025011 from the
Clinical & Translational Science Award (CTSA) program of the
National Center for Research Resources National Institutes of
Health and by NSF Awards CCF-0830569 and #0851774. The
University of Wisconsin-Madison Systems Engineering Initiative
for Patient Safety (SEIPS) provided support on this project
http://cqpi.engr.wisc.edu.
7. References
[1] Anderson, L.A. and R.F. Dedrick, Development of the Trust
in Physician scale: a measure to assess interpersonal trust in
patient-physician relationships. Psychological Reports, 1990. 67(3
Pt 2): p. 1091-1091.
[2] Ayantunde, A.A., N.T. Welch and S.L. Parsons, A survey of
patient satisfaction and use of the internet for health information.
International Journal of Clinical Practice, 2007. 61(3): p. 458-462.
[3] Bates, B.R., S. Romina, R. Ahmed and D. Hopson, The
effect of source credibility on consumers' perceptions of the
quality of health information on the Internet. Medical Informatics
and the Internet in Medicine, 2006. 31(1): p. 45-52.
[4] Benotsch, E.G., S. Kalichman and L.S. Weinhardt, HIV-
AIDS Patients' Evaluation of Health Information on the Internet:
The Digital Divide and Vulnerability to Fraudulent Claims.
Journal of Consulting and Clinical Psychology, 2004. 72(6): p.
1004-1011.
[5] Borzekowski, D.L.G. and V.I. Rickert, Adolescent
cybersurfing for health information - A new resource that crosses
barriers. Archives of Pediatrics & Adolescent Medicine, 2001.
155(7): p. 813-817.
[6] Chang, H.H. and C.S. Chang, An assessment of technology-
based service encounters & network security on the e-health care
systems of medical centers in Taiwan. Bmc Health Services
Research, 2008. 8.
[7] Cima, R.R., K.J. Anderson, D.W. Larson, E.J. Dozois, I.
Hasson, W.J. Sandborn, et al., Internet use by patients in an
inflammatory bowel disease specialty clinic. Inflammatory Bowel
Diseases, 2007. 13(10): p. 1266-1270.
[8] Dart, J., The internet as a source of health information in
three disparate communities. Australian Health Review, 2008.
32(3): p. 559-569.
[9] Dutta-Bergman, M., Trusted online sources of health
information: differences in demographics, health beliefs, and
health-information orientation. Journal of Medical Internet
Research, 2003. 5(3).
[10] Ebert, T.A.E., Interdisciplinary trust meta-analysis. Retrieved
June, 2007. 25: p. 2007-2007.
[11] El-Attar, T., J. Gray, S.N. Nair, R. Ownby and S.J. Czaja.
Older adults and internet health information seeking. in 49th
Annual Meeting of the Human Factors and Ergonomics Society,
HFES 2005, Sep 26-30 2005. 2005. Santa Monica, CA 90406-
1369, United States: Human Factors an Ergonomics Society Inc.
[12] Eysenbach, G., J. Powell, O. Kuss and E.-R. Sa, Empirical
Studies Assessing the Quality of Health Information for
Consumers on the World Wide Web: A Systematic Review.
JAMA, 2002. 287(20): p. 2691-2700.
[13] Fisher, J., F. Burstein, K. Lynch and K. Lazarenko,
"Usability plus usefulness = trust": an exploratory study of
Australian health web sites. Internet Research, 2008. 18(5): p.
477-498.
[14] Flanagin, A.J. and M.J. Metzger, The role of site features,
user attributes, and information verification behaviors on the
perceived credibility of web-based information. New Media &
Society, 2007. 9(2): p. 319-342.
[15] Fox, S. and S. Jones (2009) The Social Life of Health
Information, Pew Research Center’s Internet & American Life
Project Web 2.0, Available from: http://bit.ly/WKgoj.
[16] Fruhling, A.L. and S.M. Lee, The influence of user interface
usability on rural consumers' trust of e-health services.
International Journal of Electronic Healthcare, 2006. 2(4): p. 305-
21.
[17] Glenton, C., E.S. Nilsen and B. Carlsen, Lay perceptions of
evidence-based information - a qualitative evaluation of a website
for back pain sufferers. Bmc Health Services Research, 2006. 6.
[18] Grabner-Krauter, S. and E.A. Kaluscha, Empirical research
in on-line trust: a review and critical assessment. International
Journal of Human-Computer Studies, 2003. 58(6): p. 783-812.
[19] Greenhalgh, T., G.W. Wood, T. Bratan, K. Stramer and S.
Hinder, Patients' attitudes to the summary care record and
HealthSpace: Qualitative study. British Medical Journal, 2008.
336(7656): p. 1290-+.
[20] Halkias, D., N. Harkiolakis, P. Thurman and S. Caracatsanis,
Internet use for health-related purposes among Greek consumers.
Telemedicine Journal and E-Health, 2008. 14(3): p. 255-260.
[21] Hall, M.A., E. Dugan, B. Zheng and A.K. Mishra, Trust in
physicians and medical institutions: what is it, can it be measured,
and does it matter? The Milbank Quarterly, 2001: p. 613-639.
[22] Hesse, B.W., D.E. Nelson, G.L. Kreps, R.T. Croyle, N.K.
Arora, B.K. Rimer, et al., Trust and sources of health information
- The impact of the Internet and its implications for health care
providers: Findings from the first Health Information National
Trends Survey. Archives of Internal Medicine, 2005. 165(22): p.
2618-2624.
[23] Hong, T., The influence of structural and message features
on Web site credibility. Journal of the American Society for
Information Science and Technology, 2006. 57(1): p. 114-127.
[24] Hovland, C., I.L. Janis and J.J. Kelley, Communication and
Persuasion. 1953, New Haven, CT: Yale University Press.
[25] Huh, J., D.E. DeLorme and L.N. Reid, Factors affecting trust
in on-line prescription drug information and impact of trust on
behavior following exposure to DTC advertising. Journal of
Health Communication, 2005. 10(8): p. 711-731.
[26] Huntington, P., D. Nicholas, B. Gunter, C. Russell, R.
Withey and P. Polydoratou, Consumer trust in health information
on the web. Aslib Proceedings, 2004. 56(6): p. 373-382.
[27] Ivanitskaya, L., I. O'Boyle and A.M. Casey, Health
information literacy and competencies of information age
students: Results from the interactive online Research Readiness
Self-Assessment (RRSA). Journal of Medical Internet Research,
2006. 8(2).
[28] Jian, J.-Y., A.M. Bisantz and C.G. Crury, Foundations for an
Empirically Determined Scale of Trust in Automated Systems.
International Journal of Cognitive Ergonomics, 2000. 4(1): p. 53-
71.
[29] Kaelber, D.C., A.K. Jha, D. Johnston, B. Middleton and
D.W. Bates, A Research Agenda for Personal Health Records
(PHRs). J Am Med Inform Assoc, 2008: p. M2547-M2547.
[30] Kelton, K., K.R. Fleischmann and W.A. Wallace, Trust in
digital information. Journal of the American Society for
Information Science and Technology, 2008. 59(3): p. 363-374.
[31] Khoo, K., P. Bolt, F.E. Babl, S. Jury and R.D. Goldman,
Health information seeking by parents in the Internet age. Journal
of Paediatrics and Child Health, 2008. 44(7-8): p. 419-423.
[32] Komiak, S.X. and I. Benbasat, Understanding Customer
Trust in Agent-Mediated Electronic Commerce, Web-Mediated
Electronic Commerce, and Traditional Commerce. Information
Technology and Management, 2004. 5(1): p. 181-207.
[33] LaCoursiere, S.P., M.T. Knobf and R. McCorkle, Cancer
Patients' Self-Reported Attitudes About the Internet. Journal of
Medical Internet Research, 2005. 7(3).
[34] Larsson, M., A descriptive study of the use of the Internet by
women seeking pregnancy-related information. Midwifery, 2009.
25(1): p. 14-20.
[35] Lee, C.J. and R.C. Hornik, Physician Trust Moderates the
Internet Use and Physician Visit Relationship. Journal of Health
Communication, 2009. 14(1): p. 70-76.
[36] Lee, J.D. and K.A. See, Trust in Automation: Designing for
Appropriate Reliance. Human Factors, 2004. 46(1): p. 50-80.
[37] Lemire, M., G. Paré, C. Sicotte and C. Harvey, Determinants
of Internet use as a preferred source of information on personal
health. International Journal of Medical Informatics, 2008. 77(11):
p. 723-734.
[38] Lumsden, J. and L. MacKay. How Does Personality Affect
Trust in B2C e-Commerce? in Proceedings of the 8th
International Conference on Electronic Commerce (ICEC’2006).
2006. Fredericton, New Brunswick, Canada.
[39] Marshall, L.A. and D. Williams, Health information: does
quality count for the consumer? How consumers evaluate the
quality of health information materials across a variety of media.
Journal of Librarianship and Information Science, 2006. 38(3): p.
141-156.
[40] Mayer, R.C., J.H. Davis and D. Schoorman, An Integrative
Model of Organizational Trust. The Academy of Management
Review, 1995. 20(3): p. 709-734.
[41] McKnight, H.D. and V. Choudhury, Distrust and trust in
B2C e-commerce: do they differ? Proceedings of the 8th
international conference on Electronic commerce: The new e-
commerce: innovations for conquering current barriers, obstacles
and limitations to conducting successful business on the internet,
2006. 156: p. 482-491.
[42] Menon, A., A. Ddeshpande, M. PerriIii and G. Mzinkhan,
Trust in online prescription drug information among Internet
users: The impact on information search behavior after exposure
to direct-to-consumer advertising. Health Marketing Quarterly,
2002. 20(1): p. 17-17.
[43] Montague, E.N.H., B.M. Kleiner and W.W. Winchester Iii,
Empirically understanding trust in medical technology.
International Journal of Industrial Ergonomics, 2009.
[44] Montague, E.N.H., B.M. Kleiner and W.W. Winchester Iii,
Empirically understanding trust in medical technology.
International Journal of Industrial Ergonomics, 2009. 39(4): p.
628-634.
[45] Montague, E.N.H., B.M. Kleiner and W.W. Winchester Iii,
Empirically understanding trust in medical technology.
International Journal of Industrial Ergonomics, 2009. In Press,
Corrected Proof.
[46] Muir, B.M., Trust in automation: Part I. Theoretical issues in
the study of trust and human intervention in automated systems.
Ergonomics, 1994. 37(11): p. 1905-1922.
[47] Newnham, G.M., W.I. Burns, R.D. Snyder, A.J. Dowling,
N.F. Ranieri, E.L. Gray, et al., Attitudes of oncology health
professionals to information from the Internet and other media.
Medical Journal of Australia, 2005. 183(4): p. 197-200.
[48] Newnham, G.M., W.I. Burns, R.D. Snyder, A.J. Dowling,
N.F. Ranieri, E.L. Gray, et al., Information from the Internet:
attitudes of Australian oncology patients. Internal Medicine
Journal, 2006. 36(11): p. 718-723.
[49] Pasmore, W.A., Designing Effective Organizations: The
Sociotechnical Systems Perspective. 1988: John Wiley & Sons.
[50] Powell, J. and A. Clarke, Internet information-seeking in
mental health - Population survey. British Journal of Psychiatry,
2006. 189: p. 273-277.
[51] Rains, S.A., Perceptions of traditional information sources
and use of the world wide web to seek health information:
Findings from the Health Information National Trends Survey.
Journal of Health Communication, 2007. 12(7): p. 667-680.
[52] Rains, S.A. and C.D. Karmikel, Health information-seeking
and perceptions of website credibility: Examining Web-use
orientation, message characteristics, and structural features of
websites. Computers in Human Behavior, 2009. 25(2): p. 544-
553.
[53] Rosenbaum, S.E., C. Glenton and J. Cracknell, User
experiences of evidence-based online resources for health
professionals: User testing of The Cochrane Library. Bmc
Medical Informatics and Decision Making, 2008. 8.
[54] Rosenvinge, J.H., S. Laugerud and P. Hjortdahl, Trust in
health Websites: A surevey among Norwegian Internet users.
Journal of Telemedicine and Telecare, 2003. 9(3): p. 161-166.
[55] Rotter, J.B., Interpersonal trust, trustworthiness, and
gullibility. American Psychologist, 1980. 35(1): p. 1-7.
[56] Semere, W., H.L. Karamanoukian, M. Levitt, T. Edwards,
M. Murero, G. D'Ancona, et al., A pediatric surgery study: Parent
usage of the Internet for medical information. Journal of Pediatric
Surgery, 2003. 38(4): p. 560-564.
[57] Shapiro, S.P., The Social Control of Impersonal Trust. The
American Journal of Sociology, 1987. 93(3): p. 623-658.
[58] Sillence, E., P. Briggs, L. Fishwick and P. Harris. Trust and
mistrust of online health sites. in 2004 Conference on Human
Factors in Computing Systems - Proceedings, CHI 2004, Apr 24-
29 2004. 2004. New York, NY 10036-5701, United States:
Association for Computing Machinery.
[59] Sillence, E., P. Briggs, L. Fishwick and P. Harris. Guidelines
for developing trust in health websites. in Special interest tracks
and posters of the 14th international conference on World Wide
Web. 2005. Chiba, Japan: ACM.
[60] Sillence, E., P. Briggs, P. Harris and L. Fishwick, A
framework for understanding trust factors in web-based health
advice. International Journal of Human-Computer Studies, 2006.
64(8): p. 697-713.
[61] Sillence, E., P. Briggs, P. Harris and L. Fishwick, Going
online for health advice: Changes in usage and trust practices over
the last five years. Interacting with Computers, 2007. 19(3): p.
397-406.
[62] Sillence, E., P. Briggs, P. Harris and L. Fishwick, Health
Websites that people can trust - the case of hypertension.
Interacting with Computers, 2007. 19(1): p. 32-42.
[63] Sillence, E., P. Briggs, P.R. Harris and L. Fishwick, How do
patients evaluate and make use of online health information?
Social Science & Medicine, 2007. 64(9): p. 1853-1862.
[64] Smith, M.A., B. Shneiderman, N. Milic-Frayling, E.M.
Rodrigues, V. Barash, C. Dunne, et al. Analyzing (social media)
networks with NodeXL. in Proceedings of the fourth international
conference on Communities and technologies. 2009: ACM New
York, NY, USA.
[65] Song, J. and F.M. Zahedi, Trust in health infomediaries.
Decis. Support Syst., 2007. 43(2): p. 390-407.
[66] Taylor, M.R.G., A. Alman and D.K. Manchester, Use of the
internet by patients and their families to obtain genetics-related
information. Mayo Clinic Proceedings, 2001. 76(8): p. 772-776.
[67] Theng, Y.-L. and E.-S. Soh. An Asian study of healthcare
Web portals: Implications for healthcare digital libraries. in 8th
International Conference on Asian Digital Libraries, ICADL
2005, Dec 12-15 2005. 2005. Heidelberg, D-69121, Germany:
Springer Verlag.
[68] Thom, D.H. and B.B. Campbell, Patient‐physician trust: an
exploratory study. Journal of Family Practice, 1997. 44(2): p.
169(8).
[69] Tio, J., A. LaCaze and W.N. Cottrell, Ascertaining consumer
perspectives of medication information sources using a modified
repertory grid technique. Pharmacy World & Science, 2007.
29(2): p. 73-80.
[70] Tung, F.-C., S.-c. Chang and C.-M. Chou, An extension of
trust and TAM model with IDT in the adoption of the electronic
logistics information system in HIS in the medical industry.
International Journal of Medical Informatics, 2008. 77(5): p. 324-
335.
[71] Wainstein, B.K., K. Sterling-Levis, S.A. Baker, J. Taitz and
M. Brydon, Use of the Internet by parents of paediatric patients.
Journal of Paediatrics and Child Health, 2006. 42(9): p. 528-532.
[72] Walther, J.B., Z.M. Wang and T. Loh. The effect of top-level
domains and advertisements on health Web site credibility. 2004.
[73] Winkelman, W.J., K.J. Leonard and P.G. Rossos, Patient-
Perceived Usefulness of Online Electronic Medical Records:
Employing Grounded Theory in the Development of Information
and Communication Technologies for Use by Patients Living with
Chronic Illness. Journal of the American Medical Informatics
Association, 2005. 12(3): p. 306-314.
[74] Zahedi, F. and J. Song, Dynamics of trust revision: Using
health infomediaries. Journal of Management Information
Systems, 2008. 24(4): p. 225-248.