Systematic reviews of

qualitative research
D. Evans and A. Pearson

Systematic reviews have become an important component of evidence-based health

care. To date, these reviews have focused predominantly on randomized controlled
trials to determine the effectiveness of treatments. However, there is a growing interest
in reviews of qualitative research to summarize research addressing a phenomenon of
interest or to supplement evidence generated by randomized controlled trials. These
reviews have the potential to make an important contribution because they can bring
the person s perspective in to the decision making process. Like all systematic reviews,
optimal methods for reviewing qualitative research are still evolving, however published
reviews demonstrate considerable variability in their approach. Despite their potential
usefulness there has been little professional discussion and debate. This paper presents
the current methods used for the review of qualitative studies and highlights some areas
in need of further exploration. °C 2001 Harcourt Publishers Ltd

INTRODUCTION than the RCT. For example, because qualitative

David Evans RN.
DipN. BN. MNS,
Coordinator of Reviews
The Joanna Briggs
Institute for Evidence
Based Nursing and
Midwifery
Alan Pearson RN PhD
Professor of Nursing
La Trobe University
Correspondence to:
David Evans, The Joanna
Briggs Institute, Margaret
Graham Building, The
Royal Adelaide Hospital,
North Terrace,
Adelaide, South
Australia, Australia 5000;
Tel.: (61) 8 8303 4880;
E-mail:
d.evans@adelaide.edu.au
During the past two decades there has been a grow-
ing interest in the scientific basis of health care
interventions. In conjunction with this, the sys-
tematic review has risen to prominence as means
to identify, appraise and then synthesize research.
These reviews have focused predominantly on ran-
domized controlled trial (RCT) to determine the
effectiveness of interventions. This pivotal role of
the systematic review is reflected in the consid-
erable professional discussion, development and
refinement of review methods, addressing issues
such as search strategies (Adams et al. 1992;
Counsell 1997; Dickersin 1994), critical appraisal
(Jadad et al. 1996; Moher et al. 1995; Sindhu et al.
1997) and meta-analysis (DeMets 1987; Dickersin
and Berlin 1992; LeLorier et al. 1997).
However, despite all this activity, the scientific
basis of health care has been narrowly defined.
This is a consequence of measuring the value of
interventions primarily by effectiveness. As a re-
sult, the RCT is commonly seen as the gold stan-
dard because it is the optimal method to explore
issues related to cause and effect. This is not to
suggest that knowing whether interventions work
is not vital, rather that it is also important to know
whether interventions should and can be used.
Some of the evidence needed to answer ques-
tions about the appropriateness and feasibility of
interventions can be generated by methods other
studies explore the beliefs, expectations and un-
derstandings of patients, they have the potential to
influence health care decisions.
Despite this potential, qualitative evidence has
not played a significant role in the evidence-based
movement. Part of the reason for this is the exclu-
sion of qualitative studies from systematic reviews.
With the increasing use of systematic reviews as
the basis for health care decisions, this exclusion
has meant that qualitative research has been ex-
cluded from the decision making process.
Despite this marginalization of qualitative re-
search, there is a growing interest in the synthesis
of data generated by individual qualitative studies.
It has been suggested that by combining quali-
tative studies in a review, findings become more
significant because they draw on broader range
of participants and descriptions (Sherwood 1999).
These reviews consolidate the body of knowledge
on a topic, organizing and describing the charac-
teristics of studies, their methods and the findings.
As a result of this process a new interpretation of
the phenomenon is produced, contributing to the
understanding of the topic. For example, these re-
views allow exploration of research addressing the
impact of illness, patients experiences with health
care, or organizational aspects of care delivery.
These reviews can also contribute contextual in-
formation to supplement evidence generated by
RCTs.

Clinical Effectiveness in Nursing (2001) 5, 111–119 °

C 2001 Harcourt Publishers Ltd

doi:10.1054/cein.2001.0219, available online http://www.idealibrary.com on

112 Clinical Effectiveness in Nursing
This interest has been matched by a growing
discussion in the health care literature addressing
the synthesis of qualitative research (Estabrooks
et al. 1994; Jensen and Allen 1996; Popay et al.
1998; Sandelowski et al. 1997; Sherwood 1999). A
small number of reviews of qualitative studies have
been published addressing issues such as chronic
illness (Thorne and Paterson 1998), the concept of
caring (Sherwood 1997), the concept of wellness
and illness (Jensen 1994) and stressors in fami-
lies with a child with a chronic condition (Ogden-
Burke et al. 1998). However, methods for review-
ing, summarizing and synthesizing qualitative
research are still evolving and optimal approaches
have yet to be determined. This paper presents
the current approaches used for qualitative reviews
and a discussion of related issues.
REVIEWS OF QUALITATIVE
RESEARCH
Noblit and Hare developed an approach for the
synthesise the findings of ethnographic and other
types of qualitative studies that they termed meta-
ethnography (Noblit and Hare 1988). More re-
cently the broader term meta-synthesis has come
into use (Jensen and Allen 1996; Sandelowski
et al. 1997; Sherwood 1999), referring to the syn-
thesis of findings from different types of qualitative
research. Sherwood describes meta-synthesis as
the critical review and analysis of processed data,
or the themes, domains and descriptions generated
by multiple qualitative studies (Sherwood 1999).
The goal of meta-synthesis differs significantly
from meta-analysis, being interpretive rather than
aggregative, it seeks to generate understanding or
interpretive explanations of a phenomenon (Jensen
and Allen 1996). Through the process of synthesis,
the review allows critical examination of multiple
accounts of an event or situation, a systematic com-
parison of studies, a way of taking about specific
work and comparing it to the work of others, and
synthesis of qualitative research (Noblit and Hare
1988).
However, the optimal methods for reviewing
this research are still unclear and approaches em-
ployed in published reviews differ. While the steps
used during the review of RCT, such as formulating
a question, locating the studies, critical appraisal,
data extraction and data synthesis, also apply to re-
views of qualitative research, there are significant
differences in their application.
QUALITATIVE REVIEW METHODS
Defining the focus
Like systematic reviews, reviews of qualitative re-
search seek to answer a specific question. The
question must be broad enough to be of interest but
small enough to be manageable (Sherwood 1999).
Delineating a clear focus is particularly important
with qualitative reviews because the data is nar-
rative text, which does not lend itself to concise
summary and synthesis in the same manner as nu-
merical data. Too broad a review question will sig-
nificantly increase the time and work required to
complete the review. However there is little guid-
ance on how best to strike this balance. It is likely
that the principles used to guide the primary re-
searcher will be equally useful by the reviewer.
There are fundamental differences in the focus
of reviews of RCT compared to those involving
qualitative research. Meta-analysis aims at provid-
ing definitive evidence on the effectiveness of an
intervention rather than increasing our understand-
ing of a phenomenon. Yet each approach provides
vital information for practice. To date these two
approaches have been seen as mutually exclusive.
That is, reviews focus on either RCT or qualitative
research, never both. Yet both provide a unique fo-
cus, and so perhaps an opportunity exists to com-
bine approaches to provide a richer source of ev-
idence on which to base practice. The evidence
generated from this type of review would inform
practice not only on the effectiveness of an inter-
vention, but also what impact it has on the person.
However, this potential has yet to be explored.
Locating studies
Systematic reviews of RCT attempt to identify all
work on the topic of interest and this is reflected in
attempts to improve search strategies (Adams et al.
1992; Counsell 1997; Dickersin 1994; Hay et al.
1996; Jadad and McQuay 1993). This approach
is consistent with RCT methods that aim to con-
trol variables, use objective outcome measures and
ensure complete follow up and accounting of all
participants. However, it is not clear whether re-
views of qualitative research should adopt a sim-
ilar approach or use a sample of the total number
of available studies. Additionally, if only a sam-
ple of the literature is to be used, the size of the
sample to ensure an adequate understanding of the
phenomenon is difficult to determine (Jensen and
Allen 1996).
An exhaustive search of the literature, as is done
during systematic reviews, would ensure that all
relevant studies are included in the review. This
approach would help prevent important informa-
tion being excluded from the review. Additionally,
the findings would be strengthened because of the
broader base from which the descriptions were
generated. However, for topics with a large number
of published studies, attempting to summarise and
synthesize 100 or more qualitative studies would
be a vast and time consuming undertaking.
The alternative approach of using only a sam-
ple of available studies would make it possible

to generate a rich and detailed text on the phe-
nomenon. This type of review would not aim to
identifying every issue on the topic, rather it would
generate a text documenting the major themes and
categories from the sample of studies. In support
of this approach, Noblit and Hare (Noblit and Hare
1988) argue that unless there is some substan-
tive reason for an exhaustive search, generalisa-
tion from all studies on a particular topic yields
trite conclusions. However, with the use of a repre-
sentative sample of studies, information that could
make an important contribution to the understand-
ing of the topic may be omitted from the review.
Examining published reviews reveals a lack of
consistency in approaches used to review qualita-
tive studies. Some reviewers attempted to find all
past research on a topic (Sherwood 1997; Thorne
and Paterson 1998), others placed publication date
limits on the search (Jensen 1994; Ogden-Burke
et al. 1998; Paterson et al. 1998), while one re-
viewer chose a sample of studies, recent articles
and older classics (Bunting 1997).
While it appears that reviews of qualitative
research can be based on either a comprehensive
or representative sample of studies, the impact of
this decision on review findings is not clear. For
topics with a limited number of studies, a com-
prehensive approach would be the obvious choice.
For topics with a substantive body of research, a
representative sample may be one way in which
to deal with large numbers of studies. While there
have been no comparisons of the two approaches,
the summary provided by each would likely differ.
Reviews that use only a sample of available studies
would be at considerably greater risk of missing
information that could make an important contri-
bution to the understanding of the phenomenon.
Selecting studies for inclusion
in review
Criteria of eligibility are used during the conduct of
systematic reviews to ensure only studies address-
ing the specific topic of interest are included in
the review. These criteria are based on clear scien-
tific reasoning and are developed and documented
prior to commencement of the review (Mulrow and
Oxman; NHS Centre for Reviews and Dissemi-
nation 1996; Petitti 1994). When the focus of the
systematic review is RCT, the inclusion criteria de-
scribes the population, intervention and outcome
measures of interest. When selecting qualitative
studies for inclusion in a review Estabrooks et al.
(Estabrooks et al. 1994) suggest that criteria of
importance include:
• Studies should focus on similar populations or
themes
• Studies should have a similar research
approach, as mixing methods may lead to
difficulty in developing theory
Systematic reviews of qualitative research 113
• Reporting of themes and labels must be clear
and must be grounded in the data to be able to
demonstrate comparability of themes and
labels across studies.
Evaluating published reviews of qualitative re-
search, not all state the criteria used for study selec-
tion and few provided full details. Of the reviews
that cited information about study selection, issues
addressed by the criteria included:
• Clear description of study participants or that
the study used a specific group of participants
(Bunting 1997; Ogden-Burke et al. 1998;
Thorne and Paterson 1998)
• The study focused on a specific issue such as
illness or investigated the experience of living
with a condition such as diabetes or chronic
illness (Jensen 1994; Paterson et al. 1998;
Thorne and Paterson 1998)
• The report provided enough description of the
method to infer qualitative approach was used
(Ogden-Burke et al. 1998; Paterson et al.
1998)
• Primary researchers were nurses (Bunting
1997)
• The study was published within specific time
frame (Thorne and Paterson 1998)
• The study was reported in English (Thorne
and Paterson 1998), and
• Sufficient evidence of data trail was provided
(Thorne and Paterson 1998).
Based on information provided in these re-
views, the most common criteria used for study
selection addresses participants, study focus and
description of qualitative method. Sufficient ev-
idence of a data trail has also been highlighted
as an important issue by Thorne (Thorne and
Paterson 1998) and Estabrooks (Estabrooks et al.
1994). However, unlike criteria used to select RCT
for inclusion in systematic reviews, there appears
to be little agreement on the optimal components
of criteria for the selection of qualitative studies.
Additionally, the reporting of the criteria used is at
times inadequate, and this issue has received only
limited attention, suggesting further investigation
and clarification is needed.
Critical appraisal
Prior to inclusion in a systematic review, RCT are
critically appraised to exclude any studies whose
results are at risk of error or bias. Whether this
process should be applied to reviews of qualitative
research is less clear. Critical appraisal of quali-
tative research has been the subject of consider-
able controversy and discussion (Engel and Kuzel
1992; Koch 1994; Lincoln 1995; Sandelowski
1993; Schwandt 1996; Smith 1987), and its role
in reviews has yet to be adequately addressed.
Estabrooks et al. (Estabrooks et al. 1994) warn of
114 Clinical Effectiveness in Nursing
the risk of including studies with superficial anal-
ysis or research biases and they suggest that these
weak studies should stand out as an anomaly or
because supporting data is missing. While this
critical appraisal of qualitative studies has been
supported and used in reviews (Sherwood 1999;
Sherwood 1997), others argue against appraisal
(Jensen 1994; Jensen and Allen 1996).
Even the term qualitative adds to this difficulty
because it is used to represent a range of different
methods that are commonly categorized together.
These differing methods are often not differen-
tiated when issues related to scientific rigour are
addressed. Additionally, Burns argues that the ma-
jor obstacle to the identification of excellence in
qualitative studies is attempts to evaluate these
studies using quantitative criteria (Burns 1989).
Criteria developed by Burns to appraise qualitative
studies, addresses all types of qualitative research
(Burns 1989):
1. Descriptive Vividness
2. Methodological Congruence
• Rigour in documentation
• Procedural rigour
• Ethical rigour
• Auditability.
3. Analytical Preciseness
4. Theoretical Connectedness
5. Heuristic Relevance
• Intuitive recognition
• Relationship to existing body of knowledge
• Applicability.
The value of this tool as part of the system-
atic review process is not clear, but it is likely that
different appraisers using this tool would reach
differing conclusions.
From the perspective of the reviewer, standard-
ized criteria would aid in the conduct of reviews
of qualitative research. However, to date there has
been no agreement on these criteria. In evaluating
the approach to critical appraisal, as reported in
published qualitative reviews, there is great vari-
ation in the methods used and little consistency.
During a review of research addressing chronic
illness, Thorne and Paterson appraised all studies
to assess the rigour, epistemologic soundness, and
fruitfulness of the research methods used (Thorne
and Paterson 1998). A review addressing wellness-
illness placed no restriction on the scientific merit
of studies to avoid eliminating data germane to the
purpose of the investigation (Jensen 1994). One re-
view, focusing on how people adapted to the man-
agement of their diabetes, failed to mention critical
appraisal or study quality in the description of the
review method (Paterson et al. 1998). Another re-
view addressing stressors in families with a child
with chronic illness, also did not appraise studies,
and noted that studies published in clinical journals
that did not give details of the analytic procedures
were also included in the review (Ogden-Burke
et al. 1998).
While the debate continues on how best to ap-
praise qualitative research, minimum criteria do
emerge from completed reviews and the related
literature. A clear description of method and par-
ticipants is vital. Without this description, any at-
tempt at summary and synthesis of findings would
be difficult. An inadequate description would also
increase the risk of synthesizing dissimilar stud-
ies. Equally important is evidence of a data trail.
Estabrooks et al. note that when examining quali-
tative research findings, it is critical that labels are
closely and explicitly tied to supportive data or ex-
emplars (Estabrooks et al. 1994). This is because
categories or themes used across different studies
may not always have the same label.
However, the issue of how best to critically
appraise research is not limited to qualitative re-
search, as witnessed by the large number of check-
lists and scales that have been developed for the
appraisal of RCT (Moher et al. 1995). This sug-
gests that critical appraisal of all research is an
emerging field and will continue to grow and de-
velop in the future.
Data extraction
The aim of this phase of systematic reviews is to
collect data from studies that are reliable, valid
and free from bias (Petitti 1994). To this end, a
data collection form is used to bridge the gap be-
tween what is reported by the primary investiga-
tor and what is ultimately reported by the reviewer
(Mulrow and Oxman 1997). Information collected
during reviews of RCT include such things as bibli-
ographic details, description of study setting, pop-
ulation, details on delivery of intervention, out-
come measures, study method, any factor that may
influence the validity of findings and the actual re-
sults of the research (NHS Centre for Reviews and
Dissemination 1996).
During reviews of qualitative studies the data
collection phase is less clear and is usually reported
as being part of the synthesis process, during the
reading and rereading of the text (Jensen 1994;
Noblit and Hare 1988; Paterson et al. 1998). The
text of individual studies is treated more as a whole
during initial readings while major themes and cat-
egories are identified, and then compared and con-
trasted with those of other studies. This represents
a significant departure from the approached used
with RCT, where collection of data is a distinct
phase of the review, involving checks to maintain
rigour and pilot testing of forms.
However, the use of coding sheets have been
recommended to help track study details such
as research design, assumptions and sample
(Sherwood 1999). Jensen and Allen recommended
listing key metaphors, phrases, concepts, ideas and

categories, although this is part of the synthesis
process rather than a distinct data collection phase
(Jensen and Allen 1996). Additionally, some re-
views provide details of study characteristics in the
form of tables (Jensen 1994; Ogden-Burke et al.
1998; Sherwood 1997), suggesting that specific
data was collected at some stage of the review
process.
One review of qualitative research address-
ing chronic illness identified over 400 studies
of which 158 met the review inclusion criteria
(Thorne and Paterson 1998). Clearly, some top-
ics have a large body of research, and that deal-
ing with this volume of information would be
impossible without a formalized process of col-
lection and recording of relevant information and
study characteristics. Additionally, reporting the
characteristics of studies would be an important
component of the final review report, and would
represent part of the reviews data trail that has
been claimed to be so important for primary
research. However, this component of reviews
of qualitative studies has received only limited
attention.
Data synthesis
The analysis and synthesis of qualitative stud-
ies is commonly termed meta-synthesis, and like
meta-analysis, it is based on processed data. How-
ever, there are major differences between the ap-
proach used to synthesise the findings of RCT
and qualitative studies. Reality for the qualita-
tive researcher, and reviewer, is viewed as mul-
tiple and constructed (Sandelowski 1993), and
so undertaking meta-synthesis means that no two
reviewers will produce exactly the same results
(Jensen and Allen 1996). While meta-synthesis
provides only one interpretation, it aims to cap-
ture the essence of the phenomenon of interest
(Sherwood 1999).
First stage of the synthesis involves reading
and re-reading of texts from the individual stud-
ies, noting interpretive metaphors, themes and
categories. This phase continues throughout the
duration of the synthesis (Jensen and Allen 1996;
Sherwood 1999). Data is standardized through
common codes and outlines, and it has been sug-
gested that a table summarizing the bibliographic
information and other characteristics of studies
helps display details for comparison and com-
pilation (Sherwood 1999). Studies are sorted by
methodology, data collection instrument and data
analysis methods (Sherwood 1999), and by key
metaphors, phrases, ideas and concepts (Jensen
and Allen 1996). This process allows data to be
reduced to common denominators, as is done dur-
ing meta-analysis (Sherwood 1999). Studies are
examine for homogeneity and the initial assump-
tions about the relationships between studies are
developed (Sherwood 1999).
Systematic reviews of qualitative research 115
While heterogeneity of results is a cause for
concern during meta-analysis, greater variation
in findings should be anticipated from qualita-
tive studies. However, these differences are not
merged, rather these differences are compared
and contrasted, and areas of commonality iden-
tified as part of the synthesis (Paterson et al. 1998;
Sherwood 1997). The aim of this synthesis is
to portray an accurate interpretation of the phe-
nomenon, and to compare and contrast the con-
structs of individual studies to reach consensus on
a new construction of the phenomenon (Sherwood
1999). Studies are translated into one another,
which involves comparing both the metaphors or
concepts and the interaction of one account with
those of other accounts (Jensen and Allen 1996;
Sherwood 1999). This results in a progressive re-
finement of the understanding of the phenomenon
through a process of constant comparative analy-
sis. The aim is to generate a new construction on
which there is consensus (Jensen and Allen 1996).
Differing research methods, such as phe-
nomenology, ethnography or grounded theory, are
not mixed in a single synthesis of all qualitative
studies. If different qualitative research methods
are mixed, it will be unclear what has been
obtained from the synthesis. (Jensen and Allen
1996). The basis for this is that while qualitative
studies using differing methods may address the
same topic, each method provides its own unique
perspective.
As qualitative studies differ from RCT, meta-
synthesis also uses an approach that is markedly
different from that used during meta-analysis.
However, both approaches provide an understand-
ing that is based on a range of populations, settings
and circumstances. This broad base for generation
of evidence on a phenomenon allows for greater
confidence in the evidence. However, unlike meta-
analysis, meta-synthesis deals in multiple realities
and so it provides but one interpretation of the
phenomenon.
DISCUSSION
While reviews of qualitative research have the po-
tential to make an important contribution to nurs-
ing knowledge some issues have yet to be resolved,
and like systematic reviews, it is a method that is
still evolving. The published reviews are impor-
tant in that they represent exploration of not only
the review topic, but also of potential review meth-
ods. These reviews provide a basis for continued
development of methods and help focus profes-
sional debate. However, to date this professional
debate has been quite limited.
Of the reviews published, some provide only
limited information of the processes and meth-
ods used. For example, information related to how
studies were identified, assessed for suitability to
116 Clinical Effectiveness in Nursing
the review and critically appraised, is often inad-
equate. This limits the opportunity to appraise the
trustworthiness of the review itself. This is a sig-
nificant departure from reporting of systematic re-
view methods, which aim to provide sufficient in-
formation to permit the assessment and appraisal
of the methods used. As an emerging field, this
complete description is also important for future
endeavours, in that it allows continued refinement
and development of review methods. This lack of
information, combined with the occasional poor
description of the findings of the review, means
that the data trail that is vital for qualitative study
reports is lacking from review reports.
It is not clear what questions can reasonably
be addressed by this type of review. While it is
likely that suitable questions for the primary re-
searcher would also be appropriate for the re-
viewer, this issues has not been fully explored.
The validity of the findings of qualitative reviews
also requires further exploration. These findings
are based on the third level of interpretation of the
phenomenon, that is, the event or phenomenon is
interpreted by the person, the researcher and finally
by the reviewer. A concern is that reviews may in-
crease bias by using these interpretations rather
than the original informants descriptions (Jensen
and Allen 1996). The risk is that these reviews pro-
duce readable, but meaningless, descriptions that
are too far removed from the original event to be of
any value for generating an understanding about a
phenomenon.
The full potential of this type of review has
yet to be realised. While they are currently used
to increase the knowledge and understanding on
a phenomenon, such as the experience of chronic
illness or management diabetes, they could also
provide vital information that is complementary
to that provided by systematic reviews of RCT.
From this perspective, reviews of qualitative re-
search could complement the answer to the ques-
tion does this intervention work provided by the
RCT, with answers to should it be used or what
does this mean to the person.
These reviews have potential to bring the pa-
tient’s perspective into the decision making pro-
cess. For example, issues such as chronic pain,
incontinence or the fear of falling can be acknowl-
edged as being important because of their impact
on a person’s life. Qualitative research should play
an important role in the evidenced based health
care movement, influencing both clinical practice
and policy decisions. However, in the current ap-
proach to health care, this research has had only a
peripheral role.
CONCLUSION
Reviews of qualitative research have the potential
to make an important contribution to health care.
However optimal methods have yet to be deter-
mined and published reviews demonstrate some
variability in the approach to the summary and
synthesis of studies. However, while the methods
used to review qualitative research are still evolv-
ing, professional discussion and debate have been
limited. Issues such as what questions can best be
addressed by this approach, and the role of these
reviews within the broader evidence-based health
care movement, have yet to be fully explored. Fi-
nally, qualitative reviews are an emerging field,
and therefore the published reviews are important
in that they provide the basis for on going devel-
opment and refinement of methods.
REFERENCES
Adams CE, Lefebvre C, Chalmers I 1992 Difficulty with
MEDLINE: searches for randomised controlled trials.
The Lancet 340: 915–916
Bunting SM 1997 Applying a feminist analysis model to
selected nursing studies of women with HIV. Issues in
Mental Health Nursing 18: 523–537
Burns N 1989 Standards for qualitative research. Nursing
Science Quarterly 2: 44–52
Counsell C 1997 Formulating questions and locating primary
studies for inclusion in systematic reviews. Annals of
Internal Medicine 127: 380–387
DeMets DL 1987 Methods for combining randomised clinical
trials: strength and limitations. Statistics in Medicine 6:
341–348
Dickersin K, Scherer R, Lefebvre C 1994 Identifying relevant
studies for systematic reviews. British Medical Journal
309: 1286–1291
Dickersin K, Berlin JA 1992 Meta-analysis: state of the
science. Epidemiologic Review 14: 154–176
Engel JD, Kuzel AJ 1992 On the idea of what constitutes
good qualitative inquiry. Qualitative Health Research
2(4): 504–510
LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian
F 1997 Discrepancies between meta-analyses and
subsequent large randomised controlled trials. New
England Journal of Medicine 337: 536–542
Lincoln YS 1995 Emerging criteria for quality in qualitative
and interpretive research. Qualitative Inquiry 1(3):
275–289
Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh
S 1995 Assessing the quality of randomized controlled
trials: an annotated bibliography of scales and checklists.
Controlled Clinical Trials 16: 62–73
Mulrow CD, Oxman AD Cochrane Collaboration Handbook
[updated 1 March 1997], In: The Cochrane Library
[database on disk and CDROM]: The Cochrane
Collaboration, Oxford: Update Software
NHS Centre for Reviews and Dissemination 1996
Undertaking Systematic Reviews of Research on
Effectiveness: CRD Guidelines for Those Carrying Out
or Commissioning Reviews. York: University of York
Noblit GW, Hare RD 1988 Meta-ethography: Synthesizing
Qualitative Studies. Vol. 11. Newbury Park: Sage
Publications
Ogden-Burke S, Kauffmann E, Costello E, Wiskin N,
Harrison MB 1998 Stressors in families with a child with
a chronic condition: analysis of qualitative studies and a
framework. Canadian Journal of Nursing Research
30(1): 71–95
Paterson BL, Thorne S, Dewis M 1998. Adapting to and
managing diabetes. Image 30(1): 57–62
Petitti DB 1994 Meta-analysis, decision analysis and
 Systematic reviews of qualitative research 117

cost-effectiveness analysis. New York: Oxford
University Press
Popay J, Rogers A, Williams G 1998 Rationale and standards
for the systematic review of qualitative literature in
health services research. Journal of Qualitative Health
Research 8(3) 341–51
Sandelowski M 1993 Rigor or rigor mortis: the problem of
rigor in qualitative research revisited. Advances in
Nursing Science 16(2): 1–8
Sandelowski M, Docherty S, Emden C 1997 Qualitative
metasynthesis: issues and techniques. Research in
Nursing and Health 20: 365–371
Schwandt TA 1996 Farewell to criteriology. Qualitative
Inquiry 2(1): 58–72
Sherwood G 1999 Meta-synthesis: merging qualitative
studies to develop nursing knowledge. International
Journal of Human Caring 3(1): 37–42
Sherwood GD 1997 Meta-synthesis of qualitative analyses of
caring: defining a therapeutic model of nursing.
Advanced Practice Nursing Quarterly 3(1): 32–42
Sindhu F, Carpenter L, Seers K 1997 Development of a tool
to rate the quality assessment of randomized controlled
trials using a Delphi technique. Journal of Advanced
Nursing 25: 1262–1268
Smith ML 1987 Publishing qualitative research. American
Educational Research Journal 24: 173–182
Thorne S, Paterson B 1998 Shifting images of chronic illness.
Image: Journal of Nursing Scholarship 30(2): 173–178

COMMENTARIES
D. Evans and A. Pearson Clinical Effectiveness in Nursing (2001) 5, 111–119
Systematic reviews of qualitative research

This paper outlines the major theoretical and [albeit limited] empirical positions in the debate sur-
rounding the role of qualitative research in systematically produced reviews of evidence. Good quality
qualitative research with its rich, deep, meaningful description of a phenomenon or events is ideally
placed to fill the information gaps in decisions involving the experiences of patients receiving healthcare.
The authors recognise the value of such understanding to decision making but then go on to highlight
some of the problems associated with systematically reviewing qualitative research. Their picture is
somewhat bleak and I would like to propose a more optimistic response – borne largely of pragmatism.
Firstly, I do not agree that qualitative research has been excluded from the evidence based healthcare
agenda. The United Kingdom Health Technology Assessment (HTA) programme has published reviews
of the role of qualitative research in the evaluation of health technologies (Murphy et al. 1998) and has
recently commissioned a study into ways of developing the role of qualitative research in systematic
reviews. Moreover, both the International Cochrane (for healthcare) and Campbell (for social policy)
collaborations have methods groups concerned with the role of qualitative research in systematic reviews
of effectiveness. The limited role of qualitative research in evidence based healthcare may not be due to
the myopia of commissioners and policy makers but rather the nihuistic responses qualitative researchers
sometimes entertain when having to demonstrate the quality of their product (c.f. Barbour 2001).
Secondly, the authors rightly point out that the science of reviewing qualitative research is in its
infancy and that significant obstacles to synthesizing such knowledge exist. The problem of unfo-
cused reviews could be overcome by using clinical rather than research questions as the basis for
reviews. Focused clinical questions can be both a useful filter for the kinds of research material sought
and increasing end-user relevance. Locating studies of qualitative research is difficult, but tools such
as the qualitative search filter at the University of Rochester (http://www.urmc.rochester.edu/miner/
educ/ebnfilt.htm) are making the task easier. However, research needs to be carried out ascertain the
sensitivity and specificity of such search tools. On the difficulties involved in establishing criteria for
inclusion in reviews of qualitative research I would concur. The task here is for primary researchers to
develop and report the choices they make at the sampling stage. It is when one attempts to critically
appraise qualitative research, however, that the conflict between the needs of decision makers and the
qualitative research community reaches its zenith. Scholars may have the luxury of being able to debate
the merits and demerits of various quality criteria, but clinician systematic reviewers are not so lucky.
However, there are various quality appraisal tools in existence which can at least make the criteria used
for appraisal transparent. The Critical Appraisal Skills Programme (CASP) qualitative checklist is a
good example (http://www.publichealth.org.uk/casp/qualitative.html). As long as readers of reviews can
identify the choices made then where is the problem? Whilst (imperfect) reviews are being undertaken,
the scholars can then busy themselves with refining the science of appraisal, and reviewing qualitative
evidence will develop incrementally. Other pragmatic responses to the authors concerns could be offered
but the take home message, for this reader at least, should be that efforts at reviewing and synthesizing
qualitative evidence should continue apace. In this way raw materials for the research community to
scrutinize and improve upon will be generated.
This is an important and cogent paper, offering a brief tour of some of the major dilemmas in this
area of research synthesis. The end result of this overview is a solid foundation for the development of
further (empirically informed) debate on this topic.
118 Clinical Effectiveness in Nursing

REFERENCES

Barbour RS. Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ 322:
1115–1117.
Murphy E, Dingwall R, Greatbatch D, Parker S, Watson P. Qualitative research methods in health technology
assessment: a systematic review of the literature. Health Technology Assessment 1998; 2(16).

Carl Thompson
doi:10.1054/cein.2001.0224, available online http://www.idealibrary.com on

D. Evans and A. Pearson Clinical Effectiveness in Nursing (2001) 5, 111–119

Systematic reviews of qualitative research

This paper is a timely attempt to examine the place of systematic reviews in qualitative research.
Certainly the possibilities of such reviews are not currently widely recognized. However, there are a
number of issues I would raise for the authors to think about.
The first point, only partly addressed by Evans and Pearson in their conclusion, is that the whole
point of qualitative research is to dig deep, to examine rich detail and to gain insight into unique human
experiences. To collect, aggregate and summarize studies which report these things must necessarily
take away from these aims and I can even imagine reviewers counting instances of this of that category,
concept or phenomenon in complete contradistinction to the true aims of most, if not all qualitative
enterprise. Whilst such review activities might have some basic value as background, the methodological
fetishism which some researchers have adopted would certainly be counterproductive in this case.
My second point is that systematic reviews are generally much less critical than they might at first
seem. The mass aggregation of outcomes and bare facts from large numbers of studies defeats the
possibility of appropriate criticism except in the most abstract of terms. The way to analyze and criticise
research papers is to allocate the process a lot of words for each criterion against which the study is
to be evaluated. To come to these criteria, Evans and Pearson give a good summary of those in some
use today and they recognize that many are unsatisfactory. It seems to me that few of those criteria
mentioned address the real issues. Why ‘primary researcher were nurses’ (Bunting 1997) is important is
hard to see; the important criteria (applicable to all forms of qualitative research) and made reasonably
clear by Martin Hammersley (1990) are validity, relevance and importance. Hammersley shows that
evaluation of qualitative research requires analytic and critical depth of a type not frequently seen in
the nursing research literature and unlikely to prevail in the ‘systematic review’ as described here.
One particular aspect of assessing validity is the approach to sampling. Here I mean not that of the
original papers reviewed, but the selection of papers for inclusion. Evans and Pearson seem to feel that
such a thing as a ‘representative sample’ would be possible, but make no suggestions as to how this will
be achieved. Typically, though thankfully not here, randomness is brought into play as a mechanism for
the achievement, or at least the assumption, of representativeness of the field. It will be important to lay
this particular myth to rest early on in the evolution of qualitative systematic reviews, since random or
other chance methods for inclusion would in all probability have left out the most important works of all.
My final concern in this vein is in the suggestion, not contradicted by Evans and Pearson, that it
will be in some way wrong to mix studies which used different methods and perspectives in systematic
reviews. My views on this are put in much greater depth elsewhere (Johnson, Long and White 2001), but
sufficient to argue that my associates and I have found virtually no qualitative research which follows
exactly the prescriptions and procedures of the various exponents of ‘tight’ methods.
Methodological purity is a myth and so there surely can be little wrong in comparing and contrasting
outcomes from the various forms of qualitative research, if it can be established that the review has
value in the first place.
To conclude, the paper makes a valuable contribution to the evaluation of the place of systematic
reviews of qualitative evidence. It may be, however, that the goal of answering specific questions is
misplaced. Rather than providing answers, in my view such reviews will refine old questions and
hopefully ask new ones, surely, a much more important and interesting aim.

REFERENCES

Hammersley M 1990 Reading ethnographic research: a critical guide. Longman, London

Johnson M, Long T, White A 2001 Arguments for ‘British Pluralism’ in qualitative health research. Journal of
Advanced Nursing 33(2): 243–249

Martin Johnson
doi:10.1054/cein.2001.0223, available online http://www.idealibrary.com on
 Systematic reviews of qualitative research 119

Author’s response
The authors would like to thank the two commentators for their response to this discussion on the
conduct of systematic reviews of qualitative research. The commentators have raised many worthy
points and have presented a balanced and thought-provoking discussion. The authors agree with these
commentaries, and would like to highlight the need for an informed public debate on this important
issue.

David Evans
doi:10.1054/cein.2001.0222, available online http://www.idealibrary.com on