This article was downloaded by: [PERI Pakistan]

On: 22 April 2010

Access details: Access Details: [subscription number 778684090]
Publisher Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 3741 Mortimer Street, London W1T 3JH, UK

International Journal of Social Research Methodology

Publication details, including instructions for authors and subscription information:

http://www.informaworld.com/smpp/title~content=t713737293

Paradigm Peace and the Implications for Quality

Alan Bryman

To cite this Article Bryman, Alan(2006) 'Paradigm Peace and the Implications for Quality', International Journal of Social

Research Methodology, 9: 2, 111 126

To link to this Article: DOI: 10.1080/13645570600595280
URL: http://dx.doi.org/10.1080/13645570600595280

PLEASE SCROLL DOWN FOR ARTICLE

Full terms and conditions of use: http://www.informaworld.com/terms-and-conditions-of-access.pdf
This article may be used for research, teaching and private study purposes. Any substantial or
systematic reproduction, re-distribution, re-selling, loan or sub-licensing, systematic supply or
distribution in any form to anyone is expressly forbidden.
The publisher does not give any warranty express or implied or make any representation that the contents
will be complete or accurate or up to date. The accuracy of any instructions, formulae and drug doses
should be independently verified with primary sources. The publisher shall not be liable for any loss,
actions, claims, proceedings, demand or costs or damages whatsoever or howsoever caused arising directly
or indirectly in connection with or arising out of the use of this material.

Int. J. Social Research Methodology

Vol. 9, No. 2, April 2006, pp. 111126

Paradigm Peace and the Implications

for Quality

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

Alan Bryman

In this article it is shown that the paradigm wars that raged concerning the incompatibility
of quantitative and qualitative research have largely subsided. In the process, discussions of
epistemological and ontological issues have become less prominent. The peace that has
broken out has proved to be more favourable to research combining quantitative and qualitative research than was the case during the paradigm wars. Drawing on interviews with
social researchers who employ a mixed-methods approach and on the literature, it is shown
that a spirit of pragmatism with regard to combining quantitative and qualitative research
prevails which encourages researchers to consider using mixed-methods research when the
research question is suited to it. However, the issue of which quality criteria should be
employed in investigations combining quantitative and qualitative research has not been
given a great deal of consideration. The author argues for a contingency approach, in which
issues to do with quality are decided in relation to the nature of the study.
International
10.1080/13645570600595280
TSRM_A_159511.sgm
1364-5579
Original
Taylor
9202006
ab302@leicester.ac.uk
AlanBryman
00000April
and
&
Article
Francis
(print)/1464-5300
Francis
Journal
2006 Ltd
of Social(online)
Research Methodology

Introduction
The debate about quantitative and qualitative research has oscillated between philosophical and technical levels of discussion (Bryman, 1984). At the former level,
authors have drawn attention to the different epistemological and ontological
assumptions underpinning quantitative and qualitative approaches to social research
and the research methods with which they are associated. Such discussions are often
construed in terms of a clash between positivism or post-positivism, on the one
hand, and a broadly interpretivist approach, founded on such traditions as phenomenology, symbolic interactionism, hermeneutics and Verstehen, on the other. At the
Alan Bryman is Professor of Organizational and Social Research in the Management Centre, University of Leicester, England. His main research interests are in research methodology and leadership studies. He is the author of
Social Research Methods (Oxford University Press, 2001, 2004). Correspondence to: Alan Bryman, Management
Centre, Ken Edwards Building, University of Leicester, University Road, Leicester LE1 7RH, UK. Tel.: +44 (0)116
252 2790; Email: ab302@leicester.ac.uk
ISSN 13645579 (print)/ISSN 14645300 (online) 2006 Taylor & Francis
DOI: 10.1080/13645570600595280

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

112

A. Bryman

technical level, quantitative and qualitative research refer to collections of techniques.

Technique most often refers to the methods employed for collecting data and to the
approach taken to the analysis of data, but could be taken (and indeed has been
taken by some writers) to include other aspects of the research process, such as
sampling.
The significance of the distinction between philosophical and technical levels of
discussion is that they represent contrasting perspectives in connection with the
feasibility of combining the two approaches. Typically, writers who emphasize
epistemological issues have depicted quantitative and qualitative research as based on
incompatible principles, and therefore as not capable of being combined. Any apparent
combination tended to be portrayed as superficial and therefore as not constituting a
genuine rapprochement (e.g. Smith & Heshusius, 1986). By contrast, the technical
version of the account of the differences between quantitative and qualitative research
typically allowed for, and indeed frequently encouraged, the merging of the two
approaches within single research projects.
In this article, I examine some of the literature concerning the debate about quantitative and qualitative research and in particular the suggestion in several quarters that
the cleavages between them have been bridged, so that investigations that combine
them become feasible. I then examine some of the implications for quality criteria of
the dtente that has largely replaced the antimonies. Specifically, given that it is sometimes suggested that quantitative and qualitative research should be judged according
to different criteria, what are the implications of quality criteria for studies that
combine the two approaches?
Research Methods
Side by side with this examination of the literature, I present the views of 20 social
researchers working in the UK whom I interviewed in 2004. These researchers are
essentially a purposive sample generated from an examination of articles published in
books and journals during the period 19942003. The interviews were conducted with
a semi-structured interview guide. The interviews were concerned with the researchers
views on and practices in relation to the integration of quantitative and qualitative
research. The typical interview lasted around 45 minutes. Interviewees varied between
senior figures in the field and relatively new researchers. They conducted their research
in at least one of the following fields: sociology, social psychology, human and cultural
geography, media and cultural studies, and organization studies.
In addition, I draw very briefly upon a content analysis of articles using both
quantitative and qualitative research in refereed journals. I searched the Social Sciences
Citation Index (SSCI) for articles in which relevant key words or phrases such as quantitative and qualitative, multi(-)method, mixed method or triangulation
appeared in the title, key words or abstract. The sample therefore comprises articles
which to some degree foreground the fact that the study is based on both quantitative
and qualitative research. Searches using other kinds of key words, such as survey and
ethnography/ic, produced a far larger sample of articles than could be dealt with

International Journal of Social Research Methodology 113

within the purview of this investigation. In conducting the search, the same five fields
of social research as those referred to in relation to the interviews were the focus for the
searches. The analysis was restricted to 19942003. The fact that the findings are based
on a large corpus of articles suggests that the sample is unlikely to be overly unusual,
although it is impossible to make any claims of representativeness. Judgments about
whether an article was relevant to the investigation, in terms of whether it could be
regarded as deriving from one of the five fields previously mentioned, were made on
the basis of the journal title or information supplied in abstracts. In this way, a total of
232 articles was generated and content analysed.

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

The Paradigm Wars

The debate about quantitative and qualitative research at the epistemological level
became known in some quarters as the paradigm wars. The term stemmed from a
perception of quantitative and qualitative research as distinct and to a large extent
competing paradigms based on fundamentally different principles. The use of the term
paradigm was frequently taken to entail, following Kuhn (1970), the implication that
the two approaches cannot be combined because paradigms are incommensurable.
Interestingly, the terms of this debate were, to a large extent, set by qualitative researchers; quantitative researchers tended not to become entangled in the philosophical
distinctions that were being demarcated. To the extent that others, such as methodologists, became embroiled in the tussles, it was largely in terms of the battle lines drawn
up by qualitative researchers. One possible reason for qualitative researchers being the
main contributors to the epistemological version of the debate is that drawing out a
distinctive philosophical position for qualitative research gave an intellectual rationale
for their preferred approach in the face of the hegemony of quantitative investigations
in journals and elsewhere. Alternatively, it may be that the growth of interest in
qualitative research grew out of an awareness of philosophical positions that were
marginalized by the quantitative research orthodoxy.
Paradigm Peace
To a very large extent, the paradigm wars can be considered over and peace can be
regarded as having broken out. There are occasional skirmishes, as authors occasionally
revive the old debates, perhaps placing them in a new context or with a fresh gloss. For
example, in the field of educational research, there has been some concern about
attempts emanating largely from policy makers and others concerned about policy issues
to introduce standards for the conduct of investigations that are predominantly associated with quantitative research (Hodkinson, 2004). By implication, such a position is
inimical to qualitative research and has prompted debates about the desirability of such
standards and about what the criteria should be. This is also a field where skirmishes
still occur (e.g. Smith, 1997), but it is also clear that not all of those who work in this
field are participants in the battles (e.g. Rocco et al., 2003). Also, while classic statements
of the epistemological divisions are often cited (e.g. Guba & Lincoln, 1982; Smith &

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

114

A. Bryman

Heshusius, 1986), particularly in terms of their arguments regarding the incompatibility

of quantitative and qualitative research, such references are usually used to provide a
backcloth to the newer thinking that allows for and even promotes mixed-methods
research. Thus, the editors of a handbook concerned with mixing methods state confidently that the incompatibility thesis has now been largely discredited (Teddlie &
Tashakkori, 2003, p. 19). It could be argued that one problem with this characterization
is that it fails to reflect fully the diversity of positions among qualitative researchers
(Denzin & Lincoln, 2005). These positions vary in their stances towards, and preparedness to accommodate, quantitative research. However, in general there is a perception
that by and large quantitative and qualitative research can be meaningfully integrated.
Further evidence on this point derives from the previously mentioned content analysis of journal articles. One of the dimensions on which these articles were coded was
in terms of whether there was any reference to paradigm issues. The criteria for an article to be deemed to have made a reference to paradigm issues were extremely nominal:
all that was needed was for the author(s) to have referred to the possible relevance of
epistemological or ontological issues or paradigm conflicts to the combined use of
quantitative and qualitative research, even if the reference was one that entailed a
dismissal of the relevance of such issues. In other words, the reference to the possible
relevance of such philosophical issues to the combination of quantitative and qualitative research could be one which portrayed those issues in a positive or a negative light.
In fact, such issues were addressed in just 14 (6 per cent) of the 232 articles. Again, this
finding suggests that at least so far as practising researchers are concerned, paradigm
wars issues have little if any relevance to their work.
The view that quantitative and qualitative research can no longer be regarded as
incompatible is significant for three reasons. First and most obviously, it removes any
lingering doubts concerning whether it is intellectually legitimate to integrate the two
approaches. Second, the compatibility view marginalizes the epistemological issues and
concerns that were at the heart of the paradigm wars, though, as will be shown, that is
not to suggest that philosophical issues disappear completely. Third, the view that
quantitative and qualitative research can be combined tended to be associated with an
uncoupling of research methods from philosophical positions. Writers often argued
that the association of particular methods with philosophical stances was based on
convention and that methods are in fact independent of epistemology (e.g. Bryman,
1988) or that the connections between them were more contingent than is often
assumed (e.g. Hammersley, 1992, p. 142).
Researchers in applied fields such as evaluation research and nursing have been in
the vanguard of the paradigm peace movement. For example, Twinn writes that the
nursing literature generally has accepted the idea of mixed methods research and that
the paradigm wars have been resolved within nursing (2003, p. 549). As Tashakkori
and Teddlie put it: Most investigators using these methods have not been interested in
delving deeply into the philosophical orientations that supposedly underlie the
application of their research studies (2003a, p. x).
Moreover, there may be a cultural side to the issues. Writing as the editor of a new
journal, Evaluation, Elliot Stern noted in connection with his editorial board:

International Journal of Social Research Methodology 115

There were those present, probably more from among North American participants than
others, who believed that the paradigm wars were over. Not everyone was convinced,
however. Continuing appeals to the positivist, constructivist and scientific realist canon
were noted (Stern, 1995, pp. 78)

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

However, the position may well be changing from that identified by Stern. I asked my
interviewees, all of whom were working in the UK and none of whom were in applied
fields like evaluation research, whether they felt the view that quantitative and qualitative research cannot or should not be combined because they derive from different
epistemological and ontological positions concerned them in their own research.
Virtually all of my interviewees indicated that such issues did not affect them in their
research. In other words, they were not concerned about the possibility that their use
of mixed-methods research transgressed philosophical principles. Examples of their
views on this issue are:
No, not at allI mean it doesnt. Ive just decided that thats not aI mean Im not really
prepared to worry about it, it does seem to me, from my experience er that you need to
useto use both, and if you have an epistemology that says you can only use one, then
thats a very narrow and pointless epistemology because clearlyclearly you can generate
understandings from different kinds of methods. (Respondent 1)
Er, no, not at all. (laughter) Im aware of Smith and Heshusius and you knowI just
cant be bothered with that sort of attitude really. (laughter) I dont really want to
think a lot about philosophy, you know (yeah). I feel its a separatea somewhat separate
realm. Erm, but I dont think you have to subscribe to all the the sort of foundation or
philosophical assumptions in order to do exciting and interesting things in your research
practice, which I think is a sort of quasi autonomous realm from philosophy and so on.
(Respondent 3)

Sometimes, the interviewees acknowledged that they may be transgressing fundamental epistemological issues, but felt that in order to get on with their research, they had
to sideline such concerns:
I mean I can see intellectually, that there are kind of contradictions there, erm, but II
guess I tend to do research in more practical terms and I think, you know, you can use the
data from contradictory, structured and contradictory kind of traditions, to shed light on
the same question. So Iit doesnt really bother me in the way that some purists might, I
guess, be less happy with it. (Respondent 16)
So I suppose what Im saying is I know that if Iwhen I go back and read those epistemological debates, I know that Im going to be reminded of the impossible chasm and I dont
believe in that impossible chasm in my own research practice Im afraid I put my head
in the sand. (Respondent 12)

Such comments display a lingering anxiety about the neglect of the philosophical
issues.
It could be argued that there is a certain inevitability about such views. After all,
researchers who have actually carried out investigations that bring together quantitative and qualitative research are less likely to fret about the epistemological niceties
involved. Indeed, one interviewee noted that some of her colleagues were opposed on
principle to the combination of quantitative and qualitative research, but also hinted

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

116

A. Bryman

that they were not empirical researchers. In other words, it is quite feasible that there is
a small cadre of social scientists for whom the paradigm wars have not been resolved.
These may be either those who are relatively uninvolved in empirical research or those
who maintain a principled objection to combining quantitative and qualitative
research in their investigations. What can be said is that this cadres voices cannot be
very loud, as those of the peaceniks are far more frequently encountered in the
literature.
Occasionally, the social researchers I interviewed gave an indication that they were
not totally at ease with the way in which they had been forced (or felt forced) to sideline
the epistemological debates. Such debates were felt by a few interviewees to be both
interesting and unresolved, and it was felt that it was a shame that the philosophical
issues had been forced into a realm that had become divorced from their own research
practice. They felt that the issues should not be lost, as suggested by the following
comment: Erm, but an intellectual familiarity with the broad problems erm is
absolutely indispensable, I think (Respondent 5).
However, the interviewees typically saw themselves as needing to ignore such issues
to get on with the kind of research they wanted to do or as somehow transcending the
debates, as suggested by the following remark:
Yeah, well certainly I think ones got to keep asking oneself those questions all the time and
the discipline Im inis interesting from that point of view, because it does combine
people who are at both ends of the spectrum, in terms of the work they do and others, like
myself, who are in the middle. (Respondent 11)

By and large, in spite of occasional glimmers of discomfort, my interviewees saw

themselves as having to disregard the philosophical issues that were a feature of some
versions of the debate about quantitative and qualitative research.
The Rise of Pragmatism
Research combining quantitative and qualitative research has become imbued with an
overwhelmingly pragmatic attitude. In some versions of this attitude, the notion of
pragmatism is being construed with reference to pragmatism as a distinctive philosophy of the kind with which Charles Peirce is associated. In other versions, it is less to
do with an allegiance to a distinct set of philosophical principles than a generalized
commitment to needing to conduct research that will answer certain kinds of research
questions. The specific issue of the role of research questions will be returned to below.
The apparent commitment to pragmatism has become a way of rationalizing the
conjoint use of quantitative and qualitative research while simultaneously recognizing
that there have been debates about their supposed epistemological incompatibility.
Maxcy suggests that pragmatism seems to have emerged as both a method of inquiry
and a device for the settling of battles between research purists and more practicalminded scientists (2003, p. 79). Even when pragmatism is not explicitly mentioned, its
broad principles can be detected in remarks such as: there is growing acknowledgment
that complex social phenomena can usefully be understood by looking at them both

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

International Journal of Social Research Methodology 117

quantitatively and qualitatively (Rossman & Wilson, 1994, p. 315). Smith recalls being
told by the editor of a major educational research journal that most researchers had
become bored with philosophical debates and were more interested in getting on
with the task of doing their research (1996, pp. 162163). Thus, in place of the primacy
of philosophical principles that was associated with the era of the paradigm wars,
pragmatism is taken to advocate the pre-eminence of technical decisions about the
appropriate use of different methods (either singly or in tandem with other methods
regardless of whether they are quantitative or qualitative ones) according to particular
circumstances.
The adoption of disciplined pragmatism was also discernible in the interviews that
I carried out. Respondent 17 said: Erm, to some extent. I mean I think thatsyeah,
there are major theoretical questions there, you know, but in the end I take a more
pragmatic stance. In similar fashion, Respondent 16 replied: Erm, so I guess I dont
tend to think of it as aat a philosophical level, I tend to think of it in terms of the
outcomes and what you can do with what comes out of it. I mean Im a really pragmatic academic. Respondent 15 took a similar view: So its a fairly pragmatic
approach really. I mean I suppose I dont feel dogmatically signed up to one epistemology either, you know. Respondent 18 expressed a similar view regarding an investigation in which he was then involved: So weve taken that pragmatic decision to do it
that way because thatll generate something that either method, standing alone, is not
gonna give us.
Often, the need for pragmatism was justified in terms of the requirement to put
aside the intellectual debates in order to apply for research grants from funding bodies.
Debating the niceties of epistemological positions was seen as inconsistent with the
need for a fairly hard-nosed approach to getting financial support for ones research.
For example, Respondent 14 said: because I suppose here, traditionally, weve been
you know a self-funding, entrepreneurial centre as we were, theres always been so
much more of a pragmatic approach to doing things and perhaps there isnt in other
departments. Respondent 6 took the view:
Im much more focused on the pragmatic issues of designing a research design which, in
some ways, is strong enough to overcome thatthat gatekeeper who will referee the
proposal I guess. Its become much more strategic

He went on to add later in the interview: Erm, and again its probably thebeing very
pragmatic, that in order to get funding from x or y, you need to fall into line to their
expectations as to what is a good research design
Even when the term pragmatism was not employed, its influence was frequently
detectable. For example, Respondent 9 replied:
To me, the answersis much more an answer ofof just attempting to better understand
what it is youre trying to understand, and in that way, you then have to ask how appropriate are the sorts of methods Im using and are they going to give me the information to
understand what it is Im researching?

In suggesting that such views are indicative of pragmatism, regardless of whether the
term is mentioned, it has to be recognized that the term is typically being employed in

118

A. Bryman

a general way, rather than as a commitment to a clearly defined philosophical position. As such, pragmatism tends to denote a no-nonsense practical approach to
research.

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

The Centrality of the Research Question

One of the chief manifestations of the pragmatic approach to the matter of mixing
quantitative and qualitative research is the significance that is frequently given to the
research question. In the context of the debate about quantitative and qualitative
research and the matter of mixing the two approaches, two issues follow from this.
First, it is frequently suggested, in the new climate of pragmatism, that issues to do with
the adequacy of particular methods for answering research questions are the crucial
arbiter of which methodological approach should be adopted rather than a commitment to a paradigm and the philosophical doctrine on which it is supposedly based. For
example, Teddlie and Tashakkori write that: Pragmatist researchers consider the
research question to be more important than either the method they use or the
paradigm that underlies the method (2003, p. 20). Similarly, Erzberger and Kelle argue
that: The selection of adequate methods should not be made mainly on the basis of
sympathies toward a certain methodological camp or school. Methods are tools for the
answering of research questions and not vice versa (2003, p. 482). This position with
regard to the debate about quantitative and qualitative research prioritizes the research
question and relegates epistemological and ontological debates to the sidelines. In
doing so, it clears the path for research that combines quantitative and qualitative
research.
Second, the primacy of the research question means that research that brings
together quantitative and qualitative research is not only feasible, but more importantly desirable or even required for answering certain kinds of research question or
combinations of research questions. A writer from the field of population geography
typifies this position:
It may seem common sense to suppose that any choice of research method must be appropriate to the particular research problem in hand and the data available for addressing that
research problem. The notion that we must adopt ways of doing research best suited to the
research question and consider all methods available is not uncommon. On this view, a
multi-method research design may be appropriate to some research projects but not to
others, with appropriateness being judged in relation to the nature of the research question
and the sources of information we have at our disposal to answer that question. (Graham,
1999, p. 78)

Teddlie and Tashakkori put the matter simply: Mixed methods research can answer
research questions that the other methodologies cannot (2003, p. 14).
Similar positions were also very evident among a few of my interviewees. Respondent
16 put it this way:
Erm I dont really have a kind of problem, with a kind of clashing philosophy of using all
kinds of so-called scientific method and a more qualitative approach. I mean I can see
intellectually, that there are kind of contradictions there, erm, but II guess I tend to do

International Journal of Social Research Methodology 119

research in more practical terms and I think, you know, you can use the data from
contradictory, structured and contradictory kind of traditions, to shed light on the same
question. So Iit doesnt really bother me in the way that some purists might, I guess, be
less happy with it.

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

Respondent 6 felt that it was crucial to ask is that the appropriate research design for
this research problem or this research issue?.
A variation of this kind of argument was to suggest that mixing quantitative and
qualitative research was only acceptable if it was relevant to the research question in
which one was interested. In other words, my interviewees, all of whom were users of
a mixed-methods research approach, were often keen not to present it as a panacea or
themselves as its unequivocal advocates. Their view of mixing quantitative and qualitative research was typically that it should be viewed as something that can be adopted
to fit some research questions but not others. While discussing mixed-methods
research, Respondent 2 said:
I think itsImean, I dont think its necessarily a good or a bad thing, I think it depends
on how you want to access research questions and answer them and whether its appropriate
for the particular questions that you have.

Respondent 4 suggested:
So effectively I would say its task driven and its what you might call methodologically
appropriate that Im concerned with, and I dont like what I would call ideological
commitments to any particular kind of methodology.

Respondent 14 maintained that youve got to have a completely clear idea in your head
about why youre combining them .
These views carry the suggestion that mixed-methods research is helpful for answering certain kinds of research question. By implication, some research questions or
combinations of research questions can only be answered by combining quantitative
and qualitative research.
Implications for Quality
In this section, I consider the implications of paradigm peace and the issues addressed
in the previous sections for the matter of the quality of research that combines quantitative and qualitative approaches. Two main areas are considered: the issue of the
appropriateness of a mixed-methods approach and the extent to which there should be
alternative criteria for such research.
The Appropriateness of Mixing Quantitative and Qualitative Research
In some of the comments by the interviewees can be detected the glimmer of a concern
about quality. For example, Respondent 2 argued:
I dont think any is gonna be better quality than anything else, it depends on have you
found a way of interrogating your research questions thats going to give you in-depth

120

A. Bryman
insights. And you can do that quantitatively, qualitatively or using both. So I wouldnt say
that any one is particularly better than any other really.

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

Interviewees frequently expressed a concern that mixed-methods research should not

be seen as a universally superior or appropriate strategy. Two kinds of reservations
about the appropriateness of mixed-methods research and the possibility that quality
may be jeopardized could be discerned in interviewees remarks.
First, there was a concern that, following a period when quantitative and qualitative
research were seen as irreconcilable, combining them was increasingly being seen as
fashionable. This was viewed as raising the prospect of researchers seeking to combine
quantitative and qualitative research in ways that may not be appropriate to the
research question(s). Respondent 1 maintained:
Erm, yeah, so I do think its a bit of a fashion er I dont think its always necessary. I often
look at research proposals that Im reviewing and think you dont need this bit, you dont
need this bit, its a waste of your time, why is it there? You know, concentrate on the 20
interviews, in-depth interviews and find outyou can answer all the questions you want
from that, why bother with all the rest of it? Erm, yeah.

Respondent 6 observed:
theres now expectation that, to have aa research design which is gonna be robust, which
is going to stand up to critique and which is representativeof whatever, whatever that
may be, you need to adopt a mixed method approach. And that isnt necessarily the right
way to approach a particular research problem because it might not be the right approach
at all. I think people are pigeon holed or you pick, as I said, youyou have to, the structure
pushes you to making decisions about picking off, picking off the shelf the tailor made
mixed method approach and it might not necessarily be appropriate. But because you
almostyoure not obliged to do it but you feel constrained that you have to do this
approach to convince the gatekeepers and itsas you said, I think, you know, there are
expectations there. And of course, bodies like the Economic and Social Research Council,
you knowin some ways their quantitative turn over theover recent times, you know,
reproduces that mixed method approach.

Such comments imply a concern that mixed-methods research has become a fad and
that there is perception that it is more likely to be favoured by funding bodies, who
themselves are implicated in that fad, because it offers the best of all worlds. As
Respondent 2 succinctly put it: I think the disadvantages arise from when people just
do it because its expected or do it without thinking about how it relates to the research
questions. The problem that interviewees are expressing is that it may result in quantitative and qualitative research being combined regardless of the research question.
Second, the concern that the rationale for mixing quantitative and qualitative
research may not be fully thought through, because it is sometimes being employed
regardless of the research question, was seen by several interviewees as having implications for the quality of mixed-methods research. In particular, it was argued by several
interviewees that the quality of mixed-methods research is diminished if it is not
soundly grounded in a clear sense of purpose. For example, Respondent 7 suggested:
And I suppose if you had a kind of evangelistic belief, which I dont quite have, I wouldnt
go quite that far but you know, if you were really a fervent advocate of multiple methods,

International Journal of Social Research Methodology 121

regardless of the research problem and you always applied them, then that would really be
quite inappropriate.

Respondent 8 provided a similar position:

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

So, the combination of methods, I think is reallyit needs to be seen in the light of answering the particular research problem and just throwing all the methods in together is a
recipe for disaster, I think. Im sure theres probably projects which have tried to do that.

One interviewee felt that if, in order to implement a mixed-methods research

approach, researchers were having to conduct research using methods with which they
were unfamiliar (and perhaps unsympathetic to), the quality of the ensuing research
might jeopardized. Similarly, Respondent 7 averred that with mixed-methods research
theres more opportunities to make bad mistakes. Indeed, the problem of skills was a
recurring motif, with many interviewees voicing a concern that the training and skills
of researchers (including their own) were often a barrier to combining quantitative and
qualitative research. It was felt that too often researchers are skilled in and comfortable
with one approach rather than the other. Respondent 17 expressed a related view while
talking about some classic qualitative studies in his field:
you know, those are such sensitive studies, the meanings, the subtle fluid meanings of
peoples lives, that if youd have asked those researchers to be quantitative as well, I dont
think it would have been as high quality, you know. If it was somehow a requirement of
any study that it included both, I think it would have detracted because those people, doing
it by themselves, would have lost some bitsanalytical power and excitement.

On the other hand, there was a prevalent feeling that all things being equal, mixedmethods research would deliver quality research. As Respondent 14 succinctly put it:
it does depend on the parameters of the research and on the aims and objectives erm and
how theyre set and how you relate to them. But, instinctively Id say yes it wouldit
would provide better quality . I think there are certain things that you can learn by
adopting different approaches, you know

Thus, for most of the interviewees, mixed-methods research is not intrinsically superior in terms of quality to mono-method research, and there are even suggestions that
its quality may be endangered as a result of a slavish devotion to fashion or to a (possibly erroneous) belief that it is favoured by research funding bodies.
Quality Criteria for Research Combining Quantitative and Qualitative Research
One of the most significant developments in research methodology, as qualitative
research has become more accepted and has developed a momentum of its own, is the
focus upon quality criteria (Seale, 1999). Qualitative researchers have expressed the
concern that traditional criteria, that is, those associated largely with quantitative
researchreliability, validity, generalizability, replicability etc.are inappropriate to
their approach. There have been two main approaches to addressing this issue. One is
for qualitative researchers to adapt traditional criteria (e.g. LeCompte & Goetz, 1982).
The other is to develop alternative criteria. In the process, there has been a proliferation

122

A. Bryman

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

of different ways of expressing quality criteria for qualitative research (e.g. Lincoln &
Guba, 1985; Spencer et al., 2003; Yardley, 2000). There is also a prior issue of what the
question of quality relates to, that is, quality of what? By and large, assessments of quality tend to relate to the overall research process from formulation of research questions
to the generation of implications for theory. As such, it encapsulates all aspects of an
investigation, including problem formulation, sampling, research instruments, analysis and inferences.
What are typically given less attention in the course of these discussions are quality
criteria in relation to research that includes both quantitative and qualitative
approaches. In fact, Sale and Brazil (2004) concluded, following an examination of
appraisal criteria through journal databases, that no criteria with respect to mixedmethods studies could be found. There are several ways of conceiving of the ways in
which quality criteria should be used in mixed-methods research. The following might
be envisaged:
(1) Convergent criteriause the same criteria for both the quantitative and the
qualitative components of the research.
(2) Separate criteriause separate criteria for the quantitative and qualitative
components.
(3) Bespoke criteriadevise new criteria specifically for mixed-methods research.
One problem with all three of these solutions is that they are likely to generate controversy that may hinder rather than help the development of mixed-methods research.
Such a prospect is quite feasible in the light of the substantial and sometimes heated
arguments surrounding quality criteria for qualitative research (Sparkes, 2001). Only
one of the 20 interviewees raised the issue of criteria in relation to mixed-methods
research, implying that they have not become a major issue yet for practitioners of the
approach. Respondent 18 noted in connection with the analysis of both quantitative
and qualitative data that it can be very difficult to synchronize the two because of the
different standards of evidence youre applying and rigour. However, it may not be
desirable to think in universalistic terms when discussing or using quality criteria and
instead a contingency position might be more appropriate. Each of the three
approaches may be suitable in some circumstances but not others. For example, in
mixed-methods research in which qualitative research is used to generate hypotheses
or to assist with devising measurement devices for quantitative research, the investigation is operating more or less exclusively within a quantitative research framework. In
these circumstances, convergent criteria entailing the use of the traditional criteria of
reliability, validity and generalizability may be more appropriate than either separate
or bespoke criteria. Similarly, when quantitative research is being employed simply to
collect background data within a predominantly qualitative research project or when
the amount of quantitative data is minimal, criteria associated with interpretivist or
constructionist ideas, such as trustworthiness criteria (Lincoln & Guba, 1985), could be
more appropriate than either separate or bespoke criteria.
Bespoke criteria might appear to be the ideal solution insofar as they either fuse the
different quality criteria associated with quantitative and qualitative research or entail

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

International Journal of Social Research Methodology 123

the development of criteria that transcend these. However, it is unlikely, given the
controversy that has surrounded criteria in qualitative research (e.g. Sparkes, 2001),
that, at least in the near future, criteria could be developed that would satisfy the qualitative research community. More fundamentally, it is being argued here that such
criteria may not necessarily be relevant to a large proportion of studies using both
quantitative and qualitative research, namely, ones in which either quantitative or
qualitative is a dominant approach. In such studies, the qualitative or the quantitative
component is a procedural adjunct to a dominant quantitative or qualitative approach
respectively. It is common for writers on mixed-methods research to distinguish
between studies in which either quantitative or qualitative research is dominant or in
which they are equal (e.g. Creswell, 2003), so that the view being presented here is
consistent with such a distinction.
A further issue concerns whether the quantitative and qualitative research components are genuinely integrated or whether they stand alone. Sometimes, mixed-methods research is conducted to explore very different research questions within an overall
project. When there is little integration of the different components, separate criteria
might be appropriate. Where there is a genuine integration of quantitative and
qualitative research, whereby the quantitative and the qualitative findings are systematically related to each other rather than allowed to stand alone as separate sets of
findings, a bespoke criteria approach, if one could be successfully produced given the
controversies involved, would be appropriate.
Tashakkori and Teddlie (2003b) have proposed an approach to bespoke criteria for
mixed-methods research that is based on the notion of inference quality, which is meant
to include both the traditional notion of internal validity and the notions of trustworthiness and credibility that are associated with qualitative research (Lincoln & Guba,
1985). Inference quality is to do with the issue of whether an inference meets the fundamental criteria required for it to be credible and/or defensible. This is an interesting
approach but it clearly only captures portions of the kinds of criteria that have come to
be associated with quantitative and qualitative research criteria. It does not, for
example, address issues to do with external validity/transferability (Lincoln & Guba,
1985). More fundamentally, it is being argued here that such a criterion may only be
required when there is no dominant quantitative or qualitative approach or when the
quantitative and the qualitative components relate to distinctly separate research questions. As such, a bespoke criteria approach of the kind that Tashakkori and Teddlie
offer would primarily apply only to those studies where there is genuine integration.
Sale and Brazil (2004) have also proposed bespoke criteria based on commonalities
among criteria associated with both quantitative and qualitative research. They
propose: truth value (quantitative research criterion of internal validity vs. qualitative
research criterion of credibility); applicability (external validity/generalizability vs.
transferability/fittingness); consistency (reliability vs. dependability); and neutrality
(objectivity vs. confirmability). Interestingly, however, these authors proffer a remark
that is very much in tune with the chief argument of this section: future guidelines for
criteria might be more specific depending on the mixed-methods design (Sale &
Brazil, 2004, p. 361). As such, bespoke criteria such as those offered by Tashakkori and

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

124

A. Bryman

Teddlie and by Sale and Brazil would be employed in the context of mixed-methods
investigations in which quantitative and qualitative research components are integrated, that is, where one is not a procedural adjunct of the other and the quantitative
and qualitative components are fully integrated. However, these considerations
essentially sidestep the issues of whether there is agreement over the content and
meaning of quality criteria for quantitative research and whether it is desirable to have
parallel criteria for qualitative research (quite aside from what the criteria should be).
One further issue that is likely to make the creation of criteria for mixed-methods
research difficult arises from the fact that the outcomes are often unplanned. Research
of all kinds has the capacity to offer surprising or unexpected findings, but when quantitative and qualitative research are combined the possibilities of unplanned or unanticipated outcomes are magnified considerably. For example, including both
quantitative and qualitative components in a study may result in a realization that it
offers the opportunity to compare findings in the spirit of triangulation (that is, to
cross-check quantitative and qualitative findings), even though the investigation was
not devised with triangulation in mind (Deacon, Bryman, & Fenton, 1998; Waysman
& Savaya, 1997). Alternatively, triangulation may have been the goal of combining the
two approaches, but other implications of putting the two sets of data together may
become apparent when the data are analysed (Bryman, 2006; Rocheleau, 1995). Such
unpredictability may pose problems to mixed-methods researchers seeking to specify
quality criteria. This may arise because the kinds of criteria foreseen at the outset may
appear less appropriate towards the later stages of an investigation when the implications of combining the quantitative and qualitative components appear to be different
from the original design.
Furthermore, several writers have observed that articles reporting the results of
studies mixing quantitative and qualitative research frequently provide little explicit
discussion of the rationale for using both approaches (Bryman, 2006; Niglas, 2004;
Rocco et al., 2003). Unless mixed-methods researchers become a great deal more
candid about such issues, it will be difficult to move forward the specification and
application of criteria for mixed-methods research.
Conclusion
I have sought to show that the so-called paradigm wars, which emphasized the epistemological and ontological differences between quantitative and qualitative research,
have been replaced by a period of paradigm peace. In this new era, there is a tendency
to stress the compatibility between quantitative and qualitative research and a pragmatic viewpoint which prioritizes using any approach that allows research questions to
be answered regardless of its supposed philosophical presuppositions. Within this more
accommodating climate, the research question and the appropriateness of particular
research methods or approaches to the research question become the hub for the possible integration of quantitative and qualitative research. The issue of which quality criteria should be applied to investigations that mix quantitative and qualitative research has
not been given a great deal of attention. I show that practitioners of mixed-methods

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

International Journal of Social Research Methodology 125

research are concerned that the approach should not be seen as a panacea for all
research problems. They feel that this could be to the detriment of the quality of some
mixed-methods research, since it would mean that the approach was sometimes being
applied with insufficient attention to research questions. I also consider different ways
in which quality might be assessed in studies combining quantitative and qualitative
research and it is suggested that a contingency position might be one way forward. This
means that rather than adopting one approach to assessing the quality of all mixedmethods research, it is suggested that the approach to considering quality issues should
be influenced by the nature and goals of the investigation. Probably the most difficult
kind of mixed-methods research to fit into this approach is that which fully integrates
the two approaches, since neither quantitative nor qualitative research would suffice
and there are few generally agreed criteria that transcend the quantitativequalitative
divide. The suggestions concerning inference quality provided by Tashakkori and
Teddlie (2003) represent a starting point in this connection but their work only deals
with a limited domain of issues. This is an area that will require further attention.
Acknowledgements
The author wishes to thank the three referees of this article for their constructive
comments. He also wishes to thank the Economic and Social Research Council for
funding the research project Integrating Quantitative and Qualitative Research:
Prospects and Limits (Award number H333250003) which made possible the research
on which this article is based.
References
Bryman, A. (1984). The debate about quantitative and qualitative research: A question of method or
epistemology? British Journal of Sociology, 35, 7592.
Bryman, A. (1988). Quantity and quality in social research. London: Routledge.
Bryman, A. (2006). Integrating quantitative and qualitative research: How is it done? Qualitative
Research, 6, 97113.
Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd
ed.). Thousand Oaks, CA: Sage.
Deacon, D., Bryman, A., & Fenton, N. (1998). Collision or collusion? A discussion of the unplanned
triangulation of quantitative and qualitative research methods. International Journal of Social
Research Methodology, 1, 4763.
Denzin, N. K., & Lincoln, Y. S. (2005). The SAGE handbook of qualitative research (3rd ed.). Thousand
Oaks, CA: Sage.
Erzberger, C., & Kelle, U. (2003). Making inferences in mixed methods: The rules of integration. In
A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral
research (pp. 45788). Thousand Oaks, CA: Sage.
Graham, E. (1999). Breaking out: The opportunities and challenges of mixed-methods research in
population geography. Professional Geographer, 51, 7689.
Guba, E. G., & Lincoln, Y. S. (1982). Epistemological and methodological bases of naturalistic
inquiry. Educational Communication and Technology Journal, 3, 233252.
Hammersley, M. (1992). The paradigm wars: Reports from the front. British Journal of Sociology of
Education, 13, 131143.

Downloaded By: [PERI Pakistan] At: 08:30 22 April 2010

126

A. Bryman

Hodkinson, P. (2004). Research as a form of work: Expertise, community and methodological

objectivity. British Educational Research Journal, 30, 926.
Kuhn, T. S. (1970). The structure of scientific revolutions (2nd ed.). Chicago, IL: University of Chicago
Press.
LeCompte, M. D., & Goetz, J. P. (1982). Problems of reliability and validity in ethnographic
research. Review of Educational Research, 52, 3160.
Lincoln, Y. S., & Guba, E. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.
Maxcy, S. J. (2003). Pragmatic threads in mixed methods research in the social sciences: The search
for multiple modes of enquiry and the end of the philosophy of formalism. In A. Tashakkori
& C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 5189).
Thousand Oaks, CA: Sage.
Niglas, K. (2004). The combined use of qualitative and quantitative methods in educational research.
Tallinn, Estonia: Tallinn Pedagogical University Dissertation on Social Sciences.
Rocco, T. S., Bliss, L. A., Gallagher, S., Perez-Prado, A., Alacaci, C., Dwyer, E. S., Fine, J. C., &
Pappamihiel, N. E. (2003). The pragmatic and dialectical lenses: Two views of mixed methods
use in education. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social
and behavioral research (pp. 595615). Thousand Oaks, CA: Sage.
Rocheleau, D. (1995). Maps, numbers, text, and context: Mixing methods in feminist political ecology. Professional Geographer, 47, 458466.
Rossman, G. B., & Wilson, B. L. (1994). Numbers and words revisited: Being shamelessly eclectic.
Quality and Quantity, 28, 315327.
Sale, J. E. M., & Brazil, K. (2004). A strategy to identify critical appraisal criteria for primary mixedmethods studies. Quality and Quantity, 38, 351365.
Seale, C. (1999). The quality of qualitative research. London: Sage.
Smith, J. K. (1996). An opportunity lost? In L. Heshusius & K. Ballard (Eds.), From positivism
to interpretivism and beyond: Tales of transformation in educational and social research
(pp. 1617). New York: Teachers College Press.
Smith, J. K. (1997). The stories educational researchers tell about themselves. Educational Researcher,
26, 411.
Smith, J. K., & Heshusius, L. (1986). Closing down the conversation: The end of the quantitative
qualitative debate among educational researchers. Educational Researcher, 15, 412.
Sparkes, A. (2001). Myth 94: Qualitative health researchers will agree about validity. Qualitative
Health Research, 11, 538552.
Spencer, L., Ritchie, J., Lewis, J., & Dillon, L. (2003). Quality in qualitative evaluation: A framework
for assessing research evidence. London: Government Chief Social Researchers Office.
Stern, E. (1995). Editorial. Evaluation, 1, 59.
Tashakkori, A., & Teddlie, C. (2003a). Preface. In A. Tashakkori & C. Teddlie (Eds.), Handbook of
mixed methods in social and behavioral research (pp. ixxv). Thousand Oaks, CA: Sage.
Tashakkori, A., & Teddlie, C. (2003b). The past and future of mixed methods research: From data
triangulation to mixed model designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook of
mixed methods in social and behavioral research (pp. 671701). Thousand Oaks, CA: Sage.
Teddlie, C., & Tashakkori, A. (2003). Major issues and controversies in the use of mixed methods in
the social and behavioural sciences. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed
methods in social and behavioral research (pp. 350). Thousand Oaks, CA: Sage.
Twinn, S. (2003). Status of mixed methods research in nursing. In A. Tashakkori & C. Teddlie
(Eds.), Handbook of mixed methods in social and behavioural research (pp. 54156). Thousand
Oaks, CA: Sage.
Waysman, M., & Savaya, R. (1997). Mixed method evaluation: A case study. Evaluation Practice, 18,
227237.
Yardley, L. (2000). Dilemmas in qualitative health research. Psychology and Health, 15, 215228.