You are on page 1of 47


Talk Outline

Clarifying Concepts Mixing Research Methods

Quantitative Qualitative Purposes Steps

Data analysis process

Components Data collection

Consensual Qualitative Research

Data analysis
Auditing Writing up Advantages

Mixing Research Approaches

Practitioner Research and Evaluation Skills Training in Open and Distance Learning (

Multi-method studies use different methods of data collection and analysis within a single research paradigm. Mixed method studies attempt to bring together methods from different paradigms.

Multi-method designs

Mixed methods designs

Use more than one method but restricted to methods selected from within one worldview (i.e. quantitative or qualitative approaches).

Use and mix both qualitative and quantitative data or methods.

Comparison of quantitative and qualitative research approaches

Purpose Design Approach Tools Sample Analysis


To study relationships, cause and To understand social phenomena effect Developed prior to study Deductive; tests theory Uses standardized measurement Evolves during study Inductive; generates theory Uses face-to-face interaction

Uses large samples Statistical analysis of numeric data

Uses small samples Narrative description and interpretation

Research Criteria
(Lincoln & Guba, 1985)



Generalizability (thru sampling) Reliability (replicable)

Objectivity (limits bias)

Credibility (believable) Transferability (thru thick description) Dependability (context of research) Reflexivity (examine own biases)

Why mix methods?

Mixed method research, apart from avoiding unimethod bias, capitalizes on the strengths of quantitative and qualitative methods to provide a more complete understanding of the research problem and/or address multiple questions (Johnson & Christensen, 2004; Wiersma & Jurs, 2005).

Why mix methods?

Both quantitative and qualitative methods have different but complementary roles to play in a research process and outcome (Sogunro, 2002, p. 3).

Purposes of Mixed Method

(Green, Caracelli and Graham,1989)

Triangulation Complementarity Development Initiation Expansion


seeks convergence, corroboration, correspondence of results from the different methods.


seeks elaboration, enhancement, illustration, and clarification of the results from one method with the results from the other method.


seeks to use the results from one method to help develop or inform the other method, where development is broadly construed to include sampling and implementation, as well as measurement decisions.


seeks the discovery of paradox and contradiction, new perspectives of frameworks, the recasting of questions or results from one method with questions or results from the other method.


seeks to extend the breadth and range of inquiry by using different methods for different inquiry components (p. 259).

Complementarity Initiation Triangulation Expansion Development

Fundamental Principle of Mixed Research (Turner, 2003)

Researchers should collect multiple data using different strategies, approaches, and methods in such a way that the resulting mixture or combination is likely to result in complementary strengths and nonoverlapping weaknesses.

7-stage mixed methods data analysis process (Onwuegbuzie and

Teddlie, 2003)

data reduction data display data transformation data correlation data consolidation data comparison and data integration

Data reduction

involves reducing the dimensionality of the qualitative data (e.g., via exploratory thematic analysis, memoing) and quantitative data (e.g., via descriptive statistics, exploratory factor analysis, cluster analysis).

Data display

Involves describing pictorially the qualitative data (e.g., matrices, charts, graphs, networks, lists, rubrics, and Venn diagrams) and quantitative data (e.g., tables, graphs).

Data transformation

wherein quantitative data are converted into narrative data that can be analyzed qualitatively (i.e., qualitized; Tashakkori & Teddlie, 1998) and/or qualitative data are converted into numerical codes that can be represented statistically (i.e., quantitized; Tashakkori & Teddlie, 1998).

Data correlation

involves the quantitative data being correlated with the qualitized data or the qualitative data being correlated with the quantitized data.

Data consolidation
Wherein both quantitative and qualitative data are combined to create new or consolidated variables or data sets.

Data comparison

involves comparing data from the qualitative and quantitative data source

Data integration

characterizes the final stage, whereby both quantitative and qualitative data are integrated into either a coherent whole or two separate sets (i.e., qualitative and quantitative) of coherent wholes.

Essential Components of CQR

open-ended questions in semistructured data collection techniques (typically in interviews), several judges throughout the data analysis process to foster multiple perspectives; consensus to arrive at judgments about the meaning of the data;

Essential Components of CQR

at least one auditor to check the work of the primary team of judges and minimize the effects of groupthink in the primary team; and domains, core ideas, and crossanalyses in the data analysis.


Is an integral part of the CQR method (Hill et al., 1997)


on mutual respect, equal involvement, and shared power (p. 523).

a diversity of viewpoints is valued, honored, and protected (Williams & Barber, 2004).

Expectations and Biases

Researchers should report both .. expectations (beliefs that researchers have formed based on reading the literature and thinking about and developing research questions, Hill et al, 1997, p. 538) and biases (personal issues that make it difficult for researchers to respond objectively to the data, p. 539)

Data Collection
Samples (8-15 participants) Interviews (8-10 scripted questions for 1 hour) Mode


telephone interviews, taped face-to-face interviews, paper-and-pencil survey format, and e-mail format

Data Analysis

Domains (i.e., topics used to group or cluster data) are used to segment interview data. Core ideas (i.e., summaries of the data that capture the essence of what was said in fewer words and with greater clarity) are used to abstract the interview data within domains. Cross-analysis - to construct common themes across participants (i.e., developing categories that describe the common themes reflected in the core ideas within domains across cases).

Auditors role is to check.

whether the raw material is in the correct domain, that all important material has been faithfully represented in the core ideas, that the wording of the core ideas succinctly captures the essence of the raw data, and that the cross-analysis elegantly and faithfully represents the data.


The auditor thus provides detailed feedback at each stage of the analysis process (e.g., creating domains, constructing core ideas, creating the crossanalysis).

Stability Check

Hill et al. (1997) recommend that after the domains and core ideas are completed for all of the cases, at least two cases be withheld from the initial cross analysis and then used as a check to determine whether all of the data for these cases fit into the existing categories and whether the designations of general, typical, and variant changed substantially with the addition of the two new cases.

Charting Results
Charting results depicts visually the relationships among categories across domains, particularly for data representing sequences of events (e.g., the process of resolving a misunderstanding). A criterion of at least three cases to establish each connection between domains in the pathway.

Establishing the Trustworthiness and Accuracy of the Data

use of participants to help assess the accuracy and trustworthiness of the data sometimes called member checking

Writing Up the Results and Discussion Sections

main purpose of the Results section is to communicate the results clearly and cogently to the audience. results and conclusions of the data analysis need to be logical, account for all the data, answer the research questions and make sense to the outside reader (Hill et al, 1997, p. 558).

Organizing Findings
according to their domains and categories, according to main groupings or clusters of the data, and according to research questions.

Organizing Findings
Presenting core ideas, Presenting participant quotes, and Using a combination of core ideas and quotes to exemplify the categories and subcategories either in the text or in tables.

Organizing Findings

fully and richly describe at least the general and typical categories and provide at least one example (using the core ideas or quotes) to illustrate each category in the text.

Use the Discussion section to highlight the most important findings, relate the results back to the literature, and pull the results together in some meaningful way, perhaps by beginning to develop theory to make sense of the data.

Advantages of CQR

ideal for conducting in-depth studies of the inner experiences of individuals. good for studying events that are hidden from public view, are infrequent, occur at varying time periods, have not been studied previously, or for which no measures have been created. CQR is ideal because it involves a rigorous method that allows several researchers to examine data and come to consensus about their meaning, thus reducing the biases inherent with just one person analyzing data

Limitations of CQR
the time commitment, the repetitiousness of some of the tasks, the lack of precise guidelines for some of the steps (e.g., When have you collected enough cases? How exactly do you come to consensus?), and the difficulty of combining findings across studies

Features of Good Quality Research Conceptual innovation Methodological rigor Rich, substantive content

3 questions guiding proposal reviewers What are we going to learn as the result of the proposed project that we do not know now? Why is it worth knowing? How will we know that the conclusions are valid?

Ethics in Research

Research should not cause harm to the participants Researchers should obtain the informed consent of participants Researchers should respect the privacy of participants Researchers should avoid conflict of interest Researchers should fairly and accurately report their findings

Key References

Borrego, M., Douglas, E.P., & Amelink, C.T. (2009) Quantitative, Qualitative, and Mixed Research Methods in Engineering Education, Journal of Engineering Education 98(1): 53-66. Hill, C.E., Knox, S., Thompson, B.J., Williams, E.N., Hess, S.A., & Ladany, N. (2005) Consensual Qualitative Research: An Update, Journal of Counseling Psychology 52(2): 196-205. Johnson, R. B. & Onwuegbuzie, A. J. (2004) Mixed Method Research: A Research Paradigm Whose Time Has Come, Educational Researcher 33(7): 14-26.