DESIGNINGANDCONDUCTING MIXEDMETHODSSTUDIES

BethAngellandLisaTownsend Workshopforthe2011SocietyforSocialWorkand Researchannualmeeting

OverviewofWorkshop
DefinitionsandterminologyofMMResearch PhilosophicalAssumptions MixedMethods:NutsandBolts Break Planningyourmixedmethodsstudy
Sampling DataCollection

DataAnalysis EvaluatingMixedMethodsStudies RepresentingMixedMethodsResearch Examples QandAandTechnicalAssistance

Resources
Creswell&PlanoClark(2011)Designingand conductingmixedmethodsresearch.Thousand Oaks,CA:SagePublications,Inc. Teddlie&Tashakkori(2009)FoundationsofMixed MethodsResearch:IntegratingQuantitativeand QualitativeApproachesintheSocialand BehavioralSciences.LosAngeles:Sage Publications,Inc.

ConsensusDefinitionofMMResearch
Mixedmethodsresearchisthetypeofresearchin whicharesearcherorteamofresearchers combineselementsofqualitativeandquantitative approaches(e.g.,useofqualitativeand quantitativeviewpoints,datacollection,analysis, inferencetechniques)forthepurposeofbreadth anddepthofunderstandingandcorroboration Johnsonetal.(2007).

Keyterminology
Qualitative(QUAL)componentandQuantitative (QUAN)componentareoftenreferredtoas strands

PhilosophicalIssues

QuantitativeTradition
Philosophicalunderpinnings:positivism/post positivism Deductivelogic Dataarerepresentednumerically Associatedterms:surveyresearch,probability sampling,experimentalandquasiexperimental designs,descriptiveandinferentialstatistics

QualitativeTradition
Philosophicalunderpinnings:constructivism Inductivelogic Dataarerepresentedtextuallyorpictorially Associatedterms:groundedtheory,ethnography, casestudies,purposivesampling,categoricalvs. contextualizingstrategies,trustworthiness, credibility

MixedMethodsTradition
Philosophicalunderpinnings:pragmatism Bothdeductiveandinductive Dataarerepresentedbothnumericallyand textually/pictorially Associatedterms:concurrent(parallel)and sequentialmixeddesigns,triangulation,data conversion,inferencequality

Philosophicaldebatesaboutmixed methods
Incompatibilitythesis fundamentaldifferences betweenQUANandQUALapproachesaresogreat thatmethodscannotbemixed Pragmatism whatisthebestwaytoanswera researchquestion;bothmethodsofferdifferent waysofansweringresearchquestions

MixedMethods:NutsandBolts

CharacteristicsofMMStudies
Involvescollectionandanalysisofqualitativeand quantitativedatainwaysthatare:
rigorous framedepistemologically/theoretically

Themethodsaremixedby
orderingthemsequentially mergingthem embeddingonestrandwithintheother

Combinesthedatawithinthecontextofasinglestudyor researchprogram Encapsulatesthestrandswithinanoverallresearchdesign thatguidesthestudyasawhole

ResearchQuestionsthatCallforMM
Exploringthemeaningofaconstructor phenomenonfrommorethanoneperspective Explanationofanomalousfindingsorgetting behindthemechanismofactionofaneffect Theorydevelopmentfollowedby testing/extension Measuredevelopmentusinggroundedconcepts Augmentingevaluationstudieswithbetter understandingofinterventionimplementation

WaysThatDesignsVary
Levelofinteractionbetweenstrands Relativepriorityofstrands Timingorpacingofeachstrand Pointofinterface(atwhichpointintheresearch processarethestrandsmixed?):during interpretation,dataanalysis,datacollection? Researchstance,epistemology

TypologyofMixedMethodsDesigns
Convergentparallel Explanatorysequential Exploratorysequential Embedded Caveat:evolvingfieldwithevolvinglanguage

AdaptedfromCreswell&PlanoClark(2011)

ConvergentDesigns
QUANandQUALstrandsareconductedseparately yetconcurrentlyandmergedatthepointof interpretation Equalprioritygiventoeachstrand Usedtoformamorecompleteunderstandingofa topic,ortovalidateorcorroboratequantitative scales

ConvergentParallelDesign
QUAL DataCollection QUAL DataAnalysis

Compare orRelate

Interpretation/ Meta Inference

QUAN DataCollection

QUAN DataAnalysis

Convergent Parallel Design Example: Conceptual Adequacy of the Drug Attitude Inventory for Youth
Design Decisions
Choice of instrument Questionnaire Rating scale Sampling Convenience Representative

Design Decisions

QUAN Data Collection

Demographics Youth DAI Parent DAI Adherence Ratings Clinical Scales (CDRS,YMRS,CBCL)

Choice of method Interview Ethnography Focus group Sampling Purposive Convenience Setting

QUAL Data Collection

Semi-structured interviews Youth SEMI Parent SEMI

QUAN Data Analysis

Compare Contrast

QUAL Data Analysis

Metainference
Mixed Methods Question: Can prediction of youth attitudes toward psychotropic treatment be improved by knowledge about the factor structure of the DAI in youth and their subjective experiences experiences of treatment?

Townsend,Floersch,&Findling,2010

ExplanatorySequentialDesign
Methodsareimplementedsequentially,(QUAN QUAL) Usedwhenresearcherwishestousequalitative findingstohelpinterpretorcontextualize quantitativeresults
QUANData Collection and Analysis
Followup with

QUALData Collection andAnalysis

Interpretation/ Meta Inference

ExplanatorySequentialExample:ACT SocialNetworkStudy
QUANData Collection and Analysis QUALand QUANData Collection andAnalysis

Interpretation /Meta Inference

ACT Randomized Trial:NoSocial NetworkEffects

QUANAnalysisof SocialNetwork Predictors QUALstudyof RCTSubsample

Interpretation /Meta Inference

Angell&Test,2002;Angell,2003

ExploratorySequentialDesign
Methodsareimplementedsequentially,(QUAL QUAN) TheQUALstrandisconsideredexploratory,tobe followedbyfurthertestingandverificationduring theQUANphase

Qualitative DataCollection andAnalysis

Buildsto

Quantitative DataCollection andAnalysis

Interpretation

ExploratorySequentialDesignExample:Measuring ProceduralJustice(PJ)inPoliceEncounters
Reviewof existing QUALstrand: instruments Consumer andliterature interviews ledtoresearch Analysisof question:Do discrete existingPJ encounters instruments usinggrounded capture dimensional featuresof analysis contacts betweenpolice andcitizens Interpretation: PJexperiences withmental area)contextualizedby illness?
negativeexpectationsandb) sensitivetosmallgesturesof humanity

QUANstrand Instrument development Cognitive interviewing Expertreview Surveyof consumersusing finalinstrument (PCES) EFAandRasch Analysis
Interpretation: PCESpredicted reactionstopoliceencounter (resistance,cooperation)

Watson,Angell,Vidalon,&Davis(2010)

EmbeddedDesign
ResearcherconductingeitheraQUALorQUAN studyembedsasmallerstrandoftheother method,asanenhancement Secondarystrandcanbeconcurrentorsequential

QualitativeorQuantitativeDesign QUALorQUANDataCollectionandAnalysis QUALorQUANDataCollectionand Analysis(before,during,orafter) Interpretation/ MetaInference

EmbeddedDesignExample:CTI EvaluationandFidelityStudy
RCTofCriticalTimeIntervention(CTI)forMenLeaving Prison.QUANDataCollection,n=220 Fidelity/ProcessStudy QUALdatacollection,n=24 DataCollection Decisions: Interviews FocusGroups Fieldnotes Recordabstraction Samplingcriteria DataAnalysis Decisions: Coding(open, selective,axial) ofinterviewsand documents/ Narrative analysis?

MMResearchQuestions:Inwhatways isCriticalTimeInterventionmodifiedor adaptedwhenusedwithapopulationof recentlyreleasedprisoners?What processescontributetotheadapted programslevelofeffectiveness? Draine,Angell,Barrenger,&Kriegel(inprogress)

VariationsontheMMDesigns
Multiphaseformat Multilevelformat MonostrandConversion(nottrulymixed methods):conversionofQUALdatatoQUANor QUANdatatoQUAL,withoutadditionalstrands Transformativestance

Questions?

Break

PlanningYourMixedMethodsStudy

Sampling:GeneralConsiderations
Strategychosenshouldbeappropriatetoeach respectivestrand Balancebetweensaturationofphenomenonor theory(qualitativegoal)andrepresentativeness (quantitativegoal)

SamplingStrategies (TeddlieandTashakkori,2010)
Parallelmixedmethodssampling(paralleluseof probabilityandpurposivestrategies,either concurrentlyorwithatimelapse).
Onesamplemaybeasubsetoftheother Bothstudiesmayusesametotalsample

Sequentialmixedmethodssampling(informationfrom thefirstsampleisusedtodrawthesecond) Multilevelmixedmethodssampling:usingprobability andpurposivesamplingtechniquesatdifferentlevels ofanalysis(e.g.,cliniciansandclients)

DataCollection
(Teddlie&Tashakkori,2009)

SelfReportTechniques
Interviews Questionnaires AttitudeScales Personalityinventories Projectiveinstruments

ObservationalMethods
Participantobservation,nonparticipantobservation

Sociometry
Socialnetworkanalysis

SecondaryDataAnalysis
Archivalanalysis Metaanalysis 1998)

MultipleModesofDataCollection(Tashakkori&Teddlie,

DataAnalysis

QuantitativeDataAnalysis
Descriptive
summarizingdata,lookingfortrendsandpatterns; means,frequencies,measuresofvariability

Inferential
hypothesistesting,inferencesaboutapopulation characteristic;significancetests(2,t,F),multiple regression,ANOVA,MANOVA,MANCOVA, hierarchicallinearmodeling,timeseries,eventhistory

QualitativeDataAnalysis
Oftenongoingduringdatacollection(e.g.,purposive sampling,modificationofinterviewquestions,etc.)
Groundedtheory Thematicanalysis Negativecaseanalysis

FRACTURINGVS.CONTEXTUALIZING
Categoricalstrategies:producecategoriesthatfacilitatecomparisons;e.g., constantcomparativemethod Contextualizingstrategies:interpretnarrativedatainthecontextofthe wholetext,focusingoninterconnectionsbetweenstatements,events,etc.; e.g.,phenomenology

SIMILARITYVS.CONTRAST

MixedMethodsDataAnalysis
(CreswellandPlanoClark,2011)

QUAN+QUAL=convergeresultsCONVERGENTDESIGN QUAN qual=explainresultsSEQUENTIALEXPLANATORY DESIGN QUAL quan=generalizefindingsSEQUENTIALEXPLORATORY DESIGN QUAN(+qual)=enhanceexperimentEMBEDDEDDESIGN TRANSFORMATIVEDESIGN usesatransformativetheoretical perspectivetoadvocateforsocialchange,addresssocialinjustice, orgivevoicetomarginalized/underrepresentedgroup. MULTIPHASEDESIGN aprogramofresearchthatinvolvesseveral studies;canhavecombinationsofsequentialandconcurrent designs

MixedMethodsDataAnalysis
(Creswell&PlanoClark,2011)

Convergentparallel:mergeddataanalysisforpurposesof comparingresults
CollectandanalyzeQUALandQUANdata Strandsareanalyzedindependently(couldbequalitizing/quantitizing strategiesalso) Howwillthetwostrandsbecompared? Howwilltheyberepresented?

Explanatory:connecteddataanalysistoexplainfindings
Collectandanalyzequantitativedata;derivesecondresearchquestion Designandconductqualitativeresearch Analyzequalitativedataforanswerstosecondaryresearchquestion Linkresultsfrombothstrands howdoqualitativeresultsexplain quantitativefindings?

Convergent Parallel Design: Data Analysis of the Drug Attitude Inventory

QUAN Data Collection

QUAL Data Collection

Does the factor structure of the DAI in adults fit the youth data? If not, what is the factor structure of the DAI in youth? How well do DAI items correlate with one another? Do they measure a single construct or multiple constructs?

QUAN Data Analysis

Univariate descriptives Bivariate correlations Structural Equation Modeling Factor analysis Parallel Analysis (SPSS, LISREL)

Are there elements of youth medication experience that the DAI does not capture?

QUAL Data Analysis

In vivo codes Intermediate codes Superordinate codes Peer review Constant comparative approach (Atlas TI)

Compare

Converge

Can prediction of youth attitudes toward psychotropic treatment be improved by knowledge about the factor structure of the DAI in youth and their subjective experiences of treatment?

MixedMethodsDataAnalysis
(Creswell&PlanoClark,2011)

Exploratory:connecteddataanalysistogeneralize findings
Collectandanalyzequalitativedata;usequalitativedatatodesign quantitativecomponent Collectandanalyzequantitativedata Linkresultsfrombothstrands:howdoquantitativeresultsextend qualitativefindings?

Embeddeddesign:merged(concurrentdesign)or connected(sequentialdesign)analysis
Collectandanalyzeprimarydataset;decidehowembeddeddatawillbe usedandwheretheyshouldbeincorporatedintotheprimaryanalysis Analyzesecondarydatasetdictatedbywhereitisembeddedinthelarger design Howdotheembeddedfindingsintegratewiththeprimarystudyfindings?

EvaluatingMixedMethodsStudies

Mixed Methods Validation Framework (VF)

(Dellinger & Leech, 2007; Leech, Dellinger, Brannagan, & Tanaka, 2010)

Five elements:
Foundational element
Quality of literature review and theory base

Construct validation
Validity of QUAN, QUAL, and mixed elements

Inferential consistency
Consistency of links between various strands of the study (see table on following slide)

Utilization/historical element
Whether and how the studys findings went on to be used in future work

Consequential element
Social acceptability and consequences of study findings

Construct Validation
(Dellinger & Leech, 2007; Leech, Dellinger, Brannagan, & Tanaka, 2010)

RepresentingMixedMethodsData
(Creswell&PlanoClark,2011)

Sidebysidecomparison Jointcomparison Mergedcategory/themedisplay

Writing the article

(Creswell, 2003)

Introduction explicit integration of both paradigms from the outset Literature review integration of inductive/deductive reasoning, why the literature needs this type of study Posing the research question what are the questions and why do they call for two paradigms? Methods present both methodologies, in their respective languages, integrated under the umbrella of the research question Results present results of both modes of data collection Discussion role of meta-inference

ExampleI
FindingsfromDAIStudy (Townsend,Floersch,&Findling,2010)

StudyFlowchart
QUAN Data Collection
Demographics Youth DAI Parent DAI Decision-Making Scales Adherence Ratings Clinical Scales (CDRS,YMRS,CBCL)

QUAL Data Collection

Semi-structured interviews Youth SEMI Parent SEMI Brief Parent SEMI

QUAN Data Analysis

Univariate descriptives Bivariate correlations Structural Equation Modeling Factor analysis Parallel Analysis (SPSS, LISREL)

QUAL Data Analysis

In vivo codes Intermediate codes Superordinate codes Peer review Constant comparative approach (Atlas TI)

Compare

Converge

Can prediction of youth attitudes toward psychotropic treatment be improved by knowledge about the factor structure of the DAI in youth and their subjective experiences of treatment?

ParallelDataReductionStrategies
@stigma {0-56}

&Desire_for_Normality {0-0}

&Crazy_Identity {0-0}

&Educating_Others {0-0}

+want_to_live_normal_life_withou t_meds {1-0} +nobody_else_takes_meds {2-0}

+for_crazy_people {2-0} +Labeled_psycho {2-0}

+did_bipolar_slide_show_at_schoo l {1-0} +I_like_explaining_meds_to_peop le {1-0}

StructuralEquationModelOne
DAIOriginalFactorStructure

RMSEA .061
[ideal = <.05 (Kaplan, 2000)]

CFI .925 NNFI .913

[ideal = >.95 (Kaplan, 2000)]

X2

df 258 420.38

ExploratoryFactorAnalysis
FactorSelectionCriteria:
Maximumlikelihoodestimation Eigenvalue> 1.0 Minimumitemloading> .30 Retained4+items >4itemsbutdifferentiatedwellfromotherfactors Qualitativedataindicateretentionofitems/factorsisjustified

EFAOne
Twofactorswerenotinterpretedfurtherbecausetheywereeachcomprisedofonlyoneitem (items8and13)andhadvalues>1.0.(Heywoodcases)

EFATwo28items(SupplementedbyParallelAnalysis)
Twofactorsretained,accountingfor36.61%ofthevarianceinDAIscore. Factorlabels:
PositiveFeelingstowardMedication NegativeFeelingstowardMedication

Fouritemsdidnotloadonanycomponent(10R,11R,20R,and30) Cronbachsalpha=.889 YouthDAIcorrelatedpositivelywithyouthselfreportedadherence(r=.205,p<.05) DAIshowednosignificantcorrelationswithclinicaloutcomemeasures(CBCLCompetenceand SymptomScales).

QualitativeAnalysisMethods
Constantcomparativeapproach analysisofinvivocodes followedbyintermediatelevelcodingandsynthesisinto higherordersuperordinateconcepts.
@stigma {0-56}

&Desire_for_Normality {0-0}

&Crazy_Identity {0-0}

&Educating_Others {0-0}

+want_to_live_normal_life_withou t_meds {1-0} +nobody_else_takes_meds {2-0}

+for_crazy_people {2-0} +Labeled_psycho {2-0}

+did_bipolar_slide_show_at_schoo l {1-0} +I_like_explaining_meds_to_peop le {1-0}

SummaryofQualitativeThemes
DAIrelatedconcepts
Positivefeelings Negativefeelings Health/IllnessModel InternalLOC ExternalLOC RelapsePrevention

Conceptsnot representedintheDAI
Balancedresponsibility Ambivalence ChangeoverTime Adherenceifeffective Expectations Inclusionintreatment decisions Autonomy Stigma

SidebySideComparisonExample: ConclusionsSection
QuantitativeFindings
Insummarizingthefindingsoftheadolescent qualitativeandquantitativeanalyses,itis apparentthatthesevendimensionsfoundin theoriginalDAIemployedwithadultsare presentinteens conceptualizationsoftheir viewsofmedication.However,adolescents maythinkaboutthosedimensions,suchas sideeffectsandrelapseprevention, differentlythanadultsdo.Thesefindingsare supportedbyboththeEFAandqualitative results.EFAfindingsindicatethatsideeffects differentiatedintospecificcognitiveeffectsas wellasotherharmsforyouth.Adolescents didnotappeartoviewrelapsepreventionas theadultsdid,insteadevaluatingtheir perceivedneedformedicationinrelationto theirsymptomstabilityinformingtheir attitudestowardmedication.

QualitativeFindings
Thequalitativedatahighlightedthepresence ofeightadditionalthemesnotreflectedinthe originalfactorstructureoftheDAI,including importantthemessuchaspersonalautonomy, inclusioninmedicationdecisionmaking,the experienceofstigma,andlevelofpersonal responsibilityvs.externalcontrolover behavior.Thesethemeshighlightthat medicationexperiencesandtheformationof attitudestowardpharmacologicaltreatment foryoutharecomplexandmultiply determined,renderingitdifficulttorepresent adolescentattitudestowardmedicationfully withasingleinstrument.Thesefindingspoint tothepotentialvalueofcreating individualized,qualitativeassessmenttoolsto captureyouthexperienceswithmedication ratherthanrelyingonasinglequantitative measureorsetofsubscales.

Discussionsectionhighlights divergentresults

JointDataDisplayExample
HighlightsConvergenceofFactorAnalyticandThematicAnalysisResults
PositiveFeelings 2 goodoutweighsbad 21 thoughtsareclearer 26 happieronmeds 29 inbettercontrol QualitativeThemes Emotional Cognitive Physical NegativeFeelings 5R takeb/cofpressurefrom others 14R medicationisslowacting poison 19R Idratherbesickthan takingmedications QualitativeThemes Emotional Cognitive Physical FactorLoading .474 .789 .487 .598 Quotations Icaneithernottakeitandbeagrouch,ortakeitandbehappy.SoIwouldmuchrathertakeitandbehappy. Sittingdown,payingattentiontotheteacher,nottalking.JustpayingattentionanddoingwhatIneedtodo. Igainenergytowanttodomyworkandlisteningandfocusandtrytodomybestandgive110%. FactorLoadings .315 .714 .658 Quotations IdontlikesometimesIfeellikeoutofmybodyorjustlikenotmyself,moreanxioussometimes. WellIhadittoldmetheTopamaxorwhatevercausesshorttermmemoryloss,soIassumeitstheTopamaxorwhatever. Makemesick.Makemegetthebubblegutsandstuff.AndIwishthattherewasamedicinethatdidnthavesideeffects.

MergedDataDisplayExample
Countsofquotationsrepresentedgraphicallytodemonstrateconvergence withfactoranalyses positive andnegative factorssupportedbyallformsofQUANand QUALevaluation
Distribution of DAI-Related Quotations across Respondents
40 35 Number of Quotations 30 25 20 15 10 5 0 4 7 10 12 13 14 16 17 18 20 22 23 24 25 27 29 30 32 38 41 Respondent Positive Negative Illness Model ILOC ELOC Harm Relapse Prev.

Number of adolescent quotations linked to original DAI factors Positive Subjective Feeling 103 Negative Subjective Feeling 31
Health/Illness Model

External Locus of Control 20

Internal Locus of Control 10

Harm/ Toxicity 48

Relapse Prevention 34

21

ExampleII
ForensicAssertiveCommunityTreatment(FACT) Evaluation(Angell&Watson,inprogress) Keymixedmethodquestions:
HowisACTmodifiedinthecontextofprisonreentry? Howdoestheagencysrecoveryorientedmissionshape thetranslationofACTtoFACT? Howdoesengagementofconsumersoccurandtowhat extentisleverageinvolved? Whataretheuniquefeaturesofclientprovider relationshipsinFACT? Howdoesengagementrelatetoratesofsuccess (avoidanceofreincarceration)?

OverallDesign:ConvergentParallel
Sample=21adultswithmentalillnessleavingprisonandentering FACTprogram
PanelDesign

Client reported outcomes assessedat BL,1,3,6, 9,18mos staff reported outcomes monthly

QUAN DataCollection QUAN DataAnalysis

QUAL DataCollection QUAL DataAnalysis

Ethnographicapproach Participantand passiveobservation Interviewswith consumersandstaff Recordabstractions

WithinSubject CrossData Comparisons

Interpretation/ MetaInference

AnalysisStrategies
Sample=21adultswithmentalillnessleavingprisonandentering FACTprogram
QUAL Descriptive PrePost Comparisons Outcome differencesby risklevel

QUAN DataCollection QUAN DataAnalysis

QUAL DataCollection QUAL DataAnalysis

QUAL openandfocusedcoding creationofsynthetic participantnarratives

WithinSubject CrossData Comparisons

MixedAnalysis: Comparingphenomena Acrossdatatype Acrosstime Acrosssubjects

Interpretation/ MetaInference

QandA/TechnicalAssistance