Legacy Series: MAXIMIZING

UTILIZATION OF RESEARCH

This paper is part of a series of eight Legacy Papers synthesizing major lessons learned through research conducted under the Frontiers in Reproductive Health Program (FRONTIERS). The full set of Legacy Papers includes: -- Capacity Building -- Family Planning -- Female Genital Mutilation/Cutting -- Gender -- Integration of Services -- Sustainability of Services -- Utilization of Research Findings -- Youth Reproductive Health The complete reports referenced in these papers are available online:

Operationsresearch(OR)canonlybejudgedonlyas successfulifresultsareutilizedformakingdecisions tostrengthenRH/FPpoliciesandservicedelivery. Howcanthisbestbeachieved?Howcanboth processandimpactofORbemeasured?Drawing from10yearsofFRONTIERSORexperience,a numberofkeyprinciplesforpromotingresearch utilization,illustratedwithdocumentedexamples, arepresentedhere.

www.popcouncil.org/frontiers

Oneimportantfirststepisclarifyingtermsused, almostinterchangeably,inresearchutilization,asit incorporatesarangeofwaysinwhichresearchcan beusedformakingdecisionstostrengthenRH/FP policiesandprograms.(Conceptsunderlyingterms usedaredefinedinBox1.)Achieving,andmeasuring, utilizationofresearchdependsinpartonwhat type(s)ofutilizationisenvisioned,andsoitiscritical thespecifictypeneedstobeconsideredand specifiedbeforeresearchbegins.
Box 1. Clarifying Research Utilization Language Research utilization: making decisions concerning policy, advocacy and resource allocation, planning and management, and program systems development and strengthening, using information generated from research. Institutionalization: incorporation of a practice or intervention proven to be effective (sometimes termed a best practice) within the routine activities of a facility, program or organization. Replication: introduction of a proven intervention or practice into another setting; this may be another program or another country. Scale-up: extension of an intervention or proven practice beyond the original project site.

Moreover,anORprojectmaybe expectedtoachieveseveraltypesof utilization,dependingonprojectnature, generalizabilityoffindingstoother programsandsettings,andavailability ofresourcesformorethanonetypeof utilization.Forexample,anintervention provensuccessfulinadistrictmayfirst becomeinstitutionalizedwithinthat districtshealthprogramandthen scaleduptootherdistrictsinthesame provinceandtootherprovinces.If appropriate,itmayalsobereplicatedin anothercountry.

FRONTIERSLegacyDocuments

MaximizingUtilizationofResearch

 Mostoftheprinciplesdescribedbelowarerelevantwhetherresearchconcerns introductionofaneworimprovedRH/FPtechnologytoacountryorprogram(suchas emergencycontraception),orofanew/revisedservicedeliveryguidelineortool(suchas theBalancedCounselingStrategyorSystematicScreeningoraneducationalortraining curriculum),orthereorganizationofservicedeliverysystems(suchasintegratingFPwith HIVservices,orusingcommunitylevelworkerstoprovideservicesusuallyofferedin clinics).

CommunicatingResultstoInfluenceDecisions
Underlyingallresearchutilizationiscommunicationofinformationgeneratedbythosewho produceittothosewhocanuseitformakingdecisions.FRONTIERSexperiencesuggestsa numberofstrategiescanenhancelikelihoodthiscommunicationcanbeeffective.

Translateandsynthesize.Research,likeservice
delivery,hasitsownvocabularythatcanbe difficultfornonresearcherstounderstand. Moreover,thetypesofactiondecisionsthat couldbemadebasedonfindingsshouldbe explicitlystatedasrecommendations;simply presentingresearchfindingsrequiresthe decisionmakertointerpretthem,whichmay resultinwrongdecisions.Bringingtogethera rangeofevidencefromseveralresearch studiescangreatlystrengthenfindings. DocumentssuchastheFRONTIERSseriesof ProgramBriefs(Box2)areexamplesofhow researchfindingscanbetranslatedand synthesizedtoproduceprogrammatically usefulinformation.
Box 2. FRONTIERS Program Briefs FRONTIERS produced 13 Program Briefs, 4- to 20- page documents that synthesized findings from OR on major reproductive health issues, online at:

www.popcouncil.org/frontiers/ pubs_types/prbriefs.html
1. Meeting womens needs after abortion 2. Using men as community-based distributors of condoms 3. Enhancing quality for clients: The balanced counseling strategy 4. Postabortion family planning benefits clients and providers 5. Building capacity to utilized operations research 6. Systematic screening: A strategy for determining and meeting clients reproductive health needs 7. Make better use of provider time in public health clinics 8. How much will it cost to scale up a reproductive health pilot project? 9. Increasing womens use of the IUD for family planning 10. Meeting the family planning needs of postpartum women 11. Adapting focused antenatal care: Lessons from three African countries 12. Financial capacity building for NGO sustainability 13. Multisectoral youth interventions: The scale-up process in Kenya and Senegal

Communicateresultsthroughmultiple channelstoreachthesameaudience manytimesandmanyaudiencesatleast once.Hearingthesamemessagemanytimes

andfromdifferentsourcesincreases likelihooditwillbeused.Communicatingthe samefindingsinFinalReports,ORSummaries (www.popcouncil.org/frontiers/pubs_types/ orsummaries/ors.html),ProgramBriefs, nationalworkshops,international conferences,listservannouncements, interpersonaldiscussionsandotherformsof mediahasincreasedvisibilityofkeyfindings fromFRONTIERSprojects.

FRONTIERSLegacyDocuments

MaximizingUtilizationofResearch

Shareresultswithspecialistmediatororganizations.Recognizingtheneedforresearch
basedevidenceanddifficultiesfacedbymostresearchersineffectivelycommunicating programmaticmessagesfromfindings,anumberofspecialistorganizationsandprojects nowexistthatcanassistincommunicatingfindings(Askew,Matthews,andPartridge2006). ExamplesincludethePopulationReferenceBureau,theJohnsHopkinsCenterfor CommunicationsPrograms,andtheCChangeproject,amongothers.Aparticularlyuseful mediatorforRH/FPprogrammingistheImplementingBestPracticesinitiative, coordinatedbyWHO(Box3).
Box 3. The IBP Initiative Initiated by WHO and USAID and with supported by numerous international agencies, the IBP Initiative seeks to improve access to evidence-based practices in reproductive health. The coalition supports country and regional conferences focused various issues, from identifying best practices on a specific topic to scaling up proven practices throughout the health care system. The IBP initiative has produced guidelines on the scale-up process, and also operates a Knowledge Gateway to circulation of documents on best practices and discussion of field experiences. The Knowledge Gateway offers community discussions on a range of topics, a community library, announcements, and a calendar of events. www.ibpinitiative.org

Becomeadecisionmaker.Oneextremelyeffectivewaytocommunicatefindingsisfor
researchersthemselvestoengageindecisionmakingprocesses.Researcherscanplay extremelyusefulroleswhenservingoncommittees,steeringgroups,technicaladvisory groups,programdesignteams,etc.bybringingnotonlyinformationfromtheirown researchbutalsoathoroughunderstandingofliteraturearoundthetopic.

Managersalsohavearesponsibilitytoseekoutresearchbasedinformation.Theonus
oncommunicatingresearchresultsisusuallyplacedonthoseproducinginformation. However,adecisionmakerhastowanttousesuchinformation,mustknowhowtofindit, andshouldactivelysearchforit.FRONTIERSaddressedthisissueinanumberofways, includingdevelopingashortcourseonORspecificallytailoredformanagers(Box4), engagingnational decisionmakersin Box 4. OR for Managers designingand This three-day course is one of several introductory courses implementingOR on OR developed by FRONTIERS. The OR for Managers course is projects,supporting designed to educate managers of reproductive health and service provision programs on operations research and how to apply participationat research findings to improve programs. researchconferences, www.popcouncil.org/frontiers/OR_Course/index.htm andsolicitingtheir perspectiveson Foreit and Khan 2008 researchpriorities.

FRONTIERSLegacyDocuments

MaximizingUtilizationofResearch

 StrategiesforIncreasingResearchUtilization

Whileitisbothimpossibleandinappropriatetoproposeastandardizedmodelfor developingandimplementingORtomaximizeutilization,experiencefromalmost200 projectsover10yearshasprovidedFRONTIERSwithmanyexamplesofwhatdoesanddoes notwork.ThisexperiencehasbeencombinedwithWHOsandotherpartnersexperiences inTurningResearchintoPractice(Box5).FRONTIERSalsocollaboratedwithseveralUK basedDFIDfundedappliedresearchgroups,whichprovideanotherrichsourceofguidance onmaximizingresearchutilization(Askew,Matthews,andPartridge2006;Nath2007).The MEASUREEvaluationProjectsDataDemandandInformationUseinitiativealsooffersa usefulframeworkandseveraltoolsforensuringresearchcaninformprogramdecision makers(Foreitetal.2006).Someofthekeystrategiesemergingfromthesereviewsfollow.

PlanforutilizationBEFOREstartingresearch.
Managersandresearchersmustconsiderlong termprospectsforresearchbeforeobtaining fundingandevenbeforewritingresearch proposals.Planningneedstoaddressseveral questions:Whowillusethedata?What decisionscanbeinfluenced?Cantheycommit themselvestomakingandfundingchanges needed?

Engageandworkwithdatausersandother stakeholdersthroughoutresearch process.Involvethosewhowillusethedata

(managers,clinicians,policymakers)in researchdesigntofindoutwhattypeofdata wouldbemostpersuasiveandeffective.Give regularprogressreports(inpersonand writing)andencouragemanagerstomakesite visits.Workwiththemtointerpretdraft resultsbeforefinalizingthestudysotheycan understandandidentifyprogrammatic implications.

Box 5. TRIP Report: A framework for maximizing research utilization This report is the product of two international meetings in which researchers, policymakers, donors, and program managers developed guidance on increase the use of research findings in program development and monitor the incorporation of evidence-based practices within reproductive health programs. www.who.int/reproductivehealth/publications/trip/index.html

Undertakeoperationsresearchinafavorable context.Validmeasuresoffeasibilityand
effectivenesscanonlybeachievedwhenan interventionisfullyimplemented.Thusitis preferabletoselectprojectsitesfromamong thosethatarestable,functioning,andready todeliverintervention.

FRONTIERSLegacyDocuments

MaximizingUtilizationofResearch

Usethestrongestresearchdesignanddatacollectionmethodspossible.Fully
experimentaldesignsarenotalwayspossible,butstrongquasiexperimentaldesignsshould beusedwhereverpossible.Adheringtonationalandinternationalethicalstandardsfor researchmustbeapriority,evenifitcompromisesidealstudydesign.Becarefulnotto collecttoomuchdata;beguidedbytypesofprogrammaticdecisionsyouwouldlikedatato inform.

Analyzedataquicklyandprioritizeresultsneededfordecisions.Keepinitialanalysis
simple,yetappropriate.Involvedecisionmakersindataanalysisandinterpretationso findingscanbetranslatedintoactionablestatementsorpolicies.Prioritizecommunication offindingstodecisionmakersmostlikelytousefindingsandtocommunitiesparticipating instudiesfirst.

Planfor,budgetfor,andincludeutilizationphasewithinresearchprocess.
ORstrengtheningservicedeliveryshouldnotendwithresultsdisseminationtodecision makers;itisunreasonabletoexpectthemtosimplyturnfindingsintoaction.Resources shouldbeleveragedsothosewhoundertookpilotprojectcanprovidetechnicalassistance tohelpprogramsimplementchangesindicatedbyresearchfindings.

Onlyrecommendandadvocateforservicedeliverychangesifresultsreallydo demonstratefeasibility,effectiveness(andpreferablycosteffectiveness),and potentialforinstitutionalizationandscaleup.Researchdatacanbemisunderstoodor

misusedforexample,tosupportparticularviewpoints.Greatcareisneededtoavoid recommendinginterventionsbeadoptedwhendatadoesnotreallydemonstrate convincingeffectiveness,aswellasapproachestoservicedeliverybeingchangedunless alternativesreallyaremoreeffectiveorlesscostly.

InstitutionalizationofProvenPractices:CriticalStepinCreatingConditionsfor ScalingUpEffectiveInterventions
Technicalassistanceisnotusuallyconceptualizedasaresearchactivity.However,after researchhasidentifiedapromisingpractice,afollowonphaseoftechnicalassistanceisan effectivestrategyforenhancinginstitutionalizationofanewpracticepriortofullscaleup. Duringthisfollowonphase,individualsororganizationswhoundertookthepilotproject providesupporttohelptheRH/FPprogramenactchangesnecessarytoinstitutionalizethe practice,soitbecomesroutine.WithinFRONTIERS,thisadaptationphasewasorganizedas andtermedacreatingconditionsforscaleupproject.Theseprojectslastedfromsixto18 monthsdependingontheadaptationsneeded,andcouldincludeactivitiessuchas reorganizingstaffresponsibilitiesandskillstraininginnewprocedures,revisingtraining curricula,changingsupervisoryprocedures,restructuringrecordkeepingandreporting forms,andsoon.

FRONTIERSLegacyDocuments

MaximizingUtilizationofResearch

 Toensureinterventionsproveneffectiveduringpilotprojectscanbeinstitutionalizedinto programroutineoperatingprocedures,creatingconditionsprojectssupportexpansionof proceduresthroughoutallfacilitiesinoneormoredistrictsandusuallydistrict(s)inwhich theinterventionwaspiloted.InmostcountriesofAfricaandSouthAsia,thedistrictisthe lowestlevelofhealthservicesadministration;consequently,theseprojectsprovide technicalassistancetostaffofadistricthealthmanagementteam(oritsequivalent)by helpingcomprehensivelyinstitutionalizeneworimprovedpracticesthatis,withindistrict planning,budgeting,implementation,andreportingsystems.Experiencehasshownthis phaseoflimitedscaleupofprovenpracticesdemonstratestheinterventioncanbe incorporatedintoandfundedbyexistinghealthsystems,institutionalizedintoroutine procedures,implementedatscale,andfundedbythehealthprogram. ThisphasehasprovencriticalforanRH/FPprogramtolearnhowtoperformthese functionsbeforemorewidespreadscaleupcanbeundertaken.Examplesofcreating conditionsprojectsinclude:amaleRHservicesmodelinBangladesh(Mannanetal.2008);a QualityAssuranceapproachinGujaratState,India(Khanetal.2008);amodelforinvolving meninmaternitycareinNewDelhi,India(Varkey,Mishra,andKhan2008);andcommunity midwiferyservicesinWesternProvince,Kenya(MwangiandWarren2008).

Scalingupbestpractices:Effectiveness,efficiency,andexpansion
Therearemanyreasonswhypilotprojectsshowinghowtoeffectivelyimplementnewand improvedpracticesarenotscaledup,includinglimitedfunding(availabilityfordurationof researchprojectonly);lackofexpectationbydonorthoseimplementingpilotprojectshould scaleuppracticestested;andanassumptionservicedeliveryorganizationsareabletotake researchfindingsandscalethemupinanRH/FPprogram.Introducingacreating conditionsphaseenablesprogramstomovefromprovidingserviceseffectivelytolearning howtoprovidethemefficiently,byadaptingnewpracticestoexistingdistrictlevelsystems. Scalingupfromdistrictleveltoregionalorprovincial,andthennationallevels,requires anotherphase:learninghowtoexpandresponsibilitytoinstitutionalizeservicedelivery protocolsthroughouttheservicedeliveryprogram. FRONTIERSexperiencewithnationwidescaleupofemergencycontraception(EC)services inBangladesh(KhanandHossainForthcoming)andofexpandinganadolescentRHservicesin Kenya(Eveliaetal.2008;Joyceetal.2008)providetwoexamplesofwaysinwhichprograms havelearnedtoexpandimplementationofinterventionsproveneffectiveandefficient. Somegenerallessonsemergingfromthesetwoexperiencesinclude: Ensuringthegovernmentministryorministriesleadtheprocess,notjustintitlebutby designatingkeymanagerstomakeallrelevantdecisions; Convincingthegovernmenttocommitfundingtosustainnewwayofprovidingservices, whetherfrominternalcommitmentsornegotiationswithdevelopmentpartners; FRONTIERSLegacyDocuments MaximizingUtilizationofResearch

Engagingawiderangeofnationaland internationalstakeholderstoappear nottoadvocateforoneorganizations interests; Supportingpolicyandsystemsreviews andrevisionsfacilitatingnationwide implementationofnewservicesand procedures; Creatingatrainingcascadethrough trainingoftrainersatseverallevelsto minimizeresourcesneeded. Animportantlesson,forboththosecon Guidance for adapting and modifying cost ductingapilotprojectandforthose information obtained from a pilot project to responsibleforscaleup,istestingandimple estimate scale-up costs is provided in mentingonlyinterventionsanationalprogram FRONTIERS Program Brief no. 8 (Janowitz, Bratt, canaffordtoscaleup.Thereisnopointin Homan, and Foreit 2007). The brief shows why the costs of a pilot project alone are not sufficient to pilottestinginterventionsnotaffordableat scale.Whendesigningapilotproject,planners predict costs of scale-up, and gives examples of how costs are influenced by factors like shouldattempttoestimatecostsofscalingup economies and diseconomies of scale, resource beforeembarkingoncreatingconditionsand substitution, and intervention modification. expansionphases.Insodoing,waste associatedwithpilotingunsustainable interventionscanbeavoided.Affordableinterventionsmayproducelessspectacularresults duringthepilotphase,butbeingabletoscaleupmoremodest,affordableinterventionswill makelargerhealthimpactsthansmallscaleprojectsyieldinglargehealthbenefitsbutonlyin intensivelyresourcedpilotprojectssocalledboutiqueprojects.

ReplicatingSuccessfulInterventions

Aninterventionthatisproveneffectiveandefficientinonesettingcanbeattractiveto programsinothercountries.Aneffectivestrategytofacilitatereplicationistoholda workshoporconference,attendedbypolicymakersandprogrammanagersfromseveral countriesinaregion,atwhichpracticalguidanceinintroducingandscalingupaproven bestpracticeisprovidedbythoseresponsibleforsuccessfullypilotingtheintervention. FRONTIERSusedthisstrategytoreplicateseveraloftheinterventionsinitiallytested throughORprojects. In2002FRONTIERSjoinedaconsortiumofinternationalorganizationsinSenegaltoconvene aregionalconferencethatadvocatedforincreasingaccesstoandstrengtheningqualityof postabortioncare(PAC)services(PostabortionCare(PAC)InitiativeforFrancophoneAfrica Committee2004)(Box6).TwokeypresentationsweretheexperiencesofSenegaland BurkinaFasoindevelopingandtestingPACinterventionsthroughORprojects(Askew2006). FRONTIERSLegacyDocuments MaximizingUtilizationofResearch

 Thesesystematicallydocumentedpilotprojects formedthebasisofthemodeldevelopedfor introducingPACservicesinothercountriesinthe region.Ananalysisoftheexperiencesof introducingPACintofourothercountries (Guinea,Mali,NigerandTogo)andofscalingup thepilotexperiencesinBurkinaFasoandSenegal bytheCentredeFormationetdeRechercheen SantdelaReproduction(CenterforTrainingin ReproductiveHealthResearchorCEFOREP) describestheprocessesfollowedintheregion (seebox).

Box 6. Replicating PAC in West Africa In this model emphasis was placed on identification of national champions, advocacy to gain support for PAC services, need for an initially vertical program, a pilot phase using an operations research approach, and training of trainers. During this six country assessment, these elements were highlighted as key determinants by national stakeholders. In particular, the role of an initial OR study and utilization of findings and leadership of university professors as facilitating factors for PAC introduction was highly influential. The assessment identified 22 programmatic recommendations for guiding replication of this PAC model.

FRONTIERSusedthisstrategytoreplicate introductionofagenderaccreditationtool developedandscaledupinBolivia(Palenqueetal. 2004;Palenque,RiverosHamel,andVernon2007).In Dieng et al. 2008 June2007aworkshopinCostaRicatrained31 participantsinthestrategyfromMinistriesof Health,SocialSecurityInstitutes,multilateralorganizationsandseveralNGOsinBolivia, Peru,Ecuador,DominicanRepublic,ElSalvador,Honduras,Guatemala,andCostaRica. Subsequently,technicalassistancewasprovidedtoorganizationsinElSalvador,Honduras andPerutosupportintroductionofthetool(RiverosHamel,Martin,andVernon2008).

ThesamestrategyhasalsobeenusedrecentlyincollaborationwithWHOsAfricaRegional BureauforreplicatingamultisectoraladolescentRHmodelinseveralAfricancountries (BurkinaFaso,Ghana,Kenya,Mauritania,Mozambique,Namibia,Senegal,Tanzania,and Zimbabwe)(DiopandDiagne2008),andforenhancingpostpartum/postabortionfamily planningservicesintheArabregion(Egypt,Jordan,Sudan,andYemen)(FRONTIERS2008).

EvaluatingORUtilization
Aswithservicedeliveryprojects,evaluatingoperationsresearchactivitieshasmanybenefits:

Increasingrecognitionofvalueofevidencebasedpolicymakingandservices programming;

Demonstratingtofunderresearchismakingadifferenceandisthereforea worthwhileinvestment;

Justifiesallocationofresourcesandprioritizesfutureallocations;

Focusesresearcherattentiononutilizationandapplicationofresearchfindings;

Helpsimprovedesignandimplementationofresearch,therebyincreasinglikelihoodof producingresultsthancanbeutilized. FRONTIERSLegacyDocuments MaximizingUtilizationofResearch

 Anumberofwayshavebeentestedtomeasureanddocumentresearchutilization. ThecasestudyapproachreviewsutilizationofaspecificORactivityingreatdepthto understandwhetherfindingswereused,andifso,how.TheGettingResearchintoPolicy andPractice(GRIPP)initiativecollected18casestudies,documentedresearcheractivities promotingutilization,andcreatedawebportalforcommunicatingexperiences.GRIPPwas apartnershipbetweenFRONTIERS,JohnSnowInternational(Europe),andtwoDFIDfunded researchprograms,OpportunitiesandChoicesandSafePassagestoAdulthood.Asynthesis ofthe18casestudiesidentifiedanumberoffactorsfacilitatingutilization(Nath2007).The casestudyapproachrequiresdocumentationbeintegraltotheresearchprocessfrom inception,aneutralfacilitatordocumenttheprocess,adequateresourcesbudgetedfor documentation,andidentificationofappropriatestakeholderstoelicitdifferent perspectives. Asecondapproachreviewsabroadportfolioofresearchinitiativesundertakenina particularsettingtolearnaboutunderlyingpatternsfacilitatingorobstructingresearch utilization.SuchanapproachhasbeenusedinMexicoandGuatemala,Bangladesh, Indonesia,andEgypt.InMexico,thefocuswasondescribingtherelationshipbetween healthresearchersandpolicymakersandtoreconstructtheprocessesthroughwhich researchwasusedtomakedecisionsandpolicies(Trostle2006).InGuatemala,researchers reviewed44ORprojectsconductedbythePopulationCouncilbetween1988and2001 (Brambilaetal.2007).Projectswereonawiderangeoftopics,covereddifferenttarget groups,andwereconductedwithdiversecollaboratinginstitutionsandresearchers.This approachhaspreviouslybeenusedinBangladesh(HaggaandMaru1996),Egypt(Hegazi1997) andIndonesia(IskandarandIndrawati1996). AthirdapproachgeneratesquantitativemeasuresoftheextenttowhichlessonsfromanOR projecthavebeenused.TulaneUniversitydevelopedamethodologyforassessing utilizationofFRONTIERSprojectsthatcollecteddataon14processindicators(levelof participationofkeystakeholders,qualityofresearch,problemsinprogramimplementation) and11impactindicators(Box7),andsix contextualfactors(e.g.facilitatingfactors, Box 7. Key Impact Outcomes barriers,assessmentofcosts)(MarinandBertrand Measured by Tulane Methodology: 2000;Marin,Gage,andKhan2004).This 1. Were improvements made to methodologyevaluatedtheextentofutilization program? 2. (If proven effective) were the of64FRONTIERSprojectsthathadtestedan improvements scaled up? interventionbetween1998and2004(RamaRao 3. (If proven successful) were the andGolon2005).TheTulanemethodologyiseasy improvements replicated? touse,fairlylowcostandgeneratesquantitative 4. Was policy (re-) formulated? 5. Was increased funding made indicators,makingitattractivetothosefunding available? researchprograms.Formoretexturedprocess 6. Was organizational capacity information,thismethodologycanbe enhanced? complementedbycasestudiesandportfolio reviews.

FRONTIERSLegacyDocuments

MaximizingUtilizationofResearch