IntroductiontoCriticalAppraisal

IntroductiontoCriticalAppraisal

CHRISREDMAN
ALEXSANCHEZVIVAR

PresentationOverview
Introductiontocriticalappraisal
Definition,differences,strengthsandweaknessesof
systematicreviewsandmetaanalyses
Sourcesofsystematicreviews/metaanalyses
LevelsofEvidence
Interpretationofbasicstatisticsinmetaanalyses
confidenceintervals,forestplots
Criticalappraisalofsystematicreviews/metaanalyses

Whatiscriticalappraisal?
Criticalappraisalistheprocessof
systematicallyexaminingresearchevidenceto
assessitsvalidity,results,andrelevancebefore
usingittoinformadecision
(HillandSpittlehouse,2001,p.1)

Considerationofquantitativeandqualitative
aspects

Criticalappraisalisnot

Negativedismissalofanypieceofresearch
Assessmentonresultsalone
Basedentirelyonstatisticalanalysis
Undertakenbyexpertsonly

Whycriticallyappraise?
Tofindoutthevalidityofthestudy
arethemethodsrobust?

Tofindoutthereliabilityofthestudy
whataretheresultsandaretheycredible?

Tofindouttheapplicabilityofthestudy
isitimportantenoughtochangemypractice?

HowdoIcriticallyappraiseresearch?
Be(critically)opentoeverything
Believe(inprinciple)papersfromhighquality
journals
Read&decideyourself
Letotherpeoplereadanddecidefor(with)you
Readforyourselfandmakeastructured
appraisal

Criticalappraisal
Advantages

systematicwayofassessingvalidity,results&usefulnessof
research
contributestoimprovingpractice(quality)
encouragesobjectiveassessmentofinformation
notdifficulttodevelopskills

Disadvantages

timeconsuming
notalwaysanyeasyanswersorwhatyouhopedtofind
dispiritingifgoodevidenceislackingi.e.little/poor
researchdone

Criticalappraisal
BUTyoucanalldoitwiththerighttools&
guidance

WhatdoIneedtoknow?

Awarenessofstudydesigns
Levelsofevidence
Statistics!!
CAchecklists
CAresources

Awarenessofstudydesign

Observationalstudydesignmeasuresofdisease,measuresofrisk,andtemporality

Whatisasystematicreview?
Areviewthathasbeenprepared
usingsomekindofsystematicapproach
tominimisingbiasesandrandomerrors,
andthatthecomponentsoftheapproach
willbedocumented
inamaterialsandmethodssection
Chalmersetal,1995

What is a systematic review

Reviews

Systematic reviews

Rationaleforsystematicreviews
Informationoverload

Publicationbias

Poorqualityofreviews

VitaminCandthepreventionofthecommoncold (Pauling
1986)

Missinglink

Inhalationofhexamethonium(commentbyClarketal,2001)

Sourcesofsystematicreviews

TheCochraneLibrary
www.library.nhs.uk
DARE(inCochraneLibraryOtherreviews)
HealthTechnologyAssessments(inCochraneLibraryTechnology
Assessments)
Medline,Cinahl,Embasesearchonsystematicreviewintitle,
abstract
PubMedSystematicReviewinLimits>Topic
TRIP
www.tripdatabase.com
EvidenceBasedReviewsJournalsandDatabases
https://www.library.nhs.uk/evidence

NHSEvidence
https://www.evidence.nhs.uk/

Formatofasystematicreview

Formulationofareviewquestion
Defineinclusion/exclusioncriteria
Locatestudies
Selectstudies(inclusion/exclusion)
Assessstudyquality
Dataextraction
Analyseandpresentresults
Interpretationofresults
Eggeretal,2001

Formulationofreviewquestion
Isthequestionfocusedintermsof
Populationstudied
Intervention/exposuregiven
Outcomesconsidered
Doanticoagulantspreventstrokesinpatients
withatrialfibrillation?

Defineinclusion/exclusioncriteria
Weretherighttypesofstudiesincludedtoanswer
thequestion?Dependsonthequestion.
Canhaveobservationalstudies(cohort,case
control),diagnostic/screeningtests,prognostic,non
randomisedtrials
Studiesshouldbedefinedaccordingtotheirdesign,
participantcharacteristics,interventionsand
outcomes

Locatestudies
Comprehensivesearch

Databases
Conferenceproceedings
Handsearching
Greyliterature(reports,researchregisters)
Foreignlanguage
Followupreferences
Contactingexperts/authors

Publicationbiasunpublishedstudies
Explicit

SelectandAssessStudies
Eligibilitycriteriaforstudyselectioncanbe
applied
Morethanonereviewercanhelpreducebias
Checklists/scoringsystems

Whatdothefindingsmean?
Effectmeasuresoddsratios,relativerisk,
meandifference
Pvalues
Confidenceintervals

Usingstatistics
Assesstheweightoftheevidencethatatreatment
works(ordoesnt)
Giveanestimate(andlikelyrange)ofthe
treatmenteffect
Testtoseehowlikelyitisthatthiseffectwould
havebeenseenbychance

Oddsratio(OR)

Expressestheoddsofhavinganeventcompared
withnothavinganeventintwodifferentgroups

OR=oddsinthetreatedgroup/oddsinthecontrolgroup

 OR=1treatmenthasidenticaleffecttocontrol
OR<1eventislesslikelytohappenthannot(i.e.the
treatmentreducesthechanceofhavingtheevent)
OR>1eventismorelikelytohappenthannot
(increasesthechancesofhavingtheevent)
Clinicaltrialstypicallylookfortreatmentswhich
reduceeventrates,andwhichhaveoddsratiosof
lessthanone

Importanceofdefiningthe
outcome
Typeofoutcome
Valueof
OR/RR

Adverseoutcome(e.g. Beneficialoutcome(e.g.
death)
stoppedsmoking)

<1

Newintervention
better

Newinterventionworse

Newinterventionno
better/noworse

Newinterventionno
better/noworse

>1

Newintervention
worse

Newinterventionbetter

Pvaluessignificancetest
Apvalueisameasureofstatisticalsignificance
whichtellsustheprobabilityofaneventoccurring
duetochancealone
Pvalueresultsrangefrom0to1
Thecloserthepvalueistozero,thelesschance
thereisthattheeffectsoftwointerventionsarethe
same

Statisticalsignificance
Ingeneral,pvaluesofeither0.05or0.01areused
asacutoffvalue,althoughthisvalueisarbitrary

Pvalueof<0.05 indicatestheresultisunlikely
tobeduetochance
Pvalueof>0.05 indicatestheresultmighthave
occurredbychance.

Becareful
Apvalueinthenonsignificantrangetellsyouthateither
thereisnodifferencebetweenthegroupsorthereweretoo
fewsubjectstodemonstratesuchadifference(ideallyneed
toreportconfidenceintervals)
Thereisnotmuchdifferencebetweenp=0.049andp=0.051
Pvaluesdonotindicatethemagnitudeoftheobserved
differencebetweentreatmentsthatisneededtodetermine
theclinicalsignificance

Interpretationof

ConfidenceIntervals
Confidenceintervalistherangewithinwhichwe
haveameasureofcertaintythatthetrue
populationvaluelies
OR
Theconfidenceintervalaroundaresultobtained
fromastudysample(pointestimate)indicatesthe
rangeofvalueswithinwhichthereisaspecific
certainty(usually95%)thatthetruepopulation
valueforthatresultlies.
(MeReCBriefing2005)

WhatcanaCItellus?
Tellsuswhethertheresultissignificantornot
Thewidthoftheintervalindicatesprecision.Wider
intervalssuggestlessprecision
Showswhetherthestrengthoftheevidenceisstrongor
weak.
Thegeneralconfidencelevelis95%.Therefore,the95%
CIistherangewithinwhichweare95%certainthatthe
truepopulationvaluelies

ConfidenceIntervalsreportedon
Ratios(oddsratio,etc)

Thelineofnoeffectcentresaround1
IfaCIforanRRorORincludes1
(thelineofnoeffect)
thenweareunabletodemonstratestatistically
significantdifferencebetweenthetwogroups

Whatisametaanalysis?
Astatisticalanalysisoftheresults
fromindependentstudies,
whichgenerallyaimstoproduce
asingleestimateofthetreatmenteffect
Eggeretal,2001

Interpretationofforestplots
Effect of probiotics on the risk of antibiotic associated diarrhoea

D'Souza, A. L et al. BMJ 2002;324:1361

Interpretationofforestplots
Lookatthetitleoftheforestplot,theintervention,outcome
effectmeasureoftheinvestigationandthescale
Thenamesontheleftaretheauthorsoftheprimarystudies
includedintheMA
Thesmallsquaresrepresenttheresultsoftheindividualtrial
results
Thesizeofeachsquarerepresentstheweightgiventoeach
studyinthemetaanalysis
Thehorizontallinesassociatedwitheachsquarerepresentthe
confidenceintervalassociatedwitheachresult
Theverticallinerepresentsthelineofnoeffect,i.e.wherethere
isnostatisticallysignificantdifferencebetweenthe
treatment/interventiongroupandthecontrolgroup
Thepooledanalysisisgivenadiamondshape.Thehorizontal
widthofthediamondistheconfidenceinterval

Advantagesofasystematic
review/metaanalysis
Limitsbiasinidentifyingandexcludingstudies
Objective
Goodqualityevidence,morereliableand
accurateconclusions
Addedpowerbysynthesisingindividualstudy
results
Controloverthevolumeofliterature

Drawbackstosystematic
reviews/metaanalyses
Canbedonebadly
2systematicreviewsonsametopiccanhavedifferent
conclusions

Inappropriateaggregationofstudies
Ametaanalysisisonlyasgoodasthepapers
included
Tendtolookatbroadquestionsthatmaynotbe
immediatelyapplicabletoindividualpatients

Conclusion
Criticalappraisalofsystematicreviewsand
otherresearchiswellwithinyourcapabilities
Usearecognisedchecklist(i.e.SIGN)
Updateyourliteraturesearchingskills
regularly(contactyourlibraryskillstrainer)

D'Souza, A. L et al. BMJ 2002;324:1361

(Other)Criticalappraisalchecklists
CASP

(CriticalSkillsAppraisalProgramme)

http://www.phru.nhs.uk/casp/critical_appraisal_tools.htm

JAMAUsersGuidestotheMedicalLiterature
http://www.cche.net/usersguides/main.asp

CrombieI(1996)ThePocketGuidetoCriticalAppraisal,BMJ
Books,London
GreenhalghT(2001)HowtoReadaPaper,BMJBooks,London
BestBETsCAdatabase
http://www.bestbets.org/cgibin/browse.pl?~show=appraisal

References
Systematicreviewsinhealthcare[electronicresource]:meta
analysisincontext/editedbyMatthiasEgger,GeorgeDavey
Smith,andDouglasG.Altman.BMJBooks2001(ebook)
Whatisasystematicreview?,Whatisametaanalysis?,What
areconfidenceintervals?
http://www.evidencebased
medicine.co.uk/what_is_series.html
Understandingsystematicreviewsandmetaanalysis.Akonberg
AK.ArchivesofDiseaseinChildhood2005;90:845848.

References
CochraneOpenLearningMaterial:SystematicReviewsand
Metaanalyses(usefulForestPlotinterpretationPDF)
http://www.cochranenet.org/openlearning/HTML/mod32.htm

Funnelplots

Biasinmetaanalysisdetectedbyasimple,graphicaltest.
EggerM,etalBMJ1997(315):629634
Thecaseofthemisleadingfunnelplot.LauJ,etal.BMJ
2006(333):597600

Heterogeneity

Whatisheterogeneityandisitimportant?FletcherJBMJ
2007;334:946

The label tells you what the comparison and

outcome of interest are
Effect of probiotics on the risk of antibiotic associated diarrhoea

Scale measuring treatment effect.

Take care when reading labels!
Effect of probiotics on the risk of antibiotic associated diarrhoea

Each study has an ID (author)

Effect of probiotics on the risk of antibiotic associated diarrhoea

Treatment effect sizes for each study

(plus 95% CI)
Effect of probiotics on the risk of antibiotic associated diarrhoea

Horizontal lines are confidence intervals

Diamond shape is pooled effect
Horizontal width of diamond is confidence interval
Effect of probiotics on the risk of antibiotic associated diarrhoea

The vertical line in middle is the line of no effect

For ratios this is 1, for means this is 0
Effect of probiotics on the risk of antibiotic associated diarrhoea

Rationale for meta-analysis

Conventional
andcumulative
metaanalysisof
33trialsof
intravenous
streptokinase
foracute
myocardial
infarction.

Mulrow, C D BMJ 1994;309:597-599