Systematic Review Appraisal:

Steps in Evidence based practice:

1. Start with formulating a question
2. Search for evidence to answer that question
3. Critically appraise evidence (evaluation and assessment)
4. Apply to practice
5. Audit the practice
6. Modify

Hierarchy of evidence:
 Observational studies are a bit weaker compared to
interventional studies.
 RCT is considered the gold standard of primary studies
 Systematic reviews are considered secondary studies
which mean they deal with studies to produce new
evidence.
 Systematic reviews are v meticulous in their production
so are considered on top of the hierarchy pyramid
Systematic Reviews:
 Secondary study: several studies that are collected
together to produce a new brand study. The participants
here are previous studies.
 Primary studies (cohort, case-control, RCT): the authors
conducted their study on populations then reported the
results as is. Participants might be human beings, test
materials, animals.
 Produced by a team with a min of 2 researchers along
with statisticians.
 The mathematical results of the studies can be
aggregated/ combined together as one result. This is
called meta analysis (statistical aggregation of studies
that were once separate). Requires at least 2 studies.
So a systematic review should start with a clearly formulated
question. Accordingly, an eligibility criteria for primary
studies is formulated. The authors then select studies that fit
in. The studies are appraised to evaluate their methodological
quality and then summarized and interpret the data and, if
possible, combine them by meta analysis.
They synthesize the results of multiple primary
investigations, then researches are appraised, results
interpreted and summarized, but not statistically
combined when this is done, it is called “ Meta analysis”
which is using statistical methods to combine results of 2
or more studies.
What is a systematic Review?
A review (secondary study) of the evidence on a clearly
formulated question (PIPO) that uses systematic and
explicit methods to identify, select and critically appraise
relevant primary research, and to extract and analyse
data from the studies that are included in the review*
Statistical methods (meta-analysis- gives a combined
mathematical results) may or may not be used to analyze
and summarize the results of the included studies.”
(Cochrane Glossary)
Meta- Analysis are not easy to produce due to the
heterogeneity of the studies used to formulate a systematic
review. The differences can be manageable or too difficult to
produce a meta-analysis.
Advantages of SR:
1. Find the relevant evidence in unbiased manner
2. Appraise each paper for methodological quality
3. Synthesis the results: meta-analysis
The authors of a good SR:
1. Formulate a Question (PICO- population,
intervention, comparison, outcome)
2. Conduct comprehensive Search: will evaluate if that
review included all studies that conducted this
question.
3. Refine the search by applying predetermined
inclusion and exclusion criteria
4. Assess their quality and validity (critical appraisal)
5. Extract the appropriate data
 6. Synthesize, interpret, and report data
All steps should be described explicitly in the review

Importance of evidence for practioners:

To provide patients with best clinical practice.
Best approach for therapeutic questions- RCTs
Problem: many RCTs so how will the decision be taken.
Best way to judge: Systematic Reviews (but not all are
reliable).
Searching for evidence:
Comprehensive, reproducible search is the foundation for the
systematic review. Steps of search must be transparent and
clear.
“A comprehensive and reproducible literature search is
the foundation of a systematic review.”
How to ensure that authors done a proper job a search:
Min number of 2 databases to avoid selection bias.
 Electronic data bases (e.g. Medline)- at least 2 (the
more the better)
 “Hand searches” of relevant journals: searched
physically the journal going through reference list to
make sure that all the possible trails that were conducted
dealing with that question is included in that systematic
review.
 Writing to the experts: to ask them if they’re going
through a trial that was not published yet.
  Should aim to incorporate all available evidence,
whether published or unpublished (imp as there is a
problem with journals: they tend to publish studies that
have positive results so if only published results are
included the systematic reviews would deal with results
of positive studies) to avoid bias: Publication bias
Literature search bias:
Database Bias - No single database is likely to contain all
published studies on a given subject. (so to avoid it a min
of 2 databases should be used)
Publication Bias - selective publication of articles that
show positive treatment of effects and statistical
significance. (include published an unpublished bias)
Hence, it is important to search for unpublished studies
through a manual search of
1. Conference proceedings
2. Correspondence with experts
3. Search of clinical trials registries.
English-language bias - occurs when reviewers exclude
papers published in languages other than English- some
studies are conducted in other languages.
Unpublished Literature:
 Not all known published trials are identifiable in
Medline (depending on topic)
 Only 25% of all medical journals in Medline
 Non-English language articles are under-represented
in Medline (and developing countries)
Appraise each paper for methodological quality:
After search for studies is done, the authors should appraise
each paper or study they included in their systematic review.
All the types of bias involved.
Systematic review authors should include a methodological
quality assessment for each study.
This appraisal process is done by 2 independent researchers
and they should totally explain in the systematic review the
process of the appraisal. Which tool they chose to judge the
quality?

After appraisal of the studies, evaluating the quality of studies

in their systematic reviews.
Judge if the data can combine in one mathematical conclusion
– conduct meta-analysis or display results without combining
it
So, the 5th step would be combing results and judging whether
meta-analysis is possible or not.
Forest plot: if authors can produce a meta-analysis, display all
the results on a diagram called forest plot (to display the meta-
analysis
If can produce a meta-analysis- termed pooled estimate with
CI
The results of systematic reviews are commonly displayed
using a Forest plot.
This summarises the magnitude of the intervention effect
(e.g. odds ratio, risk ratio) and 95% CI for each of the
included studies.
This information is then combined to produce a pooled
estimate and 95% CI of the effect size.

Three elements of critical appraisal:

Part 1:
1. Are the methods valid?
Part 2: related to the results
2. What are the results?
3. Will the results help me?

Part 1: Are the methods valid?

Ask specific questions:
First question will be dealing with systematic review input.
1) Did the systematic review include only RCTs as the basic
blocks of this review?
Why is this question important?
Therapy questions (discussing the effectiveness of a specific
drug) are best answered in RCTs.
BY combining all relevant RCTs the bias is reduced and the
highest level of evidence is achieved.
SR that includes non-randomized trials will lower the quality
of evidence, should be avoided.
 2) Does the systematic review have methods section that
describes:
(a) Finding and including ALL relevant trials? Or did
it miss some trials that would offer an answer to that
question
 The method section should be scrutinized to determine
whether it describes how the investigators found ALL
relevant trials. If not drop the review and look
again.
 We should make sure the investigator looked beyond
the standard bibliographic data bases (at least 2).
 A more rigorous SR would include hand searching
journals, reference lists, conference proceedings,
data banks of pharmaceutical firms.
 Contacting the authors of published articles.
 Checking on going trials (Clinical Trials.gov)
 Negative trials are more prone to be unpublished.
 Restricting language also results in selection bias in
the included articles, and bias the conclusions of the
SR.
Ideally, the inclusion and exclusion of studies should be
represented in a flow chart showing the number of studies
retrieved from the search and how many were then excluded
and why? – PRISMA flow chart
Quantitative synthesis: meta-analysis

(b) How the validity of the individual studies was

assessed?
The methods section of systematic review should include
clear process of how the included studies were tested for
quality (validity)- 2 independent researchers
This is done by a minimum of two independent researchers
using the criteria of critical appraisal of RCTs (selection bias:
randomization – concealment of allocation – performance bias
-attrition bias& other biases)
The results could be: low risk of bias/High risk of bias /Un
clear ???? ( write to author)
Other tools to check validity are also available.
2 methods by which the systematic review can show how was
the validity of the included trials assessed.

The bar indicated the percentage of the satisfaction.

Eg. Random sequence generation: around 80% of the studies
has low risk of bias and 20% had unclear risk of bias.

Instead of percentages, the trials themselves are listed and the

score of each was given to that specific trial.
Part 2: Results
(a) Were the results consistent from study to study?-
heterogeneity
(reporting similar not identical results)
Are the studies included going the same direction?
This means whether or not most of the included studies
reported similar results in other words are there differences,
lack of consistency i.e. (heterogeneity) between the final
results reported which is usually represented by CI.
This can be tested by eye balling (overlapping CI), and
statistically.- next lecture
What are the reasons for heterogeneity?
Chance- difference in studied populations, doses of
medications, duration of therapy, outcome measurements.
(b) Importance of results
 What is the magnitude of the treatment effect?
Relative risk, or absolute risk reduction, number needed to
treat… etc
 How precise is the treatment effect?
(Confidence Interval); the narrower the CI the more precise
the results.
(c) Were individual patient data used in the analysis
(or aggregate data)? (not a common practice)
Less frequent point to consider whether the author used
individual patient data rather than summary tables or
published reports.
We would feel more confident in conclusions if individual
patient data were used as it allows for subgroup analyses.
(d) Applicability of the results:
Sometimes the results have a high magnitude but are not
suitable for that patient.
Is our patient so different from those in the study that its
results cannot apply?
Is the treatment feasible in our setting?
What are our patient’s potential benefits and harms from the
therapy?