CRITICAL APPRAISAL

Noor Pramono Noerpramana

Clinical Epidemiology and Biostatistics
RESEARCH QUESTION

This is the essential question the study is set
up to answer. Most studies are concerned
with assessing one of four types of
objectives:
to assess the magnitude of a health problem
or health factor;
to assess the efficacy of an intervention;
to asses the causal relation between one
factor or set of factors and the disease or
outcome of interest; or
to asses the natural history of disease
HYPOTHESIS

A proportion about the relationship between two or
more factors or variables of interest.
The purpose of the study is to collect data which
will allow the researcher to test the hypothesis.
In some studies the hypothesis may not be stated
but it is implicit. (However, not all studies test
hypotheses. Some times descriptive studies are
called hypothesis generating).
Hypotheses are often started in a null form, so as
to allow them to be refuted.
STUDY FACTOR

The exposure or variable of interest that is
hypothesized to be related to the health
problem, disease or outcome of interest.
As the independent variable. For example, in a
study of salt intake and its relationship to
blood pressure, the study factor is salt intake.
Must be quantifiable, but it may be assessed
by a variety of means both direct and indirect,
objective and subjective
OUTCOME FACTOR

The event or occurrence that is supposed to
have happened as a result of the influence
of the study factor.
The example: the outcome factor is blood
pressure, it being seen to be influenced by
the study factor, salt intake. The outcome
factor is also know as the dependent
variable
May also be measured by a variety of
methods direct and indirect, objective and
subjective
POPULATION
The whole collection of units
(individuals, cases, events) from
which a sample may be drawn; and
which must be defined geographically
and temporally.

REFERENCE POPULATION
This is an abstract concept of the
people to whom the researcher wishes
to refer results. Also known as the
target population.
SOURCE POPULATION

In setting up a study it is frequently
impossible or impractical to include all of
the reference population and the
researcher will have to resort to a subset
known as the source population.
The population that is included in the study
actually comes from and its specifications
will be more limited than the reference
population
SAMPLING FRAME

To access the source population the researcher
requires a sampling frame or study frame
which will represent the source population.
The simplest sampling frame would be complete
listing of all the population.
From this sampling frame a sample is collected
which may consist of all or part of the frame. As
not all of those identified in the sample will be
available or willing to participate in a study,
those that do will comprise the final study
subjects.
STUDY SAMPLE

This is the subject selected by
random or non-random means from
the sampling frame to represent the
source population.
In some studies it is possible to study
the entire source population for the
period of the study and there is no
necessity to draw a sample
SAMPLING

A process for selecting a study sample
from the sampling frame when it is
impractical to use the complete source
population.
The usual preferred method is random
sampling whereby inclusion in the study
sample is decided by chance.
The objective of sampling is to select a
non biased sample which can be studied.
The hierarchy of sampling can be
illustrated in the sample given below:
Source Population- the broad group of people from
whom the subjects will be obtained, i.e., school
children in the Hunter Region.
Sampling Frame the list of potential subjects
from which a sample will be drawn, i.e., class lists
of all Public Schools in the greater Newcastle area
Reference Population the
abstract concept of people
to whom the results would
apply-in these case-
Australian school children
Study Subjects
Sample
Sampling Frame
Source Population
Reference Population
A researcher wants to study the prevalence of
hearing defects in Australian school children.

Sampling Frame the list of potential
subjects from whom which a sample will
be drawn, i.e. class lists of all Public
Schools in the greater Newcastle area.
Sample- the subjects who are selected to
take part in the study, e.g. a random
selection of 25% of each class in each of
the 8 schools which have been randomly
selected from the 40 in the greater
Newcastle area
Sample Subjects- these provide the
data, e.g. 90% of children had parental
approval to take part in the study.
BIAS

Any effect at any stage of investigation (or
inference) tending to produce results that
depart systematically (i.e., not randomly)
from the truth.
Many varieties of bias have been
described including measurement bias,
selection bias and confounding bias

CONFOUNDER

A factor that distorts the apparent magnitude of
the effect of the study factor on the outcome
factor.
A confounder is it self a determinant of the
outcome of interest, and is unequally distributed
among the subjects that are exposed or not
exposed to the study factor.
For example, in a study of the influence of
maternal nutrition on the birth weight of
newborns (where it is hypothesised that women
with a poor diet will have low birth weight
infants), smoking is a confounding factor if
women with poor dietary habits are also more
likely to smoke (and it is known that smoking is
of itself a risk factor for low birth weight infants)
RISK
Probability that an event will occur, e.g.,
that an individual will die or become ill within a
stated period of time or age

ATTRIBUTABLE RISK
The difference between the rate of disease
in the exposed population and the rate in the
non-exposed population.
It is a measure of the amount of disease
that might be reasonably attributed
to the exposure in question.

RISK FACTOR
An attribute or exposure that increases the
probability occurrence of disease or a specified
outcome. This defines risk factor as a determinant
of the outcome, though risk factor can be used to
denote association with a specified outcome that
is not necessarily causal (i.e., a risk marker or risk
indicator, (e.g., smoking and positive parental
history are considered risk factors for coronary
artery disease.

PREVALENCE
The total number of all individuals who have the
disease or factor or interest at a particular time or
during a particular period ( the numerator)
divided by the population at risk of having the
disease at this time ( the denominator)

INCIDENCE

The number of new cases of a disease
occurring in a defined population within a
specified period of time.
Usually expressed as the incidence rate
which has the number of new cases of a
disease in a defined period of time as the
numerator and the number of persons in
the stated population in which the cases
occurred as the denominator.
RELATIVE RISK

It is one measure of the strength of effect of an
exposure on the likelihood of a particular
outcome and is the ratio of risk of disease or
death among those exposed to the study factor,
to the risk of disease or death among those not
exposed to the study factor.
Note, that to calculate relative risk, it is
necessary to use incidence figures for disease or
death. As some study designs preclude this
possibility, an approximation to relative risk,
called the odds ratio is used instead.
(STATISTICAL) POWER
This is the probability that a study will
detect a statistically significant difference
when a difference really exists.

COHORT STUDY
The study population is classified according
to exposure to the study factor prior to the
development of the outcome of interest.
The incidence of the outcome is then
ascertained in groups with different
exposure status
CASE CONTROL (OR CASE REFERENT) STUDY
The study sample consists of a group with the
outcome of interest (cases) and a sample of
controls representative of the same source
population from which the cases arose. The
prevalence of exposure to the study factor in each
group at a time prior to the development of the
outcome in the cases is compared

RANDOMIZED CONTROLLED TRIAL
An epidemiological experiment whereby the
incidence of the study outcome(s) is compared in
groups assigned by chance to different
interventions and usually one group (the control
group) who are not exposed. The aim of
randomization is to achieve an equal or at least
unbiased distribution of confounders, particularly
unmeasured or potential confounders in exposed
and non exposed groups
STUDY VALIDITY
The degree to which the inference drawn
from a study, especially generalisations
extending beyond the study sample, are
warranted when account is taken of the
study methods, the representativeness of
the study sample, and the nature of the
population from which it is drawn. Two
varieties of study validity are
distinguished (Last):
1. Internal Validity. The groups being compared (treated
vs. non-treated, exposed vs. non-exposed or cases vs.
controls) have been selected and measurements
made in such a way that the results can be considered
a good approximation to truth.

2. External Validity. The subjects in the study are
selected and described in such a way that the results,
given internal validity, can be applied or generalized
outside of the study sample.

Threats to internal validity come from systematic error or
bias. Threats to external validity come from the selection
and sampling of subjects for a study . In order to see
whether a study has external or generalizability it is
important to have a clear idea of where the subjects have
come from
MEASUREMENT ISSUES

Epidemiology is very much about making measurements
and about being critical of the measurements that are
made by ourselves and other. There are two issues with
all measurements which will be defined here. They are
Validity and repeatability.

1. Measurement validity. An expression of the degree to
which a measurement measures what it purports to
measure, e.g., an autopsy has more validity than a
biopsy or x-ray diagnosis

2. Repeatability. Critical to the validity of test is its
repeatability, i.e., the degree of stability exhibited
when the measurement is repeated under identical
conditions. Most biological measurements show some
variation when repeated.
STUDY EFFICIENCY

This addresses the issues of the cost of obtaining
information. the effects or end-results achieved
in relation to the effort expended in terms of
money, resources, and time (Last). Case control
studies for example, are considered to be more
efficient than longitudinal studies at obtaining
information about the effect of an exposure or an
outcome. This is because less effort needs to go
into acquiring data by comparing diseased and
non-diseased people for their exposure versus
comparing exposed and non-exposed people for
whether they develop an outcome
Critical Appraisal
using
Epidemiology a Basic Methods. Critical
Appraisal ( Including Study
Terminology).
University of Newcastle; 1994.
CRITICAL APPRAISAL WORKSHEET (CAW)
A General approach to critical appraisal of scientific literature
The worksheet is a matrix composed of nine rows of
questions about the paper, and three columns in which you
are:

firstly whether the information to answer the question is
available in the paper;

secondly, whether they was a problem with the way in
which it was handled and;

thirdly, whether the problem identified was sufficient to
affect the validity of the study
CRITICAL APPRAISAL WORKSHEET (CAW) (1991)
(1) (2) (3)
Can you find this
information in the
paper?
Is the way this was done a
problem?
Does this problem
threaten the
validity of the
study
1. What is the
research question
and/or hypothesis?



2. What is the study
type?


3. What is the
reference
population? What
are the sampling
frame and
sampling method?
Is it concerned with the
impact on an
intervention, causality, or
determining the
magnitude of a health
problem?
Is the study type
appropriate to the
research question ?

Is the sampling frame
appropriate for the
reference population
Is there selection bias?






If not, how useful
are the results
produced by this
type of study?
Do these threaten
the external
validity of the
study
CRITICAL APPRAISAL WORKSHEET (CAW) (1991)

(1) (2) (3)
Can you find this
information in the
paper?
Is the way this was
done a problem?
Does this problem
threaten the validity of
the study

4. What are the study
factors and how are
they measured?

5.What are the outcome
factors and how
are they
measured?


Is there measurement
error?


a. Are all relevant
outcomes assessed?
b.Is there measurement
error?


Is measurement error
an important source of
bias?

a.How important are
omitted outcomes?
b.Is measurement error
an important source
of bias?
CRITICAL APPRAISAL WORKSHEET (CAW) (1991)

(1) (2) (3)
Can you find this
information in the
paper?
Is the way this was
done a problem?
Does this problem
threaten the validity of
the study

6. Are these sources of
bias relevant to the
study? Selection
bias, recall bias,
ascertainment
bias, confounding
bias, non-random
assignment,
incomplete follow-
up

Have steps been taken
to avoid these
biases?

Is the internal validity
of the study threatened
by bias?
CRITICAL APPRAISAL WORKSHEET (CAW) (1991)
(1) (2) (3)
Can you find this
information in the
paper?
Is the way this was done a
problem?
Does this problem
threaten the
validity of the
study
7. Are sample size
issues considered?
Is the power of the
study indicated?

8. Are statistical
methods
described?

Is the sample size sufficient
to detect clinically/
socially meaningful
differences

Are the proposed statistical
methods appropriate for
addressing the stated
research question and/or
hypothesis? Are the
proposed statistical
methods appropriate for
the data?





Do the conclusions
drawn follow
logically from the
results of the
analyses?
CRITICAL APPRAISAL WORKSHEET (CAW) (1991)
(1) (2) (3)
Can you find this
information in the
paper?
Is the way this was done a
problem?
Does this problem
threaten the
validity of the
study
9. What conclusions
did the authors
reach about the
research question?
Did they generate
new hypotheses?
Do you agree with
the conclusions?
Do the results apply to the
population in which you
would be interested
Do you accept the
results of this
study?
Critical Appraisal
using
The Pocket Guide to Critical Appraisal
Iain K. Crombie

BMJ
THE STANDARD APPRAISAL QUESTIONS

ARE THE AIMS CLEARLY STATED?
WAS THE SAMPLE SIZE JUSTIFIED?
ARE THE MEASURMENTS LIKELY TO BE VALID AND
RELIABLE?
ARE THE STATISTICAL METHODS DESCRIBED?
DID UNTOWARD EVENTS OCCUR DURING THE STUDY?
WERE THE BASIC DATA ADEQUATELY DESCRIBED?
DO THE NUMBERS ADD UP
WAS THE STATISTICAL SIGNIFICANCE ASSESSED?
WHAT DO THE MAIN FINDINGS MEAN?
HOW IS NULL FINDINS INTERPRETED?
ARE IMPORTANT EFFECTS OVERLOOKED?
HOW DO THE RESULTS COMPARE WITH PREVIOUS REPORTS?
WHAT IMPLICATIONS DOES THE STUDY HAVE FOR YOUR
PRACTICE?
THE COMPLETE LIST FOR APPRAISAL
OF SURVEYS

The essential questions
Who was studied?
How was the sample obtained?
What was the response rate?.

The detailed questions*
Design
Are the aims clearly stated?
Is the design appropriate to the stated objectives?
Was the sample size justified?
Are the measurements likely to be valid and reliable?
Are the statistical methods described?
Is there a suggestion of haste?
Conduct
Did untoward events occur during the study?

Analysis
Where the basic data adequately described?
Do the numbers add up?
Was the statistical significance assessed?
Were the findings serendipitous?

Interpretation
What do the main findings mean?
How could selection bias arise?
How are null findings interpreted?
Are important effects overlooked?
Can the results compare with previous reports?
How do the results compare with previous reports?
What implications does the study have for your practice?
THE COMPLETE LIST FOR THE
APPRAISAL OF COHORT STUDIES

The essential questions
Who exactly has been studied?
Was a control group used? Should one have been used?
How adequate was the follow-up?
The detailed questions*
Design
Are the aims clearly stated?
Is the design appropriate to the stated aims?
Was the sample size justified?
Are the measurements likely to be valid and reliable?
Were relevant outcome measures ignored?
Are the statistical methods described?
Conduct
Did untoward events occur during the study?
Analysis
Did the analysis allow for the passage of time?
Do the numbers add up?
Were the basic data adequately described?
Was statistical significance assessed?
Interpretation
What do the main findings mean?
What else might influence the observed outcome?
How are null findings interpreted?
Are important effects overlooked?
How do the results compare with previous reports?
What implications does the study have for your practice?
THE COMPLETE LIST FOR THE
APPRAISAL OF CLINICAL TRIALS

The essential questions
Were treatment randomly allocated?
Were all the patients accounted for?
Were outcomes assessed bind?
The detailed questions*
Design
Are the aims clearly stated?
Was the sample size justified?
Are the measurements likely to be valid and reliable?
Could the choice of subjects influence the size of treatment effect?
Were the ambiguities in the description of the treatment and its
administration?
Are the statistical methods described?
Could lack of blinding introduce bias?
Are the outcomes clinically relevant?
Conduct
How was the randomization carried out?
Did untoward events occur during the study?
Analysis
Were the treatment groups comparable at baseline?
Were results analyzed by intention to treat?
Was the statistical significance assessed?
Were the basic data adequately described?
Do the numbers add up?
Were side-effect reported?
Interpretation
What do the main findings mean?
How are null findings interpreted?
Are important effects overlooked?
How do the results compare with previous
reports?
What implications does the study have for
your practice?
THE COMPLETE LIST FOR THE
APPRAISAL OF CASE-CONTROL STUDIES

The essential questions
How were the cases obtained?
Is the control group appropriate?
Were data collected the same way for cases and
controls?
The detailed questions*
Design
Are the aims clearly stated?
Is the method appropriate to the aims?
Was the sample size justified?
Are the measurements likely to be valid and reliable?
Are the statistical methods described?
Conduct
Did untoward events occur during the study?
Analysis
Were the basic data adequately described?
Do the numbers add up?
Was there data-dredging?
Was the statistical significance assessed?
Interpretation
What do the main findings mean?
Where are the biases?
Could there be confounding?
How are null findings interpreted?
Are important effects overlooked?
How do the results compare with previous
reports?
What implications does the study have for your
practice?
THE COMPLETE LIST FOR
APPRAISAL OF REVIEW PAPERS

The essential questions
How were the papers identified?
How was the quality of papers assessed?
How were the results summarized?
The detailed questions*
Design
Is the topic well defined?
Are the statistical methods described?
Conduct
Were the detailed study designs reviewed?
Was missing information sought?
Analysis
Were the basic data adequately described?
Was publication bias taken into account?
Was heterogeneity of effect investigated?
Interpretation
What do the main findings mean?
Are there other findings which merit attention?
Are the conclusions justified?
How do the findings compare with previous
reports?
What implications does the study have for your
practice?
Evidence Based Medicine
Evidence-based medicine (EBM) requires the integration of the best
research evidence with our clinical expertise and our patients
unique values and circumstances
By best research evidence we mean valid and clinically relevant
research, often from the basic sciences of medicine, but
especially from patient-centered clinical research into the
accuracy of diagnostic tests (including the clinical examination),
the power of prognostic markers, and the efficacy and safety of
therapeutic, rehabilitative, and preventive regimens. New
evidence from clinical research both invalidates previously
accepted diagnostic tests and treatments and replaces them
with new ones that are more accurate, more efficacious, and
safer
Evidence Based Medicine
By clinical expertise we mean the ability to use our clinical skills and
past experience to rapidly identify each patients unique health state
and prognosis, their individual risks and benefits of potential
interventions, and their personal circumstances and expectations
By patient values we mean the unique preferences, concerns and
expectations each patient brings to a clinical encounter and which
must be integrated into clinical decisions if they are to serve the
patient.
By patient circumstances we mean their individual clinical state and
the clinical setting.
Critical Appraisal
using
Evidence-Based Medicine.
How to Practice and Teach EBM.
Sharon E. Straus, W. Scott Richardson, Paul
Glasziou, R. Brian Haynes.


Third edition (CD room); Elseiver Churchill Living stone



Critical Appraisal using EBM Version needs worksheets:
1. Diagnosis and screening worksheet
2. Harm / Etiology worksheet
3. Prognosis worksheet
4. Systematic Review worksheet
5. Therapy or Prevention worksheet
6. Etc


1. DIAGNOSIS WORKSHEET
Are the results of this diagnostic study valid

Was there an independent , blind comparison
with a reference (gold) Standard of diagnosis

Was the diagnostic test evaluated in an
appropriate spectrum of patients (like those in
whom it would be used in practice)?

Was the reference standard applied regardless
of the diagnostic test result?

Was the test (or cluster of tests) validated in a
second, independent group of patients?

Are the results of this diagnostic study valid?

Sample Calculation
Target disorder
(iron deficiency anemia)
Totals
Present Absent

Diagnostic
test result
(serum
ferritin)
Positive

(<65 mmol/L)
731

a
270

B
1001

a+b
Negative

(>65 mmol/L)
78

c
1500

d
1578

c+d
Totals 809

a+c
1770

b+d
2579

a+b+c+d
Sensitivity = a/(a+c) = 731/809 = 90%

Specificity = d/(b+d) = 1500/1770 = 85%

Likelihood ratio for a positive test result =
LR+=sensitivity/(1-specifity)=90%/15%=6

Likehood ratio for a negative test result =
LR-=(1-Sensitivity)/specificity=10%/85%=0.12

Positive predictive value = a/(a+b)=731/1001=73%

Negative predictive value = d/(c+d)=1500/1578=95%

Pre-test probability (prevalence) = (a+c) / (a+b+c+d)
= 809/2579=32%

Pre-test odds=prevalence/(1-prevalence)=31%/69%=0.45

Post-test odds = ( pre-test odds) x LR

Post-test probability=(post-test odds)/post-test odds+1)

YOUR CALCULATIONS
Target disorder

Totals
Present Absent

Diagnostic
test result
Positive a b a+b
Negative c d c+d
Totals a+c b+d a+b+c+d
Can You apply this valid, important evidence about
a diagnostic test in caring for your patient?
Is the diagnostic test available,
affordable, accurate, and precise in your
setting?
Can you generate a clinically sensible
estimate of your patients pre-test
probability (from personal experience,
prevalence statistics, practice database,
or primary studies)?
Are the study patients similar to your
own?
Is it unlikely that the disease
possibilities or probabilities have
changed since the evidence was
gathered?
Can You apply this valid, important evidence about
a diagnostic test in caring for your patient?
Will the resulting post-test probabilities
affect your management and help your
patient?
Could it move you across a test-
treatment threshold ?
Would your patient be a willing partner
in carrying it out?
Would the consequences of the test help
your patient?
Additional Notes:
2. HARM WORKSHEET

Are the results of this harm study valid?
Were there clearly defined groups of
patients, similar in all important ways
other than exposure to the treatment or
other cause?
Were treatments/exposures and clinical
outcomes measured in the same ways in
both groups (was the assessment of
outcomes either objective or blinded to
exposure)?
Was the follow-up of study patients
sufficiently long and complete?
Do the results satisfy some diagnostic
tests for caution?
Are the results of this harm study valid?
Is it clear that the exposure preceded the
onset of the outcome?
Is there a dose-response gradient?
Is there positive evidence from a
dechellenge-rechallenge study?

Is the association make biological sense?

Does the association make biological
sense?
What is the magnitude of the association
between the exposure and outcome?

What is the precision of the estimate of
the association between exposure and
outcome
Are the valid results from this harm study important?
Adverse outcome

Totals
Present
(Case)
Absent
(control

Exposed to
the
treatment
Yes
(cohort)
a b a+b
No
(cohort)
c d c+d
Totals a+c b+d a+b+c+d
In randomized trial or cohort study: relative
risk=RR=[a/(a+b)]/[c/(c+d)]
In case-control study : odds ratio (or relative odds)=OR=ad/bc
Should this valid, potentially important
results changes the treatment of your
patient?
1. Do the results apply to our patient?
2. Is our patient so different from those in the study that its
results dont apply?
3. What are the patients risks of the adverse event?
4. To calculate the NNH ( number of patients we need to
treat to harm one of them) for any odds ratio (OR) and
our patients expected event rate for this adverse event
if they were not exposed to this treatment (PEER):
PEER (OR-1)+1
NNH = ------------------------------
PEER(OR-1)x(1-PEER)
5. What are our patients preferences, concerns and
expectations from this treatment?
6. What alternative treatments are available?
Additional notes:

3. PROGNOSIS WORKSHEET

Are the results of this prognosis study valid ?

1. Was a defined, representative sample of patients
assembled at a common ( usually early) point in the
course of their disease?
2. Was patient follow-up sufficiently long and complete?
3. Were objective outcome criteria applied in a blind
fashion?
4. If subgroups with different prognoses are identified:
- Was there adjustment for important prognostic factors?
- Was there validation in an independent group (test
set) of patients?
Are the valid results of this prognosis study
important?
1. How likely are the outcome over time?
2. How precise are the prognostic estimates?
IF YOU WANT CALCULATE A CONFIDENCE INTERVAL
AROUND THE MEASURE OF PROGNOSIS
Clinical measure Standard error (SE) Typical calculation of
CI
Proportion ( as in the
rate of some
prognostic event, etc.)
where:
n=the number of
patients
P=the proportion of
these patients who
experience the event

__________
[p x (1-p)/n]

Where p is proportion
and n is number of
patients
If p=24/60=0.4
(or40%) and n 60:
_____________
SE =[0.4x(1-0.4)/60]
= 0.063 (or 6.3%)
95%CI is
40%1.96x6.3% or
27.6% to 52.4 %
n from your evidence:
p from your evidence:
Our calculation:
SE: ___________
95% CI:
Can you apply this valid, important evidence
about prognosis in caring for your patient?
1. Do the results apply to our patient?
2. Is our patient so different from those in the study that its
results cannot apply?
3. Will this evidence make a clinically important impact on
our conclusions about what to offer or tell our patient?
Additional notes:
4. SYSTEMATIC REVIEW (OF THERAPY) WORKSHEET

Are the results of this systematic review valid?

1. Is this a systematic review of randomized trials?
2. Does it describe a comprehensive and detailed search for
relevant trials?
3. Were the individual studies assessed for validity?
4. Were the individual patient data used in the analysis ( or
aggregate data)?
The numbers in the body of the tables are the NNTs for the
corresponding odds ratio (OR) at that particular patients
expected event rate (PEER)


Are the valid results of this systematic review
important?

1. Are the results consistent across studies?
2. What is the magnitude of the treatment effect?
3. How precise is the treatment effect?

Translating odd ratio to NNTs

When the odds ratio (OR)<1
This table applies when a bad outcome is prevented by therapy

OR < 1


Patients
expected
event
rate
(PEER)
0.9 0.8 0.7 0.6 0.5
0.05 2.09
a
104 69 52 41
b
0.10 110 54 36 27 21
0.20 61 30 20 14 11
0.30
0.40
46
40
22
19
14
12
10
9
8
7
0.50 38 18 11 8 6
0.70 44 20 13 9 6
0.90 101
c
46 27 18 12
d
a
The relative risk reduction (RRR) here is 10%
b
The RRR here is 49%
c
For any OR, NTT is lowest when PEER=0.50
d
The RRR here is 9%

When the odds ratio (OR)>1
This table applies both when a good outcome is increased by therapy
and when a side effect is caused by therapy

OR > 1


Patients
expected
event
rate
(PEER)
1.1 1.2 1.3 1.4 1.5
0.05 212 106 71 54 43
0.10 112 57 38 29 23
0.20 64 33 22 17 14
0.30 49 25 17 13 11
0.40
0.50
43
42
23
22
16
15
12
12
10
10
0.70 51 27 19 15 13
0.90 121 66 47 38 32
Can you apply this valid, important evidence from
systematic in caring for your patient ?

1. Do these results apply to our patient ?
2. Is our patient so different from those in the study that its
results cannot apply?
3. Is the treatment feasible in our setting?
4. What are our patients potential benefits harms from the
therapy?
5. Method I : In the OR tables above, find the intersection
of the closest OR from the systematic review and our
patients expected event rate (PEER)
6. Method II : To calculate the NNT from any OR and PEER

1-[PEERx(1-OR)]
NNT= ----------------------------
(1-PEER)xPEERx(1-OR)
7. Are our patients values and preferences satisfied by the
regimen and its consequences ?
8. Do we and our patient's have clear assessment of their
values and preferences?
9. Are they met by this regimen and its consequences?
Should you believe apparent qualitative differences
in the efficacy of therapy in some subgroups of
patients? Only if you can say yes to all of the
following ?
1. Do they really make biologic and clinical sense?
2. Is the qualitative difference both clinically (beneficial for
some but useless or harmful for others) and statistically
significant?
3. Was this difference hypothesized before the study began
(rather than the product of dredging the data)?
4. Was this one of just a few subgroup analyses carried out
in this study?
5. Has the result been confirmed in other independent
studies?
Additional Notes:
5. THERAPY WORKSHEET
Are the results of this single preventive or
therapeutic trial valid?

1. Was the assignment of patients to treatments
randomized?
2. Was the randomization list concealed?
3. Was follow-up of patients sufficiently long and complete?
4. Were all patients analyzed in the groups to which they
were randomized?
5. Were patients, clinicians, and study personal kept" blind
to treatment?
6. Were the groups treated equally, apart from the
experimental treatment?
7. Were the groups similar at the start of the trial apart
from the experimental therapy?
Are the valid results of this randomized trial
important?

1. What is the magnitude of the treatment effect?
2. How precise is the estimate of the treatment effect?
SAMPLE CALCULATIONS
Occurrence of diabetic
neuropathy at 5 years among
insulin-dependent diabetics in
the DCCT trial
Relative risk
reduction
(RRR)
Absolute risk
reduction
(ARR)
Number
needed to
treat (NNT)
Usual insulin
regimen
control event
rate (CER)
Intensive
insulin
regimen
experimental
event rate
(EER)
CER EER
CER
CER - EER 1/ARR
9.6% 2.8% 9.6% - 2.8%
9.6%
=71%
9.6% - 2.8%
= 6.8 %
1/6.8%
= 15 patients
95% CI
a
4.4% to 9.2% 11 to 23
Are the valid results of this randomized trial
important?

a
95% confidence interval (CI) on an NNT
= 1/(limits on the CI of its ARR)
CER x ( 1-CER) EER x (1-EER)
= 1.96 ------------------------- + -----------------------------
number of control patients number of experimental patients

0.96 x 0.904 0.028 x 0.972
= 1.96 ----------------------- + ---------------------
730 711
= 2.4%
YOUR CALCULATIONS
Relative risk
reduction
(RRR)
Absolute risk
reduction
(ARR)
Number
needed to
treat (NNT)
CER EER CER EER
CER
CER - EER 1/ARR
95% CI
Can you apply this valid, important evidence about
therapy in caring for your patient ?

1. Do these results apply to our patient ?
2. Is our patient so different from those in the study that its
results cannot apply?
3. Is the treatment feasible in our setting?
4. What are our patients potential benefits harms from the
therapy?
5. Method I : f
Risk of the outcome in our patient, relative to patients in
the trial
Expressed as a decimal: __________
NNT/f = ---------/--------------=
(NNT for patients like ours)
6. Method II : 1/(PEER xRRR)
Our patients expected event rate if they received the
control treatment (PEER)= ___
1/(PEER x RRR) = 1/ =
(NNT for patients like ours)
7. Are our patients values and preferences satisfied by the
regimen and its consequences ?
8. Do we and our patients have a clear assessment of their
values and preferences?
9. Are they met by this regimen and its consequences?

Additional notes:
RINGKASAN

Setelah membaca artikel dan belum jelas
desain penelitiannya (study design dasar),
maka worksheet yang digunakan adalah:
I.1. Critical appraisal worksheet 1991,
Epidemiology A Basic Methods



Setelah membaca artikel dan telah jelas desain
penelitiannya (study design dasar), maka menggunakan
The Pocket Guide to Critical Appraisal By IAIN K.Crombie
dengan worksheet:
II.1. The complete list for the appraisal of surveys (p:35)
II.2. The complete list for the appraisal of cohort studies
(p:42)
II.3. The complete list for the appraisal of clinical trials
(p:49)
II.4. The complete list for the appraisal of case-control
studies (p:55)
II.5. The complete list for the appraisal of review papers
(p:62)
II.6. Baik point II.1 s/d I.5. perlu jawaban pertanyaan
&..dari I.3.1 I.3.3

RINGKASAN

Setelah membaca artikel dan telah jelas desain dan jenis
penelitiannya maka menggunakan format EBM yang
terdiri dari:
III.1 Diagnosis and screening
III.2 Harm / Etiology
III.3 Prognosis
III.4 Therapy or Prevention
III.5 Etc

The conclusions have to answer these questions:
1. Is the evidence from this study valid?
2. If valid, is this evidence important?
3. If valid and important, can you apply this evidence
in caring for your patients


RINGKASAN

DAFTAR PUSTAKA

1. Critical Appraisal ( Including Study Terminology). Epidemiology a
Basic Methods. Critical Appraisal ( Including Study Terminology).
Newcastle: University of Newcastle; 1994. p.12
2. Crombie IK. The Pocket Guide to Critical Appraisal: A Handbook
for Health Care Professionals. London: BMJ Publising group;
1996. p.23, 35, 42, 49, 62
3. Straus SE, Richardson WS,Glaziou P, Haynes RB. Evidence Based
Medicine: How to Practice and Teach EBM 3
rd
ed CD room;
Elseiver Churchill Living stone
GLOSSARY
Terms you are likely to encounter in your clinical
reading
Absolute risk reduction (ARR). See treatment effects
Allocation concealment. Occurs when the person who is enrolling a
participant into a clinical trial is unaware whether the next
participant to be enrolling will be allocated to the intervention or
control group
Case-control study. A study which involves identifying patients
who have the outcome of interest (cases) and control patients
without the same outcome, and looking back to see if they had the
exposure of interest
Case series. A report on a series of patients with an outcome of
interest. No control group is involved
Clinical practice guideline. A systematically developed statement
designed to assist clinician and patient decisions about appropriate
health care for specific clinical circumstances
Cohort study. Involves identification of two groups (cohort) of
patients, one that received the exposure of interest, and one that
did not, and following these cohorts forward for they outcome of
interest.
Confidence interval (CI). Quantifies the uncertainly in
measurement. It is usually reported as 95% CI, which is
the range of values within which we can be 95% sure
that the true value for the whole population lies. For
example, for an NNT of 10 with a 95% CI or 5 to 15, we
would have 95% confidence that the true NNT values
lies between 5 and 15
Control event rate (CER). See treatment effects
Cost- benefit analysis.
Assesses whether the cost of an intervention as worth
the benefit measuring both in the same units; monetary
units are usually used
Cost-minimization analysis. If health effects are
known to be equal, only costs are analyzed and the least
costly alternative is chosen

Cost-utility analysis. Converts health effects into
personal preferences (or sterilities) and describes how
much it costs for some additional quality gain(e.q. cost
per additional quality gain (e.q. cost per additional
quality-adjusted life-year, or QALY)
Crossover study design. The administration of two or
more experimental therapies one after the other in a
specified or random order to the same group of patients
Cross-sectional study, The observation of defined
population at a single point in time or time interval.
Exposure and outcome are determined simultaneously
Decision analysis ( or clinical decision analysis).
The application of explicit, quantitative methods that
quantity prognoses, treatment effects, and patient
values in order to analyze a decision under conditions of
uncertainty

Event rate. The proportion of patients in a group in
whom the event is observed. Thus, if of 100 patients,
the event is observed in 27, the event rate is 0.27.
Control event rate(CER) and experimental event rate
(EER) are used to refer to this in control and
experimental groups of patients, respectively. The
patient expected event rate (PEER) refers to the rate of
events we'd expect in a patient who received no
treatment or conventional treatment. See treatment
effects.
Evidence-based health care. Extends the application
of the principles of evidence-based medicine (see
below) to all professions associated with health care,
including purchasing management.
Evidence-based medicine (EBM). The conscientious,
explicit, and judicious use of current best evidence in
making decisions about the care of individual patients.
The practice of evidence-based medicine requires the
integration of individual clinical expertise with the best
available external clinical evidence from systematic
research and our patients unique values and
circumstances.
Experimental event rate (EER). See treatment
effects.
Inception cohort. A group of patients who are
assembled near the onset of the target disorder
Incidence The proportion of new cases of the target
disorder in the population at risk a specified time
interval.
Intention-to-treat analysis. A method of analysis for
randomized trials in which all randomly assigned to one
of the treatments are analyzed together, regardless of or
not they completed or received that treatment, in order
to preserve randomization.
Likelihood ratio (LR). The likelihood that a given test
result would be expected in a patient with the target
disorder compared with the likelihood that the same
result would be expected in a patient without the target
disorder. See Table A2.2 for calculations.
Meta-analysis. A systematic review that uses
quantitative methods to synthesize and e the results.
n-of-1-trials. In such trials, the patient undergoes pairs
of treatment periods organized so that one period
involves the use of the experimental treatment and the
other involves the use of an alternate or placebo
therapy. The patient and physician are blinded, if
possible, and outcomes are monitored. Treatment
periods are replicated until the clinician and patient are
convinced that the treatments are definitely different or
definitely not different.
Negative predictive value. Proportion of people with
a negative test who are free of the target disorder. See
also likelihood ratio.
needed to treat (NNT). The inverse of the absolute
risk reduction and the number of patients that need to
be treated to prevent one bad outcome. See treatment
effects
Odds. A ratio of the number of people incurring an
event to the number of people who have an event.
Odds ratio (OR). The ratio of the odds of having the
target disorder in the experimental group relative to the
odds in favor of having the target disorder in the control
group (in cohort studies or systematic reviews), or the
odds in favor of being exposed in participants with target
disorder divided by the odds in favor of being exposed in
control participants (without the target disorder). See
Table A2A for calculations.
Overview. See systematic review.
Patient expected event rate (PEER). See treatment
effects.
Positive predictive value. Proportion of people with a
positive test who have the targ disorder. See also
likelihood ratio.
Post-test odds. The odds that the patient has the
target disorder after the test is carried out [(pre-test
odds) x (likelihood )]
Post-test probability. The proportion of patients with
that particular test result who have the target disorder
[(post-test odds) / (1 + post-test odds)].
Pre-test odds. The odds that the patient has the target
disorder before the test is carried out [(pre-test
probability) / (1 - pre-test probability)].
Pre-test probability/prevalence. The proportion of
people with the target disorder in the population at risk
at a specific time (point prevalence) or time interval
(period prevalence). See also likelihood ratio.
Randomization (or random allocation). Method
analogous to tossing a coin to assign patients to
treatment groups (the experimental treatment is
assigned if the coin lands "heads"; a conventional,
"control" or "placebo" treatment is given if the coin lands
"tails
Randomized controlled clinical trial (RCT).
Participants are randomly allocated into an- experimental
group or a control group and followed over time for the
variables/outcomes of interest.
Relative risk reduction (RRR). See treatment effects.
Risk ratio (RR). The ratio of the risk in the treated
group (EER) to the risk in the control group (CER); used
in randomized trials and cohort studies. RR=ERR/CER.
Also called relative risk.
Sensitivity. Proportion of people with the target
disorder who have a positive test result. It is used to
assist in assessing and selecting a diagnostic
test/sign/symptom. See also likehood ratio.
SnNout. When a sign/test/symptom has a high
sensitivity (5n), a negative result (N) can help rule out
(out) the diagnosis. For example, the sensitivity of a
history of ankle swelling for diagnosing ascites is 93%;
therefore, if person does not have a history of ankle
swelling, it is highly unlikely that the person has ascites.
Specificity. Proportion of people without the target
disorder who have a negative test result. It is used to
assist in assessing and selecting a diagnostic
test/sign/symptom, See also likelihood ratio.
SpPin. When a sign/test/symptom has a high specificity
(Sp), a positive result (P) can help to rule in (in) the
diagnosis. For example, the specificity of a fluid wave for
diagnosing ascites is 92%; therefore, if a person does
have a fluid wave, he/she may have ascites.
Systematic review. A summary of the medical literature that uses
explicit methods to perform a comprehensive literature search and
critical appraisal of individual studies, and that uses appropriate
statistical techniques to combine these valid studies.
Treatment effects. The evidence-based journals (Evidence Based
Medicine and ACP journal Club) have achieved consensus on some
of the terms they use to describe both the good and the bad effects
of therapy. We will bring them to life with a synthesis of three
randomized trials in diabetes which individually showed that several
years of intensive insulin therapy reduced the proportion of patients
with worsening retinopathy to 13% from 38, raised the proportion
of patients with satisfactory hemoglobin A
1c
levels to 60% from
about 30%, and increased the proportion of patients with at least
one episode of symptomatic hypoglycemia to 57% from 23%. Note
that, in each case, the first number constitutes the "experimental
event rate" (EER), and the second number the "control event rate"
(CER). We will use the following terms and calculations to describe
these effects treatment.
When the experimental treatment reduces the probability of a bad
outcome (worsening - diabetic retinopathy)
RRR (relative risk reduction). The proportional
reduction in rates of bad outcomes between
experimental and control participants in a trial,
calculated as [EER CER]/CER accompanied by a 95%
confidence interval (CI). In the case of worsening
diabetic retinopathy, [EER CER]/CER = [13% -
38%]/38% = 66%.
AAR (absolute risk reduction). The absolute
arithmetic difference in rates of bad outcomes between
experimental and control participants in a trial,
calculated as [EER CER], and accompanied by a 95%
CI. In this case, [EER CER] = [13% - 38%] = 25(This
sometimes called the risk difference.)
NNT (number needed to treat). The number of
patients who need to be treated to achieve one
additional favorable outcome, calculated as l/ARR and
accompanied by a 95% CI. In this case, 1/ARR = 1/25%
= 4.
when the experimental treatment increases the
probability of a good outcome (satisfactory hemoglobin
A
1c
levels)
RBI (relative benefit increase). The proportional
increase in rates of good outcomes between experimental
and control patients in a trial, calculated as [EER
CER]/CER, and accompanied by a 95% confidence interval
(CI). In the case of satisfactory hemoglobin A
1c
levels,
[EER CER]/CER = [60% - 30%]/30% = 100%.
ABI (absolute benefit increase). The absolute
arithmetic difference in rates of good outcomes between
experimental and control patients in a trial, calculated as
[EER CER], and accompanied by a 95% CI. In the case
of satisfactory hemoglobin A
1c
levels, [EER CER]= [60%
- 30%]= 30%
NNT (number needed to treat). The number of
patients who need to be treated to achieve one additional
good outcome, calculated as 1/ARR and accompanied by a
95% C is case, 1/ARR = 1/30% = 3.
When the experimental treatment increases the
probability of a bad outcome (episodes of hypoglycemia))
RRI (relative risk increase). The proportional
increase in rates of bad outcomes between experimental
and control patients in a trial, calculated as [EER CER]/
CER, and accompanied by a 95% confidence interval
(CI). In the case of hypoglycemic episodes, [EER-CER]
= [57% - 23% ]/ 23% = 148%. (RRI is also used in
assessing the impact of "risk factors" for disease.)
ARI (absolute risk increase). The absolute arithmetic
difference in rates of bad outcomes between
experimental and control patients in a trial, calculated as
[EER CER], and accompanied by a 95% C1. In the
case of hypoglycemic episodes, [EER CER] = [57% -
23%] = 34%. (ARI is also used in assessing the impact
of "risk factors" for disease.)
NNH (number needed to harm). The number of patients
who, if they received the experimental treatment, would
result in one additional patient being harmed, compared
with patients who received the control treatment;
calculated as I/AAR and accompanied by a 95% CI. In
the case, 1/ARR=1/34% =3



Table A2.1
How to calculate LRs




Table A2.2
We can assume that there are four possible groups of
patients, as indicated ( a to d ) in the table. From these
we can determine the sensitivity and specificity as
follows:
Sensitivity = a/(a + c)
Specificity = d/ (b + d)
We can now use these to calculate the likelihood
ratio for a positive test result (LR+):
LR+ = sensitivity/ (1 specificity)
= [a/(a +c)]/[b/(b+d)]
Similarly, we can calculate the likelihood ratio for a
negative test result (LR-):
LR - = (1-sensitivity)/specificity
= [c/(a+c)]/[d/(b+d)]
Positive predictive value = a/(a+b)
Negative predictive value = d/(c+d)
Pre-test probability = (a+c)/(a+b+c+d)
SAMPLE CALCULATION

Suppose you have a patient with anemia and a
serum feritin of 60 mmol/L You come across a
systematic review* of serum ferritin as a
diagnostic
test for iron deficiency anemia, with the results
summarized in the following table.






Table A2.3
The results indicate that 90% or the patients with iron deficiency
anemia have a positive test result (serum ferritin < 65 mmol/L).
This is known as the sensitivity and is calculated as:

Sensitivity = a/(a+c) = 731/809 = 90%

The results also show that 85% of patients who do not have iron
deficiency anemia have a negative test result. This is referred to as
the specificity, calculated as:

Spesitivity = d/(b+d) = 1500/1770 = 85%

From the sensitivity and specificity, the positive (LR+) and negative
(LR-) likelhood ratios can be determinated

LR+ = sensitivity/(1-specificity) = 90%/15% = 6
LR- = (1 sensitivity/specificity) = 10%/85% = 0.12

The results indicate that 90% or the patients with iron deficiency
anemia have a positive test result (serum ferritin < 65 mmol/L).
This is known as the sensitivity and is calculated as:

Sensitivity = a/(a+c) = 731/809 = 90%

The results also show that 85% of patients who do not have iron
deficiency anemia have a negative test result. This is referred to as
the specificity, calculated as:

Spesitivity = d/(b+d) = 1500/1770 = 85%

From the sensitivity and specificity, the positive (LR+) and negative
(LR-) likelhood ratios can be determinated

LR+ = sensitivity/(1-specificity) = 90%/15% = 6
LR- = (1 sensitivity/specificity) = 10%/85% = 0.12

Thus from your calculation of LR+, you determine that your patients
positive test result would be about six times more likely to be seen
in someone with iron deficiency anemia than in someone without
the disorder.

Calculation of odds ratio/relative risk

The table below can be used to calculate the odds
ratio/relative risk for the use of trimethoprim-
sulfamethoxazole prophylaxis in cirrhosis:






Table A2.4