HSH719 Economic Evaluation 1

Topic 3:
CONTEXT: Methods and
Stages:
The Key Stages of an
Economic Evaluation
1

Deakin Health Economics

Economic Evaluation:
Context, Method and Stages
Learning objectives:
1. Know the logical stages of an economic
evaluation
2. Be familiar and know how to use quality
assessment criteria for economic
evaluations
3. Understand the different ways in which
the results of economic evaluations can be
presented.

Know how to calculate a simple ICER

Deakin Health Economics

Fig 1: The Steps in Undertaking an Economic Appraisal

Step One: Deciding upon the study question
Step Two: Clear statement of alternatives to be appraised
Step Three: Assessment of costs and benefits of both alternatives
a) Identification of the appropriate costs and benefits to include in the
appraisal
b) Measurement of resources used and saved by the program alternatives,
and the outcomes produced by each
c) valuing resources used (and saved) and valuing outcomes

Step Four: Adjusting for timing

Discounting for the time stream of costs and outcomes

Step Five: Adjusting for risk and uncertainty

Modeling and sensitivity analysis

Step Six: Making a decision

3

Calculating and using decision rules

a) Net present value of programme
Deakin Health Economics
b) Comparing cost-effectiveness
ratios

Figure 2: Basic Types of Economic Evaluation

COSTS

CONSEQUENCES
Identification

Measurement

Valuation
Health state preferences
(U)

Health state
changed

Effects (E)
or

Healthcare sector
(C1 )
Willingness-to-pay
(W)
Resources
consumed Patient and family
(C2 )

HEALTH CARE
PROGRAM

Other value created

(V)

Other value
created
Healthcare sector (S1 )

Other sectors (C3 )

Patient and family

Resources saved

(S2 )
Other sectors (S3 )

Deakin Health Economics

Resources saved

Global
willingness
-to-pay
(W)

Update of Figure 2
CONSEQUENCES
Identification

COSTS

Measurement

Valuation
Health state preferences (U)

Health state changed

Effects (E)

or
Healthcare sector
(C1 )

Resources
consumed

Willingness-to-pay (W)

Other sectors (C2)

Patient & family (C3)

Other value created

Other value created (V)

HEALTH CARE
PROGRAM

Global
willingnessto-pay (W)

Productivity
losses (C4)

Resources
saved

Health sector (S1 )

Other sectors (S2)
Patient & family (S3)
Productivity gains
(S4)

Drummond textbook p19 Figure 2.1: Components of economic evaluation in health care
5

Deakin Health Economics

Key Questions
Is the methodology employed in the study appropriate and are
the results valid?
If the results are valid, would they apply to my setting?

Reference: Drummond et al (2005)

CHEERS checklist

Deakin Health Economics

Study Frame
Gold et al 1995 (Washington Panel) outline two key aspects
on an economic appraisal
The Study Frame and the Study Design

The Study Frame covers the vital conceptualization of the

evaluation purpose and methods this is really about a
GOOD study question
Objectives; decision context; intended audience
Perspective and choice of evaluation techniques (CEA; CUA; CBA; CCA;
PBMA: etc) and ICERs
Choice of intervention(s) and comparators
Target population
Study boundaries
Time horizon (for intervention and tracking costs/outcomes)
7

Deakin Health Economics

Study Design
The Study Design specifies the key elements of data collection
and analysis
There are four basic decisions
Appropriateness of the analyses used to describe the intervention and
its effects on health outcomes (modelled versus a within trial
design)
Appropriateness of data collections for activities, costs and outcomes
for the intervention and comparators
Appropriateness of the analysis used to combine the information
(modelling assumptions; uncertainty analysis; sensitivity analysis;
discounting; shadow pricing)
Appropriateness of the interpretation/ conclusions, given the above

Deakin Health Economics

How can we decide if the study is any

good?
Criteria checklists
There are a few in the literature
Best known are:
Drummond 10 point Checklist (Chapter 3
textbook)
CHEERs checklist (in your readings)

Fundamentally cover similar ground

CHEERS includes more explicit criteria around
modelling studies
9

Deakin Health Economics

Drummonds Checklist
1. Was a well-defined question posed in answerable
form?
Did study examine both costs and effects of the service(s)
or programme(s)?
Did the study involve a comparison of alternatives?
Was a viewpoint for the analysis stated and was the study
placed in any particular decision-making content?

10

Deakin Health Economics

10

Drummonds Checklist
2. Was a comprehensive description of the competing
alternatives given?
(ie can you tell who did what to whom, where and how
often?)

Were any important alternatives omitted?

Was (Should) a do-nothing alternative (be) considered?

11

Deakin Health Economics

11

Drummonds Checklist
3. Was the effectiveness of the programmes or services
established?
Was this done through a RCT? If so, did the trial protocol
reflect what would happen in regular practice?
Was effectiveness established through an overview of clinical
studies?
Were observational data or assumptions used to established
effectiveness? If so, what are the potential biases in results?

12

Deakin Health Economics

12

Drummonds Checklist
4. Were all the important and relevant costs and
consequences for each alternative identified?
Was the range wide enough for the research question at
hand?
Did it cover all relevant viewpoints?
Were capital costs, as well as operating costs, included?

13

Deakin Health Economics

13

Drummonds Checklist
5. Were costs and consequences measured accurately in
appropriate physical units?
(eg hours of nursing time, number of physician visits, lost
work-days, gained life-years?)

Were any of the identified items omitted from

measurement? If so, does this mean that they carried no
weight in the subsequent analysis?

Were there any special circumstances (eg joint use of

resources) that made measurement difficult? Were
these circumstances handled appropriately?
14

Deakin Health Economics

14

Drummonds Checklist
6. Were costs and consequences valued credibly?
Were the sources of all values clearly identified? (Possible
sources include market values, patient or client preferences
and views, policy-makers views and health professionals
judgements).
Were market values employed for changes involving resources
gained or depleted?
Where market values were absent (eg volunteer labour) what
valuation sources were used?
Was the valuation of consequences appropriate for the
question posed.
CEA vs CUA vs CBA

15

Deakin Health Economics

15

Drummonds Checklist
7. Were costs and consequences adjusted for differential
timing?

Were costs and consequences which occur in

the future discounted to their present values?
Was any justification given for the discount rate
used?

16

Deakin Health Economics

16

Drummonds Checklist
8. Was an incremental analysis of costs and
consequences of alternatives performed?

Were the additional (incremental) costs generated

by one alternative over another compared to the
additional effects, benefits or utilities generated?
More on this latter!!

17

Deakin Health Economics

17

Drummonds Checklist
9. Was allowance made for uncertainty in the estimates
of costs and consequences?
If data on costs or consequences were stochastic, were
appropriate statistical analyses performed?
If a sensitivity analysis was employed, was justification
provided for the ranges of values (for key study parameters)?

Were study results sensitive to changes in the values (within

the assumed range for sensitivity analysis, or within the
confidence interval around the ratio of costs to
consequences)?
18

Deakin Health Economics

18

Drummonds Checklist
10. Did the presentation and discussion of study results
include all issues of concern to users?

19

Were the conclusions of the analysis based on some overall

index or ratio of costs to consequences (eg costeffectiveness ratio)? If so, was the index interpreted
intelligently or in a mechanistic fashion?
Were the results compared with those of others who have
investigated the same question? If so, were allowances
made for potential differences in study methodology?
Did the study discuss the generalisability of the results to
other settings and patient/client groups?
Deakin Health Economics

19

Drummonds Checklist
10. Did the presentation and discussion of study results
include all issues of concern to users? cont/d..

20

Did the study allude to, or take account of, other

important factors in the choice or decision under
consideration (eg distribution of costs and
consequences, or relevant ethical issues)?
Did the study discuss issues of implementation, such as
the feasibility of adopting the preferred programme
given existing financial or other constraints, and whether
any freed resources could be redeployed to other
worthwhile programmes?
Deakin Health Economics

20

Economic Evaluation: Traps for Beginners

1.

No comparator
The essential question economic appraisal asks is what
difference does it make?

2.

Study perspective not specified

1 and 2 lead to poorly specified research question, and
problems with identification/measurement of costs and
benefits.

3.

Inadequate description of programme and comparator

Who does what to who, when and where?
Leads to inadequate measurement of costs and benefits, and
poorly identified data sources.

21

Deakin Health Economics

21

Economic Evaluation: Traps for Beginners

4.

Failure to specify inclusion/exclusion criteria for costs and

benefits
Leads to internal and external validity issues. Financial costing
rather than economic approach. Failure to consider if
outcome measure captures all relevant benefits.

5.

Failure to undertake marginal analysis

Reliance on average C/E results can hide important
information.

6.

Failure to undertaken sensitivity analysis

No provision for uncertainty.

22

Deakin Health Economics

22

Economic Evaluation: Traps for Beginners

7.

Failure to incorporate discounting

No provision for when costs and benefits are experienced.

8.

Failure to consider evaluability

Is the project ready to be evaluated from output/outcome
perspective?

9.

Insufficient thought given to time period of the study

10. Double counting of benefits

(Life years saved plus forgone productivity).

23

Deakin Health Economics

23

Presentation of Results
Economic Evaluation has become a lot more
sophisticated over the last 15 years
When I first started:
ICER =

1 2
1 2

ICER = Comparative, costs, benefits

Now also have:

Cost-Effectiveness Planes (CEP)
Cost-Effectiveness Acceptability Curves (CEAC)
Net-Monetary Benefits (NMB)
24

Deakin Health Economics

Example a simple ICER

Societal Cost of Drug 1 = $10,000

Societal Cost of Drug 2 = $5,000
Benefit of Drug 1 = 10 life years
Benefit of Drug 2 = 8 life years

ICER =
ICER =

10,0005,000
108
5000
2

ICER = $2,500/LY saved

Is this good value for money???
25

Deakin Health Economics

Decision rules
Incremental benefit of Program B vis-vis Program A

Incremental
cost of
Program B
compared
with
Program A

MORE

MORE

SAME

LESS
Dominated

SAME

Dominant

LESS

Dominant

Dominated

X = study reduces from CEA to a cost-minimisation

analysis.
26
Deakin Health Economics

27

Problems with League Tables

Comparability of methods, for example:
Perspective
Time Horizon
Settings (USA vs Aust)

Outcomes
Uncertainty in results
Rarely just one single point estimate!

28

Deakin Health Economics

Example
Mihalopoulos et al, 2011, ANZJP, 45, 36-44
This study evaluated 2 interventions designed to
prevent depression in adults
Brief bibliotherapy
Group based psychological therapy

29

Comparator was treatment as usual for both

Perspective was health sector
Time horizon was 5 years
Modelled economic evaluation
Part of a larger project called ACE-Prevention
which had a detailed protocol of methods
Deakin Health Economics

Mihalopoulos et al, 2011 (ANZJP, 45, 36-44)

Results from paper
point
estimate (95%
uncertainty
interval)
MedianMedian
point
estimate
(95%
uncertainty
interval)

Brief bibliotherapy
Group-based
psychological
Brief bibliotherapy
intervention
Group-based
psychological
intervention
treatment
treatment

DALYs averted

DALYs averted

2,600 (-1,500
6,700)
2,600 (-1,500
6,700)
Govt: $760,000 ($520,000 -

Govt: $760,000
$1,000,000)($520,000 $1,000,000)
Cost of the interventionPrivate

Cost of the intervention

Private patient :$21M ($13M -

patient
$31M) :$21M ($13M -

1,700 (518
3,000)
1,700
(518

3,000)

Govt: $20M ($13M - $30M)

Govt: $20M ($13M - $30M)

Private patient :$3M ($2M $5M)

Private patient :$3M ($2M - $5M)

$31M)
Private other: $15M ($10M-

Private other:
20M) $15M
Total costs

$37M ($24M - $52M)

$37M ($24M
- $52M)

Cost offsets

Cost offsets

$6M ($-5M - $15M)

ICER (without1cost-offsets) 1

$11,000

ICER (with cost-offsets)

ICER (without cost-offsets)

30

$6M ($-5M - $15M)

$8,600

$8,600
$11,000

Private other: $14M ($9M-20M)

20M)
($10M-20M)

Total costs

1
ICER (with
1 cost-offsets)

Private other: $14M ($9M-

Deakin Health Economics

$38M ($26M - $53M)

$38M ($26M - $53M)

$4M ($1M - $8M)

$4M ($1M - $8M)

$20,000 ($12,000 -

$20,000
$66,000)

($12,000 - $66,000)

$23,000 ($14,000 - $

$23,000
68,000)

($14,000 - $ 68,000)

Box 3.2: The Cost-effectiveness

Plane
In the diagram the horizontal axis represents the difference in effect between the intervention of interest (A)
and the relevant alternative (O), and the vertical axis represents the difference in cost. The alternative (O)
could be the status quo or a competing program.
If point A is in quadrants II or IV the choice between the programs is clear. In quadrant II the intervention of
interest is both more effective and less costly than the alternative. That is, it dominates the alternative. In
quadrant IV the opposite is true. In quadrants I and III the choice depends on the maximum costeffectiveness ratio one is willing to accept. The slope of the line OA gives the cost-effectiveness ratio.

IV

COST DIFFERENCE

Intervention more effective and

more costly than O

Intervention less effective and

more costly than O

Intervention less effective and less

costly than O

III
31

EFFECT DIFFERENCE

Intervention more effective and

less costly than O

Deakin Health Economics

II

Previous example as a CE Plane

Mihalopoulos
et al, 2011

http://basecase.com/articles/cost-effectiveness-plane-explanation/
32

Deakin Health Economics

Thresholds
Threshold values to denote value for money
E.g. $50,000/QALY, 30,000/QALY in UK
These can change in different jurisdictions
PBAC does not have a stated threshold
NICE does
McKie et al (2011) found that in Australia societal values changed
according to constructs such as hope. (Health Economics, (20), 945957)

Are largely subjective value judgements

What is something worth

WHO Commission on Macroeconomics developed a rule of

thumb
1 DALY for less than the average per capita GDP for a given country is
very cost-effective
Even up to 3 times per GDP is still cost-effective
Aust per capita GDP is: $37,000
33

Deakin Health Economics

Acceptability Curve
http://bjp.rcpsych.org/content/187/2/106
Excellent description of CEAC written for noneconomists

Shows the probability that an intervention is

cost-effective over a range of costeffectiveness value for money thresholds
Are constructed using the same data used in
the CEP
34

Deakin Health Economics

Acceptability Curves

35

Deakin Health Economics

Net Monetary Benefits

This is a composite measure (part CEA and part
CBA)
Uses a threshold value for money
E.G. $50,000/QALY

W>

E.g. ig C = $5000 & B = 2QALYs

= 50,000*2>5,000
Therefore intervention is deemed cost-effective
Studies will often adopt different thresholds and
calculate probability that intervention is CE
36

Deakin Health Economics

Further Reading
Drummond et al (2005), Chapter 3
Weblinks in the lecture
CHEERS Checklist in Unit Readings

37

Deakin Health Economics