Presentation given at the National Judicial College of Australia Expert Evidence Conference, 12 & 13 February 2011

http://www.njca.com.au/Professional%20Development/programs%20by%20year/2011/Expert%20Evidence%20Conference/Expert%20Evidence%202011.htm
Click on references to articles etc. below for hyperlinks.

The New Paradigm in Forensic Science

Geoffrey Stewart Morrison

Quotations
? D. V. Lindley:

Numeracy is not favoured by British justice.

? R. A. Carr-Hill:

I believe Lindleys suggestion is not only mad, it is extremely

dangerous.

Lindley, D. V. (1977). Probability and statistics. The Statistician, 27(3), 203220.

Imagine you are driving to the airport...

Imagine you are driving to the airport...

? This is Bayesian reasoning

It is about logic
It is not about mathematical formulae
There is nothing complicated or unnatural about it
It is the logically correct way to think about many problems

Imagine you work at a shoe recycling depot...

? You pick up two shoes of the same size

Does the fact that they are of the same size mean they were
worn by the same person?
Does the fact that they are of the same size mean that it is
highly probable that they were worn by the same person?

Imagine you work at a shoe recycling depot...

? You pick up two shoes of the same size

Does the fact that they are of the same size mean they were
worn by the same person?
Does the fact that they are of the same size mean that it is
highly probable that they were worn by the same person?
? Both similarity and typicality matter

Imagine you are a forensic shoe comparison expert...

? The footprint at the crime scene is size 10
? The suspects shoe is size 10

What is the probability of the footprint at the crime scene

being size 10 if it had been made by the suspects shoe?
(similarity)
? Half the shoes at the recycling depot are size 10

What is the probability of the footprint at the crime scene

being size 10 if it had been made by the someone elses shoe?
(typicality)

Imagine you are a forensic shoe comparison expert...

? The footprint at the crime scene is size 14
? The suspects shoe is size 14

What is the probability of the footprint at the crime scene

being size 14 if it had been made by the suspects shoe?
(similarity)
? 1% of the shoes at the recycling depot are size 14

What is the probability of the footprint at the crime scene

being size 14 if it had been made by the someone elses shoe?
(typicality)

Imagine you are a forensic shoe comparison expert...

? The footprint at the crime science is size 10

similarity / typicality = 1 / 0.5 = 2

? The footprint at the crime science is size 14

similarity / typicality = 1 / 0.01 = 100

? If you didnt have a database, could you have made subjective

guesses at relative proportions of different shoe sizes in the

population and applied the same logic to arrive at a
conceptually similar answer?

similarity / typicality = likelihood ratio

The New Paradigm for Forensic-Comparison Science

? Use of the likelihood-ratio framework for the evaluation of evidence

logically correct
adopted for DNA in the mid 1990s
? Use of objective measurements, databases representative of the

relevant population, and statistical models

transparent and replicable
? Empirical testing of validity and reliability under conditions reflecting

those of the case at trial

The New Paradigm for Forensic-Comparison Science

? Morrison, G. S. (2009). Forensic voice comparison and the paradigm shift.

Science & Justice, 49, 298308.

? Morrison, G. S. (2010). Forensic voice comparison. In I. Freckelton, & H.

Selby (Eds.), Expert Evidence (Ch. 99). Sydney, Australia: Thomson

Reuters.
? Morrison, G. S. (submitted). Measuring the validity and reliability of

forensic likelihood ratios. Manuscript submitted for publication, minor

revisions requested.

The Likelihood-Ratio Framework

for the Evaluation of Evidence

Given that it is a cow, what is the probability of it having four legs?

p( 4 legs | cow ) = ?

Given that it has four legs, what is the probability that it is a cow?

p( cow | 4 legs ) = ?

Given two voice samples with acoustic properties x1 and x2,

what is the probability that they were produced by the same speaker?

p( same speaker | acoustic properties x1, x2 ) = ?

0.05
0.04
0.03
0.02
0.01
0
-0.01
-0.02
-0.03
-0.04

0.06
0.05
0.04
0.03
0.02
0.01
0
-0.01
-0.02
-0.03

p( same speaker | acoustic properties x1, x2 ) = ?

p( cow | x legs ) = ?

Bayes Theorem:
posterior odds

p( same speaker | acoustic properties x1, x2 )

p( different speaker | acoustic properties x1, x2 )
=
p( acoustic properties x1, x2 | same speaker ) p( same speaker )
p( acoustic properties x1, x2 | different speaker ) p( different speaker )
likelihood ratio

prior odds

 However !!!
The forensic scientist acting as an expert witness
canNOT give the posterior probability. They canNOT
give the probability that two speech samples were
produced by the same speaker.

Why not?
? The forensic scientist does not know the priors.
? Determining the probability of guilt (same speaker) is the task of

the trier of fact (judge, panel of judges, or jury), not the

forensic scientist.
? The task of the forensic scientist is to present the strength of

evidence which can be extracted from the speech samples.

posterior odds

p( same speaker | acoustic properties x1, x2 )

p( different speaker | acoustic properties x1, x2 )
=
p( acoustic properties x1, x2 | same speaker ) p( same speaker )
p( acoustic properties x1, x2 | different speaker ) p( different speaker )
likelihood ratio

prior odds

Example
? The likelihood ratio is 100
? Whatever the trier of facts belief as to the relative probabilities of

the same-speaker versus the different-speaker hypotheses

before being presented with the likelihood ratio, after being
presented with the likelihood ratio they should be 100 times
more likely than before to believe that the voices on the two
recordings belongs to the same speaker rather than to different
speakers.

Calculating forensic likelihood ratios

using objective measurements,
databases representative of the
relevant population,
and statistical models

Likelihood Ratio:
p( acoustic properties x1, x2 | same speaker )
p( acoustic properties x1, x2 | different speaker )
p( x legs | cow )
p( x legs | not a cow )

1
cows
not cows

proportion

0.8

0.6

0.4

0.2

legs

For continuous data rather than histograms, probability density

functions (PDFs) must be used.
0.014

(a)

(b)

0.012

0.010

0.008

0.006

0.004

0.002

20

40

60

80

100

120

140

160

180

rectangle width: 10

200 0

20

40

60

80

100

120

140

160

180

200

80

100

120

140

160

180

200

rectangle width: 5

0.014

(c)

(d)

0.012

0.010

0.008

0.006

0.004

0.002

20

40

60

rectangle width: 2.5

80

100

120

140

160

180

200 0

20

40

60

rectangle width: 0.1

suspect model
background model
offender value

0.025

0.020

LR = 11.35
0.015

0.010

0.005

0
20

40

60

80

100

120

140

160

180

200

Empirically Testing the Validity of a

Forensic-Comparison System

Measuring Validity
? Test set consisting of a large number of pairs known to be same

origin and a large number of pairs known to be different origin

? Test set must represent the relevant population and reflect the

conditions of the case at trial

? Use forensic-comparison system to calculate LR for each pair
? Compare output with knowledge about input

Measuring Validity
? Goodness is extent to which LRs from same-origin pairs > 1, and

LRs from different-origin pairs < 1

? Goodness is extent to which log(LR)s from same-origin pairs > 0,

and log(LR)s from different-origin pairs < 0

LR
1/1000

1/100

1/10

10

100

1000

-3

-2

-1

+1

+2

+3

log10(LR)

? A metric which captures the gradient goodness of a set of likelihood

ratios derived from test data is the log-likelihood-ratio cost, Cllr

1 1
Cllr =
2 N ss

1
1
+
log 2 1 +

LRssi N ds

i =1
N ss

N ds

log
j =1

(1 + LR )
ds j

9
8
7
6

Cllr

5
4
3
2
1

-3

-2

-1

Log10 Likelihood Ratio

Regina versus T
[2010] EWCA Crim 2439

RvT
? 32. It is clear that likelihood ratios have been used in other areas of

expertise by forensic experts when expressing their

conclusions. We are solely concerned in this appeal with the
use in relation to footwear mark evidence.
? 61. [The Forensic Science Regulator] suggested that it was not

logical to adopt the position that the Bayesian or likelihood

ratio approach could be used in some areas, but not in others...
? 76. ...We do not agree with the observations of the Regulator that a

similar approach is justified in all areas of forensic expertise.

Each area requires a separate analysis because of the
differences that there are in the nature of the underlying data.

RvT
? 79. The paper by Jackson, Champod and Evett [2001] rejected the

suggestion that hard data were needed to evaluate a likelihood

ratio...
? 80. We cannot agree with this in so far as it suggests that a

mathematical formula can be used. An approach based on

mathematical calculations is only as good as the reliability of
the data used...

RvT
? 83. ... the data on footwear distribution and use is quite unlike

DNA. A persons DNA does not change and a solid statistical

base has been developed which enable accurate figures to be
produced...
? 84. Use of the FSSs own database could not have produced

reliable figures as it had only 8,122 shoes whereas some 42

million are sold every year...

RvT
? 87. It is of course regrettable that there are, at present, insufficient

data for a more certain and objective basis for expert opinion
on footwear marks, but it cannot be right to seek to achieve
objectivity by reliance on data which does not enable this to be
done. We entirely understand the desire of the experts to try
and achieve the objectivity in relation to evidence of footwear
marks, but the work done has never before, as we understand
it, been subject to open scrutiny by a court.

Further reading
? R v George [2007] EWCA Crim 2722
? R v GK [2001] NSWCCA 504
? Morrison, G. S. (2009). Comments on Coulthard & Johnsons (2007) portrayal of the

likelihood-ratio framework. Australian Journal of Forensic Sciences, 41,

155161.
? Rose, P., & Morrison, G. S. (2009). A response to the UK position statement on

forensic speaker comparison. International Journal of Speech, Language and the

Law, 16, 139163.
? Balding D. J. (2005). Weight-of-evidence for forensic DNA profiles. Chichester, UK:

Wiley.
? Robertson, B., & Vignaux, G. A. (1995). Interpreting evidence. Chichester, UK: Wiley.

Thank You
http://geoff-morrison.net
http://forensic-voice-comparison.net
http://forensic.unsw.edu.au