1

SummarizingPerformanceData
ConfidenceIntervals
Impoitant
Easy to Bifficult
Waining: some mathematical content
Contents
1. Summaiizeu uata
2. Confiuence Inteivals
S. Inuepenuence Assumption
4. Pieuiction Inteivals
S. Which Summaiization to 0se .
2
3
1SummarizingPerformanceData
Bow uo you quantify:
Cential value
Bispeision (vaiiability)
old
new
4
Histogramisoneanswer
old
new
5
ECDFalloweasycomparison
old
new
6
SummarizedMeasures
Neuian, Quantiles
Neuian
Quaitiles
Pquantiles
Nean anu stanuaiu ueviation
Nean
Stanuaiu ueviation
What is the inteipietation of stanuaiu ueviation .
A: if uata is noimally uistiibuteu, with 9S% piobability, a new uata sample lies in
the inteival
Example
7
mean and standard deviation
quantiles
8
CoefficientofVariationSummarizes
Variability
Scale fiee
Seconu oiuei
Foi a uata set with n samples
Exponential uistiibution: Cov =1
What uoes Cov = u mean .
LorenzCurve Gapis anAlternativetoCoV
Alteinative to Cov
Foi a uata set with n samples
Scale fiee, inuex of unfairness
9
Jains Fairness Indexis anAlternativetoCoV
Quantifies faiiness of x;
Ranges fiom
1: all x
i
equal
1n: maximum unfaiiness
Faiiness anu vaiiability aie
two siues of the same coin
10
LorenzCurve
0lu coue, new coue: is }FI laigei . uap .
uinis inuex is also useu; Bef: 2 x aiea between uiagonal anu Loienz cuive
Noie oi less equivalent to Loienz cuive gap
11
Lorenz Curve gap
Perfect equality (fairness)
12
WhichSummarizationShouldOneUse?
Theie aie (too) many synthetic inuices to choose fiom
Tiauitional measuies inengineeringaie stanuaiu ueviation, mean anu Cov
Tiauitional measuies incomputerscienceaie mean anu }FI
}FI is equivalent to Cov
In economy, gap anu uinis inuex (a vaiiant of Loienz cuive gap)
Statisticians like meuians anu quantiles (iobust to statistical assumptions)
We will come back to the issue aftei uiscussing confiuence inteivals
13
14
2.ConfidenceInterval
Bo not confuse with predictioninterval
Quantifies uncertainty about an estimation
15
mean and standard deviation
quantiles
16
ConfidenceIntervalsforMeanofDifference
Nean ieuuction =
u is outsiue the confiuence inteivals foi mean anu foi meuian
Confiuence inteival foi meuian
17
ComputingConfidenceIntervals
This is simple if we can assume that the uata comes fiom an iiu mouel
Inuepenuent Iuentically Bistiibuteu
18
CIformedian
Is the simplest of all
Robust: always tiue pioviueu iiu assumption holus
19
20
ConfidenceIntervalforMedian,level95%
n = S1
n = S2
21
Examplen ,confidenceintervalfor
median
The meuian estimate is
X
(S0)
+X
S1
2
Confiuence level 9S%
] = Su - 9.8 = 4u
k = S1 + 9.8 = 6u
a confiuence inteival foi the meuian is
|X
40
; X
61
]
Confiuence level 99%
] = Su - 12.8 = S7
k = S1 + 12.8 = 64
a confiuence inteival foi the meuia is
|X
37
; X
64
]
22
CIformeanandStandardDeviation
This is anothei methou, most commonly useu methou
But iequiies some assumptions to holu, may be misleauing if they uo not
holu
Theie is no exact theoiem as foi meuian anu quantiles, but theie aie
asymptotic iesults anu a heuiistic.
23
CIformean,asymptoticcase
If cential limit theoiem holus
(in piactice: n is laige anu uistiibution is not wilu)
24
Example
n =1uu ; 9S% confiuence level
CI foi mean:
s
n
amplituue of CI uecieases in
compaie to pieuiction
inteival
25
NormalCase
Assume uata comes fiom an iiu + normal uistiibution
0seful foi veiy small uata samples (n <Su)
26
Example
n =1uu ; 9S% confiuence level
CI foi mean:
CI foi stanuaiu ueviation:
same as befoie except
s insteau of
foi all n insteau of 1.98 foi n=1uu
In piactice both (noimal case anu laige n
asymptotic) aie the same if n > Su
But laige n asymptotic uoes not iequiie noimal
assumption
27
Tablesin[WeberTables]
28
StandardDeviation:norn1?
29
BootstrapPercentileMethod
A heuiistic that is iobust (iequiies only iiu assumption)
But be caieful with heavy tail, see next
but tenus to unueiestimate CI
Simple to implement with a computei
Iuea: use the empiiical uistiibution in place of the theoietical (unknown)
uistiibution
Foi example, with confiuence level = 9S%:
the uata set is S=
Bo i=1 to i=999
(ieplay expeiiment) Biaw n bootstiap ieplicates withreplacement fiom S
Compute sample mean T
i
Bootstiap peicentile estimate is (T
(2S)
, T
(97S)
)
30
Example:CompilerOptions
Boes uata look noimal .
No
Nethous 2.S.1 anu 2.S.2 give same
iesult (n >Su)
Nethou 2.S.S (Bootstiap) gives same
iesult
=>Asymptotic assumption valiu
ConfidenceIntervalforFairnessIndex
0se bootstiap if uata is iiu
31
32
We testasystem10000timeforfailures
andfind 200failures:give a95%confidence
interval forthefailure probability .
33
We testasystem10000timeforfailures
andfind 200failures:give a95%confidence
interval forthefailure probability .
Let X

= u oi 1 (failuie success); E X

= p
So we aie estimating the mean. The asymptotic theoiy
applies (no heavy tail)
p
n
= u.u2
s
n
2
=
1
n
X

2
=1n
-p
n
2
=
1
n
X

=1n
-p
n
2
= p
n
-p
n
2
= p
n
1 -p
n
= u.u2 u.98 = u.u2
s
n
= u.u2 = u.14
Confiuence Inteival: p
n
_
qs
n
10000
= u.u2 _u.uuS at level u.9S
34
We testasystem10timeforfailures and
find 0failure:give a95%confidenceinterval
forthefailure probability .
1. |u ; uj
2. |u ; u.1j
S. |u ; u.11j
4. |u ; u.21j
S. |u; u.S1j
35
ConfidenceIntervalforSuccessProbability
Pioblem statement: want to estimate pioba of failuie; obseive n outcomes;
no failuie; confiuence inteival .
Example: we test a system 1u time foi failuies anu finu u failuie: give a 9S%
confiuence inteival foi the failuie piobability p.
Is this a confiuence inteival foi the mean . (explain why)
The geneial theoiy uoes not give goou iesults when mean is veiy small
36
37
38
We testasystem10000timeforfailures andfind 200failures:give
a95%confidenceinterval forthefailure probability p.
Apply foimula 2.29 (z = 2uu 6 anu n - z 6)
u.u2 _
1.96
1uuuu
2uu 1 - u.u2 = u.u2 _
1.96
1uuuu
1u 2 = u.u2 _ u.uuS
39
40
TakeHomeMessage
Confiuence inteival foi meuian (oi othei quantiles) is easy to get fiom the
Binomial uistiibution
Requiies iiu
No othei assumption
Confiuence inteival foi the mean
Requiies iiu
Anu
Eithei if uata sample is noimal anu n is small
0i uata sample is not wilu anu n is laige enough
The boostiap is moie iobust anu moie geneial but is moie than a simple
foimula to apply
Confiuence inteival foi success piobability iequiies special attention when
success oi failuie is iaie
To we neeu to verify the assumptions
3.TheIndependenceAssumption
41
Confiuence Inteivals iequiie that we can assume that the uata comes fiom
an iiu mouel
Inuepenuent Iuentically Bistiibuteu
Bow uo I know if this is tiue .
Contiolleu expeiiments: uiaw factois ianuomly with ieplacement
Simulation: inuepenuent ieplications (with ianuom seeus)
Else: we uo not know in some cases we will have methous foi time seiies
42
Whatdoesindependencemean?
Example
Pietenu uata is iiu:
CI foi mean is |69;
69.8j
Is this biaseu .
43
data ACF
Whathappensifdataisnotiid?
If uata is positively coiielateu
Neighbouiing values look similai
Fiequent in measuiements
CI is unueiestimateu: theie is less infoimation in the uata than one thinks
44
45
4.PredictionInterval
CI foi mean oi meuian summaiize
Cential value + unceitainty about it
Pieuiction inteival summaiizes vaiiability of uata
46
PredictionIntervalbasedonOrderStatistic
Assume uata comes fiom an iiu mouel
Simplest anu most iobust iesult (not well known, though):
47
PredictionIntervalforsmalln
Foi n=S9, |x
min
, x
max
j is a pieuiction inteival at level 9S%
Foi n <S9 theie is no pieuiction inteival at level 9S% with this methou
But theie is one at level 9u% foi n > 18
Foi n = 1u we have a pieuiction inteival |x
min
, x
max
j at level 81%
PredictionIntervalbasedonMean
48
Prediction Interval based onMean
If uata is not noimal, theie is no geneial iesult bootstiap can
be useu
If uata is assumeu noimal, how uo CI foi mean anu Pieuiction
Inteival baseu on mean compaie .
49
Prediction Interval based onMean
If uata is not noimal, theie is no geneial iesult bootstiap can
be useu
If uata is assumeu noimal, how uo CI foi mean anu Pieuiction
Inteival baseu on mean compaie .
p =estimateu mean
s
2
=estimateu vaiiance
CI foi mean at level 9S % = p _
1.96
n
s
Pieuiction inteival at level 9S% = p _1.96 s
50
51
ReScaling
Nany iesults aie simple if the uata is noimal, oi close to it (i.e. not wilu). An
impoitant question to ask is: can I change the scale of my uata to have it look
moie noimal.
Ex: log of the uata insteau of the uata
A geneiic tiansfoimation useu in statistics is the BoxCox tiansfoimation:
Continuous in s
s=u : log
s=1: 1x
s=1: iuentity
52
PredictionIntervalsforFileTransferTimes
mean and
standard deviation
on rescaled data
mean and
standard deviation
order statistic
53
WhichSummarizationShouldIUse?
Two issues
Robustness to outlieis
Compactness
54
QQplotiscommontoolforverifyingassumption
Noimal Qqplot
Xaxis: stanuaiu noimal quantiles
Yaxis: 0iueieu statistic of sample:
If uata comes fiom a noimal uistiibution, qqplot is close to a stiaight line
(except foi enu points)
visual inspection is often enough
If not possible oi uoubtful, we will use tests latei
55
QQPlotsofFileTransferTimes
56
TakeHomeMessage
The inteipietation of as measuie of
vaiiability is meaningful if the uata is
noimal (oi close to noimal). Else, it is
misleauing. The uata shoulu be best ie
scaleu.
5.WhichSummarizationtoUse?
Issues
Robustness to outlieis
Bistiibution assumptions
57
ADistributionwithInfiniteVariance
58
True mean
True median
True mean
True median
CI based on std dv
CI based on bootsrp
CI for median
59
OutlierinFileTransferTime
60
RobustnessofConf/PredictionIntervals
mean +std dev
CI for median
geom mean
Outlier removed
Outlier present
Order stat
Based on
mean +std dev
Based on
mean +std dev
+re-scaling
FairnessIndices
Confiuence Inteivals obtaineu by Bootstiap
Bow .
}FI is veiy uepenuent on one outliei
As expecteu, since }FI is essentially Cov, i.e. stanuaiu ueviation
uap is sensitive, but less
Boes not use squaiing ; why .
61
62
Compactness
If noimal assumption (oi, foi CI; asymptotic iegime) holus, anu aie moie
compact
two values give both: CIs at all levels, pieuiction inteivals
Beiiveu inuices: Cov, }FI
In contiast, CIs foi meuian uoes not give infoimation on vaiiability
Pieuiction inteival baseu on oiuei statistic is iobust (anu, INB0, best)
63
TakeHomeMessage
0se methous that you unueistanu
Nean anu stanuaiu ueviation make sense when uata sets aie not wilu
Close to noimal, oi not heavy taileu anu laige uata sample
0se quantiles anu oiuei statistics if you have the choice
Rescale
Questions
64
Questions
65
Questions
66