Simulation studies of the bootstrap percentile and

bootstrap t confidence intervals

Xiou Cao

Introduction

The nonparametric methods serve an important part of statistical analysis. Compared to

conventional parametric methods, nonparametric methods requires less fewer assumptions,
making them widely used in practical situations where the underlying population distribution
is unknown. Due to the robustness and simplicity of nonparametric methods, they can be
applied to some certain cases, even when the use of parametric methods is justified. However,
in most cases, nonparametric methods have less power when parametric methods are appropriate. This report focuses on the bootstrap method and compares its power to Students t
confidence interval.
The bootstrapping method was introduced by Bradley Efron in 1979. Consistent with
other nonparametric methods, the bootstrap is usually applied to small samples where the
original population is unknown. The basic principle of the bootstrap is to simulate the sampling distribution of statistic by repeatedly sampling from the empirical distribution function,
which is generated by the original sample. In other words, by sampling the original sample repeatedly, we can approximately get an idea of the sampling distribution, which, in parametric
methods, is accomplished by transformation and the knowledge of the population distribution.
There are two major methods for bootstrap confidence intervals (CI), which are percentile
bootstrap and Studentized bootstrap (bootstrap t-intervals). The percentile bootstrap forms
the CI by adopting percentiles of the bootstrap distribution. In contrast, Studentized bootstrap replaced the quantiles of t distribution by the quantiles from the bootstrap distribution
of the Students t-test.
This report examines the empirical coverage probability (ECP) of bootstrap percentile
and bootstrap t confidence intervals of the mean by performing the simulations with different
sample sizes (n) and different numbers of bootstrap samples (B ). In order to compare the
power of bootstrap to parametric methods, the t confidence intervals was also calculated for
reference.

Method

The simulation was performed by using software R. Two functions, bt per and bt t were
designed to accomplish the generation of confidence intervals of the mean with certain probability using the given sample.
For percentile bootstrap (bt per), the following steps were designed:
1

1. Obtain random samples of size n with replacement and compute the mean of bootstrap
samples.
2. Repeat step 1 for B times and store the results in bt stat matrix.
3. Compute the quantiles to form confidence intervals with given preset probability (p =
0.95).
Similar steps were devised for computing bootstrap t confidence interval (bt t):
1. Obtain random samples of size n with replacement and compute Z for each bootstrap
samples as follows:
x
(i) x

Z (i) = p
V ar[x (i)]/n
Here, i = 1, 2, ..., B, x is the mean of the original sample, x
(i) is the mean of the ith
bootstrap sample.
2. Repeat step 1 for B times and store the results in bt t stat matrix.
3. Compute the sample quantiles of {Z1 , Z2 , . . . , Zi } and serve as t(1p/2) , t(p/2) and store
in t int.
4. Compute bootstrap t confidence intervals as shown below and store in bt t int.
100(1 p)% CI:

s
s
t(p/2) )
(
x t(1p/2) , x
n
n

Three underlying distributions, Normal(1, 1), 21 and Exponential(1) were selected based
on their different performance on skewness; the mean of the three populations are the same
( = 1). For each distribution, N =10,000 data sets were generated from the population. The
bootstrap percentile and bootstrap t confidence intervals (p = 0.95) were calculated for each
data set using the two functions above. Six sample sizes (n = 5, 10, 15, 20, 25, 50) and three
bootstrap numbers (B = 10, 50, 500) were used for each data set. The t confidence intervals
were also calculated using the function t.test. Each confidence intervals were examined for
the coverage of the population mean ( = 1) immediately. The empirical coverage probability
were then calculated by dividing the number of CIs that straddle the population mean by
the total number of data sets (N = 10, 000). The source code for generating CI from normal
distribution was attached in the Appendix.

Results

The results of the simulations performed were summarized in Table 1, 2 and 3. The confidence intervals formed by t distribution assume that the samples are drawn from a normal
distribution. Indeed, the CI obtained from t estimator behaves with high reliability for normally distributed population regardless of the sample size. All of the five empirical coverage
probabilities (ECP) of t confidence intervals are around 0.95. In contrast, the bootstrap confidence intervals show the dependence on both the sample size and the number of bootstrap
samples. With small sample size and small number of bootstrap samples, the ECP of the 95%
CI generated by percentile bootstrapping is less than 0.90. For instance, the ECP of the CI
2

(n = 10, B = 10) is only 0.7749. However, increasing the sample size and/or the bootstrap
number could substantially improve the power of percentile bootstrap. When n = 50 and
B = 500, the ECP of the CI is 0.9415, quite close to the preset probability 0.95. Interestingly, bootstrap t confidence intervals seem not dependent on the sample size. With the same
bootstrap number, increasing the sample size does not significantly improve the empirical
coverage probability. For example, With B = 500, the ECPs are 0.9447, 0.9461, 0.9434,
0.9481, 0.9489 and 0.9478, respectively (from n = 5 to 50). In addition, large number of
bootstrap samples could enhance the power of Studentized bootstrapping. When n = 5, the
ECP could be increased from 0.8073 to 0.9447 if the bootstrap number is raised from 10 to 500.
Table. 1 The empirical coverage probability of t, bootstrap percentile and bootstrap t confidence
intervals given different sample size (n). The underlying population is Normal(0,1). The second row
shows the number of bootstrap samples (B) .

5
10
15
20
25
50

0.9474
0.9461
0.9503
0.9478
0.9477
0.9532

bootstrap percentile
10
50
500
0.7178
0.7749
0.7939
0.7961
0.7983
0.8181

0.8103
0.8738
0.8927
0.8950
0.9085
0.9137

0.8361
0.9004
0.9168
0.9224
0.9322
0.9415

bootstrap t
10
50
500
0.8073
0.7996
0.7873
0.7902
0.7949
0.7868

0.9162
0.9167
0.9132
0.9137
0.9207
0.9198

0.9447
0.9461
0.9434
0.9481
0.9489
0.9478

When the population distribution was changed to exponential distribution, the ECPs of
t confidence interval decrease, especially for small sample sizes. The ECP is only 0.8964
when the sample size n = 10, consistence with notion that t confidence intervals can only be
applied to normally distributed populations. But increasing the sample size could somewhat
balance the inaccuracy brought by the skewness of the underlying population and improve the
reliability of t confidence intervals. Percentile bootstrap and Studentized bootstrap exhibit
the similar behavior as they do for the normal distribution. With high bootstrap numbers,
bootstrap t confidence interval is more reliable than t confidence interval for small sample size.
For instance, when the sample size is 20 and bootstrap number is 500, the ECP of bootstrap
t confidence interval is 0.9481 but the ECP is 0.9216 for t confidence interval.
Table. 2 The empirical coverage probability of t, bootstrap percentile and bootstrap t confidence
intervals given different sample size (n). The underlying population is Exp(1). The second row shows
the number of bootstrap samples (B) .

5
10
15
20
25
50

0.8794
0.8964
0.9106
0.9216
0.9231
0.9348

bootstrap percentile
10
50
500
0.6740
0.7375
0.7646
0.7739
0.7812
0.8082

0.7638
0.8409
0.8631
0.8765
0.8858
0.9035

0.7884
0.8660
0.8939
0.9023
0.9096
0.9301

bootstrap t
10
50
500
0.7867
0.7871
0.7881
0.7840
0.7868
0.7945

0.9037
0.9080
0.9101
0.9095
0.9133
0.9189

0.9286
0.9392
0.9406
0.9392
0.9422
0.9484

For highly skewed population, 21 , t estimator loses a substantial part of its power to deduce a reliable confidence interval even with a large sample size. When n = 50, the ECP of
t interval is only 0.9158, still much less than the proposed probability, 0.95. In contrast, the
bootstrap t confidence interval can achieve around 0.94 ECP with 500 bootstrap cycles. It
should be noted that even with a very small sample size, n = 5, the bootstrap t confidence
interval can also reach an ECP (0.9189) higher than that achieved by t at n = 50 (0.9158),
indicating its strong potential for small sample statistical analysis. The percentile bootstrap
follows the same trend in normal and exponential distribution. With large sample size and
large number of bootstrap samples, the bootstrap percentile confidence interval gives higher
ECP, which is comparable to that given by t but significant lower than that of bootstrap t
confidence intervals.
Table. 3 The empirical coverage probability of t, bootstrap percentile and bootstrap t confidence
intervals given different sample size (n). The underlying population is 21 . The second row shows the
number of bootstrap samples (B) .

n
5
10
15
20
25
50

t
0.8219
0.8659
0.8750
0.8933
0.8969
0.9158

bootstrap percentile
10
50
500
0.6345
0.7124
0.7426
0.7642
0.7730
0.7874

0.7113
0.8045
0.8350
0.8591
0.8721
0.8874

0.7366
0.8291
0.8623
0.8830
0.8960
0.9175

10

bootstrap t
50
500

0.7754
0.7714
0.7808
0.7871
0.7881
0.7897

0.8946
0.8981
0.9000
0.9068
0.9107
0.9091

0.9189
0.9310
0.9352
0.9405
0.9410
0.9433

Discussion

Nonparametric methods, such as bootstrap provide an almost universal way of statistical

analysis for the data obtained from any type of underlying distribution without losing too
much power of the test. In this report, the simulation studies that aim to evaluate the performance of conventional t, percentile bootstrap and Studentized bootstrap were accomplished.
It was verified in the report that the confidence interval formed by t statistic is powerful and
reliable for normally distributed population. Whereas, for non-normal distribution, t confidence interval is not fully reliable especially for small sample size, where t statistic is often
deployed. Percentile bootstrap and Studentized bootstrap can be applied to any underlying
population. However, bootstrap percentile confidence interval requires both large sample size
and bootstrap number to reach a high empirical coverage probability. In contrast, only large
number of bootstrap samples are needed by forming a bootstrap t confidence interval with
high ECP. The high ECP of bootstrap t confidence interval given small sample size is quite
impressive and valuable considering the situations where we know nothing about the underlying population.
This report only examines the power of bootstrap on estimating the population mean. Actually the bootstrap method can be applied to almost any statistic. In addition, there exist
other bootstrapping methods to generate confidence interval such as bias-corrected bootstrap

and accelerated bootstrap. More simulation studies could be to done to validate the performance of bootstrap for more statistics and other bootstrapping methods.

Appendix

Source code
1
2
3

num <- c (5 , 10 , 15 , 20 , 25 , 50) # sample size

B <- c (10 , 50 , 500)
# bootstrap number
N <- 10000
# number of data sets

4
5
6
7
8
9
10
11

# create matrice for data storage

mydata _ 1 <- matrix ( c ( rep (0 , N * num [1])) ,
mydata _ 2 <- matrix ( c ( rep (0 , N * num [2])) ,
mydata _ 3 <- matrix ( c ( rep (0 , N * num [3])) ,
mydata _ 4 <- matrix ( c ( rep (0 , N * num [4])) ,
mydata _ 5 <- matrix ( c ( rep (0 , N * num [5])) ,
mydata _ 6 <- matrix ( c ( rep (0 , N * num [6])) ,

nrow =N ,
nrow =N ,
nrow =N ,
nrow =N ,
nrow =N ,
nrow =N ,

ncol = num [1] ,

ncol = num [2] ,
ncol = num [3] ,
ncol = num [4] ,
ncol = num [5] ,
ncol = num [6] ,

byrow = TRUE )
byrow = TRUE )
byrow = TRUE )
byrow = TRUE )
byrow = TRUE )
byrow = TRUE )

12
13
14

# data storage list

data _ list <- list ( mydata _ 1 , mydata _ 2 , mydata _ 3 , mydata _ 4 , mydata _ 5 , mydata _ 6)

15
16
17
18
19

# store coverage probability

store _ cov _ 1 <- matrix ( c ( rep (0 , 3 * 6)) , nrow =6 , ncol =3 , byrow = TRUE )
store _ cov _ 2 <- matrix ( c ( rep (0 , 3 * 6)) , nrow =6 , ncol =3 , byrow = TRUE )
store _ cov _ 3 <- c ( rep (0 , 6))

20
21
22
23

y _ 1 <- 0
y _ 2 <- 0
y _ 3 <- 0

24
25
26

# function of generating bootstrap percentile CI . Arguments are sample ( sam ) ,

# sample size ( n ) , bootstrap number ( B ) , probability ( p ) level of CI .

27
28
29
30
31
32
33
34
35
36
37

bt _ per <- function ( sam , n , B , p )

{
bt _ stat <- c ( rep (0 , B ))
for ( i in 1: B )
{
bt _ stat [ i ] <- mean ( sample ( sam , n , replace = TRUE , prob = NULL ))
}
bt _ per _ int <- quantile ( bt _ stat , prob = c ( p / 2 , 1 - p / 2) , type =2)
return ( bt _ per _ int )
}

38
39
40

# function of generating bootstrap t CI . Arguments are sample ( sam ) ,

# sample size ( n ) , bootstrap number ( B ) , probability ( p ) level of CI .

41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

bt _ t <- function ( sam , n , B , p )

{
sam _ mean <- mean ( sam )
sam _ var <- var ( sam )
bt _ t _ sam <- matrix ( c ( rep (0 , B * n )) , nrow =B , ncol =n , byrow = TRUE )
bt _ t _ stat <- c ( rep (0 , B ))
bt _ t _ int <- c ( rep (0 ,2))
for ( i in 1: B )
{
bt _ t _ sam [i ,] <- sample ( sam , n , replace = TRUE , prob = NULL )
bt _ t _ stat [ i ] <- ( mean ( bt _ t _ sam [i ,]) - sam _ mean ) / sqrt ( var ( bt _ t _ sam [i ,]) / n )
}
t _ int <- quantile ( bt _ t _ stat , prob = c (1 - p / 2 , p / 2))
for ( j in 1:2)
{
bt _ t _ int [ j ] <- sam _ mean - t _ int [ j ] * sqrt ( sam _ var / n )
}
return ( bt _ t _ int )

60

61
62
63
64
65
66
67
68
69

# generate data sets

for ( i in 1:6)
{
for ( j in 1: N )
{
data _ list [[ i ]][ j ,] <- rnorm ( num [ i ] , 1)
}
}

# draw samples from Normal (1)

70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95

# compute ECP for bootstrap percentile and bootstrap t

for ( i in 1:6)
{
for ( j in 1:3)
{
for ( k in 1: N )
{
x _ 1 <- bt _ per ( data _ list [[ i ]][ k ,] , num [ i ] , B [ j ] , 0.05)
x _ 2 <- bt _ t ( data _ list [[ i ]][ k ,] , num [ i ] , B [ j ] , 0.05)
# judge if the CI straddles the mean
if ( x _ 1[1] <=1 & x _ 1[2] >=1)
{
y _ 1 <- y _ 1+1
}
if ( x _ 2[1] <=1 & x _ 2[2] >=1)
{
y _ 2 <- y _ 2+1
}
}
store _ cov _ 1[ i , j ] <- round ( y _ 1 /N ,5)
store _ cov _ 2[ i , j ] <- round ( y _ 2 /N ,5)
y _ 1 <- 0
y _ 2 <- 0
}
}

96
97
98
99
100
101
102
103
104
105
106
107
108
109
110

# compute ECP for t

for ( i in 1:6)
{
for ( k in 1: N )
{
x _ 3 <- t . test ( data _ list [[ i ]][ k ,]) $ conf . int
if ( x _ 3[1] <=1 & x _ 3[2] >=1)
{
y _ 3 <- y _ 3+1
}
store _ cov _ 3[ i ] <- round ( y _ 3 /N ,5)
}
y _ 3 <- 0
}