Parameter Estimation

Module 2.2: Limit Theorems

c University of New South Wales
School of Risk and Actuarial Studies

1/26
 Parameter Estimation

Chebyshevs Inequality

Convergence concepts

Application of strong convergency: Law of Large Numbers

Central Limit Theorem

Applications of Convergence in Distributions

2/26
 Parameter Estimation
Chebyshevs Inequality

Chebyshevs Inequality
I The Chebyshevs inequality, states that for any random
variable X with mean and variance 2 , the following
probability inequality holds for all  > 0:
2
Pr (|X | > ) .
2
I Note that this applies to all distributions, hence also
non-symmetric ones! This implies that:
2
Pr (X > ) Pr (X < ) .
2
I Interesting example: set  = k then:
1
Pr (|X | > k ) .
k2
3/26
 Parameter Estimation
Chebyshevs Inequality

Application: Chebyshevs Inequality

The distribution of fire insurance claims does not have a special
distribution. We do know that the mean claim size in the portfolio
is $50 million with a standard deviation of $150 million.
Question: What is an upper bound for the probability that
the claim size is larger than $500 million?

Solution: We have:

Pr (X > k ) Pr (|X | > k )

= Pr (|X 50| > k 150)
1 1
2 = .
k 9
Thus, Pr (X > 500) 1/9.
4/26
 Parameter Estimation
Convergence concepts

Convergence concepts

I Suppose X1 , X2 , . . . form a sequence of r.v.s. Example: Xi is

the sample variance using the first i observations.
I Xn is said to converge almost surely (a.s.) to the random
variable X as n if and only if:

Pr ( : Xn () X () , as n ) =1,
a.s.
and we write Xn X , as n .
I Sometimes called strong convergence. It means that beyond
some point in the sequence (), the difference will always be
less than some positive , but that point is random.

5/26
 Parameter Estimation
Convergence concepts

Convergence in probability

I Xn converges in probability to the random variable X as

n if and only if, for every  > 0,

Pr (|Xn X | > ) 0, as n ,
p
and we write Xn X , as n .
I Difference converges in probability and converges almost
surely: Pr (|Xn X | > ) goes to zero instead of equals zero
p a.s.
as n goes to infinity (hence is weaker than ).

6/26
 Parameter Estimation
Convergence concepts

Convergence in distribution
I Xn converges in distribution to the random variable X as
n if and only if, for every x,

FXn (x) FX (x) , as n .

d
and we write Xn X , as n . Sometimes called weak
convergence.
I Convergence of MGFs implies weak convergence.
I Applications:
- Cental Limit Theorem;
- Xn Bin(n, p) and X N(n p, n p (1 p));
- Xn Poi(n ), with n and X N(n , n ).
7/26
 Parameter Estimation
Application of strong convergency: Law of Large Numbers

The Law of Large Numbers

I Suppose X1 , X2 , . . . , Xn are independent random variables with

common mean E[Xk ] = and common variance
Var (Xk ) = 2 , for k = 1, 2, . . . , n.
I Define the sequence of sample means as:
n
1X
Xn = Xk .
n
k=1

I Then, according to the law of large numbers, for any  > 0, we

have:
2 2
lim Pr X n  >  = lim 2n = lim
 

= 0,
n n  n n 2

8/26
 Parameter Estimation
Application of strong convergency: Law of Large Numbers

The Law of Large Numbers

I The law of large numbers (LLN) is sometimes written as:

 

Pr X n  >  0, as n .

I The result above is sometimes called the (weak) law of large

p
numbers and sometimes we write X n , because this is the
same concept as convergence in probability to a constant.

9/26
 Parameter Estimation
Application of strong convergency: Law of Large Numbers

The Law of Large Numbers

I However, there is also what we call the (strong) law of large

numbers which simply states that the sample mean converges
almost surely to :
a.s.
X n , as n .

Important result in Probability and Statistics!

I Intuitively, the law of large number states that the sample

mean X n converges to the true value .

I How accurate the estimate is will depend on:

I) how large the sample size is; II) the variance 2 .

10/26
 Parameter Estimation
Application of strong convergency: Law of Large Numbers

Application of LLN: Pooling of Risks in Insurance

I Individuals may be faced with large and unpredictable losses.
Insurance may help reduce the financial consequences of such
losses by pooling individual risks. This is based on the LLN.
I If X1 , X2 , . . . , Xn are the amount of losses faced by n different
individuals, but homogeneous enough to have a common
distribution, and if these individuals pool together and each
agrees to pay:
n
1 X
Xn = Xk .
n
k=1

I Then, the LLN tells us that the amount each person will end
up paying becomes more predictable as the size of the group
increases. In effect, this amount will become
closer to , the average loss each individual expects.
11/26
 Parameter Estimation
Central Limit Theorem

Central Limit Theorem

I Suppose X1 , X2 , . . . , Xn are independent, identically distributed
random variables with finite mean and finite variance 2 . As
before, denote the sample mean by X n .

I Then, the central limit theorem states:

Xn d
 N (0, 1) , as n .
n

This holds for all r.v. with finite mean and variance, not only
normal r.v.!

Prove & rewriting CLT: see next slides.

12/26
 Parameter Estimation
Central Limit Theorem

Rewriting Central Limit Theorem

I We can write this result as:
!
Xn
lim Pr  x = (x) ,
n n
for all x where () denotes the cdf of a standard normal r.v..
Xn
Zn = 
n
is approximately standard normally distributed.
I The Central Limit Theorem
P is usually expressed in terms of the
standardized sums Sn = nk=1 Xk . Then the CLT applies to
the random variable:
Sn n d
Zn = N (0, 1) , as n .
n
13/26
 Parameter Estimation
Central Limit Theorem

Proof of the Central Limit Theorem

I Let X1 , X2 , . . . be a sequence of independent

Pn r.v.s with mean
2
and variance and denote Sn = i=1 Xi . Prove that

Sn n
Zn =
n

converges to the standard normal distribution.

d
General procedure to prove Xn X :
1. Find the m.g.f. of Z : MZ (t);
2. Find the m.g.f. of Zn : MZn (t);
3. Take the limit n of m.g.f. of Zn : lim MZn (t) and
n
rewrite it. This should be equal to MZ (t).
14/26
 Parameter Estimation
Central Limit Theorem

Assumptions

Note that we only assumed that:

MXi (t) =f t, 2 ;

E [Xi ] =;
Var (Xi ) = 2 < ,

hence, for any distribution Xi with mean and finite variance!

15/26
 Parameter Estimation
Central Limit Theorem

1. Proof: Consider the case with = 0 and assuming the MGF

2


exists for X , then we have: MZ (t) = exp t /2 .
n
P
n Xi
P S i=1
2. Recall Sn = Xi , the m.g.f. of Zn = n
n
=
n
is
i=1
obtained by:
 
t
MZn (t) =Msn
n
  n
t
= MXi
n

* using MaX (t) = MX (a t)

** using Sn is the sum of n i.i.d. random variables Xi , thus
MPni=1 Xi (t) = MXn i (t).

16/26
 Parameter Estimation
Central Limit Theorem

Note: lim b nc = 0, for b R and c > 0.

n
Consider Taylor series around zero for any MX (t):
i
X t (i)

MX (t) = MX (t)

i! | {z t=0}
i=0
i th moment
(1)
 1 2 (2)

=MX (0) + t MX (t) + t MX (t) + O(t 3 ),
 
t=0 2 t=0

where O(t 3 ) covers all terms ck t k , with ck R for k 3.

We have,

MX (0) =E[e 0X ] = 1

(1)
MXi (t) =E [Xi ] = 0,

t=0
(2)
=E Xi2 = Var (Xi ) + (E [Xi ])2 = 2 .
 
MXi (t)

t=0
17/26
 Parameter Estimation
Central Limit Theorem

I Now we can align the results from the previous two slides.
I We take the following limit,

lim MZn (t)

n
16

n
= lim MXi t/( n)
n

i !n
17
X t/( n) (i)

= lim MXi (t)

n i! t=0
i=0
 2  3 !!n
17 1 t 2 t
= lim 1 + 0 + +O
n 2 n / n

18/26
 Parameter Estimation
Central Limit Theorem

I Next we take log and have

lim log (MZn (t))

n
 2  3/2 !!
1 t 2 1
= lim n log 1 + +O
n 2 n n
 3/2 !  1/2 ! 
t 2 t2
 
1 1 1
= lim n +O +O = ,
n 2 n n n 2
| {z }
3/2 2 1/2
      
=n O ( 1n ) +O ( 1n ) =O ( 1n ) 0, if n

P (1)i+1 ai
* using log(1 + a)=

 i=1 i = a + O(a2 ), with
t2 3/2
a= n + O n1 .

19/26
 Parameter Estimation
Central Limit Theorem

Application CLT

I An insurer offers builders risk insurance. It has yearly 400

contracts and offers the product already 9 years. The sample
mean of a claim is $10 million and the sample standard
deviation is $25 million.

I Question: What is the probability that in a year the claim size

is larger than $5 billion?

20/26
 Parameter Estimation
Central Limit Theorem

Application CLT
Solution: Using CLT (why is sample s.d.?)

Xn d
 N (0, 1) , as n
n

2 
n X n N n , n 2


X n N , / n

Let X be the total loss, then we have

X = 400 X 400 N(400 10, 400 252 ).

The probability we need to calculate is

5000 4000
P(X > 5000) = P(Z > ) = 1 0.9772 = 0.0228
20 25
21/26
 Parameter Estimation
Applications of Convergence in Distributions

Normal Approximation to the Binomial

I We know: a Binomial random variable is the sum of Bernoulli
random variables. Let Xk Bernoulli (p). Then:

S = X1 + X2 + . . . + Xn

has a Binomial(n, p) distribution.

I Applying the Central Limit Theorem, S must be approximately
normal with mean E[S] = n p and variance Var (S) = n p q,
so that approximately for large n we have:
S np
N (0, 1) .
npq
I Question: What is the probability that X = 60 if
X Bin(1000, 0.06)? Not in Binomial tables!
22/26
 Parameter Estimation
Applications of Convergence in Distributions

I In practice, for large n and for p around 0.5 (but in particular

np > 5 and np (1 p) > 5 or n > 30) then can approximate
the binomial probabilities with the Normal distribution.

Use = n p and 2 = n p (1 p).

I Continuity correction for binomial: note that Binomial random
variable X takes integer values k = 0, 1, 2, . . . but Normal
probability is continuous so that for value:

Pr (X = k) ,

we require the Normal approximation:

!
k 12 k+ 21



Pr <Z <

and similarly for probability that Pr (X k).

23/26
 Parameter Estimation
Applications of Convergence in Distributions

Normal approximation to Binomial

probability mass function Binomial(5,0.1) p.m.f. Binomial(10,0.1) p.m.f.

probability mass function

0.4
n = 5, p = 0.1 n = 10, p = 0.1
0.4 0.3
p.d.f. N(0.5,0.45) p.d.f. N(1,0.9)
0.2
0.2
0.1

0 0
0 2 4 0 5 10
x x
Binomial(30,0.1) p.m.f. Binomial(200,0.1) p.m.f.
probability mass function

0.2
n = 30, p = 0.1 probability mass function 0.08 n = 200, p = 0.1

0.15 p.d.f. N(3,2.7) 0.06 p.d.f. N(20,18)

0.1 0.04
0.05 0.02
0 0
0 10 20 30 0 100 200
x x
24/26
 Parameter Estimation
Applications of Convergence in Distributions

Normal approximation to the Poisson

I Approximation of Poisson by Normal for large values of .

Let Xn be a sequence of Poisson random variables with

increasing parameters 1 , 2 , . . . such that n .

We have:

E[Xn ] =n
Var (Xn ) =n

I Standardize the random variable (i.e., subtract mean and

divide by standard deviation):

Xn E[Xn ] Xn n d
Zn = p = Z N(0, 1).
Var (Xn ) n
25/26
 Parameter Estimation
Applications of Convergence in Distributions

Normal approximation to Poisson

Poisson(0.1) p.m.f. Poisson(1) p.m.f.
probability mass function

probability mass function

= 0.1 =1
1 0.3
p.d.f. N(0.1,0.1) p.d.f. N(1,1)
0.2
0.5
0.1

0 0
0 1 2 0 2 4 6
x x
Poisson(10) p.m.f. Poisson(100) p.m.f.
probability mass function

probability mass function

= 10 = 100
0.1 0.03
p.d.f. N(10,10) p.d.f. N(100,100)
0.02
0.05
0.01

0 0
0 10 20 30 0 100 200
x x

26/26