4.

Binomial Random Variable Approximations,

Conditional Probability Density Functions
and Stirlings Formula
Let X represent a Binomial r.v as in (3-42). Then from
(2-30) n k2 k2

P k1 X k2
k k1

Pn ( k ) p k q n
(4-1)
k k k
k
.
1

n n!

k (n k )! k!

Since the binomial coefcient grows quite

rapidly with n, it is difcult to compute (4-1) for large n.
In this context, two approximations are extremely
useful.
n
4.1 The
npq Normal Approximation (Demoivre-Laplace 1
PILLAI
 n k n k 1 ( k np
p q
2

(4-2)
) / 2 npq
e .
k 2 npq
Thus if and k in (4-1) are within or around the
k
neighborhood of the interval
1 2
we
np npq , np npq ,
can approximate the summation in (4-1) by an
integration. In that case (4-1) reduces to
k2 1 x2 1
P k1 X k2 e
( x np ) 2 / 2 npq
dx e
y 2 /2
dy , (4-3)
k1
2 npq x1
2
where
k1 np k2 np
x1 , x2 .
npq npq
We can express (4-3) in terms of the normalized
integral
erf ( x )
1

x
e
y2 /2
dy erf ( x ) (4-4)
2 0

that has been tabulated extensively (See Table 4.1).

2
PILLAI
For example, ifx andx 1 2 are both positive ,we obtain
P k1 X k 2 erf ( x2 ) erf ( x 1 ). (4-5)

Example 4.1: A fair coin is tossed 5,000 times. Find

the probability that the number of heads is between 2,
475 to 2,525.
Solution: We needP ( 2 , 475 X 2 , 525 ). Here n is large so
that we can use the normal approximation. In this case p
1
2
,

so that np 2 , 500 and npq 35 . Sincenp npq 2 , 465 ,

and np npq 2 , 535 , the approximation is valid for k 2 , 475 1

and k 2 , 525 . Thus

2

x2 1

y2
P k1 X k2 e
/2
dy .
x1 2
Here k1 np 5 k2 np 5
x1 , x2 .
npq 7 npq 7
3
PILLAI
 1 x 1

y2
erf ( x ) e
/2
dy G ( x)
2 0
2

Table 4.1
4
PILLAI
Since x 0 , from Fig. 4.1(b), the above probability is
1

given
P 2 , by
475 X 2 , 525 erf ( x ) erf ( x ) erf ( x ) erf (| x |)
2 1 2 1

5
2 erf 0 . 516 ,
7
where we have used Table 4.1
erf ( 0 . 7 ) 0 . 258 .

1 x 2 /2
1 x 2 /2
e e
2 2

x x
x1 x2 x1 x2

(a) x1 0, x2 0 (b) x1 0, x2 0
Fig. 4.1
4.2. The Poisson Approximation
As we have mentioned earlier, for large n, the Gaussian
approximation of a binomial r.v is valid only if p is xed,
i.e., only if np 1 andnpq 1 . what if np is small, or if it
does not increase with n? 5
PILLAI
Obviously that is the case if, for example,
p 0
as
n ,
such that np is a xed number.
Many random phenomena in nature in fact follow this
pattern. Total number of calls on a telephone line,
claims in an insurance company etc. tend to follow
this type of behavior. Consider random arrivals such
as telephone calls over a line. Let n represent the total
number of calls in the interval From our
( 0 , T ). T n
experience, as we have so that we may
n T .
assume Consider a small interval of duration
as in Fig. 4.2. If there is only a single call coming in,
the probability p of that single call occurring in that
interval must depend on its relative size with respect
1 2 n
to T.

0 T 6
Fig. 4.2 PILLAI

Hence we may assume T . Note that
p p 0
Tas
.

However in this casenp T
T
is a
constant, and the normal approximation is invalid here.
Suppose the interval in Fig. 4.2 is of interest to us. A
call inside that interval is a success (H), whereas one
outside is a failure (T ). This is equivalent to the coin
tossing situation, and hence the probability
P (k ) of n

obtaining k calls (in any order) in an interval of

duration is given by the binomial p.m.f. Thus
n!
Pn ( k ) p (1 p )
k n k
, (4-6)
(n k )! k !

and here asn , p 0 such that np . It is

easy to obtain an excellent approximation to (4-6) in
that situation. To see this, rewrite (4-6) as 7
PILLAI
 n(n 1) ( n k 1) ( np )
k
n k
Pn ( k ) k
( 1 np / n )
n k!
(4-7)
1 2 k 1 (1 / n )
k n

1 1 1 .
k! (1 / n )
k
n n n

Thus k

lim Pn ( k ) e , (4-8)
n , p 0 , np k!
k 1
since the nite products n as well
1 2
1 1 1
n n

k


1 n ,
as n tend to unity as and

n

lim 1 e .
n
n
The right side of (4-8) represents the Poisson p.m.f
and the Poisson approximation to the binomial r.v is
valid in situations where the binomial r.v parameters n
and p diverge to two extremes
( n , p 0 ) such
that their product np is a constant. 8
PILLAI
Example 4.2: Winning a Lottery: Suppose two million
lottery tickets are issued with 100 winning tickets
among them. (a) If a person purchases 100 tickets,
what is the probability of winning? (b) How many
tickets should one buy to be 95% condent of having a
winning ticket? Solution: The probability of
buying a winning ticket
No. of winning tickets 100
p 5 10 5 .
2 10
6
Total no. of tickets
Here n 100 , and the number of winning tickets X in
the n purchased tickets has an approximate Poisson
distribution with
parameter
np 100 5 10

0 . 005 . 5

Thus
k

P(X k) e ,
k!

P ( X 1) 1 P ( X 0 ) 1 e 0 . 005 .
and (a) Probability of winning 9
PILLAI
(b) In this case we need
P(X 1) 0 . 95 .

P(X 1) 1 e 0 . 95 implies ln 20 3.

But np n 5 10 3 or n 60 , 000 . Thus one needs to

5

buy about 60,000 tickets to be 95% condent of

having a winning ticket!
Example 4.3: A space craft has 100,000 components n
The probability of any one component being defective
is 2 10 ( p 0 ). The mission will be in danger if ve
5

or more components become defective. Find the

probability of such an event.
Solution: Here n is large and p is small, and hence
Poisson approximation is valid. Thus
np 100 , 000 2 10 2 , 5

and the desired probability is given by

10
PILLAI
 4
k 4
k
P(X 5) 1 P ( X 4 ) 1 e
1e 2
k!
k 0 k! k 0

4 2
1 e 2 1 2 2 0 . 052 .
3 3

Conditional Probability Density Function

For any two events A and B, we have dened the
conditional probability of A given B as
P(A B) (4-9)
P(A |B) , P(B) 0.
P(B)

Noting that the probability distribution function

F ( x) X
is
given by
F X ( x) P X ( ) x , (4-10)
we may dene the conditional distribution of the r.v X
given the event B as
11
PILLAI
 P X ( ) x B
FX (x |B) P X ( ) x | B . (4-11)
P(B)
Thus the denition of the conditional distribution
depends on conditional probability, and since it obeys
all probability axioms, it follows that the conditional
distribution has the same properties as any distribution
function. In particular
P X ( ) B P(B)
F X ( | B ) 1,
P(B) P(B)
(4-12)
P X ( ) B P ( )
F X ( | B ) 0.
P(B) P(B)

Further
P x1 X ( ) x2 B
P ( x1 X ( ) x2 | B )
P(B)
F X ( x2 | B ) FX ( x 1 | B ), (4-13)
12
PILLAI
Since for x x ,
2 1

X ( ) x2 X ( ) x1 x1 X ( ) x2 . (4-14)

The conditional density function is the derivative of

the conditional distribution function. Thus
dF (x |B)
fX ( x | B ) X
, (4-15)
dx

and proceeding as in (3-26) we obtain

x
FX (x |B)
f X ( u | B ) du . (4-16)

Using (4-16), we can also rewrite (4-13) as

(4-17)
x2
P x1 X ( ) x2 | B f X ( x | B ) dx .
x1

13
PILLAI
Example 4.4: Refer to example 3.2. Toss a coin and
X(T)=0, X(H)=1. Suppose
B { H }. Determine
F ( x | B ). X

Solution: From Example 3.2,F ( x ) X has the following

form. WeFneed
(x |B)
X for all x.
For x 0 , X ( ) x , so that X ( ) x B ,
and F ( x | B ) 0 .
X

F X ( x) FX (x | B)

1 1
q
x x
1 1
(a) (b)
Fig. 4.3 14
PILLAI
For 0 x 1, X ( ) x T , so that
X ( ) x B T H and FX (x |B) 0.
For x 1, X ( ) x , and
P(B)
X ( ) x B B { B} and FX (x |B) 1
P(B)
(see Fig. 4.3(b)).
Example 4.5: GivenF ( x ), supposeB X ( ) a .
X
Find
f ( x | B ). X

Solution: We will rst determineF ( x | B ). From (4-11) X

and B as given above, we have

P X xX a
FX (x |B) . (4-18)
P X a

15
PILLAI
For x a , X xX a X x so that
P X x F ( x)
F (x | B) X
. (4-19)

X
P X a F X
(a )

For x a , X xX a ( X a) so thatF X
(x |B) 1.
Thus
F X ( x)
, x a,
FX (x |B) F X (a ) (4-20)

1, x a,

and hence
fX ( x)
d , x a,
fX ( x | B ) FX (x |B) F X (a ) (4-21)
dx
0, otherwise.

16
PILLAI
 FX (x |B)
fX ( x | B )
1
fX ( x)
F X ( x)

x x
a a
(a) (b)
Fig. 4.4

Example 4.6: Let B represent the event a X ( ) b with

b a.

For a givenF ( x ), determineF ( x | B ) and

X
f ( x | B ).
X X

Solution:
P X ( ) x a X ( ) b
FX (x |B) P X ( ) x | B
P a X ( ) b
P X ( ) x a X ( ) b
. (4-22)
F X (b) FX (a )
For x a , we have X ( ) x a X ( ) b , and
hence F ( x | B ) 0 .
X
(4-23)
17
PILLAI
For a x b , we have X ( ) x a X ( ) b { a X ( ) x }
and hence
P a X ( ) x F X ( x) FX (a )
FX (x |B) . (4-24)
F X (b) FX (a ) F X (b) FX (a )

For x b , we have X ( ) x a X ( ) b a X ( ) b
so that F ( x | B ) 1 .
X (4-25)
Using (4-23)-(4-25), we get (see Fig. 4.5)
fX ( x)
, a x b,
fX ( x | B ) FX (b) FX (a )

(4-26)
0, otherwise.

fX ( x | B )

fX ( x)

x
a b
Fig. 4.5 18
PILLAI
We can use the conditional p.d.f together with the
Bayes theorem to update our a-priori knowledge about
the probability of events in presence of new
observations. Ideally, any new information should be
used to update our knowledge. As we see in the next
example, conditional p.d.f together with Bayes
theorem allow systematic updating. For any two events
A and B, Bayes theorem gives
P ( B | A) P ( A)
P(A |B) .
P(B) (4-27)
B x 1 X ( ) x2
Let so that (4-27) becomes (see (4-13)
and (4-17))
P ( x1 X ( ) x 2 ) | A P ( A )
P A | ( x1 X ( ) x 2 )
P x1 X ( ) x2
x2

F X ( x2 | A ) FX ( x1 | A )
P ( A)

x1
f X ( x | A ) dx
P ( A ). (4-28)
FX x2
F X ( x2 ) ( x1 )
x1
f X ( x ) dx
19
PILLAI
Further, let x 1
x, x2 x , 0, 0,
so that in the limit as
fX ( x | A)
lim P A | ( x X ( ) x ) P A | X ( ) x P ( A ). (4-29)
0 fX ( x)

or
P(A | X x) fX ( x)
f X |A ( x | A ) .
(4-30)
P ( A)

From (4-30), we also get

P ( A) f(
x | A ) dx
X
P(A | X x ) f X ( x ) dx , (4-31)

or
1

(4-32)
P ( A)
P(A | X x ) f X ( x ) dx

and using this in (4-30), we get the desired result

P(A | X x) fX ( x)
(4-33)
f X |A ( x | A )
.

P(A | X x ) f X ( x ) dx 20
PILLAI
To illustrate the usefulness of this formulation, let us
reexamine the coin tossing problem.
Example 4.7: Let represent the probability of
p P(H )
obtaining a head in a toss. For a given coin, a-priori p
can possess any value in the interval (0,1). In the
absence of any additional information, we may
f ( p)
assume the a-priori p.d.f to be a uniform P

distribution in that interval. Now suppose we actually

perform an experiment of tossing the coin n times,
and k heads are observed. This is new information.
f ( p) ?
How can we update P

Solution: Let A= k heads in n specic tosses. Since

f ( p)
these tosses result in a specic sequence, P

P(A |P p) p q
k n k
, (4-34) p
0 1
21
Fig.4.6
PILLAI
and using (4-32) we get
(n k )! k !
(4-35)
1 1

n k
P ( A) P(A |P p ) f P ( p ) dp p (1 p )
k
dp .
0 0
( n 1 )!
The a-posteriori p.d.f represents the updated
f ( p | A)
information given the event A, and from (4-30) P |A

f P |A ( p | A )
P(A |P p ) fP ( p )
f P |A ( p | A )
P ( A)
p
(n 1)!

k
p q
n k
, 0 p 1 ( n , k ). (4-36) 0 1

(n k )! k ! Fig. 4.7
Notice that the a-posteriori p.d.f of p in (4-36) is not a
uniform distribution, but a beta distribution. We can
use this a-posteriori p.d.f to make further
predictions, For example, in the light of the above
experiment, what can we say about the probability of a
head occurring in the next (n+1)th toss?
22
PILLAI
Let B= head occurring in the (n+1)th toss, given that k
heads have occurred in n previous tosses.
Clearly and from (4-32)
P(B | P p) p,

(4-37)
1
P(B) P(B | P p ) f P ( p | A ) dp .
0

Notice that unlike (4-32), we have used the a-posteriori

p.d.f in (4-37) to reect our knowledge about the
experiment already performed. Using (4-36) in (4-37),
we get
1 (n 1)! k 1
P(B) p k
p q
n k
dp . (4-38)
0
(n k )! k ! n 2

Thus, if n =10, and k = 6, then

7
P(B) 0 . 58 ,
12

which is more realistic compare to p = 0.5. 23

PILLAI
To summarize, if the probability of an event X is
unknown, one should make noncommittal judgement
about its a-priori probability density
f ( x ).
function
Usually the uniform distribution is a reasonable
X

assumption in the absence of any other information.

Then experimental results (A) are obtained, and out
knowledge about X must be updated reecting this
new information. Bayes rule helps to obtain the a-
posteriori p.d.f of X given A. From that point on, this a-
f ( x | A)
posteriori p.d.f X |A
should be used to make further
predictions and calculations.

24
PILLAI
Stirlings Formula : What is it?

Stirlings formula gives an accurate approximation for n!

as follows:
n 1
n! ~ 2 n 2
e
n (4-39)
in the sense that the ratio of the two sides in (4-39) is ne
to one; i.e., their relative error is small, or the percentage
error decreases steadily as n increases. The approximat
is remarkably accurate even for small n. Thus 1! = 1 is
approximated as2 / e 0 .9 2 2 1, and 3! 6 is
approximated as 5.836.
Prior to Stirlings work, DeMoivre had established
the same formula in (4-39) in connection with binomial
distributions in probability theory. However DeMoivre
did not establish the constant 25
PILLAI
term 2 in (4-39); that was done by James Stirling
( 1730).

How to prove it?

We start with a simple observation: The function
log x is a monotone increasing function, and hence we h
k k 1

k 1 l og x dx l og k k l og x dx .
l og x

Summing overk 1, 2, , n
we get l og k

n n 1

0 l og x dx l og n ! 1 l og x dx
1
x
k 1 k k 1

or
l og x dx x l og x x

n l og n n l og n ! ( n 1) l og( n 1) n . (4-40) 26
PILLAI
The double inequality in (4-40) clearly suggests that log n
is close to the arithmetic mean of the two extreme numb
there. However the actual arithmetic mean is complicate
and it involves several terms. Since (n12 + )log n n is
quite close to the above arithmetic mean, we consider
the difference1
an l og n ! ( n 1
) l og n n . (4-41)
This gives 2

a n a n 1 l og n ! ( n 12 ) l og n n l og( n 1) !
( n 23 ) l og( n 1) ( n 1)

l og( n 1) ( n 12 ) l og n ( n 23 ) l og( n 1) 1
n 1 1
n 1
( n ) l og
1
2 n
1 ( n ) l og
1
2 n
2 2
1.
(n + )log n n is due
1 1
1According to W. Feller this clever idea to use the approximate mean
2 2

to H.E. Robbins, and it leads to an elementary proof. 1

2
27
PILLAI
 Hence1 1 1

a n a n 1 ( n 12 ) l og( 1 2 n 1
1 )1
2 n 1

2 ( n 12 ) 1 1 1 1
2 n 1 3( 2 n 1) 3
5( 2 n 1) 5

1 1 1 0 (4-42)
2 4 6
3( 2 n 1) 5( 2 n 1) 7 ( 2 n 1)
Thus {an} is a monotone decreasing sequence and let c
represent its limit, i.e.,
lim an c (4-43)
n
From (4-41), as this is equivalent to
n
n 1 n
To nd the constant n ec in
n ! ~ eterm (4-44) use
. (4-44), we can make
c 2

a formula due to Wallis ( 1655).

1By Taylor series expansion

1 2 1 3 1 4 1 1 2 1 3 1 4
28
l og( 1 x) x x x x ; l og ( 1 x ) x x x x
2 3 4 2 3 4 PILLAI
 The well known functionsi nx x si n c x si n x
x

goes to zero atx n ;

moreover these are the only n n
x
zeros of this function. Also
si n x has no nite poles.
x
(All poles are at innity). As a result we can write
si n x (1 x 2 ) (1 x 2 ) (1 x
2
)
x 2 4
2
n
2
2

or [for a proof of this formula, see chapter 4 of Dienes,

The Taylor Series]
si n x x (1 x ) (1 x ) (1 x ) ,
2 2 2

4
2
n
2 2 2

/2
which for x gives the Wallis formula
1 (1 ) (1 1 ) ( 13 ) ( 35 ) ( 57 ) (
( 2 n 1 ) ( 2 n 1 )
1
) (1 1
)
2 2
2
4
2
(2n) 2 2
2
4
2
6
2 2
(2n)
2
29
PILLAI
or 2

) ( ( 2 n 1 ) ( 2 n 1 ) )
2 2 2 2 (2n)
4n
( 2 n 1 ) ( 2 n 1 )
( 2
1 3
)( 4
3 5
)( 6
5 7
2 n 1

2 4 62n
( 1 3 5 ( 2 n 1 ) ) . 2 1
2 n 1

Thus asn , this gives

2 4 6 2n 1

1 3 5 ( 2 n 1)

n 1
2

( 2 4 6 2 n )2 1 2n ( n! )2 1
2 .
( 2 n )! ( 2 n )!
n 1 n 1
Thus asn 2 2

l og 2 n l og 2 2 l og n ! l og( 2 n ) ! 1 l og( n 1
) (4-45)
2 2

But from (4-41) and (4-43)

l i m l og n ! c l i m { ( n 12 ) l og n n } (4-46)
n n

and hence lettingn

in (4-45) and making use
30
PILLAI
of (4-46) we get
l og c 1 l og 2 l i m 1 l og(1 1
) c 1 l og 2
which gives 2 n 2
2n 2

With (4-47) in (4-44) .

c
2 obtain
e we (4-47)
(4-39), and this proves
the Stirlings formula.
Upper and Lower Bounds
It is possible to obtain reasonably good upper and
lower bounds for n! by elementary reasoning as well.
To see this, note that from (4-42) we get

an a n 1 1 1 1
3 2 4
( 2 n 1) ( 2 n 1)
1 1 1 1
so that {an 3[1/( 212 n
n 1)} 2is
1]a monotonically
1 2 n ( n 1) 1 2 n increasing
1 2 ( n 1)
31
PILLAI
sequence whose limit is also c. Hence for any nite n
a n 1 21 n c a n c 1 21 n

and together with (4-41) and (4-47) this gives

n 1 n 1 1
(4-48)
2 n
2
n! 2
e 12 n

Similarly from (4-42) we also have

a n a n 1 1 1 1 0
3( 2 n 1) 2 1 2 n 1 1 2 ( n 1) 1

so that {an 1/(12n+1)} is a monotone decreasing seque

whose limit also equals c. Hence
a n 1 2 1n 1 c a n c 1 2 1n 1
or
n! 2 n
n 1
2
e
n ( 1 1 2 n (1n 1 ) )
.
(4-49)
Together with (4-48)-(4-49) we obtain 32
PILLAI
 n 1 n
1
n 1 n
1

2 n 2
e e 1 2 n 1
n! 2 n 2
e e 12 n
. (4-50)

Stirlings formula also follows from the asymptotic

expansion for the Gamma function given by
x x 1
( x) 2 e 1 1 139 o ( 14 )
x 1
2

12 x 286 x2 51840 x3 x
(4-51)
Together with ( x 1) x ( x ) , the above expansion can
be used to compute numerical values for real x.
For a derivation of (4-51), one may look into Chapter 2
of the classic text by Whittaker and Watson (Modern
Analysis).

We can use Stirlings formula to obtain yet another

approximation to the binomial probability mass 33
PILLAI
function. Since
n k n k n! k n k
p q p q , (4-52)
k (n k )! k !
using (4-50) on the right side of (4-52) we obtain

n k n k n np k nq n k

p q c1 2 ( n k ) k k n k
k
and
n k n k n np k nq n k

p q c2 2 ( n k ) k k n k
k
where
1 2 n 1 1 2 ( n k ) 1 2 k
1 1 1

c1 e
and
1 2 n 1 2 ( n k ) 1 1 2 k 1
1 1 1

c2 e .
Notice that the constants c1 and c2 are quite close
34
to each other. PILLAI