Lecture - 5

February 10, 2012

Introduction
There are two approaches for estimation techniques. First one is frequentist approach where we usually assume that parameters are xed. The other is Bayesian approach. Here we assume that parameters are random and they have a probability distribution. In this lecture we concentrate on bayesian approach. In bayesian approach, since we have prior knowledge about the distribution of the parameters, we call it as prior distributions.

Bayes Estimation :
The likelihood function is interpreted as the conditional density of the data Y given and written as f Y | . If prior distribution of is (). E |Y = Then f Y, = ()f Y | f |Y d = ()f Y | d ()f Y | d

The marginal density of Y is f Y = and conditional density of given Y is f |Y = ()f Y | ()f Y | d ()f Y | d

is another form of bayes law. which is called posterior density of given observed data Y . The most common bayes estimators are mode and mean of posterior density. The mode is called the maximum a posterior estimator or MAP estimator. The mean of the posterior is E |Y = f |Y d = ()f Y | d ()f Y | d

This is sometimes called as bayes estimate. Similarly bayes estimate for h() is E h()|Y . Important Remark: We may nd diculty in calculating E h()|Y . Therefore we adapt monte carlo techniques to approximate this integration. Here I am going to talk about one such important monte carlo techniques i.e. Importance Sampling. It is very popular in application of various pricing in nance. Algorithm is as follows: X g(x). [Assumption : its easy to draw a random sample from g(x).] E h()|Y =
h(Xi )p(Xi |Y ) n i=1 g(Xi ) p(Xi |Y ) n i=1 g(Xi )

. [ Motivation is discussed in the class.]

Important Remark: We may have two types of advantages out of this methods. 1. Suppose p(Xi |Y ) = k pu (Xi ), where k is the normalizing constant. E h()|Y =
h(Xi )pu (Xi ) n i=1 g(Xi ) pu (Xi ) n i=1 g(Xi )

We rewrite

. Computing k can be dicult sometime. Therefore this

expression is advantageous as it uses pu (Xi ) instead of p(Xi |Y ). 2

2. This is useful in reducing the variance of the estimator. Intuition is as follows : we choose g(x) in such a way that whenever h() is small, h() is large,
pu (Xi ) g(Xi ) pu (Xi ) g(Xi )

will be large and whenever

will be small. [Please recall such choice explained in the class.]

Example :

Suppose, Y be the number of times the stock price increases on three

consecutive days. then Y is binomial with parameter n = 3 and . Suppose prior distribution of is () = 6(1 ) , . Suppose we have observed Y = 3, then what is MAP and bayes estimate of . likelihood function of Y in general f Y | = ; Here f Y | = 3 Now, posterior distribution of given by f (|3) = Therefore bayes estimate is
1

0<<1

3 y (1 )3y y

as Y = 3

()f 3| = ()f 3| d

64 (1 )
1 0

64 (1

)d

= 304 (1 )

305 (1 )d =
0

30 = 0.7143 42

and MAP estimate is

4 5

The posterior CDF is F |Y = 3 = 30 5 6 , 5 6 01

0.05 and 0.95 posterior quantities of are 0.418 and 0.9372 respectively. 3

Therefore 90% credible intervals is [0.4182, 0.9392] or 90% posterior probability that is between 0.4182 and 0.9372. Credible intervals are bayesian analogues of condence intervals. Important Remark: There can be several such intervals with same condence level, we generally work with shortest interval given a condence co-ecient.