Estimation Theory Estimation Theory

Chapter 6
Best Linear Unbiased Estimators (BLUE)
Tried 1) CRLB Theory
2) Linear Model 3) Sufficient St at ist ics
All of t hem assume PDF knowledge, what if we
don t know ( or can t make assumpt ion on) dat a PDF ?
R t i t t i t t b Li Rest rict est imat or t o be Linear

1
] [

N
u
and find MVU est imat or wit hin t his class of est imat ors

=
=
0
] [
n
n
n X a u
Linear MVU Est imat or BLUE

OPTI MALI TY OF BLUE?

Best Linear Unbiased Estimators
] [
1

Recall
1
in WGN
level DC
|
|
.
|

\
|
= =

N
n X
N
X u
1

0
in WGN
=
. \
=
n
n
N
a
N
be to happens estimator MVU
if BLUE Estimator MVU =
known) A" (" estimation variance AWGN : Example Counter
noise) Gaussian with model data linear in (like linear
) ) ( (
1

Estimator MVU
1
0
2 2
= =

=
N
n
A n x
N
o

data in linear Not
0

n
Best Linear Unbiased Estimators
Any linear est imat or is sub- opt imal ( in general)
Somet imes BLUE is t ot ally wrong Somet imes, BLUE is t ot ally wrong
ance with vari WGN W[n] X[n]
2
o = Example :
] [
N
1

Estimator MVU
ance with vari WGN W[n] X[n]
1
2 2
o
o
= =

N
n X
Example :
3.6) Example (see
N
1
0

=
N
n
)

E( bi d b T
] [

BLUE
2 2
1
0
2
o = =

=
N
n
n
n X a
)

E( unbiased be To
2 2
o o =
Best Linear Unbiased Estimators
1 N
0 ]] [ [ )

E( But
2
1
0
2
o o n X E a
N
n
n
= = =

=
. Estimator Linear use t Can'
unbiased.

make to ' a find to Impossible

2
n
o s
Finding the BLUE
]) [ ( )

E( : Constraint Unbiased
1
n X E a
N
n
= =

u u
]) [ ( ] [ )

V ( V i
]) [ ( ) (
2
0
X E X E
n
n
(
(

|
|

|
=

u
] [ L t
]) [ ( ] [ ) Var( : Variance n X a E n X a E
T
n n
n n
(
(

|
.

\
=

u
| |
] [ a Let
1 1 0
a a a
T
N
=

( ) | | ) ( )

(
2
X E a X a E Var
T T
= u

Finding the BLUE
( ) =
2
] [ ) X E( X ( a E
T
( )( ) = ] [
C
a ) X E( X ) X E( X a E
T
T T
( )( ) =
=
) ( ) ( C where

] [ X E X X E X E
a C a
T
T
( )( )
=
h ld i bi d F
X of Matrix Covariance
) ( ) ( ] [

1 - N
]) [ ( )

E(
hold to constraint unbiased For
u u X E

=
= =
0 n
n
]) [ ( a ) E( u u n X E
Finding the BLUE
ly equivalent or in linear be be must E(X[n]) u
known ] [ S[n] E(X[n]) n S = u
] [ S[n]
E(X[n])] - [X[n] E(X[n]) X[n] Since
n W + =
+ =
u ] [ S[n] n W + u
linear model but
without Gaussian noise assumption
BLUE Applicable to Amplitude Estimation of Signals In Noise
over minimize must
1 - N
)

Var( , a a C a
T
= u Summary
( )
( )
constraint o subject t
1 N
0 n
n
] [ a n X E =

=
u
( )
( )
where
] [ ] [
) ( X[n] E
n S a n X E a
n S
n n
= =
=

u u
u
( )
1 1 ] [ a
] [ ] [
S a n S
n S a n X E a
T
n
n
n
n
= =

u u
mean scaled the is 1)] - N S(1)....S( ) 0 ( [ S where
T
n
n
1 1 ] [ a
S
S a n S
=

ea sca ed t e s )] N S( )....S( ) 0 ( [ S w e e S
S C
T T
) 1 ( F
Technique Multiplier Lagrangian Use : Solution

A
Y A
S a a C
T
T
T T
= =
c
+ =
A for Y 2A
Y
g Usin
) 1 ( a F
Y
A
Y
c
= =
c
T
b
A for Y 2A g Usin
b
Y
Y
T
=
c
cb

Gradient
Y
g
Y
g
Y
g
T
N
=
(

c
c
c
c
c
c
=
c
c

2 1
Y
g
e Wher
N 2 1
S a C
a
F
0 2 = + =
c
c

S C a
1
2

=

S C S S
T T 1
1 a
constraint use ) multiplier n (Lagrangia find To

S C S S
1
2
-
1
2
a
=
= =

S C
S C S
T
1
1
a
2

=
S C S
T 1
OPT
a

=
To Find Minimum Variance
S C C C
a C a Var
T
OPT
T
OPT
T
1 1
S
)

(

= u
S C S
S C C C
T
T
1
2
1
) (
S

=
S C S S C S
S C
T T
T
1
2
1
1
1
) (
S

= =
X C
X a
T
T
1
S

= u
S C S
X C
T 1
1

= u
BLUE
S C S
T 1
1

Var

= u

) (
)

E(
unbiased is that show To
1 1
u
u
u
u
S C S X E C S
T T
need we BLUE find To
) (
) E(
1 1
= = =

u u
S C S S C S
T T
S d ) f C i ( C 2)
Moment First - Mean) Scaled ( S 1)
need we BLUE, find To
4) Chapter in as ( PDF entire not But
Moment Second ) X of Covariance ( C 2)
& first as long as data d distribute - uniformly
as data Gaussian for estimator same get We
s pdf' both for same are moments second
& first as long as data d distribute uniformly
+ =
2
with var noise white W[n] A X[n]
Noise in White Level DC
o
Example :
(not necessarily Gaussian)
+ =
i d d il W[ ] f S l
W[n] of PDF know t Don'
- with var noise white W[n] A X[n] o
(Cant apply CRLB Theorem)
PDF) Gaussian assuming not re we' (since ed uncorrelat only
t, independen y necessaril not are W[n] of Samples

= = =
2
1
1 1 1 S
A

1 1 ] [ A E(X[n]) But
T T T
X X I X C
S n S
o

= = =
1
2
1
1
1 1
1
1 1 1
1 1
S
S
A
N
T T T
X
I
X I
S C
X C
o
o

=
=
1
0
] [
N
1

N
n
n X
Example (Contd)
1
1 1
)

(
2
1
N
S C S
A Var
T
T
o = = =

data, of PDF of t independen BLUE is mean Sample
1
1
1
2
I
S C S
T
o
OPTIMAL?
data, of PDF of t independen BLUE is mean Sample
Not necessarily MVU but for Gaussian noise, it is efficient
and hence MVU using the CRLB Theorem
C or colored is W[n] Now
noise colored in Level DC
2
= I o
Example :
1 1
1
A

C or colored is W[n] Now
1
1
=
=

T
T
C
X C
I o
Consider . C some assume to Need
1 1
1
(
T
C
0
C
2
1
2
0
(
(
(

= o
o
variances different have but ed uncorrelat are Samples
0
2
1
(
(

N
o
variances. different have but ed uncorrelat are Samples
(
(
(

2
0
0
1
o
(
(
(
(

2
1
0
1
1
0
1
C
o
o
(
(

2
1
0
1
1
0
N
o
(
(

(
(
(

2
1
2
0
2
1
2
0
] 1 [
] 0 [
1
1
0
0
1

N
T
N
T
N X
X
X
A
o
o
o
o

(
(
(

=
(
(
(

=

2
0
1
2
0
1
1
1
1
] [

1
1
0
1
1

T
N
T
N
A
o o
(
(

(
(

2
1
2
1
1
1
0
N
N
o
o
1 N
] [

1
1
0
2
=

=
o
N
N
n
n
n X
1 1
1
1
0
2

=
o
N
n
n
1
1
1 1
1
)

(
1
2
1
= =

o
N
n
T
C
A Var
i ll
have they if heavily more samples weighs BLUE
0 = n
? A

is what , 0 If
riance. smaller va
2
0
o
( ) ] [ X[ ] E
model Our
S u ( )
since Model Linear General he actually t is
] [ X[n] E = n S u
])) [ ( ] [ ( S[n] X[n]
Mean) (Zero W[n]
+ = n X E n X u
Gaussian W assuming ithout Although w
X + = W Su
Gaussian, is X or Gaussian is W If
Gaussian W assuming ithout Although w

Estimator MVU BLUE
BLUE FOR VECTOR PARAMETER

] [
1
2 1

u u u u
N
T
p
N p is A

Or
1,2,....p i ] [

Propose
0 =

= =

u
u
X A
n X a
n
iN i
unbiased be To
N p is A Or
1
= u X A
N

]) [ ( )

E(
1
0
p 1,2,.., i

=
= = =

u u n X E a
i
N
n
in i
constraint satisfy To
) ( A )

E( Or = = u u X E
satisfied be must ) X E(
Known |
= u H
BLUE FOR VECTOR PARAMETER
] 0 [
assumed we case scalar For
S
(
] 1 [
] 1 [
] 0 [
) X E(
N S
S
S
(
(
(

u
p N is H Now
] 1 [
1 N H
N S

i b A l N
) ( A
p N is H Now
I H A H A X E = = =

u u u
i i A
as written be A let Now
th
1
T
T
a
(
(

row i is a A
th
T
i
T
p
a
(
(

=

BLUE FOR VECTOR PARAMETER
] [ H Also
Columns
2 1
(
=
|
p
h h h
| |
1
1
=
(
(

=
p T
p
T
I h h
a
a
I H A
a a a
T T T
T
1 2
T
1 1
T
1
(
(

p
p
I
h h h
a
a a a

T
p
T
2 2
T
2 1
T
2
=
(
(
(

p
p
I
h
h h h

j i
j i

0
1
a is constraint unbiased The
=
=
)
`

= =
ij j
T
i
h o

j i 0
p 1,2...., j 1,2,....p; i for = =
=
)
BLUE FOR VECTOR PARAMETER
1 2 i )

V (
before) as (same is minimized be to variance The
C
T
u
6B) Appendix See ( Result
p 1,2,..., i ) Var( = = a C a
i i i
u
) H ( C
) H (

1
1
1
1
1
=

H C
X C H H C
T
T
T
u
where Model Linear General as result Same
) H ( C
1

= H C
u
estimator MVU is BLUE Gaussian is X If
) , 0 ( W X

~ + = C N W Hu
estimator MVU is BLUE Gaussian, is X If
SUMMARY
X
form has data If : Theorem Markov Gauss
0 ) W E( 1 = + = W H N u
is BLUE then , arbitrary) is W of (PDF
X

C Covariance
0 ) W E( 1
1
=
=
| |
+W H N
p p N
u
is variance minimum The
) (

1
1
1

= X C H H C H
T T
u
] ) [( )

Var(
is variance minimum The
1
1

= H C H
ii
T
i
u
) ( C
is matrix covariance estimator And
1
1

= H C H
T

) ( C H C H
u
SK Problem 6.1 p. 146 p
( ) x C H H C H A
1 T 1 - 1 T
^
=

( )
| |
( )
I C ; r r 1 H , where
n
1 N
2 T 1 - N
o = =

( )
( )
r
r n x
r n x
1
r
1
A
1 N
n 2
n
0 n
n
1 N
0 n
2
1
1 N
0 n
n 2
2
^
=
o
|
.
|

\
|
o
=

=
r
0 n

=
1 r iff 0 A var ;
r
H C H
1
A var
^
1 N
n 2
2
1 T
^
>
|
.
|

\
|
o
= =
|
.
|

\
|

0 n

=
SK Problem 6.12 p. 149 p
invertible is B & W H X Since + = u
B
invertible is B & W H
b
X Since
+ =
+
u o
u
( ) B
1 -
b = o u
( )
B H B H X
B H X
1 - 1 -
1 -
W b
W b
+ =
+ =
o
o
B H B H X
` `
1 - 1 -
W b
H X
+ = + o


Problem 6.12 (Cont.) ob e 6 (Co )
( ) X C H H C H
T T

' 1 ' 1 ' 1 '

^
( )
( ) ( )
( ) ( )
b HB X C H B
X C H H C H
T T
+ =
=

1
1 1 1 1 - 1 - T T -
1 1 1
HB C H B
o
( ) ( )
b BB
b HB X C H
T
+ =
+ =

1
^
1 1
1
1 - T
B
H C H B
u
b
b BB
+ =
+
^
B
B
u
u
= + = 1 H w A H x
6.16) (SK Exercise
|
.
|

\
|
=
= + =
1
^
T 1 1
^
T
^
x C H H C H A
1 H w A H x
=
|
.
|

\
|
=
(

. \

^
1
^
T 1 1
^
T
^
! C any for unbiased A HA C H H C H A E
(

|
.
|

\
|
=
(

. \
2
^ ^
A A E A var
(

|
.
|

\
|

|
.
|

\
|
=

. \
2
HA
1
^
T 1 1
^
T
A x C H H C H E

. \
. \
+w HA
Exercise (Contd)
(

|
.
|

\
|
|
.
|

\
|
=

w C H H C H E
2 1
^
T 1 1
^
T
| |
| |
=
(

. \
. \

H C ww E C H
1
^
T 1
^
T
|
.
|

\
|

H C H
^ ^
2 1
^
T
|
.
|

\
|
=

H C H
H C C C H
2 1
^
T
1 1 T
(
(
(

=
(

= =
|
.

\
1
0
0 1
C
0
0 1
C I; C for
1 -
^ ^
(
(

o
(

o
1
0
0
;
Exercise (Contd)
2
^
1
1+
(
2
2

1
1
A var
o
o
|
.
|

\
|
+
=
(

( )
2
^
A var
(

( )
( )
2
2
1
^
1
1 2
A var
A var
,
o
o
c
+
+
=
(

= Let
1 o

=