Estimation: parts of Chapters 12-13

Wiener and Kalman Filtering

Natasha Devroye
devroye@ece.uic.edu
http://www.ece.uic.edu/~devroye

Spring 2011

Summary
Wiener filtering

!"#$%&"'()*+,--$."'()*/&%0"1$"-'
!"#$%&"'()*+,--$."'()*/&%0"1$"-'
!"#$%&"'()*+,--$."'()*/&%0"1$"-'
Terminology
Terminology for three
Terminology
for three different
for three
different ways
different
ways toways
to cast castWiener
the thecast
to Wiener filter problem
the Wiener
filter problem filter problem
!"#$%&"'( !"#$%&"'(
!"#$%&"'( +,--$."'( /&%0"1$"-'
+,--$."'(
+,--$."'( /&%0"1$"-'
/&%0"1$"-'
Given: ![0],![0],
Given: ![1], ![1],
Given: ![0],
…, !["] ![1], …,
…, !["] !["]
Given: ![0], ![1],
Given: ![0], ![0],
![#-1]
![1],
Given:
…, ![1],Given:
…, ![#-1] …, ![#-1]
![0], ![1],
Given: ![0], ![0],
…,![1],
Given: …,![1],
![#-1] …, ![#-1]
![#-1]
$ˆ[" ] $ˆ[" ] Find: $ˆ[" ]
Find:Find: $ˆ[0], $ˆ$ˆ[[10],],
Find: Find: Find: [0!
!$ˆ[,1$ˆ],[$ˆ#
! 1],!!
],, 1$ˆ$ˆ[][# !ˆ![ #
1], $ˆ[ #Find:
Find: [,# %#"% ]0!ˆ,[ #%#"%0] , % " 0
1] # %!ˆ]Find:
Note!! Note!! Note!!
!["] !["] !["] !["] !["] !["] !["] !["] !["]

2 2 2 2 2 2 2 2 2
1 "1 " 1 "1 1" 4 51 " 3 "
1 3 3 " 3 13 3 "3 13 3 4 5 4" 5
$̂[0] $̂[0] $̂[0] $̂[1] $̂[2] $̂[30]] $̂[1] $̂[2] $̂[3] !ˆ[5]
$̂[0] $̂[0] $̂[1] $̂[2] $̂[3] !ˆ[5]
!ˆ[5]
$̂[1] $̂[1]
$̂[1]
All three solved usingsolved
All three General LMMSE
using Est.LMMSE Est.
General
$̂[2] $̂[2] All three solved using General LMMSE Est.
$̂[2]
!1 !1
$̂[3] $̂[3] !ˆ $ 3
ˆ! $ 3!ˆ22
!23 $23!!2
1 3 22 2
!2 3 22 2
http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_28%20Ch_12B.pdf
$̂[3]
11 11
11

Wiener filtering

!"#$%&"'()*+,--$."'()*/&%0"1$"-'
!"#$%&"'()*+,--$."'()*/&%0"1$"-'
!"#$%&"'()*+,--$."'()*/&%0"1$"-'
Terminology
Terminology for three
Terminology
for three different
for three
different ways
different
ways toways
to cast castWiener
the thecast
to Wiener filter problem
the Wiener
filter problem filter problem
!"#$%&"'( !"#$%&"'(
!"#$%&"'( +,--$."'( /&%0"1$"-'
+,--$."'(
+,--$."'( /&%0"1$"-'
/&%0"1$"-'
Given: ![0],![0],
Given: ![1], ![1],
Given: ![0],
…, !["] ![1], …,
…, !["] !["]
Given: ![0], ![1],
Given: ![0], ![0],
![#-1]
![1],
Given:
…, ![1],Given:
…, ![#-1] …, ![#-1]
![0], ![1],
Given: ![0], ![0],
…,![1],
Given: …,![1],
![#-1] …, ![#-1]
![#-1]
$ˆ[" ] $ˆ[" ] Find: $ˆ[" ]
Find:Find: $ˆ[0], $ˆ$ˆ[[10],],
Find: Find: Find: [0!
!$ˆ[,1$ˆ],[$ˆ#
! 1],!!
],, 1$ˆ$ˆ[][# !ˆ![ #
1], $ˆ[ #Find:
Find: [,# %#"% ]0!ˆ,[ #%#"%0] , % " 0
1] # %!ˆ]Find:
Note!! Note!! Note!!
!["] !["] !["] !["] !["] !["] !["] !["] !["]

2 2 2 2 2 2 2 2 2
1 "1 1 "1 1"
1 3 3 " 3 " 13 3 "3 13 4 531 4" 53 4" 5 "
$̂[0] $̂[0] $̂[0] $̂[1] $̂[2] $̂[30]] $̂[1] $̂[2] $̂[3] !ˆ[5]
$̂[0] $̂[0] $̂[1] $̂[2] $̂[3] !ˆ[5]
ˆ ![5]
$̂[1] $̂[1]
$̂[1]
All three solved usingsolved
All three General LMMSE
using Est.LMMSE Est.
General
$̂[2] $̂[2] All three solved using General LMMSE Est.
$̂[2]
!1 !1
$̂[3] $̂[3] !ˆ $ 3 3!ˆ22
$323!!2
1 3 22 2
!ˆ $
!23 2
http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_28%20Ch_12B.pdf
$̂[3]
!2 22 11 11
11
Wiener filtering

!1
!ˆ " "!! " !! !

Filtering Smoothing Prediction

! " "[! ] (scalar) ! " # (vector) ! " '[ ) ! 1 ( ( ] (scalar)

# $
"'! " % "[! ]! # "!! " % #! ## $ x '()*s!

" % #"[! ] # $ #
" % # #( # ( & ) $ # #
"'' " % '[ ) ! 1 ( ( ] ! # $
" %$"" [! ]! $"" [0]& " % # ## ( #& $
# # " %$'' [ ) ! 1 ( ( ]! $'' [( ]&

"~
$""T (vector! ) " % "" (Matrix! ) "~
$''T (vector! )

#
" '' " % (# ( & )( # ( & )# $ #
" '' " % (# ( & )( # ( & )# $
" % ### #
( && # $ " % ### #
( && # $ # $
" '' " % !! #
" % ''
" % "" ( % && " % "" ( % &&

$""# ( % "" ( % && ) !1 !

"ˆ[! ] " ~ #ˆ " % "" ( % "" ( % && ) !1 ! 'ˆ[ ) ! 1 ( ( ] " ~ # !1
$'' % '' !
%1)( ! (1)&%( ! (1))( ! (1)&%( ! (1))1& % ) ) ) &% ) ) ) &% ) )1& %1) ) &% ) ) ) &% ) )1&
12

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_28%20Ch_12B.pdf

Wiener filtering

!"##$%&'("%()*+&$,*%-.()/0(1*$%$,
,##! ( 0 ## " 0 "" ) #1 2 ! 3! 2
#ˆ[$ ] ! ~
$ !! !#!!! "
3! $
#ˆ[$ ] ! & & ( $ ) [ ' ] ([ $ # ' ]
$ %
4 ! & ( $ ) [0] & ( $ ) [1] ... & ( $ ) [$ ]
! ' !0
Wiener Filter as
Time-Varying FIR
! $% $ % $ #1 ... % 0 % !
Filter
• Causal!
• Length Grows!
Wiener-Hopf Filtering Equations
, )(( [0] )(( [1] & )(( [$ ] ) , & ( $ ) [0]) , )## [0])
* ) [1] * '
( 0 ## " 0 "" )4 ! ,##
$! !#!! " * (( )(( [0] & )(( [$ # 1]'' * & ( $ ) [1] ' ** )## [1] ''
!
0 (( * % % ' % '* % ' * % '
* ' * ' * '
,## ! $)## [0] )## [1] ... )## [$ ]% ! )(( [$ ] )(( [$ # 1] & ) [0] & ( $ ) [$ ]'( + )## [$ ](
+$ !!!!!!#!!!!((!!"( *+
Symmetric & Toeplitz

In Principle: Solve WHF Eqs for filter 4 at each $

In Practice: Use Levinson Recursion to Recursively Solve 13

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_28%20Ch_12B.pdf
Kalman filtering

Borrowed heavily from the excellent notes

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13A.pdf

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13B.pdf

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13C.pdf

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13D.pdf

Kalman filtering

• Rudolf Kalman developed in 1960s

• discrete-time and continuous time versions

• used in Control systems, Navigation systems, Tracking systems

Background

• Wiener filter: LMMSE of changing signal (varying parameter)

• Sequential LMMSE: sequentially estimate fixed parameter

• State-space models: dynamical models for varying parameters

• Kalman filter: sequential LMMSE estimation for a time-varying parameter

vector that follows a ``state-space’’ dynamical model (i.e. not arbitrary
dynamics)

State-space / dynamical models

• System state: variables needed to predict system at future times in absence

of inputs (i.e. what you need to keep track of)

• Example: constant velocity aircraft in 2-D

Vector Gauss-Markov Model

Thm. 13.1 (Vector Gauss-Markov Model)

Example - constant velocity 2-D aircraft

Ex. Set of “Constant-Velocity” A/C Trajectories

14000 Red Line is Non-Random
Constant Velocity Trajectory
12000

10000

Y position (m)
8000 Acceleration of
(5 m/s)/1s = 5m/s2
6000
# ! 1 sec

4000 $ u ! 5 m/s

rx [ "1] ! ry [ "1] ! 0 m
2000
v x [ "1] ! v y [ "1] ! 100 m/s
0
0 2000 4000 6000 8000 10000 12000 14000 16000 18000
X position (m)
12

Have state model, now observation model

• Have a state-space or dynamical system model for the desired signal

• Need a model for the noisy measurements and how it relates to the states
(depends on how we acquire the data)
Estimation problem

Scalar state, scalar observation Kalman

Scalar state, scalar observation Kalman

Let’s derive this!!!

Some observations

• dynamical model provides update from estimate to prediction

• in the Kalman filter, prediction acts like the prior information about the state at
time n before we observe the data at time n

• must know noise and initial state distributions and dynamical and observation
model
!"#$"%&'(#)*+,&-."#"+&-)")*&&/&&-."#"+&012*+3")(4%
s[n] $ as[n # 1] " u[n] u[n] WGN; WSS; ~ N (0, ! u )
2
-)")*&546*#:
Varies
012*+3")(4%&546*#: x[n] $ s[ n] " w[n] w[n] WGN; ~ N (0, ! n2 ) with n

sˆ[#1 | #1] $ E{s[#1]} $ % s 572)&!%48: :%%s,s,!!s2s, ,a,a,!!u2,u,!!n2n

572)&!%48 2 2 2
9%()("#(:")(4%:
M [#1 | #1] $ E{( s[#1]} # sˆ[#1 | #1]) 2 } $ ! s2

;+*6(.)(4%: sˆ[n | n # 1] $ asˆ[n # 1 | n # 1]

;+*6<&5-=: M [n | n # 1] $ a 2 M [n # 1 | n # 1] " ! u2

M [n | n # 1]
!"#$"%&>"(%: K [ n] $
! " M [n | n # 1]
2
n

?@6")*: sˆ[n | n] $ sˆ[n | n # 1] " K [n]& x[n] # sˆ[n | n # 1]'

=2)<&5-=: M [n | n] $ &1 # K [n]'M [n | n # 1] 13

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13B.pdf

!"#$"%&'(#)*+,&-*.)/+&0)")*&&1&&0."#"+&234*+5")(/%
0)")*&</=*#: 4[n ] $ ;4[n # 1] " :9[n ] 4 p ! 1; ; p ! p; : p ! r; 9 ~ N (78 6) r ! 1

234*+5")(/%&</=*#: x[n] $ > [n]4[n] " w[n]; > [n] p ! 1

T T
w[n] WGN; ~ N (0, % n2 )

<94)&!%/A:((s,s,??s,s,;,
<94)&!%/A: ;,:, 6,%%n2n
:,>,>,6,
2
B%()("#(C")(/%: 4ˆ[#1 | #1] $ E{4[#1]} $ @ s
&
<[#1 | #1] $ E (4[#1]} # 4ˆ[#1 | #1])(4[#1]} # 4ˆ[#1 | #1])T $ ? s '
D+*=(.)(/%: 4ˆ[n | n # 1] $ ;4ˆ[n # 1 | n # 1]
D+*=E&<0F&Gp!pH: <[n | n # 1] $ ;<[n # 1 | n # 1];T " :6:T
<[n | n # 1]>[n]
!"#$"%&I"(%&Gp!JH: ![ n ] $
% " >$T !
2
n <!
[n]! [n#| n!#! >["
1]! n]
1!1

KL=")*: 4ˆ[n | n] $ 4ˆ[n | n # 1] " 

$!!#!!"
ˆ [ n|n #1]
$!!! !#x! !!! "
~
x [ n ]: innovations

F4)E&<0F&Gp!pH: : )
<[n | n] $ B # ![n]>T [n] <[n | n # 1] * 14

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13B.pdf
!"#$"%&'(#)*+,&-*.)/+&0)")*&&1&&-*.)/+&234*+5")(/%
0)")*&</=*#: 4[n] $ ;4[n # 1] " :9[n] 4 p !1; ; p ! p; : p ! r ; 9 ~ N(78 6) r !1

234*+5")(/%: A[n] $ @[n]4[n] " ?[n]; A M !1; @[n] M ! p; ?[n] ~ N(78 >[n]) M !1

C%()("#(D")(/%: 4ˆ[#1 | #1] $ E{4[#1]} $ B s <94)&!%/?: :''s,s,>>s,s,;;, ,::, ,@@, ,66, ,>>[n]}
<94)&!%/? [n]}
%
<[#1 | #1] $ E (4[#1]} # 4ˆ[#1 | #1])(4[#1]} # 4ˆ[#1 | #1])T $ > s &
E+*=(.)(/%: 4ˆ[n | n # 1] $ ;4ˆ[n # 1 | n # 1]
E+*=F&<0G&Hp!pI: <[n | n # 1] $ ;<[n # 1 | n # 1];T " :6:T
#1
- *
!"#$"%&J"(%&Hp!MI: ![ n] $ <[ n | n # 1]@ [n]+ >[ n] " @[ n]<[ n | n # 1]@ T [ n] (
T
+ $!!!#!!!" (
, M !M )
KL=")*: 4ˆ[n | n] $ 4ˆ[n | n # 1] " 
$!!#!!"
[ n|n #1]
$!!! !#Aˆ! !!! "
~
A [ n ]: innovations

G4)F&<0G&Hp!pI: :
<[n | n] $ .C # ![n]@[n]/<[n | n # 1] 15

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13B.pdf

!"#$"%#&'()'**+,',-.%/0.%(1
5#1='**+, >($C#'7%1#0$
K--6/#'' “Squared” Cost Function K14'LB>M'
506--%01 !ˆ ! E{! | :} ?1(&1'N19 *(/#1.-

2(%1.34' $1
!ˆ ! E{!} % F !: F :: ": $ E{:}# 7**+,
506--%01

804#-%01' 7**+,'
7%1#0$' "
!ˆ ! H ! % F! GT GF! GT % F w #
$1
": $ GH ! # 7%1#0$'
*(9#3 *(9#3

!;.%/03' 7%1#0$'
+#<='>%3.#$ &
!ˆ n ! !ˆ n $1 % J n x[n ] $ ITn !ˆ n $1 ' +#<='>%3.#$
(No Dynamics) @A('B410/%C-D

!;.%/03' 7%1#0$'
?03/01'>%3.#$ -ˆ[n | n ] ! -ˆ[n | n $ 1] % ? [n ]( :[n ] $ G[n ]K-ˆ[n $ 1 | n $ 1]) ?03/01'>%3.#$
(w/ Dynamics) @&E'B410/%C-D17

http://www.ws.binghamton.edu/fowler/fowler%20personal%20page/EE522_files/EECE%20522%20Notes_29%20Ch_13B.pdf
Properties of the Kalman filter

• extension of sequential MMSE (fixed parameter) to time-varying parameter

with a known dynamical model

• Kalman filter is a time-varying filter (Kalman gain changes withup-down

5. There is a natural n) progression in the error
• The Prediction Stage increases the error
• The Update Stage decreases the error M[n|n – 1] > M[n|n]

• Kalman filter computes and uses its performance measure M[n|n] • This is OK… prediction is just a natural, intermediate step in the Optimal
processing
M[5|4]
M[6|5]
M[7|6]
• prediction increases error, update decreases error
M[5|5]
M[6|6] M[8|7]
M[7|7]

5 6 7 8 n

• as n→! Kalman filter reaches ``steady-state’’ and6.becomes•

a linear time-
Prediction is an integral part of the KF
And it is based entirely on the Dynamical Model!!!
invariant filter (i.e. k[n] constant, M[n|n] constant) 7. After a “long” time (as n ! ") the KF reaches “steady-state”
operation… and the KF becomes a Linear Time-Invariant filter
• M[n|n] and M[n|n – 1] both become constant

• Kalman filter created uncorrelated sequence of ``innovations’’ •

… but still have M[n|n – 1] > M[n|n]
•
Thus, the gain k[n] becomes constant, too.
2

• Kalman filter is optimal for Gaussian, if not Gaussian, optimal Linear MMSE

• M[n|n-1], M[n|n], K[n] can be computed off-line (ahead of time)

More observations

• Kalman vs Wiener?

• What about if don’t have linear observation and dynamical models?

• Much more to know about Kalman filter!!!