Sie sind auf Seite 1von 40

Lecture 1
Introduction
Dr. Tahir Zaidi

Course Content

(Tentative)

Textbook:
S.Haykin, Adaptive Filter Theory, 4th Ed., Prentice Hall, 2002

Introduction
BACKGROUND REVIEW
2.
Discrete-time and random signals
3.
Mathematical Tools
OPTIMAL LINEAR FILTERS
4.
Wiener Filters
5.
Linear Prediction
6.
7.
Family of LMS Algorithms
8.
Method of Least Squares
9.
Recursive Least Squares (RLS) Algorithm
10.
Square-Root Algorithms
11.
LMS and RLS Algorithms: Practical Issues
12.
Kalman Filtering (?)
APPLICATIONS
13.
Spectrum Estimation
14.
Array Processing
1.

## S.Haykin, Adaptive Filter Theory, 4th Ed.

Definition of filtering

A filter
is commonly used to refer to a system that is designed to
extract information
prescribed quantity of interest
from
noisy data.

Filter

## Often specified in frequency domain: lowpass, highpass etc. filters to

remove out of band signals.

Often the desired signals and undesired signals are at the same
frequency band. In such a case statistical characterization for filter
design is needed.

## A filter is linear if its output is a linear function of the input, i.e.,

Input

Output

x1

y1

x2

y2

a1 x1 + a2 x2

a1 y1 + a2 y2

Applications

Speech/Image Processing; echo and noise cancellation,
biomedical engineering
Others; seismology, financial engineering, etc.

Applications

## !!! Noise and errors are statistical in nature !!!

We will use statistical tools.

## Basic Kinds of Estimation

Filtering
(real-time operation)

Smoothing
(off-line operation)

Prediction
(real-time operation)

Filter
Linear

Non-Linear
Otherwise, it is non-linear.

## A filter is linear if the filtered,

smoothed or predicted quantity
at the output of the filter is a
linear function of the observations
applied to the filter input.
u(t)

Linear

or

Non-linear

or

y(t)
Filter

u(n)

y(n)

## !!! Non-linear filters may be hard to

Analyse, if not impossible !!!

Optimum Filter

## Definition: Solution of an optimization problem wrt.

a certain criterion with statistical parameters.

Nonlinear:
Maximum Likelihood (ML) sense (very difficult to implement)
Linear:
Minimum Mean Square Error (MMSE) sense

## Wiener filters, (Stationary environment)

Kalman filters, (Non-stationary environment)

Etc.

10

filters

## Requires a priori information about the environment

Example: Under certain conditions the so called Wiener filter is
optimal in the mean-squared sense

algorithm

priori

## Wiener Filter requires -a priori information of several statistics

-estimation (knowledge of the system)
is needed before filtering
-inversion of a huge matrix
-computationally inefficient!

Recursive algorithm
No complete a priori information required

## If the environment is stationary converges to the Wiener soln.

non-stationary tracks the changes.

12

## Rate of convergence (to the optimum Wiener soln.)

Misadjustment (deviation from the optimum Wiener
soln.)
Tracking (the variations in a non-stationary environment)
Robustness(to disturbances of small energy)
Computational Requirements/Cost
Numerical Properties (Numerical stability & accuracy)

13

## Linear Filter Structure

Structure:
FIR (Finite-duration Impulse Response)

IIR

Lattice
Systolic array

## (Infinite-duration Impulse Response)

14

Transversal Filter
Convolution
Sum

unit-delay
element

multiplier

15

Transversal Filter

## xH: Hermitian transpose

xH=(xT)*

16

Lattice Predictor

## Predictor order (# of stages): M

Forward prediction error

## m: the mth reflection coefficient

Input seq. u(n) is correlated, backward prediction error
b(n) is uncorrelated
Together with m, b(n) approximates d(n) (innovations
process).

17

Lattice Predictor

18

Systolic Array

## Represents a parallel computing network

Used for efficient pipelined operation

Matrix multiplication
Triangularisation
Back substitution (Matrix eqn. solving)

## Example 3x3 matrix/3x1 vector

19

Systolic Array

Boundary cell

Internal cell
s

y3

20

IIR Filter
May have stability problems,
We will prefer FIR filters for

21

Error
Difference

between
the filter output
a desired response
Mean Square Error
Weighted

Filter

y(n)

d(n)

(n) : Error
+
-

Error Squares

22

## Cost Function depends on Mean-Square Error criterion

(!!! Stochastic, depends on second order statistics !!!)
Solution of Wiener-Hopf Equations

updated value

of tap - weight

vector

old value

of tap - weight

vector

parameter
of

Requires
Expectations
E{.}

updated value

of tap - weight

vector

old value

of tap - weight

vector

error

parameter
input

signal

## (step size) vector

23

Least-Squares Estimation

## Cost Function depends on sum of weighted error squares

Low computational complexity due to recursive operation

Three

Standard RLS

## Relies on Matrix Inversion Lemma

Numerically unstable, high computational complexity

## Square-root RLS algorithm

categories

Based on QR-decomposition
Numerically stable

## Exploits certain matrix structures to reduce complexity.

24

Applications

Four Classes

Identification

Inverse modeling

deconvolution
equalisation

Prediction

system identification
layered earth modeling

## linear predictive coding

spectrum analysis
signal detection

Interference cancellation

noise canceling
echo cancellation

25

Parameters

## u=input of adaptive filter=input to plant

d=desired response=output of plant
e=d-y=estimation error

Applications:

System identification

System Identification

## Observing the output of a

plant(system), given the input
signal, tries to estimate the
IR of the plant.
Filter coefficient are found by

27

Inverse Modeling

Parameters

## u=input of adaptive filter=output to plant

d=desired response=delayed system input
e=d-y=estimation error

Applications:

Equalization

Removes intersymbol
interference (ISI).
Filter coefficient are found

29

Prediction

random signal

Parameters

## u=input of adaptive filter=delayed version of random signal

d=desired response=random signal
e=d-y=estimation error=system output

Applications:

Parametric (AR)
model
Linear AR filter

## input: white noise

output: observed
signal
aim: find the model
parameters by an

31

Interference Cancellation

Parameters

## u=input of adaptive filter=reference signal

d=desired response=primary signal
e=d-y=estimation error=system output

Applications:

Echo cancellation

Electrocardiography (ECG)
Acoustic noise in speech
Active noise cancellation

33

Echo Cancellation

## Coupling due to imperfect

balancing in hybrid
transformer creates an echo
in analog telephone lines.
Echo signal can be estimated
by an adaptive filter and the
subtracted out.

34

Multiple sensors
(antenna, microphone,
etc) used to steer the
beam to a specific
position.
Commun. systems,
Astrophysical
exploration,
Biomedical signal
processing, etc.

35

Historical Notes

## To understand a science it is necessary to know its history.

Auguste Comte (1798-1857)

## Method of least squares, Gauss, 1795

Minimum mean square error estimation, late 1930s
Discrete-time Wiener-Hopf equation, Levinson, 1947
Kalman filter, Swerling, 1958 and Kalman, 1960

## Stochastic approximation, Robins and Monro, 1951

LMS algorithm, Widrow and Hoff, 1959

36

Historical Notes

## Kalman filter, Godard Algorithm, Godard, 1974

Relationship between RLS and Kalman, Sayed and Kailath, 1994
QR decomposition based systolic array, Gentleman & Kung, 1981
Fast RLS algorithm, 1970s, Morf

Neural Networks

## Logical calculus for neural networks, McCulloch and Pitts, 1943

Perceptron, Rosenblatt, 1958
Back-propagation algorithm, Rumelhart, et al., 1986

37

Applications

## Zero-forcing equaliser, Lucky, 1965

MMSE equaliser, Gersho, 1969, Proakis&Miller, 1969
Godard Algorithm, Godard, 1974
Fractionally Spaced Equaliser (FSE), Brady, 1970
Decision Feedback Equaliser (DFE), Austin 1967, MMSE,
Monsen, 1971.

Speech Coding

## Maximum Likelihood speech prediction, Saito and Itakura, 1966

Linear Predictive Coding (LPC), Atal and Hanauer 1970-1
Adaptive Lattice Predictor, Nakhoul and Cossell, 1981

38

Applications

## Maximum entropy method, Burg, 1967

Method of multiple windows, Thomson, 1982

## Intermediate Frequency (IF) sidelobe canceller, Howells, 1950

Control law for adaptive array antenna, Applebaum, 1966
Application of LMS, Widrow et al., 1967
Minimum Variance Distortionless Response (MVDR)
beamformer, Capon, 1969

39

Performance Measures of