1 Tutorial EUSIPCO2012 PDF

Teaching Signal Processing with Geometry
Martin Vetterli and Vivek Goyal

Based upon textbook with Jelena Kovačević
Includes slides by Andrea Ridolfi and Amina Chebira
August 27, 2012

Outline
1 Introduction
Why a tutorial on teaching?
Overview
2 Basic Principles in Teaching SP with Geometry
Unified view
Hilbert space tools—Part I: Basics through projections
Basic results I: Best approximation
Hilbert space tools—Part II: Bases through discrete-time systems
3 Example Lecture: Sampling and Interpolation
Motivation
Classical View and Historical Notes
Operator View
Basis Expansion View
Extensions
Summary
4 Teaching Materials
Textbooks
Supplementary materials
5 Wrap up
Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 2 / 96
Introduction Why a tutorial on teaching?
Teaching about teaching?

Teaching should not remain static
Applications change (and we should want them to)
◮ Signal processing thinking should be applied broadly
◮ Global Fourier techniques relatively less important than in the past
Computing platforms change (and we should want them to)

◮ Classical DSP architectures relative less important than in the past
◮ Likely to use high-level programming languages
Students change (and we should want them to)

◮ Different base of knowledge
◮ Biology, economics, social sciences, . . .
Eternal challenge of the educator

◮ Knowledge grows, time in school does not
◮ Must be willing to cull details to convey big picture
◮ Should teach what is most reusable and generalizable
















Introduction Overview
Overview
Goals of the tutorial:
See that geometric notions unify (simplify!) signal processing
Learn/review basics of Hilbert space view
See Hilbert space view in action
Learn about textbooks Foundations of Signal Processing and
Fourier and Wavelet Signal Processing
Structure of the tutorial:

Developing unified view of signal processing
◮ A few key results (best approximation and the projection theorem)
Example lecture: Sampling made easy
Textbooks and related materials

Overview


Overview


Overview


Basic Principles in Teaching SP with Geometry Unified view
Unifying principles
Signal processing has various dichotomies
continuous time vs. discrete time
infinite intervals vs. finite intervals
periodic vs. aperiodic
deterministic vs. stochastic
Each can placed in a common framework featuring geometry
Example payoffs:
Unified understanding of best approximation (projection theorem)
Unified understanding of Fourier domains
Unified understanding of signal expansions (including sampling)

Unifying principles
Example payoffs:

Unifying principles
Example payoffs:

Unifying principles
Example payoffs:

Unifying framework: Hilbert spaces

Examples of Hilbert spaces:
finite-dimensional vectors (basic linear algebra)
sequences on {. . . , −1, 0, 1, . . .} (discrete-time signals)
sequences on {0, 1, . . . , N − 1} (N-periodic discrete-time signals)
functions on (−∞, ∞) (continuous-time signals)
functions on [0, T ] (T -periodic continuous-time signals)
scalar random variables
More abstraction. More mathematics. More difficult?

With framework in place, can go farther, faster
Leverage “real world” geometric intuition

Unifying framework: Hilbert spaces

Examples of Hilbert spaces:
finite-dimensional vectors (basic linear algebra)
sequences on {. . . , −1, 0, 1, . . .} (discrete-time signals)
sequences on {0, 1, . . . , N − 1} (N-periodic discrete-time signals)
functions on (−∞, ∞) (continuous-time signals)
functions on [0, T ] (T -periodic continuous-time signals)
More abstraction. More mathematics. More difficult?

With framework in place, can go farther, faster
Leverage “real world” geometric intuition

Mathematical rigor
Everything should be made as simple as possible, but no simpler.
– Common paraphrasing of Albert Einstein
Make everything as simple as possible without being wrong.
– Our variant for teaching
Correct intuitions are separate from functional analysis details

Teach the difference among
◮ rigorously true, with elementary justification
◮ rigorously true, justification not elementary (e.g., Poisson sum formula)
◮ convenient and related to rigorous statements (e.g., uses of Dirac delta)
. . . if whether an airplane would fly or not depended on whether some

function . . . was Lebesgue but not Riemann integrable, then I would not
fly in it.
– Richard W. Hamming

Mathematical rigor
Everything should be made as simple as possible, but no simpler.
– Common paraphrasing of Albert Einstein
Make everything as simple as possible without being wrong.
– Our variant for teaching
Correct intuitions are separate from functional analysis details

Teach the difference among
◮ rigorously true, with elementary justification
◮ rigorously true, justification not elementary (e.g., Poisson sum formula)
◮ convenient and related to rigorous statements (e.g., uses of Dirac delta)
. . . if whether an airplane would fly or not depended on whether some

function . . . was Lebesgue but not Riemann integrable, then I would not
fly in it.
– Richard W. Hamming

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part I: Basics through projections
Vector spaces
A vector space generalizes easily beyond the R2 Euclidean plane
Axioms
A vector space over a field of scalars C (or R) is a set of vectors V together
with operations
◮ vector addition: V × V → V
◮ scalar multiplication: C × V → V
that satisfy the following axioms:
1. x +y =y +x
2. (x + y ) + z = x + (y + z)
3. ∃ 0 ∈ V s.t. x + 0 = x for all x ∈ V
4. α(x + y ) = αx + αy
5. (α + β)x = αx + βx
6. (αβ)x = α(βx)
7. 0x = 0 and 1x = x

Vector spaces
Examples
CN : complex (column) vectors of length N
CZ : sequences – discrete-time signals

(write as infinite column vector)
CR : functions – continuous-time signals
polynomials of degree at most K
discrete-time stochastic processes

Vector spaces
Key notions
Subspace
◮ S ⊆ V is a subspace when it is closed under vector addition and scalar
multiplication:
⋆ For all x, y ∈ S, x + y ∈ S
⋆ For all x ∈ S and α ∈ C, αx ∈ S
Span
◮ S: set of vectors (could be infinite)
◮ span(S) = set of all finite linear combinations of vectors in S:
( N−1 )
X
S = αk ϕk | αk ∈ C, ϕk ∈ S and N ∈ N
k=0
◮ span(S) is always a subspace

Vector spaces
Key notions
Subspace
◮ S ⊆ V is a subspace when it is closed under vector addition and scalar
multiplication:
⋆ For all x, y ∈ S, x + y ∈ S
⋆ For all x ∈ S and α ∈ C, αx ∈ S
Span
◮ S: set of vectors (could be infinite)
◮ span(S) = set of all finite linear combinations of vectors in S:
( N−1 )
X
S = αk ϕk | αk ∈ C, ϕk ∈ S and N ∈ N
k=0
◮ span(S) is always a subspace

Vector spaces
Key notions
Linear independence
N−1
◮ S = {ϕk }k=0 is linearly independent when:
N−1
X
αk ϕk = 0 only when αk = 0 for all k
k=0
◮ If S is infinite, we need every finite subset to be linearly independent
Dimension
◮ dim(V ) = N if V contains a linearly independent set with N vectors and every
set with N + 1 or more vectors is linearly dependent
◮ V is infinite dimensional if no such finite N exists

Vector spaces
Key notions
Linear independence
N−1
◮ S = {ϕk }k=0 is linearly independent when:
N−1
X
αk ϕk = 0 only when αk = 0 for all k
k=0
◮ If S is infinite, we need every finite subset to be linearly independent
Dimension
◮ dim(V ) = N if V contains a linearly independent set with N vectors and every
set with N + 1 or more vectors is linearly dependent
◮ V is infinite dimensional if no such finite N exists

Inner products
Inner products generalize angles (especially right angles) and orientation
Definition (Inner product)

An inner product on vector space V is a function h·, ·i : V × V → C
satisfying
1 Distributivity: hx + y , zi = hx, zi + hy , zi
2 Linearity in the first argument: hαx, y i = αhx, y i
3 Hermitian symmetry: hx, y i∗ = hy , xi
4 Positive definiteness: hx, xi ≥ 0 and hx, xi = 0 if and only if x = 0
Note: hx, αy i = α∗ hx, y i

Inner products
Examples
N−1
X
On CN : hx, y i = xn yn∗ = y ∗ x
n=0
X
On CZ : hx, y i = xn yn∗ = y ∗ x
n∈Z
Z ∞
On CR : hx, y i = x(t)y ∗ (t) dt
−∞
On C-valued random variables: hx, yi = E[ xy∗ ]

Geometry in inner product spaces

Drawn in R2 and true in general:
hx, y i = x1 y1 + x2 y2
= kxk ky k cos α
= product of 2-norms times the cos
of the angle between the vectors
hx, e1 i = x1 = kxk cos αx

Geometry in inner product spaces

Drawn in R2 and true in general:
hx, y i = x1 y1 + x2 y2
= kxk ky k cos α
= product of 2-norms times the cos
of the angle between the vectors
hx, e1 i = x1 = kxk cos αx

Orthogonality
Let S = {ϕi }i ∈I be a set of vectors
Definition (Orthogonality)
x and y are orthogonal when hx, y i = 0 written x ⊥ y
S is orthogonal when for all x, y ∈ S, x 6= y we have x ⊥ y
S is orthonormal when it is orthogonal and for all x ∈ S, hx, xi = 1
x is orthogonal to S when x ⊥ s for all s ∈ S, written x ⊥ S
S0 and S1 are orthogonal when every s0 ∈ S0 is orthogonal to S1 , written

S0 ⊥ S1
Right angles (perpendicularity) extends beyond Euclidean geometry

Norm
Norms generalize length in ordinary Euclidean space
Definition (Norm)
A norm on V is a function k·k : V → R satisfying
1 Positive definiteness: kxk ≥ 0 and kxk = 0 if and only if x = 0
2 Positive scalability: kαxk = |α| kxk
3 Triangle inequality: kx + y k ≤ kxk + ky k with equality if and only if y = αx
Any inner product induces a norm

p
kxk = hx, xi
Not all norms are induced by an inner product

Norm
Norms generalize length in ordinary Euclidean space
Definition (Norm)
A norm on V is a function k·k : V → R satisfying
1 Positive definiteness: kxk ≥ 0 and kxk = 0 if and only if x = 0
2 Positive scalability: kαxk = |α| kxk
3 Triangle inequality: kx + y k ≤ kxk + ky k with equality if and only if y = αx
Any inner product induces a norm

p
kxk = hx, xi
Not all norms are induced by an inner product

Norms induced by inner products

p
Any inner product induces a norm: kxk = hx, xi
Examples
N−1
!1/2
p X
N 2
On C : kxk = hx, xi = |xn |
n=0
!1/2
p X
On CZ : kxk = hx, xi = |xn |2
n∈Z
Z 1/2
p ∞
On CR : kxk = hx, xi = |x(t)|2 dt
−∞
p
On C-valued random variables: kxk = hx, xi = E |x|2

Properties
Pythagorean theorem
◮ x ⊥y ⇒ kx + y k2 = kxk2 + ky k2

X 2 X
◮ {xk }k∈K orthogonal ⇒
xk
=
kxk k2
k∈K k∈K
x +y
y

Properties
Parallelogram Law
kx + y k2 + kx − y k2 = 2(kxk2 + ky k2 )
x
x +y
y y
x −y

Properties
Cauchy–Schwarz inequality
|hx, y i| ≤ kxk ky k

Norms not necessarily induced by inner products
Examples
P 1/p
N−1
On CN : kxkp = n=0 |xn |
p
, p ∈ [1, ∞)
P p
1/p
On CZ : kxkp = n∈Z |xn | , p ∈ [1, ∞)
kxk∞ = sup|xn |
n∈Z
R 1/p
∞
On CR : kxkp = −∞
|x(t)|p dt , p ∈ [1, ∞)
kxk∞ = ess sup|x(t)|

t∈R
Only induced by inner products for p = 2

p
Geometry of ℓ : Unit balls
x1
kxk∞ = 1
kxk4 = 1
kxk2 = 1
kxk1 = 1
kxk1/2 = 1
x0
Valid norm (and convex unit ball) for p ≥ 1; ordinary geometry for p = 2

Normed vector spaces
A normed vector space is a set satisfying axioms of a vector space where the
norm is finite
ℓ2 (Z): square-summable sequences (“finite-energy discrete-time signals”)

!1/2
X
2
kxk = |xn | < ∞
n∈Z
L2 (R): square-integrable functions (“finite-energy continuous-time signals”)

Z ∞ 1/2
kxk = |x(t)|2 dt < ∞
−∞
x and y are the same when kx − y k = 0

No harm in considering only functions with finitely-many discontinuities

A normed vector space is a set satisfying axioms of a vector space where the
norm is finite
ℓ2 (Z): square-summable sequences (“finite-energy discrete-time signals”)

!1/2
X
2
kxk = |xn | < ∞
n∈Z
L2 (R): square-integrable functions (“finite-energy continuous-time signals”)

Z ∞ 1/2
kxk = |x(t)|2 dt < ∞
−∞
x and y are the same when kx − y k = 0

No harm in considering only functions with finitely-many discontinuities

Hilbert spaces: Convergence
Definition
A sequence of vectors x0 , x1 , . . . in a normed vector space V is said to converge
to v ∈ V when limk→∞ kv − xk k = 0, or for any ε > 0, there exists Kε such that
kv − xk k < ε for all k > Kε .
Choice of the norm in V is key
Example
For k ∈ Z+ , let
1, for t ∈ [0, 1/k ];
xk (t) =
0, otherwise.
v (t) = 0 for all t. Then for p ∈ [1, ∞),
Z ∞ 1/p 1/p
1 k→∞
kv − xk kp = |v (t) − xk (t)|p dt = −→ 0,
−∞ k
For p = ∞: kv − xk k∞ = 1 for all k

Definition
Example
For k ∈ Z+ , let
1, for t ∈ [0, 1/k ];
xk (t) =
0, otherwise.
Z ∞ 1/p 1/p
1 k→∞
kv − xk kp = |v (t) − xk (t)|p dt = −→ 0,
−∞ k

Definition
Example
For k ∈ Z+ , let
1, for t ∈ [0, 1/k ];
xk (t) =
0, otherwise.
Z ∞ 1/p 1/p
1 k→∞
kv − xk kp = |v (t) − xk (t)|p dt = −→ 0,
−∞ k

Hilbert spaces: Completeness
Definitions
A sequence {xn } is a Cauchy sequence in a normed space when for any
ε > 0, there exists kε such that kxk − xm k < ε for all k, m > kε
A normed vector space V is complete if every Cauchy sequence converges in V
A complete normed vector space is called a Banach space
A complete inner product space is called a Hilbert space

Hilbert spaces
Examples
Q is not a complete space
X∞
1 π2
◮ → ∈ R, ∈
/Q
n=1
n2 6
X∞
1
◮ → e ∈ R, ∈
/Q
n=0
n!
e
5 8 65
1 2 2 3 24
R is a complete space

Hilbert spaces
Examples
Q is not a complete space
X∞
1 π2
◮ → ∈ R, ∈
/Q
n=1
n2 6
X∞
1
◮ → e ∈ R, ∈
/Q
n=0
n!
e
5 8 65
1 2 2 3 24
R is a complete space

Hilbert spaces
Examples
All finite dimensional spaces are complete
ℓp (Z) and Lp (R) are complete

◮ ℓ2 (Z) and L2 (R) are Hilbert spaces
C q ([a, b]), functions on [a, b] with q continuous derivatives, are not complete
except for q = 0 under L∞ norm

Hilbert spaces
Examples


Hilbert spaces
Examples


Summary on spaces
Vector spaces
Inner product spaces Banach spaces
Hilbert
spaces
• ℓ1 (Z)
•Q N
•C N • ℓ∞ (Z)
• C ([a, b]) • ℓ2 (Z) • L1 (R)
• L2 (R) • L∞ (R)
• (C 1 ([a, b]), k · k∞ ) • (V , d)
• ℓ0 (Z)

Linear operators
Linear operators generalize matrices
Definitions
A : H0 → H1 is a linear operator when for all x, y ∈ H0 , α ∈ C:
1 Additivity: A(x + y ) = Ax + Ay
2 Scalability: A(αx) = α(Ax)
Null space (subspace of H0 ): N (A) = {x ∈ H0 , Ax = 0}
Range space (subspace of H1 ): R(A) = {Ax ∈ H1 , x ∈ H0 }
Operator norm: kAk = sup kAxk
kxk=1
A is bounded when: kAk < ∞

Inverse: Bounded B : H1 → H0 inverse of bounded A if and only if:
BAx = x, for every x ∈ H0
ABy = y , for every y ∈ H1

Linear operators
Definitions
kxk=1


Linear operators
Definitions
kxk=1


Linear operators
Definitions
kxk=1


Linear operators: Illustration
R(A) is the plane 5y1 + 2y2 + 8y3 = 0

Adjoint operators
Adjoint generalizes Hermitian transposition of matrices
Definition (Adjoint and self-adjoint operators)

A∗ : H1 → H0 is the adjoint of A : H0 → H1 when
hAx, y iH1 = hx, A∗ y iH0 for every x ∈ H0 , y ∈ H1
If A = A∗ , A is self-adjoint or Hermitian
Note that N (A∗ ) = R(A)⊥

Adjoint operator: Illustration
N (A∗ ) is the line 51 y1 = 1

2 y2 = 1
8 y3

Adjoint operators
Theorem (Adjoint properties)

Let A : H0 → H1 be a bounded linear operator
1 A∗ exists and is unique
2 (A∗ )∗ = A
3 AA∗ and A∗ A are self-adjoint
4 kA∗ k = kAk
5 If A is invertible, (A−1 )∗ = (A∗ )−1
6 If B : H0 → H1 is bounded, (A + B)∗ = A∗ + B ∗
7 If B : H1 → H2 is bounded, (BA)∗ = A∗ B ∗

Adjoint operators: Local averaging

Z k+1/2
A : L2 (R) → ℓ2 (Z) (Ax)k = x(t)dt
k−1/2
Z !
X X n+1/2 XZ n+1/2
hAx, y iℓ2 = (Ax)n yn∗ = x(t) dt yn∗ = x(t)yn∗ dt
n∈Z n∈Z n−1/2 n∈Z n−1/2
XZ n+1/2 Z ∞
= x ∗ (t) dt =
x(t)b x ∗ (t) dt = hx, b
x(t)b x iL2 = hx, A∗ y iL2
n∈Z n−1/2 −∞
x(t) yn = (Ax)n
10 10
5 5
t n
5 10 15 5 10 15
−5 −5
−10 −10
x (t) = (A∗ Ax)(t)

b x(t), yn , b
x (t)
10 10
5 5
t t
5 10 15 5 10 15
−5 −5
−10 −10

Unitary operators
Definition (Unitary operators)

A bounded linear operator A : H0 → H1 is unitary when:
1 A is invertible
2 A preserves inner products: hAx, Ay iH1 = hx, y iH0 for every x, y ∈ H0
If A is unitary, then kAxk2 = kxk2
A is unitary if and only if A−1 = A∗

Unitary operators
Definition (Unitary operators)

A bounded linear operator A : H0 → H1 is unitary when:
1 A is invertible
2 A preserves inner products: hAx, Ay iH1 = hx, y iH0 for every x, y ∈ H0
If A is unitary, then kAxk2 = kxk2
A is unitary if and only if A−1 = A∗

Projection operators
Definition (Projection, orthogonal projection, oblique projection)

P is idempotent when P 2 = P
A projection operator is a bounded linear operator that is idempotent
An orthogonal projection operator is a self-adjoint projection operator
An oblique projection operator is not self adjoint

Projection operators
Definition (Projection, orthogonal projection, oblique projection)

P is idempotent when P 2 = P
A projection operator is a bounded linear operator that is idempotent
An orthogonal projection operator is a self-adjoint projection operator
An oblique projection operator is not self adjoint
Theorem
If A : H0 → H1 , B : H1 → H0 bounded and A is a left inverse of B, then BA is
a projection operator. If B = A∗ then, BA = A∗ A is an orthogonal projection

Basic Principles in Teaching SP with Geometry Basic results I: Best approximation
Best approximation: Euclidean geometry
x is a point in Euclidean space
S is a line in Euclidean space

x•


x•
x ∈ S that is closest to x
Nearest point problem: Find b


x•
•
•′ b
x
x


x•
•
•′ b
x
x
Solution uniquely determined by x − b
x⊥X
◮ Circle must touch S in one point, radius ⊥ tangent


x
•
•′ b
x
x
Solution uniquely determined by x − b
x⊥X
◮ Circle must touch S in one point, radius ⊥ tangent

Best approximation: Hilbert space geometry
S closed subspace of a Hilbert space
Best approximation problem:
Find b
x = argminkx − sk
b
s∈S
•
•′ b
x
x

Best approximation by orthogonal projection
Theorem (Projection theorem)

Let S be a closed subspace of Hilbert space H and let x ∈ H.
Existence: There exists xb ∈ S such that kx − b
x k ≤ kx − sk for all s ∈ S
Orthogonality: x − b
x ⊥ S is necessary and sufficient to determine b
x
Uniqueness: b
x is unique
Linearity: b
x = Px where P is a linear operator
Idempotency: P(Px) = Px for all x ∈ H
Self-adjointness: P = P ∗
All “nearest vector in a subspace” problems in Hilbert spaces are the same


x
Uniqueness: b
x is unique
Linearity: b


x
Uniqueness: b
x is unique
Linearity: b

Example 1: Least-square polynomial approximation
Consider: x(t) = cos( 32 πt) ∈ L2 ([0, 1])
Find the degree-1 polynomial closest to x (in L2 norm)
Solution: Use orthogonality



Z 1 2 1
0 = hx(t) − b
x (t), 1it = cos( 32 πt) − (a0 + a1 t) 1 dt = − − a0 − a1
0 3π 2
Z 1 4 + 6π 1 1
0 = hx(t) − b
x (t), tit = cos( 32 πt) − (a0 + a1 t) t dt = − − a0 − a1
0 9π 2 2 3
1 1
0.8 0.8
0.6 x(t) 0.6
0.4 0.4
0.2 0.2
0
−0.2
x (t)
b 0
−0.2
−0.4 −0.4
−0.6 −0.6
−0.8 −0.8
−1 −1
0 0.2 0.4
t 0.6 0.8 1 0 0.2 0.4
t 0.6 0.8 1
Approx. with degree 1 polynomial Approx. with higher degree polynomials

Example 2: MMSE estimate
Consider: Real-valued random variable x

Find the constant c that minimizes E (x − c)2
Note:
◮ Expected square is a Hilbert space norm
◮ Constants are a closed subspace in vector space of random variables

◮ c determined uniquely by E[ (x − c) αc ] = 0 for all α ∈ R
◮ c = E[ x ]
Alternative:
◮ Expand into quadratic function of c and minimize with calculus
◮ Not too difficult, but lacks insight


Note:

◮ c = E[ x ]
Alternative:


Note:

◮ c = E[ x ]
Alternative:


Note:

◮ c = E[ x ]
Alternative:


Note:

◮ c = E[ x ]
Alternative:

Example 3: Wiener filter

Consider: Jointly wide-sense stationary
y b
x
discrete-time stochastic processes x and y h
Find the linear −
shift-invariant
filter h that x
+ e
minimizes E |xn − b xn |2 where b x =h∗y
Note:
◮ Expected squared absolute value is a Hilbert space norm
◮ LSI filtering puts b
xn in a closed subspace
Solution: Use orthogonality (extended for processes)
◮ h determined uniquely by relation between cross- and autocorrelations:
cx,bx,k = abx,k , k ∈Z
Cx,y (e jω )
◮ DTFT-domain version: H(e jω ) = , ω∈R
Ay (e jω )
Alternative:
◮ Expand into quadratic function of h and minimize with calculus
◮ Messy (!)—especially in complex case—and lacks insight


y b
x
Find the linear −
shift-invariant
filter h that x
+ e
Note:
Cx,y (e jω )
Ay (e jω )
Alternative:


y b
x
Find the linear −
shift-invariant
filter h that x
+ e
Note:
Cx,y (e jω )
Ay (e jω )
Alternative:


y b
x
Find the linear −
shift-invariant
filter h that x
+ e
Note:
Cx,y (e jω )
Ay (e jω )
Alternative:

Example 4: Best piecewise-constant approximation
Local averaging
Z k+ 21
2 2
A : L (R) → ℓ (Z) (Ax)k = x(t)dt
k− 12
has adjoint A∗ : ℓ2 (Z) → L2 (R) that produces staircase function
AA∗ is identity, so A∗ A is orthogonal projection

x(t) yn = (Ax)n
10 10
5 5
5 10 15t 5 10 15n
−5 −5
−10 −10 x(t)
10
5
x (t) = (A∗ Ax)(t)
b x(t), yn , b
x (t)
t
10 10 5 10 15
5 5 −5
5 10 15t 5 10 15t −10
−5 −5
−10 −10

Example 5: Approximations of “All is vanity” image—Haar
1 2
(−π, π) (π, π)
2
(−π, −π) (π, −π)

Example 5: Approximations of “All is vanity” image—Haar
(−π, π) (π, π)
3
(−π, −π) (π, −π)

Example 5: Approximations of “All is vanity” image—sinc
1 2
(−π, π) (π, π)
2
(−π, −π) (π, −π)

Basic Principles in Teaching SP with Geometry Hilbert space tools—Part II: Bases through discrete-time systems
Bases
Definition (Basis)
Φ = {ϕk }k∈K ⊂ V is a basis when
1 Φ is linearly independent and
2 Φ is complete in V : V = span(Φ)
X
Expansion formula: for any x ∈ V , x= αk ϕk
k∈K
{αk }k∈K : is unique
αk : expansion coefficients
Example
The standard
basis for RN T
ek = 0 0 · · · 0 1 0 0 ··· 0 , k = 0, . . . , N − 1
N−1
X
any x ∈ RN , x= xk ek
k=0

Bases
Definition (Basis)
X
k∈K
Example
The standard
basis for RN T
ek = 0 0 · · · 0 1 0 0 ··· 0 , k = 0, . . . , N − 1
N−1
X
k=0

Bases
Definition (Basis)
X
k∈K
Example
The standard
basis for RN T
ek = 0 0 · · · 0 1 0 0 ··· 0 , k = 0, . . . , N − 1
N−1
X
k=0

Bases
Examples
α1 α1 ϕ1
x x
ϕ
e1 ϕ1
ϕ1 ϕ1
ϕ0
ϕ0 ϕ0
α0 α0 ϕ0 ϕ
e0
Orthonormal basis Biorthogonal basis . . . with dual basis

Operators associated with bases
Definition (Basis synthesis operator)

Synthesis operator
X
◮ Φ : ℓ2 (K) → H Φα = αk ϕk
k∈K
◮ Adjoint: Let α ∈ ℓ2 (Z) and y ∈ H

X X
hΦα, y i = αk ϕk , y = αk hy , ϕk i∗
k∈K k∈K
Definition (Basis analysis operator)

Analysis operator
◮ Φ∗ : H → ℓ2 (K) (Φ∗ x)k = hx, ϕk i, k ∈ K
Note that the analysis operator is the adjoint of the synthesis operator


Synthesis operator
X
◮ Φ : ℓ2 (K) → H Φα = αk ϕk
k∈K

X X
k∈K k∈K

Analysis operator
◮ Φ∗ : H → ℓ2 (K) (Φ∗ x)k = hx, ϕk i, k ∈ K


Synthesis operator
X
◮ Φ : ℓ2 (K) → H Φα = αk ϕk
k∈K

X X
k∈K k∈K

Analysis operator
◮ Φ∗ : H → ℓ2 (K) (Φ∗ x)k = hx, ϕk i, k ∈ K

Orthonormal bases
Definition (Orthonormal basis)

Φ = {ϕk }k∈K ⊂ H is an orthonormal basis for H when
1 Φ is a basis for H and
2 Φ is an orthonormal set
hϕi , ϕk i = δi −k for all i, k ∈ K
If Φ is an orthogonal set, then it is linearly independent
If span(Φ) = H and Φ is an orthogonal set,

then Φ is an orthogonal basis for H
If we also have kϕk k = 1, then Φ is an orthonormal basis

Orthonormal bases



Orthonormal bases



Orthonormal basis expansions
Definition (Orthonormal basis expansions)

Φ = {ϕk }k∈K orthonormal basis for H, then for any x ∈ H:
αk = hx, ϕk i for k ∈ K, or α = Φ∗ x, and α is unique
X
Synthesis: x = hx, ϕk iϕk = Φα = ΦΦ∗ x
k∈K
Example


X
k∈K
Example


X
k∈K
Example
ϕ2 = e2 x
hx, ϕ1 i
ϕ1 = e1
ϕ0 = e0
hx, ϕ0 i
x01

Orthonormal basis: Parseval equality
Theorem (Parseval’s equalities)

Φ = {ϕk }k∈K orthonormal basis for H
X
kxk2 = |hx, ϕk i|2 = kΦ∗ xk2 = kαk2
k∈K
In general:
hx, y i = hΦ∗ x, Φ∗ y i = hα, βi
where αk = hx, ϕk i, βk = hy , ϕk i


X
k∈K
In general:
y
β
Φ∗
x
θ Φ
θ α
H ℓ2 (K)

X
k∈K
In general:
y
β
Φ∗
x
θ Φ
θ α
H ℓ2 (K) isometry

Orthogonal projection and decomposition
Theorem
Φ = {ϕk }k∈I ⊂ H, I ⊂ K
X
PI x = hx, ϕk iϕk = ΦI Φ∗I x
k∈I
is the orthogonal projection of x onto SI = span({ϕk }k∈I )
Φ induces an orthogonal decomposition

M
H = S{k} where S{k} = span(ϕk )
k∈K

Orthogonal projection and decomposition
Theorem
Φ = {ϕk }k∈I ⊂ H, I ⊂ K
X
PI x = hx, ϕk iϕk = ΦI Φ∗I x
k∈I
is the orthogonal projection of x onto SI = span({ϕk }k∈I )
Φ induces an orthogonal decomposition

M
H = S{k} where S{k} = span(ϕk )
k∈K

Biorthogonal pairs of bases
Definition
e = {ϕ
Φ = {ϕk }k∈K ⊂ H and Φ ek }k∈K ⊂ H is a biorthogonal pair of bases
when
1 e are both bases for H
Φ and Φ
2 e are biorthogonal: hϕi , ϕ
Φ and Φ ek i = δi −k for all i, k ∈ K
e are interchangeable
Roles of Φ and Φ
Example
       
1 0 1 1 0 1
ϕ0 = 1, ϕ1 = 1, ϕ2 = 1, Φ = 1 1 1
0 1 1 0 1 1
       
0 −1 1 0 1 −1
−1
e0 =  1 ,
ϕ e1 =  1 ,
ϕ e2 = −1,
ϕ Φ = −1 1 0
−1 0 1 1 −1 1
Definition
e = {ϕ
when
Φ and Φ
Roles of Φ and Φ
Example
       
1 0 1 1 0 1
ϕ0 = 1, ϕ1 = 1, ϕ2 = 1, Φ = 1 1 1
0 1 1 0 1 1
       
0 −1 1 0 1 −1
−1
e0 =  1 ,
ϕ e1 =  1 ,
ϕ e2 = −1,
ϕ Φ = −1 1 0
−1 0 1 1 −1 1
Definition
e = {ϕ
when
Φ and Φ
Roles of Φ and Φ
Example
       
1 0 1 1 0 1
ϕ0 = 1, ϕ1 = 1, ϕ2 = 1, Φ = 1 1 1
0 1 1 0 1 1
       
0 −1 1 0 1 −1
−1
e0 =  1 ,
ϕ e1 =  1 ,
ϕ e2 = −1,
ϕ Φ = −1 1 0
−1 0 1 1 −1 1
Biorthogonal basis expansion
Theorem
e = {ϕ
Φ = {ϕk }k∈K , Φ ek }k∈K biorthogonal pair of bases for H
Any x ∈ H has expansion coefficients
αk = hx, ϕ e ∗x
ek i for k ∈ K, or α = Φ
X
Synthesis: x = hx, ϕ
ek iϕk = Φα = ΦΦ e ∗x
k∈K
X
Also x = hx, ϕk iϕ
ek
k∈K
e1 ϕ1
ϕ
ϕ0
ϕ
e0
Theorem
e = {ϕ
αk = hx, ϕ e ∗x
X
k∈K
X
ek
k∈K
e1 ϕ1
ϕ
ϕ0
ϕ
e0
Theorem
e = {ϕ
αk = hx, ϕ e ∗x
X
k∈K
X
ek
k∈K
e1 ϕ1
ϕ
ϕ0
ϕ
e0
Frames: Overcomplete representations

X
x = hx, ϕk iϕk = ΦΦ∗ x without {ϕk }k∈J being linearly independent
k∈J
Example
R2

1 0 −1
Given Φ =
ϕ2 = e2 x 0 1 −1
hx, ϕ1 i
ϕ0 = e0
ϕ1 = e1 {ϕ0 , ϕ1 , ϕ2 } is a frame for R2
hx, ϕ0 i
x01

Change of basis: Orthonormal basis
How are the expansion coefficients in two orthonormal bases related?
Assume x = Φα = Ψβ
Then β = Ψ∗ x = Ψ∗ Φα
Change of basis from Φ to Ψ that maps α to β is the operator

CΦ,Ψ : ℓ2 (K) → ℓ2 (K) s.t. CΦ,Ψ = Ψ∗ Φ
As a matrix
 
.. .. ..
 . . . 
· · · hϕ−1 , ψ−1 i hϕ0 , ψ−1 i hϕ1 , ψ−1 i · · ·
 
 
CΦ,Ψ = · · · hϕ−1 , ψ0 i hϕ0 , ψ0 i hϕ1 , ψ0 i · · ·
 
· · · hϕ−1 , ψ1 i hϕ0 , ψ1 i hϕ1 , ψ1 i · · ·
 
.. .. ..
. . .


CΦ,Ψ : ℓ2 (K) → ℓ2 (K) s.t. CΦ,Ψ = Ψ∗ Φ
As a matrix
 
.. .. ..
 . . . 
· · · hϕ−1 , ψ−1 i hϕ0 , ψ−1 i hϕ1 , ψ−1 i · · ·
 
 
 
· · · hϕ−1 , ψ1 i hϕ0 , ψ1 i hϕ1 , ψ1 i · · ·
 
.. .. ..
. . .


CΦ,Ψ : ℓ2 (K) → ℓ2 (K) s.t. CΦ,Ψ = Ψ∗ Φ
As a matrix
 
.. .. ..
 . . . 
· · · hϕ−1 , ψ−1 i hϕ0 , ψ−1 i hϕ1 , ψ−1 i · · ·
 
 
 
· · · hϕ−1 , ψ1 i hϕ0 , ψ1 i hϕ1 , ψ1 i · · ·
 
.. .. ..
. . .


CΦ,Ψ : ℓ2 (K) → ℓ2 (K) s.t. CΦ,Ψ = Ψ∗ Φ
As a matrix
 
.. .. ..
 . . . 
· · · hϕ−1 , ψ−1 i hϕ0 , ψ−1 i hϕ1 , ψ−1 i · · ·
 
 
 
· · · hϕ−1 , ψ1 i hϕ0 , ψ1 i hϕ1 , ψ1 i · · ·
 
.. .. ..
. . .

Matrix representation of operator: Orthonormal basis
Let y = Ax with A : H → H
How are expansion coefficients of x and y related?
◮ {ϕk }k∈K orthonormal basis of H
◮ x = Φα, y = Φβ
Matrix representation allows computation of A directly on coefficient
sequences
Γ : ℓ2 (K) → ℓ2 (K) s.t. β = Γα
As a matrix:  
.. .. ..
 . . . 
· · · hAϕ−1 , ϕ−1 i hAϕ0 , ϕ−1 i hAϕ1 , ϕ−1 i · · ·
 
 
Γ = · · · hAϕ−1 , ϕ0 i hAϕ0 , ϕ0 i hAϕ1 , ϕ0 i · · ·
 
· · · hAϕ−1 , ϕ1 i hAϕ0 , ϕ1 i hAϕ1 , ϕ1 i · · ·
 
.. .. ..
. . .

◮ x = Φα, y = Φβ
sequences
Γ : ℓ2 (K) → ℓ2 (K) s.t. β = Γα
.. .. ..
 . . . 
 
 
 
 
.. .. ..
. . .

◮ x = Φα, y = Φβ
sequences
Γ : ℓ2 (K) → ℓ2 (K) s.t. β = Γα
.. .. ..
 . . . 
 
 
 
 
.. .. ..
. . .

H H
A
x • • y
Φ∗ Φ Φ∗ Φ
α • • β
Γ
2
ℓ (K) ℓ2 (K)
When orthonormal bases are used, matrix representation of A∗ is Γ∗

H H
A
x • • y
Φ∗ Φ Φ∗ Φ
α • • β
Γ
2
ℓ (K) ℓ2 (K)
When orthonormal bases are used, matrix representation of A∗ is Γ∗

Example: Averaging Operator I
Example
Z 2(ℓ+1)
1
Let A : H0 → H1 , y (t) = Ax(t) = 2 x(τ ) dτ
2ℓ
for 2ℓ ≤ t < 2(ℓ + 1), ℓ∈Z
H0 : piecewise-constant, finite-energy functions with breakpoints at integers
H1 : piecewise-constant, finite-energy functions with breakpoints at even
integers

1, for t ∈ I ;
Given χI (t) =
0, otherwise
Let Φ = {ϕk (t)}k∈Z = {χ[k,k+1) (t)}k∈Z
Ψ = {ψi (t)}i ∈Z = { √12 χ[2i ,2(i +1))(t)}i ∈Z
be orthonormal bases for H0 , H1 respectively

Example: Averaging Operator II
Example (Cont.)
Z 2
1 1 √1 √1
Aϕ0 (t) = 2 χ[0,2) (t) ⇒ hAϕ0 , ψ0 i = 2 2 dτ = 2
0
 
.. .. .. .. .. .. ..
. . . . . . .
 
· · · 1 1 0 0 0 0 · · ·
 
Then Γ = √12 
· · · 0 0 1 1 0 0 · · ·
· · · 0 0 0 0 1 1 · · ·
 
.. .. .. .. .. .. ..
. . . . . . .
Simplicity of matrix representation depends on the basis!

Discrete-time systems
A linear system A : ℓ2 (Z) → ℓ2 (Z) has a matrix representation H with

respect to the standard basis
For a linear shift-invariant (LSI) system, the matrix H is Toeplitz:

  .. .. .. .. ..  
.. .. .. ..
. . . . . . . . .
    
y−2  . . . h0 h−1 h−2 h−3 h−4 . . .  x−2 
    
y−1  . . . h1 h0 h−1 h−2 h−3 . . .  x−1 
    
y = 
 y 0  = . . .
  h2 h1 h0 h−1 h−2 . . .  
  x0  = Hx
 y1  . . . h3 h2 h1 h0 h−1 
. . .  1 
 x
   
 y2  . . . . . .  
   h4 h3 h2 h1 h0   x2 
.. .. .. .. .. .. .. .. .
..
. . . . . . . .
| {z }
H

Discrete-time systems
Matrix representation of A∗ is H ∗ [Note: using orthonormal basis]

 
.. .. .. .. .. .
 . . . . . .. 
. 
 .. h0∗ h1∗ h2∗ h3∗ h4∗ 
 
. .. 
 .. ∗
h−1 h0∗
h1∗ ∗
h2 h3 ∗ .
 
. . 
H = 
∗
 .
. ∗
h−2 ∗
h−1 h0∗ h1∗ h2∗ . .

 
. . . . 
 . ∗
h−3 ∗ ∗
h−2 h−1 h0 h1 ∗ ∗ .
 
 .. 
 ∗
h−4 ∗
h−3 ∗
h−2 ∗
h−1 h0∗ .
 
. .. .. .. .. .
.. . . . . ..
∗
Adjoint of filtering by hn is filtering by h−n

Discrete-time systems: Periodic sequences
For an N-periodic setting, the matrix H is circulant:

    
y0 h0 hN−1 hN−2 . . . h1 x0
 y1   h1 h0 hN−1 . . . h2   x1 
    
 y2   h2 h h0 ... 
h3   x2 
y =   =  1   = Hx
 ..   .. .. .. .. ..   .. 
 .   . . . . .  . 
yN−1 hN−1 hN−2 hN−3 . . . h0 xN−1
| {z }
H
Equally well in both cases (and continuous time as well):

◮ Eigenvectors (-sequences, -signals) lead to diagonal representation of H
◮ Fourier transform follows logically from the class of operators
◮ Convolution theorem follows logically from the definition of the Fourier
transform

Example Lecture: Sampling and Interpolation Motivation
Sampling and Interpolation

Sampling and interpolation bridge the analog and digital worlds
Sampling: discrete-time sequence from a continuous-time function
Interpolation: continuous-time function from a discrete-time sequence
Doing all computation in discrete time is the essence of digital signal processing:
yn wn
x(t) Sampling DT processing Interpolation v (t)
Interpolation followed by sampling occurs in digital communication:
y (t) v (t)
xn Interpolation CT channel Sampling b
xn

Why Study Sampling and Interpolation Further?
Real-world sampling not pure mathematical idealization

◮ Don’t/can’t sample at one point
◮ Causal, non-ideal filters
Many practical architectures different from classical structure

◮ Multichannel, time-interleaved
Most information acquisition is intimately related to sampling

◮ Digital photography
◮ Computational imaging (magnetic resonance, space-from-time, ultrasound,
computed tomography, synthetic aperture radar, . . . )
◮ Reflection seismology, acoustic tomography, . . .
Goals from this lecture:

1 Understand classical sampling as a special case of a Hilbert space theory
2 Gain a generalizable understanding of sampling
















Example Lecture: Sampling and Interpolation Classical View and Historical Notes
Bandwidth
Definition (Bandwidth of sequence)

A sequence x is bandlimited when there exists ω0 ∈ [0, 2π) such that the
discrete-time Fourier transform X satisfies
X (e jω ) = 0 for all ω with |ω| ∈ (ω0 /2, π].
The smallest such ω0 is called the bandwidth of x.
0.5 æ æ
0.5 0.5
æ
ææ æææ æææææ
æ æ æ ææ æ ææ æ æ
æ ææ æ æ æ
æ æ æ æ æ æ æ æ æ æ
æ æ æ æ æ
æ æ æ æ æææææ æ æ æ æææ
æ
æ
æ æ æ æ ææææ
æ æ æ æ æ æ æ ææ
æ æ æ æ æ æ æ æ ææ ææ
æ æ ææ æ æ æ æ æ æ æ
ææ ææ æææææ
æ æ ææ
æ æ
æ æ ææ æ
æ ææ
ææ æ
æ
-0.5 -0.5 -0.5
ω0 = π ω0 = π/2 ω0 = π/4

Bandwidth
Definition (Bandwidth of function)

A function x is bandlimited when there exists ω0 ∈ [0, ∞) such that the Fourier
transform X satisfies
X (ω) = 0 for all ω with |ω| > ω0 /2.
Definition (Bandlimited sets)

The set of sequences in ℓ2 (Z) with bandwidth at most ω0 and the set of functions
in L2 (R) with bandwidth at most ω0 are denoted BL[−ω0 /2, ω0 /2]
If ω0 < ω1 then BL[−ω0 /2, ω0 /2] ⊂ BL[−ω1 /2, ω1 /2]

A bandlimited set is always a subspace
◮ Subspace is closed in Hilbert space ℓ2 (Z) or L2 (R)

Bandwidth




Bandwidth




Classical Sampling

6 0;
(sin t)/t, for t =
Recall: sinc(t) =
1, for t = 0
Theorem (Sampling theorem)

Let x be a function and let T > 0. Define
X
b
x (t) = x(nT ) sinc( Tπ (t − nT )).
n∈Z
If x ∈ BL[−ω0 /2, ω0 /2] with ω0 ≤ 2π/T , then b

x = x.
Exact recovery for (sufficiently) bandlimited signals
Nyquist period for bandwidth ω0 : T = 2π/ω0
Nyquist rate for bandwidth ω0 : T −1 = ω0 /2π
Easier in cycles/sec rather than radians/sec:

Need two samples per cycle of fastest component
Classical Sampling

6 0;
(sin t)/t, for t =
Recall: sinc(t) =
1, for t = 0
Theorem (Sampling theorem)

Let x be a function and let T > 0. Define
X
b
x (t) = x(nT ) sinc( Tπ (t − nT )).
n∈Z
If x ∈ BL[−ω0 /2, ω0 /2] with ω0 ≤ 2π/T , then b

x = x.
Exact recovery for (sufficiently) bandlimited signals
Nyquist period for bandwidth ω0 : T = 2π/ω0
Nyquist rate for bandwidth ω0 : T −1 = ω0 /2π
Easier in cycles/sec rather than radians/sec:

Need two samples per cycle of fastest component
History
Many names:
◮ Shannon sampling theorem
◮ Nyquist–Shannon sampling theorem
◮ Nyquist–Shannon–Kotel’nikov sampling theorem
◮ Whittaker–Shannon–Kotel’nikov sampling theorem
◮ Whittaker–Nyquist–Shannon–Kotel’nikov sampling theorem
Well-known people associated with sampling (but less often so):
◮ Cauchy (1841) – apparently not true
◮ Borel (1897)
◮ de la Vallée Poussin (1908)
◮ E. T. Whittaker (1915)
◮ J. M. Whittaker (1927)
◮ Gabor (1946)
Less-known people, almost lost to history:
◮ Ogura (1920)
◮ Küpfmüller (Küpfmüller filter)
◮ Raabe [PhD 1939] (assistant to Küpfmüller)
◮ Someya (1949)
◮ Weston (1949)
Best approach: use no names?
◮ Cardinal Theorem of interpolation theory
History
Many names:
◮ Borel (1897)
◮ Gabor (1946)
◮ Ogura (1920)
◮ Someya (1949)
◮ Weston (1949)
History
Many names:
◮ Borel (1897)
◮ Gabor (1946)
◮ Ogura (1920)
◮ Someya (1949)
◮ Weston (1949)
History
Many names:
◮ Borel (1897)
◮ Gabor (1946)
◮ Ogura (1920)
◮ Someya (1949)
◮ Weston (1949)
Sampling theorem: Conventional justification

P
Let b
x (t) = n∈Z x(nT ) g (t − nT ) [will deduce that g should be sinc]


P
Let b
R∞
Since x(nT ) g (t − nT ) = −∞ x(τ ) g (t − τ ) δ(τ − nT ) dτ ,
Z ∞ X
b
x (t) = g (t − τ ) x(τ ) δ(τ − nT ) dτ
−∞ n∈Z


P
Let b
R∞
Z ∞ X
b
x (t) = g (t − τ ) x(τ ) δ(τ − nT ) dτ
−∞ n∈Z
Recall: Fourier transform of Dirac comb

X
FT 2π X 2π
δ(t − nT ) ←→ δ ω− k
T T
n∈Z k∈Z


P
Let b
R∞
Z ∞ X
b
x (t) = g (t − τ ) x(τ ) δ(τ − nT ) dτ
−∞ n∈Z
| {z }
h(τ )
Recall: Fourier transform of Dirac comb

X
FT 2π X 2π
δ(t − nT ) ←→ δ ω− k
T T
n∈Z k∈Z
Take Fourier transforms, using convolution theorem for right side:

X
b (ω) = G (ω) 1
X X ω−
2π
k
T T
k∈Z
| {z }
H(ω)

X
b (ω) = 1 G (ω)
X X ω−
2π
k
T T
k∈Z
b has “spectral replication”

Reconstruction X
b (ω) = X (ω) for all ω?
How can we have X
◮ x ∈ BL[−π/T , π/T ] implies replicas do not overlap

T , for |ω| < π/T ;
◮ G (ω) = selects “desired” replica with correct gain
0, for t = 0
Shows recovery and deduces correctness of sinc interpolator

Dissatisfaction
Mathematical rigor of derivation:
X
FT 2π X 2π
δ(t − nT ) ←→ δ ω− k
T T
n∈Z k∈Z
◮ Not a convergent Fourier transform (in elementary sense)
Mathematical/technological plausibility of use:

X
b
x (t) = x(nT ) sinc( Tπ (t − nT ))
n∈Z
◮ At each t, reconstruction is an infinite sum

◮ Very slow decay of terms makes truncation accuracy poor
Technological implementability:
◮ Point measurements difficult to approximate physically
◮ Causality of reconstruction
Dissatisfaction
X
FT 2π X 2π
δ(t − nT ) ←→ δ ω− k
T T
n∈Z k∈Z

X
b
n∈Z

Dissatisfaction
X
FT 2π X 2π
δ(t − nT ) ←→ δ ω− k
T T
n∈Z k∈Z

X
b
n∈Z

Example Lecture: Sampling and Interpolation Operator View
Operator view: Interpolation
Definition (Interpolation operator)

For a fixed positive T and interpolation postfilter g (t), let Φ : ℓ2 (Z) → L2 (R) be
given by X
(Φ y )(t) = yn √1T g (t/T − n), t∈R
n∈Z
Φ
y g (t) b
x
T
Generalizes sinc interpolation

P
For simplicity, we will consider only T = 1: (Φ y )(t) = n∈Z yn g (t − n)

Reconstruction space
Range of interpolation operator has special form
Definition (Shift-invariant subspace of L2 (R))

A subspace W ⊂ L2 (R) is a shift-invariant subspace with respect to shift T ∈ R+
when x(t) ∈ W implies x(t − kT ) ∈ W for every integer k. In addition,
w ∈ L2 (R) is called a generator of W when W = span({w (t − kT )}k∈Z).
Range of Φ is a shift-invariant subspace generated by g

Operator view: Sampling
Definition (Sampling operator)

For a fixed positive T and sampling prefilter g ∗ (−t), let Φ∗ : L2 (R) → ℓ2 (Z) be
given by
(Φ∗ x)n = hx(t), √1T g (t/T − n)it , n∈Z
Φ∗
T
x g ∗ (−t) y
For simplicity, we will consider only T = 1:
(Φ∗ x)n = hx(t), g (t − n)it

Adjoint relationship between sampling and interpolation
Theorem
Sampling and interpolation operators are adjoints
Let x ∈ L2 (R) and y ∈ ℓ2 (Z)

X
hΦ∗ x, y i = hx(t), g (t − n)it yn∗
n∈Z
X Z ∞
= x(t) g ∗ (t − n) dt yn∗
n∈Z −∞
Z !
∞ X
∗
= x(t) g (t − n) yn∗ dt
−∞ n∈Z
Z !∗
∞ X
= x(t) yn g (t − n) dt
−∞ n∈Z
= hx, Φ y i

Adjoint relationship between sampling and interpolation
Theorem
Sampling and interpolation operators are adjoints
Let x ∈ L2 (R) and y ∈ ℓ2 (Z)

X
hΦ∗ x, y i = hx(t), g (t − n)it yn∗
n∈Z
X Z ∞
= x(t) g ∗ (t − n) dt yn∗
n∈Z −∞
Z !
∞ X
∗
= x(t) g (t − n) yn∗ dt
−∞ n∈Z
Z !∗
∞ X
= x(t) yn g (t − n) dt
−∞ n∈Z
= hx, Φ y i

Relationships between sampling and interpolation

x = Φ Φ∗ x
Sampling followed by interpolation: b
T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)
b
x is best approximation of x within shift-invariant subspace generated by g if
P = Φ Φ∗ is an orthogonal projection operator
P is automatically self-adjoint: P ∗ = (Φ Φ∗ )∗ = P
Need P idempotent: P 2 = Φ Φ∗ Φ Φ∗ = P
Require Φ∗ Φ = I ⇒ study interpolation followed by sampling


x = Φ Φ∗ x
T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)
b


x = Φ Φ∗ x
T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)
b


x = Φ Φ∗ x
T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)
b


Interpolation followed by sampling: yb = Φ∗ Φ y
b
x (t) T
yn
T
g (t) g ∗ (−t) ybn
Consider output due to input y = δ
ybn = hg (t − n), g (t)it
Shifting input shifts output
Φ∗ Φ = I if and only if hg (t − n), g (t)it = δn


b
x (t) T
yn
T
g (t) g ∗ (−t) ybn


b
x (t) T
yn
T
g (t) g ∗ (−t) ybn

Sampling for shift-invariant subspaces
Theorem
Let g be orthogonal to its integer shifts: hg (t − n), g (t)it = δn . The system
T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)
yields b
x = P x where P is the orthogonal projection operator onto the
shift-invariant subspace S generated by g .
Corollaries:
If x ∈ S, then x is recovered exactly from samples y
If x 6∈ S, then b
x is the best approximation of x in S

Sampling for shift-invariant subspaces
Theorem
Let g be orthogonal to its integer shifts: hg (t − n), g (t)it = δn . The system
T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)
yields b
shift-invariant subspace S generated by g .
Corollaries:
If x ∈ S, then x is recovered exactly from samples y
If x 6∈ S, then b
x is the best approximation of x in S

Reinterpreting classical sampling
T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)
Case of g (t) = sinc(πt)
sinc(πt) is orthogonal to its integer shifts

◮ Immediately, orthogonal projection property holds
Prefilter bandlimits (“anti-aliasing”)
g ∗ (−t) = g (t)

Reinterpreting classical sampling
T
yn
x(t) g ∗ (−t) T
g (t) b
x (t)
Case of g (t) = sinc(πt)
sinc(πt) is orthogonal to its integer shifts

◮ Immediately, orthogonal projection property holds
Prefilter bandlimits (“anti-aliasing”)
g ∗ (−t) = g (t)

Discrete-time version (downsampling)
Definition (Sampling operator)

For a fixed positive N and sampling filter g , let Φ∗ : ℓ2 (Z) → ℓ2 (Z) be given by
(Φ∗ x)k = hxn , gn−kN in , k ∈Z
Φ∗
∗ y
x g−n N
Definition (Interpolation operator)

For a fixed positive N and interpolation filter g , let Φ : ℓ2 (Z) → ℓ2 (Z) be given by
X
(Φ y )n = yk gn−kN , n∈Z
k∈Z
Φ
y N gn b
x

Discrete-time version (downsampling)
Definition (Shift-invariant subspace of ℓ2 (Z))

A subspace W ⊂ ℓ2 (Z) is a shift-invariant subspace with respect to shift L ∈ Z+
when xn ∈ W implies xn−kL ∈ W for every integer k. In addition, w ∈ ℓ2 (Z) is
called a generator of W when W = span({wn−kL }k∈Z ).
Theorem
Let g be orthogonal to its shifts by multiples of N: hgn−kN , gn in = δk . The system
∗
yn
xn g−n N N gn b
xn
yields b
shift-invariant subspace S generated by g with shift N.

Beyond orthogonal case
T
yn
x(t) ge(t) T
g (t) b
x (t)
e ∗ , interpolation operator Φ
Sampling operator Φ

T
yn
x(t) ge(t) T
g (t) b
x (t)
e ∗ generally not self-adjoint, but can still be a projection operator
ΦΦ
e ∗ Φ = I for an oblique projection
Check Φ

T
yn
x(t) ge(t) T
g (t) b
x (t)
e ∗ generally not self-adjoint, but can still be a projection operator
ΦΦ
e ∗ Φ = I for an oblique projection
Check Φ
e
S
·
S
•x
•
b
x

Example Lecture: Sampling and Interpolation Basis Expansion View
Review: Operators associated with bases

Synthesis operator associated with basis {ϕk }k∈K for H
X
◮ Φ : ℓ2 (K) → H Φα = αk ϕk
k∈K

Analysis operator associated with basis {ϕk }k∈K for H
◮ Φ∗ : H → ℓ2 (K) (Φ∗ x)k = hx, ϕk i, k ∈ K
P
When {ϕk }k∈K is an orthonormal set, k∈K hx, ϕk iϕk is an orthogonal
projection
Special case of a shift-variant space and a basis obtained from shifts of
function:
ϕk = g (t − k)

Example Lecture: Sampling and Interpolation Basis Expansion View
Review: Operators associated with bases

Synthesis operator associated with basis {ϕk }k∈K for H
X
◮ Φ : ℓ2 (K) → H Φα = αk ϕk
k∈K

Analysis operator associated with basis {ϕk }k∈K for H
◮ Φ∗ : H → ℓ2 (K) (Φ∗ x)k = hx, ϕk i, k ∈ K
P
When {ϕk }k∈K is an orthonormal set, k∈K hx, ϕk iϕk is an orthogonal
projection
Special case of a shift-variant space and a basis obtained from shifts of
function:
ϕk = g (t − k)

Example Lecture: Sampling and Interpolation Extensions
Variations
Multichannel sampling
2T y0,n
x(t) ge0 (t)
2T y1,n
ge1 (t)
Sample signal and derivatives

Periodic nonuniform sampling (time-interleaved ADC)
Many inverse problems have linear forward models, perhaps not shift-invariant
Similar subspace geometry holds

Variations
Multichannel sampling
2T y0,n
x(t) ge0 (t)
2T y1,n
ge1 (t)
Sample signal and derivatives

Periodic nonuniform sampling (time-interleaved ADC)
Many inverse problems have linear forward models, perhaps not shift-invariant
Similar subspace geometry holds

Beyond subspaces
Definition (Semilinear signal model)

A subset W ⊂ L2 (R) has rate of innovation ρ with respect to generator g if any
x ∈ W can be written as X
x = αk g (t − tk )
k∈Z
where
#{tk in [−T /2, T /2]} ρ
lim sup =
T →∞ T 2
W is not a subspace
For some g , exact recovery from uniform samples at rate ρ is possible
Many classes of techniques

Example Lecture: Sampling and Interpolation Summary
Summary
Adjoints
◮ Time reversal between sampling and interpolation
Subspaces
◮ Shift-invariant, range of interpolator Φ
◮ Null space of sampler Φ∗
Projection
◮ Φ Φ∗ always self adjoint
◮ Φ∗ Φ = I implies Φ Φ∗ is a projection operator
◮ Together, orthogonal projection operator, best approximation
Basis expansions
◮ Sampling produces analysis coefficients for basis expansion
◮ Interpolation synthesizes from expansion coefficients

Teaching Materials Textbooks
Textbooks
Two books:
M. Vetterli, J. Kovačević, and V. K. Goyal,
Foundations of Signal Processing
J. Kovačević, V. K. Goyal, and M. Vetterli,

Manuscripts distributed in draft form online (originally as

a single volume and with some variations in titles) since
2010 at
http://www.fourierandwavelets.org
Free, online versions have gray scale images, no PDF

hyperlinks, no exercises with solutions or exercises

Textbooks
Foundations of Signal Processing
1 On Rainbows and Spectra
2 From Euclid to Hilbert
3 Sequences and Discrete-Time Systems
4 Functions and Continuous-Time Systems
5 Sampling and Interpolation
6 Approximation and Compression
7 Localization and Uncertainty
Features:
About 640 pages illustrated with more than 200 figures
More than 200 exercises (more than 30 with solutions within the text)
Solutions manual for instructors
Summary tables, guides to further reading, historical notes
Textbooks
1 Filter Banks: Building Blocks of Time-Frequency Expansions
2 Local Fourier Bases on Sequences
3 Wavelet Bases on Sequences
4 Local Fourier and Wavelet Frames on Sequences
5 Local Fourier Transforms, Frames and Bases on Functions
6 Wavelet Bases, Frames and Transforms on Functions
7 Approximation, Estimation, and Compression

Prerequisites
Textbook is a mostly self-contained treatment

Mathematical maturity
◮ Mechanical use of calculus not enough
◮ Sophistication to read and write precise mathematical statements needed
(or could be learned here)
Linear algebra
◮ Basic facility with matrix algebra very useful
◮ Abstract view built carefully within the book
Probability
◮ Basic background (e.g., first half of Introduction to Probability by Bertsekas
and Tsitsiklis) needed (else all stochastic material could be skipped)
Signals and systems
◮ Basic background (e.g., Signals and Systems by Oppenheim and Willsky)
helpful but not necessary
Teaching Materials Supplementary materials
Solutions manual
Convolution of Derivative and Primitive
Let h and x be differentiable functions, and let
Z t Z t
h(1) (t) = h(τ ) dτ and x (1) (t) = x(τ ) dτ
−∞ −∞
be their primitives. Give a sufficient condition for h ∗ x = h(1) ∗ x ′ based on

integration by parts.

Solutions manual
From the definition of convolution, (4.35),
Z ∞
(h ∗ x)(t) = x(τ )h(t − τ ) dτ.
−∞
We wish to apply definite integration by parts, (2.204b), to get to a form

involving h(1) and x ′ . With the associations
u(τ ) = x(τ ) and v ′ (τ ) = h(t − τ ),
we obtain
u ′ (τ ) = x ′ (τ ) and v (τ ) = −h(1) (t − τ ).
Substituting these into (2.204b) gives

Z ∞
t=∞
(h ∗ x)(t) = −x(τ ) h(1) (t − τ )t=−∞ + h(1) (t − τ ) x ′ (τ ) dτ. (1)
−∞
This yields the desired result of
(h ∗ x)(t) = (h(1) ∗ x ′ )(t), for all t ∈ R,
provided that the first term of (1) is zero:
lim x(τ ) h(1) (t − τ ) = 0, for all t ∈ R.

τ →±∞

Mathematica figures
|X10 (e jω )| |X100 (e jω )|
2 2
Π Π 3Π
Π
ω Π Π 3Π
Π
ω
4 2 4 4 2 4
(a) N = 10 (b) N = 100
|X1000 (e jω )| |X1000 (e jω )|
2
Π Π 3Π
Π
ω 9Π Π ω
4 2 4 20 2
(c) N = 1000 (d) Detail of (c).
Figure: Truncated DTFT of the sinc sequence,

illustrating the Gibbs phenomenon. Shown are
|XN (e jω )| from (3.84) with different N. Observe how
oscillations narrow from (a) to (c), but their amplitude
remains√ constant (the topmost grid line in every plot),
1.089 2.
Wrap up
Why rethink how signal processing is taught?
Signal processing is an essential and vibrant field
Geometry is key to gaining intuition and understanding

1 Tutorial EUSIPCO2012 PDF

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

1 Tutorial EUSIPCO2012 PDF

Hochgeladen von

Copyright:

Verfügbare Formate

Teaching Signal Processing with Geometry

Martin Vetterli and Vivek Goyal

August 27, 2012

Teaching about teaching?

Computing platforms change (and we should want them to)

Students change (and we should want them to)

Eternal challenge of the educator

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 3 / 96

Teaching about teaching?

Computing platforms change (and we should want them to)

Students change (and we should want them to)

Eternal challenge of the educator

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 3 / 96

Teaching about teaching?

Computing platforms change (and we should want them to)

Students change (and we should want them to)

Eternal challenge of the educator

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 3 / 96

Teaching about teaching?

Computing platforms change (and we should want them to)

Students change (and we should want them to)

Eternal challenge of the educator

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 3 / 96

Structure of the tutorial:

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 4 / 96

Structure of the tutorial:

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 4 / 96

Structure of the tutorial:

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 4 / 96

Structure of the tutorial:

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 4 / 96

Each can placed in a common framework featuring geometry

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 5 / 96

Each can placed in a common framework featuring geometry

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 5 / 96

Each can placed in a common framework featuring geometry

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 5 / 96

Each can placed in a common framework featuring geometry

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 5 / 96

Unifying framework: Hilbert spaces

More abstraction. More mathematics. More difficult?

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 6 / 96

Unifying framework: Hilbert spaces

More abstraction. More mathematics. More difficult?

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 6 / 96

Correct intuitions are separate from functional analysis details

. . . if whether an airplane would fly or not depended on whether some

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 7 / 96

Correct intuitions are separate from functional analysis details

. . . if whether an airplane would fly or not depended on whether some

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 7 / 96

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 8 / 96

CZ : sequences – discrete-time signals

CR : functions – continuous-time signals

polynomials of degree at most K

scalar random variables

discrete-time stochastic processes

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 9 / 96

◮ span(S) is always a subspace

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 10 / 96

◮ span(S) is always a subspace

Vetterli & Goyal www.FourierAndWavelets.org August 27, 2012 10 / 96

◮ If S is infinite, we need every finite subset to be linearly independent