Trigonometric Polynomial Approximation

The Discrete Fourier Transform

Trigonometric Least Squares Solution

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Outline

Numerical Analysis and Computing

Lecture Notes #14
Approximation Theory
Trigonometric Polynomial Approximation
Joe Mahaffy,
hmahaffy@math.sdsu.edui

Trigonometric Polynomial Approximation

Introduction
Fourier Series

The Discrete Fourier Transform

Introduction
Discrete Orthogonality of the Basis Functions

Trigonometric Least Squares Solution

Expressions
Examples

Department of Mathematics

Dynamical Systems Group

Computational Sciences Research Center

San Diego State University

San Diego, CA 92182-7720

http://www-rohan.sdsu.edu/jmahaffy

Spring 2010
Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Introduction
Fourier Series

(1/22)

Trigonometric Polynomials: A Very Brief History

P(x) =

an cos(nx) + i

n=0

an sin(nx)

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Introduction
Fourier Series

(2/22)

For each positive integer n, the set of functions

{0 , 1 , . . . , 2n1 }, where

n=0

Jean Le Rond dAlembert used finite sums of sin and cos

to study vibrations of a string.

17xx

Use adopted by Leonhard Euler (leading mathematician at

the time).

17xx

Daniel Bernoulli advocates use of infinite (as above) sums

of sin and cos.

18xx

Jean Baptiste Joseph Fourier used these infinite series

to study heat flow. Developed theory.
Trig. Polynomial Approx.

Trig. Polynomial Approx.

Fourier Series: First Observations

1750s

Joe Mahaffy, hmahaffy@math.sdsu.edui

Joe Mahaffy, hmahaffy@math.sdsu.edui

(3/22)

1
2
k (x) = cos(kx), k = 1, . . . , n

n+k (x) = sin(kx), k = 1, . . . , n 1

0 (x) =

is an Orthogonal set on the interval [, ] with respect to the

weight function w (x) = 1.

Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

(4/22)

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Introduction
Fourier Series

Orthogonality

The Fourier Series, S(x)

For f C [, ], we seek the continuous least squares
approximation by functions in Tn of the form

Orthogonality follows from the fact that integrals over [, ] of

cos(kx) and sin(kx) are zero (except cos(0)), and products can be
rewritten as sums:

sin 1 sin 2 =

cos 1 cos 2 =

sin 1 cos 2 =

n1
k=1

cos(1 2 ) cos(1 + 2 )
2

where, thanks to orthogonality

cos(1 2 ) + cos(1 + 2 )
2

1
ak =

sin(1 2 ) + sin(1 + 2 )
.
2

Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Introduction
Fourier Series

2
|x| dx =

ak

1
bk =

f (x) cos(kx) dx,

f (x) sin(kx) dx.

S(x) = lim Sn (x)

n

is called the Fourier Series of f .

(5/22)

1 of 2

x dx = .

The limit

First we note that f (x) and cos(kx) are even functions on [, ]

and sin(kx) are odd functions on [, ]. Hence,
Z

Definition (Fourier Series)

Example: Approximating f (x) = |x| on [, ]

X
a0
(ak cos(kx) + bk sin(kx)) ,
+ an cos(nx) +
2

Sn (x) =

Let Tn be the set of all linear combinations of the functions

{0 , 1 , . . . , 2n1 }; this is the set of trigonometric
polynomials of degree n.

a0

Introduction
Fourier Series

Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Introduction
Fourier Series

Example: Approximating f (x) = |x| on [, ]

We can write down Sn (x) =

k=1

f(x)
S0(x)
S1(x)
S3(x)
S5(x)
S7(x)

1
2
|x| cos(kx) dx =
x cos(kx) dx

0

Z
2 sin(kx)
2
1 sin(kx) dx
=

k
k 0
0
{z
}
|

2 of 2

n
2 X (1)k 1

+
cos(kx)
2
k2

(6/22)

=
bk

2
2
[cos(k) cos(0)] =
(1)k 1 .
2
2
k
k
1

|x| sin(kx)
| {z }

dx = 0.

even odd = odd.

Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

(7/22)

-2

Joe Mahaffy, hmahaffy@math.sdsu.edui

2
Trig. Polynomial Approx.

(8/22)

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Introduction
Discrete Orthogonality of the Basis Functions

The Discrete Fourier Transform: Introduction

Introduction
Discrete Orthogonality of the Basis Functions

The Discrete Fourier Transform

The discrete Fourier transform, a.k.a. the finite Fourier transform,

is a transform on samples of a function.
It, and its cousins, are the most widely used mathematical
transforms; applications include:

Suppose we have 2m data points, (xj , fj ), where

xj = +

j
, and fj = f (xj ),
m

j = 0, 1, . . . , 2m 1.

The discrete least squares fit of a trigonometric polynomial

Sn (x) Tn minimizes

Signal Processing
Image Processing
Audio Processing

Data compression

E (Sn ) =

A tool for partial differential equations

2m1
X

[Sn (xj ) fj ]2 .

j=0

etc...
Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

(9/22)

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Introduction
Discrete Orthogonality of the Basis Functions

Orthogonality of the Basis Functions?

Trig. Polynomial Approx.

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Introduction
Discrete Orthogonality of the Basis Functions

(10/22)

Orthogonality of the Basis Functions! (A Lemma)...

We know that the basis functions

0 (x) =

(x)
=
cos(kx),
k = 1, . . . , n
k

n+k (x) = sin(kx), k = 1, . . . , n 1

Lemma
If the integer r is not a multiple of 2m, then
2m1
X

are orthogonal with respect to integration over the interval.

k (xj )l (xj ) = k k,l

cos(rxj ) =

j=0

The Big Question: Are they orthogonal in the discrete case? Is

the following true:
2m1
X

Joe Mahaffy, hmahaffy@math.sdsu.edui

sin(rxj ) = 0.

j=0

Moreover, if r is not a multiple of m, then

2m1
X

[cos(rxj )]2 =

j=0

???

2m1
X

2m1
X

[sin(rxj )]2 = m.

j=0

j=0

Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

(11/22)

Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

(12/22)

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Introduction
Discrete Orthogonality of the Basis Functions

Proof of Lemma

1 of 3

Recalling long-forgotten (or quite possible never seen) facts from

Complex Analysis Eulers Formula:

Proof of Lemma
2 of 3
P
irj/m is a geometric series with first term 1, and
Since 2m1
j=0 e
ratio e ir /m 6= 1, we get

e i = cos() + i sin().

2m1
X

Thus,
2m1
X

cos(rxj ) + i

sin(rxj ) =

2m1
X

2m1
X

j=0

j=0

[cos(rxj ) + i sin(rxj )] =

j=0

j=0

1 e 2ir = 1 cos(2r ) i sin(2r ) = 1 1 i 0 = 0.

e irxj = e ir (+j/m) = e ir e irj/m ,

2m1
X

cos(rxj ) + i

2m1
X

sin(rxj ) = e

ir

j=0

j=0

2m1
X

This shows the first part of the lemma:

e

irj/m

2m1
X

Trig. Polynomial Approx.

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Introduction
Discrete Orthogonality of the Basis Functions

Proof of Lemma

(13/22)

3 of 3

j=0

2m1
X
j=0

2m1
X 1
1 + cos(2rxj )
=
= m.
2
2
j=0

Similarly (use cos2 + sin2 = 1)

2m1
X

2m1
X

sin(rxj ) = 0.

j=0

Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Introduction
Discrete Orthogonality of the Basis Functions

(14/22)

Showing Orthogonality of the Basis Functions

Recall

If r is not a multiple of m, then

[cos(rxj )]2 =

cos(rxj ) =

j=0

j=0

Joe Mahaffy, hmahaffy@math.sdsu.edui

2m1
X

1 e 2ir
1 (e ir /m )2m
=
.
1 e ir /m
1 e ir /m

This is zero since

e irxj .

Since
we get

e irj/m =

j=0
2m1
X

Introduction
Discrete Orthogonality of the Basis Functions

sin 1 sin 2 =

cos 1 cos 2 =

sin 1 cos 2 =

cos(1 2 ) cos(1 + 2 )
2
cos(1 2 ) + cos(1 + 2 )
2
sin(1 2 ) + sin(1 + 2 )
.
2

Thus for any pair k 6= l

[sin(rxj )]2 = m.

j=0

2m1
X

This proves the second part of the lemma.

We are now ready to show that the basis functions are orthogonal.

k (xj )l (xj )

j=0

is a zero-sum of sin or cos, and when k = l, the sum is m.

Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

(15/22)

Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

(16/22)

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Expressions
Examples

Finally: The Trigonometric Least Squares Solution

Example: Discrete Least Squares Approximation

Using
[1] Our standard framework for deriving the least squares solution
set the partial derivatives with respect to all parameters
equal to zero.
[2]

j
0
1
2
3
4
5
6
7
8
9

We find the coefficients in the summation

n1

X
a0
(ak cos(kx) + bk sin(kx)) :
Sn (x) =
+ an cos(nx) +
2
k=1

2m1
1 X
bk =
fj sin(kxj ).
m

j=0

j=0

Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Expressions
Examples

(17/22)

Example: Discrete Least Squares Approximation

2 of 3

1 of 3

Let f (x) = x 3 2x 2 + x + 1/(x 4) for x [, ].

Let xj = + j/5, j = 0, 1, . . . , 9., i.e.

The orthogonality of the basis functions.

2m1
1 X
ak =
fj cos(kxj ),
m

Expressions
Examples

xj
-3.14159
-2.51327
-1.88495
-1.25663
-0.62831
0
0.62831
1.25663
1.88495
2.51327

fj
-54.02710
-31.17511
-15.85835
-6.58954
-1.88199
-0.25
-0.20978
-0.28175
1.00339
5.08277

Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Expressions
Examples

Example: Discrete Least Squares Approximation

(18/22)

3 of 3

We get the following coefficients:

a0 = 20.837, a1 = 15.1322, a2 = 9.0819, a3 = 7.9803
b1 = 8.8661, b2 = 7.8193, b3 = 4.4910.
20

f(x)
s1(x)
s3(x)
s5(x)

-20

-40

-60

-2

[1]

The approximation get better as n .

[2]

Since all the Sn (x) are 2-periodic, we will always have a

problem when f () 6= f (). [Fix: Periodic extension.] On
the following two slides we see the performance for a 2periodic f .

[3]

It seems like we need O(m2 ) operations to compute

a and
m sums, with m additions and multiplications. There
b
is however a fast O(m log2 (m)) algorithm that finds these
coefficients. We will talk about this Fast Fourier Transform
next time.

S1 (x), S3 (x), S5 (x)

Joe Mahaffy, hmahaffy@math.sdsu.edui

Notes:

Trig. Polynomial Approx.

(19/22)

Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

(20/22)

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Trigonometric Polynomial Approximation

The Discrete Fourier Transform
Trigonometric Least Squares Solution

Expressions
Examples

Example(2): Discrete Least Squares Approximation

1 of 2

Expressions
Examples

Example(2): Discrete Least Squares Approximation

2 of 2

We get the following coefficients:

Let f (x) = 2x 2 + cos(3x) + sin(2x), x [, ].
Let xj = + j/5, j = 0, 1, . . . , 9., i.e.
j
0
1
2
3
4
5
6
7
8
9

xj
-3.14159
-2.51327
-1.88495
-1.25663
-0.62831
0
0.62831
1.25663
1.88495
2.51327

a0 = 8.2685,

a1 = 2.2853, a2 = 0.2064,
b1 = 0, b2 = 1, b3 = 0.

a3 = 0.8729

20

fj
18.7392
13.8932
8.5029
1.7615
-0.4705
1.0000
1.4316
2.9370
7.3273
11.9911

15

f(x)
s1(x)
s3(x)
s5(x)

10

0
-4
Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

(21/22)

-2

S1 (x), S3 (x), S5 (x)

Joe Mahaffy, hmahaffy@math.sdsu.edui

Trig. Polynomial Approx.

4
(22/22)