Review

Matrix Equations
Convergence Properties
Summary

C ONVERGENCE P ROPERTIES OF I TERATION

S CHEMES

Dr. Johnson

School of Mathematics

Semester 1 2008

Dr. Johnson MATH65241

 Review
Matrix Equations
Convergence Properties
Summary

O UTLINE

1 R EVIEW

2 M ATRIX E QUATIONS
Rearranging the Matrix
Iteration Matrix Equations

3 C ONVERGENCE P ROPERTIES
The Iteration Matrix
Properties of Convergence
Further Convergence Properties

4 S UMMARY

Dr. Johnson MATH65241

 Review
Matrix Equations
Convergence Properties
Summary

O UTLINE

1 R EVIEW

2 M ATRIX E QUATIONS
Rearranging the Matrix
Iteration Matrix Equations

3 C ONVERGENCE P ROPERTIES
The Iteration Matrix
Properties of Convergence
Further Convergence Properties

4 S UMMARY

Dr. Johnson MATH65241

 Review
Matrix Equations
Convergence Properties
Summary

O UTLINE

1 R EVIEW

2 M ATRIX E QUATIONS
Rearranging the Matrix
Iteration Matrix Equations

3 C ONVERGENCE P ROPERTIES
The Iteration Matrix
Properties of Convergence
Further Convergence Properties

4 S UMMARY

Dr. Johnson MATH65241

 Review
Matrix Equations
Convergence Properties
Summary

O UTLINE

1 R EVIEW

2 M ATRIX E QUATIONS
Rearranging the Matrix
Iteration Matrix Equations

3 C ONVERGENCE P ROPERTIES
The Iteration Matrix
Properties of Convergence
Further Convergence Properties

4 S UMMARY

Dr. Johnson MATH65241

 Review
Matrix Equations
Convergence Properties
Summary

R ELAXATION M ETHODS

The model equations can be rewritten

wi+1,j − 2wi,j + wi−1,j wi,j+1 − 2wi,j + wi,j−1

+ = fi,j ,
∆x2 ∆y2

How up-to-date the values are define different schemes

We can over-relax to improve convergence

q+1 q ∗
wi,j = (1 − ω )wi,j + ωwi,j

and also solve a line (or block) using a direct method.

Dr. Johnson MATH65241

 Review
Matrix Equations
Convergence Properties
Summary

R ELAXATION M ETHODS

The model equations can be rewritten

wi+1,j − 2wi,j + wi−1,j wi,j+1 − 2wi,j + wi,j−1

+ = fi,j ,
∆x2 ∆y2

How up-to-date the values are define different schemes

We can over-relax to improve convergence

q+1 q ∗
wi,j = (1 − ω )wi,j + ωwi,j

and also solve a line (or block) using a direct method.

Dr. Johnson MATH65241

 Review
Matrix Equations Rearranging the Matrix
Convergence Properties Iteration Matrix Equations
Summary

O UTLINE

1 R EVIEW

2 M ATRIX E QUATIONS
Rearranging the Matrix
Iteration Matrix Equations

3 C ONVERGENCE P ROPERTIES
The Iteration Matrix
Properties of Convergence
Further Convergence Properties

4 S UMMARY

Dr. Johnson MATH65241

 Review
Matrix Equations Rearranging the Matrix
Convergence Properties Iteration Matrix Equations
Summary

M ATRIX E QUATION

Given some PDE problem, it can always be reduced to the

solution of a linear system

Ax = b
where A = (ai,j ) is an (n × n) matrix, and x, b are (n × 1)
column vectors.

Dr. Johnson MATH65241

 Review
Matrix Equations Rearranging the Matrix
Convergence Properties Iteration Matrix Equations
Summary

M ATRIX E QUATION

Let write the matrix A in the form A = D + L + U where D,

L, U are the diagonal matrix, lower and upper triangular
parts of A, ie
 
a1,1 0
 0 a2,2 0 
 

 0 a3,3 0 

 .. 
D=
 . 

 .. 

 . 

 0 an−1,n−1 0 
0 an,n

Dr. Johnson MATH65241

 Review
Matrix Equations Rearranging the Matrix
Convergence Properties Iteration Matrix Equations
Summary

M ATRIX E QUATION

 
0 a1,2 . . .
 0 0 a2,3 
 

 0 0 a3,4 

 .. 
U= 0 . 
..
 
 .. 

 0 . . 

 0 0 an−1,n 
0 0

Dr. Johnson MATH65241

 Review
Matrix Equations Rearranging the Matrix
Convergence Properties Iteration Matrix Equations
Summary

M ATRIX E QUATION

 
0 0
 a2,1 0 0 
 
 .. 
 . a3,2 0 0 
 
L=
 .. 
 . 

 ..
.

 
 
 an−1,n−2 0 0 
... an,n−1 0

Dr. Johnson MATH65241

 Review
Matrix Equations Rearranging the Matrix
Convergence Properties Iteration Matrix Equations
Summary

M ATRIX E QUATION

Then we may write

Ax = b
as

Lx + Dx + Ux = b.

In this way we can express our iteration schemes in terms

of matrices.

Dr. Johnson MATH65241

 Review
Matrix Equations Rearranging the Matrix
Convergence Properties Iteration Matrix Equations
Summary

O UTLINE

1 R EVIEW

2 M ATRIX E QUATIONS
Rearranging the Matrix
Iteration Matrix Equations

3 C ONVERGENCE P ROPERTIES
The Iteration Matrix
Properties of Convergence
Further Convergence Properties

4 S UMMARY

Dr. Johnson MATH65241

 Review
Matrix Equations Rearranging the Matrix
Convergence Properties Iteration Matrix Equations
Summary

I TERATIVE METHODS

Jacobi iteration may be written as:

x(k +1) = D−1 ( b − ( L + U ) x(k ) )

Gauss-Seidel iteration may be written as:

x(k+1) = (D + L)−1 (b − Ux(k) )

Dr. Johnson MATH65241

 Review
Matrix Equations Rearranging the Matrix
Convergence Properties Iteration Matrix Equations
Summary

I TERATIVE METHODS

Jacobi iteration may be written as:

x(k +1) = D−1 ( b − ( L + U ) x(k ) )

Gauss-Seidel iteration may be written as:

x(k+1) = (D + L)−1 (b − Ux(k) )

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

O UTLINE

1 R EVIEW

2 M ATRIX E QUATIONS
Rearranging the Matrix
Iteration Matrix Equations

3 C ONVERGENCE P ROPERTIES
The Iteration Matrix
Properties of Convergence
Further Convergence Properties

4 S UMMARY

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

G ENERAL I TERATION S CHEME

In general an iteration scheme may be written as

x(k+1) = Px(k) + Q,

where P is the iteration matrix

For the Jacobi scheme scheme we have

P = PJ = D−1 (−L − U )

For the Gauss-Seidel scheme

P = PG = (D + L)−1 (−U ).

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

G ENERAL I TERATION S CHEME

In general an iteration scheme may be written as

x(k+1) = Px(k) + Q,

where P is the iteration matrix

For the Jacobi scheme scheme we have

P = PJ = D−1 (−L − U )

For the Gauss-Seidel scheme

P = PG = (D + L)−1 (−U ).

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

O UTLINE

1 R EVIEW

2 M ATRIX E QUATIONS
Rearranging the Matrix
Iteration Matrix Equations

3 C ONVERGENCE P ROPERTIES
The Iteration Matrix
Properties of Convergence
Further Convergence Properties

4 S UMMARY

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

E RROR V ECTORS

The exact solution to the linear system satisfies

x = Px + Q.

The approximation at the kth iteration has some error

defined by

x(k ) = x + e(k )

then the error satisfies the equation

e(k+1) = Pe(k) = P2 e(k−1) = Pk+1 e(0) .

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

D IMINISHING ERRORS

In order that the error diminishes as k → ∞ we must have

||Pk || → 0 as k → ∞,

Since

||Pk || = ||P||k

we see that we require

||P|| < 1.

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

O UTLINE

1 R EVIEW

2 M ATRIX E QUATIONS
Rearranging the Matrix
Iteration Matrix Equations

3 C ONVERGENCE P ROPERTIES
The Iteration Matrix
Properties of Convergence
Further Convergence Properties

4 S UMMARY

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

S PECTRAL R ADIUS

From linear algebra

||P|| < 1

is equivalent to the requirement that

ρ(P) = max |λj | < 1

i

where λi are the eigenvalues of the matrix P, and ρ(P) is

called the spectral radius of P.

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

C ONVERGENCE R ATE
Note also that for large k

||e(k+1) || = ρ||e(k) ||

How many iterations does it takes to reduce the initial

error by a factor ǫ?
We need q iterations where q is the smallest value for which

ρq < ǫ

giving

log ǫ
q ≤ qd = .
log ρ

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

C ONVERGENCE R ATE
Note also that for large k

||e(k+1) || = ρ||e(k) ||

How many iterations does it takes to reduce the initial

error by a factor ǫ?
We need q iterations where q is the smallest value for which

ρq < ǫ

giving

log ǫ
q ≤ qd = .
log ρ

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

C ONVERGENCE R ATE
Note also that for large k

||e(k+1) || = ρ||e(k) ||

How many iterations does it takes to reduce the initial

error by a factor ǫ?
We need q iterations where q is the smallest value for which

ρq < ǫ

giving

log ǫ
q ≤ qd = .
log ρ

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

S PECTRAL R ADIUS - M ODEL P ROBLEM

Thus iteration matrices where the spectral radius is close to

1 will converge slowly.
For the model problem it can be shown that for Jacobi
iteration

cos πn + β2 cos m
¡ ¢ ¡π¢
ρ = ρ ( PJ ) = ,
1 + β2

and for Gauss-Seidel

ρ = ρ(PG ) = [ρ(PJ )]2 .

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

C ONVERGENCE P ROPERTIES

If we take n = m, dx = dy and n >> 1 then for Jacobi

iteration we have

log ǫ 2n2
qd = ³ ´ = − 2 log ǫ
log 1 − 2nπ2 π
2

For Gauss-Seidel we have that ρ(PG ) = [ρ(PJ )]2 so that

n2
qd = − log ǫ
π2
Gauss-Seidel converges twice as fast as Jacobi.

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

C ONVERGENCE P ROPERTIES

If we take n = m, dx = dy and n >> 1 then for Jacobi

iteration we have

log ǫ 2n2
qd = ³ ´ = − 2 log ǫ
log 1 − 2nπ2 π
2

For Gauss-Seidel we have that ρ(PG ) = [ρ(PJ )]2 so that

n2
qd = − log ǫ
π2
Gauss-Seidel converges twice as fast as Jacobi.

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

O PTIMAL SOR

We can relate the eigenvalues λ of the point SOR iteration

matrix to the eigenvalues µ of the point Jacobi iteration
matrix by the equation

(λ + ω − 1)2 = λω 2 µ2

From this equation it can be proved that the optimum

value of omega is

2
ωb = p
1+ 1 − [ρ(PJ )]2

There exist methods to estimate ρ(PJ ), but for the model

problem we have a formula.
Dr. Johnson MATH65241
 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

O PTIMAL SOR

We can relate the eigenvalues λ of the point SOR iteration

matrix to the eigenvalues µ of the point Jacobi iteration
matrix by the equation

(λ + ω − 1)2 = λω 2 µ2

From this equation it can be proved that the optimum

value of omega is

2
ωb = p
1+ 1 − [ρ(PJ )]2

There exist methods to estimate ρ(PJ ), but for the model

problem we have a formula.
Dr. Johnson MATH65241
 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

C ONVERGENCE R ATES FOR POINT SOR

For point SOR on the model problem with optimum values

and n = m it can be shown that

1 − sin πn
ρ=
1 + sin πn

giving

n
qd = − log ǫ.
2π

Dr. Johnson MATH65241

 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

C ONVERGENCE R ATES FOR LINE SOR

For line SOR on the model problem with optimum values
and n = m it can be shown that
¶2
1 − sin πn
µ
ρ=
1 + sin πn

giving
n
qd = − √ log ǫ.
2 2π
√
Line SOR converges 2 times faster than point SOR for the
model problem.
When choosing a method we must decide whether faster
convergence compensates for extra computations required.
Dr. Johnson MATH65241
 Review
The Iteration Matrix
Matrix Equations
Properties of Convergence
Convergence Properties
Further Convergence Properties
Summary

C ONVERGENCE R ATES FOR LINE SOR

For line SOR on the model problem with optimum values
and n = m it can be shown that
¶2
1 − sin πn
µ
ρ=
1 + sin πn

giving
n
qd = − √ log ǫ.
2 2π
√
Line SOR converges 2 times faster than point SOR for the
model problem.
When choosing a method we must decide whether faster
convergence compensates for extra computations required.
Dr. Johnson MATH65241
 Review
Matrix Equations
Convergence Properties
Summary

A linear system may be expressed as

Lx + Dx + Ux = b.

so that in a general iterative scheme may be written

x(k+1) = Px(k) + Q,

The condition for convergence on the iteration matrix P is

||P|| < 1.

Dr. Johnson MATH65241