Iterative Methods

BerlinChen
DepartmentofComputerScience&InformationEngineering
NationalTaiwanNormalUniversity

Reference:
1. Applied Numerical Methods with MATLAB for Engineers, Chapter 12 & Teaching material

Chapter Objectives
Understanding the difference between the Gauss-Seidel
and Jacobi methods
Knowing how to assess diagonal dominance and
knowing what it means
Recognizing how relaxation can be used to improve
convergence of iterative methods
Understanding how to solve systems of nonlinear
equations with successive substitution and NewtonRaphson

[A] {x} = {b}

NM Berlin Chen 2

Gauss-Seidel Method
The Gauss-Seidel method is the most commonly used
iterative method for solving linear algebraic equations
[A]{x}={b}
The method solves each equation in a system for a
particular variable, and then uses that value in later
equations to solve later variables
For a 3x3 system with nonzero elements along the
diagonal, for example, the jth iteration values are found
from the j-1th iteration using:
j1
j1
a11 x1 a12 x2 a13 x3 b1
a 21 x1 a 22 x2 a 23 x3 b2
a31 x1 a32 x2 a33 x3 b3

x1j

b1 a12 x2 a13 x3
a11

b2 a21 x1j a23 x3j1

x
a22
j
2

b3 a31 x1j a32 x2j

x
a33
j
3

NM Berlin Chen 3

Gauss-Seidel Method: Convergence

The convergence of an iterative method can be
calculated by determining the relative percent change of
each element in {x}. For example, for the i th element in
the j th iteration,

xij xij1
a,i
100%
j
xi
The method is ended when all elements have converged
to a set tolerance

NM Berlin Chen 4

Gauss-Seidel Method: An Example (1/2)

Example 12.1

NM Berlin Chen 5

Gauss-Seidel Method: An Example (2/2)

Example 12.1

NM Berlin Chen 6

Tactic of Gauss-Seidel Method

As each new x value is computed for the Gauss-Seidel
method, it is immediately used in the next equation to
determine another x value
Thus, if the solution is converging, the best available
estimates will be employed for the Gauss-Seidel method

NM Berlin Chen 7

Jacobi Iteration
The Jacobi iteration is similar to the Gauss-Seidel
method, except the j-1th information is used to update all
variables in the j th iteration:
Gauss-Seidel
Jacobi

NM Berlin Chen 8

Diagonal Dominance
The Gauss-Seidel method may diverge, but if the
system is diagonally dominant, it will definitely
converge
Diagonal dominance means:
n

aii aij
j1
ji

NM Berlin Chen 9

MATLAB Program

NM Berlin Chen 10

Relaxation
To enhance convergence, an iterative program
can introduce relaxation where the value at a
particular iteration is made up of a combination
of the old value and the newly calculated value:

xinew xinew 1 xiold

where is a weighting factor that is assigned a value between 0
and 2
0<<1: underrelaxation
=1: no relaxation
1<2: overrelaxation
The choice of a proper value for is highly problem-specific and
is often determined empirically
NM Berlin Chen 11

Gauss-Seidel with Relaxation: An Example (1/3)

Example 12.2

NM Berlin Chen 12

Gauss-Seidel with Relaxation: An Example (2/3)

Example 12.2

NM Berlin Chen 13

Gauss-Seidel with Relaxation: An Example (3/3)

Example 12.2

NM Berlin Chen 14

Nonlinear Systems
Nonlinear systems can also be solved using the same
strategy as the Gauss-Seidel method
Solve each system for one of the unknowns and update each
unknown using information from the previous iteration

This is called successive substitution

x12 x1 x2 10
x2 3x1 x22 57

NM Berlin Chen 15

Successive Substitution: An Example (1/2)

x12 x1 x2 10
x2 3x1 x22 57

Example 12.3
NM Berlin Chen 16

Successive Substitution: An Example (2/2)

1.Forsuccessivesubstitution,convergence
oftendependsonthemannerinwhich
theequationsareformulated
2.Divergencealsocanoccuriftheinitial
guessesareinsufficientlyclosetothetrue
solution

Example 12.3
NM Berlin Chen 17

Newton-Raphson (1/3)
Nonlinear systems may also be solved using the
Newton-Raphson method for multiple variables
For a two-variable system, the Taylor series
approximation and resulting Newton-Raphson equations
are:
f
f
f1,i 1 f1,i x1,i 1 x1,i

1,i

x2,i 1 x2,i

1,i

x1
x2
f 2,i
f 2,i
f 2,i 1 f 2,i x1,i 1 x1,i
x2,i 1 x2,i
x1
x2
f1,i 1 0

f 2,i 1 0

Recall in Chapter 6, Newton- Raphonfor

the root of a singleequation:

f xi 1 f xi xi 1 xi f xi 0

(first - order Taylor series expansionof f )

xi 1 xi

f xi
f xi

f1,i
f
f
f
x1,i 1 1,i x2,i 1 f1,i 1,i x1,i 1,i x2,i
x1
x2
x1
x2
f1,i
f
f
f
x1,i 1 1,i x2,i 1 f1,i 1,i x1,i 2,i x2,i
x1
x2
x1
x2
NM Berlin Chen 18

Newton-Raphson (2/3)
After algebraic manipulation
f 2 ,i
x2

f 1,i f 2 ,i
x1 x 2
f 1,i
f 2 ,i
x1

f 1,i f 2 ,i
x1 x 2
f 1,i

x 1,i 1 x 1,i

x 2 ,i 1 x 2 ,i

f 1, i

J fx1
2 ,i
x
1

f 1, i
x 2
f 2 ,i
x 2

ReferredtoastheJacobian ofthesystem

f 1,i
x2
f 1,i f 2 ,i

x 2 x1
f 2 ,i
f 1,i
x1
f 1,i f 2 ,i

x 2 x1
f 2 ,i

Referredtoasthedeterminant
oftheJacobian ofthesystem

NM Berlin Chen 19

Newton-Raphson (3/3)
Matrix notion of Newton-Raphson
Also can be generalized to n simultaneous equations

J x i 1 f J x i

x i 1 x i J 1 f

NM Berlin Chen 20

Newton-Raphson: An Example

x12 x1 x2 10
x2 3x1 x22 x12 57

f1 x1 , x2 x12 x1 x2 10 0

f 2 x1 , x2 x2 3x1 x22 x12 57 0

Example 12.4
NM Berlin Chen 21

MATLAB Program

NM Berlin Chen 22