Introduction to Numerical Analysis II:

Solving Linear System

Xiaozhe Hu

The Pennsylvania State University

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 1 / 10

Solving Linear System

Linear system:
Ax = b

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 2 / 10

Solving Linear System

Linear system:
Ax = b

Direct Method
I Gaussian Elimination
I PA = LU
Iterative Method
I Linear Iterative Method
I Krylov Subspace Method

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 2 / 10

Basic Linear Iterative Method

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 3 / 10

Basic Linear Iterative Method

General Form of Linear Iterative Method

Form Residual:
r = b Ax k
Solve Residual Equation Approximately:

Ae = r e = Br , B A1

Update:
x k+1 = x k + e

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 3 / 10

Basic Linear Iterative Method

General Form of Linear Iterative Method

Form Residual:
r = b Ax k
Solve Residual Equation Approximately:

Ae = r e = Br , B A1

Update:
x k+1 = x k + e

x k+1 = x k + B(b Ax k )

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 3 / 10

Examples

A=D +L+U

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 4 / 10

Examples

A=D +L+U

Richardson: B = I
Jacobi: B = D 1
Damped Jacobi: B = D 1
Forward Gauss-Seidel: B = (D + L)1
Backward Gauss-Seidel: B = (D + U)1
Symmetric Gauss-Seidel: B = (D + U)1 D(D + L1 )
SOR: B = (D + L)1
Symmetric SOR: B = w (1 w )(D + U)1 D(D + L)1

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 4 / 10

Convergence

e k+1 = (I BA)e k e k := x x k

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 5 / 10

Convergence

e k+1 = (I BA)e k e k := x x k

Theorem (Necessary and sufficient condition)

The linear iterative method converges if and only if

(I BA) < 1.

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 5 / 10

Convergence

e k+1 = (I BA)e k e k := x x k

Theorem (Necessary and sufficient condition)

The linear iterative method converges if and only if

(I BA) < 1.

Theorem (A sufficient condition when A is SPD)

If A is SPD, then the linear iterative method converges if

B 1 + B T A

is SPD

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 5 / 10

Example

2 2
Richardson Method: 0 < < max (A) , and opt = min (A)+max (A) .

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 6 / 10

Example

2 2
Richardson Method: 0 < < max (A) , and opt = min (A)+max (A) .

Jacobi Method: 2D A is SPD.

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 6 / 10

Example

2 2
Richardson Method: 0 < < max (A) , and opt = min (A)+max (A) .

Jacobi Method: 2D A is SPD.

Gauss-Seidel Method: Always converge when A is SPD.

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 6 / 10

Example

2 2
Richardson Method: 0 < < max (A) , and opt = min (A)+max (A) .

Jacobi Method: 2D A is SPD.

Gauss-Seidel Method: Always converge when A is SPD.

2
SOR Method: 0 < < 2 and opt =
1+ 1(I D 1 A)2

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 6 / 10

Nonnegative Matrix

Definition (Nonnegative Matrix)

A matrix A is nonnegative, if aij 0. We use notation A 0.

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 7 / 10

Nonnegative Matrix

Definition (Nonnegative Matrix)

A matrix A is nonnegative, if aij 0. We use notation A 0.

Theorem (Perron-Frobenius Theory)

Let A 0, then
A has a nonnegative real eigenvalue equal to its spectral radius.
To (A), there corresponds a nonzero eigenvector x 0.

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 7 / 10

Convergence: General Case

Definition (Regular Splitting of A)

A=M N
M is nonsingular, M 1 and N is nonnegative.

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 8 / 10

Convergence: General Case

Definition (Regular Splitting of A)

A=M N
M is nonsingular, M 1 and N is nonnegative.

I BA = I M 1 A = M 1 N

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 8 / 10

Convergence: General Case

Definition (Regular Splitting of A)

A=M N
M is nonsingular, M 1 and N is nonnegative.

I BA = I M 1 A = M 1 N

Theorem (Necessary and sufficient condition)

Let M and N be regular splitting of a matrix A, then (M 1 N) < 1 if and
only if A is nonsingular and A1 is nonnegative.

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 8 / 10

M-matrix

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 9 / 10

M-matrix
Definition (M-matrix)
A Rnn , aij 0 for all i 6= j. A is a M-matrix if A is nonsingular and
A1 0.

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 9 / 10

M-matrix
Definition (M-matrix)
A Rnn , aij 0 for all i 6= j. A is a M-matrix if A is nonsingular and
A1 0.

Definition (irreducible)
nn , if there exists a permutation matrix P, such that
For A R 
A11 A12
P T AP = , then A is reducible. Otherwise, A is irreducible/
0 A22

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 9 / 10

M-matrix
Definition (M-matrix)
A Rnn , aij 0 for all i 6= j. A is a M-matrix if A is nonsingular and
A1 0.

Definition (irreducible)
nn , if there exists a permutation matrix P, such that
For A R 
A11 A12
P T AP = , then A is reducible. Otherwise, A is irreducible/
0 A22

Definition (Diagonally Dominant)

Let A Rnn , if |aii |
P
i6=j |aij |, then A is diagonally dominant.

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 9 / 10

M-matrix

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 10 / 10

M-matrix
Definition (Strictly Diagonally Dominant)
X
|aii | > |aij |, for all i
i6=j

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 10 / 10

M-matrix
Definition (Strictly Diagonally Dominant)
X
|aii | > |aij |, for all i
i6=j

Definition (Irreducibly Diagonally Dominant)

A is diagonally dominant and ireducible. And
X
|aii | > |aij |, for at least one i
i6=j

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 10 / 10

M-matrix
Definition (Strictly Diagonally Dominant)
X
|aii | > |aij |, for all i
i6=j

Definition (Irreducibly Diagonally Dominant)

A is diagonally dominant and ireducible. And
X
|aii | > |aij |, for at least one i
i6=j

Theorem
If aii > 0 and aij 0, i 6= j. If A is SDD or IDD, then A is M-matrix.

X. Hu (Penn State) MATH/CMPSC 456 Feb. 13, 2012 10 / 10