Numerical Analysis-II

Math-604
BS Part-IV Mathematics
Solution of Linear Equations
and
Matrix Inversion

Solution of Linear Equations

1. Direct Methods
Exact solution in a finite number of steps.

2. Indirect (Iterative)
methods
A sequence of approximate solutions
hopefully converging to the exact solution.

Solution of Linear Equations

1. Direct Methods

LU decomposition
Crouts method
Cholesks method

2. Indirect (Iterative) methods

Gauss-Jacobi method
Gauss-Seidel method
Successive Over Relaxation (SOR)

LU Decomposition
Method

Introduction
LU Factorization
LU Decomposition how it works
LU Decomposition method with
examples
Crouts method with examples
Cholesks method with examples
4

Inverse of a matrix
Methods
LU Decomposition method
Cholesks method

LU Decomposition
Method
A=LU

Introduction
In numerical analysis, another way of solving a
system of equations is by using a factorization
technique for matrices called LU decomposition.
This factorization involves two matrices, one lower
triangular matrix and one upper triangular matrix.
An LU decomposition of a matrix A is the product
of a lower triangular matrix and an upper
triangular matrix that is equal
to A.
The LU decomposition was introduced by
mathematician Alan Turing in 1948.

Introduction (Contd)
LU decomposition was originally derived by Lagrange, in the
very first paper in his collected works (1759) derives the
algorithm, called
Gaussian elimination. Later Turing
introduced the LU decomposition of a matrix in 1948 that is
used to solve the system of linear equation.

J-L Lagrange
(1736 1813)

A. M. Turing
(1912-1954)

LU Factorization
A system of equations AX = B
is decomposed into two factors
L and U. Where L is a lower
triangular matrix and U is an
0 triangular
0
u11 u12 u13
upper
matrix.
11

L 21
31

22
32

0 and U 0
0
33

u22
0

u23
u33

LU Factorization (Contd)
Such a factorization is unique if the diagonal elements of L or U are unity. If
diagonal elements of L are unity, the resulting scheme has two methods:

(i) If a matrix A is not symmetric and positive definite then this scheme is
called LU Decomposition method.
(ii) If a matrix A is symmetric positive definite matrix, i.e., A=AT and det(A) >0
and determinant of a minor matrix is also positive then this scheme is
called Cholesky method. In this method decomposition is defined as
A=L LT
(iii) If we choose the diagonal elements of U as unity, the resulting scheme is
called Crouts method or Crouts reduction.

10

LU Factorization
LU factorization involves
two steps:
Factorization to decompose
the [A] matrix into a
product of a lower
triangular matrix [L] and an
upper triangular matrix [U].
[L] has 1 for each entry on
the diagonal.
Substitution to solve for
{x}

Gauss elimination can be

implemented using LU
factorization

Summary of LU
Decomposition

Methods for solving system

of equations
1. Decomposition Method
2. Crouts Method or Reduction Method
3. Cholesky Method

13

1. LU Decomposition
This method Method
can be used to solve system of
linear equations if a matrix A is not symmetric
and positive definite. In this case the diagonal
elements of a lower triangular matrix L are
taken as unity and decomposition is defined as
A LU
a11
a
21
a31

a12
a22
32

a13 1
a23 21
a33 31

0
1
32

0
0
1

u11
0

u12
u 22
0

u13
u 23
u33

14

Algorithm
Solve A X = B
Step 1. Decompose A into L and U
A= LU
Step 2. Solve L Y = B for Y
(Forward Substitution)
Step 3. Solve U X = Y for X
(Backward Substitution)

15

Example 1. Using Decomposition method to find

L and U matrix.
1
A 3
2

2
8
6

4
14
13

Where
1

L 21
31

0
1
32

0
1

u11

and U 0
0

u12
u22
0

u13

u23
u33
16

Multiplying LU and setting the answer

equal to A.
LU A
1

21
31
u11
l u
21 11
l31u11

0
1
32

0 u11

0 0
1 0

u12
l21u12 u 22
l31u12 l32u 22

u12
u22
0

u13 1

u23 3
u33 2

2 4

8 14
6 13

u13
1
l21u13 u 22u 23 3
l31u13 l32u23 u33 2

2 4
8 14
6 13
17

We begin by running along the top row to

find the entries in L and U.
u 1,
11
l u 3,
21 11

u 2, and u 4.
12
13
l 1 3, l 3.
21
21

l u u 8,
21 12 22
l u u 14,
21 13 23

3 2 u 8, u 2.
22
22
3 4 u 14, u 2.
23
23

This pattern continues on the last

row
l u 2,
31 11

l u l u 6,
31 12 32 22
l u l u u 13,
31 13 32 23 33

l 1 2, l 2.
31
31

2 2 l 2 6, l 1.
32
32
(2 4) (1 2)u 13, u 3.
33
33
18

We have Shown that

1

1
1
0
0

8 14 3 1 0 0

6 13 2 1 1 0

2
2
0

19

Example 2. Use
Decomposition method to solve
the system of equations
x1 3x2 2 x3 19

2 x1 x2 x3 13
4 x1 2 x2 2 x3 31

Solution: A=LU
1

21
31

0
1
32

0
0
1

u11
0

LU A
u12
u22
0

u13 1
u23 2
u33 4

3 2
1 1
2 3
20

Multiplying LU and setting the answer

equal to A.
u11
l u
21 11
l31u11

u12

u13

l21u12 u 22
l31u12 l32u22

l21u13 u 23
l31u13 l32u 23 u33

2
4

3 2

1 1
2 3

We begin by running along the top row to find the

entries in L and U.
u 1,
11

u 3, and u 2.
12
13

l u 2,
21 11

l 1 2, l 2.
21
21

l u u 1,
21 12 22

23 u 8, u 2.
22
22

l u u 1,
21 13 23

2 2 u 1, u 3.
23
23

21

This pattern continues on the last

l u 4, l row
1 4, l 4.
31 11

31

l u l u 2,
31 12 32 22
l u l u u 3,
31 13 32 23 33

31

43 l (5) 2, l 2.
32
32
(4 2) (2)(3) u 3, u 1.
33
33

We have Found that

A LU

3 2

1 0 0 1

1 1 2 1 0 0

2 3

4
2
1

22

STEP 2: Solve for Y

Set [L][Y] = [B]

Solve for [Y]

1
2

0
1
2

0
0
1

y1 19
y 13
2
y3 31

y1 19

(i )

2 y1 y2 13

(ii )

4 y1 2 y2 y3 31

(iii )

23

STEP 2: Solve for Y

(Contd..)
Complete the forward substitution to solve for [Y]

From (i )

y1 19

From (ii ) y2 13 2(19) 25

From (iii ) y3 31 4(19) 2(25) 5

y1 19
Y y2 25
y3 5

24

STEP 3: Solve for X

Set [U][X] = [Y]

Solve for [X]

1
0

3
5
0

2
3
1

x1 19
x 25
2

x3 5

x1 3x2 2 x3 19 (i )
5 x2 3x3 25 (ii )
x3 5

(iii )

25

STEP 3: Solve for X

(Contd..)
Complete the forward substitution to solve for [Y]

From (iii )

x3 5

From (ii ) 5 x2 15 25 x2 2
From (i ) x1 3(2) 2(5) 19 x1 3

x1
X x2
x3

3
2
5

Which is required solution .

26

2. Crouts Method
This method can be used to solve
system of equations if a matrix A is not
symmetric and positive definite. In this
case the diagonal elements of a upper
triangular matrix U are taken as unity
and decomposition is defined as
A LU
a11
a
21
a31

a12
a22
32

a13 11

a23 21
a33 31

0
22
32

0
33

1 u12
0 1

u13

u23
1

27

Algorithm
Solve A X = B
Step 1. Decompose A into L and U
A= LU
Step 2. Solve L Y = B for Y
(Forward Substitution)
Step 3. Solve U X = Y for X
(Backward Substitution)

28

Example 1. Use Crouts

method to solve the system of
equations
x1 2 x2 3 x3 16

3 x1 5 x2 8 x3 43
4 x1 9 x2 10 x3 57

Solution: A=LU
11

21
31

0
22
32

0
0
33

u11
0

LU A
u12
u22
0

u13 1
u23 3
u33 4

2 3
5 8
9 10
29

Multiplying LU and setting the answer

equal to A.
l11u11
l u
21 11
l31u11

l11u12
l21u12 l22u 22
l31u12 l32u 22

l11u13
l21u13 l22u23
l31u13 l32u 23 l33u33

3
4

9 10

For convenience, we put u u u 1

11 22 33

l11
l
21
l31

l11u12
l21u12 l22
l31u12 l32

l11u13
1
l21u13 l22u 23 3
l31u13 l32u 23 l33 4

2 3
5 8
9 10
30

STEP 1: Find the entries in L and U.

l 1,
11

l u 2 or u 2 and l u 3 or u 3
11 12
12
11 13
13

l 3,
21

l u l 5 or (3)(2) l 5 or l 1
21 12 22
22
22

l u l u 8 or (3)(3) (1)u 8 or u 1
21 13 22 23
23
23
l 4,
31

l u l 9 or (4)(2) l 9 or l 1
31 12 32
32
32

l u l u l 10 or (4)(3) (1)(1) l 10 or l 3
31 13 32 23 33
33
33

A 3

0 1

1 0 0

1 3 0

2
1
0

1
31

STEP 2: Solve for Y

Set [L][Y] = [B]

Solve for [Y]

1
3

5.76

0
1
1

0
0
3

y1 16
y 43
2
y3 57

y1 16

(i )

3 y1 y2 43

(ii )

4 y1 y2 3 y3 57

(iii )

32

STEP 2: Solve for Y

(Contd..)
Complete the forward substitution to solve for [Y]

y1 16
y2 3(16) 43 5
3 y3 4(16) (5) 57 12 or y3 4

y1 16
Y y2 5
y3 4

33

STEP 3: Solve for X

Set [U][X] = [Y]

Solve for [X]

1
0

2 3 x1 16
1 1 x2 5
0 1 x3 4

The 3 equations become

x1 2 x2 3x3 16 (i )
x2 x3 5 (ii )
x3 4 (iii )
34

STEP 3: Solve for X

(Contd..)
Complete the forward substitution to solve for [X]

x3 4
x2 5 4 1
x1 2(1) 3(4) 16 or x1 2

x1
X x2
x3

2
1
4

It is a required
solution.
35

3. Cholesky Method
This method can be used to solve the
system of equations if a matrix A is
symmetric and positive definite. In
this method, the diagonal elements of
a lower triangular matrix L are to be
taken not as unity and decomposition
T
T.
A

LL
is defined as A=LL
a11
a
21
a31

a12
a22
a32

a13
11
a23 21
31
a33

0
22
32

0
0
33

11
0

21
22
0

31
32
33
36

Algorithm
Solve A X = B
Step 1. Decompose A into L and U, where U= LT
A=L LT

Step 2. Solve L Y = B for Y

(Forward Substitution)
Step 3. Solve LT X = Y for X
(Backward Substitution)

37

Example 1. Use Choleskys

method to solve the system of
equations
9 x1 6 x2 3 x3 30
6 x1 5 x2 8 x3 19
3 x1 x2 4 x3 57

x1
9 6 3
30

Here A 6 5 1 , X x2 and B 19
x3
3 1 4
17

38

Example: Choleskys method

(Contd..)
We see that
(i)Matrix is symmetric (ii) Det of
A=18
(iii) Det of minors is also positive.
Thus conditions of Choleskys
method are satisfied. Now we
can apply this method to find the
solution of the given equations.
39

Step 1.
9 6 3 11

T
A LL 6 5 1 21
3 1 4 31

9
6
3

6 3 l112

5 1 l21l11
1 4 l31l11

0
22
32

l11l21
2
2
l21
l22
l31l21 l32l22

0
33

11
0

21
22
0

31

32
33

l11l31

l21l31 l22l32
l312 l322 l332

40

Step 1 (Contd..)
2
11 9 or 11 3

2
2
1121 6 or 21 2 21 22 5 or 22 1

3111 3 or 31 1

3121 32 22 1 or 32 1

2
2
2
31 32 33 4 or 33 4 1 1 2

41

Step 2. Solve for Y

By taking
LY=B.

3
2

0 y1
0 y 2
2 y3

0
1
1

30
19
17

3 y1 30

y1 10

2 y1 y 2 19

y 2 1

y1 y 2

2 y3 17 y3 3

42

Step 3. Solve for X.

For finding X
consider LT X=Y

By backward
substitution,
we get

3
0

1 x1
10
1
1 x2 1
3 2
0
2 x3
3 x1 2 x2 x3 10 (i )
2

x2 x3 1

(ii )

2 x3 3 2

(iii )

From (iii ) x3 3
From (ii )

x2 3 1 or x2 2

From (i ) 3 x1 2( 2) 3 10 or x1 1
Thus x1 1, x2 2 and x3 3 .

It is a required
solution.

43

Short Cut Method to find

L and U
Algorithm
Solve A X = B
Step 1. Using elementary row operations to
convert a
matrix A in upper
triangular matrix U.
Step 2. For Lower triangular matrix L taking
negative
of the times row which is
a 31
a 32
a 21
Where

subtracted from the

row
for
making
it
21
31
32
a 11
a 11
a 22
zero. Diagonals of L are to be taken 1.
Step 3. Solve LY = B for Y
(Forward Substitution)
Step 4. Solve U X = Y for X
44

Example 1. Use Short cut method to find U

matrix and L matrix
2 1

4 2
6 1

2
U 0
0

3
5 R2 2 R1 and L
2

1
4
4

5
11

3
1
2

l
21
l31

1
R3 3R1 and L l21
l31

0
1
l32
0
1
l32

0
0

l21 ve of 2 2

0
0

l31 ve of 3 3

1
45

Example 1 (Contd..)
1
4

2
U 0
0

3
5 R3 R2 and L
6

1
l
21

0
1

l31

l32

0
0 l32 ve of 1 1
1

Thus we have found L

and U
2

4
6

2
1

1 2
3
2

0
1
1

0
0
1

2
0

3
4 5
0
6
46

Example 2. Use Short cut method to find U

matrix and L matrix
Using the Forward Elimination Procedure of Gauss Elimination to
convert U in upper triangular matrix

25 5 1
64 8 1

144 12 1

5
1
25
64
Step 1:
2.56; Row2 Row1 2.56 0 4.8 1.56
25
144 12
1
5
1
25
144
5.76; Row3 Row1 5.76 0 4.8 1.56
25
0 16.8 4.76
47

Finding the [U] Matrix

Matrix after Step
1:

5
1
25
0 4.8 1.56

0 16.8 4.76

1
25 5
16.8
3.5; Row3 Row2 3.5 0 4.8 1.56
Step 2:
4.8
0
0
0.7

1
25 5
U 0 4.8 1.56
0
0
0.7
48

Now Finding the L matrix

1

21
31

0
1
32

0
0

Using the multipliers used during the Forward Elimination Procedure

a
64
21 21
2.56
5 1
From the first step 25
a11 25
of forward
elimination

64 8 1

144 12 1

31

a31 144

5.76
a11
25

49

Finding the [L] Matrix

From the second
step of forward
elimination

5
1
25
0 4.8 1.56

0 16.8 4.76

L 2.56

32

a32 16.8

3.5
a 22
4.8

1 0
5.76 3.5 1
50

Does [L][U] = [A]?

0 25 5
1
LU 2.56 1 0 0 4.8 1.56
5.76 3.5 1 0
0
0.7
1

Yes its product is equal

to A.
51

Example 3. Using LU
Decomposition method to solve
System of linear equations
25 5 1 x1 106.8
64 8 1 x 177.2

144 12 1 x3 279.2

Solve the following set of

linear equations using LU
Decomposition.

Using the procedure for finding the [L] and [U] matrices

0 25
5
1
A LU 2.56 1 0 0 4.8 1.56
5.76 3.5 1 0
0
0.7
1

52

Example 3: Solve for Z

Set [L][Z] = [B]

Solve for [Z]

1
0 0 z1
106.8
2.56 1 0 z 2 177.2
279.2
5.76 3.5 1 z 3

z1 10
2.56 z1 z 2 177.2
5.76 z1 3.5 z 2 z 3 279.2

53

Example: Solve for Z

(Contd..)
Complete the forward substitution to solve for [Z]

z1 106.8
z 2 177.2 2.56 z1

177.2 2.56106.8

96.2
z3 279.2 5.76 z1 3.5 z 2

279.2 5.76106.8 3.5 96.21

z1 106.8
Z z2 96.21
z3 0.735

0.735

54

Example 3: Solve for X

Set [U][X] = [Z]

Solve for [X]

1
25 5
0 4.8 1.56

0
0
0.7

x1 106.8
x 96.21
2

x3 0.735

The 3 equations become

25 x1 5 x2 x3 106.8
4.8 x2 1.56 x3 96.21
0.7 x3 0.735
55

Example 2 (Contd..)
From the 3 equation
rd

Substituting in x3 and using the

second equation

0.7 x3 0.735

4.8 x2 1.56 x3 96.21

0.735
x3
0.7
x3 1.050

96.21 1.56 x3
x2
4.8
96.21 1.561.050
x2
4.8
x2 19.70

56

Example 3: Solve for X

(Contd..)
Substituting in x3 and x2 using
the first equation

25 x1 5 x2 x3 106.8
106.8 5 x2 x3
25
106.8 519.70 1.050

25
0.2900

x1

Hence the Solution Vector is:

x1 0.2900
x 19.70
2

x3 1.050

It is a required
solution.
57

Inverse of a matrix
Methods

Inverse of
Inverse of
Inverse of
Inverse of
method
Inverse of

a
a
a
a

matrix
matrix
matrix
matrix

by
by
by
by

Adjoint method
Matrix method
Jordan method
LU Decomposition

a matrix by Cholesks method

58