Solving Linear Systems of Equations

Lecture8
Whats the big deal? ....cost
Look at 1D:
3 equations
3 unknowns
each unknown coupled to its neighbor
2x
1
x
2
= 5.8
x
1
2x
2
x
3
= 13.9
x
2
2x
3
= 0.03
or

2 1 0
1 2 1
0 1 2

x
1
x
2
x
3

5.8
13.9
0.03

Easy in 1d

2 1
.
.
.
1 2 1
.
.
.
1 2

n points in the grid

3 (n 2) + 2 + 2 or about 3n
nonzeros in the matrix
trididagonal (easy)
2D: harder

8 1 1
.
.
.
1 1 8 1 1
.
.
.
1 1 8

n points in one direction

n
2
points in grid
about 9 nonzeros in each row
about 9n
2
nonzeros in the matrix
n-banded (harder...we will see)
3D: hardest
n points in one direction
n
3
points in grid
about 27 nonzeros in each row
about 27n
3
nonzeros in the
matrix
n
2
-banded (yikes!)
Applications get harder and harder...
courtesy of LLNL courtesy of TrueGrid
courtesy of Rice courtesy of Warwick U.
Solving is a problem...
dim unknowns storage example
1D n 3n 10 100 1000
2D n
2
9n
2
10
2
10
4
10
6
3D n
2
27n
3
10
3
10
6
10
9
Moore...
Whats the problem?
humans: milliFLOPS
hand calculators: 10 FLOPS
desktops: a few GFLOPS (10
9
FLOPS)
look at the basic operations!
vec-vec, mat-vec, mat-mat
inner product of u and v both [n 1]
= u
T
v = u
1
v
1
+ + u
n
v
n
n multiplies, n 1 additions
O(n) ops
vec-vec, mat-vec, mat-mat
mat-vec of A ([n n]) and u ([n 1])
1 for i = 1, . . . , n
2 for j = 1, . . . , n
3 v(i) = a(i, j)u(j) + v(i)
4 end
5 end
n
2
multiplies, n
2
additions
O(n
2
) ops
vec-vec, mat-vec, mat-mat
mat-mat of A ([n n]) and B ([n n])
1 for j = 1, . . . , n
2 for i = 1, . . . , n
3 for k = 1, . . . , n
4 C(k, j) = A(k, i)B(i, j) + C(k, j)
5 end
6 end
7 end
n
3
multiplies, n
3
additions
O(n
3
) ops
vec-vec, mat-vec, mat-mat
Operation FLOPS
u
T
v O(n)
Au O(n
2
)
AB O(n
3
)
matlab test
four tests:
matrix-matrix multiply
inner product
matrix-vector multiply
n matrix-vector multiplies
order them...fastest to slowest
testop.m
Gaussian Elimination
Solving Diagonal Systems
Solving Triangular Systems
Gaussian Elimination Without Pivoting

Hand Calculations

Cartoon Version

The Algorithm
Gaussian Elimination with Pivoting

Row or Column Interchanges, or Both

Implementation
Solving Systems with the Backslash Operator
Solving Diagonal Systems
The system dened by
A =

1 0 0
0 3 0
0 0 5

b =

1
6
15

is equivalent to
x
1
= 1
3x
2
= 6
5x
3
= 15
The solution is
x
1
= 1 x
2
=
6
3
= 2 x
3
=
15
5
= 3
Solving Diagonal Systems
The system dened by
A =

1 0 0
0 3 0
0 0 5

b =

1
6
15

is equivalent to
x
1
= 1
3x
2
= 6
5x
3
= 15
The solution is
x
1
= 1 x
2
=
6
3
= 2 x
3
=
15
5
= 3
Solving Diagonal Systems
The system dened by
A =

1 0 0
0 3 0
0 0 5

b =

1
6
15

is equivalent to
x
1
= 1
3x
2
= 6
5x
3
= 15
The solution is
x
1
= 1 x
2
=
6
3
= 2 x
3
=
15
5
= 3
Solving Diagonal Systems (1)
Listing 1: Diagonal System Solution
1 given A, b
2 for i = 1 . . . n
3 x
i
= b
i
/a
i,i
4 end
In M:
1 >> A = ... % A is a diagonal matrix
2 >> b = ...
3 >> x = b./diag(A)
This is the only place where element-by-element division (.*) has anything to
do with solving linear systems of equations.
Operations?
Try...
Sketch out an operation count to solve a diagonal system of equations...
cheap!
one division n times O(n) FLOPS
Operations?
Try...
Sketch out an operation count to solve a diagonal system of equations...
cheap!
one division n times O(n) FLOPS
Triangular Systems (1)
The generic lower and upper triangular matrices are
L =

l
11
0 0
l
21
l
22
0
.
.
.
.
.
.
.
.
.
l
n1
l
nn

and
U =

u
11
u
12
u
1n
0 u
22
u
2n
.
.
.
.
.
.
.
.
.
0 u
nn

The triangular systems

Ly = b Ux = c
are easily solved by forward substitution and backward substitution,
respectively
Solving Triangular Systems (2)
A =

2 1 2
0 3 2
0 0 4

b =

9
1
8

is equivalent to
2x
1
+ x
2
+ 2x
3
= 9
3x
2
+ 2x
3
= 1
4x
3
= 8
Solve in backward order (last equation is solved rst)
x
3
=
8
4
= 2 x
2
=
1
3
(1 + 2x
3
) =
3
3
= 1
x
1
=
1
2
(9 x
2
2x
3
) =
4
2
= 2
Solving Triangular Systems (3)
A =

2 1 2
0 3 2
0 0 4

b =

9
1
8

is equivalent to
2x
1
+ x
2
+ 2x
3
= 9
3x
2
+ 2x
3
= 1
4x
3
= 8
Solve in backward order (last equation is solved rst)
x
3
=
8
4
= 2 x
2
=
1
3
(1 + 2x
3
) =
3
3
= 1
x
1
=
1
2
(9 x
2
2x
3
) =
4
2
= 2
Solving Triangular Systems (4)
A =

2 1 2
0 3 2
0 0 4

b =

9
1
8

is equivalent to
2x
1
+ x
2
+ 2x
3
= 9
3x
2
+ 2x
3
= 1
4x
3
= 8
Solve in backward order (last equation is solved rst)
x
3
=
8
4
= 2 x
2
=
1
3
(1 + 2x
3
) =
3
3
= 1
x
1
=
1
2
(9 x
2
2x
3
) =
4
2
= 2
Solving Triangular Systems (5)
Solving for x
1
, x
2
, . . . , x
n
for a lower triangular system is called forward
substitution.
1 given L, b
2 x
1
= b
1
/
11
3 for i = 2 . . . n
4 s = b
i
5 for j = 1 . . . i 1
6 s = s
i,j
x
j
7 end
8 x
i
= s/
i,i
9 end
Using forward or backward substitution is sometimes referred to as performing
a triangular solve.
Solving Triangular Systems (6)
Solving for x
1
, x
2
, . . . , x
n
for a lower triangular system is called forward
substitution.
1 given L, b
2 x
1
= b
1
/
11
3 for i = 2 . . . n
4 s = b
i
5 for j = 1 . . . i 1
6 s = s
i,j
x
j
7 end
8 x
i
= s/
i,i
9 end
Using forward or backward substitution is sometimes referred to as performing
a triangular solve.
Operations?
Try...
Sketch out an operation count to solve a triangular system of equations...
cheap!
begin in the bottom corner: 1 div
row -2: 1 mult, 1 add, 1 div, or 3 FLOPS
row -3: 2 mult, 2 add, 1 div, or 5 FLOPS
row -4: 3 mult, 3 add, 1 div, or 7 FLOPS
.
.
.
row -j: about 2j FLOPS
Total FLOPS?

n
j=1
2j = 2
n(n+1)
2
or O(n
2
) FLOPS
Operations?
Try...
Sketch out an operation count to solve a triangular system of equations...
cheap!
begin in the bottom corner: 1 div
row -2: 1 mult, 1 add, 1 div, or 3 FLOPS
row -3: 2 mult, 2 add, 1 div, or 5 FLOPS
row -4: 3 mult, 3 add, 1 div, or 7 FLOPS
.
.
.
row -j: about 2j FLOPS
Total FLOPS?
n
j=1
2j = 2
n(n+1)
2
or O(n
2
) FLOPS
Gaussian Elimination
Goal is to transform an arbitrary, square system into the equivalent upper
triangular system so that it may be easily solved with backward substitution.
The formal solution to Ax = b, where A is an n n matrix is
x = A
1
b
In M:
1 >> A = ...
2 >> b = ...
3 >> x = A\b
Gaussian Elimination Hand Calculations (1)
Solve
x
1
+ 3x
2
= 5
2x
1
+ 4x
2
= 6
Subtract 2 times the rst equation from the second equation
x
1
+ 3x
2
= 5
2x
2
= 4
This equation is now in triangular form, and can be solved by backward
substitution.
Gaussian Elimination Hand Calculations (2)
The elimination phase transforms the matrix and right hand side to an
equivalent system
x
1
+ 3x
2
= 5
2x
1
+ 4x
2
= 6

x
1
+ 3x
2
= 5
2x
2
= 4
The two systems have the same solution. The right hand system is upper
triangular.
Solve the second equation for x
2
x
2
=
4
2
= 2
Substitute the newly found value of x
2
into the rst equation and solve for x
1
.
x
1
= 5 (3)(2) = 1
Gaussian Elimination Hand Calculations (3)
When performing Gaussian Elimination by hand, we can avoid copying the x
i
by using a shorthand notation.
For example, to solve:
A =

3 2 1
6 6 7
3 4 4

b =

1
7
6

Form the augmented system

A = [A b] =

3 2 1
6 6 7
3 4 4

1
7
6

The vertical bar inside the augmented matrix is just a reminder that the last
column is the b vector.
Gaussian Elimination Hand Calculations (4)
Add 2 times row 1 to row 2, and add (1 times) row 1 to row 3

A
(1)
=

3 2 1
0 2 5
0 2 3

1
9
7

Subtract (1 times) row 2 from row 3

A
(2)
=

3 2 1
0 2 5
0 0 2

1
9
2

Gaussian Elimination Hand Calculations (5)

The transformed system is now in upper triangular form

A
(2)
=

3 2 1
0 2 5
0 0 2

1
9
2

Solve by back substitution to get

x
3
=
2
2
= 1
x
2
=
1
2
(9 5x
3
) = 2
x
1
=
1
3
(1 2x
2
+ x
3
) = 2
Gaussian Elimination Cartoon Version (1)
Start with the augmented system

x x x x x
x x x x x
x x x x x
x x x x x

The xs represent numbers, they are not necessarily the same values.
Begin elimination using the rst row as the pivot row and the rst element of
the rst row as the pivot element

x x x x x
x x x x x
x x x x x
x x x x x

Gaussian Elimination Cartoon Version (2)

Eliminate elements under the pivot element in the rst column. x

indicates a
value that has been changed once.

x x x x x
x x x x x
x x x x x
x x x x x

x x x x x
0 x

x x x x x
x x x x x

x x x x x
0 x

0 x

x x x x x

x x x x x
0 x

0 x

0 x

Gaussian Elimination Cartoon Version (3)

The pivot element is now the diagonal element in the second row. Eliminate
elements under the pivot element in the second column. x

indicates a value
that has been changed twice.

x x x x x
0 x

0 x

0 x

x x x x x
0 x

0 0 x

0 x

x x x x x
0 x

0 0 x

0 0 x

Gaussian Elimination Cartoon Version (4)

The pivot element is now the diagonal element in the third row. Eliminate
elements under the pivot element in the third column. x

indicates a value
that has been changed three times.

x x x x x
0 x

0 0 x

0 0 x

x x x x x
0 x

0 0 x

0 0 0 x

Gaussian Elimination Cartoon Version (5)

Summary
Gaussian Elimination is an orderly process of transforming an augmented
matrix into an equivalent upper triangular form.
The elimination operation is
a
kj
= a
kj
( a
ki
/ a
ii
) a
ij
Elimination requires three nested loops.
The result of the elimination phase is represented by the image below.

x x x x x
x x x x x
x x x x x
x x x x x

x x x x x
0 x

0 0 x

0 0 0 x

Gaussian Elimination Algorithm

Listing 2: Gaussian Elimination
1 form

A = [Ab]
2
3 for i = 1 . . . n 1
4 for k = i + 1 . . . n
5 for j = i . . . n + 1
6 a
kj
= a
kj
( a
ki
/ a
ii
) a
ij
7 end
8 end
9 end
GEshow
The GEshow function in the NMM toolbox uses Gaussian elimination to solve a
system of equations. GEshow is intended for demonstration purposes only.
The Need for Pivoting (1)
Solve:
A =

2 4 2 2
1 2 4 3
3 3 8 2
1 1 6 3

b =

4
5
7
7

Note that there is nothing wrong with this system. A is full rank. The solution
exists and is unique.
Form the augmented system.

2 4 2 2
1 2 4 3
3 3 8 2
1 1 6 3

4
5
7
7

The Need for Pivoting (2)

Subtract 1/2 times the rst row from the second row,
add 3/2 times the rst row to the third row,
add 1/2 times the rst row to the fourth row.
The result of these operations is:

2 4 2 2
0 0 5 2
0 3 5 5
0 3 5 4

4
7
1
5

The next stage of Gaussian elimination will not work because there is a zero
in the pivot location, a
22
.
The Need for Pivoting (3)
Swap second and fourth rows of the augmented matrix.

2 4 2 2
0 3 5 4
0 3 5 5
0 0 5 2

4
5
1
7

Continue with elimination: subtract (1 times) row 2 from row 3.

2 4 2 2
0 3 5 4
0 0 0 1
0 0 5 2

4
5
4
7

The Need for Pivoting (4)

Another zero has appear in the pivot position. Swap row 3 and row 4.

2 4 2 2
0 3 5 4
0 0 5 2
0 0 0 1

4
5
7
4

The augmented system is now ready for backward substitution.

Pivoting Strategies
Partial Pivoting: Exchange only rows
Exchanging rows does not affect the order of the x
i
For increased numerical stability, make sure the largest possible pivot
element is used. This requires searching in the partial column below the
pivot element.
Partial pivoting is usually sufcient.
Partial Pivoting
To avoid division by zero, swap the row having the zero pivot with one of the
rows below it.
0
*
Rows completed in
forward elimination.
Rows to search for a
more favorable pivot
element.
Row with zero pivot element
To minimize the effect of roundoff, always choose the row that puts the largest
pivot element on the diagonal, i.e., nd i
p
such that |a
i
p
,i
| = max(|a
k,i
|) for
k = i, . . . , n
Pivoting Strategies (2)
Full (or Complete) Pivoting: Exchange both rows and columns
Column exchange requires changing the order of the x
i
For increased numerical stability, make sure the largest possible pivot
element is used. This requires searching in the pivot row, and in all rows
below the pivot row, starting the pivot column.
Full pivoting is less susceptible to roundoff, but the increase in stability
comes at a cost of more complex programming (not a problem if you use
a library routine) and an increase in work associated with searching and
data movement.
Full Pivoting
0
*
Rows completed in
forward elimination.
Columns to search for a more
favorable pivot element.
Row with zero pivot element
Rows to search for a
more favorable pivot
element.
*
Gauss Elimination with Partial Pivoting
Listing 3: Gaussian Elimination with Partial Pivoting
1 form

A = [Ab]
2 for i = 1 . . . n 1
3 find i
p
such that
4 max(| a
i
p
i
|) max(| a
ki
|) for k = i . . . n
5 exchange row i
p
with row i
6 for k = i + 1 . . . n
7 for j = i . . . n + 1
8 a
k,j
= a
k,j
( a
k,i
/ a
i,i
) a
i,j
9 end
10 end
11 end
Gauss Elimination with Partial Pivoting
GEshow
The GEshow function in the NMM toolbox uses naive Gaussian elimination
without pivoting to solve a system of equations.
GEpivshow
The GEpivshow function in the NMM toolbox uses Gaussian elimination with
partial pivoting to solve a system of equations. GEpivshow is intended for
demonstration purposes only.
GEshow and GEpivshow are for demonstration purposes only. They are also a
convenient way to check your hand calculations.
The Backslash Operator (1)
Consider the scalar equation
5x = 20 = x = (5)
1
20
The extension to a system of equations is, of course
Ax = b = x = A
1
b
where A
1
b is the formal solution to Ax = b
In M notation the system is solved with
1 x = A\b
The Backslash Operator (2)
Given an n n matrix A, and an n 1 vector b the \ operator performs a
sequence of tests on the A matrix. M attempts to solve the system with
the method that gives the least roundoff and the fewest operations.
When A is an n n matrix:
1
M examines A to see if it is a permutation of a triangular system
If so, the appropriate triangular solve is used.
2
M examines A to see if it appears to be symmetric and positive
denite.
If so, M attempts a Cholesky factorization
and two triangular solves.
3
If the Cholesky factorization fails, or if A does not appear to be
symmetric,
M attempts an LU factorization
and two triangular solves.
Limits on Numerical Solution to Ax = b
Machine Limitations
RAM requirements grow as O(n
2
)
op count grows as O(n
3
)
The time for data movement is an important speed bottleneck on modern
systems
An architecture observation...
Disk
FPU CPU
internal bus
cache
system bus
RAM
network
Computer
Data Access Time:
shortest
longest
Computer Architecture Affecting the Speed of Data Access
(a highIy simpIified view)
Stand alone computer
Computer Computer
Operation FLOPS
u
T
v O(n)
Au O(n
2
)
AB O(n
3
)
quiz...
order each in terms of speed
order each in terms of efciency