Chapter 5

METHODS TO SOLVE A
SYSTEM OF LINEAR
EQUATIONS
Introduction
Engineers frequently encounter problems involving the solution of sets of simultaneous algebric equations. Very frequently the equations are linear, so here we will
consider methods applicable for solving sets of linear equations.
Problems involving simultaneous linear equations arise in the areas of elasticity,
electric-circuit analysis, heat transfer, vibrations, and so forth. Also, the numerical integration of some types of ordinary and partial differential equations may be
reduced to the solution of such a set of equations.
To solve the sets of linear equations, the basic concepts and notation of matrices
are required. A matrix is a rectangular array of numbers in which not only the
value of the number is important but also its position in the array. The number of
its rows and columns describes the size of the matrix. A matrix of n rows and m
columns is said to be n m.

A=

a11
a21
a31

an1

a12 a13
a22 a23
a32 a33

an2 an3

a1m
a2m
a3m
anm

= [aij ],

i = 1, 2, 3, , n, j = 1, 2, 3, , m

Two matrices of the same size may be added or subtracted. The sum of A =
[aij ] and B = [bij ] is the matrix whose elements are the sum of the corresponding
elements of A and B:
C = A B = [aij bij ] = [cij ].
73

If

4 2
1 5

A = 4 2.5 and B = 2 6
3 0.0
4 3
Here A is 3 2 because it has three rows and two columns. B is also 3 2. Their
sum C is also 3 2:

5
3

C = A + B = 2 8.5
7
3
The difference D of A and B is

3 7

D = A B = 6 3.5
1 3
Multiplication of two matrices is defined when the number of columns of first matrix
is equal to the number of rows of second matrix i.e., when A is nm and B is mr :

[aij ][bij ] = [cij ] =

cij =

(a11 b11
(a21 b11
(a31 b11

(an1 b11
m
X

+
+
+

aik bkj ,

+ a1m bm1 ) (a11 b1r + + a1m bmr )

+ a2m bm1 ) (a21 b1r + + a2m bmr )
+ a3m bm1 ) (a31 b1r + + a3m bmr )
+ anm bm1 ) (an1 b1r + + anm bmr )

i = 1, 2, 3, , n, j = 1, 2, 3, , r.

k=1

If A is nm, B must have m rows or else they are said to be nonconformable

for multiplication and their product is undefined. In general, AB 6= BA, so the
order of factors must be preserved in matrix multiplication.
If k is a scalar then
If

kA = C,

cij = kaij

Matrix with only one column n 1 in size, is termed as a column vector, and one
of only one row, 1 m in size, is called a row vector. Normally, term vector is
used for a column vector. If A is n n, it is called a square matrix.
Suppose

3 7 1
3 7 3 2
6
3 2

A = 6 3 2 1 , B =

1 3 5
1 3 5 4
6
3 2
74

Then

60 45 8

A B = 32
48 12
;
34 5 26
and

BA=

50 45
0
3
38
45 2 1

;
10 17 16 15
2 39 34 23

Since A is 3 4 and B is 4 3, they are conformable for multiplication and their

product is 3 3. When we form the product of B A, it is 4 4. Observe that not
only is AB 6= BA; AB and BA are not even the same size.
The definition of matrix multiplication permits to write the set of n linear equations in m unknowns
a11 x1 + a12 x2 + a13 x3 + + a1m xm
a21 x1 + a22 x2 + a23 x3 + + a2m xm
a31 x1 + a32 x2 + a33 x3 + + a3m xm

an1 x1 + an2 x2 + an3 x3 + + anm xm

=
=
=

b1
b2
b3

= bn

(5.1)

much more simply in matrix notation, as

Ax = b,
where

A=

a11 a12 a13 a1m

a21 a22 a23 a2m
a31 a32 a33 a3m
..
.
an1 an2 an3 anm

x=

x1
x2
x3
..
.
xm

b=

b1
b2
b3
..
.
bm

A very important special case is the multiplication of two vectors, e.g.,

[1 3 2] 2
= [16]
3
gives a matrix of one row and one column. The result is a pure number, a scalar.
This product is called the scalar product of the vectors, also called the inner
product.
75

Reverse the order of multiplication of these two vectors, we get

4
4 12 8

2
6 4

[1
3
2]
=

2
.
3
3 9 6
This product is called the outer product, very important in nonlinear optimization
problems.
If all the elements above the diagonal are zero, a matrix is called lowertriangular; it is called upper-triangular when all the elements below the diagonal
are zero. For example,

4 0 0

L = 2 6 0 ,
3 9 6

4 12 8

U = 0 6 0 .
0 0 6

The diagonal elements are the line of elements aii from upper left to lower right of
the matrix. If only the diagonal terms are nonzero, the matrix is called a diagonal
matrix. When the diagonal elements are each equal to unity while all off-diagonal
elements are zero, the matrix is said to be the identity matrix of order n e.g.,

1
0
0
0

0
1
0
0

0
0
1
0

0
0
0
1

= I4

Tridiagonal matrices are those that have nonzero elements only on the diagonal and in the positions adjacent to the diagonal; they will be of special importance
in certain partial differential equations. An example is

4
2
0
0
0

2
6
9
0
0

0
3
6
1
0

0
0
1
3
2

0
0
0
1
4

When a matrix is square, a quantity called its trace is defined, which is the sum of
the elements on the main diagonal. Trace of the above matrix is
tr(A) = 4 + 6 + 6 + 3 + 4 = 23
All the elements of the null matrix are zero:

A=

0
0
0
0

0
0
0
0
76

0
0
0
0

0
0
0
0

For a matrix defined by A = [aij ], its transpose is defined by AT = [aji ], e.g.,

A=

2
3 4
1 3 2
3
3 4
6
1 6

2 1 3 6

then AT = 3 3 3 1
4
2 4 6

The inverse of a matrix A is written as A1 and satisfies AA1 = A1 A = I.

A matrix that has orthonormal columns is called an orthogonal matrix. It
satisfies
QT Q = I, QQT = I, QT = Q1
A vector whose lenght is one is called a unit vector. A vector that has all its
elements equal to zero except one element, which has a value of unity, is called a
unit basis vector. There are three distinct unit basis vectors for order-3 vectors:

u = 0 ,
0

v = 1 ,
0

w = 0 .
1

Null vectors are defined as the vectors, with all the elements zero:

u = 0 .
0
Transpose vector of a vector

x1

u = x2
x3
is given by
uT = [x1

x2

Determinant
minor

5.1

Sigma Notation
n
X

xi = x1 + x2 + x3 + + xn

i=1

77

x3 ]

n
X

c = c

i=1
1
X

n
X

1 == c(1 + 1 + 1 + + 1) = cn

i=1

xi = x1

i=1
n
X

xi =

i=1

n
X

n
X

xj =

j=1

xk .

i, j, k are dummy index

k=1

System (??) can be written as

m
X

a1j xj =

b1

j=1
m
X

a2j xj =

b2

j=1
m
X

a3j xj =

j=1

m
X
j=1

5.2

b3

anj xj =

bn

m
X

aij xj = bi

i = 1, 2, 3, , n

j=1

Product Notaion
n
Y

x1 = x1 x2 x3 xn

i=1

5.3

The Gaussian Elimination Method

The first method usually presented in algebra for the solution of the set of equations is one in which the unknowns are eliminated by combining equations. Such a
method is known as an elimination method. It is called Gaussian elimination
if a particular systematic scheme, attributed to Gauss, is used in the elimination
process. This method is the basic pattern of a large number of methods that are
classified as direct methods.
Using Gausss method, a set of n equations in n unknowns is reduced to an
equivalent triangular set (an equivalent set is a set having identical solution values),
which is then easily solved by back substitution.
Consider the general form of the set of n equations in n unknowns

a11 x1 + a12 x2 + a13 x3 + + a1n xn = b1

78

(a)

a21 x1 + a22 x2 + a23 x3 + + a2n xn = b2

a31 x1 + a32 x2 + a33 x3 + + a3n xn = b3
..
..
.
.
an1 x1 + an2 x2 + an3 x3 + + ann xn = bn

(b)
(c)

(5.2)

If the bs are not all zero, the set of equations is nonhomogeneous, and all
equations must be independent to obtain unique solutions. If the bs are all zero,
the set of equations is homogeneous, then nontrivial solutions exist only if all the
equations are not independent.
The Gaussian elimination method is accomplished in the following manner:
1. Equation (??a) is divided by the coefficient of x1 in that equation to obtain
x1 +

a12
a13
a1n
b1
x2 +
x3 + +
xn =
a11
a11
a11
a11

(5.3)

Equation (??) is next multiplied by the coefficient of x1 in equation (??b),

and the resulting equation is subtracted from (??b), thus eliminating x1 from
equation (??b). Equation (??), is then multiplied by the coefficient of x1 in
equation (??c), and the resulting equation is subtracted from equation (??c)
to eliminate x1 from equation (??c). In a similar manner, x1 is eliminated
from all equations of the set except the first, so that the set assumes the form
a11 x1 + a12 x2 + a13 x3 + + a1n xn
a022 x2 + a023 x3 + + a02n xn
a032 x2 + a033 x3 + + a03n xn
..
.
0
0
0
an2 x2 + an3 x3 + + ann xn

= b1
= b02
= b03
..
.
= b0n

(a)
(b)
(c)

(5.4)

where the prime superscripts indicate the new coefficients which are formed
in the reduction process.
The equation used to eliminate the unknowns in the equations which follow it
is called the pivot equation(eq. (??a) in the preceding steps). In the pivot
equation the coefficient of the unknown, which is to be eliminated from the
subsequent equations, is known as the pivot coefficient(a11 in the preceding
steps).
2. Following the above steps, (??b) becomes the pivot equation, and the steps of
part 1 are repeated to eliminate x2 from all the equations following this pivot
79

equation. This elimination yields

a11 x1 + a12 x2 + a13 x3 + + a1n xn
a022 x2 + a023 x3 + + a02n xn
a0033 x3 + + a003n xn
..
.
00
00
an3 x3 + + ann xn

= b1
= b02
= b003
..
.
= b00n

(a)
(b)
(c)

(5.5)

3. Equation (??c) is next used as the pivot equation, and the procedure described
is used to eliminate x3 from all the equations following this pivot equation
(??c). This procedure, using successive pivot equations, is continued until the
original set of equations has been reduced to a triangular set
a11 x1 + a12 x2 + a13 x3 + + a1n1 xn1 + a1n xn
a022 x2 + a023 x3 + + a02n1 xn1 + a02n xn
a0033 x3 + + a003n1 xn1 + a003n xn
..
.
(n2)
(n2)
an1n1 x3 + an1n xn
a(n1)
xn
nn

= b1
(a)
= b02
(b)
00
= b3
(c)
..
.
(n2)
= bn1
= b(n1)
n

(5.6)

4. After the triangular set of equations has been obtained, the last equation in
this equivalent set yields the value of xn directly. This value is then substituted
into the next -to-the-last equation of the triangular set to obtain a value of
xn1 , which is, in turn, used along with the value of xn in the second-to-the-last
equation to obtain a value of xn2 , and so on. This is the back-substitution
procedure.
To illustrate the method, let us apply these procedurs to solve the following set
of equations:

2x1 + 8x2 + 2x3 = 14

x1 + 6x2 x3 = 13

2x1 x2 + 2x3 = 5
Using the first equation as the pivot
obtain
2x1 + 8x2 +
2x2
9x2

(5.7)

equation (the pivot coefficient is 2), we

80

2x3
= 14

2x3
=
6

(0)x3 = 9

(5.8)

Next, using the second equation of (??) as the pivot equation and repeating the
procedure, we obtain the following triangular set of equations
2x1 + 8x2 + 2x3 = 14
2x2 2x3 = 6
9x3 = 18

(5.9)

Obviously, x3 = 2 from the 3rd equation. Backward substitution into the

second equation gives x2 = 1 and substitution these into the first equation gives
x1 = 5.
In the above example the solution yielded values, which were exact, since only
whole numbers were encountered in the elimination process. In most instances, however, fraction will be encountered in the reduction process. The computer handles
fractions in decimal form to a certain limited number of decimal places, introducing
round-off error in the computer solution.
When only a small number of equations are to be solved, the round-off error is
small and usually does not substantially affect the accuracy of the results, but if
many equations are to be solved simultaneously, the cumulative effect of round-off
error can introduce relatively large solution errors. For this reason, the number of
simultaneous equations, which can be satisfactorily solved by Gausss elimination
method, using seven to ten significant digits in the arithmetic operations, is generally
limited to 15 to 20 when most of all of the unknowns are present in all of the
equations (the coefficient matrix is dense). On the other hand if only a few unknowns
are present in each equation (the coefficient matrix is sparse), many more equations
may be satisfactorily handled.
The number of equations which can be accurately solved also depends to a great
extent on the condition of the system of equations. If a small relative change in
one or more of the coefficients of a system of equations results in a small relative
change in the solution, the system of equations is called a well-conditioned system.
If, however, a small relative change in one or more of the coefficient values results
in a large relative change in solution values, the system of equations is said to be
ill conditioned. Since small changes in the coefficients of an equation may result
from round-off error, the use of double-precision arithmetic and partial pivoting
or complete pivoting becomes very important in obtaining meaningful solutions
of such sets of equations. If coefficient errors result from experimental error, then
no amount of accuracy in the solution method can be expected to yield meaningful
results.
One method that has been used to measure the condition of a set of equations
is to divide each equation by the square root of the squares of its coefficients, and
then evaluate the determinant of the coefficient matrix. The smaller the magnitude
in comparison with 1, the more ill conditioned the set is. An extensive discussion
of the solution of ill conditioned linear equations is given by Wilkinson ??.
81

To implement the Gausss elimination method on the computer, we present the

same problem in matrix notation. Since the computer can handle only numerical
data, the use of matrices is required in programming.

2 8
2
x1
14

1 6 1 x2 = 13
2 1
2
x3
5

(5.10)

Its augmented matrix

.
2 .. 14

.
6 1 .. 13

..
2 1
2 . 5

A/b = 1

(5.11)

The computer solution is involved with the reduction of this augmented matrix
to the augmented matrix of the equivalent triangular set of equations. The necessary
successive matrix reductions are accomplished by the same procedures given in steps
1 through 3, above although in working with matrices we are now concerned with
the pivot row and pivot element rather than with the pivot equation and pivot
coefficient, as before.
The first matrix reduction should result in the augmented matrix

.
2 .. 14
.
2 2 ..
6
..
0 9
0 . 9

0
A = 0

(5.12)

Reviewing the procedure outlined in step 1, the elements of the reduced matrix A0
can be written directly from the original matrix A, using the following formula:
a0ij

aik
= aij
akj
akk

where
a
a0
i
j
k
n
m

=
=
=
=
=
=
=

an elements of original matrix A

an elements of reduced matrix A0
row number of matrices
column number of matrices
number identifying pivot row
number of rows in matrices
number of columns in matrices

82

k j m
k+1 i n

(5.13)

With k = 1, elements of the matrix A0 given by (??) can be conformed. With k = 2,

we get

..
.
14
..
.
6
..
0 0 9 . 18

2 8
2

00
A = 0 2 2

(5.14)

which is determined from the matrix A0 utilizing (??). Note that the matrix of
equation(??) is the augmented matrix of equation (??). After obtaining the augmented matrix of the equivalent triangular set of equations, the xi values are obtained by back substitution, giving the same results.
For an efficient way of programming Gausss elimination method for the computer, we write a general procedure for reducing matrices as
(

k1
akj
k
k1
k1
aij = aij k1 aik
akk

k+1 j m
k+1 i n

(5.15)

where the is, js, ks, and so on, are as previously defined. The superscripts shown
merely correspond to the primes used in identifying successive reduced matrices,
and are not needed in a computer program.
The back substitution procedure may be generalized in the form of the following
set of equations:
anm
ann
P
aim nj=i+1 aij xj
=
aii

xn =
xi

(5.16)
i = n 1, n 2, , 1

(5.17)

There are two points, yet to be considered. First, we must guard against dividing by zero. Observe that zeros may be created in the diagonal positions even
if they are not present in the original matrix of coefficients. A useful strategy to
avoid (if possible) such zero divisors is to rearrange the equations so as to put the
coefficient of large magnitude on the diagonal at each step. This is called pivoting.
Complete pivoting may require both row and column interchanges. This is not
frequently done. Partial pivoting which places a coefficient of larger magnitude
on the diagonal by row interchanges only, will guarantee a nonzero divisor if there
is a solution to the set of equations, and will have the added advantage of improved
arithmetic precision. The diagonal elements that result are called pivot elements.
The second important point is the effect of the magnitude of the pivot elements
on the accuracy of the solution. If the magnitude of the pivot element is appreciably
smaller than the magnitude (absolute), in general, of the other elements in the
83

matrix, the use of the small pivot element will cause a decrease in the solution
accuracy. Therefore, for overall accuracy, using as a pivot row the row having the
largest pivot element should make each reduction. Such a provision should always
be incorporated in a computer program that is to solve fairly large numbers of
simultaneous equations.
We note that in terms of matrix operations, we transform the augmented coefficient matrix by elementary row transformations until an upper triangular matrix
is created on the left. Then back substitution gives the results (solution).
There exists the possibility that the set of equations has no solution or that
the prior procedure will fail to find it. During the triangularization step, if a
zero is encountered on the diagonal, we cannot use that row to eliminate coefficients
below that zero element. However, in that case, we will continue by interchanging
rows and eventually achieve an upper triangular matrix of coefficients. The real
stumbling block is finding a zero on the diagonal after we have trangularized. If
that occurs, the back substitution fails, for we cannot divide by zero. It also means
that the determinant is zero: there is no solution.

5.4

Gaussian Elimination Used in Determinant

Evaluation

The use of determinant method (Cramers rule) is not recommended for the solution
of linear simultaneous equations. The reason is that, except for systems of only two
or three equations, this method is too inefficient. For example, for a set of 10
simultaneous equations, about 70,000,000 multiplications and divisions are required
if the usual method of expansion in terms of minors is used. A more efficient method
of evaluating the determinants can reduce this to about 3000 multiplications, but
even this is inefficient compared to Gaussian elimination, which would require about
380. It may at times be necessary to evaluate a determinant. For large determinants,
the use of Gaussian elimination is much more efficient than expansion by minors.
From the definition of a determinant and the rules of expansion by minors, it may
be shown that
1. A determinant is changed only in sign by interchanging any two of its rows
(or columns).
2. A determinant is not changed in value if we add (or subtract) a constant
multiple of the elements of a row (or column) to the corresponding elements
of another row (or column).
The Gaussian reduction procedure is illustrated by evaluating the determinant of
the coefficient matrix of equation (??). The determinant is
84