Lecture Notes on

Computational and Applied Mathematics

(MAT 3310, 3 units)

Academic year 2009/2010: second term

Prepared by
Jun ZOU (2006)a and Patrick CIARLET (2010)b

We have prepared the lecture notes purely for the convenience of our teaching. We gratefully
acknowledge the contribution of Prof. I-Liang CHERN (National Taiwan University, Taipei, Taiwan),
made during his visit to the Dept of Mathematics of CUHK in 2009.
a
Department of Mathematics, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong
b
Dept of Applied Mathematics, Ecole Nationale Superieure de Techniques Avancees, Paris, France

Introduction to mathematical modeling discrete versus continuous models

In this chapter, we shall build some mathematical models, made up of either algebraic
or differential equations. They will be used throughout the notes as the basic models on
which the analytical or numerical methods we propose will be tested. Since the models
we propose originate from physical considerations, we also spend here some time on their
derivation.
Physical systems are usually modeled by discrete systems or continuous systems.
For instance, one kind of discrete system consist of nodes connected by bonds (or edges)
representing interactions among nodes. A spring-mass system is one of such discrete
systems. The masses are the nodes, whereas the springs are the bonds. In the solid mechanics, we may also image the atoms are the nodes and the chemical bonds connecting
them are the bonds. Continuous systems can be viewed as continuous limits of discrete
systems. Solid mechanics, fluid mechanics are such systems.
If the systems do not vary in time, we call them in equilibria. In this situation,
discrete systems are usually modeled by algebraic equations, whereas continuous systems
are usually formulated by differential equations.
Below, We shall use the spring-mass system and elastic bar to explain how to build
mathematical models for discrete and continuous systems.
We also recall a popular method for solving the discrete models, the Gaussian elimination, which we reinterpret as the LU factorization of a matrix.

1.1

Modeling spring-mass systems at equilibrium

Consider a spring-mass system which consists of n masses placed vertically between one
or two walls. Below, we call the downward direction the positive direction.
First, the case of two walls (see Figure 1.1(a)). The n masses mi , i = 1, ..., n, and
the two end walls are connected by n + 1 springs si , i = 1, ..., n + 1. If all masses are
zeros, the springs are at rest states: this is called the reference configuration. When
the masses are greater than zeros, the springs are elongated due to the gravitation force.
From its reference position, the mass mi moves a distance ui , called the displacement.
The goal is to find the displacements ui of the masses mi , i = 1, ..., n.

(a)

(b)

s1
m1

s2

s1
m1

m2

m2

mn

mn

s2

(c)
u 1 >0

m1

sn+1
Figure 1.1: Spring-mass systems: (a) two walls ; (b) one wall ; (c) displacement.
In this model, the nodes are the masses mi . We may treat the end walls as fixed
masses, and call them m0 and mn+1 , respectively. The corresponding displacements u0
and un+1 are both equal to 0. The edges (or the bonds) are the springs. Let us number
the spring connecting mi and mi+1 by i, i = 1, ..., n + 1, and suppose the spring si has
spring constant ci > 0. According to Hookes law, the elongation ei of the spring si is
in direct proportion1 with the load applied to it: so, the restoration force exerted by
the spring is equal to ci ei (in modulus) and it aims at contracting itself if ei > 0, and
extending itself is ei < 0.
Second, the case of a single wall (see Figure 1.1(b)). The only difference with the
previous model is that the lowest mass (labeled mn ), is connected to a single spring
(labeled sn ). So, there is a single fixed mass, here m0 , whereas the other endpoint is
free.
Let me start from the simplest case: n = 1 and no bottom wall. The mass m1
elongates the spring s1 by a displacement u1 , where the displacement is measured with
respect to the position at rest (see Figure 1.1(c)). The elongated spring has a restoration force c1 u1 acting on m1 .2 At equilibrium, this force must be balanced with the
1

This relation is true up to the so-called elastic limit. When the applied load becomes too large,
some irreversible phenomena occur: the spring begins to deform plastically.
2
The minus sign is due to the direction of force, which is upward.

gravitational force acting on m1 .3 Thus, we have

c1 u1 + f1 = 0,
where f1 = m1 g, the gravitation force on m1 , and g is the gravitation constant. From
this, we get
f1
u1 = .
c1
Next, let us consider the case where there is a bottom wall. In this case, both springs
s1 and s2 exert upward forces on m1 . The balance law becomes
c1 u1 c2 u1 + f1 = 0.
This results in the displacement u1 = f1 /(c1 + c2 ).
Let us jump to a slightly more complicated case, say n = 3. The unknowns are the
displacements u1 , u2 , u3 , whereas the displacements
u0 = 0, u4 = 0,

(1.1)

at the walls are known (and equal to 0). The displacements u1 , u2 , u3 cause elongations
of the springs, that are respectively equal to:
ei = ui ui1 , i = 1, 2, 3, 4.

(1.2)

(Above, we relate the elongations to both the unknown displacements of the masses,
and to the known displacements of the walls).
The restoration force of spring si is
wi = ci ei .

(1.3)

The force exerted to mi by spring si is wi = ci ei . In fact, when ei < 0, the spring

is shortened and it pushes downward to mass mi (the sign is positive), hence the force
is ci ei > 0. On the other hand, when ei > 0, the spring is elongated and it pulls mi
upward. We still get the force wi = ci ei < 0. Similarly, the force exerted to mi by
spring si+1 is wi+1 = ci+1 ei+1 . Indeed, when ei+1 > 0, the spring si+1 is elongated and
it pulls mi downward, the force is wi+1 = ci+1 ei+1 > 0. When ei+1 < 0, it pushes mi
upward, and the force wi+1 = ci+1 ei+1 < 0. In both cases, the force exterted to mi by
spring si+1 is wi+1 .
3

The mass m1 is in equilibrium.

At equilibrium, the force balance law on mi is

wi+1 wi + fi = 0, i = 1, 2, 3.

(1.4)

There are three algebraic equations for three unknowns u1 , u2 , u3 . In principle, we can
solve it.
Let us express the above equations in matrix form. First, the elongation:

~e = A~u, or

e1
e2
e3
e4

w1
w2
w3
w4

1
0
0
1 1
0
0 1 1
0
0 1

u1

u2

u3

where ~e R4 , A R43 , ~u R3 .
The restoration force:

w
~ = C~e, or

c1 0 0 0
0 c2 0 0
0 0 c3 0
0 0 0 c4

e1
e2
e3
e4

where w
~ R4 , C R44 .
The force balance laws:

1 1 0
0

T
A w
~ = f~, or 0 1 1 0

0 0
1 1

w1
w2
w3
w4

f
1

= f2

f3

where AT R34 is the transpose of A, and f~ R3 .

Eliminating the elongation ~e, we can write the above equations in block matrix form
as
C 1 A
AT 0

w
~
~u

!
=

0
f~

!
.

(1.5)

This kind of block matrix appears commonly in many other physical systems, for instance, network flows, fluid flows. In fact, any discrete optimization system with constraint can be written in this form. Here, the constraint part is the second set of
equations, written AT w
~ = f~. We shall come back to this point in the next chapter.
6

One way to solve the above block matrix system is to eliminate further the variable
w
~ and get
K~u = f~.
(1.6)
The matrix K := AT CA R33 is a symmetric positive definite matrix4 . It is called
the stiffness matrix. We get finally

c1 + c2 c2
0

K = c2 c2 + c3 c3
0
c3 c3 + c4
In section 1.4, we use the Gaussian elimination method to solve the system of algebraic
equations (1.6). We shall propose several other methods for solving this system, and also
for solving the constrained system of algebraic equations (1.5), in the Lecture Notes.

1.2
1.2.1

Modeling an elastic bar

Strain and Stress

Consider a (continuous) elastic bar5 of length ` = 1 and cross sectional area a, which is
hanged vertically. (it elongates6 due to gravity).
Set up an x-axis along the bar, with positive direction pointing downwards and origin
located at the top of the elastic bar. Consider any point at x along the bar: the position
is at x if no external force is applied: it is called the reference configuration (see Fig.
1.2(c) left). Then it moves down to x + u(x) because of the action of the external force
of gravity (see Fig. 1.2(c) right). The function u(x) is called the displacement. The
stretching at all points is measured by the derivative e(x) = du/dx(x), called the strain.
If u is a constant, the elastic bar is unstretched: e = 0 in this case. On the other hand,
the stretching of the bar produces an internal force called stress (one can experience
this force easily by pulling the two ends of an elastic bar). According to experimental
measurements, one can check that this internal force is proportional to the strain in the
4

Prove this important result.

The length ` of a bar is much bigger than the size of its cross-section a. Moreover, an elastic bar
may be pulled back and forth.
6
Alternately, the bar can contract if one compresses it, by applying opposite forces at its ends,
directed towards it. Or, one can imagine superposition of several applied external forces, that result in
elongations and compressions of the bar.
5

(a)

(b)

(c)

x
ref.
config.

x+u(x)
with
applied forces

l
Figure 1.2: Elastic bar: (a) bar with both ends fixed ; (b) bar with only top end fixed ;
(c) displacement u(x) with respect to reference configuration.
bar7 i.e.

du
(x) ,
dx
where c(x) is determined by the elastic material. This is Hookes law in the case of an
elastic bar. c is either a constant, or a function of x if the material is inhomogeneous.
In any case, it is strictly positive and bounded away from 0:
(internal force) w(x) = c(x)

there exist c0 > 0 and c1 > 0 such that c0 c(x) c1 , for all x ]0, 1[.

(1.7)

To set up the model, we take a small piece of the bar [x, x + 4x], of volume (4x) a.
At equilibrium8 , all forces acting on it are balanced. This yields


ac

 du 
du 
(x + 4x) ac
(x) + (x)(4x a)g = 0,
dx
dx

(1.8)

where g is the gravitational constant and (x) is the mass density at position x.
7

This relation is valid up to the elastic limit of the material composing the bar. When the load
becomes too large, which can be expressed mathematically as w(x0 ) > wlimit , for some position x0 , the
material deforms plastically.
8
The sign in front of the upper and bottom stress contributions can be determined by analogy with
the discrete case (the spring-mass system).

Dividing both sides of equation (1.8) by a4x, then taking 4x 0, we get

d  du 

c
(x) = f (x)
dx dx

(1.9)

where f (x) = g (x) is the external force per unit length.

The equation (1.9) must be supplemented with appropriate physical boundary conditions, to ensure it is well-posed.
1.2.2

Boundary conditions

There are many possibilities. For instance

(a) Both ends of the elastic bar are fixed (see Fig. 1.2(a)), so no displacements there:
u|x=0 = 0,

u|x=1 = 0.

These are called Dirichlet boundary conditions 9 . By definition, these conditions

are said to be homogeneous, since the values are 0.
(b) Top end of the elastic bar is fixed (no displacement), while the other end is free
(see Fig. 1.2(b)). Indeed, there is no internal force since it is in the air10 :
 du 
u|x=0 = 0, w|x=1 = c
|x=1 = 0.
dx
The first one is called a Dirichlet boundary condition (see (a)), while the second
one is called a Neumann boundary condition 11 . Again, the boundary condition
 du 
c
|x=1 = 0
dx
is said to be homogeneous. On the other hand, boundary conditions like
or u|x=1 = 2,

u|x=0 = 1,
or

 du 
c
|x=1 = 3
dx
are all called non-homogeneous boundary conditions.
9

In general, Dirichlet boundary conditions involve only the value of u, and not the value of any of
its higher order derivatives, at the endpoints.
10
See the end of the section for a justification.
11
In general, Neumann boundary conditions involve only the value of du/dx, and not the value of u
nor of any of its higher order derivatives, at the endpoints.

So the complete model, or full model, for an elastic bar is :

d  du 

c
(x) = f (x), 0 < x < 1
dx dx
supplemented with the set of boundary conditions
(a)

u|x=0 = 0,

u|x=1 = 0

or

 du 
c
|x=1 = 0.
dx
This differential equation, together with the two boundary conditions, is called a twopoint boundary value problem 12 .
(b)

1.2.3

u|x=0 = 0,

Solutions of the elastic bar model

We now try to find the solution of the following boundary value problem




d c du (x) = f (x), 0 < x < 1

dx dx 
du 

u|x=0 = 0,
c
|x=1 = 0 .
dx

(1.10)

Solution. Integrating the equation (1.10) over13 ]x, 1[, we obtain

h du i1 Z 1
f (t) dt .
c
=
dx x
x
Using the Neumann boundary condition, we have
Z 1
du
c(x) (x) =
f (t)dt ,
dx
x
or
du
1
(x) =
dx
c(x)

f (t)dt .
x

Integrating over ]0, x[ and using the Dirichlet boundary condition gives
Z x
Z 1
1
u(x) =
f (t)dt ds ,
0 c(s) s
this is the exact solution to the problem (1.10). ]
12
13

Why do we need two boundary conditions?

Why do we begin by integrating from x to 1?

10

(1.11)

Remark 1.1. The solution given by (1.11) is an abstract solution. In other words, one
might not be able to derive an explicit formula for u(x)...
Example 1.1. Find the exact solution to the problem

d u (x) = x2 , 0 < x < 1

dx2

u|x=0 = 0, du |x=1 = 0 .
dx

(1.12)

Solution. Integrating the equation (1.12) over ]x, 1[, we obtain

h du i1 Z 1
1 1

=
t2 dt = x3 .
dx x
3 3
x
Using the Neumann boundary condition, we have
1 1
du
(x) = x3 .
dx
3 3
Integrating over ]0, x[ and using the Dirichlet boundary condition gives
Z x
1
1
1 1
u(x) =
( s3 )ds = x x4 .
3 3
3
12
0

(1.13)

It is easy to verify that this u(x) is really the solution of the system (1.12). ]
Example 1.2. What happens when one considers another boundary condition? For
instance, let us try to find the exact solution to the following problem
d  du 

c
(x) = f (x), 0 < x < 1
dx dx
with the boundary conditions
u|x=0 = 1, u|x=1 = 1 .
Solution. Integrating the equation over ]x, 1[, we get
Z 1
du
c(x) (x) = C0 +
f (t) dt ,
dx
x
where C0 is an integration constant14 . This implies
Z 1
du
C0
1
(x) =
+
f (t) dt .
dx
c(x) c(x) x
14

The value of C0 , here cux|x=1 , is unknown. Indeed, it is not given by the problem.

11

Now integrating over ]0, x[ and using the Dirichlet boundary condition at x = 0 gives
Z x
Z x
Z 1
1
1
u(x) = 1 + C0
f (t) dt ds .
dt +
0 c(t)
0 c(s) s
Using the boundary condition u|x=1 , we can find the integration constant C0 . Its exact
value is


R1 1 R1
2 0 c(s) s f (t) dt ds
R

C0 =
. ]
1 1
dt
0 c(t)

1.3

Connections between the continuous and discrete models

The analogy between the continuous model and the discrete model. We
provide a table to show the analogy between the spring-mass model and the elastic bar
model. By analogy, the operator d/dx (with discrete counterpart A) appears as the
variables and relations spring-mass
displacement
ui , i = 1, , n
elongation/strain
ej , j = 1, , n (or n + 1)
restoration force/stress wj
gravitation force
fi
connection relation
~e = A~u
Hookes law
w
~ = C~e
Force balance law
AT w
~ = f~

elastic bar
u(x), x ]0, 1[
e(x)
w(x)
f (x)
e(x) = du
(x)
dx
w(x) = c(x)e(x)
dw
(x) = f (x)
dx

conjugate of the operator d/dx (with discrete counterpart AT ). Why is it so? Given
A Rnp , we recall that its transpose AT Rpn can be defined by
(AT ~x, ~y )Rp = (~x, A~y )Rn ,

~x Rn , ~y Rp .

Now, we consider the following functional spaces:

C 1 ([0, 1]) := {u : [0, 1] R | u is continuously differentiable over [0, 1]}
C01 ([0, 1]) := {u C 1 ([0, 1]) | u(0) = u(1) = 0} .
These are vector spaces15 . We can define an inner product
Z 1
(u, v) =
uv dx .
0
15

Evidently, the differential operator d/dx maps C 1 ([0, 1]) into C 0 ([0, 1]).

12

(1.14)

The integration by parts formula gives

Z 1
Z 1
h ix=1
du
dv
u dx + uv
v dx =
.
dx
x=0
0 dx
0
When u, v C01 ([0, 1]), we get
(

d
d
u, v) = (u, v).
dx
dx

With respect to the inner product (1.14) defined above, this explains why the conjugate
operator of d/dx is d/dx.
Remark 1.2. One could also choose
1
C00
([0, 1]) := {u C 1 ([0, 1]) | u(0) = 0} , and
1
C01
([0, 1]) := {u C 1 ([0, 1]) | u(1) = 0} .

And prove that d/dx is also the conjugate operator of d/dx with respect to (1.14), for
1
1
u C00
([0, 1]) and v C01
([0, 1]).
The elastic bar model seen as a continuous limit of the spring-mass system.
In the continuous model (1.9) with ` = 1, we can divide the domain [0, 1] into n+1 small
subintervals uniformly16 , every one of them of length 4x = `/(n + 1). We label the grid
points i4x by xi , for i = 0, 1, , n + 1. We further imagine there are masses mi at the
positions xi , for i = 1, , n, with springs connecting them consecutively. Each spring
has length 4x when it is at rest. According to the spring-mass model, we have
ci (ui ui1 ) ci+1 (ui+1 ui ) = mi g,

i = 1, , n .

(1.15)

Here, ci is the spring constant of the spring connecting xi to xi+1 . Over the cross-section
(area a), we have, when 4x is small
mi (xi )a4x, ci ac(xi1/2 )/4x.
Above, is the mass density, xi1/2 = (i 1/2)4x is the mid-point of the spring, and
the spring constant is proportional17 to 1/4x and a.
16

This is the case of the bar with both ends fixed. Otherwise (one fixed end, one free end), one follows
the same path with 4x = `/n.
17
Think about the problem: given two neighboring masses, let us have many parallel springs over
the cross-section, linking one mass to its neighbor. Then all restoration forces add up, with the same

13

Now, with this approximation, we get that for small 4x, the spring-mass system
yields
1 
(ui ui1 )
(ui+1 ui ) 
c(xi1/2 )
c(xi+1/2 )
= (xi )g.
(1.16)
4x
4x
4x
As we take 4x 0, and assuming that u(xi ) ui , we recover the equation for the
elastic bar:
d  du 

c
(x) = f (x),
dx dx
where f = g.
Considering the spring-mass system with two endwalls (see Figure 1.1(a)), we recall
that the end displacements u0 and un+1 satisfy the fix-end boundary conditions
u0 = 0, un+1 = 0.
They correspond to the homogeneous Dirichlet boundary condition on u in the elastic
bar model:
u|x=0 = 0, u|x=1 = 0.
Or, considering the spring-mass system with one free endpoint at the bottom location
xn = ` (see Figure 1.1(b)), we notice that we have
cn (un un1 ) = mn g .
Therefore, when 4x is small, we derive


u(xn ) u(xn1 )
c(xn1/2 )
= g (4x) .
4x
Passing to the limit 4x 0, we recover the homogeneous Neumann boundary condition
 du 
c
|x=1 = 0 .
dx
elongation, so the global spring constant is equal to the sum of spring constants. It is therefore proportional to a. On the other hand, having a series of consecutive springs is trickier. However, one can
prove that if all springs have the same characteristic (same spring constant c for the Nsp connected
springs), then the resulting glogal spring constant is given by
1
cglobal

k=Nsp

X 1
Nsp
=
.
c
c

k=1

In the present case, Nsp 1/4x, so the global spring constant is indeed equivalent to 1/4x.

14

1.4

Solving the spring-mass system Gaussian elimination

In this section, we shall solve the square linear system A~x = ~b by the Gaussian elimination method. Consider solving the system of equations of the following form
A~x = ~b ,
where A is a non-singular n n matrix and ~b Rn is a vector. Let us recall the basic
principle underlying the method. Gaussian elimination is a process to reduce the n n
system of equations A~x = ~b, or
a11 x1 + a12 x2 + + a1n1 xn1 + a1n xn = b1
a21 x1 + a22 x2 + + a2n1 xn1 + a2n xn = b2

an1 x1 + an2 x2 + + ann1 xn1 + ann xn = bn

into an upper triangular system of equations
a11 x1 + a12 x2 + + a1n1 xn1 + a1n xn = b1
0 x1 + a
22 x2 + + a
2n1 xn1 + a
2n xn = b2

0 x1 + 0 x2 + + 0

xn1 + a
nn xn = bn .

This is equivalent to a process of reducing the full matrix

A=

a11 a12
a21 a22

an1 an2

a1n1 a1n
a2n1 a2n

ann1 ann

into an upper triangular matrix

A =

a11 a12
0 a
22

0
0

15

a1n1 a1n
a
2n1 a
2n

0
a
nn

1.4.1

LU factorization

The Gaussian elimination method is equivalent to factoring A into LU , a product of

a lower triangular matrix and an upper triangular matrix, where L is normalized with
unity diagonal entries. The problem of solving A~x = ~b is then decomposed into two
steps:
L~c = ~b, U~x = ~c.
The former can be solved by forward substitution, whereas the latter can be solved by
backward substitution. We give a detailed description below. In the case when A is
symmetric, we can further factor A into A = LDLT , where D is a diagonal matrix. The
equation A~x = ~b can be split into
~
Ld~ = ~c, Dd~ = ~c, LT ~x = d.
They can be solved by forward substitution, direct inversion and backward substitution,
respectively. Below, we give elementary versions of the corresponding algorithms.
A lower triangular system can be solved by forward substitution. Consider the linear equation L~c = ~b in R4 , or
c 1 = b1 ,
l21 c1 + c2 = b2 ,
l31 c1 + l32 c2 + c3 = b3 ,
l41 c1 + l42 c2 + l43 c3 + c4 = b4 .
You can solve c1 first, then substitute into the 2nd equation to get c2 and so on:
c1 = b 1 ,
c2 = b2 l21 c1 ,
c3 = b3 l31 c1 l32 c2 ,
c4 = b4 l41 c1 l42 c2 l43 c3 .
This is called method of forward substitution.

16

 A diagonal system can be solved directly. Consider the linear equation

Dd~ = ~c in R4 . You can solve all unknowns simultaneously:
d1 = c1 /d11 ,
d2 = c2 /d22 ,
d3 = c3 /d33 ,
d4 = c4 /d44 .
An upper triangular system can be solved by backward substitution.
Consider the linear equation U~x = d~ in R4 , or
u11 x1 + u12 x2 + u13 x3 + u14 x4 = d1 ,
u22 x2 + u23 x3 + u24 x4 = d2 ,
u33 x3 + u34 x4 = d3 ,
u44 x4 = d4 .
You can solve x4 first, then substitute into the third equation to get x3 and so on:
x4 = d4 /u44 ,
x3 = (d3 u34 x4 )/u33 ,
x2 = (d2 u23 x3 u24 x4 )/u22 ,
x1 = (d1 u12 x2 u13 x3 u14 x4 )/u11 .
This is called method of backward substitution.
Next, let us review some basic properties of lower triangular matrices.
Understanding row

1
0

L2 A
0
0

operation by

0 0 0

1 0 0

l32 1 0
l42 0 1

a11

a21

=
a31 + l32 a21
a41 + l42 a21

a lower elementary triangular matrix:

a11 a12 a13 a14

a21 a22 a23 a24

a31 a32 a33 a34

a41 a42 a43 a44

a12
a13
a14

a22
a23
a24

.
a32 + l32 a22 a33 + l32 a23 a34 + l32 a24
a42 + l42 a22 a43 + l42 a23 a44 + l42 a24
17

If you look at the above operations carefully, you will come to the conclusion:
the actions of adding row 2 multiplied by l32 and l42 respectively to row 3 and row
4 respectively, are equal to the operation L2 A.
The product of two elementary lower triangular matrices. Now consider
the following two matrices:

1 0 0 0
1 0 0 0
l

0 1 0 0
21 1 0 0

L1 =
, L2 =
,
l31 0 1 0
0 l32 1 0
l41 0 0 1

0 l42 0 1

we can directly check18

1
l21
l31
l41

L1 L2 =

0
1
l32
l42

0
0
1
0

0
0
0
1

You may also define19

1
0
0
0

L3 =

0
1
0
0

0
0
1
l43

0
0
0
1

The inverse of elementary lower triangular matrices. As we saw, you can

solve L1~c = ~b by forward substitution. Considering successively




0
0
0
1
0
0
1
0

~b =
, ~b = , ~b = , ~b = ,
0
1
0
0
0

you can compute all columns of L1

1 , and

1
l
21
L1
1 =
l31
l41
18
19

Check the product L2 L1 : is it equal to L1 L2 ?

Find the product L1 L2 L3 .

18

you can check that

0 0 0
1 0 0

.
0 1 0
0 0 1

Similar results are true for the other Li matrices and more generally for all any
n n elementary lower triangular matrix.
1.4.2

Simple cases

Below, we will explain the Gaussian elimination and the LU factorization for two simple
examples: one is a 2 2 system of equations, the other is a 3 3 system. If you can
understand these two simple examples, then you may easily carry out the Gaussian
elimination and LU factorization for more general n n systems.
Gaussian elimination and LU factorization for a 2 2 matrix Let us consider
the following 2 2 system:
(
2x1 + 4x2 = 2
4x1 + 11x2 = 1
Eliminating x1 in the 2nd equation one needs to add eq. 1 multiplied by 2 to eq. 2,
which yields
(
2x1 + 4x2 = 2
3x2 = 3
It follows that x2 = 1, and then x1 = 3. This is the Gaussian elimination.
Let us look now at the LU factorization. The algebraic equations may be written
!
!
!
2 4
x1
2
A~x :=
=
=: ~b .
(1.17)
4 11
x2
1
Eliminating x1 in the 2nd equation one needs to add row 1 multiplied by 2 to row 2,
that equals
!
!
!
1
0
2
4
2
4
:=
LA
=
=: U.
2 1
4 11
0 3
is elementary, the above equation gives, with L = L
1 ,
Since the matrix L
!
!
!
2 4
1 0
2 4
=
=: L U.
A=
4 11
2 1
0 3
This yields a LU factorization of A.
Using the above factorization, solving the system
A~x = ~b
19

is equivalent to solving the system

LU~x = ~b,
which can be done as follows:
L~c = ~b,

U~x = ~c.

Applying this process to equation (1.17), we have

!
!
1
0
c
1
L~c = ~b
=
2 1
c2

2
1

!
,

which gives
c1
c2

!
=

Then
U~x = ~c

2
3

2 4
0 3

which gives the solution to equation (1.17):

!
x1
=
x2

!
.

x1
x2

3
1

!
=

2
3

!
,

!
.

Gaussian elimination and LU factorization for a 3 3 matrix Let us consider

one additional simple example for the LU factorization20 :

0
1 1 1
x1

(1.18)
A~x := 3 6 4 x2 = 2 =: ~b .
1/3
1 2 1
x3
Eliminating x1 in the 2nd equation, one needs to add row 1 multiplied by 3 to row 2;
eliminating x1 in the 3rd equation, one needs to add row 1 multiplied by 1 to row 3,
that equals

1 0 0
1 1 1
1 1 1

1 A :=
L
3 1 0 3 6 4 = 0 3 1 =: U1 .
1 0 1
1 2 1
0 1 0
20

Carry out yourself the Gaussian elimination for the corresponding 3 3 system of equations.

20

Now eliminating x2 in the

row 3, that equals

1
0
0

2L
1 A := 0
L
1
0
0 1/3 1

3rd equation, one needs to add row 2 multiplied by 1/3 to

1 0 0
1 1 1
1 1
1

1 =: U.
3 1 0 3 6 4 = 0 3
1 0 1
1 2 1
0 0 1/3

1 and L
2 are elementary, the above equation gives
Since the matrices L

1 1 1

A = 3 6 4
1 2 1

1 0 0
1 0 0
1 1
1

= 3 1 0 0 1 0 0 3
1
1 0 1
0 1/3 1
0 0 1/3

1 1
1
1 0 0

= 3 1 0 0 3
1
0 0 1/3
1 1/3 1
=: L U.
This completes a LU factorization of A.
Using this factorization, solving the system
A~x = ~b
is equivalent to solving the system
LU~x = ~b,
which can be done as before:
L~c = ~b,

U~x = ~c.

Applying this process to equation (1.18), we have

0
1 0 0
c1

L~c = ~b 3 1 0 c2 = 2 ,
1 1/3 1
c3
1/3
which gives

c1
0

c2 = 2 .
c3
1
21

Then

U~x = ~c

1 1
1
x1
0

1 x2 = 2 ,
0 3
0 0 1/3
x3
1

which gives the solution of equation (1.18):

x1
8/3

x2 = 1/3 .
x3
3
1.4.3

LU factorization for an n n matrix

The previous LU factorization can be carried out for more general n n matrices. In
general, it can be shown that if all the main submatrices21 of A are non-singular, then
we can factor A as
A = LU
where L is a lower triangular matrix with 1 as its diagonal entries, and U is an upper
triangular matrix. Let
D = diag(U ),
then we can further factor A as follows:
A = L D U0
where L and U 0 are lower and upper triangular matrices respectively, both matrices with
1 as their diagonal entries, and D is a diagonal matrix22 .
What happens when one of the main submatrices of the matrix A is singular? One
has to consider a new strategy, which involves pivoting, corresponding to rearrangements
of rows of A. Let us give an example below.
Let us begin by the Gaussian elimination of the system

x1 + 2x2 + 3x3 = 1
3x1 + 6x2 + 5x3 = 2

4x1 + 2x2 + 7x3 = 3

21

The n main submatrices of A are the k k matrices (aij )1i,jk , for k = 1, 2, , n. See the end
of the paragraph, when one the main submatrices is singular.
22
By writing the product L D U 0 by row or by column, derive the formulas to compute the entries of
L, D and U 0 (or check the Assignments for more details).

22

Eliminating x1 in the 2nd equation, one needs to add eq. 1 multiplied by 3 to eq. 2;
eliminating x1 in the 3rd equation, one needs to add eq. 1 multiplied by 4 to eq. 3, to
find

1
x1 + 2x2 + 3x3 =
4x3 = 1

6x2 5x3 = 1
The coefficient in front of x2 in eq. 2 is equal to 0: indeed, the 2 2 main submatrix is
singular, since


 1 2 


 = 0!

 3 6 
To carry on, one simply swaps eq. 2 and eq. 3, to find

1
x1 + 2x2 + 3x3 =
6x2 5x3 = 1

4x3 = 1
Then, this system can be solved by backward substitution, to find successively x3 = 1/4,
x2 = 1/24 and x1 = 1/3.
Let us proceed with the LU factorization of A in

1
x1
1 2 3
~

A~x := 3 6 5 x2 = 2 =: b .
3
x3
4 2 7
Eliminating x1 in the 2nd equation, one needs to add row 1 multiplied by 3 to row 2;
eliminating x1 in the 3rd equation, one needs to add row 1 multiplied by 4 to row 3,
that equals

1 0 0
1 2 3
1
2
3

1 A :=
L
0 4 =: U1 .
3 1 0 3 6 5 = 0
4 0 1
4 2 7
0 6 5
Observe that the coefficient in front of x2 is equal to 0 on row 2. Therefore, it is not
possible to eliminate x2 on row 3 by adding row 2 multiplied by an ad hoc factor... To
carry on, one idea is to check for non-zero coefficients on column 2 (same index as row
2), below the main diagonal: here 6 is the answer. Then, one swaps row 2 and row 3, to
bring the (non-zero) coefficient at the desired location. This strategy is called pivoting,

23

and it can be achieved by the multiplication by a permutation matrix P :

1 0 0
1
2
3
1
2
3

1 A :=
PL
0 4 = 0 6 5 =: U.
0 0 1 0
0 1 0
0 6 5
0
0 4
Above, we have obtained an upper triangular matrix U , as before. If that were not the
case, we could nonetheless proceed with the factorization process, since the diagonal
coefficient is not equal to 0 anymore.
To conclude, we note that we can write

1 0 0
1 0 0

1 = L
01 P :=
PL
4 1 0 0 0 1 .
3 0 1
0 1 0
In other words, we can put the multiplication by the permutation matrix P in front of
A so that we have finally
0 (P A) = U, or P A = L U.
L
1
As before, L (respectively U ) is a lower (resp. upper) triangular matrix. The novelty is
that there is an additional permutation matrix P , to account for the pivoting of rows.
You can then solve the initial system A~x = ~b: indeed, one has P A~x = P~b, so it follows
that

1/3
1

L U ~x = 3 , and ~x = 1/24 .
1/4
2
More generally, you can check that given any non-singular n n matrix A and any
vector ~b of Rn , one can solve the system A~x = ~b by performing the LU factorization
with pivoting:
P A = LU ,
where P is a permutation matrix23 (possibly equal to the identity matrix, when no
pivoting is needed!). As before, L is a lower triangular matrix with 1 as its diagonal
entries, and U is an upper triangular matrix.
23

The matrix P is usually built as the product of elementary, 2-row permutation, matrices, and as
such, it is also a permutation matrix.

24

1.4.4

LU factorization of a symmetric positive definite matrix

Let A be an n n symmetric and positive definite matrix. In other words, A = AT

(symmetry) and for all ~x 6= 0, (~x, A~x)Rn > 0 (positive definiteness)24 . A has a unique
decomposition:
A = LDU .
Since A is positive definite, dii > 0, for i = 1, 2, , n.
Since A is symmetric, there holds
A = L D U = U T D LT .
By the uniqueness of the decomposition, we have U = LT , that is,
A = L D LT .
If we write
p
D1/2 = diag( dii ),
then
A = (L D1/2 ) (L D1/2 )T .
This indicates that for any symmetric and positive definite matrix A, we have the factorization of the form
L
T
A=L
is a lower triangular matrix. This is called the Cholesky factorization of a
where L
positive definite matrix.
in the Cholesky factorization are not necessarily
Note that the diagonal entries of L
equal to 1, unlike the entries of L in the LU factorization25 .
1.4.5

Solving the spring-mass system

Now, we go back to the spring-mass system:

K~u = AT CA~u = f~
24

(1.19)

Here, (, )Rn > 0 denotes the scalar product of Rn . The condition could be written equivalently:
for all ~x 6= 0, ~xT (A~x) > 0.
25
L
T by row or by column, derive the formulas to compute the entries of L

By writing the product L

(or check the Assignments for more details).

25

Due to the symmetry of our matrix K, we can thus factor K into

K = LDLT
where L is a lower triangular matrix

L = l21 1

0 l32 1
and D = diag(d11 , d22 , d33 ) is a diagonal matrix. Solving K~u = f~ is now decomposed
into three steps:
L~y = f~, Dw
~ = ~y , LT ~u = w.
~
Each of them can be solved by substitution or direct inversion.
To find the matrices L and D, we multiply LDLT and get the entry terms as

d11
d11 l21
0

2
LDLT = d11 l21 d11 l21
+ d22
d22 l32

2
0
d22 l32
d22 l32 + d33
By comparing the entries of K and LDLT , we get
d11 = c1 + c2
d11 l21 = c2
2
d11 l21
+ d22 = c2 + c3

d22 l32 = c3
2
d22 l32

+ d33 = c3 + c4

We can solve d11 , l21 , d22 , l32 , d33 successively from the above equations.

26