Numerical Methods Notes

Approximation and Errors
Significant Figures
4 significant figures
1.845
0.01845
0.0001845
43,500 ? confidence
4.35 x 10
4
4.350 x 10
4
4.3500 x 10
4
Accuracy and Precision
Accuracy - how closely a computed or
measured value agrees with the true value
Precision - how closely individual
computed or measured values agree with
each other
number of significant figures
spread in repeated measurements or
computations
increasing accuracy
i
n
c
r
e
a
s
i
n
g

p
r
e
c
i
s
i
o
n

Error Definitions
Numerical error - use of approximations to
represent exact mathematical operations and
quantities
true value = approximation + error
absolute error, E
t
= true value - approximation

subscript t represents the true error
shortcoming....gives no sense of magnitude
normalize by true value to get true relative error
Error definitions cont.
t
true error true value estimate
100% 100%
true value true value
c = =
True relative percent error
But we may not know the true answer a priori
May not know the true answer a priori
a
approximate error
100%
approximation
c =
This leads us to develop an iterative
approach of numerical methods
a
approximate error
100%
approximation
present approx. previous approx.
100%
present approx.
c =
=
Usually not concerned with sign, but with
tolerance
Want to assure a result is correct to n
significant figures
( )% 10 5 . 0
n 2
s
s a
= c
c < c
Example
Consider a series expansion to estimate trigonometric
functions
< < + + = x .....
! 7
x
! 5
x
! 3
x
x x sin
7 5 3
Estimate sin t / 2 to three significant figures

Error Definitions cont.
Round off error - originates from the fact
that computers retain only a fixed number
of significant figures
Truncation errors - errors that result from
using an approximation in place of an exact
mathematical procedure
Computer Storage of Numbers
Computers use a binary base system for logic
and storage of information
This is convenient due to on/off circuitry
The basic unit of computer operation is the bit
with value either 0 (off) or 1 (on)
Numbers can be represented as a string of bits
110
2
= 0*2
0
+ 1*2
1
+ 1*2
2
= 0+2+4 = 6
10
A string of 8 bits is called a byte
Computer Storage of Integers
It is convenient to think of an integer as being
represented by a sign bit and m number bits
This is not quite correct
Perhaps you remember that underflow & overflow
are -4,294,967,296 & +4,294,967,295 for a 4 byte
(32 bit or single precision) integer
With a sign bit this should be 2
(32-1)
- 1 =
4,294,967,295
Because +0 and -0 are redundant, a method of
storage called 2s complement is used
2s Complement
The easiest way to understand 2s
complement is in terms of a VCR counter
When you rewind a tape and the counter goes
past zero, does it register negative values?
0 0 2
2s Complement & -1
Use a 1 byte (8 bit) integer as an example
Use all 1s (our highest digit in base 2) for -1
Thus we use 11111111 to represent -1
We furthermore represent 0 as all zeros
What happens we we add -1 and 1 using these?
1 1 1 1 1 1 1 1
+ 0 0 0 0 0 0 0 1
1 0 0 0 0 0 0 0 0
Carried Bit is Lost
8 Bit Integer Example, Cont.
With our system we dont need separate
algorithms for addition and subtraction.
We simply add the negative of a number
So -1 plus -1 is 1 1 1 1 1 1 1 1
+ 1 1 1 1 1 1 1 1
(-2) 1 1 1 1 1 1 1 1 0
And 1 less than -2 is 1 1 1 1 1 1 1 0
+ 1 1 1 1 1 1 1 1
(-3) 1 1 1 1 1 1 1 0 1
8 Bit Integer Example, Cont.
It is not hard to see from the trend that the
most negative number is 1000000
2
= -128 (-2
7
)
and the most positive is 0111111
2
= 127 (2
7
-1)
The algorithm for negation is to reverse
all bits and add 1, thus twos complement.
So that - 0111111
2
= 1000001
2
= -127
This even works for 0 1 1 1 1 1 1 1 1
+ 0 0 0 0 0 0 0 1
1 0 0 0 0 0 0 0 0
8 Bit Real Variable Example
Real numbers must be represented by a
mantissa and an exponent (each with a sign)
Computers use 0.XXX E
2
+ZZ rather than
the X.XX E
10
+YY of scientific notation
Divide the 8 bits up into 3 bits for the
exponent and 5 bits for the mantissa, with
each using a sign bit (not really, but nearly)
So we have Z Z X X X X
Exponent Mantissa
8 Bit Real, Machine c
Machine c is the error relative to real #
representation of 1 0 1 1 0 0 0
Addition of two reals requires that their
exponents be the same 0 1 0 0 0 1
So the smallest number which can be added
to real 1without losing the mantissa bit is
shown above (0x2
-1
+ 0x2
-2
+ 0x2
-3
+1x2
-4
) x 2
1
= 0.125 : Machine c here
Exponent Mantissa
Exponent Mantissa
32 Bit Real Representations
IEEE standard real numbers have 8 bit
exponents and 24 bit mantissas
Machine c for this system is then
.00000000000000000000001
2
= 2
-23

= 1.192093 x 10
-7
Overflow for this system is then
.11111111111111111111111
2
x E
2
2
1111111

= (1- c
machine
) x 2
127+1
= 3.40282 x 10
+38
(note that the first mantissa bit after . is
assumed as 1--thus 127+1 factors of 2)
TAYLOR SERIES
Provides a means to predict a function value
at one point in terms of the function value
and its derivative at another point
Zero order approximation
( ) ( )
i 1 i
x f x f ~
+
This is good if the function is a constant.
Taylor Series Expansion
First order approximation
{

slope multiplied by distance
Still a straight line but capable of predicting
an increase or decrease - LINEAR
( ) ( ) ( )( )
i 1 i i i 1 i
x x x ' f x f x f + ~
+ +
Second order approximation - captures
some of the curvature
( ) ( ) ( )( )
( )
( )
2
i 1 i
i
i 1 i i i 1 i
x x
! 2
x ' ' f
x x x ' f x f x f + + ~
+ + +
( ) ( ) ( ) ( )
( )
( ) ( )
( )
( )
( )
1 i i
1 n
1 n
n
i 1 i
n
n
i
n
......
3
i
2
i
i i i 1 i
x x h
! 1 n
f
R
x x size step h where
R h
! n
x f
h
! 3
x ' ' ' f
h
! 2
x ' ' f
h x ' f x f h x f x f
+
+
+
+
+ +
+
< <
+

=
= =
+ +
+ + ~ + =
Example
Use zero through fourth order Taylor series expansion
to approximate f(1) from x = 0 (i.e. h = 1.0)
( ) 2 . 1 x 25 . 0 x 5 . 0 x 15 . 0 x 1 . 0 x f
2 3 4
+ =
-1
-0.5
0
0.5
1
1.5
0 0.5 1 1.5
x
f
(
x
)
Note:
f(1) = 0.2
-1.5
-1
-0.5
0
0.5
1
1.5
0 0.5 1 1.5
x
f
(
x
)
n = 3
n = 0
n = 1
n = 2
act ual value
Functions with infinite number of
derivatives
f(x) = cos x
f `(x) = -sin x
f ``(x) = -cos x
f ```(x) = sin x
Evaluate the system where x
i
= t/4 and
x
i+1
= t /3
h = t /3 - t /4 = t /12
Functions with infinite number of
derivatives
Zero order
f(t /3) ~ cos (t /4 ) = 0.707 c
t
= 41.4%
First order
f(t /3) ~ cos (t /4 ) - sin (t /4 )(t /12)
c
t
= 4.4%
Second order
f(t /3) ~ 0.4978 c
t
= 0.45%
By n = 6 c
t
= 2.4 x 10
-6
%
Roots (Quadratic Formula)
0 c bx ax ) x ( f
a 2
ac 4 b b
x
2
2
= + + =

=
This equation gives us the roots of the algebraic function
f(x)

i.e. the value of x that makes f(x) = 0

How can we solve for f(x) = e
-x
- x?
Roots of Equations
Plot the function and determine where it
crosses the x-axis
Lacks precision
Trial and error
-10
-5
0
5
10
15
-5 0 5 10
x
f
(
x
)
f(x) = e
-x
- x
Overview of Methods
Bracketing methods
Graphing method
Bisection method
False position
Open methods
One point iteration
Newton-Raphson
Secant method
Bracketing Methods
Graphical
Bisection method
False position method
Graphical
(limited practical value)
x
f(x)
x
f(x)
x
f(x)
x
f(x)
consider lower
and upper bound
same sign,
no roots or
even # of roots
opposite sign,
odd # of roots

Bisection Method
Takes advantage of sign changing
f(x
l
)f(x
u
) < 0 where the subscripts refer to
lower and upper bounds
There is at least one real root
x
f(x)
x
f(x)
x
f(x)
-10
-5
0
5
10
15
-5 0 5 10
x
f
(
x
)
f(x) = e
-x
- x
f(x) = e
-x
- x
x
l
= -1
x
u
= 1
PROBLEM STATEMENT

Use the bisection method
to determine the root
SOLUTION
f(x) = e
-x
- x
x
l
= -1 x
u
= 1
check if f(x
l
) f(x
u
) < 0
f(-1) f(1) = (3.72)(-0.632) < 0
x
r
= (x
l
+ x
u
) / 2 = 0
f(0) = 1 exchange so that x
l
= 0
x
l
= 0 x
u
= 1
Solution cont.
x
l
= 0 x
u
= 1 SWITCH LOWER LIMIT
check if f(x
l
) f(x
u
) < 0
f(0) f(1) = (1) (-0.632) < 0
x
r
= (x
l
+ x
u
) / 2 = (0 + 1)/2 = 0.5
f (0.5) = 0.1065
x
l
= 0.5 x
u
= 1 SWITCH LOWER LIMIT
x
r
= (x
l
+ x
u
) / 2 = (0.5 + 1)/2 = 0.75
f (0.75) = -0.2776
Solution cont.
x
l
= 0.5 x
u
= 0.75 SWITCH UPPER LIMIT
x
r
= (x
l
+ x
u
) / 2 = (0.5 + 0.75)/2 = 0.625
f (0.625) = -0.090
x
l
= 0.5 x
u
= 0.625 SWITCH UPPER LIMIT
x
r
= (0.5

+ 0.625) / 2 = 0.5625
f (0.5625) = 0.007
Solution cont.
Here we consider an error that is not
contingent on foreknowledge of the root
c
a
= f (present and previous approx.)
-10
-5
0
5
10
15
-5 0 5 10
x
f
(
x
)
f(x) = e
-x
- x
False Position Method
Brute Force of bisection method is
inefficient
Join points by a straight line
Improves the estimate
Estimating the curve by a straight line gives
the false position
x
l
x
u
f(x
l
)
f(x
u
)
next estimate
real root
DEVELOP METHOD BASED ON
SIMILAR TRIANGLES
x
l
x
u
f(x
l
)
f(x
u
) next estimate, x
r
( ) ( )
( )( )
( ) ( )
u l
u l u
u r
u r
u
l r
l
x f x f
x x x f
x x
x x
x f
x x
x f
Based on
similar
triangles
Example
f(x) = x
3
- 98
x = 98
1/3
= 4.61
x
l
= 4.55 f(x
l
) = -3.804
x
u
= 4.65 f(x
u
) = 2.545
x
r
= 4.65 - (2.545)(4.55-4.65)/(-3.804-2.545)
x
r
= 4.6099 f(x
r
) = -0.03419
if f(x
l
)f(x
r
) > 0 x
l
= x
r
if f(x
l
)f(x
r
) < 0 x
u
= x
r
Example (continued)
x
l
= 4.61 f(x
l
) = -0.034
x
u
= 4.65 f(x
u
) = 2.545
x
r
= 4.610
x
r
= 4.6104 f(x
r
) = -0.0004
c
a
= 0.011%
Pitfalls of False Position Method
f(x) = x
10
- 1
-5
0
5
10
15
0 0.5 1 1.5
x
f
(
x
)
Open Methods
Fixed point iteration
Newton-Raphson method
Secant method
Multiple roots
In the previous bracketing methods, the root
is located within an interval prescribed by
an upper and lower boundary
Open Methods cont.
Such methods are said to be convergent
solution moves closer to the root as the
computation progresses
Open method
single starting value
two starting values that do not necessarily
bracket the root
These solutions may diverge
solution moves farther from the root as the
computation progresses
Pick initial
estimate x
i
x
i
f(x)
x
f(x
i
)
draw a tangent
at f(x
i
)
x
i
f(x)
x
f(x
i
)
At the intersection
with the x-axis
we get x
i+1
and
f(x
i+1
)
f(x)
x
x
i+1
f(x
i+1
)
The tangent
gives next
estimate.
x
i
f(x)
x
f(x
i
)
x
i+1
f(x
i+1
)
Solution can overshoot
the root and potentially
diverge
x
0
f(x)
x
x
1
Solution can overshoot
the root and potentially
diverge
x
0
f(x)
x
x
1
x
2
Fixed point iteration
Open methods employ a formula to predict
the root
In simple fixed point iteration, rearrange the
function f(x) so that x is on the left hand
side of the equation
i.e. for f(x) = x
2
- 2x + 3 = 0
x = (x
2
+ 3) / 2
Simple fixed point iteration
In simple fixed point iteration, rearrange the
function f(x) so that x is on the left hand
side of the equation
i.e. for f(x) = sin x = 0
x = sin x + x
Let x = g(x)
New estimate based on
x
i+1
= g(x
i
)
Example
Consider f(x) = e
-x
-3x
g(x) = e
-x
/ 3
Initial guess x = 0
f(x) = e
-x
-3x
-15
-10
-5
0
5
10
15
-4 -2 0 2 4 6
x
f
(
x
)
Initial guess 0.000
g(x) f(x) c
a
0.333 -0.283
0.239 0.071 39.561
0.263 -0.018 9.016
0.256 0.005 2.395
0.258 -0.001 0.612
0.258 0.000 0.158
0.258 0.000 0.041
f(x) = e
-x
-3x
-15
-10
-5
0
5
10
15
-4 -2 0 2 4 6
x
f
(
x
)
Newton Raphson
most widely used
x
i+2
f(x)
x
Note how the
new estimates
converge
i.e. x
i+2

is closer to the
root, f(x) = 0.
Newton Raphson
( )
( )
( )
( )
i
i
i 1 i
1 i i
i
i
x ' f
x f
x x
rearrange
x x
0 x f
x ' f
' f
dx
dy
gent tan
=

=
= =
+
+
f(x
i
)
x
i
tangent
x
i+1
Newton Raphson Pitfalls
f(x)
(x)
Newton Raphson Pitfalls
f(x)
(x)
solution diverges
Example
f(x) = x
2
- 11
f '(x) = 2x
initial guess x
i
= 3
f(3) = -2
f '(3) = 6
f(x) = x
2
- 11
-20
0
20
40
60
80
100
0 5 10
x
f
(
x
)
Secant method
( )
( ) ( )
i 1 i
i 1 i
x x
x f x f
x ' f
Approximate derivative using a finite divided difference

What is this? HINT: dy / dx = lim Ay / Ax

Substitute this into the formula for Newton Raphson
Secant method
( )
( )
( )( )
( ) ( )
i 1 i
i 1 i i
i 1 i
i
i
i 1 i
x f x f
x x x f
x x
x ' f
x f
x x

=
=
+
+
Substitute finite difference
approximation for the
first derivative into this
equation for Newton
Raphson
Secant method
Requires two initial estimates
f(x) is not required to change signs, therefore this
is not a bracketing method
( )( )
( ) ( )
i 1 i
i 1 i i
i 1 i
x f x f
x x x f
x x
+
Secant method
new estimate
{

initial estimates
slope
between
two
estimates
f(x)
x
Example
Lets consider f(x) = e
-x
- x
-10
-5
0
5
10
15
-5 0 5 10
x
f
(
x
)
f(x) = e
-x
- x
Example cont.
Choose two starting points
x
0
= 0 f(x
0
) =1
x
1
= 1.0 f(x
1
) = -0.632
Calculate x
2
x
2
= 1 - (-0.632)(0 - 1)/(1+0.632) = 0.6127
x
f(x)
1
2
new est.
x
f(x)
1
new est.
2
FALSE POSITION
SECANT METHOD
The new estimate
is selected from the
intersection with the
x-axis
Multiple Roots
Corresponds to a point where
a function is tangential to the
x-axis
i.e. double root
f(x) = x
3
- 5x
2
+ 7x -3
f(x) = (x-3)(x-1)(x-1)
i.e. triple root
f(x) = (x-3)(x-1)
3
f(x) = (x-3)(x-1)
2
-4
-2
0
2
4
6
8
10
0 1 2 3 4 5
x
f
(
x
)
double root
Difficulties
Bracketing methods wont work
Limited to methods that may diverge
f(x) = (x-3)(x-1)
2
-4
-2
0
2
4
6
8
10
0 1 2 3 4 5
x
f
(
x
)
double root
f(x) = 0 at root
f '(x) = 0 at root
Hence, zero in the
denominator for
Newton-Raphson
and Secant
Methods

f(x) = (x-3)(x-1)
3
-3
-2
-1
0
1
2
3
4
5
6
0 1 2 3 4
x
f
(
x
)
triple root
Multiple Roots
( ) ( )
( ) | | ( ) ( )
i i
2
i
i i
i 1 i
x ' ' f x f x ' f
x ' f x f
x x
=
+
f(x) = (x-3)(x-1)
4
-4
-2
0
2
4
6
8
10
12
14
16
0 1 2 3 4
x
f
(
x
)
quadruple
root
Systems of Non-Linear Equations
We will later consider systems of linear
equations
f(x) = a
1
x
1
+ a
2
x
2
+...... a
n
x
n
- C = 0
where a
1
, a
2
.... a
n
and C are constant
Consider the following equations
y = -x
2
+ x + 0.5
y + 5xy = x
3
Solve for x and y
Systems of Non-Linear Equations cont.
Set the equations equal to zero
y = -x
2
+ x + 0.5
y + 5xy = x
3
u(x,y) = -x
2
+ x + 0.5 - y = 0
v(x,y) = y + 5xy - x
3
= 0
The solution would be the values of x and y
that would make the functions u and v equal
to zero
Recall the Taylor Series
( ) ( ) ( )
( ) ( )
( )
i 1 i
n
n
i
n
......
3
i
2
i
i i 1 i
x x size step h where
R h
! n
x f
h
! 3
x ' ' ' f
h
! 2
x ' ' f
h x ' f x f x f
= =
+
+ + + ~
+
+
+
+
Write 2 Taylor series with respect
to u and v
( ) ( )
( ) ( ) HOT y y
y
v
x x
x
v
v v
HOT y y
y
u
x x
x
u
u u
i 1 i
i
i 1 i
i
i 1 i
i 1 i
i
i 1 i
i
i 1 i
+
c
c
+
c
c
+ =
+
c
c
+
c
c
+ =
+ + +
+ + +
The root estimate corresponds to the point where
u
i+1
= v
i+1
= 0
First Order Taylor Series
Approximation
( ) ( )
( ) ( )
i i 1 i
i
i 1 i
i
i i 1 i
i
i 1 i
i
v y y
y
v
x x
x
v
u y y
y
u
x x
x
u
=
c
c
+
c
c
=
c
c
+
c
c
+ +
+ +
Defining A x = x
i+1
x
i
& A y = y
i+1
y
i

Then in matrix form, the equations are
(
(
(
(
=
(
(
(
(
A
A
(
(
(
(
c
c
c
c
c
c
c
c
i
i
i i
i i
v
u
y
x
y
v
x
v
y
u
x
u
This can be solved for x
i+1
& y
i+1

This is a 2 equation version of Newton-Raphson
x
v
y
u
y
v
x
u
x
v
u
x
u
v
y y
x
v
y
u
y
v
x
u
y
u
v
y
v
u
x x
i i i i
i
i
i
i
i 1 i
i i i i
i
i
i
i
i 1 i
c
c
c
c
c
c
c
c
c
c
c
c
=
c
c
c
c
c
c
c
c
c
c
c
c
=
+
+
Therefore
THE DENOMINATOR
OF EACH OF THESE
EQUATIONS IS
FORMALLY
REFERRED TO
AS THE DETERMINANT
OF THE
JACOBIAN
This is a 2 equation version of Newton-Raphson
x
v
y
u
y
v
x
u
x
v
u
x
u
v
y y
x
v
y
u
y
v
x
u
y
u
v
y
v
u
x x
i i i i
i
i
i
i
i 1 i
i i i i
i
i
i
i
i 1 i
c
c
c
c
c
c
c
c
c
c
c
c
=
c
c
c
c
c
c
c
c
c
c
c
c
=
+
+
Example
Determine the roots of the following
nonlinear simultaneous equations
y = -x
2
+ x + 0.5
y + 5xy = x
3
u(x,y) = -x
2
+ x + 0.5 - y = 0
v(x,y) = y + 5xy - x
3
= 0
Use an initial estimate of x = 0, y =1
Or alternately
i i
i i
i 1 i i
i i i i
i i
i i
i 1 i i
i i i i
v u
u v
y y
x x x
u v u v
x y y x
u v
v u
x x
y y y
u v u v
x y y x
+
+
c c
c c
= A =
c c c c
c c c c
c c
c c
= A =
c c c c
c c c c
THE
SAME
DETERMINANT
OF THE
JACOBIAN
This is the form of the equations
Example cont.
x 5 1
y
v
x 3 y 5
x
v
1
y
u
1 x 2
x
u
2
+ =
c
c
=
c
c
=
c
c
+ =
c
c
First iteration: [J] = 6
x = -0.08333
y = 0.41667
System of Linear Equations
We have focused our last lectures on
finding a value of x that satisfied a single
equation
f(x) = 0
Now we will deal with the case of
determining the values of x
1
, x
2
, .....x
n
, that
simultaneously satisfy a set of equations
System of Linear Equations
Simultaneous equations
f
1
(x
1
, x
2
, .....x
n
) = 0
f
2
(x
1
, x
2
, .....x
n
) = 0
.............
f
3
(x
1
, x
2
, .....x
n
) = 0
Methods will be for linear equations
a
11
x
1
+ a
12
x
2
+...... a
1n
x
n
=c
1
a
21
x
1
+ a
22
x
2
+...... a
2n
x
n
=c
2

..............

a
n1
x
1
+ a
n2
x
2
+...... a
nn
x
n
=c
n
Mathematical Background
Matrix Notation
a horizontal set of elements is called a row
a vertical set is called a column
first subscript refers to the row number
second subscript refers to column number
| |
(
(
(
(
=
mn 3 m 2 m 1 m
n 2 23 22 21
n 1 13 12 11
a ... a a a
. . . .
a ... a a a
a ... a a a
A
This matrix has m rows an n column.

It has the dimensions m by n (m x n)
note
subscript
| |
(
(
(
(
=
mn 3 m 2 m 1 m
n 2 23 22 21
n 1 13 12 11
a ... a a a
. . . .
a ... a a a
a ... a a a
A
| |
(
(
(
(
=
mn 3 m 2 m 1 m
n 2 23 22 21
n 1 13 12 11
a ... a a a
. . . .
a ... a a a
a ... a a a
A
row 2
column 3
Note the consistent
scheme with subscripts
denoting row,column
Row vector: m=1
Column vector: n=1 Square matrix: m = n
| | | |
n 2 1
b ....... b b B =
| |
(
(
(
(
(
(
=
m
2
1
c
.
.
c
c
C
| |
(
(
(
=
33 32 31
23 22 21
13 12 11
a a a
a a a
a a a
A
| |
(
(
(
=
33 32 31
23 22 21
13 12 11
a a a
a a a
a a a
A
The diagonal consists of the elements

a
11
a
22
a
33

Symmetric matrix
Diagonal matrix
Identity matrix
Upper triangular matrix
Lower triangular matrix
Banded matrix
Symmetric Matrix

a
ij
= a
ji
for all is and js
| |
(
(
(
=
8 7 2
7 3 1
2 1 5
A
Does a
23
= a
32
?
Yes. Check the other elements
on your own.
Diagonal Matrix

A square matrix where all elements off
the main diagonal are zero
| |
(
(
(
(
=
44
33
22
11
a 0 0 0
0 a 0 0
0 0 a 0
0 0 0 a
A
Identity Matrix

A diagonal matrix where all elements on
the main diagonal are equal to 1
| |
(
(
(
(
=
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
A
The symbol [I] is used to denote the identify matrix.
Upper Triangle Matrix

Elements below the main diagonal are
zero
| |
(
(
(
=
33
23 22
13 12 11
a 0 0
a a 0
a a a
A
Lower Triangular Matrix

All elements above the main diagonal
are zero

| |
(
(
(
=
8 7 2
0 3 1
0 0 5
A
Banded Matrix

All elements are zero with the exception
of a band centered on the main diagonal

| |
(
(
(
(
=
44 43
34 33 32
23 22 21
12 11
a a 0 0
a a a 0
0 a a a
0 0 a a
A
Matrix Operating Rules
Addition/subtraction
add/subtract corresponding terms
a
ij
+ b
ij
= c
ij

Addition/subtraction are commutative
[A] + [B] = [B] + [A]
Addition/subtraction are associative
[A] + ([B]+[C]) = ([A] +[B]) + [C]
Multiplication of a matrix [A] by a scalar g
is obtained by multiplying every element of
[A] by g
| | | |
(
(
(
(
(
(
(
(
= =
mn 2 m 1 m
n 2 22 21
n 1 12 11
ga . . . ga ga
. . . . . .
. . . . . .
. . . . . .
ga . . . ga ga
ga . . . ga ga
A g B
The product of two matrices is represented as
[C] = [A][B]

n = column dimensions of [A]
n = row dimensions of [B]
=
=
n
1 k
kj ik ij
b a c
Simple way to check whether
matrix multiplication is possible
[A] m x n [B] n x k = [C] m x k
interior dimensions
must be equal
exterior dimensions conform to dimension of resulting matrix
Matrix multiplication
If the dimensions are suitable, matrix
multiplication is associative
([A][B])[C] = [A]([B][C])
If the dimensions are suitable, matrix
multiplication is distributive
([A] + [B])[C] = [A][C] + [B][C]
Multiplication is generally not commutative
[A][B] is not equal to [B][A]
Inverse of [A]
Transpose of [A]
| || | | | | | | | I A A A A
1 1
= =

| |
(
(
(
(
(
(
(
(
=
mn n 2 n 1
2 m 22 12
1 m 21 11
t
a . . . a a
. . . . . .
. . . . . .
. . . . . .
a . . . a a
a . . . a a
A
bc ad
d c
b a
=
(
Determinants
Denoted as det A or A

for a 2 x 2 matrix
bc ad
d c
b a
=
(
Determinants cont.
There are different schemes used to compute the determinant.

Consider cofactor expansion
- uses minor and cofactors of the matrix

Minor: the minor of an entry a
ij
is the determinant of the
submatrix obtained by deleting the ith row and the jth
column

Cofactor: the cofactor of an entry a
ij
of an n x n matrix
A is the product of (-1)
i+j
and the minor of a
ij
(
(
(
=
33 32 31
23 22 21
13 12 11
32
a a a
a a a
a a a
a of minor
Minor: the minor of an entry a
ij
is the determinant of the
submatrix obtained by deleting the ith row and the jth
column.

Example: the minor of a
32
for a 3x3 matrix is:

13 22 23 12
23 22
13 12
33 32 31
23 22 21
13 12 11
31
a a a a
a a
a a
a a a
a a a
a a a
a of minor = =
(
(
(
=
Cofactor: A
ij
, the cofactor of an entry a
ij
of an n x n matrix
A is the product of (-1)
i+j
and the minor of a
ij

i.e. Calculate A
31
for a 3x3 matrix

First calculate the minor a
31
Minors and cofactors are used to calculate the determinant
of a matrix.

Consider an n x n matrix expanded around the ith row
in i2 i1
A A A
in 2 i 1 i
a ...... a a A + + =
Consider expanding around the jth column
nj 2j 1j
A A A
nj j 2 j 1
a ...... a a A + + =
(for any one value of i)
(for any one value of j)
32 31
22 21
13
33 31
23 21
12
33 32
23 22
11
33 32 31
23 22 21
13 12 11
a a
a a
a
a a
a a
a
a a
a a
a
a a a
a a a
a a a
D + = =
( ) ( ) ( ) ( )
( ) ( )
31 22 32 21 13
3 1
31 23 33 21 12
2 1
32 23 33 22 11
1 1
a a a a a 1
a a a a a 1 a a a a a 1 det
+
+ =
+
+ +
Example:
Calculate the determinant of the following 3x3 matrix.
First, calculate it using the 1st row (the way you
probably have done it all along).
Then try it using the 2nd row.

(
(
(
5 1 6
2 3 4
9 7 1
Properties of Determinants
det A = det A
T

If all entries of any row or column are zero,
then det A = 0
If two rows or two columns are identical,
then det A = 0
How to represent a system of
linear equations as a matrix
[A]{X} = {C}

where {X} and {C} are both column vectors
| |
(
(
(
=
= + +
= + +
= + +
44 . 0
67 . 0
01 . 0
x
x
x
5 . 0 3 . 0 1 . 0
9 . 1 1 5 . 0
1 52 . 0 3 . 0
} C { } X { A
44 . 0 x 5 . 0 x 3 . 0 x 1 . 0
67 . 0 x 9 . 1 x x 5 . 0
01 . 0 x x 52 . 0 x 3 . 0
3
2
1
3 2 1
3 2 1
3 2 1
Graphical Method
2 equations, 2 unknowns
22
2
1
22
21
2
12
1
1
12
11
2
2 2 22 1 21
1 2 12 1 11
a
c
x
a
a
x
a
c
x
a
a
x
c x a x a
c x a x a
+
|
|
.
|
\
|
=
+
|
|
.
|
\
|
=
= +
= +
x
2
x
1
( x
1
, x
2
)
1 x
2
1
x
9 x
2
3
x
2 x 2 x
18 x 2 x 3
1 2
1 2
2 1
2 1
+
|
.
|
\
|

=
+
|
.
|
\
|
=
= +
= +
x
2
x
1
( 4 , 3 )
3
2
2
1
9
1
Check: 3(4) + 2(3) = 12 + 6 = 18
Special Cases
No solution
Infinite solution
Ill-conditioned
x
2
x
1
( x
1
, x
2
)
a) No solution - same slope
f(x)
x
b) infinite solution

f(x)
x
-1/2 x
1
+ x
2
= 1
-x
1
+2x
2
= 2
c) ill conditioned
so close that the points of
intersection are difficult to
detect visually
f(x)
x
Lets consider how we know if the system is
ill-conditions. Start by considering systems
where the slopes are identical
If the determinant is zero, the slopes are
identical

2 2 22 1 21
1 2 12 1 11
c x a x a
c x a x a
= +
= +
Rearrange these equations so that we have an
alternative version in the form of a straight
line: i.e. x
2
= (slope) x
1
+ intercept

22
2
1
22
21
2
12
1
1
12
11
2
a
c
x
a
a
x
a
c
x
a
a
x
+ =
+ =
If the slopes are nearly equal (ill-conditioned)
0 a a a a
a a a a
a
a
a
a
12 21 22 11
12 21 22 11
22
21
12
11
~
~
~
A det
a a
a a
22 21
12 11
=
Isnt this the determinant?
If the determinant is zero the slopes are equal.
This can mean:
- no solution
- infinite number of solutions

If the determinant is close to zero, the system is ill
conditioned.

So it seems that we should use check the determinant
of a system before any further calculations are done.

Lets try an example.

Example
Determine whether the following matrix is ill-conditioned.

)
`
=
)
`
12
22
x
x
5 . 2 2 . 19
7 . 4 2 . 37
2
1
Solution
( )( ) ( )( )
76 . 2
2 . 19 7 . 4 5 . 2 2 . 37
5 . 2 2 . 19
7 . 4 2 . 37
=
=
What does this tell us? Is this close to zero? Hard to say.

If we scale the matrix first, i.e. divide by the largest
a value in each row, we can get a better sense of things.
-80
-60
-40
-20
0
0 5 10 15
x
y
This is further justified
when we consider a graph
of the two functions.

Clearly the slopes are
nearly equal
1 0126
1 0130
0 004
.
.
. =
Cramers Rule
Not efficient for solving large numbers of
linear equations
Useful for explaining some inherent
problems associated with solving linear
equations.
| |{ } { } b x A
b
b
b
x
x
x
a a a
a a a
a a a
3
2
1
3
2
1
33 32 31
23 22 21
13 12 11
=
(
(
(
Cramers Rule
to solve for
x
i
- place {b} in
the ith column
33 32 3
23 22 2
13 12 1
1
a a b
a a b
a a b
A
1
x =
33 32 3
23 22 2
13 12 1
1
a a b
a a b
a a b
A
1
x =
Cramers Rule
to solve for
x
i
- place {b} in
the ith column
Cramers Rule
to solve for
x
i
- place {b} in
the ith column
3 32 31
2 22 21
1 12 11
3
33 3 31
23 2 21
13 1 11
2
33 32 3
23 22 2
13 12 1
1
b a a
b a a
b a a
A
1
x
a b a
a b a
a b a
A
1
x
a a b
a a b
a a b
A
1
x
=
= =
Example: Use of Cramers Rule
)
`
=
)
`

= +
=
5
5
x
x
1 1
3 2
5 x x
5 x 3 x 2
2
1
2 1
2 1
Note the substitution
of {c} in [A]
( )( ) ( )( )
( )( ) ( )( ) | |
( )( ) ( )( ) | | 1
5
5
1 5 5 2
5
1
5 1
5 2
5
1
x
4
5
20
5 3 1 5
5
1
1 5
3 5
5
1
x
5 3 2 1 3 1 2 A
5
5
x
x
1 1
3 2
2
1
2
1
= =
|
.
|
\
|
= =
= =
|
.
|
\
|
=
=
= + = =
)
`
=
)
`

Elimination of Unknowns
( algebraic approach)
( )
( )
2 11 2 22 11 1 11 21
1 21 2 12 21 1 11 21
11 2 2 22 1 21
21 1 2 12 1 11
2 2 22 1 21
1 2 12 1 11
c a x a a x a a
SUBTRACT c a x a a x a a
a c x a x a
a c x a x a
c x a x a
c x a x a
= +
= +
= +
= +
= +
= +
Elimination of Unknowns
( algebraic approach)
21 12 22 11
2 12 1 22
1
11 22 21 12
2 11 1 21
2
11 2 21 1 2 11 22 2 21 12
2 11 2 22 11 1 11 21
1 21 2 12 1 ` 2 1 11 21
a a a a
c a c a
x
a a a a
c a c a
x
a c a c x a a x a a
c a x a a x a a
SUBTRACT c a x a a x a a
=
=
= +
= +
NOTE: same result as
Cramers Rule
Gauss Elimination
One of the earliest methods developed for
solving simultaneous equations
Important algorithm in use today
Involves combining equations in order to
eliminate unknowns
Blind (Naive) Gauss Elimination
Technique for larger matrix
Same principles of elimination
manipulate equations to eliminate an unknown
from an equation
Solve directly then back-substitute into one of
the original equations
Two Phases of Gauss Elimination
(
(
(
(
(
(
' '
3
' '
33
'
2
'
23
'
22
1 13 12 11
3 33 32 31
2 23 22 21
1 13 12 11
c | a 0 0
c | a a 0
c | a a a
c | a a a
c | a a a
c | a a a
Forward
Elimination

Note: the prime indicates
the number of times the
element has changed from
the original value. Also,
the extra column for cs
makes this matrix form
what is called augmented.
Two Phases of Gauss Elimination
( )
( )
11
3 13 2 12 1
1
'
22
3
1
23
'
2
2
' '
33
' '
3
3
' '
3
' '
33
'
2
'
23
'
22
1 13 12 11
a
x a x a c
x
a
x a c
x
a
c
x
c | a 0 0
c | a a 0
c | a a a

=
=
=
(
(
(
Back substitution
Example
3 x 9 x 5 x 2
1 x 7 x 4 x 4
1 x 3 x x 2
3 2 1
3 2 1
3 2 1
= + +
= + +
= + +
( ) ( ) ( ) ( )
1 x x 2 x 0
2 1 x 6 7 x 2 4 x 4 4
3 2 1
3 2 1
= + +
= + +
2 3 1
4 4 7 1
2 5 9 3
1 2 3
1 2 3
1 2 3
x x x
x x x
x x x
+ + =
+ + =
+ + =
3 x 9 x 5 x 2
1 x 7 x 4 x 4
2 x 6 x 2 x 4
3 2 1
3 2 1
3 2 1
= + +
= + +
= + +
a
21
/ a
11
= 4/2 = 2 (called pivot factor for row 2)
Subtract the (temporary) revised first equation from
the second equation

Multiply equation 1 by a
31
/a
11
= 2/2 = 1
and subtract it from equation 3
( ) ( ) ( ) ( )
2 x 6 x 4 x 0
1 3 x 3 9 x 1 5 x 2 2
3 2 1
3 2 1
= + +
= + +
The equivalent matrix algorithm for calculating
the revised entries in row 3 (a
31
, a
32
, a
33
, & a
34
) is
note that a
31
can be assumed 0
k k
k i
j k j i
'
j i
a
a
a a a =
Where i is the row being modified 3 in this case
j is the column in row i 1,2,3, & 4 in this case
k is index of the x eliminated 1 in this case
2 x 6 x 4
1 x x 2
1 x 3 x x 2
3 x 9 x 5 x 2
1 x 7 x 4 x 4
1 x 3 x x 2
3 2
3 2
3 2 1
3 2 1
3 2 1
3 2 1
= +
= +
= + +
= + +
= + +
= + +
We now replace equations 2 and 3 in the revised matrix
(Note that 1st eq. is still original)
Continue the
computation
by multiplying
the second equation
by a
32
/a
22
= 4/2 =2

4 x 4
1 x x 2
1 x 3 x x 2
2 x 6 x 4
1 x x 2
1 x 3 x x 2
3
3 2
3 2 1
3 2
3 2
3 2 1
=
= +
= + +
= +
= +
= + +
THIS DERIVATION OF
AN UPPER TRIANGULAR MATRIX
IS CALLED THE FORWARD
ELIMINATION PROCESS
From the system we immediately calculate:
1
4
4
x
3
= =
Continue to back substitute
4 x 4
1 x x 2
1 x 3 x x 2
3
3 2
3 2 1
=
= +
= + +
( )
2
1
2
1 3 1
x
1
2
1 1
x
1
2
=

=
=

=
THIS SERIES OF
STEPS IS THE
BACK
SUBSTITUTION
Pitfalls of the Elimination
Method
Division by zero
Round off errors
magnitude of the pivot element is small
compared to other elements
Ill conditioned systems
Division by Zero
When we normalize i.e. a
12
/a
11
we need to
make sure we are not dividing by zero
This may also happen if the coefficient is
very close to zero
5 x 6 x x 2
3 x 7 x 6 x 4
8 x 3 x 2
3 2 1
3 2 1
3 2
= + +
= + +
= +
Techniques for Improving the
Solution
Use of more significant figures
Pivoting
Scaling
| |{ } { } b x A
b
b
b
x
x
x
a a a
a a a
a a a
3
2
1
3
2
1
33 32 31
23 22 21
13 12 11
=
(
(
(
Use of more significant figures

Simplest remedy for ill conditioning
Extend precision
computational overhead
memory overhead
Pivoting
Problems occur when the pivot element is
zero - division by zero
Problems also occur when the pivot element
is smaller in magnitude compared to other
elements (i.e. round-off errors)
Prior to normalizing, determine the largest
available coefficient
Pivoting
Partial pivoting
rows are switched so that the largest element is
the pivot element
Complete pivoting
columns as well as rows are searched for the
largest element and switched
rarely used because switching columns changes
the order of the xs, adding unjustified
complexity to the computer program
Division by Zero - Solution
Pivoting has been developed
to partially avoid these problems
5 x 6 x x 2
8 x 3 x 2
3 x 7 x 6 x 4
3 2 1
3 2
3 2 1
= + +
= +
= + +
5 x 6 x x 2
3 x 7 x 6 x 4
8 x 3 x 2
3 2 1
3 2 1
3 2
= + +
= + +
= +
Scaling
Minimizes round-off errors for cases where some
of the equations in a system have much larger
coefficients than others
In engineering practice, this is often due to the
widely different units used in the development of
the simultaneous equations
As long as each equation is consistent, the system
will be technically correct and solvable
Scaling
998 , 49 x 999 , 49
000 , 100 x 000 , 100 x 2
2 x x 2 x x
1 x x 00002 . 0 000 , 100 x 000 , 100 x 2
2
2 1
2 1 2 1
2 1 2 1
=
= +
= + = +
= + = +
1 x x 00 . 1 x 00 . 0 x
2 1 2 1
= = = =
P
i
v
o
t

r
o
w
s

t
o

p
u
t

t
h
e

g
r
e
a
t
e
s
t

v
a
l
u
e

o
n

t
h
e

d
i
a
g
o
n
a
l

EXAMPLE

Use Gauss Elimination to solve the following set
of linear equations
4 x 8 x 4
45 x x 6 x 2
50 x 13 x 3
3 1
3 2 1
3 2
= +
= +
=
4 x 8 x 4
45 x x 6 x 2
50 x 13 x 3
3 1
3 2 1
3 2
= +
= +
=
SOLUTION
First write in matrix form, employing short hand
presented in class.
0 3 13 50
2 6 1 45
4 0 8 4

(
(
(
We will clearly run into

problems of division
by zero.

Use partial pivoting
(
(
(

(
(
(
(
(
(

50 13 3 0
43 3 6 0
4 8 0 4
50 13 3 0
45 1 6 2
4 8 0 4
4 8 0 4
45 1 6 2
50 13 3 0
Begin developing
upper triangular matrix
( )
( )
( ) ( ) okay 50 966 . 1 13 149 . 8 3
CHECK
931 . 2
4
966 . 1 8 4
x
149 . 8
6
966 . 1 3 43
x 966 . 1
5 . 14
5 . 28
x
5 . 28 5 . 14 0 0
43 3 6 0
4 8 0 4
50 13 3 0
43 3 6 0
4 8 0 4
1
2 3
=
=
=
=
+
= =
=
(
(
(

(
(
(
...end of
problem
Matrix Inversion with Gauss-
Jordan Method
Gauss-Jordan (Like Gauss Elim.--Direct Sol.)
primary motive for introducing this method is that
it provides and simple and convenient method for
computing the matrix inverse
Gauss-Seidel (Ch. 11)
fundamentally different from Gauss elimination
this is an approximate, iterative method
particularly good for large number of equations
Gauss-Jordan
Variation of Gauss elimination
When an unknown is eliminated, it is eliminated
from all other equations, rather than just the
subsequent one
All rows are normalized by dividing them by their
pivot elements
Elimination step results in and identity matrix
rather than an UT matrix
| |
(
(
(
=
33
23 22
13 12 11
a 0 0
a a 0
a a a
A
| |
(
(
(
(
=
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
A
( )
( )
( )
( )
( )
( ) n
3 3
n
2 2
n
1
n
3
n
2
n
c x
c x
c x
c | 1 0 0
c | 0 1 0
c | 0 0 1
1
1
=
=
=
(
(
(
Graphical depiction of Gauss-Jordan

( )
( )
( )
(
(
(
(
(
(
n
3
n
2
n
' '
3
' '
33 32 31
'
2
'
23
'
22 21
1 13 12 11
c | 1 0 0
c | 0 1 0
c | 0 0 1
c | a a a
c | a a a
c | a a a
1
Matrix Inversion
[A] [A]
-1
= [A]
-1
[A] = I
One application of the inverse is to solve
several systems differing only by {c}
[A]{x} = {c}
[A]
-1
[A] {x} = [A]
-1
{c}
[I]{x}={x}= [A]
-1
{c}
One quick method to compute the inverse is
to augment [A] with [I] instead of {c}
Graphical Depiction of the Gauss-Jordan
Method with Matrix Inversion

| | | |
| | | |
1
1
33
1
32
1
31
1
23
1
22
1
21
1
13
1
12
1
11
33 32 31
23 22 21
13 12 11
A I
a a a 1 0 0
a a a 0 1 0
a a a 0 0 1
1 0 0 a a a
0 1 0 a a a
0 0 1 a a a
I A

(
(
(
(
(
(
Note: the superscript

-1 denotes that
the original values
have been converted
to the matrix inverse,
not 1/a
ij
EXAMPLE

Use the Gauss-Jordan Method to solve the
following problem.
Note: You can use this solution to practice Gauss
elimination method.
65 x 5 x 5
29 x 7 x 2
28 x 6 x 5 x 4
2 1
3 1
3 2 1
=
=
= +
(
(
(
65 0 5 5
29 7 0 2
28 6 5 4
65 x 5 x 5
29 x 7 x 2
28 x 6 x 5 x 4
2 1
3 1
3 2 1
=
=
= +
SOLUTION
First, write the coefficients
and the right hand vector
as an augmented matrix
4 5 6 28
2 0 7 29
5 5 0 65
1 125 15 7
2 0 7 29
5 5 0 65
(
(
(
(
(
(
. .
Normalize the first row by
dividing by the pivot
element 4
Example: -6/4 = -1.5
1 125 15 7
2 0 7 29
5 5 0 65
. .
(
(
(
The x
1
term in the second
row can be eliminated by
subtracting 2 times the
first row from the second row
In other words, we want this
term to be zero
1 125 15 7
2 0 7 29
5 5 0 65
1 125 15 7
5 5 0 65
. .
. .
? ? ? ?
(
(
(
(
(
(
The x
1
term in the second
All values
in this row
will change
1 125 15 7
2 0 7 29
5 5 0 65
1 125 15 7
5 5 0 65
. .
. .
?
(
(
(
(
(
(
The x
1
term in the second
2 - (1)
1 125 15 7
2 0 7 29
5 5 0 65
1 125 15 7
5 5 0 65
. .
. .
?
(
(
(
(
(
(
The x
1
term in the second
2
1 125 15 7
2 0 7 29
5 5 0 65
1 125 15 7
5 5 0 65
. .
. .
?
(
(
(
(
(
(
The x
1
term in the second
2 - (1)(2)
1 125 15 7
2 0 7 29
5 5 0 65
1 125 15 7
0
5 5 0 65
. .
. .
(
(
(
(
(
(
The x
1
term in the second
2 - (1)(2) = 0
1 125 15 7
2 0 7 29
5 5 0 65
1 125 15 7
0
5 5 0 65
. .
. .
?
(
(
(
(
(
(
The x
1
term in the second
0
(
(
(
(
(
(
65 0 5 5
? 0
7 5 . 1 25 . 1 1
65 0 5 5
29 7 0 2
7 5 . 1 25 . 1 1
The x
1
term in the second
0 - (2)
(
(
(
(
(
(
65 0 5 5
? 0
7 5 . 1 25 . 1 1
65 0 5 5
29 7 0 2
7 5 . 1 25 . 1 1
The x
1
term in the second
0 - (2)(1.25)
(
(
(

(
(
(
65 0 5 5
5 . 2 0
7 5 . 1 25 . 1 1
65 0 5 5
29 7 0 2
7 5 . 1 25 . 1 1
The x
1
term in the second
0 - (2)(1.25) = -2.5
(
(
(

(
(
(
65 0 5 5
15 4 5 . 2 0
7 5 . 1 25 . 1 1
65 0 5 5
29 7 0 2
7 5 . 1 25 . 1 1
The x
1
term in the second
These term are derived
following the same algorithm.
(
(
(

(
(
(

30 5 . 7 25 . 1 0
15 4 5 . 2 0
7 5 . 1 25 . 1 1
65 0 5 5
15 4 5 . 2 0
7 5 . 1 25 . 1 1
The x
1
term in the third
subtracting -5 times the
Now we need to reduce the x
2
term from
the 1st and 3rd equation

(
(
(

30 5 . 7 25 . 1 0
15 4 5 . 2 0
7 5 . 1 25 . 1 1

2
term from

(
(
(

30 5 . 7 25 . 1 0
15 4 5 . 2 0
7 5 . 1 25 . 1 1

2
term from

Normalize the second row by a
22
(
(
(

30 5 . 7 25 . 1 0
15 4 5 . 2 0
7 5 . 1 25 . 1 1

2
term from

22
(
(
(

30 5 . 7 25 . 1 0
15 4 5 . 2 0
7 5 . 1 25 . 1 1
1 125 15 7
0 1 16 6
0 125 7 5 30
. .
.
. .
(
(
(

2
term from

22
1 125 15 7
0 1 16 6
0 125 7 5 30
. .
.
. .
(
(
(
-2.5/(-2.5) = 1
(
(
(

30 5 . 7 25 . 1 0
15 4 5 . 2 0
7 5 . 1 25 . 1 1
1 125 15 7
0 1 16 6
0 125 7 5 30
. .
.
. .
(
(
(

2
term from

22
15/(-2.5) = -6
(
(
(

30 5 . 7 25 . 1 0
15 4 5 . 2 0
7 5 . 1 25 . 1 1
(
(
(

(
(
(
30 5 . 7 25 . 1 0
6 6 . 1 1 0
0 1
30 5 . 7 25 . 1 0
6 6 . 1 1 0
7 5 . 1 25 . 1 1
1.25 - 1.25 (1) = 0

(
(
(

(
(
(
30 5 . 7 25 . 1 0
6 6 . 1 1 0
? 0 1
30 5 . 7 25 . 1 0
6 6 . 1 1 0
7 5 . 1 25 . 1 1
Before going any further

calculate the new coefficient
for a
13
(
(
(

(
(
(
30 5 . 7 25 . 1 0
6 6 . 1 1 0
? 0 1
30 5 . 7 25 . 1 0
6 6 . 1 1 0
7 5 . 1 25 . 1 1
Before going any further

calculate the new coefficient
for a
13

Your answer should
follow the following scheme
(
(
(

(
(
(
30 5 . 7 25 . 1 0
6 6 . 1 1 0
? 0 1
30 5 . 7 25 . 1 0
6 6 . 1 1 0
7 5 . 1 25 . 1 1
-1.5
(
(
(

(
(
(
30 5 . 7 25 . 1 0
6 6 . 1 1 0
? 0 1
30 5 . 7 25 . 1 0
6 6 . 1 1 0
7 5 . 1 25 . 1 1
-1.5 - 1.6
-1.5 - 1.6 (1.25)
The element to get the
zero at a
12
(
(
(

(
(
(
30 5 . 7 25 . 1 0
6 6 . 1 1 0
? 0 1
30 5 . 7 25 . 1 0
6 6 . 1 1 0
7 5 . 1 25 . 1 1
(
(
(

(
(
(
30 5 . 7 25 . 1 0
6 6 . 1 1 0
? 0 1
30 5 . 7 25 . 1 0
6 6 . 1 1 0
7 5 . 1 25 . 1 1
-1.5 - 1.6 (1.25)

Recall, 1.25
was need to establish
a zero at a
12
(
(
(
(
(
(
30 5 . 7 25 . 1 0
6 6 . 1 1 0
5 . 3 0 1
30 5 . 7 25 . 1 0
6 6 . 1 1 0
7 5 . 1 25 . 1 1
-1.5 - 1.6 (1.25) = -3.5

Follow the same scheme for c
1

and for the third row
(
(
(
(
(
(
(
(
(
5 . 22 5 . 9 0 0
6 6 . 1 1 0
5 . 14 5 . 3 0 1
30 5 . 7 25 . 1 0
6 6 . 1 1 0
5 . 14 5 . 3 0 1
30 5 . 7 25 . 1 0
6 6 . 1 1 0
7 5 . 1 25 . 1 1
(
(
(
(
(
(
(
(
(
5 . 22 5 . 9 0 0
6 6 . 1 1 0
5 . 14 5 . 3 0 1
30 5 . 7 25 . 1 0
6 6 . 1 1 0
5 . 14 5 . 3 0 1
30 5 . 7 25 . 1 0
6 6 . 1 1 0
7 5 . 1 25 . 1 1
1 0 35 14 5
0 1 16 6
0 0 9 5 22 5
1 0 0 22 79
0 1 0 9 79
0 0 1 2 37
(
(
(
(
(
(
. .
.
. .
.
.
.
Now we need
to reduce x
3

from the 1st and
2nd equation

In addition, we
will complete the
identity matrix
......end of problem
LU Decomposition Methods
Chapter 10
Two fundamentally different approaches for solving
systems of linear algebraic equations
Elimination methods
Gauss elimination
Gauss Jordan
LU Decomposition Methods
Iterative methods
Gauss Seidel
Jacobi

Naive LU Decomposition
[A]{x}={c}
Suppose this can be rearranged as an upper
triangular matrix with 1s on the diagonal
[U]{x}={d}
[A]{x}-{c}=0 [U]{x}-{d}=0
Assume that a lower triangular matrix exists
that has the property
[L]{[U]{x}-{d}}= [A]{x}-{c}
Naive LU Decomposition
[L]{[U]{x}-{d}}= [A]{x}-{c}
Then from the rules of matrix multiplication
[L][U]=[A]
[L]{d}={c}
[L][U]=[A] is referred to as the LU
decomposition of [A]. After it is
accomplished, solutions can be obtained
very efficiently by a two-step substitution
procedure
Consider how Gauss elimination can be
formulated as an LU decomposition

U is a direct product of forward
elimination step if each row is scaled by
the diagonal
| |
(
(
(
=
1 0 0
a 1 0
a a 1
U
23
13 12
Although not as apparent, the matrix [L] is also
produced during the step. This can be readily
illustrated for a three-equation system
(
(
(
3
2
1
3
2
1
33 32 31
23 22 21
13 12 11
c
c
c
x
x
x
a a a
a a a
a a a
The first step is to multiply row 1 by the factor
11
21
21
a
a
f =
Subtracting the result from the second row eliminates a
21
(
(
(
3
2
1
3
2
1
33 32 31
23 22 21
13 12 11
c
c
c
x
x
x
a a a
a a a
a a a
Similarly, row 1 is multiplied by
11
31
31
a
a
f =
The result is subtracted from the third row to eliminate a
31
In the final step for a 3 x 3 system is to multiply the modified
row by
22
32
32
' a
' a
f =
Subtract the results from the third
row to eliminate a
32
The values f
21
, f
31
, f
32
are in fact the elements
of an [L] matrix
| |
(
(
(
=
1 f f
0 1 f
0 0 1
L
32 31
21
CONSIDER HOW THIS RELATES TO THE
LU DECOMPOSITION METHOD TO SOLVE
FOR {X}
[A] {x} = {c}
[U][L]
[L] {d} = {c}
{d}
[U]{x}={d}
{x}
Crout Decomposition
Gauss elimination method involves two
major steps
forward elimination
back substitution
Efforts in improvement focused on
development of improved elimination
methods
One such method is Crout decomposition
Crout Decomposition
Represents and efficient algorithm for decomposing [A]
into [L] and [U]
(
(
(
=
(
(
(
(
(
(
33 32 31
23 22 21
13 12 11
23
13 12
33 32 31
22 21
11
a a a
a a a
a a a
1 0 0
u 1 0
u u 1
0
0 0

Recall the rules of matrix multiplication.

The first step is to multiply the rows of [L] by the
first column of [U]
( )( ) ( )( ) ( )( )
31 31
21 21
11 11 11
a
a
0 0 0 0 1 a

=
=
= + + = Thus the first
column of [A]
is the first column
of [L]
(
(
(
=
(
(
(
(
(
(
33 32 31
23 22 21
13 12 11
23
13 12
33 32 31
22 21
11
a a a
a a a
a a a
1 0 0
u 1 0
u u 1
0
0 0

Next we multiply the first row of [L] by the columns

of [U] to get
13 13 11
12 12 11
11 11
a u
a u
a
=
=
=
(
(
(
=
(
(
(
(
(
(
33 32 31
23 22 21
13 12 11
23
13 12
33 32 31
22 21
11
a a a
a a a
a a a
1 0 0
u 1 0
u u 1
0
0 0

n ,..... 3 , 2 j for
a
u
a
u
a
u
a u
a u
a
11
j 1
j 1
11
13
13
11
12
12
13 13 11
12 12 11
11 11
= =
=
=
=
=
=
Once the first row of [U] is established

the operation can be represented concisely
(
(
(
=
(
(
(
(
(
(
33 32 31
23 22 21
13 12 11
23
13 12
33 32 31
22 21
11
a a a
a a a
a a a
1 0 0
u 1 0
u u 1
0
0 0

n ,..... 3 , 2 j for
a
u
11
j 1
j 1
= =
Schematic
depicting
Crout
Decomposition
(
(
(
=
(
(
(
(
(
(
33 32 31
23 22 21
13 12 11
23
13 12
33 32 31
22 21
11
a a a
a a a
a a a
1 0 0
u 1 0
u u 1
0
0 0

Schematic
depicting
Crout
Decomposition
=
=
+ + =
=
+ = =
=
= =
= =
1 n
1 k
kn nk nn nn
jj
1 j
1 i
ik ji jk
jk
1 j
1 k
kj ik ij ij
11
j 1
j 1
il 1 i
u a
n .... 2 j , 1 j k for
u a
u
n ,..... 1 j , j i for u a
1 n ...... 3 , 2 j For
n ,..... 3 , 2 j for
a
u
n ,....., 2 , 1 i for a

The Substitution Step

[L]{[U]{x}-{d}}= [A]{x}-{c}
[L][U]=[A]
[L]{d}={c}
[U]{x}={d}
Recall our earlier graphical depiction of the
LU decomposition method
[A] {x} = {c}
[U][L]
[L] {d} = {c}
{d}
[U]{x}={d}
{x}
| |{ } { }
1 ,..... 2 , 1
,...... 3 , 2
1
1
1
11
1
1
= =
=
=
=
=
=
+ =
=
n n i for x u d x
d x
d x U recall ution substit Back
n i for
d c
d
c
d
n
i j
j ij i i
n n
ii
i
j
j ij i
i
Example
Use LU decomposition to solve the following matrix
3 a
2 a
1 a
n ,....., 2 , 1 i for a
39
22
15
x
x
x
7 4 3
3 4 2
2 3 1
31 31
21 21
11 11
il 1 i
3
2
1
= =
= =
= =
= =
(
(
(
2
a
u
3
1
3 a
u
n ,..... 3 , 2 j for
a
u
11
13
13
11
12
12
11
j 1
j 1
= =
= = =
= =
( )
( ) 5 3 3 4 u a
2 3 2 4 u a
n ,..... 1 j , j i for u a
1 2
1 k
2 k k 3 32 32
1 2
1 k
2 k k 2 22 22
1 j
1 k
kj ik ij ij
= = =
= = =
+ = =
=

( )
5 . 0
2
2 2 3
u a
u
n .... 2 j , 1 j k for
u a
u
22
1 2
1 i
3 i i 2 23
23
jj
1 j
1 i
ik ji jk
jk
=
=
+ + =
( )( ) ( )( ) 5 . 3 5 . 0 5 2 3 7 u a
u a
1 3
1 k
3 k k 3 33 33
1 n
1 k
kn nk nn nn
= = =
=
=

Therefore the L and U matrices are
| | | |
(
(
(
=
(
(
(
=
1 0 0
5 . 0 1 0
2 3 1
U
5 . 3 5 3
0 2 2
0 0 1
L
| |{ } { }
15
1
15 c
d
c d L solving and
39
22
15
c
c
c
11
1
1
3
2
1
= = =
=
Recall the original column matrix

( )
( ) ( )( )
4
5 . 3
4 5 15 3 39
d c
d
4
2
15 2 22
d c
d
n ,...... 3 , 2 i for
d c
d
ii
1 3
1 j
j j 3 3
3
ii
1 2
1 j
j j 2 2
2
ii
1 i
1 j
j ij i
i
=

=
=
=
=
=
Forward substitution
Backsubstitution
( )
( ) ( ) 1 4 2 2 3 15 x u d x
2 4 5 . 0 4 x u d x
1 ,..... 2 n , 1 n i for x u d x
4 d x
d x
3
1 1 j
j j 1 1 1
3
1 2 j
j j 3 2 2
n
1 i j
j ij i i
3 3
n n
= = =
= = =
= =
= =
=
+ =
+ =
+ =
...end of example
Matrix Inversion
[A] [A]
-1
= [A]
-1
[A] = I
One application of the inverse is to solve
several systems differing only by {c}
[A]{x} = {c}
[A]
-1
[A] {x} = [A]
-1
{c}
[I]{x}={x}= [A]
-1
{c}
One quick method to compute the inverse is
to augment [A] with [I] instead of {c}
Graphical Depiction of the Gauss-Jordan
Method with Matrix Inversion

| | | |
| | | |
1
1
33
1
32
1
31
1
23
1
22
1
21
1
13
1
12
1
11
33 32 31
23 22 21
13 12 11
A I
a a a 1 0 0
a a a 0 1 0
a a a 0 0 1
1 0 0 a a a
0 1 0 a a a
0 0 1 a a a
I A

(
(
(
(
(
(
Note: the superscript

-1 denotes that
the original values
have been converted
to the matrix inverse,
not 1/a
ij
Matrix Inversion with LU
Decomposition Method
11 1(1) 11 1(2)
21 22 2(1) 21 22 2(2)
31 32 33 3(1) 31 32 33 3(2)
11 1(3)
21 22 2(3)
31 32 33 3(3)
0 0 d 1 0 0 d 0
0 d 0 0 d 1
d 0 d 0
0 0 d 0
0 d 0
d 1

( (

( (
= =
` ` ` `
( (

( (
) )
) )

(

(
=
`
(

(

)

`

)
Solving for unknowns using {d}
(1)
gives the first column of [I]
and so forth for {d}
(2)
through {d}
(n)

Chapter 11:Iterative Methods
Solution by Gauss-Seidel Iteration
Solution by Jacobi Iteration

Gauss Seidel Method
An iterative approach
Continue until we converge within some
pre-specified tolerance of error
Round off is no longer an issue, since you
control the level of error that is acceptable
Gauss-Seidel Method
If the diagonal elements are all nonzero, the
first equation can be solved for x
1

Solve the second equation for x
2
, etc.
11
n n 1 3 13 2 12 1
1
a
x a x a x a c
x

=

To assure that you understand this, write the equation for x
2
n , n
1 n 1 n , n 2 3 n 1 1 n n
n
33
n n 3 2 32 1 31 3
3
22
n n 2 3 23 1 21 2
2
11
n n 1 3 13 2 12 1
1
a
x a x a x a c
x
a
x a x a x a c
x
a
x a x a x a c
x
a
x a x a x a c
x

=

=

=

=
Gauss-Seidel Method
Start the solution process by guessing
values of x
A simple way to obtain initial guesses is to
assume that they are all zero
Calculate new values of x
i
starting with
x
1
= c
1
/a
11

Progressively substitute through the
equations
Repeat until tolerance is reached
( )
( )
( )
( )
( )
( )
3 33 2 32 1 31 3 3
2 22 23 1 21 2 2
1
11
1
11 13 12 1 1
33 2 32 1 31 3 3
22 3 23 1 21 2 2
11 3 13 2 12 1 1
' x a / ' x a ' x a c x
' x a / 0 a ' x a c x
' x
a
c
a / 0 a 0 a c x
a / x a x a c x
a / x a x a c x
a / x a x a c x
= =
= =
= = =
=
=
=
Gauss-Seidel Method for 3 eq. System
Example
Given the following augmented matrix,
complete one iteration of the Gauss
Seidel method.
2 3 1 2
4 1 2 2
3 2 1 1
(
(
(
2 3 1 2
4 1 2 2
3 2 1 1
(
(
(
( )
( )( ) ( )( )
( )
( )( ) ( )( )
( )
( )( ) ( )( )
10
1
12 3 1
1
6 2 1 3 1
x
' x a / ' x a ' x a c x
6
1
4 2
1
0 2 1 4 2
x
' x a / 0 a ' x a c x
1
2
2
2
0 1 0 3 2
x
' x
a
c
a / 0 a 0 a c x
3
3 33 2 32 1 31 3 3
2
2 22 23 1 21 2 2
1
1
11
1
11 13 12 1 1
=
+
=

=
= =
=

=

=
= =
= =

=
= = =
GAUSS SEIDEL
Gauss-Seidel Method
convergence criterion
s
j
i
1 j
i
j
i
i , a
100
x
x x
c <
= c

as in previous iterative procedures in finding the roots,
we consider the present and previous estimates.

As with the open methods we studied previously with one
point iterations

1. The method can diverge
2. May converge very slowly
Convergence criteria for two
linear equations
( )
( )
0
x
v
a
a
x
v
a
a
x
u
0
x
u
v and u of s derivative partial the consider
x
a
a
a
c
x , x v
x
a
a
a
c
x , x u
2 22
21
1
11
12
2 1
1
22
21
22
2
2 1
2
11
12
11
1
2 1
=
c
c
=
c
c
=
c
c
=
c
c
=
=
Class question:
where do these
formulas come from?
Convergence criteria for two linear
equations cont.
1
y
v
y
u
1
x
v
x
u
<
c
c
+
c
c
<
c
c
+
c
c
Criteria for convergence
in class text material
for nonlinear equations.

Noting that x = x
1
and
y = x
2

Substituting the previous equation:
equations cont.
1
a
a
1
a
a
11
12
22
21
< <
This is stating that the absolute values of the slopes must
be less than unity to ensure convergence.

Extended to n equations:
i j excluding n , 1 j where a a
ij ii
= = >

equations cont.
i j excluding n , 1 j where a a
ij ii
= = >
This condition is sufficient but not necessary; for convergence.

When met, the matrix is said to be diagonally dominant.
x
2
x
1
Review the concepts
of divergence and
convergence by graphically
illustrating Gauss-Seidel
for two linear equations
99 x 9 x 11 : v
286 x 13 x 11 : u
2 1
2 1
=
= +
x
2
x
1
Note: initial guess is
x
1
= x
2
= 0
286 x 13 x 11 : u
99 x 9 x 11 : v
2 1
2 1
= +
=
x
2
x
1
Note: initial guess is
x
1
= x
2
= 0
286 x 13 x 11 : u
99 x 9 x 11 : v
2 1
2 1
= +
=
u
v
x
2
x
1
286 x 13 x 11 : u
99 x 9 x 11 : v
2 1
2 1
= +
=
Solve 2
nd
eq. for x
2

v
u
x
2
x
1
286 x 13 x 11 : u
99 x 9 x 11 : v
2 1
2 1
= +
=
Solve 1st eq. for x
1

v
u
x
2
x
1
286 x 13 x 11 : u
99 x 9 x 11 : v
2 1
2 1
= +
=
Solve 2
nd
eq. for x
2

v
u
x
2
x
1
286 x 13 x 11 : u
99 x 9 x 11 : v
2 1
2 1
= +
=
Solve 1st eq. for x
1

v
u
x
2
x
1
286 x 13 x 11 : u
99 x 9 x 11 : v
2 1
2 1
= +
=
Solve 2
nd
eq. for x
2

v
u
x
2
x
1
Note: we are converging
on the solution
286 x 13 x 11 : u
99 x 9 x 11 : v
2 1
2 1
= +
=
v
u
x
2
x
1
Change the order of
the equations: i.e. change
direction of initial
estimates
99 x 9 x 11 : v
286 x 13 x 11 : u
2 1
2 1
=
= +
Solve 2
nd
eq. for x
2

u
v
x
2
x
1
99 x 9 x 11 : v
286 x 13 x 11 : u
2 1
2 1
=
= +
Solve 1st eq. for x
1

u
v
x
2
x
1
99 x 9 x 11 : v
286 x 13 x 11 : u
2 1
2 1
=
= +
Solve 2
nd
eq. for x
2

u
v
x
2
x
1
99 x 9 x 11 : v
286 x 13 x 11 : u
2 1
2 1
=
= +
Solve 1st eq. for x
1

u
v
x
2
x
1
This solution is diverging!
99 x 9 x 11 : v
286 x 13 x 11 : u
2 1
2 1
=
= +
u
v
Improvement of Convergence
Using Relaxation
This is a modification that will enhance slow convergence.

After each new value of x is computed, calculate a new value
based on a weighted average of the present and previous
iteration.
( )
old
i
new
i
new
i
x 1 x x + =
Improvement of Convergence Using
Relaxation
if = 1 unmodified
if 0 < < 1 underrelaxation
nonconvergent systems may converge
hasten convergence by dampening out oscillations
if 1< < 2 overrelaxation
extra weight is placed on the present value
assumption that new value is moving to the correct
solution by too slowly
( )
old
i
new
i
new
i
x 1 x x + =
Jacobi Iteration
Iterative like Gauss Seidel
Gauss-Seidel immediately uses the value of
x
i
in the next equation to predict x
i+1

Jacobi calculates all new values of x
i
s to
calculate a set of new x
i
values
( ) ( )
( ) ( )
( ) ( )
( ) ( )
( )
FIRST ITERATION
x c a x a x a x c a x a x a
SECOND ITERATION
x c a x a x a x c a x
1 1 12 2 13 3 11 1 1 12 2 13 3 11
2 2 21 1 23 3 22 2 2 21 1 23 3 22
3 3 31 1 32 2 33 3 3 31 1 32 2 33
1 1 12 2 13 3 11 1 1 12 2 13 3 11
2 2 21 1 23 3 22 2 2 21 1
= =
= =
= =
= =
= =
/ /
/ /
/ /
/ /
/
( )
( ) ( )
a x a
23 3 22
3 3 31 1 32 2 33 3 3 31 1 32 2 33
/
/ / = =
Graphical depiction of difference between Gauss-Seidel and Jacobi
( ) ( )
( ) ( )
( ) ( )
( ) ( )
( )
FIRST ITERATION
SECOND ITERATION
1 1 12 2 13 3 11 1 1 12 2 13 3 11
2 2 21 1 23 3 22 2 2 21 1 23 3 22
3 3 31 1 32 2 33 3 3 31 1 32 2 33
1 1 12 2 13 3 11 1 1 12 2 13 3 11
2 2 21 1 23 3 22 2 2 21 1
= =
= =
= =
= =
= =
/ /
/ /
/ /
/ /
/
( )
( ) ( )
a x a
23 3 22
3 3 31 1 32 2 33 3 3 31 1 32 2 33
/
/ / = =
( ) ( )
( ) ( )
( ) ( )
( ) ( )
( )
FIRST ITERATION
SECOND ITERATION
1 1 12 2 13 3 11 1 1 12 2 13 3 11
2 2 21 1 23 3 22 2 2 21 1 23 3 22
3 3 31 1 32 2 33 3 3 31 1 32 2 33
1 1 12 2 13 3 11 1 1 12 2 13 3 11
2 2 21 1 23 3 22 2 2 21 1
= =
= =
= =
= =
= =
/ /
/ /
/ /
/ /
/
( )
( ) ( )
a x a
23 3 22
3 3 31 1 32 2 33 3 3 31 1 32 2 33
/
/ / = =
( ) ( )
( ) ( )
( ) ( )
( ) ( )
( )
FIRST ITERATION
SECOND ITERATION
1 1 12 2 13 3 11 1 1 12 2 13 3 11
2 2 21 1 23 3 22 2 2 21 1 23 3 22
3 3 31 1 32 2 33 3 3 31 1 32 2 33
1 1 12 2 13 3 11 1 1 12 2 13 3 11
2 2 21 1 23 3 22 2 2 21 1
= =
= =
= =
= =
= =
/ /
/ /
/ /
/ /
/
( )
( ) ( )
a x a
23 3 22
3 3 31 1 32 2 33 3 3 31 1 32 2 33
/
/ / = =
2 3 1 2
4 1 2 2
3 2 1 1
(
(
(
( )
( )( ) ( )( )
( )
( )( ) ( )( )
( )
( )( ) ( )( )
10
1
12 3 1
1
6 2 1 3 1
x
' x a / ' x a ' x a c x
6
1
4 2
1
0 2 1 4 2
x
' x a / 0 a ' x a c x
1
2
2
2
0 1 0 3 2
x
' x
a
c
a / 0 a 0 a c x
3
3 33 2 32 1 31 3 3
2
2 22 23 1 21 2 2
1
1
11
1
11 13 12 1 1
=
+
=

=
= =
=

=

=
= =
= =

=
= = =
RECALL
GAUSS SEIDEL
2 3 1 2
4 1 2 2
3 2 1 1
(
(
(
( )
( )( ) ( )( )
( )
( )( ) ( )( )
( )
( )( ) ( )( )
1
1
1
1
0 2 0 3 1
x
' x a / 0 a 0 a c x
2
1
2
1
0 2 0 4 2
x
' x a / 0 a 0 a c x
1
2
2
2
0 1 0 3 2
x
' x
a
c
a / 0 a 0 a c x
3
3 33 32 31 3 3
2
2 22 23 21 2 2
1
1
11
1
11 13 12 1 1
= =

=
= =
=
=

=
= =
= =

=
= = =
... end of problem
JACOBI
Optimization Methods
Overview
Unconstrained Searches
1-D (one independent variable)
Golden Search
Quadratic Interpolation
Newtons Method
2-D (two or more independent variables)
Steepest Ascent (Descent)
Conjugate Gradients
Newton and Quasi-Newton Methods
Constrained Optimization
Linear Programming

Optimization Uses in Engineering
Maximizing Strength/Weight
Minimizing Cost or Maximizing Profit
Minimizing Wait Time
Maximizing Efficiency
Minimizing Virtual Work
1-D Searches
Objective is to maximize a function Objective Function
1-D searches have objective functions of one independent
variable f(x)
Analogy to root finding :
Bracketed Method Interval Search or Limited
Search
Open Method Unlimited or Free Search
Golden Section Search
Defining
2
/
1
as R and taking the reciprocal of the above
equation
1
2
2 1
1
1
2
0
1
or
=
+
=
Define the search interval as [x

,x
u
]

Define the search interval length as
0
= |x
u
- x
|

Divide
0
up into
1
and
2
such that
0
=
1
+
2

Also make sure that the ratio of the first sub-length
to the original length is the same as the first
sub-length to second sub-length ratio
618 . 0
2
1 5
R 1 R R
R
1
R 1
2
~
= = + = +
Golden Section Search, Continued
Once subdivided, we now evaluate values of f(x)
Since the designations
1
and
2
are arbitrary, test f(x

+
1
)
and f(x
u
-
1
) to determine the new search interval
So let x
1
= (x

+ d) and x
2
= (x
u
- d), where d =
1

Then if f(x
1
) > f(x
2
) the maximum must lie in the interval
[x
u
- d, x
u
] call x
u
- d (= x
2
) the new x

(Case 1)
otherwise call x

+ d (= x
1
) the new x
u
(Case 2)
With a little thinking you can convince yourself that for
(Case 1), x
2
for this new search interval is x
1
from the
previous, because
0
/
1
=
1
/
2
(Hint: d = R
0
)
And for (Case 2) Old x
1
x
2
x
x
u

d
d
x
2
x
1

First Step
x
x
u

Old x
2
Old x
1

Second Step
x
1
x
x
2

d
f (x)
f (x)
Text Example 13.1
Use Golden Search to find max (2 sinx -x
2
/10) on [0,4]
First Step: Calculate d = R
0
= 0.61803*4 = 2.4721
Second Step: Calculate x
2
= 4 - 2.4721 = 1.5279
and x
1
= 0 +2.4721 = 2.4721
Third Step: Calculate f(x
2
) = 1.7647
and f(x
1
) = 0.6300
Now because f(x
1
) is not > f(x
2
), set x
u
to 2.4721
& set new x
1
= 1.5279
Graphically
Text Example 13.1 f(x)
-4
-3
-2
-1
0
1
2
3
0 1 2 3 4 5
x
f
(
x
)
<-------- l
0
= 4 ------------>
x
1
x
2

f=1.7647
f=0.6300
Newtons Method for Optimization in 1-D
Basis: Maximum (or minimum) of f(x) will occur where
f (x) = 0
If we call f (x) by the name g(x) and apply Newton-
Raphson to find where g(x) = 0
x
i +1
= x
i
- g (x) / g (x)
or
x
i +1
= x
i
- f (x) / f (x)
Method converges to nearest max or min (local max/min)
Method can blow up near points where f (x) = 0
if f (x) is not going to 0 as fast (or at all) as f (x)
0.0 0.5 1.0 1.5 2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
f
g
f

(
x
)

&

g

(
x
)
x
f(x) and its Derivative in Examples on [0,2]
2-D Search Methods
The Multi-D Equivalent of f is the
Gradient V f or [cf/cx
1
i, cf/cx
2
j, cf/cx
3
k...]
T

All 2-D Methods Use V f
We can Conceptualize the Gradient as the
Topographic Elevation Slope in x-y-z space
+ x
1
x
+ x
2
y
+ f z
-50
-40
-30
-20
-10
0
10
-3
-2
-1
0
1
2
3
4
5
6
-2
-1
0
1
2
3
3D Elevation Plot
3-D Elevations Contour Plot
V f
Contour Graph of f (x,y)
X
-2 -1 0 1 2 3 4 5
Y
-1
0
1
2
3
-5
-5
-5
-10
-10
-15
-20
0
0
0
0
0
0
0
-5
-5
-5
-5
-10
-10
-10
-15
-15
-20
-20
-25
-30
-35
Steepest Ascent/Descent
Follow V f = g direction (- g for descent)
How far should we go in this direction?
Call the magnitude of the step h
Now maximize f along the g direction as a
function of h called a line search
Step x
1
i+1
= x
1
i
+ g
1
h & x
2
i+1
= x
2
i
+ g
2
h

Text Example 14.4
Search for max of f = 2 x
1
x
2
+ 2x
1
- x
1
2
- 2x
2
2

from the point x
1
= -1 and x
2
= 1
Calculate g (= 6 i - 6 j), because
+ g
1
= 2 x
2
+ 2 - 2 x
1
= 2 (1) + 2 - 2 (-1) = 6
+ g
2
= 2 x
1
- 4 x
2
= 2 (-1) - 4 (1) = -6
Text Example 14.4, Continued
Now search for max of f (x
1
1
, x
2
1
) subject to
+ x
1
1
= x
1
0
+ h g
1
= -1+6h & x
2
1
= x
2
0
+ h g
2
= 1 - 6h

+f (x
1
1
, x
2
1
) = -180 h
2
+72 h - 7 = g(h)
+Max {g(h)} g(h) =0 -360 h +72 = 0 h=0.2
Now stepping along g (= 6 i - 6 j) gives
+x
1
i+1
= x
1
i
+ g
1
h = - 1 +(6)(0.2) = 0.2
+x
2
i+1
= x
2
i
+ g
2
h = 1 +(-6)(0.2) = -0.2

Graphical Depiction of Steepest Ascent
Steps in Example 14.4
X
-2 -1 0 1 2 3 4 5
Y
-1
0
1
2
3
-5
-5
-5
-10
-10
-15
-20
0
0
0
0
0
0
0
-5
-5
-5
-5
-10
-10
-10
-15
-15
-20
-20
-25
-30
-35
2-D Newton Method
The 2-D Version of Newton Maximization
Gradient 0 or V f = 0 g
1
= g
2
= 0
Applying Multi-variate Newton-Raphson
(
A
A
+
(
=
(
=
(
A
A
(
(
(
(
c
c
c
c
c
c
c
c
+
2
1
i
2
1
1 i
2
1
2
1
2
1
2
2
1
2
2
1
1
1
x
x
x
x
x
x
and
g
g
x
x
x
g
x
g
x
g
x
g
The first matrix is the Jacobian of the Gradient
. AKA the Hessian
| |
(
(
(
(
c
c
c c
c
c c
c
c
c
=
(
(
(
(
c
c
c
c
c
c
c
c
=
2
2
2
2 1
2
2 1
2
2
1
2
2
2
1
2
2
1
1
1
x
f
x x
f
x x
f
x
f
x
g
x
g
x
g
x
g
H
2-D Newton Method
Text describes Newtons equation as
0
g
g
x
x
H H
H H
2
1
2
1
22 21
12 11
=
)
`
+
)
`
A
A
(
The system is then solved by matrix inversion

)
`
)
`
=
)
`
+
2
1
1
22 21
12 11
i
2
1
1 i
2
1
g
g
H H
H H
x
x
x
x
Newton Method for Text Example 14.4
Search for max of f(x
1
,x
2
) = 2 x
1
x
2
+ 2x
1
- x
1
2
- 2x
2
2

from the point x
1
= -1 and x
2
= 1
g
1
= 2 x
2
+ 2 - 2 x
1
= 2 (1) + 2 - 2 (-1) = 6
g
2
= 2 x
1
- 4 x
2
= 2 (-1) - 4 (1) = -6
H
11
= -2, H
12
= 2, H
21
= 2, H
22
= -4
Solving,
( )
( )
(
=
(
A
A
(
=
(
A
A
(
0
3
x
x
get we
6
6
x
x
4 2
2 2
2
1
2
1
So x
1
= 2, x
2
= 1 f = 2 Maximum (Why?)
Graphical Depiction of Newton Step
X
-2 -1 0 1 2 3 4 5
Y
-1
0
1
2
3
-5
-5
-5
-10
-10
-15
-20
0
0
0
0
0
0
0
-5
-5
-5
-5
-10
-10
-10
-15
-15
-20
-20
-25
-30
-35
Constrained Optimization
Search for maximum is limited by constraints
Problem related constraints (resource limits)
+Plant operates no more than 80 hours per week
+Raw materials can not be purchased for less than
$30/ton
Feasibility constraints
+Efficiency can not be greater than unity
+Cost can not be negative
+Mass can not be negative
Linear Programming
Multi-D Objective Function Z(x
1
, x
2
, x
3
)
is a linear function of x
1
, x
2
, x
3

Constraints must be formulated as linear
inequality statements in terms of x
1
, x
2
, x
3

Multi-D Problem Statement
+ Z = c
1
x
1
+ c
2
x
2
+ c
n
x
n
+ a
i1
x
1
+ a
i2
x
2
+... a
in
x
n
s b
i
( i = 1, m)
Text Example
Maximize profit (Z) in the production of
two products (x
1
, x
2
)
2-D Problem
+ Z = 150 x
1
+ 175

x
2

+ 7 x
1
+ 11 x
2

s 77 (1)
+ 10 x
1
+ 8 x
2

s 80 (2)
+ x
1
s 9 (3)
+ x
2
s 6 (4) (m=4)
+ x
1
> 0 (5) (Note that these are
+ x
2
> 0 (6) not s constraints)

Graphical Depiction of Constraints
x
1

x
2

Graphical Depiction Z(x
1
, x
2
) Subject to Constraints
x
1

x
2

Z = 0
Z = 600
Z = 1400
175
Z
x
175
150
x
1 2
+ =
Things to Note from Graphical Solution
Region of possible solutions which meet all
constraints (feasible solutions) is a polygon
Brute force approach could find max Z by
evaluating Z at all vertices
Vertices are simultaneous solution of 2
constraints (inequalities) converted to 2
equations (equalities) x
1
= x
1
*
& x
2
= x
2
*

Z evaluation is of Z (x
1
*
, x
2
*
)
Simplex Method
Converts relations to m equations in m+n
unknowns we must have at least m = 1
Constraint inequalities are converted to equalities
with new variables called slack variables S
i

a
i1
x
1
+ a
i2
x
2
+... + a
in
x
n
+ S
i
= b
i
(i = 1, m)
Inequality i will then be at its limit when S
i
= 0
The independent variables (xs) are called
structural variables
Simplex Method, Continued
Slack variables will have positive values if the
values of structural variables meet the inequality
with room to spare
There will be Comb(n+m,m) different possible
constraint intersections (including non-negativity
constraints)
Simplex method uses Gauss Jordan elimination
to check only vertices on the feasible polygon
starting with all xs = 0
Simplex Tableau for Text Example
Basic Z x
1
x
2
S
1
S
2
S
3
S
4
b
i
___________________________________________________________________________________

6 1 0 0 0 1 0 0 S
9
80
77
0
0 1 0 0
0 0 1 0
0 0 0 1
0 0 0 0
0 1 0 S
8 10 0 S
11 7 0 S
175 150 1 Z
4
3
2
1

Note that we never do Gauss Jordan elimination on Z row
1st row keeps track of current estimate of Z in b column
Basic variables start out as Ss
Non-basic variables start out as xs and are assumed zero
Simplex Elimination
Basic Z x
1
x
2
S
1
S
2
S
3
S
4
b
i
___________________________________________________________________________________

6 1 0 0 0 1 0 0 S
9
80
77
0
0 1 0 0
0 0 1 0
0 0 0 1
0 0 0 0
0 1 0 S
8 10 0 S
11 7 0 S
175 150 1 Z
4
3
2
1

Arbitrarily choose x
1
for Gauss Jordan elimination (not standard)
This makes it a basic variable (enters basic list: entering variable)
Pivot about S
2
row, because 80/10 is smallest b
i
/ a
i1
-- any other
choice results in negative Ss after elimination
This choice makes S
2
non-basic (it is the leaving variable)
First Simplex Step - Eliminate x
1

6 1 0 0 0 1 0 0 S
1
8
21
1200
0 1 1 . 0 0
0 0 1 . 0 0
0 0 7 . 0 1
0 0 15 0
8 . 0 0 0 S
8 . 0 1 0 x
4 . 5 0 0 S
55 0 1 Z
4
3
1
1
S
1
row can be read as S
1
= 21 (reduced from 77), because x
2
and
S
2
are non-basic variables (their values can be taken as zero)
Similarly x
1
row is read x
1
= 8
Equivalent to moving from [0,0] to [8,0] for location of max
This moves our estimate of max Z from 0 to 1200 (see Z eq.)
Basic Z x
1
x
2
S
1
S
2
S
3
S
4
b
i
___________________________________________________________________________________

Next Elimination Choice
6 1 0 0 0 1 0 0 S
1
8
21
1200
0 1 1 . 0 0
0 0 1 . 0 0
0 0 7 . 0 1
0 0 15 0
8 . 0 0 0 S
8 . 0 1 0 x
4 . 5 0 0 S
55 0 1 Z
4
3
1
1
Now choose x
2
as the entering variable (most neg. Z eq. Coef.)
The S
1
row is chosen as pivot row (note: 1/(-0.8) not allowed-why?)
So now S
1
is our leaving variable

Basic Z x
1
x
2
S
1
S
2
S
3
S
4
b
i
___________________________________________________________________________________

Next Simplex Step (Elimination)
11 . 2 1 0 13 . 0 18 . 0 0 0 0 S
11 . 4
89 . 4
89 . 3
1414
0 1 20 . 0 15 . 0
0 0 20 . 0 15 . 0
0 0 13 . 0 18 . 0
0 0 87 . 7 18 . 10
0 0 0 S
0 1 0 x
1 0 0 x
0 0 1 Z
4
3
1
2

We know we are done, because all coefficients in Z eq. are positive
The non- basic variables are S
1
& S
2
, so vertex is at the intersection
of the equations from constraint (1) and constraint (2)
The optimal Z is then 1414
Basic Z x
1
x
2
S
1
S
2
S
3
S
4
b
i
___________________________________________________________________________________

Graphical Depiction of Simplex Steps
x
1

x
2

Actual Simplex Path
We want to find the best fit of a curve through the data.
Here we see :
a) Least squares fit
b) Linear interpolation
Can you suggest another?
f(x)
x
Curve Fitting
The prerequisite mathematical background
for interpolation is found in the material on
the Taylor series expansion and finite
divided differences
Simple statistics
average
standard deviation
normal distribution
Normal Distribution
A histogram used
to depict the distribution
of exam grades.
x
x 2
95%
o
x o
68%
Material to be Covered in Curve
Fitting
Linear Regression
Polynomial Regression
Multiple Regression
General Linear Least Squares
Nonlinear Regression
Interpolation
Newtons Polynomial
Lagrange Polynomial
Coefficients of Polynomials
Least Squares Regression
Simplest is fitting a straight line to a set of
paired observations
(x
1
,y
1
), (x
2
, y
2
).....(x
n
, y
n
)
The resulting mathematical expression is
y = a
o
+ a
1
x + e
We will consider the error introduced at
each data point to develop a strategy for
determining the best fit equations
f(x)
x
| | ( )
2
n
1 i
i 1 o i
n
1 i
2
i r
x a a y e S

= =
+ = =
i 1 o i
x a a y
To determine the values for a
o
and a
1
, differentiate
with respect to each coefficient
( )
( ) | |
=
c
c
=
c
c
i i 1 o i
1
r
i 1 o i
o
r
x x a a y 2
a
S
x a a y 2
a
S
Note: we have simplified the summation symbols.
What mathematics technique will minimize S
r
?
( )
2
n
1 i
i 1 o i
n
1 i
2
i r
x a a y e S

= =
= =
( )
( ) | |
=
c
c
=
c
c
i i 1 o i
1
r
i 1 o i
o
r
x x a a y 2
a
S
x a a y 2
a
S
Setting the derivative equal to zero will minimizing S
r
.
If this is done, the equations can be expressed as:

=
=
2
i 1 i o i i
i 1 o i
x a x a x y 0
x a a y 0

=
=
2
i 1 i o i i
i 1 o i
x a x a x y 0
x a a y 0
If you recognize that:

you have two equations, with two simultaneous equations
with two unknowns, a
o
and a
1
.

What are these equations? (hint: only place terms with
a
o
and a
1
on the LHS of the equations)

What are the final equations for a
o
and a
1
?
=
o o
na a
( )
x a y a
x x n
y x y x n
a
y x x a x a
y x a na
1 o
2
i
2
i
i
i i i
1
i i
2
i 1 i o
i i 1 o
=

=
= +
= +

These first two
equations are called
the normal equations
n
y
y &
n
x
x that note Also
y x
y
a
a
x x
x n
i
_
i
_
i i
i
1
0
2
i i
i
E
=
E
=
)
`
E
E
=
)
`
E E
E
These second two
result from solving
Error
( )
2
n
1 i
i 1 o i
n
1 i
2
r
x a a y e S
i

= =
= =
Recall:
f(x)
x
( )
1 n
S
s
y y S
t
y
2
i t
=
=
The most common measure of the spread of a sample is the

standard deviation about the mean:
Introduce a term to measure the standard error of the estimate:
2
|
=
n
S
s
r
x y
Coefficient of determination r
2
:
t
r t
2
S
S S
r

=
r is the correlation coefficient
t
r t
2
S
S S
r

=
The following signifies that the line explains 100 percent
of the variability of the data:

S
r
= 0
r = r
2
= 1

If r = r
2
= 0, then S
r
= S
t
and the fit is invalid.

Data 1 Data 2 Data 3 Data 4
10 8.04 10 9.14 10 7.46 8 6.58
8 6.95 8 8.14 8 6.77 8 5.76
13 7.58 13 8.74 13 12.74 8 7.71
9 8.81 9 8.77 9 7.11 8 8.84
11 8.33 11 9.26 11 7.81 8 8.47
14 9.96 14 8.10 14 8.84 8 7.04
6 7.24 6 6.13 6 6.08 8 5.25
4 4.26 4 3.10 4 5.39 19 12.50
12 10.84 12 9.13 12 8.15 8 5.56
7 4.82 7 7.26 7 6.42 8 7.91
5 5.68 5 4.74 5 5.73 8 6.89
Consider the following four sets of data
Data Set 1
0.00
2.00
4.00
6.00
8.00
10.00
12.00
0 5 10 15
x
f
(
x
)
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
0 5 10 15 20
f
(
x
)
Data Set 2
0.00
2.00
4.00
6.00
8.00
10.00
0 5 10 15
x
f
(
x
)
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
0 5 10 15
0.00
2.00
4.00
6.00
8.00
10.00
0 5 10 15
Data Set 3
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
0 5 10 15
x
f
(
x
)
0.00
2.00
4.00
6.00
8.00
10.00
12.00
0 5 10 15
Data Set 4
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
0 5 10 15 20
x
f
(
x
)
GRAPHS OF FOUR DATA SETS
Data Set 1
0.00
2.00
4.00
6.00
8.00
10.00
12.00
0 5 10 15
x
f
(
x
)
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
0 5 10 15 20
f
(
x
)
Data Set 2
0.00
2.00
4.00
6.00
8.00
10.00
0 5 10 15
x
f
(
x
)
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
0 5 10 15
0.00
2.00
4.00
6.00
8.00
10.00
0 5 10 15
Data Set 3
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
0 5 10 15
x
f
(
x
)
0.00
2.00
4.00
6.00
8.00
10.00
12.00
0 5 10 15
Data Set 4
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
0 5 10 15 20
x
f
(
x
)
Linear Regression Data Set 1
0.50 3.00
0.117906 1.124747
0.666542 1.236603
17.98994 9
0.50 3.00
0.117964 1.125302
0.666242 1.237214
17.96565 9
0.50 3.00
0.117878 1.124481
0.666324 1.236311
17.97228 9
0.50 3.00
0.117819 1.123921
0.666707 1.235695
18.00329 9
All equations are y = 0.5x + 3 R
2
= 0.67
GRAPHS OF FOUR DATA SETS
Linearization of non-linear
relationships

Some data is simply ill-suited for
linear least squares regression....

or so it appears.
f(x)
x
P
t
ln P
t
slope = r
intercept = ln P
0
rt
o
e P P =
L
i
n
e
a
r
i
z
e

why?
EXPONENTIAL
EQUATIONS
( )
( ) ( )
( ) rt P ln
e ln P ln
e P ln P ln
e P P
0
rt
0
rt
0
rt
0
+ =
+ =
=
=
slope = r

intercept = ln P
o
Can you see the similarity
with the equation for a line:

y = b + mx

where b is the y-intercept
and m is the slope?
lnP

t
( )
( ) ( )
( ) t r P ln
e ln P ln
e P ln P ln
e P P
0
t r
0
t r
0
t r
0
+ =
+ =
=
=
ln P
0
t
slope = r intercept = ln P
0
After taking the natural log
of the y-data, perform linear
regression.
From this regression:
The value of b will give us
ln (P
0
). Hence, P
0
= e
b

The value of m will give us r
directly.
POWER EQUATIONS
log Q
log H
Q
H
a
H c Q =
Here we linearize
the equation by
taking the log of
H and Q data.
What is the resulting
intercept and slope?
(Flow over a weir)
( )
H log a c log
H log c log
H c log Q log
H c Q
a
a
a
+ =
+ =
=
=
log Q
log H
slope = a
intercept = log c

S
1/
1/ S
SATURATION-GROWTH
RATE EQUATION
S K
S
s
max
+
=
slope = K
s
/
max

intercept = 1/
max

Here, is the growth rate of
a microbial population,
max
is the maximum
growth rate, S is the
substrate or food
concentration, K
s
is the
substrate concentration at a
value of =
max
/2
General Comments of Linear
Regression
You should be cognizant of the fact that
there are theoretical aspects of regression
that are of practical importance but are
beyond the scope of this book
Statistical assumptions are inherent in the
linear least squares procedure
Regression
x has a fixed value; it is not random and is
measured without error
The y values are independent random
variable and all have the same variance
The y values for a given x must be normally
distributed
Regression
The regression of y versus x is not the same
as x versus y
The error of y versus x is not the same as x
versus y
f(x)
x
y
-
d
i
r
e
c
t
i
o
n

x-direction
Polynomial Regression
One of the reasons you were presented with
the theory behind linear regression was to
allow you the insight into similar
procedures for higher order polynomials
y = a
0
+ a
1
x
mth - degree polynomial
y = a
0
+ a
1
x + a
2
x
2
+....a
m
x
m
+ e
Based on the sum of the squares
of the residuals
( )
=
2
m
i m
2
i 2 i 1 o i r
x a ...... x a x a a y S
1. Take the derivative of the above equation with respect to
each of the unknown coefficients: i.e. the partial with respect
to a
2
( )
=
c
c
m
i m
2
i 2 i 1 o i
2
i
2
r
x a ..... x a x a a y x 2
a
S
2. These equations are set to zero to minimize S
r
., i.e.
minimize the error.

3. Set all unknowns values on the LHS of the equation.
Again, using the partial of S
r
. wrt a
2
4. This set of normal equations result in m+1 simultaneous
equations which can be solved using matrix methods to
determine a
0
, a
1
, a
2
......a
m

= + + + +
+
i
2
i
2 m
i m
4
i 2
3
i 1
2
i o
y x x a ..... x a x a x a
Multiple Linear Regression
A useful extension of linear regression is
the case where y is a linear function of two
or more variables
y = a
o
+ a
1
x
1
+ a
2
x
2

We follow the same procedure
y = a
o
+ a
1
x
1
+ a
2
x
2
+ e
Multiple Linear Regression
For two variables, we would solve a 3 x 3 matrix
in the following form:
(
(
(

i i 2
i i 1
i
2
1
0
2
i 2 i 2 i 1 i 2
i 2 i 1
2
i 1 i 1
i 2 i 1
y x
y x
y
a
a
a
x x x x
x x x x
x x n
[A] and {c} are clearly based on data given for x
1
, x
2
and y
to solve for the unknowns in {x}.

Interpolation
General formula for an n-th order
polynomial
y = a
0
+ a
1
x + a
2
x
2
+....a
m
x
m
For m+1 data points, there is one, and only
one polynomial of order m or less that
passes through all points
Example: y = a
0
+ a
1
x
fits between 2 points
1st order
Interpolation
We will explore two mathematical methods
well suited for computer implementation
Newtons Divided Difference Interpolating
Polynomials
Lagrange Interpolating Polynomial
Newtons Divided Difference
Interpolating Polynomials
Linear Interpolation
General Form
Errors
Linear Interpolation
Temperature, C
Density, kg/m
3

0 999.9
5 1000.0
10 999.7
15 999.1
20 998.2

How would you approach estimating the density at 17 C?
Temperature, C
Density, kg/m
3

0 999.9
5 1000.0
10 999.7
15 999.1
20 998.2

T
15
20
9
9
8
.
2

9
9
9
.
1

???
999.1 > > 998.2
( ) ( )
( ) ( )
( )
0
0 1
0 1
o 1
x x
x x
x f x f
x f x f
+ =
Note: The notation f
1
(x) designates that this is a first order
interpolating polynomial
15 17
1 . 999
15 20
1 . 999 2 . 998
f(x)
x
true solution
smaller intervals
provide a better estimate
1
2
f(x)
x
true solution
Alternative approach would be to
include a third point and estimate
f(x) from a 2nd order polynomial.
( ) ( ) ( )( )
1 0 2 0 1 0 2
x x x x b x x b b x f + + =
Prove that this a 2nd order polynomial of
the form:
( )
2
2 1 0
x a x a a x f + + =
( )
1 2 0 2 1 0 2
2
2 0 1 1 0 2
x x b x x b x x b x b x b x b b x f + + + =
First, multiply the terms
( )
2
2 1 0
x a x a a x f + + =
( ) ( ) ( )( )
1 0 2 0 1 0 2
Collect terms and recognize that:
2 2
1 2 0 2 1 1
1 0 2 0 1 0 0
b a
x b x b b a
x x b x b b a
=
=
+ =
f(x)
x
x
2
, f(x
2
)
x
1
, f(x
1
)
x
0
, f(x
0
)
x, f(x)
Procedure for
Quadratic
Interpolation
( )
( ) ( )
( ) ( ) ( ) ( )
0 2
0 1
0 1
1 2
1 2
2
0 1
0 1
1
0 0
x x
x x
x f x f
x x
x f x f
b
x x
x f x f
b
x f b
=
=
Procedure for Quadratic
Interpolation
( )
( ) ( )
( ) ( ) ( ) ( )
0 2
0 1
0 1
1 2
1 2
2
0 1
0 1
1
0 0
x x
x x
x f x f
x x
x f x f
b
x x
x f x f
b
x f b
=
=
( ) ( ) ( )( )
1 0 2 0 1 0 2
Example
998
998.5
999
999.5
1000
1000.5
0 10 20 30
Temp
D
e
n
s
i
t
y
Temperature, C
Density, kg/m
3

0 999.9
5 1000.0
10 999.7
15 999.1
20 998.2

Include 10 degrees in
your calculation of the
density at 17 degrees.
Example
Temperature, C
Density, kg/m
3

0 999.9
5 1000.0
10 999.7
15 999.1
20 998.2

Include 10 degrees in
your calculation of the
density at 17 degrees.
( ) ( ) ( )( )
1 0 2 0 1 0 2
( )
( ) ( )
( ) ( ) ( ) ( )
0 2
0 1
0 1
1 2
1 2
2
0 1
0 1
1
0 0
x x
x x
x f x f
x x
x f x f
b
x x
x f x f
b
x f b
=
=
General Form of Newtons
Interpolating Polynomials
for the nth-order polynomial
( ) ( ) ( )( ) ( )
1 n 1 0 n 0 1 0 n
x x x x x x b .... x x b b x f

+ + + =
To establish a methodical approach to a solution define
the first finite divided difference as:
| |
( ) ( )
j i
j i
j i
x x
x f x f
x , x f
=
| |
( ) ( )
j i
j i
j i
x x
x f x f
x , x f
=
if we let i=1 and j=0 then this is b
1
( ) ( )
0 1
0 1
1
x x
x f x f
b
=
Similarly, we can define the second finite divided difference,
which expresses both b
2
and the difference of the first two
divided difference
Similarly, we can define the second finite divided difference,
which expresses both b
2
and the difference of the first two
divided difference
( ) ( ) ( ) ( )
| |
| | | |
k i
k j j i
k j i
0 2
0 1
0 1
1 2
1 2
2
x x
x , x f x , x f
x , x , x f
x x
x x
x f x f
x x
x f x f
b
=
Following the same scheme, the third divided difference is
the difference of two second finite divided difference.
This leads to a scheme that can easily lead to the
use of spreadsheets
i x
i
f(x
i
) first second third

0 x
0
f(x
0
) f[x
1
,x
0
] f[x
2
,x
1
,x
0
] f[x
3
,x
2
,x
1
,x
0
]

1 x
1
f(x
1
) f[x
2
,x
1
] f[x
3
,x
2
,x
1
]

2 x
2
f(x
2
) f[x
3
,x
2
]

3 x
3
f(x
3
)
( ) ( ) ( )( ) ( )
1 n 1 0 n 0 1 0 n

+ + + =
These difference can be used to evaluate the b-coefficient s.

The result is the following interpolation polynomial called
the Newtons Divided Difference Interpolating Polynomial
( ) ( ) | |( )
| |( )( ) ( )
1 n 1 0 0 1 n n
0 0 1 0 n
x x x x x x x , , x , x f
.... x x x , x f x f x f

+
+ + =

To determine the error we need an extra point.
The error would follow a relationship analogous to the error
in the Taylor Series.
Lagrange Interpolating
Polynomial
( ) ( ) ( )
( )
[
=
=
=
=
=
n
i j
0 j
j i
j
i
n
0 i
i i n
x x
x x
x L
x f x L x f
where H designates the product of
The linear version of this expression is at n = 1
( ) ( ) ( )
( )
( ) ( )
1
0 1
0
0
1 0
1
1
n
i j
0 j
j i
j
i
n
0 i
i i n
x f
x x
x x
x f
x x
x x
) x ( f
x x
x x
x L
x f x L x f
=
=
[
=
=
=
Linear version: n = 1
Your text shows you how to do n = 2 (second order).
What would third order be?
( ) ( ) ( )
( )
( )( )( )
( )( )( )
( )
.......
x f
x x x x x x
x x x x x x
) x ( f
x x
x x
x L
x f x L x f
0
3 0 2 0 1 0
3 2 1
3
n
i j
0 j
j i
j
i
n
0 i
i i n

=
=
=
[
=
=
=
Note:
x
0
is
not being subtracted
from the constant
term x

( ) ( ) ( )
( )
( )( )( )
( )( )( )
( )
.......
x f
x x x x x x
x x x x x x
) x ( f
x x
x x
x L
x f x L x f
0
3 0 2 0 1 0
3 2 1
3
n
i j
0 j
j i
j
i
n
0 i
i i n

=
=
=
[
=
=
=
Note:
x
0
is
from the constant
term x
or x
i
= x
0
in
the numerator
or the denominator
j= 0
( ) ( ) ( )
( )
( )( )( )
( )( )( )
( )
.......
x f
x x x x x x
x x x x x x
) x ( f
x x
x x
x L
x f x L x f
0
3 0 2 0 1 0
3 2 1
3
n
i j
0 j
j i
j
i
n
0 i
i i n

=
=
=
[
=
=
=
( ) ( ) ( )
( )
( )( )( )
( )( )( )
( )
( )( )( )
( )( )( )
( )
( )( )( )
( )( )( )
( )
( )( )( )
( )( )( )
( )
3
2 3 1 3 0 3
2 1 0
2
3 2 1 2 0 2
3 1 0
1
3 1 2 1 0 1
3 2 0
0
3 0 2 0 1 0
3 2 1
3
n
i j
0 j
j i
j
i
n
0 i
i i n
x f
x x x x x x
x x x x x x
x f
x x x x x x
x x x x x x
x f
x x x x x x
x x x x x x
x f
x x x x x x
x x x x x x
) x ( f
x x
x x
x L
x f x L x f

+

+

+

=
=
=
[
=
=
=
Note:
x
3
is
from the constant
term x or x
i
= x
3
in
the numerator
or the denominator
j= 3
Example
998
998.5
999
999.5
1000
1000.5
0 10 20 30
Temp
D
e
n
s
i
t
y
Temperature, C
Density, kg/m
3

0 999.9
5 1000.0
10 999.7
15 999.1
20 998.2

Determine the density
at 17 degrees.
In fact, you can derive Lagrange directly from
Newtons Interpolating Polynomial
( ) 776 . 998 496 . 279 244 . 839 964 . 119 17 f
2
= + + =
( )
( ) 74 . 998 17 f
776 . 998 17 f
1
2
=
=
Using Newtons
Interpolating Polynomial
Coefficients of an Interpolating
Polynomial
( ) ( ) ( )( ) ( )
1 n 1 0 n 0 1 0 n

+ + + =
y = a
0
+ a
1
x + a
2
x
2
+....a
m
x
m
HOW CAN WE BE MORE STRAIGHT
FORWARD IN GETTING VALUES?
( )
( )
( )
2
2 2 2 1 0 2
2
1 2 1 1 0 1
2
0 2 0 1 0 0
x a x a a x f
x a x a a x f
x a x a a x f
+ + =
+ + =
+ + =
This is a 2nd order polynomial.

We need three data points.

Plug the value of x
i
and f(x
i
)
directly into equations.

This gives three simultaneous equations
to solve for a
0
, a
1
, and a
2
Example
998
998.5
999
999.5
1000
1000.5
0 10 20 30
Temp
D
e
n
s
i
t
y
Temperature, C
Density, kg/m
3

0 999.9
5 1000.0
10 999.7
15 999.1
20 998.2

Determine the density
at 17 degrees.
(
(
(
2 . 998
1 . 999
7 . 999
a
a
a
20 20 1
15 15 1
10 10 1
2
1
0
2
2
2
Temperature, C
Density, kg/m
3

0 999.9
5 1000.0
10 999.7
15 999.1
20 998.2

( ) ( )
78 . 998
17 006 . 0 17 03 . 0 1000
006 . 0
03 . 0
1000
a
a
a
2
17
2
1
0
=
+ =
Numerical Differentiation and

Integration
Calculus is the mathematics of change.
Engineers must continuously deal with
systems and processes that change, making
calculus an essential tool of our profession.
At the heart of calculus are the related
mathematical concepts of differentiation
and integration.
Differentiation
Dictionary definition of differentiate - to
mark off by differences, distinguish; ..to
perceive the difference in or between
Mathematical definition of derivative - rate
of change of a dependent variable with
respect to an independent variable
( ) ( )
x
x f x x f
x
y
i i
A
A +
=
A
A
( ) ( )
x
x f x x f
x
y
i i
A
A +
=
A
A
( )
i
x f
( ) x x f
i
A +
Ay
Ax
f(x)
x
Integration
The inverse process of differentiation
Dictionary definition of integrate - to bring
together, as parts, into a whole; to unite; to
indicate the total amount
Mathematically, it is the total value or summation
of f(x)dx over a range of x. In fact the
integration symbol is actually a stylized capital S
intended to signify the connection between
integration and summation.
f(x)
x
( )
}
=
b
a
dx x f I
? dx e
?
x
dx
? dx a
? du u
? udv
ax
bx
n
=
=
=
=
=
}
}
}
}
}
? ) uv (
dx
d
? u
dx
d
x of functions are v and u if
? x ln
dx
d
? x
dx
d
? x tan
dx
d
? a
dx
d
? x cos
dx
d
? e
dx
d
? x sin
dx
d
n
n
x
x
= =
=
= =
= =
= =
Newton-Cotes Integration
Common numerical integration scheme
Based on the strategy of replacing a complicated
function or tabulated data with some
approximating function that is easy to integrate
( ) ( )
( )
n
n 1 0 n
b
a
n
b
a
x a .... x a a x f
dx x f dx x f I
+ + + =
~ =
} }
( ) ( )
( )
n
n 1 0 n
b
a
n
b
a
x a .... x a a x f
dx x f dx x f I
+ + + =
~ =
} }
f
n
(x) is an nth order
polynomial
The approximation of an integral by the area under
- a first order polynomial
- a second order polynomial
We can also approximated the integral by using a
series of polynomials applied piece wise.
An approximation of an integral by the area under
five straight line segments.
Newton-Cotes Formulas
Closed form - data is at the beginning and
end of the limits of integration
Open form - integration limits extend
beyond the range of data.
Trapezoidal Rule
First of the Newton-Cotes closed integration
formulas
Corresponds to the case where the
polynomial is a first order
( ) ( )
( ) x a a x f
dx x f dx x f I
1 0 n
b
a
1
b
a
+ =
~ =
} }
( ) ( )
( ) x a a x f
dx x f dx x f I
1 0 n
b
a
1
b
a
+ =
~ =
} }
A straight line can be represented as:
( ) ( )
( ) ( )
( ) a x
a b
a f b f
a f x f
1

+ =
Integrate this equation. Results in the trapezoidal rule.
( )
( ) ( )
2
b f a f
a b I
+
~
( ) ( )
( )
( ) ( )
( )
b b
1
a a
b
a
I f x dx f x dx
f b f a
f a x a dx
b a
= ~
(
= +
(

} }
}
( )
( ) ( )
2
b f a f
a b I
+
~
Recall the formula for computing the area of a trapezoid:

height x (average of the bases)
h
e
i
g
h
t

base
base
The concept is the same but the trapezoid is on its side.
width
h
e
i
g
h
t

h
e
i
g
h
t

h
e
i
g
h
t

base
base
( )
( ) ( )
2
b f a f
a b I
+
~
Error of the Trapezoidal Rule
( )( )
b a where
a b ' ' f
12
1
E
3
t
< <
=
This indicates that is the function being integrated is
linear, the trapezoidal rule will be exact.

Otherwise, for section with second and higher order
derivatives (that is with curvature) error can occur.

A reasonable estimate of is the average value of
b and a
Multiple Application of the
Trapezoidal Rule
Improve the accuracy by dividing the
integration interval into a number of smaller
segments
Apply the method to each segment
Resulting equations are called multiple-
application or composite integration
formulas
( ) ( ) ( ) ( ) ( ) ( )
( )
h
n
x x
given
2
x f x f
h
2
x f x f
h
2
x f x f
h I
dx ) x ( f dx ) x ( f dx ) x ( f I
0 n
n 1 n 2 1 1 0
x
x
x
x
x
x
n
1 n
2
1
1
0
=
+
+
+
+
+
~
+ + + =
} } }

Multiple Application of the

Trapezoidal Rule
where there are n+1 equally spaced base points.
( ) ( ) ( ) ( ) ( ) ( )
( )
( ) ( ) ( )
( )
( ) ( ) ( )
n 2
x f x f 2 x f
a b
2
x f x f 2 x f
n
a b
I
2
x f x f
h
2
x f x f
h
2
x f x f
h I
dx ) x ( f dx ) x ( f dx ) x ( f I
1 n
1 i
n i 0
1 n
1 i
n i 0
n 1 n 2 1 1 0
x
x
x
x
x
x
n
1 n
2
1
1
0

} } }
+ +
=
+ +
~
+
+
+
+
+
~
+ + + =
We can group terms to express a general form

}

}

width average height
( )
( ) ( ) ( )
n 2
x f x f 2 x f
a b I
1 n
1 i
n i 0
=
+ +
~
}

}

The average height represents a weighted average
of the function values

Note that the interior points are given twice the weight
of the two end points
( )
' ' f
n 12
a b
E
2
3
a
=
Example
Evaluate the following integral using the
trapezoidal rule and h = 0.1
( )
( ) ( ) ( )
n 2
x f x f 2 x f
a b I
1 n
1 i
n i 0
=
+ +
~
}
=
6 . 1
1
x
dx e I
2
Example Problem
0
10
20
30
40
50
60
0 0.5 1 1.5 2 2.5
x
f
(
x
)
n
a b
h

=
Solution
( ) ( ) ( ) ( ) ( ) ( ) ( ) | | 6 . 1 f 5 . 1 f 2 4 . 1 f 2 3 . 1 f 2 2 . 1 f 2 1 . 1 f 2 1 f
n 2
a b
I + + + + + +
=
Example Problem
0
10
20
30
40
50
60
0 0.5 1 1.5 2 2.5
x
f
(
x
)
x f(x)
1 2.718
1.1 3.353
1.2 4.221
1.3 5.419
1.4 7.099
1.5 9.488
1.6 12.936
( )
( ) 741 . 3 816 . 74
6 2
1 6 . 1
dx e I
6 . 1
1
x
2
=
~ =
}
.....end of example
Simpsons 1/3 Rule
Corresponds to the case where the function
is a second order polynomial
( ) ( )
( )
2
2 1 0 n
b
a
2
b
a
x a x a a x f
dx x f dx x f I
+ + =
~ =
} }
Simpsons 1/3 Rule
Designate a and b as x
0
and x
2
, and
estimate f
2
(x) as a second order Lagrange
polynomial
( ) ( )
( )( )
( )( )
( ) dx ....... x f
x x x x
x x x x
dx x f dx x f I
2
0
x
x
0
2 0 1 0
2 1
b
a
2
b
a
}
} }
(
+

=
~ =
Simpsons 1/3 Rule
After integration and algebraic
manipulation, we get the following
equations
( ) ( ) ( ) | |
( )
( ) ( ) ( )
6
x f x f 4 x f
a b
0 2
x x
h where x f x f 4 x f
3
h
I
2 1 0
0 2
2 1 0
+ +
~

= + + ~
}

}

Error
( )( )
b a where
a b ' ' f
12
1
E
3
t
< <
=
Single application of Trapezoidal Rule.
Single application of Simpsons 1/3 Rule
( )( )
5
) 4 (
t
a b f
2880
1
E =
Multiple Application of
Simpsons 1/3 Rule
( )
( ) ( ) ( ) ( )
( )
( ) 4
4
5
a
1 n
.. 5 , 3 , 1 i
2 n
.. 6 , 4 , 2 j
n j i 0
x
x
x
x
x
x
f
n 180
a b
E
n 3
x f x f 2 x f 4 x f
a b I
dx ) x ( f dx ) x ( f dx ) x ( f I
n
1 n
2
1
1
0
=
+ + +
~
+ + + =

} } }
( )
( ) ( ) ( ) ( )
n 3
x f x f 2 x f 4 x f
a b I
1 n
.. 5 , 3 , 1 i
2 n
.. 6 , 4 , 2 j
n j i 0
=
+ + +
~
The odd points represent the middle term for each application.
Hence carry the weight 4
The even points are common to adjacent applications
and are counted twice.
Simpsons 3/8 Rule
Corresponds to the case where the function
is a third order polynomial
( ) ( )
( )
( ) ( ) ( ) ( ) | |
0 3
x x
h where
x f x f 3 x f 3 x f
8
h 3
I
x a x a x a a x f
dx x f dx x f I
0 3
3 2 1 0
3
3
2
2 1 0 n
b
a
3
b
a
=
+ + + ~
+ + + =
~ =
} }
Integration of Unequal Segments
Experimental and field study data is often
unevenly spaced
In previous equations we grouped the term (i.e. h
i
)
which represented segment width.
( )
( ) ( ) ( )
( ) ( ) ( ) ( )
( ) ( )
2
x f x f
h
2
x f x f
h
2
x f x f
h I
n 2
x f x f 2 x f
a b I
n 1 n
2 1 1 0
1 n
1 i
n i 0
+
+
+
+
+
~
+ +
~
Integration of Unequal Segments

We should also consider alternately using
higher order equations if we can find data in
consecutively even segments
trapezoidal
rule
1/3
rule
3/8
rule
trapezoidal
rule
Example
Integrate the following using the trapezoidal rule,
Simpsons 1/3 Rule and a multiple application of
the trapezoidal rule with n=2. Compare results with
the analytical solution.
}
4
0
x 2
dx xe
0
5000
10000
15000
20000
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
x
f
(
x
)
Solution
( ) 927 . 5216 1 x 2
4
e
dx xe
4
0
x 2
4
0
x 2
= =
}
First, calculate the analytical solution for this problem.
Consider a single application
of the trapezoidal rule.
f(4) = 11923.83
f(0) = 0

( )
( ) ( )
( )
% 357 100
93 . 5216
66 . 23847 93 . 5216
66 . 23847
2
83 . 11923 0
0 4
2
b f a f
a b I
t
=
= c
=
+
~
+
~
0
5000
10000
15000
20000
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
x
f
(
x
)
( )
( ) ( ) ( )
( )
( )
( )
% 133 100
93 . 5216
22 . 12142 93 . 5216
22 . 12142
2 2
83 . 11923 196 . 109 2 0
0 4
n 2
x f x f 2 x f
a b I
t
1 n
1 i
n i 0
=
= c
=
+ +
~
+ +
~

=
Multiple Application of
the Trapezoidal Rule
We are obviously not
doing very well on our
estimates.
Lets consider a scheme
where we weight the
estimates
....end of example
0
5000
10000
15000
20000
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
x
f
(
x
)
Integration of Equations
Integration of analytical as opposed to
tabular functions
Romberg Integration
Richardsons Extrapolation
Romberg Integration Algorithm
Gauss Quadrature
Improper Integrals
Richardsons Extrapolation
Use two estimates of an integral to compute a third more
accurate approximation
The estimate and error associated with a multiple
application trapezoidal rule can be represented generally
as:
I = I(h) + E(h)
where I is the exact value of the integral
I(h) is the approximation from an n-segment application
E(h) is the truncation error
h is the step size (b-a)/n
Make two separate estimates using step sizes
of h
1
and h
2
.

I(h
1
) + E(h
1
) = I(h
2
) + E(h
2
)

Recall the error of the multiple-application of the trapezoidal
rule
' ' f h
12
a b
E
2
=
Assume that is constant regardless of the step size
' ' f
( )
( )
2
2
2
1
2
1
h
h
h E
h E
~
( )
( )
( ) ( )
2
2
1
2 1
2
2
2
1
2
1
h
h
h E h E
h
h
h E
h E
|
|
.
|
\
|
~
~
Substitute into previous equation:

I(h
1
) + E(h
1
) = I(h
2
) + E(h
2
)
( )
( ) ( )
2
2
1
2 1
2
h
h
1
h I h I
h E
|
|
.
|
\
|
=
Thus we have developed an estimate of the truncation
error in terms of the integral estimates and their step
sizes. This estimate can then be substituted into:

I = I(h
2
) + E(h
2
)

to yield an improved estimate of the integral:
( ) ( ) ( ) | |
1 2
2
2
1
2
h I h I
1
h
h
1
h I I
(
(
(
(
|
.
|
\
|
+ ~
( )
( ) ( )
2
2
1
2 1
2
h
h
1
h I h I
h E
|
|
.
|
\
|
=
( ) ( ) ( ) | |
1 2
2
2
1
2
h I h I
1
h
h
1
h I I
(
(
(
(
|
.
|
\
|
+ ~
What is the equation for the special case where
the interval is halved?

i.e. h
2
= h
1
/ 2
( ) ( ) ( ) | |
( ) ( )
1 2
1 2
2
2
2
2
2
1
h I
3
1
h I
3
4
I
terms collecting
h I h I
1 2
1
h I I
2
h
h 2
h
h
~
+ ~
=
|
.
|
\
|
=
|
.
|
\
|
Example
Use Richardsons extrapolation to evaluate:
}
4
0
x 2
dx xe
0
5000
10000
15000
20000
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
x
f
(
x
)
Solution
Recall our previous example where we calculated the integral
using a single application of the trapezoidal rule, and then
a multiple application of the trapezoidal rule by dividing the
interval in half.
0
5000
10000
15000
20000
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
x
f
(
x
)
SINGLE APPLICATION
OF TRAPEZOIDAL RULE

MULTIPLE APPLICATION
OF TRAPEZOIDAL RULE (n=2)

RICHARDSONS
EXTRAPOLATION
357%

133%

57.96%
....end of example
( ) ( )
( ) ( )
( ) ( )
1 4
I I 4
I
h I
63
1
h I
63
64
I
h I
15
1
h I
15
16
I
h I
3
1
h I
3
4
I
1 k
1 k , j 1 k , 1 j
1 k
k , j
m
m
1 2
~
~
~
~
We can continue to improve the estimate by successive

halving of the step size to yield a general formula:
k = 2; j = 1
ROMBERG INTEGRATION
Note:
the subscripts
m and refer to
more and less
accurate estimates
k = 3
k = 4
Following a similar pattern to Newton divided differences,
Rombergs Table can be produced
Error orders for j values
i = 1 i = 2 i = 3 i = 4
j O(h
2
) O(h
4
) O(h
6
) O(h
8
)

1 h I
1,1
I
1,2
I
1,3
I
1,4

2 h/2 I
2,1
I
2,2
I
2,3

3 h/4 I
3,1
I
3,2

4 h/8 I
4,1

Trapezoidal Simpsons 1/3 Booles

Rule Rule Rule

( ) ( )
h I
3
1
h I
3
4
I
m
~
i = 1 i = 2 i = 3 i = 4
j O(h
2
) O(h
4
) O(h
6
) O(h
8
)

1 h I
1,1
I
1,2
I
1,3
I
1,4

2 h/2 I
2,1
I
2,2
I
2,3

3 h/4 I
3,1
I
2,3

4 h/8 I
4,1


Rule Rule Rule

( ) ( )
h I
3
1
h I
3
4
I
m
~
i = 1 i = 2 i = 3 i = 4
j O(h
2
) O(h
4
) O(h
6
) O(h
8
)

1 h I
1,1
I
1,2
I
1,3
I
1,4

2 h/2 I
2,1
I
2,2
I
2,3

3 h/4 I
3,1
I
3,2

4 h/8 I
4,1


Rule Rule Rule

( ) ( )
h I
15
1
h I
15
16
I
m
~
i = 1 i = 2 i = 3 i = 4
j O(h
2
) O(h
4
) O(h
6
) O(h
8
)

1 h I
1,1
I
1,2
I
1,3
I
1,4

2 h/2 I
2,1
I
2,2
I
2,3

3 h/4 I
3,1
I
3,2

4 h/8 I
4,1


Rule Rule Rule

( ) ( )
h I
3
1
h I
3
4
I
m
~
i = 1 i = 2 i = 3 i = 4
j O(h
2
) O(h
4
) O(h
6
) O(h
8
)

1 h I
1,1
I
1,2
I
1,3
I
1,4

2 h/2 I
2,1
I
2,2
I
2,3

3 h/4 I
3,1
I
3,2

4 h/8 I
4,1


Rule Rule Rule

( ) ( )
h I
15
1
h I
15
16
I
m
~
i = 1 i = 2 i = 3 i = 4
j O(h
2
) O(h
4
) O(h
6
) O(h
8
)

1 h I
1,1
I
1,2
I
1,3
I
1,4

2 h/2 I
2,1
I
2,2
I
2,3

3 h/4 I
3,1
I
3,2

4 h/8 I
4,1


Rule Rule Rule

( ) ( )
h I
63
1
h I
63
64
I
m
~
Gauss Quadrature
f(x)
x
f(x)
x
Extend the area
under the straight
line
Method of Undetermined
Coefficients
Recall the trapezoidal rule
( )
( ) ( )
2
b f a f
a b I
+
~
This can also be expressed as
( ) ( ) b f c a f c I
1 0
+ ~
where the cs are constant
Before analyzing
this method,
answer this
question.
What are two
functions that
should be evaluated
exactly
by the trapezoidal
rule?
The two cases that should be evaluated exactly
by the trapezoidal rule: 1) y = constant
2) a straight line
f(x)
x
y = 1
(b-a)/2
-(b-a)/2
f(x)
x
y = x
(b-a)/2
-(b-a)/2
Thus, the following equalities should hold.
( ) ( )
( )
( )
( )
( )
}
}
~
|
.
|
\
|

+
|
.
|
\
|

~ +
+ ~
2
a b
2
a b 1 0
2
a b
2
a b 1 0
1 0
xdx
2
a b
c
2
a b
c
dx 1 c c
b f c a f c I
For y = 1
since f(a) = f(b) =1
For y = x
since f(a) = x =-(b-a)/2
and
f(b) = x =(b-a)/2
Evaluating both integrals
0
2
1 b
c
2
a b
c
a b c c
1 0
1 0
=
= +
For y = 1

For y = x
Now we have two equations and two unknowns, c
0
and c
1
.

Solving simultaneously, we get :

c
0
= c
1
= (b-a)/2

Substitute this back into:

( ) ( ) b f c a f c I
1 0
+ ~
( )
( ) ( )
2
b f a f
a b I
+
~
We get the equivalent of the trapezoidal rule.
DERIVATION OF THE TWO-POINT
GAUSS-LEGENDRE FORMULA

( ) ( )
1 1 0 0
x f c x f c I + ~
Lets raise the level of sophistication by:
- considering two points between -1 and 1
- i.e. open integration
f(x)
x
-1 x
0
x
1
1
Previously ,we assumed that the equation
fit the integrals of a constant and
linear function.

Extend the reasoning by assuming that
it also fits the integral of a parabolic and a cubic function.
( ) ( )
( ) ( )
( ) ( )
( ) ( )
}
}
}
}
= = +
= = +
= = +
= = +
1
1
3
1 1 0 0
1
1
2
1 1 0 0
1
1
1 1 0 0
1
1
1 1 0 0
0 dx x x f c x f c
3 / 2 dx x x f c x f c
0 xdx x f c x f c
2 dx 1 x f c x f c
We now have four
equations and four
unknowns

c
0
c
1
x
0
and x
1

What equations are
you solving?
( ) ( ) ( ) ( )
( ) ( )
( ) ( )
( ) ( ) 0 x c x c 0 dx x x f c x f c
3
2
x c x c
3
2
dx x x f c x f c
0 x c x c 0 xdx x f c x f c
2 1 c 1 c 2 dx 1 x f c x f c
3
1 1
3
0 0
1
1
3
1 1 0 0
2
1 1
2
0 0
1
1
2
1 1 0 0
` 1 1 0 0
1
1
1 1 0 0
1 0
1
1
1 1 0 0
= + = = +
= + = = +
= + = = +
= + = = +
}
}
}
}
Solve these equations simultaneously

f(x
i
) is either 1, x
i
, x
i
2
or x
i
3
|
.
|
\
|
+ |
.
|
\
|
~
=
=
= =
3
1
f
3
1
f I
3
1
x
3
1
x
1 c c
1
0
1 0
This results in the following
The interesting result is that the simple addition
of the function values at
3
1
and
3
1
However, we have set the limit of integration at -1 and 1.

This was done to simplify the mathematics. A simple
change in variables can be use to translate other limits.

Assume that the new variable x
d
is related to the
original variable x in a linear fashion.

x = a
0
+ a
1
x
d

Let the lower limit x = a correspond to x
d
= -1 and the upper
limit x=b correspond to x
d
=1

a = a
0
+ a
1
(-1) b = a
0
+ a
1
(1)

a = a
0
+ a
1
(-1) b = a
0
+ a
1
(1)
SOLVE THESE EQUATIONS
SIMULTANEOUSLY
2
a b
a
2
a b
a
1 0
=
+
=
( ) ( )
d
d
d 1 0
x
2
a b
2
a b
x or
2
x a b a b
x a a x
|
.
|
\
|

+
+
=
+ +
= + =
substitute
d
d d 1 0
dx
2
a b
dx
x
2
a b
2
a b
x a a x
|
.
|
\
|

=
|
.
|
\
|

+
+
= + =
These equations are substituted for x and dx respectively.

Lets do an example to appreciate the theory
behind this numerical method.
Example
Estimate the following using two-point Gauss Quadrature:
}
4
0
x 2
dx xe
0
5000
10000
15000
20000
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
x
f
(
x
)
( )
( )
}
+
+
1
1
d
x 2 2 2
d
dx 2 e x 2 2
d
Now evaluate the integral
55 . 3477 38 . 3468 17 . 9
2 e
3
1
2 2 2 e
3
1
2 2
3
1
f
3
1
f I
3
1
2 2 2
3
1
2 2 2
= + =
|
|
.
|
\
|
|
.
|
\
|
+ +
|
|
.
|
\
|
|
.
|
\
|
+ =
|
.
|
\
|
+ |
.
|
\
|
~
|
|
.
|
\
|
|
.
|
\
|
+
|
|
.
|
\
|
|
.
|
\
|
+
....end of problem
SINGLE APPLICATION
OF TRAPEZOIDAL RULE

MULTIPLE APPLICATION
OF TRAPEZOIDAL RULE (n=2)

RICHARDSONS
EXTRAPOLATION

2-POINT GAUSS
QUADRATURE
357%

133%

58%

33 %
Higher-Point Gauss Formulas
( ) ( ) ( )
1 n 1 n 1 1 0 0
x f c x f c x f c I

+ + + ~
For two point, we determined that c
0
=c
1
= 1

For three point:

c
0
= 0.556 (5/9) x
0
=-0.775 = - (3/5)
1/2

c
1
= 0.889 (8/9) x
1
=0.0
c
2
= 0.556 (5/9) x
2
=0.775 = (3/5)
1/2

Higher-Point Gauss Formulas
For four point:

c
0
= {18-(30)
1/2
}/36 x
0
= -{525+70(30)
1/2
}
1/2
/35
c
1
= {18+(30)
1/2
}/36 x
1
= -{525-70(30)
1/2
}
1/2
/35
c
2
= {18+(30)
1/2
}/36 x
2
= +{525-70(30)
1/2
}
1/2
/35
c
3
= {18-(30)
1/2
}/36 x
3
= +{525+70(30)
1/2
}
1/2
/35

Your text goes on to provide additional weighting
factors (c
i
s) and function arguments (x
i
s)
in Table 22.1 p. 626.
Numerical Differentiation
Forward finite divided difference
Backward finite divided difference
Center finite divided difference
All based on the Taylor Series
( ) ( ) ( )
( )
......... h
! 2
x ' ' f
h x ' f x f x f
2
i
i i 1 i
+ + + =
+
Forward Finite Difference
( ) ( ) ( )
( ) ( )
( )
( ) ( )
( )
( )
( ) ( ) ( )
( )
2
i i 1 i
i
i 1 i
i
3
i
2
i
i i 1 i
h O h
2
x ' ' f
h
x f x f
x ' f
h O
h
x f x f
x ' f
......... h
! 3
x ' ' ' f
h
! 2
x ' ' f
h x ' f x f x f
+
=
+
=
+ + + + =
+
+
+
Forward Divided Difference
f(x)
x
(x
i
, y
i
)
(x
i+1
,y
i+1
)
( )
( ) ( )
( ) ( ) h O
h
f
x x O
x x
x f x f
x ' f
i
i 1 i
i 1 i
i 1 i
i
+
A
= +
=
+
+
+
first forward divided difference
Error is proportional to
the step size
O(h
2
) error is proportional to the square of the step size

O(h
3
) error is proportional to the cube of the step size
( ) ( ) h O
h
f
x ' f
i
i
+
A
=
f(x)
x
(x
i
,y
i
)
(x
i-1
,y
i-1
)
( ) ( ) ( )
( )
( ) ( ) ( )
( )
( )
( ) ( )
h
f
h
x f x f
x ' f
..... h
! 2
x ' ' f
h x ' f x f x f
...... h
! 2
x ' ' f
h x ' f x f x f
i 1 i i
i
2
i
i i 1 i
2
i
i i 1 i
A
=
=
+ + =
+ + + =
+
Backward Difference Approximation of the
First Derivative

Expand the Taylor series backwards
The error is still O(h)
Centered Difference Approximation of the
First Derivative and Second Derivative

Subtract and add backward Taylor expansion
from and to the forward Taylor series expansion
( ) ( ) ( )
( ) ( )
( ) ( ) ( )
( ) ( )
( )
( ) ( )
( )
( )
( ) ( ) ( )
( )
2
2
1 i i 1 i
i
2
1 i 1 i
i
3
i
2
i
i i 1 i
3
i
2
i
i i 1 i
h O
h
x f x f 2 x f
x ' ' f
h O
h 2
x f x f
x ' f
h
6
x ' ' ' f
h
2
x ' ' f
h x ' f x f x f
.... h
6
x ' ' ' f
h
2
x ' ' f
h x ' f x f x f
+ +
=
=
+ + =
+ + + + =
+
+
Subtracting
Adding
Forward
Backward
f(x)
x
(x
i
,y
i
)
(x
i-1
,y
i-1
)
(x
i+1
,y
i+1
)
( )
( ) ( )
( )
2
1 i 1 i
i
h O
h 2
x f x f
x ' f
=
+
Numerical Differentiation
Forward finite divided differences Fig. 23.1
Backward finite divided differences Fig. 23.2
Centered finite divided differences Fig. 23.3
First - Fourth derivative
Error: O(h), O(h
2
), O(h
4
) for centered only
Derivatives with Richardson
Extrapolation
Two ways to improve derivative estimates
decrease step size
use a higher order formula that employs more
points
Third approach, based on Richardson
extrapolation, uses two derivatives
estimates to compute a third, more accurate
approximation
Richardson Extrapolation
( ) ( ) ( )
( ) ( )
( ) ( )
2 2 1
2
1
2
1
2
2 1
2 1
1
I I h I h I h
h
1
h
h
Special case where h
2
4 1
I I h I h
3 3
In a similar fashion
4 1
D D h D h
3 3
( ~ +

| |

|
\ .
=
~
~
For a centered difference
approximation with
O(h
2
) the application of
this formula will yield
a new derivative estimate
of O(h
4
)
Example
Given the function:

f(x) = -0.1x
4
- 0.15x
3
- 0.5x
2
- 0.25x +1.2

Use centered finite divided difference to estimate
the derivative at 0.5.

f(0) = 1.2
f(0.25) =1.1035
f(0.75) = 0.636
f(1) = 0.2

Example Problem
0
0.2
0.4
0.6
0.8
1
1.2
1.4
0 0.2 0.4 0.6 0.8 1 1.2
x
f
(
x
)
Using centered finite divided difference for h = 0.5

f(0) = 1.2
f(1) = 0.2

D(0.5) = (0.2 - 1.2)/1 = -1.0 c
t
= 9.6%

Using centered finite divided difference for h = 0.25

f(0.25) =1.1035
f(0.75) = 0.636
D(0.25) = (0.636 - 1.1035)/0.5 =-0.934 c
t
= 2.4%

Derivatives of Unequally Spaced
Data
Common in data from experiments or field studies
Fit a second order Lagrange interpolating polynomial to
each set of three adjacent points, since this polynomial
does not require that the points be uniformly spaced
Differentiate analytically
( ) ( )
( )( )
( )
( )( )
( )
( )( )
1 i 1 i i 1 i
1 i i
1 i
1 i i 1 i i
1 i 1 i
i
1 i 1 i i 1 i
1 i i
1 i
x x x x
x x x 2
x f
x x x x
x x x 2
x f
x x x x
x x x 2
x f x ' f
+ +
+
+
+
+
+

+

+

=
Derivative and Integral Estimates
for Data with Errors
In addition to unequal spacing, the other problem related to
differentiating empirical data is measurement error
Differentiation amplifies error
Integration tends to be more forgiving
Primary approach for determining derivatives of imprecise
data is to use least squares regression to fit a smooth,
differentiable function to the data
In absence of other information, a lower order polynomial
regression is a good first choice
0
100
200
300
0 10 20
t
y
0
10
20
30
0 5 10 15
t
d
y
/
d
t
0
50
100
150
200
250
0 5 10 15
t
y
0
10
20
30
40
0 5 10 15
0
50
100
150
200
250
0 5 10 15
0
10
20
30
40
0 5 10 15
t
d
y
/
d
t
Ordinary Differential Equations
A differential equation defines a
relationship between an unknown function
and one or more of its derivatives
Physical problems using differential
equations
electrical circuits
heat transfer
motion
The derivatives are of the dependent
variable with respect to the independent
variable
First order differential equation with y as
the dependent variable and x as the
independent variable would be:
dy/dx = f(x,y)
A second order differential equation would
have the form:
}

does not necessarily have to include
all of these variables
|
.
|
\
|
=
dx
dy
, y , x f
dx
y d
2
2
An ordinary differential equation is one
with a single independent variable.
Thus, the previous two equations are
ordinary differential equations
The following is not:
( ) y , x , x f
dx
dy
2 1
1
=
The analytical solution of ordinary
differential equation as well as partial
differential equations is called the closed
form solution
This solution requires that the constants of
integration be evaluated using prescribed
values of the independent variable(s).
An ordinary differential equation of order n
requires that n conditions be specified.
Boundary conditions
Initial conditions
consider this beam where the
deflection is zero at the boundaries
x= 0 and x = L
These are boundary conditions
consider this beam where the
deflection is zero at the boundaries
x= 0 and x = L
These are boundary conditions
a
y
o
P
In some cases, the specific behavior of a system(s)
is known at a particular time. Consider how the deflection of a
beam at x = a is shown at time t =0 to be equal to y
o
.
Being interested in the response for t > 0, this is called the
initial condition.
At best, only a few differential equations
can be solved analytically in a closed form.
Solutions of most practical engineering
problems involving differential equations
require the use of numerical methods.
Review of Analytical Solution
C
3
x 4
y
dx x 4 dy
x 4
dx
dy
3
2
2
+ =
=
=
} }
At this point lets consider
initial conditions.

y(0)=1
and
y(0)=2
( )
( )
( )
( )
2 C and
C
3
0 4
2
2 0 y for
1 C then
C
3
0 4
1
1 0 y for
C
3
x 4
y
3
3
3
=
+ =
=
=
+ =
=
+ =
What we see are different
values of C for the two
different initial conditions.

The resulting equations
are:

2
3
x 4
y
1
3
x 4
y
3
3
+ =
+ =
0
4
8
12
16
0 0.5 1 1.5 2 2.5
x
y
y(0)=1
y(0)=2
y(0)=3
y(0)=4
One Step Methods
Focus is on solving ODE in the form
y
x
slope = |
y
i
y
i+1
h
This is the same as saying:
new value = old value + slope * step size
( )
h y y
y , x f
dx
dy
i 1 i
| + =
=
+
Eulers Method
The first derivative provides a direct
estimate of the slope at x
i
The equation is applied iteratively, or one
step at a time, over small distance in order
to reduce the error
Hence this is often referred to as Eulers
One-Step Method
Example
4413 . 1 y
4413 . 0 x
3
4
1 y
dx x 4 dy
solution Analytical
1 . 0 size step
1 x at 1 y . C . I
x 4
dx
dy
1 . 1
1
3
1 . 1
1
2
y
1
2
=
= =
=
= =
=
} }
( ) ( ) ( ) | |( )
( ) ( ) ( ) | |( ) 1 . 0 1 4 1 f 1 . 1 f
: Note
4 . 1 1 . 0 1 4 1 f 1 . 1 f
h y y
x 4
dx
dy
2
2
i 1 i
2
+ =
= + =
| + =
=
+
I.C.
Slope
step size
Recall the analytical solution was 1.4413
If we instead reduced the step size to to 0.05 and
apply Eulers twice
Error Analysis of Eulers Method
Truncation error - caused by the nature of
the techniques employed to approximate
values of y
local truncation error (from Taylor Series)
propagated truncation error
sum of the two = global truncation error
Round off error - caused by the limited
number of significant digits that can be
retained by a computer or calculator
Example
0
2
4
6
8
10
12
0 1 2 3
x
y
Analyt ical
Solut ion
Numerical
Solut ion
....end of example
Higher Order Taylor Series
Methods
This is simple enough to implement with
polynomials
Not so trivial with more complicated ODE
In particular, ODE that are functions of both
dependent and independent variables require
chain-rule differentiation
Alternative one-step methods are needed
( )
( )
2
i i
i i i 1 i
h
2
y , x ' f
h y , x f y y + + =
+
Modification of Eulers Methods
A fundamental error in Eulers method is
that the derivative at the beginning of the
interval is assumed to apply across the
entire interval
Two simple modifications will be
demonstrated
These modification actually belong to a
larger class of solution techniques called
Runge-Kutta which we will explore later.
Heuns Method
Consider our Taylor expansion
( )
( )
2
i i
i i i 1 i
h
2
y , x ' f
h y , x f y y + + =
+
Approximate f as a simple forward difference
( )
( ) ( )
h
y , x f y , x f
y , x ' f
i i 1 i 1 i
i i
~
+ +
Substituting into the expansion
h
2
f f
y
2
h
h
f f
h f y y
i 1 i
i
2
i 1 i
i i 1 i
|
.
|
\
|
+
+ =
|
.
|
\
|

+ + =
+ +
+
Heuns Method Algorithm
Determine the derivatives for the interval @
the initial point
end point (based on Euler step from initial point)
Use the average to obtain an improved
estimate of the slope for the entire interval
We can think of the Euler step as a test step
y
x
i
x
i+1
h
y
x
i
x
i+1
Take the average of these
two slopes
y
x
i
x
i+1
y
x
x
i
x
i+1
( ) ( )
h
2
y , x f y , x f
y y
1 i 1 i i i
i 1 i
+ +
+
+
+ =
Improved Polygon Method
Another modification of Eulers Method
(sometimes called the Midpoint Method)
Uses Eulers to predict a value of y at the
midpoint of the interval

This predicted value is used to estimate the
slope at the midpoint
( )
2
h
y , x f y y
i i i 2 / 1 i
+ =
+
( )
2 / 1 i 2 / 1 i 2 / 1 i
y , x f ' y
+ + +
=
Improved Polygon Method
We then assume that this slope represents a valid
approximation of the average slope for the entire
interval

( ) h y , x f y y
2 / 1 i 2 / 1 i i 1 i + + +
+ =
Use this slope to extrapolate linearly from x
i
to
x
i+1
using Eulers algorithm

We could also get this algorithm from
substituting a forward difference in f to i+1/2 into
the Taylor expansion for f, i.e.

h f y
2
h
2 / h
f f
h f y y
2 / 1 i i
2
i 2 / 1 i
i i 1 i +
+
+
+ =
|
.
|
\
|

+ + =
y
x
x
i
x
i+1/2
f(x
i+1/2
)
y
x
x
i
x
i+1/2
f(x
i+1/2
)
y
x
x
i
x
i+1/2
x
i+1
h
Extend your slope
now to get f(x
i+1
)
Runge-Kutta Methods
RK methods achieve the accuracy of a Taylor
series approach without requiring the calculation
of a higher derivative
Many variations exist but all can be cast in the
generalized form:
( ) h h , y , x y y
i i i 1 i
| + =
+
{

| is called the incremental function
| , Incremental Function
can be interpreted as a representative slope
over the interval
( )
( )
( )
( ) h k q h k q h k q y , h p x f k
h k q h k q y , h p x f k
h k q y , h p x f k
y , x f k
: are s ' k the and t tan cons are s ' a the where
k a k a k a
1 n 1 n , 1 n 2 2 , 1 n 1 1 , 1 n i n i n
2 22 1 21 i 2 i 3
1 11 i 1 i 2
i i 1
n n 2 2 1 1

+ + + + + =
+ + + =
+ + =
=
+ + + = |
NOTE:
ks are recurrence relationships,
that is k
1
appears in the equation for k
2

and each appear in the equation for k
3
This recurrence makes RK methods efficient for
computer calculations
( )
( )
( )
h k q y , h p x f k
y , x f k
k a k a k a
1 n 1 n , 1 n 2 2 , 1 n 1 1 , 1 n i n i n
2 22 1 21 i 2 i 3
1 11 i 1 i 2
i i 1
n n 2 2 1 1

+ + + + + =
+ + + =
+ + =
=
+ + + = |
Second Order RK Methods

( )
( )
( ) h k q y , h p x f k
y , x f k
where
h k a k a y y
1 11 i 1 i 2
i i 1
2 2 1 1 i 1 i
+ + =
=
+ + =
+
( )
( )
( )
h k q y , h p x f k
y , x f k
k a k a k a
1 n 1 n , 1 n 2 2 , 1 n 1 1 , 1 n i n i n
2 22 1 21 i 2 i 3
1 11 i 1 i 2
i i 1
n n 2 2 1 1

+ + + + + =
+ + + =
+ + =
=
+ + + = |
Second Order RK Methods

We have to determine values for the constants a
1
,
a
2
, p
1
and q
11
To do this consider the Taylor series in terms of
y
i+1
and f(x
i
,y
i
)
( )
( ) ( )
2
h
y , x ' f h y , x f y y
h k a k a y y
2
i i i i i 1 i
2 2 1 1 i 1 i
+ + =
+ + =
+
+
( )
( )
2
h
dx
dy
y
f
x
f
h y , x f y y
ansion exp to in substitute
dx
dy
y
f
x
f
y , x ' f
2
i i i 1 i
i i
|
|
.
|
\
|
c
c
+
c
c
+ + =
c
c
+
c
c
=
+
Now, f(x
i
, y
i
) must be determined by the
chain rule for differentiation
The basic strategy underlying Runge-Kutta methods
is to use algebraic manipulations to solve for values
of a
1
, a
2
, p
1
and q
11

( )
( ) ( ) ( ) ( )
2
h
y , x
dx
dy
y , x
y
f
y , x
x
f
h y , x f y y
h k a k a y y
2
i i i i i i i i i 1 i
2 2 1 1 i 1 i
|
|
.
|
\
|
c
c
+
c
c
+ + =
+ + =
+
+
By setting these two equations equal to each other and
recalling:
( )
( ) h k q y , h p x f k
y , x f k
1 11 i 1 i 2
i i 1
+ + =
=
we derive three equations to evaluate the four unknown
constants
2
1
q a
2
1
p a
1 a a
11 2
1 2
2 1
=
=
= +
Because we have three equations with four unknowns,
we must assume a value of one of the unknowns.

Suppose we specify a value for a
2
.

What would the equations be?
2
11 1
2 1
a 2
1
q p
a 1 a
= =
=
Because we can choose an infinite number of values
for a
2
there are an infinite number of second order
RK methods.

Every solution would yield exactly the same result
if the solution to the ODE were quadratic, linear or a
constant.

Lets review three of the most commonly used and
preferred versions.
( )
( )
( )
2
1
q a
2
1
p a
1 a a
h k q y , h p x f k
y , x f k
where
h k a k a y y
11 2
1 2
2 1
1 11 i 1 i 2
i i 1
2 2 1 1 i 1 i
=
=
= +
+ + =
=
+ + =
+
Consider the following:

Case 1: a
2
= 1/2

Case 2: a
2
= 1

These two methods
have been previously
studied.

What are they?
( )
( ) h k y , h x f k
y , x f k
where
h k
2
1
k
2
1
y y
1
a 2
1
q p
2
1
q a
2
1
p a
2 / 1 2 / 1 1 a 1 a
1 i i 2
i i 1
2 1 i 1 i
2
11 1
11 2
1 2
2 1
+ + =
=
|
.
|
\
|
+ + =
= = =
=
=
= = =
+
Case 1: a
2
= 1/2

This is Heuns Method with
a single corrector.

Note that k
1
is the slope at
the beginning of the interval
and k
2
is the slope at the
end of the interval.
( )
( )
( ) h k q y , h p x f k
y , x f k
where
h k a k a y y
1 11 i 1 i 2
i i 1
2 2 1 1 i 1 i
+ + =
=
+ + =
+
( )
|
.
|
\
|
+ + =
=
+ =
= = =
=
=
= = =
+
h k
2
1
y , h
2
1
x f k
y , x f k
where
h k y y
2
1
a 2
1
q p
2
1
q a
2
1
p a
0 1 1 a 1 a
1 i i 2
i i 1
2 i 1 i
2
11 1
11 2
1 2
2 1
( )
( )
( ) h k q y , h p x f k
y , x f k
where
h k a k a y y
1 11 i 1 i 2
i i 1
2 2 1 1 i 1 i
+ + =
=
+ + =
+
Case 2: a
2
= 1

This is the Improved Polygon
Method.
Ralstons Method

Ralston (1962) and Ralston and Rabinowitiz (1978)
determined that choosing a
2
= 2/3 provides a minimum
bound on the truncation error for the second order RK
algorithms.

This results in a
1
= 1/3 and p
1
= q
11
= 3/4
( )
|
.
|
\
|
+ + =
=
|
.
|
\
|
+ + =
+
h k
4
3
y , h
4
3
x f k
y , x f k
where
h k
3
2
k
3
1
y y
1 i i 2
i i 1
2 1 i 1 i
Example
( ) ( )
1 . 0 h size step
1 1 y . e . i 1 x at 1 y : . C . I
y x 4
dx
dy
2
=
= = =
=
As a class problem, lets
consider two steps.

Some of you folks do the
analytical solution,
others do either:

Ralstonss
Heuns
Improved Polygon
Third Order Runge-Kutta Methods
Derivation is similar to the one for the second-order
Results in six equations and eight unknowns.
One common version results in the following
( )
( )
( )
2 1 i i 3
1 i i 2
i i 1
3 2 1 i 1 i
hk 2 hk y , h x f k
h k
2
1
y , h
2
1
x f k
y , x f k
where
h k k 4 k
6
1
y y
+ + =
|
.
|
\
|
+ + =
=
(
+ + + =
+
Note the third term
NOTE: if the derivative is a function of x only, this reduces to Simpsons 1/3 Rule
Fourth Order Runge Kutta
The most popular
The following is sometimes called the classical
fourth-order RK method
( )
( )
( )
3 i i 4
2 i i 3
1 i i 2
i i 1
4 3 2 1 i 1 i
hk y , h x f k
hk
2
1
y , h
2
1
x f k
h k
2
1
y , h
2
1
x f k
y , x f k
where
h k k 2 k 2 k
6
1
y y
+ + =
|
.
|
\
|
+ + =
|
.
|
\
|
+ + =
=
(
+ + + + =
+
Note that for ODE that are a function of x alone
that this is also the equivalent of Simpsons 1/3
Rule
( )
( )
( )
3 i i 4
2 i i 3
1 i i 2
i i 1
4 3 2 1 i 1 i
hk y , h x f k
hk
2
1
y , h
2
1
x f k
h k
2
1
y , h
2
1
x f k
y , x f k
where
h k k 2 k 2 k
6
1
y y
+ + =
|
.
|
\
|
+ + =
|
.
|
\
|
+ + =
=
(
+ + + + =
+
Example
Use 4th Order RK to solve the following differential equation:
( ) 1 1 y . C . I
x 1
xy
dx
dy
2
=
+
=
using an interval of h = 0.1
Solution
( )
( )
( )
3 i i 4
2 i i 3
1 i i 2
i i 1
4 3 2 1 i 1 i
hk y , h x f k
hk
2
1
y , h
2
1
x f k
h k
2
1
y , h
2
1
x f k
y , x f k
where
h k k 2 k 2 k
6
1
y y
+ + =
|
.
|
\
|
+ + =
|
.
|
\
|
+ + =
=
(
+ + + + =
+
We will determine
different estimates
of the slope
i.e. k
1
, k
2
, k
3
and k
4
( )
( ) ( ) ( ) ( )
05119 . 1
1 . 0 52323 . 0 51219 . 0 2 51189 . 0 2 5 . 0
6
1
1
h k k 2 k 2 k
6
1
y y
4 3 2 1 i 1 i
=
(
+ + + + =
(
+ + + + =
+
.....end of problem
Higher Order RK Methods
When more accurate results are required,
Buchers (1964) fifth order RK method is
recommended
There is a similarity to Booles Rule
The gain in accuracy is offset by added
computational effort and complexity
Systems of Equations
Many practical problems in engineering and
science require the solution of a system of
simultaneous differential equations
( )
( )
( )
n 2 1 n
n
n 2 1 2
2
n 2 1 1
1
y , , y , y , x f
dx
dy
y , , y , y , x f
dx
dy
y , , y , y , x f
dx
dy
=
=
=
Solution requires n initial conditions
All the methods for single equations can be used
The procedure involves applying the one-step
technique for every equation at each step before
proceeding to the next step
( )
( )
( )
n 2 1 n
n
n 2 1 2
2
n 2 1 1
1
y , , y , y , x f
dx
dy
y , , y , y , x f
dx
dy
y , , y , y , x f
dx
dy
=
=
=
Note that higher order ODEs can be reformulated
as simultaneous first order ODEs
( )
2 1
2
2
1
2 1
2
2
y , y , x g
dx
dy
y
dx
dy
so ,
dx
dy
y & y y defining by
d transforme be can
dx
dy
, y , x g
dx
y d
=
=

|
.
|
\
|
=
Partial Differential Equations
An equation involving partial derivatives of an
unknown function of two or more independent
variables
The following are examples. Note: u depends on
both x and y
x
y
u
xu
x
u
y 5 u 8
y
u
x
y x
u
x
y x
u
6
x
u
1 u
y
u
xy 2
x
u
2
2
2
2 2
2
3
3
2
2
2
2
2
2
=
c
c
+
c
c
= +
c
c
+
c c
c
=
c c
c
+
|
|
.
|
\
|
c
c
= +
c
c
+
c
c
Partial Differential Equations
Because of their widespread application in
engineering, our study of PDE will focus on linear,
second-order equations
The following general form will be evaluated for
B
2
- 4AC (note that text does not list E, F, G terms)
0 Gu
y
u
F
x
u
E D
y
u
C
y x
u
B
x
u
A
2
2 2
2
2
=
|
|
.
|
\
|
+
c
c
+
c
c
+ +
c
c
+
c c
c
+
c
c
B
2
-4AC Category Example
< 0 Elliptic Laplace equation (steady state with
2 spatial dimensions

= 0 Parabolic Heat conduction equation (time variable
with one spatial dimension

>0 Hyperbolic Wave equation (time-variable with one
spatial dimension
0
y
T
x
T
2
2
2
2
=
c
c
+
c
c
t
T
x
T
k
2
2
c
c
=
c
c
2
2
2 2
2
t
y
c
1
x
y
c
c
=
c
c
y or t
x
set up a grid
estimate the dependent
variable at the center
or intersections
of the grid
Typically used to characterize steady-state
boundary value problems
Before solving, the Laplace equation will be
solved from a physical problem
0
y
u
x
u
0 D
y
u
C
y x
u
B
x
u
A
2
2
2
2
2
2 2
2
2
=
c
c
+
c
c
= +
c
c
+
c c
c
+
c
c
Finite Difference:
Elliptic Equations
B
2
- 4AC < 0
The Laplace Equation
Models a variety of problems involving the
potential of an unknown variable
We will consider cases involving
thermodynamics, fluid flow, and flow through
porous media
0
y
u
x
u
2
2
2
2
=
c
c
+
c
c
The Laplace equation
Lets consider the case of a plate heated from the
boundaries
How is this equation derived from basic concepts
of continuity?
How does it relate to flow fields?
0
y
T
x
T
2
2
2
2
=
c
c
+
c
c
Consider the plate below, with thickness Az.
The temperatures are known at the boundaries.
What is the temperature throughout the plate?

T = 200 T= 200
T = 400
T = 200
T = 200 T= 200
T = 400
T = 200
y
x
Divide into a grid, with increments by Ax and Ay
T = 200 T= 200
T = 400
T = 200
y
x
What is the temperature here, if
using a block centered scheme?
T = 200 T= 200
T = 400
T = 200
y
x
What is the temperature here, if
using a grid centered scheme?
Consider the element shown below on the face of
a plate A z in thickness.

The plate is illustrated everywhere by at its edges or
boundaries, where the temperature can be set.
y
x
A x
A y
q(y + A y)
q(x + A x)
q(x)
q(y)
By continuity, the flow of heat in must equal the flow of heat
out.
( ) ( )
( ) ( ) t z x y y q t z y x x q
t z x y q t z y x q
A A A A + + A A A A +
= A A A + A A A
Consider the heat flux q
in and out of the elemental volume.
( ) ( )
( ) ( ) t z x y y q t z y x x q
t z x y q t z y x q
A A A A + + A A A A +
= A A A + A A A
Divide by Az and A t and collect terms, this equation
reduces to:
0
y
q
x
q
=
c
c
+
c
c
Again, this is our
continuity equation
0
y
q
x
q
=
c
c
+
c
c
The link between flux and temperature is provided by Fouriers
Law of heat conduction
i
T
C k q
i
c
c
=
Where q
i
is the heat flux in the direction i.
Substitute B into A to get the Laplace equation
Equation A
Equation B
0
y
q
x
q
=
c
c
+
c
c
i
T
C k q
i
c
c
=
Equation A
Equation B
0
y
T
x
T
y
T
C k
y x
T
C k
x y
q
x
q
2
2
2
2
=
c
c
+
c
c
|
|
.
|
\
|
c
c

c
c
+
|
.
|
\
|
c
c

c
c
=
c
c
+
c
c
Consider Fluid Flow
In fluid flow, where the fluid is a liquid or a gas, the
continuity equation is:
0
y
V
x
V
y
x
=
c
c
+
c
c
The link here can by either of the following sets of equations:
The potential function:

Stream function:
y
V
x
V
y x
c
c|
=
c
c|
=
x
V
y
V
y x
c
c
=
c
c
=
0
y x
or 0
y x
2
2
2
2
2
2
2
2
=
c
c
+
c
c
=
c
| c
+
c
| c
The Laplace equation is then
0
y
V
x
V
y
x
=
c
c
+
c
c
y
V
x
V
y x
c
c|
=
c
c|
=
x
V
y
V
y x
c
c
=
c
c
=
Flow in Porous Media
0
y
q
x
q
=
c
c
+
c
c
i
H
K q
i
c
c
=
Darcys Law
The link between flux and the pressure head is
provided by Darcys Law
0
y
h
x
h
2
2
2
2
=
c
c
+
c
c
) y , x ( f
y
u
x
u
2
2
2
2
=
c
c
+
c
c
For a case with sources and sinks within the 2-D
domain, as represented by f(x,y), we have the
Poisson equation.
Now lets consider solution techniques.
Evaluate these equations based on the grid and
central difference equations
(i,j)
(i+1,j)
(i-1,j)
(i,j+1)
(i,j-1)
2
1 j , i j , i
1 j ,
i
2
2
2
j , 1 i j , i j , 1 i
2
2
y
u u 2 u
y
u
x
u u 2 u
x
u
A
+
=
c
c
A
+
=
c
c
+
+
0
y
u u 2 u
x
u u 2 u
2
1 j , i j , i
1 j ,
i
2
j , 1 i j , i j , 1 i
=
A
+
+
A
+

+ +
(i,j)
(i+1,j)
(i-1,j)
(i,j+1)
(i,j-1)
If A x = A y
we can collect the terms
to get:
0 u 4 u u u u
j , i 1 j , i 1 j , i j , 1 i j , 1 i
= + + +
+ +
(i,j)
(i+1,j)
(i-1,j)
(i,j+1)
(i,j-1)
0 u 4 u u u u
j , i 1 j , i 1 j , i j , 1 i j , 1 i
= + + +
+ +
This equation is referred
to as the Laplacian difference
equation.

It can be applied to all
interior points.

We must now consider
what to do with the
boundary nodes.
Boundary Conditions
Dirichlet boundary conditions: u is specified at the
boundary
Temperature
Head
Neumann boundary condition: the derivative is specified
q
i

Combination of both u and its derivative (Mixed BC)
i i
x
h
or
x
T
c
c
c
c
The simplest case is where the boundaries are
specified as fixed values.

This case is known as the Dirichlet boundary
conditions.
u
1
u
2
u
3
u
4
u
1
u
2
u
3
u
4
Consider how we can deal with the lower node shown, u
1,1
-4u
1,1
+u
1,2
+u
2,1
+u
1
+u
4
= 0
1,2
1,1 2,1
Note:
This grid would result
in nine simultaneous
equations.
Lets consider how to model the Neumann boundary condition
x 2
u u
x
u
j , 1 i j , 1 i
A
~
c
c
+
centered finite divided difference
approximation
c
c
c
c
2
2
2
2
0
h
x
h
y
+ =
c
c
h
x
= 0
c
c
h
x
= 0
c
c
h
y
= 0
suppose we wanted to consider this end
grid point
1,2
1,1 2,1
0
x
h
=
c
c
0
y
h
=
c
c
The two boundaries are consider
to be symmetry lines due to the
fact that the BC translates
in the finite difference form
to:
h
i+1,j
= h
i-1,j

and

h
i,j+1
= h
i,j-1

1,2
1,1 2,1
h
1,1
= (2h
1,2
+ 2 h
2,1
)/4

h
1,2
= (h
1,1
+ h
1,3
+2h
22
)/4
2,2
0
x
h
=
c
c
0
y
h
=
c
c
Example
The grid on the next slide is designed to solve the LaPlace
equation
0
y x
2
2
2
2
=
c
u c
+
c
u c
Write the finite difference equations for the nodes (1,1),
(1,2), and (2,1). Note that the lower boundary is a
Dirichlet boundary condition, the left boundary is a
Neumann boundary condition, and Ax = Ay.
Solution
0
x
=
c
u c
20 = u
(1,1) (2,1)
(1,2)
( )
( )
( )
4
20
4
20 2
4
2
31 22 11
21
21 12
11
11 22 13
12
+ u + u + u
= u
+ u + u
= u
u + u + u
= u
The Liebmann Method
Most numerical solutions of the Laplace
equation involve systems that are much
larger than the general system we just
evaluated
Note that there are a maximum of five
unknown terms per line
This results in a significant number of terms
with zeros
The Liebmann Method
In addition to the fact that they are prone to
round-off errors, using elimination methods
on such sparse systems wastes a great
amount of computer memory storing zeros
Therefore, we commonly employ
approaches such as Gauss-Seidel, which
when applied to PDEs is also referred to as
Liebmanns method.
The Liebmann Method
In addition the equations will lead to a matrix that
is diagonally dominant.
Therefore the procedure will converge to a stable
solution.
Over relaxation is often employed to accelerate the
rate of convergence
( )
old
j , i
new
j , i
new
j , i
j , i 1 j , i 1 j , i j , 1 i j , 1 i
u 1 u u
0 u 4 u u u u
+ =
= + + +
+ +
( )
old
j , i
new
j , i
new
j , i
j , i 1 j , i 1 j , i j , 1 i j , 1 i
u 1 u u
0 u 4 u u u u
+ =
= + + +
+ +
As with the conventional Gauss Seidel method, the iterations
are repeated until each point falls below a pre-specified
tolerance:
% 100
u
u u
new
j , i
old
j , i
new
j , i
s

= c
Groundwater Flow Example
c
c
c
c
2
2
2
2
0
h
x
h
y
+ =
c
c
h
x
= 0
c
c
h
x
= 0
c
c
h
y
= 0
Modeling 1/2 of the system shown, we can develop the following
schematic where A x = A y = 20 m

The finite difference equations can be solved using a
a spreadsheet.
100 =A1+0.05*20 =B1+0.05*20 =C1+0.05*20 =D1+0.05*20 =E1+0.05*20 =F1+0.05*20 =G1+0.05*20 =H1+0.05*20 =I1+0.05*20 =J1+0.05*20
=(A1+2*B2+A3)/4 =(B1+C2+B3+A2)/4 =(C1+D2+C3+B2)/4 =(D1+E2+D3+C2)/4 =(E1+F2+E3+D2)/4 =(F1+G2+F3+E2)/4 =(G1+H2+G3+F2)/4 =(H1+I2+H3+G2)/4 =(I1+J2+I3+H2)/4 =(J1+K2+J3+I2)/4 =(K1+K3+2*J2)/4
=(2*A5+2*B6)/4 =(2*B5+C6+A6)/4 =(2*C5+D6+B6)/4 =(2*D5+E6+C6)/4 =(2*E5+F6+D6)/4 =(2*F5+G6+E6)/4 =(2*G5+H6+F6)/4 =(2*H5+I6+G6)/4 =(2*I5+J6+H6)/4 =(2*J5+K6+I6)/4 =(2*K5+2*J6)/4
100 =A1+0.05*20 =B1+0.05*20 =C1+0.05*20 =D1+0.05*20 =E1+0.05*20 =F1+0.05*20 =G1+0.05*20 =H1+0.05*20 =I1+0.05*20 =J1+0.05*20
100
=(A1+2*B2+A3)/4
=(A2+2*B3+A4)/4
=(A3+2*B4+A5)/4
=(A4+2*B5+A6)/4
=(2*A5+2*B6)/4
You will get an
error message in
Excel that states that
it will not resolve
a circular reference.
CAN USE EXCEL DEMONSTRATION
100 =A1+0.05*20 =B1+0.05*20 =C1+0.05*20 =D1+0.05*20 =E1+0.05*20 =F1+0.05*20 =G1+0.05*20 =H1+0.05*20 =I1+0.05*20 =J1+0.05*20
=B1+0.05*20
=(C1+D2+C3+B2)/4
=(C2+D3+C4+B3)/4
=(C3+D4+C5+B4)/4
=(C4+D5+C6+B5)/4
=(2*C5+D6+B6)/4
After selecting the
appropriate command,
EXCEL with perform
the Liebmann method
for you.

In fact, you will be able
to watch the iterations.
Table 2: Results of finite difference model.
A B C D E F G H I J K
1 100 101 102 103 104 105 106 107 108 109 110
2 101.6 102 102.6 103.4 104.2 105 105.8 106.6 107.4 108 108.4
3 102.5 102.7 103.1 103.7 104.3 105 105.7 106.3 106.9 107.3 107.5
4 103 103.1 103.4 103.9 104.4 105 105.6 106.1 106.6 106.9 107
5 103.3 103.3 103.6 104 104.5 105 105.5 106 106.4 106.7 106.7
6 103.3 103.4 103.7 104 104.5 105 105.5 106 106.3 106.6 106.7
...end of problem.
Secondary Variables
Because its distribution is described by the
Laplace equation, temperature is considered
to be the primary variable in the heated
plate problem
A secondary variable may also be of interest
In this case, the second variable is the rate
of heat flux across the place surface
i
T
C k q
i
c
c
=
|
|
.
|
\
|
= u
+ =
A
=
A
=
c
c
=
+
+
y
x
1
2
y
2
x n
1 j , i 1 j . i
y
j , 1 i j . 1 i
x
i
q
q
tan
q q q
y 2
T T
' k q
x 2
T T
' k q
i
T
C k q
FINITE DIFFERENCE
APPROXIMATION
BASED ON RESULTS
THE RESULTING
FLUX IS A VECTOR
WITH MAGNITUDE
AND DIRECTION
Finite Difference: Parabolic
Equations B
2
- 4AC = 0
0 D
y
u
C
y x
u
B
x
u
A
2
2 2
2
2
= +
c
c
+
c c
c
+
c
c
These equations are used to characterize
transient problems.

We will first study this in one spatial direction (1-D).
t
T
x
T
k
2
2
c
c
=
c
c
Consider the heat-conduction equation
As with the elliptic PDEs, parabolic equations can be solved
by substituting finite difference equations for the
partial derivatives.

However we must now consider changes in time
as well as space
t
x
y
x
l
i
u
spatial
{
temporal
{

Centered finite divided difference
Forward finite divided
difference
( )
T
T T
x
T T 2 T
k
t
T
x
T
k
l
i
1 l
i
2
l
1 i
l
i
l
1 i
2
2
A
=
A
+
c
c
=
c
c
+
+
( )
( )
( )
2
l
1 i
l
1 i
l
1 i
l
i
1 l
i
l
i
1 l
i
2
l
1 i
l
i
l
1 i
x
t k
where
T T 2 T T T
T
T T
x
T T 2 T
k
A
A
=
+ + =
A
=
A
+
+
+
+
+
We can further reduce the equation:
NOTE:

Now the temperature
at a node is estimated
as a function of
the temperature at the
node, and surrounding
nodes, but at a previous time
Example
Consider a thin insulated rod 10 cm long with
k = 0.835 cm
2
/s

Let A x = 2 cm and A t = 0.1 sec.

At t=0 the temperature of the rod is zero.
h
o
t

c
o
l
d

Now subject the two ends to temperatures of
100 and 50 degrees

Set the nodes with these boundary and initial conditions.

This is what we consider as the conditions
at t=0
100 0 0 0 0 50
i= 0 1 2 3 4 5
x= 0 2 4 6 8 10
100

0

0

0

0

50
t = 0
Consider the temperature
at this node at time t+At
( )
l l l l l
1 i i 1 i i
1
i
T T 2 T T T
+
+
+ + =
0

2

4

6

8

10
x
( ) ( ) 0 0 2 1 0 0 0 T
t t
1
+ + =
A +
0

1

2

3

4

5
i

= 0.020875
t 0 0.1 0.2
x
0 100 100 100
2 0 =B5+$B$1*(B6-2*B5+B4) =C5+$B$1*(C6-2*C5+C4)
4 0 =B6+$B$1*(B7-2*B6+B5) =C6+$B$1*(C7-2*C6+C5)
6 0 =B7+$B$1*(B8-2*B7+B6) =C7+$B$1*(C8-2*C7+C6)
8 0 =B8+$B$1*(B9-2*B8+B7) =C8+$B$1*(C9-2*C8+C7)
10 50 50 50
A B C D
1
2
3
4
5
6
7
8
9
( )
( )
2
1 i i 1 i i
1
i
x
t k
where
T T 2 T T T
A
A
=
+ + =
+
+ l l l l l
0
10
20
30
40
50
60
70
80
90
100
0 5 10
x (cm)
t

(
s
e
c
)
5 sec
10 sec
15 sec
...end of example
Convergence and Stability
Convergence means that as A x and A t
approach zero, the results of the numerical
technique approach the true solution
Stability means that the errors at any stage
of the computation are attenuated, not
amplified, as the computation progresses
The explicit method is stable and
convergent if
2
1
s
Derivative Boundary Conditions
T
o

T
L

In our previous example T
o
and T
L
were constant values.

However, we may also have derivative boundary conditions
( )
i
1
i
0
i
1
i
0
1 i
0
T T 2 T T T

+
+ + =
Thus we introduce an imaginary point at i = -1
This point provides the vehicle for providing the derivative BC
q
0
=

0

T
L

For the case of q
o
= 0, so T
-1
= T
1
.

So in this case the balance at node 0 is:
( )
l l l l
0 1 0
1
0
T 2 T 2 T T + =
+
q
0
=

1
0

T
L

For the case of q
o
= 10, we need to know k [= k/(C)].
Assuming k =1, then 10 = - (1) dT/dx,
or dT/dx = -10
( )
|
|
.
|
\
|

A
+ =
'
A
=
A
'
=
+
l l l l
l l
l l
0 1 0
1
0
1 1
1 1
T 2
1
x
20 T 2 T T
k
x
20 T T
x 2
T T
k 10
Implicit Method
Explicit methods have problems relating to
stability
Implicit methods overcome this but at the
expense of introducing a more complicated
algorithm
In this algorithm, we develop simultaneous
equations
Explicit Implicit
grid point involved with space difference
grid point involved with time difference

With the implicit method, we develop a set of simultaneous
equations at step in time
( )
2
1 l
1 i
1 l
i
1 l
1 i
x
T T 2 T
A
+
+
+ +
+
( )
2
l
1 i
l
i
l
1 i
x
T T 2 T
A
+
+
( )
( )
( )
1 l
0
1 l
0
l
i
1 l
1 i
1 l
i
1 l
1 i
l
i
1 l
i
2
1 l
1 i
1 l
i
1 l
1 i
t f T
T T T 2 1 T
t
T T
x
T T 2 T
k
+ +
+
+
+ +
+ +
+ +
+
=
= + +
A
=
A
+
which can be expressed as:
For the case where the temperature level is given at the end
by a function f
0
i.e. x = 0
( )
( )
( ) ( )
1 l
0
l
i
1 l
1 i
1 l
i
1 l
0
1 l
0
l
i
1 l
1 i
1 l
i
1 l
1 i
t f T T T 2 1
t f T
T T T 2 1 T
+ +
+
+
+ +
+
+
+ +
+ = +
=
= + +
Substituting
In the previous example problem, we get a 4 x 4 matrix
to solve for the four interior nodes for each time step

Numerical Methods Notes

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Numerical Methods Notes

Hochgeladen von

Copyright:

Verfügbare Formate

Approximation and Errors

Approximate derivative using a finite divided difference

Use of more significant figures

We will clearly run into

Graphical depiction of Gauss-Jordan

Note: the superscript

Now we need to reduce the x

Now we need to reduce the x

Now we need to reduce the x

Now we need to reduce the x

Now we need to reduce the x

1.25 - 1.25 (1) = 0

Before going any further

Before going any further

-1.5 - 1.6 (1.25)

-1.5 - 1.6 (1.25) = -3.5

Recall the rules of matrix multiplication.

Next we multiply the first row of [L] by the columns

Once the first row of [U] is established

The Substitution Step

Recall the original column matrix

Note: the superscript

Convergence criteria for two linear

This condition is sufficient but not necessary; for convergence.

The system is then solved by matrix inversion

The most common measure of the spread of a sample is the

Numerical Differentiation and

Multiple Application of the

We can group terms to express a general form

Integration of Unequal Segments

We can continue to improve the estimate by successive

Solve these equations simultaneously

However, we have set the limit of integration at -1 and 1.

Second Order RK Methods

Second Order RK Methods

Das könnte Ihnen auch gefallen