Scientic Computation (COMS 3210)

Bigass Study Guide

Spring 2012
1 Overview
Depending on how tired I am when I nish this, I may compose a owery and exciting introduction to the
eld of Scientic Computation. At the moment, this seems unlikely. That being said, this is an important
course whose motivation might be summed up as follows:
1. Science is frought with continuous problems, solving these problems on a discrete device like most mod-
ern computers usually means developing a numerical method or other means of approximation.
2. When working with scientic models and approximation, one must be aware of the computational
hazards that come with the territory. These include many forms of error, algorithmic instability,
and ill-conditioned problems.
Some examples of continuous problems that permeate many elds of science:
1. Integration, sometimes through many dimensions
2. Systems of linear / nonlinear equations
3. Dierential equations
2 Some grounding theorems from calculus
Proving the convergence on many of the numerical methods we study in this course requires knowlege of
some key theorems from calculus. A description of some of these theorems follows
1
The Mean-Value Theorem
For some function f, if f is continuous and dierentiable on the interval [a, b], then there exists some value
c [a, b] such that
f

(c) =
f(b) f(a)
b a
In plain English: theres a location somewhere between a and b where the slope of the function f is equal to
the slope of the line segment that connects a and b.
The Intermediate-Value Theorem
For some function f, if f is continuous on the interval [a, b], then for any value u such that f(a) u f(b)
there exists some value c [a, b] such that f(c) = u
The Wierstrass Corollary:
If f(a)f(b) < 0 (there is a sign change), then application of IVT guarantees a root on the interval
The Extreme-Value Theorem
Roughly: a continuous, bounded function achieves its minimum and maximum values on an interval at
either:
1. Locations where the derivative of the function is zero
2. The endpoints of the interval
This theorem is most useful when computing error bounds, as it can be used to maximize the value of a
function on an interval.
2
3 Taylors Theorem - The most important theorem in this course
For any function f that is continuous and n+1 times dierentiable, this function may be expressed exactly
as:
f(x) = P
n
(x) + R
n
(x)
Here, P
n
is an n
th
degree polynomial function of the form:
P
n
(x) =
n

i=0
f
(i)
(x
0
)
i!
(x x
0
)
i
where x
0
is some value that will act as the center of our constructed polynomial. We say P
n
is constructed
around x
0
. Meanwhile, R
n
is a remainder term of the form:
R
n
(x) =
f
(n+1)
()
(n + 1)!
(x x
0
)
n+1
Notice that R
n
looks very much like the next-order term of P
n
, but the function evaluation on f is performed
at some mysterious value. This value is mysterious indeed. We cannot know its identity exactly. All we
can say for certain is that [x
0
, x]
1
.
Lets take a moment to appreciate the gravity of the claim weve just made. Taylors theorem asserts that
any function can be expressed exactly as the sum of a polynomial and a remainder term. This is fantastic
because polynomials are simple, tractable, and easy to evaluate. Whenever we linearize a function, or perhaps
attempt to express it as a parabola, we are invoking Taylors theorem. Thus, the remainder term is crucial
to understanding the implications of our approximations.
Its not surprising that we cannot express this error term exactly, that we must rely on some unknown . If
we did know precisely what this term was, then it would not be error! We would add it to our polynomial
and have a foolproof way of turning any function into a polynomial. Clearly, this is not so. Thus, if we want
to develop an understanding of the error we incur with our approximation, the best we can hope to do is
bound the remainder term. Such an operation is known as computing an error bound. To bound
the error on an approximation of f(x) on a Taylor polynomial constructed around x
0
we may nd
max
x

[x0,x]
f
(n+1)
(x

)
And applying this function value to the full remainder term.
1
NB: Dont be fooled by the order of the values in the interval, it is not necessarily the case that x
0
< x
3
A quick note on error bounds:
Say we determine that some approximation g(x) on the function value f(x) has an error bound EB. This
guarantees that:
|f(x) g(x)| EB
Note that while |f(x) g(x)| = EB is possible, it is unusual.
4 Root Finding
Given a function f, nd the x for which f(x) = 0.
Bisection
The bisection method can be used to nd the root of a function on some interval (a, b) when a and b have
opposite signs, ab < 0. The function f must also be continuous for bisection to work. Note that these
requirements also guarantee that a root exists on the interval (a, b) thanks to the Intermediate-Value
Theorem.
The bisection method is performed as follows:
Initialize values left = a and right = b
Compute mid =
left+right
2
Evaluate val = f(mid)
If |val| return mid as the root
Else If val > 0 start over, setting a = mid if a > 0 or b = mid if b > 0
Else If val < 0 start over, setting a = mid if a < 0 or b = mid if b < 0
The bisection method is similar to a continuous binary search. Because it is continuous, we establish a
tolerance, known as , which keeps us from having to iterate indenitely towards an exact zero.
4
Convergence
Bisection is guaranteed to converge. It will always nd a root on its interval, provided a legal interval
was used.
The rate of convergence follows
2
n
Meaning that in order to approximate the root with accuracy
n = lg(
|b a|
2
)
Function evaluations (iterations on the algorithm) are required.
Newtons Method
Bisection is nice because it is guaranteed to converge. It can also be parallelized quite eectively, but thats
a story for another day. However, its convergence is comparatively very slow. Much faster is Newtons
Method. The idea behind Newtons method is simple. For some function f
1. Pick a starting point x
0
.
2. Linearize the function around x
0
3. Eortlessly nd the zero of this linearized function, call it x
1
4. Evaluate f(x
1
) and start again from this point.
5. Pray for convergence
Expressed as a recurrence, this is:
x
i+1
= x
i

f(x
i
)
f

(x
i
)
5
Because this method involves linearization, we expect to be able to understand it more fully through Taylors
theorem. In its linear form, Taylors theorem states that
f(x
0
+ ) = f(x
0
) + f

(x
0
) + R
1
(x
0
+ )
We want to nd some to add to x
0
so that f(x
0
+ ) = 0 so
0 = f(x
0
) + f

(x
0
) + R
1
(x
0
+ )
f

(x
0
) = f(x
0
) + R
1
(x
0
+ )
=
f(x
0
)
f

(x
0
)

R
1
(x
0
+ )
f

(x
0
)
Giving us our recurrence. The remainder term in the above equation tells us exactly how far x
i
+ will be
from the real root. Under the denition of R
1
(x) this is
f

()
2f

(x
0
)

2
Which yeilds Newtons iterative error equation:
e
n+1

f

(x
n
)
2f

(x
n
)
e
2
n
Not only is this error recurrence an approximation, it is also not guaranteed to converge! This is why the
last step in the description of the algorithm above was pray for convergence. When Newtons method does
converge, it does so quadratically, but Newton is not guranateed to converge. Convergence depends on
the initial point x
0
, as well as the properties of the function itself. In practice, we overcome this by bounding
the number of iterations we are willing to perform before we try from a new initial point. As a guideline
for how many iterations we may expect to have to perform, we may solve the error recurrence to obtain the
following:
n = lg
_
lg
_
1

__
Where n is the expected number of function evaluations necessary. But there is another problem! Because
we do not know our actual root, we have no way of knowing during any given iteration. How can we know
how close we are to the root if we dont know the root in the rst place? This means we must evaluate our
progress using a heuristic. Commonly used stopping points are
x
n+1
x
n

|f(x
n+1
)|
Newton may not be quite so straightforward as Bisection, but its quirks are well-tolerated. It requires very
little space to run, and is an algorithm that is generalizable to many other problems, like solving systems of
equations.
6
Secant Method
Secant method is very similar to Newtons method, so the reader excuse my brevity when describing it. The
primary dierence between Secant and Newton is that Secant method makes use of the discrete derivative,
when Newton uses the continuous derivative. Thus, Secant methods recurrence is:
x
i+1
= x
i

f(x
i
)
f[x
i
, x
i1
]
where
f[a, b] =
f(b) f(a)
b a
The error iteration then becomes
e
n+1

f

(x
n
)
2f[x
n
, x
n1
]
e
n
e
n1
Say e
n
and e
n1
may both be expressed as exponential values of the same base (for instance, 2
2
and 2
1
).
Since these exponents will add each time we take the product of these two values, we see that
log(e
n
)
log(e
n1
)
Simulates the Fibonacci sequence. For the Fibonacci sequence, we know that the ratio between consecutive
numbers approaches a special value, =
1+

5
2
as the sequence approaches innity. Thus
lim
n
log(e
n
)
log(e
n1
)
=
So where Newtons convergence rate was order 2 (quadratic). Secants is order . Remember, though,
convergence is not guaranteed for either method.
2
So why would anyone ever use Secant over Newton? It doesnt x the convergence issue. It takes longer
when it does converge, and it requires slightly more memory to implement (the previous TWO x values must
be stored). The answer is that it oers the invaluable benet of not requiring a derivative. When operating
on a function whose derivative is unknown (say, for data plotted from experimental measurement), secant
oers a viable alternative to bisection.
2
It is possible to guarantee convergence with Secant method by using a technique called root bracketing. This technique
was not discussed in class, but basically consists of ensuring that the two points that form the secant on the function have
function values with opposite signs
7
A note on some applications of root nding
By cleverly disguising one problem as another, we may use root nding techniques to perform lots of other
tasks. For instance, say we wanted to compute

2. We may do so by nding the positive root of the function.

x
2
2 = 0
Another interesting application for root nding is an implementation of the division operator. To compute
1
A
we may apply Newton to the equation
1
x
A = 0. The resulting recurrence
x
i+1
= x
i
(2 Ax
i
)
may converge to the desired reciprocal, and requires no divisions.
5 Interpolation
Say we have some set of points, {(x
1
, y
1
), (x
2
, y
2
), ..., (x
n
, y
n
)}. We wish to develop a function f that passes
through each of these points (y
i
= f(x
i
)). In doing so, we compute all the points that lie between two
members of our set of points on a curve. This process is known as interpolation.
Lagrange Polynomial
Lagrange interpolation tries to construct an n-degree polynomial in order to interpolate the n points in
our interpolation set {(x
1
, f(x
1
)), (x
2
, f(x
2
)), ..., (x
n
, f(x
n
))}. We may think of such a polynomial in a very
intuitive manner:
Consider a polynomial function l
k
. Dene l
k
such that
l
k
(x
j
) =
_
1, if j = k
0, if j = k
In general, a function that behaves this way is known as the Kroneker Delta, but we shall call l
k
a
Lagrange basis polynomial. Now, we may think of each l
k
value as a binary switch that turns on
at its respective x
k
and turns o all other x
j
. Note that the behavior of l
k
for values of x outside the
interpolation set is not so neatly dened. If we then multiply each of these values l
k
by their respective
f(x
k
) then we have the interpolating polynomial we set out to build.
Consider some points x
1
, x
2
in the interpolation set. When we evaluate h(x) = f(x
1
)l
1
(x) + f(x
2
)l
2
(x) at
h(x
1
), then l
1
(x
1
) turns on and becomes 1 and is multiplied by f(x
1
) to give the desired product f(x
1
).
Meanwhile, l
2
(x
1
) is turned o, becoming a zero and nullifying the eect of the factor f(x
2
). The reverse,
of course, is true for h(x
2
). We have interpolated these two points!
Generally, we may express the Lagrange interpolating polynomial as
8
L(x) =
n

i=1
f(x
i
)l
i
(x)
But wait! We have utilized some mysterious polynomial function l
k
without dening it! Fortunately, we may
dene such a polynomial very easily. Keeping in mind that l
k
(x
j
) = 0 when k = j we may start by writing
l

k
(x) =
n

j=1;i=k
(x x
j
)
Ensuring a value of zero whenever we have some x
i
in the interpolating set where j = k. Happily, we have
done this with an n 1 degree polynomial. Since our goal is to sum these basis polynomials to build an n
degree polynomial, it would seem that we are in good shape. Our last step is to ensure our other restriction
on l
k
, which is that l
k
(x
k
) = 1. To do this, we simply normalize our polynomial
l
k
(x) =

n
j=1;i=k
(x x
j
)

n
j=1;i=k
(x
k
x
j
)
As before l
k
is an n 1 degree polynomial. We have merely divided it by a constant to ensure unity at a
special point.
An alternative way to express l
k
is:
l
k
(x) =
g(x)
(x x
k
)g

(x
k
)
where
g(x) =
n

i=0
(x x
i
)
Under this denition,
l
k
(x
j
) =
g(x
j
)
(x
j
x
k
)g

(x
k
)
producing a zero in the numerator and some constant in the denominator, yielding zero overall. Meanwhile,
lim
xx
k
l
k
(x) =
g(x)
(x x
k
)
1
g

(x
k
)
And taking the limit, keeping in mind the denition of the derivative we have
l
k
(x
k
) = g

(x
k
)
1
g

(x
k
)
= 1
As desired.
However we choose to express l
i
, our full interpolating polynomial is
L(x) =
n

i=1
f(x
i
)l
i
(x)
which is an n-degree polynomial. Using such a function to interpolate a set of points is not without its
drawbacks. For one, high-degree interpolating polynomials suer from Runges phenomenon, or heavy
oscillation at the edges of the interval.
9
In general, a high-degree interpolating polynomial may be oscillatory (they must change direction at least n
times!), which makes them poor candidates if the goal is to achieve a smooth interpolation of a data set.
Spline
Spline interpolation is a type of interpolation that uses piecewise polynomial functions to overcome the
negative aspects of Lagrange interpolation. It is much better at tting large datasets, and typically ts
data in a smoother and more intuitive fashion. Cubic splines are often used in practice, meaning that each
polynomial embedded in the spline is of degree 3.
6 Solving Inhomogeneous Linear Recurrences with Constant Co-
ecients
A recurrence relation is a discrete dynamical system in which each subsequent value of the system may be
described as a function of the previous values of the system. This course oers a brief overview of the basic
strategies involved with computing analytical solutions to recurrences, but the techniques discussed are far
from comprehensive.
The following is an example of an Inhomogeneous Linear Recurrence with Constant Coecients:
x
i+2
3x
i+1
+ 4x
i
= 6
The recurrence is
Inhomogeneous: Because the RHS = 0
Linear: Because each recurrence term is linear in nature (as opposed to having 4x
2
i
, for instance)
2nd Order: Because each the dierence between the rst and last sequence terms in the equation is
(i + 2) i = 2
In order to nd a general solution to an inhomogeneous recurrence, we apply the following rule
Gen
Inhomogeneous
= Gen
Homogeneous
+ Part
Inhomogeneous
That is, the general solution to the inhomogenous equation can be described as a general solution to the
homogeneous part, plus a particular solution to the inhomogenous equation.
10
Finding the General Solution to the Homogenous Equation
Our rst step in solving this recurrence is solving its homogeneous counterpart. First, to construct the
counterpart, we must only make the recurrence relation evaluate to zero on the RHS. In this case, we do
this by dropping the 2.
x
i+2
3x
i+1
4x
i
= 0
Now we must try to nd some expression that will allow us to give an arbitrary x
i
in this system in terms
of i. Since we recognize that the systems growth is linearly proportional to its value, we try an exponential
function.
x
i
=
i
This gives

i+2
3
i+1
4
i
= 0

2
3 4 = 0
The above quadratic equation is known as the indicial equation for this system. We may nd a general
solution using the roots of the indicial equation

2
3 4 = 0
( 4)( + 1) = 0
= 4or 1
Since our assumption was that
x
i
=
i
We may express our general solution to the homogeneous part as a linear combination of these roots,
substituted into this equation
x
i
= C
1
(4)
i
+ C
2
(1)
i
11
Finding a Particular Solution to the Inhomogenous Equation
We are now tasked with nding some particular solution to the inhomogeneous equation. Our trick here is
to consider the possibility of a constant recurrence that obeys the inhomogenous equation, meaning that
x
i
=
Substituting this into the original recurrence, we nd
3 4 = 6
6 = 6
= 1
So one particular solution to the inhomogeneous part is
x
i
= 1
Finding the General Solution to the Inhomogeneous Equation
Using our guiding principle of
Gen
Inhomogeneous
= Gen
Homogeneous
+ Part
Inhomogeneous
we easily obtain
x
i
= C
1
(4)
i
+ C
2
(1)
i
+ 1
Turning the General Solution into a Unique Solution
What we see in the general solution to the inhomogeneous recurrence is a whole family of solutions. Each
one of these solutions ts the recurrence given. Well need to know the initial conditions of the system if
were to pick out just one. We must dene two unknown constants, so we need at least two initial conditions.
Say were given
x
0
= 3
x
1
= 4
We set up the system of equations
x
0
= C
1
(4)
0
+ C
2
(1)
0
+ 1
x
1
= C
1
(4)
1
+ C
2
(1)
1
+ 1
And nd that C
1
= C
2
= 1. Our unique solution is then
x
i
= (4)
i
+ (1)
i
+ 1
12
Verifying the Results
To ensure that all our assumptions did not lead us astray, we must diligently verify our results. As with
an induction proof, we must verify the base case, and the inductive case. For recurrences, the base case
corresponds to the initial conditions, and the inductive case corresponds to the system itself.
Verifying the Initial Conditions:
x
i
= (4)
i
+ (1)
i
+ 1
x
0
= (4)
0
+ (1)
0
+ 1 = 3
x
1
= (4)
1
+ (1)
1
+ 1 = 4
Verifying the Recurrence:
x
i
= (4)
i
+ (1)
i
+ 1
x
i+2
3x
i+1
4x
i
= 6
(4)
i+2
+ (1)
i+2
+ 1 +
3(4)
i+1
3(1)
i+1
3 +
4(4)
i
4(1)
i
4 = 6
16(4)
i
+ (1)
i
+ 1 +
12(4)
i
+ 3(1)
i
3 +
4(4)
i
4(1)
i
+ 4 = 6
6 = 6
Congratulations! Youve made it through the first semester. Now have a cup of coffee and
catch your breath.
3
7 The Logistic Map: An Example of a Nonlinear Recurrence
The Logistic Map is a quadratic recurrence of the form:
x
i+1
= k(x
i
)(1 x
i
)
Unlike our linear recurrences from before, the Logistic Map cannot be so easily solved. In fact, the recurrence
will behave very dierently depending on its k parameter.
3
Though this advice was directed toward the author and not his audience, the reader is welcome to indulge him or herself
as well
13
When we describe the long-term behavior of a dynamical system, we may dene it in terms of an attractor.
An attractor is dened as the set of points towards which the system tends over time. If an attractor is
a singleton set, it is known as a point attractor. If an attractor contains two values, it is known as a
two-point attractor.
k parameter behavior
k < 1 point attractor at 0
1 k < 3 point attractor at
x1
x
3 k 3.45 2 point attractor at a value dependent on k
3.45 k 3.54 bifurcation (cardinality of attractor rapidly doubles)
3.54 k strange attractor
A strange attractor is an attractor with non-integer, or fractal, dimension. Trajectories within a strange
attractor often appear to skip around randomly, instead of proceeding in an organized fashion. As such,
the evolution of a logistic map possessing such an attractor is highly sensitive to small changes in intial
conditions.
8 Chaos
Foundations
Originally, it was believed that understanding the evolution of a system, and its initial conditions meant
being able to determine the state of the system indenitely far in the future. As weve just seen, this is not
the case. Some systems are so sensitive to tiny changes in initial conditions, that their evolution may be
fundamentally altered by the slightest perturbation. These dynamical systems are said to be chaotic
Discovery
Chaos was discovered by Edward Lorenz in 1961. Lorenz was running simulations on the hydrodynamic
equations a set of partial dierential equations instrumental to weather prediction. What follows is an
excerpt from the Wikipedia article on chaos, and describes Lorenz discovery in detail.
Lorenz was using a simple digital computer, a Royal McBee LGP-30, to run his weather simu-
lation. He wanted to see a sequence of data again and to save time he started the simulation in
the middle of its course. He was able to do this by entering a printout of the data corresponding
to conditions in the middle of his simulation which he had calculated last time.
To his surprise the weather that the machine began to predict was completely dierent from the
weather calculated before. Lorenz tracked this down to the computer printout. The computer
worked with 6-digit precision, but the printout rounded variables o to a 3-digit number, so a
value like 0.506127 was printed as 0.506. This dierence is tiny and the consensus at the time
would have been that it should have had practically no eect. However Lorenz had discovered
that small changes in initial conditions produced large changes in the long-term outcome
14
If youve ever wondered why we can put a man on the moon, but are somehow unable to accurately predict
the weather more than a week or two in the future, this is why. Note that weather prediction is dierent
than climate studies. Weather prediction involves evaluating the exact state of a dynamical system at a
future time. Climate is a study of averages (which have the law of large numbers to shield them from chaos).
Studying climate change means understanding how modifying the parameters of a dynamical system (e.g:
the mass of greenhouse gas in the atmosphere) will aect the overall evolution of that system.
Implications
On the surface, perhaps chaos doesnt seem like such a big roadblock. Sure, there may be some dynamical
systems that are highly sensitive to changes in initial conditions, but why cant we just be extra sure that
were using the right initial conditions?
The answer, of course, is that there may not be any right initial conditions insofar as computers are
concerned. For one thing, taking innitely-precise, innitely-accurate measurements of any set of initial
conditions is impossible. However, even if we were to accomplish such a Herculean task, wed still need to
enter that information into a computer to actually operate on it. Computers are nite-precision devices, being
unable to operate on, or even store innitely-precise data. We may also incur round-o or approximation
error as we proceed in our calculation, causing the death-blow perturbations we set out to avoid. We should
know from this class that when dealing with continuous models, error happens. Thus, the behavior of a
chaotic system is impossible to predict long-term.
9 Problem Conditioning
A problem is said to be ill-conditioned when a small change in input to the problem results in a dispropor-
tionately large change in output. This is similar to, but distinct from chaos, which is a term used exclusively
for dynamical systems, wherein small changes in initial conditions produce entirely dierent evolutions on
the system.
Poor problem conditioning represents a ckle issue in scientic computing. Because conditioning is a property
of the problem, and not the algorithm used to address the problem, there is really no way around an ill-
conditioned problem. Small perturbations are going to blow up, no matter what solution strategy is used.
The Wilkinson Polynomial
Finding the roots of Wilkinsons Polynomial is a classic example of poor problem conditioning. The poly-
nomial
f(x) =
20

i=1
(x i)
has a set of clearly-dened roots at x = 1, 2, ..., 20. However, modify the coecient on the x
19
in the expanded
polynomial by just 2
23
and this set of friendly looking integer roots becomes a wholly dierent set of ten
real and ten complex roots. One of these roots, x 20.846 is the perturbed counterpart of x = 20. When
comparing the input perturbation to the resulting change in the root, we nd a ratio of
.846
2

23
7, 000, 000
As we know, error is unavoidable. The perturbation of 2
23
was literally outside the precision that Wilkin-
sons computer could even express. Ill-conditioning is important in scientic computation because when a
problem is not tolerant of any error whatsoever, it destroys our ability to compute a reliable answer
15
10 Integration
The need to evaluate integrals is a problem that can be found throughout almost every scientic eld. For
those integrals that cannot be tackled analytically, numerical integration oers a solution.
The rst few methods of numerical integration we discuss in this course are also known as quadrature rules,
sometimes called the Newton-Cotes formulas. The basis of each of these rules is to replace a curve that
is arbitrarily dicult to integrate with a polynomial approximation on that curve, something signicantly
easier to integrate.
Of course, we will use our good friend Taylors theorem as our way of replacing an arbitrary function with
a polynomial.
Trapezoidal Rule
Trapezoidal rule, named for the shape of the resulting approximation, works by linearizing our function
f before integrating it. As such, trapezoidal rule computes the integral of the degree 1 taylor polynomial
approximation of the function.
_
b
a
f(x)dx = (b a)
f(a) + f(b)
2

f

()
12
(b a)
3
Note that in order to achieve a bound on the error on the approximation given by the trapezoidal rule,
the function must be continuous, and twice dierentiable on the interval (the requirements for having a
well-formed remainder term on a degree 1 Taylor polynomial)
The error on the approximation given by Trapezoidal rule scales with the size of the integral cubed.
4
4
Just because there is a minus sign on the remainder term does not mean that Trapezoidal rule always overapproximates.
Remember that f

() may itself be positive or negative. The minus sign DOES indicate that a positive second derivative on
the function necessitates an overapproximation, however. This is something that can be understood graphically. The integral
of a concave-up curve will always be overapproximated by a trapezoid.
16
Simpsons Rule
Though it may not look it, Simpsons rule is a simple step up from trapezoidal rule. It works by approximating
the function as a degree 2 polynomial, a parabola.
5
_
b
a
f(x)dx =
(b a)
6
(f(a) + 4f(
a + b
2
) + f(b))
f
(4)
()
2880
(b a)
5
As before, in order to achieve a bound on the error on the approximation given by Simpsons rule, the
function must be continuous, and four times dierentiable on the interval. (We see a term that looks like R
3
here and not R
2
because of a cancellation that occurs when analyzing the Taylor formula for the interpolated
curve)
The error on the approximation given by Simpsons rule scales with the size of the integral to the fth.
Compound Quadrature
A major problem with our simple formula above is that their approximation error scales as a polynomial
term with the size of the interval of integration. We want to be able to evaluate arbitrarily large integrals
with good accuracy, but in the approximations above the error will only blow up with an increased interval
of integration.
The solution, as usual, is to invoke more function evaluations. We may break our interval up into several
smaller intervals, each of length h, executing the quadrature on each of these intervals and summing the
components to compute the grand total. Note that if an interval of length (a b) is broken into intervals of
length h, then n =
(ab)
h
function evaluations will be required to compute the value at each interval endpoint.
As long as we keep h << 1, the error on our approximations can be very good.
5
The parabola formed is actually the Lagrange interpolation of the left, right, and midpoint of the interval of integration
17
The process of splitting up an interval of integration into many parts so a quadrature rule may be applied to
each is known as compound quadrature. The resulting rules are called Composite Trapezoid Rule:
_
b
a
f(x)dx =
h
2
(f(a) + f(b) + 2
n1

j=1
f(x
j
))
f

()
12
(b a)h
2
and Composite Simpsons Rule:
_
b
a
f(x)dx =
h
4
(f(a) + 2
n/21

j=1
f(x
2j
) + 4
n/2

j=1
f(x
2j1
))
f
(4)
()
180
(b a)h
4
In each case, both the approximation and remainder terms can be derived from the original quadrature
formula by performing a summation on intervals of length h. The approximation terms you see are factored
versions of these summations. For instance, the trapezoidal rule takes advantage of the fact that to compute
the area of all trapezoids of length h, each inner function evaluation is a summand twice (once with its
neighbor to the left, once with its neighbor to the right), and each function evaluation that borders the
interval is a summand once. All these summands are part of an average of two heights to compute the area
of a trapezoid of length h, hence the factored
h
2
.
The composite trapezoid remaidner term, meanwhile, can be found with a simple summation of the remainder
from performing several singular trapezoid approximations using intervals of length h:
n

i=1
f

()
12
(h)
3
Since f

() is always going to be bounded above by its maximum on (a, b), we may keep it constant. h is
also held constant on the interval. The expression can thus be simplied
h
2
f

()
12
n

i=1
(h)
Giving the expected
18
h
2
f

()
12
(b a)
We have, therefore, accomplished our goal. Our error term still scales linearly with the size of the interval
for both rules, but for h << 1, the quadratic and quartic scaling is more than enough to shrink the linear
interval length term down to nothing.
An identical analysis may be applied to derive the remainder term in Simpsons rule, but one must take
care to remember that Simpsons intervals are non-overlapping. Furthermore, because Simpsons parabolae
require three function evaluations and not two, and since we promised ourselves a function evaluation after
every interval of length h, the full length of each parabola is 2h, meaning there are half as many parabolae
as there were trapezoids.
A common question is this: given these error terms, how many function evaluations are necessary to achieve
a given error bound? Once again, we look to composite trapezoidal rule as an example for deriving an
expression for how function evaluations scale against error bounds. Thanks to our remainder term, we may
safely observe an error bound of
Error max
x

[a,b]
f

(x

)
h
2
(b a)
12
Since, as we said h =
(ba)
n
:
Error max
x

[a,b]
f

(x

)
(b a)
3
12n
2
Abstracting out max
x

[a,b]
f

(x

)
(ba)
3
12
as some parameter of the function and the interval K.
Error
K
n
2
So if is our error bound then we may perform
n =
_
K

many function evaluations to achieve it. A similar technique may be applied to the composite Simpsons
error term to nd its scaling rule of
n =
4
_
K

19
11 The Curse of Dimensionality
We have seen that by performing more function evaluations, we may make our approximation of an integral
more accurate. We may be satised with a scaling law like O(
4
_
1

), thinking that all of our integration woes

may be solved with a suciently patient compound Simpsons rule. In doing so, however, we overlook the
extraordinary common issue of integrating through multiple dimensions.
With each dimension we add to our integral, the number of function evaluations required to achieve a given
accuracy grows exponentially. This is intuitive. If we require O(
4
_
1

) equally spaced function evaluations

along one dimension to achieve a given error, then surely we must have a grid of O(
4
_
1

)
2
function evaluations
in two dimensions to achieve that same error. In general, integration through d dimensions using compound
Simpsons rule will cost us O(
4
_
1

d
) function evaluations. This is the so-called curse of dimensionality.
And it destroys the plausisbility of using compound quadrature rules for the high-dimensional integrals that
appear in many scientic problems.
Vanquishing the Curse with the Monte-Carlo Method
The root of the curse of dimensionality was a methodological determinism. By performing function evalu-
ations a grid of evenly spaced points, we guaranateed that their number would increase exponentially with
the dimension of the grid. It makes sense, then, that the curse might be vanquished by something other
than determinism. Something altogether non-deterministic.
Using the Monte-Carlo method of integration, an n-dimensional integral on f is evaluated as follows
1. Randomly select a list of n-dimensional vectors at which f will be evaluated. Each component of each of
these vectors should correspond to a dimension of integration, and its value should be somewhere on the
interval put forth by the limits of integration given. For instance, when evaluating
_
3
1
_
0.5
0
f(x, y)dxdy,
one might randomly select the vectors 1.512, .0231, 2.251, .4112
2. Evaluate the function at each one of these vectors, keeping track of the average value of the function
evaluation as you go
3. Multiply the average function value by the volume of integration (from our example, this is 3(.5) = 1.5).
The result is your approximation on the integral
More formally, the method may be expressed as:
I = V
1
N
N

i=1
f(x)
Where V is the volume of integration and each x is some vector randomly chosen from within that volume.
The miracle of the Monte-Carlo method is that it allows us to escape the Curse of Dimensionality through
randomness. By weakening our error guarantee to an expected error value, we remove dimensionality
from our scaling law. The expected error on Monte-Carlo scales like
=
V ar(f)

N
Where V ar(f) is the variance of f, a constant property of the function over the volume of integration. Note
that Monte-Carlo cannot possibly guarantee any error bound. It is based entirely upon a random
process, which, though unlikely, may utterly fail to approximate the integral appropriately.
20
12 Solving Linear Inhomogeneous Ordinary Dierential Equations
with Constant Coecients
An ordinary dierential equation is an for which there is only one independent variable and one or more
derivatives of a dependent variable with respect to the independent variable. A solution to a dierential
equation is a function that describes the dependent variable as a function of the independent variable. This
course oers a brief overview of the basic strategies involved with computing analytical solutions to ODEs,
but the techniques discussed are far from comprehensive.
The following is an example of an Inhomogeneous Linear ODE with Constant Coecients:
y

3y

4y = 4t + 1
ODEs are analagous in many ways to recurrences, the ODE above is
Inhomogeneous: Because the RHS = 0
Linear: Because each dependent variable term is linear in nature (as opposed to having 3y
2
, for instance)
2nd Order: Because each the highest derivative is order 2 (y

)
As with recurrences, in order to nd a general solution to an inhomogeneous ODE, we apply the following
rule
Gen
Inhomogeneous
= Gen
Homogeneous
+ Part
Inhomogeneous
That is, the general solution to the inhomogenous equation can be described as a general solution to the
homogeneous part, plus a particular solution to the inhomogenous equation.
Finding the General Solution to the Homogenous Equation
Our rst step in solving this ODE is solving its homogeneous counterpart. First, to construct the counterpart,
we must only make the recurrence relation evaluate to zero on the RHS. In this case, we do this by dropping
the 2.
y

3y

4y = 0
Now we must try to nd some expression that will allow us to give an arbitrary y in this system in terms
of t. Since we recognize that the systems growth is linearly proportional to its value, we try an exponential
function.
y = e
t
This gives
21

2
e
t
3e
t
4e
t
= 0

2
3 4 = 0
The above quadratic equation is known as the charactaristic equation for this system. We may nd a
general solution using the roots of the charactaristic equation

2
3 4 = 0
( 4)( + 1) = 0
= 4or 1
Since our assumption was that
y = e
t
We may express our general solution to the homogeneous part as a linear combination of these roots,
substituted into this equation
y = C
1
e
4t
+ C
2
e
t
Finding a Particular Solution to the Inhomogenous Equation
We are now tasked with nding some particular solution to the inhomogeneous equation. Our trick here is
to consider the possibility of a function of same degree as the rhs of the inhomogeneous equation that obeys
the inhomogenous equation, meaning that. Since the rhs of our inhomogenous equation was 4t +1 we will
try
y = at + b
Substituting this into the original recurrence, we nd
3a 4(at + b) = 4t + 1
3a 4t(a + 1) + 4b 1 = 0
t(4a 4) + 4b 3a 1 = 0
So for all t, we can say that
4a 4 = 0 = 4b + 3a + 1
Solving for a and b we nd
a = 1
b = 1
So one particular solution to the inhomogeneous part is
y = t + 1
22
Finding the General Solution to the Inhomogeneous Equation
Using our guiding principle of
Gen
Inhomogeneous
= Gen
Homogeneous
+ Part
Inhomogeneous
we easily obtain
y = C
1
e
4t
+ C
2
e
t
+ t + 1
Turning the General Solution into a Unique Solution
What we see in the general solution to the inhomogeneous ODE is a whole family of solutions. Each one of
these solutions ts the ODE given. Well need to know the initial conditions of the system if were to pick
the right one. We must dene two unknown constants, so we need at least two initial conditions. Say were
given
y(0) = 3
y

(0) = 4
We set up the system of equations
y(0) = C
1
e
4(0)
+ C
2
e
0
+ 0 + 1 = 3
y

(0) = 4C
1
e
4(0)
C
2
e
0
+ 0 + 1 = 4
y(0) = C
1
+ C
2
+ 1 = 3
y

(0) = 4C
1
C
2
+ 1 = 4
And nd that C
1
= C
2
= 1. Our unique solution is then
y(t) = e
4t
+ e
t
+ t + 1
23
Verifying the Results
To ensure that all our assumptions did not lead us astray, we must diligently verify our results. As with an
induction proof, we must verify the base case, and the inductive case. For ODEs, the base case corresponds
to the initial conditions, and the inductive case corresponds to the system itself.
Verifying the Initial Conditions:
y = e
4t
+ e
t
+ t + 1
y(0) = 1 + 1 + 1 = 3
y

(0) = 4 + 1 + 1 = 4
Verifying the ODE:
y = e
4t
+ e
t
+ t + 1
y

3y

4y = 4t + 1
4
2
e
4t
+ (1)
2
e
t
+
3(4)e
4t
3e
t
3 +
4e
4t
4e
t
4t 4 = 4t + 1
16e
4t
+ e
t
+
12e
4t
+ 3e
t
3 +
4e
4t
4e
t
4t + 4 = 4t + 1
4t + 1 = 4t + 1
13 Numerical Solutions to Dierential Equations (Initial Value
Problems)
An initial value problem is an Ordinary Dierential Equation combined with a specied value from which
the general solution to the ODE may be made unique. As with recurrences, we are given information on
how the system starts out, and how it evolves, and we are asked to describe it at an arbitrary point in time.
Why are Numerical Solutions Needed?
Consider the following dierential equation, which is used to express the amount of a chemical reagent
present in a reaction system over time:
dx
dt
= k(n
1

x
2
)
2
(n
2

x
2
)
2
(n
3

3x
4
)
3
It is considerably more dicult to nd an analytic solution to this ODE.
In fact, only the simplest of dierential equations allow for analytical solutions.
As weve seen many times in this course, when analytic solutions escape us, approximation provides an
answer.
24
A Rudimentary Method for Numerically Solving Initial Value Problems
Consider the general initial value problem:
y

(x) = f(x, y(x))

y(k) =
Analyticaly, solving the initial value problem means building a function y that can be used to express y(x)
at any point in the systems evolution. This is not always possible.
Lets say we decide to settle for approximating some y(k +). This way, we can approximate the value of y
at any real number by using an appropriate .
We can use Taylors theorem!
y(k + ) = y(k) + y

(k) +
y

()
2
2
Notice that the errror term is O(
2
). If we want to examine the solution system at points far away from the
initial value k (i.e. >> 1), then well need another trick.
Based on what youve seen in the course so far, what do you think this trick might be?
Eulers Method
The problem with O(
2
) error is that it blows up if (i.e. > 1). However, if (i.e. < 1), were in good
shape. How do we overcome this and still be able to access arbitrary deltas?
Answer: divide into smaller intervals, h, being sure to satisfy h << 1.
Just like when we compounded our quadrature techniques, we are able to improve on the accuracy of our
approximation by invoking more computation.
Taylor using our new strategy - Eulers Method:
Dene w
i
to be y(k + ih)
w
0
= y(k)
w
i+1
= w
i
+ hy

(k + ih)
Thus, we have replaced the dierential equation (continuous) with a recurrence (discrete).
Rather than generate a function as a solution to our Initial Value Problem, we have generated a series of
points
y = {(k, y(k)), (k + h, y(k + h)), (k + 2h, y(k + 2h)), ..., (k + , y(k + ))}
Aside: What could we do if we wanted to turn this vector into a function? Interpolate the points!
Our solution may also be described as a vector:
y = [y(k), y(k + h), y(k + 2h), ..., y(k + )]
In general, a solution is considered to be good if:
|y(k + ih) y
i
| ; i {1, 2, ...,

h
}
What is the order of the error bound in using Eulers Method?
As long as both both the derivative and the second derivative are bounded on the interval:
25
Local Error: O(h
2
)
Global Error: O(h)
Whats this Global Error?
We incur an error on every function evaluation. But every error incurred will propagate through to create
more error in the subsequent function evaluations.
The factor of h dierence between local and global error we saw with Euler is typical for numerical solutions
to dierential equations.
Beating Euler
First, what are the pros and cons of Euler
Pros:
Easy to program
Easy to understand
Cons:
O(h) global error
Given that we used a linear Taylor series to build Eulers method, whats a trivial way of improving upon
it? Use a higher degree Taylor series.
Degree n Eulers Method:
Dene w
i
to be y(k + ih)
w
0
= y(k)
w
i+1
= w
i
+
n

j=1
h
j
y
(j)
(k + ih)
What does the error look like then?
As long as both both the derivative and the second derivative are bounded on the interval:
26
Local Error: O(h
n+1
)
Global Error: O(h
n
)
Pros and cons here?
Pros:
Arbitrarily good error bounds.
Cons:
Have to compute (n 1) derivatives!
Runge Kutta Methods:
We may improve the performance of Eulers method without having to compute derivatives with a simple
strategy. Instead of having steps linearize the full interval of length h using the value of y

at their current
location, do the following:
Find y(x +
h
2
) by linearizing the interval of length
h
2
Compute y

at this point
Take the derivative you found and use it to linearize the full interval of length h
Thus, the recurrence may be expressed as follows
Midpoint Method:
Dene w
i
to be y(k + ih)
w
0
= y(k)
w
i+1
= w
i
+ hy

(w
i
+
h
2
y

(k + ih))
Error Terms
As long as both both the derivative and the second derivative are bounded on the interval:
Local Error: O(h
3
)
Global Error: O(h
2
)
27
This is just one of innitely many Runge Kutta methods. The most famous of which is often called the
classical Runge Kutta method, or RK4
RK4 (Classical Runge Kutta):
Dene our Initial Value Problem to be y

= f(x, y); y(k) =

y
n+1
= y
n
+
1
6
(k
1
+ 2k
2
+ 2k
3
+ k
4
)
k
1
= hf(x
n
, y
n
)
k
2
= hf(x
n
+
h
2
, y
n
+
k
1
2
)
k
3
= hf(x
n
+
h
2
, y
n
+
k
2
2
)
k
4
= hf(x
n
+ h, y
n
+ k
3
)
What makes this so awesome?
As long as both both the derivative and the second derivative are bounded on the interval:
Local Error: O(h
5
)
Global Error: O(h
4
)
A Note on the Algorithmic Stability of Eulers Method
Eulers Method has another major aw in that it can be a numerically unstable algorithm. A nu-
merically unstable algorithm is an algorithm for which a small change in initial condition will result in a
disproportionately large change in the behavior of the algorithm over time. The gure below shows the
convergence (and nonconvergence) of Eulers method on the same function, with the same starting point,
using slightly dierent values for the interval h.
28
Another example of an unstable algorithm weve encountered in this class is Newtons Method. When x
0
is
near a local extremum, or a horizontal asymptote, a small change in initial value will likely result in very
dierent algorithmic behavior.
Stability diers from conditioning in that it is a property of the algorithm, not the problem itself. One may
surmount an numerically unstable algorithm by merely selecting a new algorithm. This is not so with an
ill-conditioned problem.
Eulers method, for instance, has a more stable counterpart known as Backward Euler:
w
0
= y(k)
w
i+1
= w
i
+ hy

(k + (i + 1)h)
Backward Euler uses the y

value at the next point in the sequence (w

i+1
) to compute w
i+1
. This process in
itself represents a problem that must be solved numerically, but the benet is a far more stable algorithm.
14 Finite Precision Arithmetic
As we have seen, computers must discretize all information that is being sent through them. When working
with continuous models, this presents the inherent problem of modeling an innitely precise continuum
with a xed number of bits. Using oating point values, computers may achieve very good precision in
comparatively little space, but it is important to consider the implications of nite precision arithmetic:
1. Rounding errors:
As we saw in our discussion on root nding in Wilkinsons polynomial, a computer may not be capable
of understanding the dierence between two arbitrarily close values. For ill-conditioned problems,
chaotic systems, or unstable algorithms, this presents a serious problem
2. Precision loss:
Consider a simplied oating point model on a computer that has the storage capacity for a one digit
exponent and a six digit mantissa (signicand). The number A = 30000 may be expressed like
A = 3.00000 10
4
Meanwhile the number B = .09 may be expressed as
B = 9.00000 10

2
When we try to sum these two numbers and computer A+B, the computer will attempt to align their
values with equal exponents, executing the addition operation
3.0000000 10 4
+ 0.0000050 10 4
3.0000050 10 4
However, the computer is only capable of storing a six digit mantissa, so the least signicant digits
are dropped producing A + B = 3.00000 10
4
. We have added a nonzero value to 30000 and recieved
exactly 30000 in return. This is known as a loss of precision.
29
15 Moores Law
Moores law is considered one of the most important laws in computing. It states that the number of
transistors that can be placed aordably on a chip doubles every two years. The implications are that
computational power increases exponentially with time.
Moores law will face some major hurdles in the coming years, however. Some experts believe that the
famous trend has already come to an end. Some primary adversaries of Moorian progress are:
1. Quantum tunnelling (transistors have been small enough for tunnelling to be an issue for years now)
2. Finite speed of light
3. Finite size of the atom
Some possible solutions include:
1. Layered circuitry
2. Quantum computing
Good luck with your studies! Im going to bed.
30