Convergence Rates on Root Finding

Com S 477/577
Oct 5, 2004

A sequence xi R converges to if for each > 0, there exists an integer N () such that
|xl | > for all l N (). The Cauchy convergence criterion states that a sequence xi R
is convergent if and only if for each > 0 there exists an N () such that |xl xm | < for all
l, m N ().
Let a sequence xi R be generated by an iteration function , that is,
xi+1 = (xi ),

i = 0, 1, 2, . . .

Let be a fixed point of , that is, = (). Suppose that the sequence {xi } is generated in the
neighborhood of . The corresponding iteration method is said to be of at least pth order if there
exists a neighborhood N () of such that for all x0 N () the generated sequence xi+1 = (xi ),
i = 0, 1, . . ., satisfies
|xi+1 | C|xi |p ,
where C < 1 if p = 1. In the case of first order convergence, for instance, we have
|xi | C|xi1 | C 2 |xi2 | C i |x0 |.
Since C < 1, it follows that
lim |xi | = lim C i |x0 | = 0,

namely, the sequence {xi } will converge to .

Now suppose is sufficiently often differentiable in N (). If xi N () and if (k) () = 0 for
k = 1, 2, . . . , p 1 but (p)() 6= 0, that is, is a zero of order p, then it follows from the Taylor
expansion that
xi+1 = (xi )
= () +



(p) ()
(xi )p + O (xi )p+1 .
p!

Because () = , we obtain
xi+1
(p) ()
=
.
i (xi )p
p!
lim

For p = 2, 3, . . ., the method is of (precisely) pth order.

The method is of first order if p = 1 and | ()| < 1. When 0 < () < 1, the sequence {xi }
will converge monotonically to as shown in the left figure below. When 1 < () < 0, the
sequence will alternate about during convergence as shown in the right figure.
1

( x)

( x)

xi 0

x i+1

x i +2

xi

x i +2

x i+1

In the below we study the convergence rates of several root finding methods introduced before.

Quadratic Convergence of Newtons Method

Newtons method has the iteration function

f (x)
(x) = x
f (x)

with

f () = 0.

Suppose f is sufficiently continuously differentiable in some neighborhood N (). In the nondegenerate case, f () 6= 0. So we have
() = ,

2


f (x) f (x)f (x) 

() = 1


2


f (x)
x=
= 0,

since f () = 0,


2


2

f (x) f (x)f (x) 

2f (x)f (x) f (x)f (x) f (x)f (x) f (x) 2f (x)f (x)

() =


4

f (x)
x=

3

2

2
f () f () f () f () f () + 2f ()f () f ()
=

4
f ()

3
f () f ()
,
since f () = 0
=

4
f ()
f ()
f ()
6
=
0.

So Newtons method is quadratically convergent.

In the degenerate case, is an m-fold zero of f , for some m > 1, that is, f (i) () = 0, for
i = 0, 1, . . . , m 1. We will leave to the students to determine the order of convergence in this case.
2

Linear Convergence of Regula Falsi

For clarity of analysis we let xi = bi for all i. We make some simplification assumptions for the
discussion of the convergence behavior: f exists and for some k the following conditions hold: (a)
xk < ak ; (b) f (xk ) < 0 and f (ak ) > 0; (c) f (x) 0 for all x [xk , ak ].

x k +1

xk

ak

(a)

(b)

Under these assumptions, either f (xk+1 ) = 0 or f (xk+1 )f (xk ) > 0 and consequently
xi < xi+1 < ai+1 = ai .
To see this, use the remainder formula for polynomial interpolation at xk and ak :
f (x) p(x) = (x xk )(x ak )

f ()
2

for x [xk , ak ] and a suitable [xk , ak ]. Under condition (c), the above equation implies that
f (x) p(x) 0. In particular, f (xk+1 ) p(xk+1 ) 0, which in turn implies that f (xk+1 ) 0 since
p(xk+1 ) = 0.
Unless f (xk+1 ) = 0, in which case the iteration stops at xk+1 , we can see that conditions (a),
(b), and (c) hold for all i k. Therefore ai = ak = a and
xi+1 =

af (xi ) xi f (a)
f (xi ) f (a)

for all i k. Furthermore, {xi } for i k form a monotone increasing sequence bounded by a. So
limi xi = exists. Consequently,
f () 0,

f (a) > 0,

and

af () f (a)
,
f () f (a)

which gives
( a)f () = 0.
But < a since f () 0 < f (a). Hence f () = 0 and {xi } converges to a zero of f .
The above discussion enables us to look at the order of convergence through the iteration
function
af (x) xf (a)
xi+1 = (xi ),
where
(x) =
.
f (x) f (a)
3

Since f () = 0, we obtain that



af () f (a) f (a) + f (a)f ()
a
() =
= 1 f ()
.
2
f (a)
f () f (a)
By the mean value theorem, there exist 1 , 2 such that
f () f (a)
a
f (xi ) f ()
xi

=
=

f (a)
= f (1 ),
a
f (xi )
= f (2 ),
xi

< 1 < a;

(1)

xi < 2 < .

(2)

Since f (x) 0, f (x) increases monotonically in [xi , a], So f (2 ) f () f (1 ). Meanwhile,

condition (2), xi < , and f (xi ) < 0 together imply that 0 < f (2 ). Therefore 0 < f () f (1 ).
We have thus shown that
f ()
< 1.
0 () = 1
f (1 )
So the regula falsi method converges linearly.
From the previous discussion we see that the method of regula falsi will almost always end up
with the one-sided convergence demonstrated before.

Superlinear Convergence of Secant Method

In secant method, the iteration is in the form

xi+1 = xi

f (xi )
,
f [xi1 , xi ]

i = 0, 1, . . .

(3)

To determine the local convergence rate, without loss of generality we assume that the sequence
{xi } is in a small enough neighborhood of the zero and that f is twice differentiable. Subtract
from both sides of (3):
f (xi )
xi+1 = (xi )
f [xi1 , xi ]


f (xi )
f [xi , ]
f (xi ) f ()
=
= (xi ) 1
,
since f [xi , ] =
f [xi1 , xi ]
xi
xi
f [xi1 , xi ] f [xi , ]
= (xi )(xi1 )
(xi1 )f [xi1 , xi ]
f [xi1 , xi , ]
= (xi )(xi1 )
.
f [xi1 , xi ]
From error estimation of polynomial interpolation, we learned that
f [xi1 , xi ] = f (1 ),
1
f (2 ),
f [xi1 , xi , ] =
2
4

1 I[xi1 , xi ];
2 I[xi1 , xi , ],

(4)

where I[xi1 , xi ] is the smallest interval containing xi1 and xi , and I[xi1 , xi , ] the smallest
interval containing xi1 , xi ,
n 


If is a simple zero, that is, f () 6= 0, there exists a bound M and an interval J = x  |x|
o
for some > 0 such that


 1 f (2 ) 


(5)
 2 f (1 )  M,
for any 1 , 2 J.
Let ei = M |xi | and e0 , e1 < min{1, M }. By induction and using (4) and (5) we can easily
show that
ei ei1
ei+1 = M |xi+1 | M

M = ei ei1 ,
M M
and
|ei | min{1, M },
for i = 1, 2, . . ..
Let q = (1 + 5)/2 be the root of the equation z 2 z 1 = 0. Then we have
i

ei K q ,

where K = max{e0 ,

i = 0, 1, 2, . . .

q
e1 } < 1. This is because (by induction)
i

ei+1 ei ei1 K q K q

i1

= Kq

i1 (q+1)

= Kq

i1 q 2

= Kq

i+1

Thus the secant method converges at least as well as a method of order p = 1+2 5 = 1.618 . . ..
One-step secant requires one additional function evaluation. But one-step Newton requires two
(f and f ). Therefore two secant steps are as expensive as single Newton step. But two secant
steps has a convergence order of (1.618)2 2.618. This explains why in practice the secant method
always dominates Newtons method with numerical derivatives.

References
[1] J. Stoer and R. Bulirsch. Introduction to Numerical Analysis. Springer-Verlag New York, Inc.,
2nd edition, 1993.
[2] M. Erdmann. Lecture notes for 16-811 Mathematical Fundamentals for Robotics. The Robotics
Institute, Carnegie Mellon University, 1998.
[3] W. H. Press, et al. Numerical Recipes in C++: The Art of Scientific Computing. Cambridge
University Press, 2nd edition, 2002.