Introduction to Numerical Analysis II:

Newtons Method

Xiaozhe Hu

The Pennsylvania State University

X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 1/8

Newtons Method
Recall Newtons method for solving one nonlinear equation:

f (x) = 0.

X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 2/8

Newtons Method
Recall Newtons method for solving one nonlinear equation:

f (x) = 0.

Newtons Method:
Given initial guess x 0 :

For (k = 0, 1, 2 )
f (x k )
x k+1 = x k
f 0 (x k )
End For

X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 2/8

Newtons Method
Recall Newtons method for solving one nonlinear equation:

f (x) = 0.

Newtons Method:
Given initial guess x 0 :

For (k = 0, 1, 2 )
f (x k )
x k+1 = x k
f 0 (x k )
End For

Question:
Why does Netwons method converge quadratically?

X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 2/8

Another Derivation of Newtons Method
Let r be the root, then
1
0 = f (r ) = f (x k ) + f 0 (x k )(r x k ) + f 00 ( k )(r x k )2 , (1)
2
where k is between r and x k . Drop the high order term, we have

0 = f (r ) f (x k ) + f 0 (x k )(r x k ).

Then we can introduce the new approximation x k+1 r , such that

0 = f (x k ) + f 0 (x k )(x k+1 x k ), (2)

which leads to the Newtons Method

f (x k )
x k+1 = x k .
f 0 (x k )

X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 3/8

Quadratic Convergence Rate
Subtract (1) and (2), we have

1
(r x k+1 )f 0 (x k ) + f 00 ( k )(r x k )2 = 0
2
rearrange the terms and take absolute value on both sides,

|r x k+1 | f 00 ( k )
= | |
|r x k |2 2f 0 (x k )

take the limit on both side, we end up with the following theorem:

X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 4/8

Quadratic Convergence Rate
Subtract (1) and (2), we have

1
(r x k+1 )f 0 (x k ) + f 00 ( k )(r x k )2 = 0
2
rearrange the terms and take absolute value on both sides,

|r x k+1 | f 00 ( k )
= | |
|r x k |2 2f 0 (x k )

take the limit on both side, we end up with the following theorem:
Theorem
Assume f is twice continuously differentiable and f 0 (r ) 6= 0, then Newtons
method is locally and quadratically convergent to r , and

|r x k+1 | f 00 (r )
lim = | |
k |r x k |2 2f 0 (r )

X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 4/8

Newtons Method to System of Nonlinear Equations
Simple Example: two equations

f1 (x1 , x2 ) = 0
= F(x) = 0
f2 (x1 , x2 ) = 0

X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 5/8

Newtons Method to System of Nonlinear Equations
Simple Example: two equations

f1 (x1 , x2 ) = 0
= F(x) = 0
f2 (x1 , x2 ) = 0

Taylor expansion F(x):

0 = F(r) = F(xk )+??(r xk ) + O(|r xk |2 )

What is ??:

X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 5/8

Newtons Method to System of Nonlinear Equations
Simple Example: two equations

f1 (x1 , x2 ) = 0
= F(x) = 0
f2 (x1 , x2 ) = 0

Taylor expansion F(x):

0 = F(r) = F(xk )+??(r xk ) + O(|r xk |2 )

What is ??:
f1 k f1 k
x1 (x ) x2 (x )
J(xk ) =


f2 k f2 k
x1 (x ) x2 (x )

which is called Jacobian matrix.

X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 5/8

Newtons Method to System of Nonlinear Equations
Simple Example: two equations

f1 (x1 , x2 ) = 0
= F(x) = 0
f2 (x1 , x2 ) = 0

Taylor expansion F(x):

0 = F(r) = F(xk )+??(r xk ) + O(|r xk |2 )

What is ??:
f1 k f1 k
x1 (x ) x2 (x )
J(xk ) =


f2 k f2 k
x1 (x ) x2 (x )

which is called Jacobian matrix. Then introduce xk+1 r such that

0 = F(xk ) + J(xk )(xk+1 xk ) = xk+1 = xk J(xk )1 (F(xk ))

X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 5/8

Newtons Method to System of Nonlinear Equations
General case:


f1 (x1 , x2 , , xn ) = 0
f2 (x1 , x2 , , xn ) = 0

.. = F(x) = 0


.
fn (x1 , x2 , , xn ) = 0

X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 6/8

Newtons Method to System of Nonlinear Equations
General case:


f1 (x1 , x2 , , xn ) = 0
f2 (x1 , x2 , , xn ) = 0

.. = F(x) = 0


.
fn (x1 , x2 , , xn ) = 0

fi
Jacobian Matrix: (J)ij = xj

X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 6/8

Newtons Method to System of Nonlinear Equations
General case:


f1 (x1 , x2 , , xn ) = 0
f2 (x1 , x2 , , xn ) = 0

.. = F(x) = 0


.
fn (x1 , x2 , , xn ) = 0

fi
Jacobian Matrix: (J)ij = xj

Newtons Method
Given initial guess x0
For (k = 0, 1, 2, ,)
1 Solve: J(xk )xk = F(xk ) (solve a linear system)
2 Update: xk+1 = xk + xk
End For
X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 6/8
Example

Example 1.
x13 + x2 = 0
x12 + x22 1 = 0
with initial guess x0 = (1, 2)T .

X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 7/8

Quadratic Convergence Rate Again

Theorem
Suppose that J is Lipschitz continuous with Lipschitz constant in a ball
of radius r around r, and J(r) is nonsingular with kJ(r)1 k . If
kx0 rk min[r , 1/(2)], then the sequence determined by Newtons
method is well-defined, convergence to r, and satisfies

kxk+1 rk kxk rk2

X. Hu (Penn State) MATH/CMPSC 456 Jan. 11, 2012 8/8