Sie sind auf Seite 1von 6

Implementation and Analysis of Conjugate Gradient Method

Hsuan Yang h.yang@uci.edu June 10, 2009

http://www.ics.uci.edu/~hyang4/cs206_project.html

CS206: Scientific Computing - Term Project Instructor: Professor James Arvo University of California, Irvine

1.

Introduction

Conjugate Gradient Method (CG) is one of the most popular iterative methods in solving large-scale system of linear equations. Generally, CG could be treated as either a direct method or an iterative method. In practice, CG is used as an iterative method in order to be applied to a sparse system which is too large to be handled by direct method.

In this paper, I will focus on the efficiency and capability of CG. I will show the implementation and analysis based on the massive computation of the programs. If you are interested in how CG works, a detailed introduction on CG is provided by Shewchuk [1]. Base on the materials from Shewchuk’s paper, I present this paper as an introduction to provide the classmates in CS206 an insight into CG without recurring the materials which are covered in class already.

For the following section, the main idea of the CG algorithm is described in section 2; the implementation details are described in section 3; the analysis of the algorithm is in section 4; conclusion and future works are described in section 5.

1

2.

Suppose we want to solve the following system of linear equations

 where is a known -by- , symmetric ( ), positive-definite ( 0 for all non-zero vectors in ), and real matrix; is an unknown vector in ; and is a

known vector in .

By using CG as an iterative method, we will need an initial guess for the unique solution of the system. Without losing generality, we could set 0. Next, we need a metric to evaluate how close we’ve gotten to the solution , which is unknown. Base on the explanation by Shewchuk [1], given a quadratic function

1 2

where is a matrix, and are vectors, if is symmetric and positive-definite, is minimized by the solution of . turns out to be the metric we are looking for.

From the initial guess , we let the first direction to be the gradient

at , which equals .

Now we define

as the residual at the th step

.

Since equals the negative gradient of at , CG would be to move in the direction of . But instead of following the direction of , which would be called the method of steepest descent (SD), CG insists that every direction are conjugate to each other, so it picks a conjugate direction which is closest to the residual .

2

Now we can put together all the materials in one piece, the method of Conjugate Gradients is:

3.

Implementation

.

Based on the pseudocodes provided by Shewchuk [1], in this section I will present the implementation of the following algorithms:

a) the method of Steepest Descent;

b) the method of Conjugate Gradients,

in order to test and compare the algorithms with numerical inputs. The entire implementation

3.1 The Method of Steepest Descent

The Matlab code are provided below

function [x,i] = SteepestDescent(A,b) i_max = 10^7; epsilon = 10^-5;

x = zeros(size(b));

i = 0;

r = b - A*x; delta = r'*r;

delta_0 = delta; while (i < i_max) && (delta > epsilon^2*delta_0)

i = i

q = A*r;

alpha = delta / (r'*q); xold = x;

x = x + alpha*r;

if mod(i,50) == 0

+ 1;

 r = b - A*x; else r = r - alpha*q;

end delta = r'*r;

end

end

3

3.2 The Method of Conjugate Gradients

The Matlab code are provided below

function [x,i] = ConjugateGradients(A,b) i_max = 10^7; epsilon = 10^-5;

 x = zeros(size(b)); i = 0; r = b - A*x; d = r;

delta = r'*r; delta_0 = delta; while (i < i_max) && (delta > epsilon^2*delta_0)

 i = i + 1; q = A*d;

alpha = delta / (d'*q);

xold = x;

x = x + alpha*d;

if mod(i,50) == 0

 r = b - A*x; else r = r - alpha*q;

end delta_old = delta; delta = r'*r;

beta = delta / delta_old;

d = r + beta*d;

end

end

4. Numerical Analysis

In Schewchuk’s paper [1], the following is used as a sample problem

3

2

6 2 ,

2

8

.

After I implemented the both algorithms, I take the above data as test input and have my first numerical results in Table 1. From the table we could see the difference between the efficiency of SD and CG. The result of SD, which appears as some tailing zeros, is due to the floating accuracy that omits the tiny fraction of the error. The graphical result is presented in Figure 1.

Furthermore, I experimented with larger matrices and presented the results in Table 2. Not surprisingly, CG shows the efficiency which is much faster than SD.

 Steepest Descent Conjugate Gradients Iterations 39 2 Results 2.000 2.000 2 2

Table 1: The results of the implementation with input

4

3

2

2

6 ,

2

8 .

2
1
0
-1
-2
-3
-4
-2
-1
0
1
2
3
4

2

is also presented to give a better understanding of the function’s characteristic.

function

3

2

Figure

graphical

results

input

2 8 .

1:

the

of

6 ,

The

contour

of

the

The result of SD is presented as blue line, and the result of CG is presented as red line.

 Size of Number of Iteration (n) Steepest Descent Conjugate Gradients Best case Worst case Average Best case Worst case Average 2-by-2 5 465 93 2 2 2 4-by-4 96 17,666 4,373 4 4 4 6-by-6 595 2,306,344 395,070 6 6 6 8-by-8 23,590 10,000,000 (*) 5,097,507 8 9 8.8 10-by-10 36,350 10,000,000 (*) 5,692,141 11 12 11.6 12-by-12 2,204,172 10,000,000 (*) 7,520,434 15 24 18.8 14-by-14 4,233,967 10,000,000 (*) 8,902,891 17 25 23.1 16-by-16 6,697,330 10,000,000 (*) 9,069,733 23 28 25.7

Table 2: results compared with SD and CG. The number of iteration is experimented under 100 times of random inputs under each size of . (*) labeled elements indicate that the maximum number of iteration (which is defined as 10 7 ) has been exceeded.

5

5.

Conclusion and Future Work

Conjugate Gradient Method is an algorithm used widely for solving system of linear equations. The results of the experiments showed the efficiency of the algorithm comparing with other implementations.

CG can deal with only symmetric, positive-definite matrices, which could be seemed as its limitation. However, further research had been done and various type of CG had developed in order to solve more problems. The Preconditioned Conjugate Gradient Method is a way to ensure fast convergence; the Biconjugate Gradient Method provides a generalization to non-symmetric matrices. Much more implementation and analysis could be done in the future in order to understand these powerful tools and use them in practice.

6. References

[1]

Jonathan Richard Shewchuk, “An Introduction to the Conjugate Gradient Method Without the Agonizing Pain,” August 4, 1994.

[2] R. Barrett, M. Berry, T. F. Chan, J. Demmel, J. Donato, J. Dongarra, V. Eijkhout, R. Pozo, C. Romine, and H. van der Vorst, Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods, SIAM, Philadelphia, Pennsylvania, 1993.

[3] Magnus R. Hestenes, Eduard Stiefel, "Methods of Conjugate Gradients for Solving Linear Systems," Journal of Research of the National Bureau of Standards, Vol. 49, No. 6, December 1952.

[4]

John K. Reid, On the Method of Conjugate Gradients for the Solution of Large Sparse Systems of Linear Equations, Large Sparse Sets of Linear Equations (London and New York) (John K. Reid, ed.), Academic Press, London and New York, 1971, pp. 231–254.

[5] Stanley C. Eisenstat, Efficient Implementation of a Class of Preconditioned Conjugate Gradient Methods, SIAM Journal on Scientific and Statistical Computing, Vol. 2, Issue 1, March 1981.

[6]

R. Fletcher, Conjugate gradient methods for indefinite systems, Numerical Analysis, Lecture Notes in Mathematics, Springer Berlin / Heidelberg, Vol. 506, p. 73–89, 1976.

6