CS4413 Algorithm Analysis

Chapter 1

(These materials are used in the classroom only)

Outline
Mathematical Induction Concept: Algorithms design and analysis process Efficiency, Analysis, and Order Some algorithms Asymptotic notation

Mathematical Induction

Mathematical Induction

A very important type of reasoning which permeates much of mathematical thought. Review (Proof by induction).

Mathematical Induction
Mathematical induction is used to establish with logical certainty the correctness of a theorem for an infinite sequence of cases. Let A denote a statement referring to an arbitrary natural number n. For example, A might be the statement The sum of the interior angles in a simple polygon of n + 2 sides is n times 180 degrees or n.

General formulation
To prove a statement of this type it is not sufficient to prove it for the first 10 or the first 100 or even the first 1000 values of n. Need principle of mathematical induction.

Principle of mathematical induction

The possibility of proof depends on two factors: 1) a general proof has to be given showing that the statement Ar+1 is correct whenever Ar is correct, and 2) the statement A must be proved.

These two conditions are sufficient to prove the correctness of all A1, A2, A3, constitutes the Principle of mathematical induction.

Proof By Induction
Claim: S(n) is true for all n >= k
Basis Step:
Show formula is true when n = k

Inductive hypothesis Step:

Assume formula is true for an arbitrary n Show that formula is then true for n+1

Review: Induction
Suppose
S(k) is true for fixed constant k
Often k = 0

S(n) S(n+1) for all n >= k

Then S(n) is true for all n >= k

Induction Example: Gaussian Closed Form

Prove 1 + 2 + 3 + + n = n(n+1) / 2
Basis Step:
If n = 0, then 0 = 0(0+1) / 2

Inductive hypothesis Step:

Assume 1 + 2 + 3 + + n = n(n+1) / 2

(show true for n+1):

1 + 2 + + n + n+1 = (1 + 2 + + n) + (n+1) = n(n+1)/2 + n+1 = [n(n+1) + 2(n+1)]/2 = (n+1)(n+2)/2 = (n+1)(n+1 + 1) / 2.

Induction Example: Gaussian Closed Form

Prove that 12 + 22 + 32 + + n2 = (2n3 + 3n2 + n)/6 (sum of squares) What is this called? Polynomial Series.

Induction Example: Geometric Closed Form

Prove a0 + a1 + + an = (an+1 - 1)/(a - 1) for all a 1
Basis Step: show that a0 = (a0+1 - 1)/(a - 1)
a0 = 1 = (a1 - 1)/(a - 1)

Inductive hypothesis Step:

Assume a0 + a1 + + an = (an+1 - 1)/(a - 1)

(show true for n+1):

a0 + a1 + + an+1 = a0 + a1 + + an + an+1 = (an+1 - 1)/(a - 1) + an+1 = (an+1+1 - 1)/(a - 1)

Exercises on Induction: Gaussian Closed Form

Prove that (1 + x)n >= 1 + nx whenever n is a nonnegative integer and x is a nonnegative real number.
Prove that k=1 n [a + (k 1)d] = an + [n(n- 1)d]/2 (arithmetic series)

Exercises on Induction: Gaussian Closed Form

Prove the Binomial Theorem: The statement A of the theorem is represented by the formula

(a + b)n = an + (n/1)an - 1b + (n(n -1)/(1 x 2))an2b2 + (n(n -1)(n -2)/(1 x 2 x 3))an-3b3 + + (n(n -1)(n -2) x x 2 x 1)/(1 x 2 x 3 x x n))bn

Exercises on Induction: Gaussian Closed Form

Prove that For n > 1, 2 + 22 + 23 + 24 + ... +

2n = 2n+1 2.
Prove that For n > 1, 12 + 23 + 34 + ... + (n)(n+1) = (n)(n+1)(n+2)/3.

Exercises on Induction: Gaussian Closed Form

Prove that For n > 5, 4n < 2n.
Prove that For all n > 1, 8n 3n is divisible by 5.

Induction
Weve been using weak induction Strong induction also holds
Basis: show S(0) Hypothesis: assume S(k) holds for arbitrary k <= n Step: Show S(n+1) follows

Another variation:
Basis: show S(0), S(1) Hypothesis: assume S(n) and S(n+1) are true Step: show S(n+2) follows

Algorithms
Problem: is a question to which we seek an answer. Algorithm: Problem solutions. Example: Sort a list S of n numbers in nondecreasing order. The answer is the numbers in sorted sequence. The purpose of studying these techniques (algorithms) is that when confronted with a new problem, you have a repertoire of techniques to consider as possible ways to solve the problem.
Note: This starts Chapter 1.

Algorithms
1.1 - Sequential search 1.2 - Binary search

Note: These are from Chapter 1.

Algorithms
1.1 - Sequential search Examples:
Sequential Search Add Array Members Exchange Sort Matrix Multiplication

Note: These are from Chapter 1.

Algorithms
Binary search algorithm

Note: These are from Chapter 1.

Developing efficient Algorithms

Sequential search versus binary search

Which one is more efficient?

Table 1.1 A comparison

Array Size

Number of comparisons by Sequential Search

128 128 1,024 1,048,576 4,294,967,296

Number of Comparisons by Binary Search

8 11 21 33

1,024 1,048,576 4,294,967,296

The Fibonacci Sequence

The Fibonacci Sequence

Nth Fibonacci Term algorithm (recursive) Nth Fibonacci Term algorithm (iterative)

Table 2: A comparison
n n+1 2n/2 Execution time (Iterative)
41 ns 61 ns 81 ns 101 ns 121 ns 161 ns 201 ns

Lower Bound on Execution time (Recursive)

1048 us 1s 18 min 13 days 36 years 3.8 x 107 years 4 x 1013 years

40 60 80 100 120 160 200

41 61 81 101 121 161 201

1,048,576 1.1 x 109 1.1 x 1012 1.1 x 1015 1.2 x 1018 1.2 x 1024 1.3 x 1030

1 ns = 10-9 second 1 us = 10-6 second

Analysis of Algorithms

Complexity analysis We analyze the efficiency of an algorithm in terms of time.

Actual number of CPU cycles? Count every instruction executed?

Analysis of Algorithms
1) Input size? 2) Basic operation? - One instruction or group of instructions. 3) T(n) is defined as the number of times the algorithm does the basic operation for an instance of size n. T(n) is called the every-case time complexity of the algorithm. The determination of T(n) is called an every-case time complexity analysis.

Analysis of Algorithms

Analysis of algorithm 1.2 Add Array Members

Analysis of algorithm 1.3 Exchange Sort Analysis of algorithm 1.4 Matrix Multiplication

Analysis of Algorithms

Analysis of algorithm 1.1 Sequential Search

Worst-case time complexity Average-case time complexity Best-case time complexity

Complexity functions
What is a function?
A complexity function can be any function that maps the positive integers to the nonnegative reals. We will use f(n) and g(n) to represent complexity functions. Example 1.6 (page 23).

Order What is order?

An intuitive introduction to order A rigorous introduction to order Using a limit to determine order

Table 1.3: The quadratic term eventually dominates

0.1n2

0.1n2 + n + 100

10 20 50 100 1,000

10 40 250 1,000 100,000

120 160 400 1,200 101,100

The running time of a program

Order
Is 0.01n2 > 100n? Linear-time algorithms Quadratic-time algorithms ( (n2 ) algorithm) Pure quadratic Complete quadratic

Asymptotic Performance

In this course, we care most about asymptotic performance

How does the algorithm behave as the problem size gets very large?
Running time Memory/storage requirements Bandwidth/power requirements/logic gates/etc.

Asymptotic Notation

By now you should have an intuitive feel for asymptotic (big-O) notation:
What does O(n) running time mean? O(n2)? O(n log n)? How does asymptotic running time relate to asymptotic memory usage?

Our first task is to define this notation more formally and completely

Analysis of Algorithms
Analysis is performed with respect to a computational model We will usually use a generic uniprocessor randomaccess machine (RAM) All memory equally expensive to access No concurrent operations All reasonable instructions take unit time Except, of course, function calls Constant word size Unless we are explicitly manipulating bits

Input Size
Time and space complexity This is generally a function of the input size E.g., sorting, multiplication How we characterize input size depends: Sorting: number of input items Multiplication: total number of bits Graph algorithms: number of nodes & edges etc

Running Time
Algorithm FindMax Int findMax (E, n) 1. max = E [0]; 2. for (index = 1; index < n; index ++) 3. if (max < E [index]) 4. max = E [index]; 5. return max;

Running Time
Number of primitive steps that are executed Except for time of executing a function call, most statements roughly require the same amount of time y=m*x+b c = 5 / 9 * (t - 32 ) z = f(x) + g(y) We can be more exact if need be

Analysis
Worst case
Provides an upper bound on running time An absolute guarantee

Average case
Provides the expected running time Very useful, but treat with care: what is average?
Random (equally likely) inputs Real-life inputs

Summary

Principle of mathematical induction. Algorithmic design and analysis concepts (chapter 1). Computational models for analysis. Asymptotic notation. Appendix A (textbook).

The End

Questions & Suggestions?