Algorithm Analysis

Algorithms that are equally correct

can vary in their utilization of
computational resources
time and memory
a slow program it is likely not to be used
a program that demands too much
memory may not be executable on the
machines that are available
E.G.M. Petrakis

Algorithm Analysis

Memory - Time
It is important to be able to predict
how an algorithm behaves under a
range of possible conditions
usually different inputs

The emphasis is on the time rather

than on the space efficiency
memory is cheap!
E.G.M. Petrakis

Algorithm Analysis

Running Time
The time to solve a problem increases with
the size of the input
measure the rate at which the time increases
e.g., linearly, quadratically, exponentially

This measure focuses on intrinsic

characteristics of the algorithm rather
than incidental factors
e.g., computer speed, coding tricks,
optimizations, quality of compiler
E.G.M. Petrakis

Algorithm Analysis

Additional Consideration
Sometimes, simple but less efficient
algorithms are preferred over more
efficient ones

the more efficient algorithms are not always

easy to implement
the program is to be changed soon
time may not be critical
the program will run only few times (e.g., once a
year)
the size of the input is always very small

E.G.M. Petrakis

Algorithm Analysis

Bubble sort
Count the number of steps executed by
the algorithm

step: basic operation

ignore operations that are executed a constant
number of times

for ( i = 1; i < ( n 1); i ++)

for (j = n; j < (i + 1); j --)
if ( A[ j 1 ] > A[ j ] ) T(n) = c n2
swap(A[ j ] , A[ j 1 ]); //step

E.G.M. Petrakis

Algorithm Analysis

Asymptotic Algorithm Analysis

Measures the efficiency of a program as
the size of the input becomes large
focuses on growth rate: the rate at which the
running time of an algorithm grows as the size
of the input becomes big
Complexity => Growth rate
focuses on the upper and lower bounds of the
growth rates
ignores all constant factors
E.G.M. Petrakis

Algorithm Analysis

Growth Rate
Depending on the order of n an algorithm can beof
Linear time Complexity: T(n) = c n
Quadratic Complexity: T(n) = c n2
Exponential Complexity: T(n) = c nk

The difference between an algorithm whose

running time is T(n) = 10n and another with running
time is T(n) = 2n2 is tremendous
for n > 5 the linear algorithm is must faster despite the
fact that it has larger constant
it is slower only for small n but, for such n, both
algorithms are very fast

E.G.M. Petrakis

Algorithm Analysis

E.G.M. Petrakis

Algorithm Analysis

Upper Lower Bounds

Several terms are used to describe the running
time of an algorithm:
The upper bound to its growth rate which is
expressed by the Big-Oh notation
f(n) is upper bound of T(n) T(n) is in O(f(n))
The lower bound to it growth rate which is
expressed by the Omega notation
g(n) is lower bound of T(n) T(n) is in (g(n))
If the upper and lower bounds are the same
T(n) is in (g(n))
E.G.M. Petrakis

Algorithm Analysis

Big O (upper bound)

Definition:T(n) (g(n)) if

T(n) cg(n) n n0

n n0 , c 1

Example:
3
3
T(n) = n T(n) (n )
T(n) = 6n 2n + 7 T(n) O(n )
2

T(n) = n + 10 n T(n) O(n )

3

E.G.M. Petrakis

Algorithm Analysis

10

Properties of O
f (g)
O is transitive: if
f (h)
g (h)

If f O(g) f + g O(g)
f O(f )
If g O(g ) f g O(f g )

f O(f )
f + g O(f + g )
If
g O(g )

E.G.M. Petrakis

Algorithm Analysis

11

More on Upper Bounds

T(n) O(g(n)) g(n) :upper bound
for T(n)
There exist many upper bounds
e.g., T(n) = n2 O(n2) , O(n3) O(n100) ...

Find the smallest upper bound!!

O notation hides the constants
e.g., cn2 , c2n2 , c!n2 ,ckn2 O(n2)
E.G.M. Petrakis

Algorithm Analysis

12

O in Practice
Among different algorithms solving
the same problem, prefer the
algorithm with the smaller rate of
growth O
it will be faster for large n

(1) < (log n) < (n) < (n log n) < (n2) < (n3)
< < (2n) < O(nn)
(n), (n log n), (n2) < (n3): polynomial
(log n): logarithmic
O(nk), O(2n), (n!), O(nn): exponential!!
E.G.M. Petrakis

Algorithm Analysis

13

n
100

10-6 sec/command
Complexity

1000n

20

200 500 100

.2
.5
1.0
0

.02

.5

.1

.9

.3

.6

1.5

4.5

10

100n 2

.04

.25

1.0

4.0

25

2min

100n 3

.08

1.0

10

1.0min

n logn

.4

1.1

2 n/3

.0001

0.1

2n

1.0

1000nlogn

nn
E.G.M. Petrakis

220da 125cent

21min 27hrs
5 108 cent

4
2.7hrs 3 10 cent

35yrs 3 10 4 cent

58min 2 109 cent

Algorithm Analysis

14

Omega (lower bound)

Definition: f(n) (g(n)) if c 1

f(n) cg(n)
g(n) is a lower bound of the rate of
growth of f(n)
f(n) increases faster than g(n) as n
increases
e.g: T(n) = 6n2 2n + 7 =>T(n) in (n2)
E.G.M. Petrakis

Algorithm Analysis

15

More on Lower Bounds

provides lower bound of the
growth rate of T(n)
if there exist many lower bounds
T(n) = n4 (n) , (n2) , (n3),
(n4)
find the larger one => T(n) (n4)
E.G.M. Petrakis

Algorithm Analysis

16

Thet
If the lower and upper bounds
are the same:
f O(g)
f (g)
f (g)

e.g.: T(n) = n3 +135n (n3)

E.G.M. Petrakis

Algorithm Analysis

17

Theorem: T(n)= i ni (nm)

T(n) = nnm +n-1nm-1++1n+ 0 O(nm)

Proof:
T(n) |T(n)| |nnm|+|m-1nm-1|+|1n|+|0|
nm{|m| + 1/n|m-1|++1/nm-1|1|+1/nm|0|}
nm{|m|+|m-1|++|1|+|0|}
T(n) O(nm)

E.G.M. Petrakis

Algorithm Analysis

18

Theorem (cont.)
b) T(n) = mnm + m-1nm-1 + 1n + 0 >=
cnm + cnm-1 + cn + c >= cnm
where c = min{m, m-1, 1, 0} =>
T(n) (nm)

(a), (b) => (n) (nm)

E.G.M. Petrakis

Algorithm Analysis

19

Theorem:
(a)
(b)

T(n) = i k (n k +1 ), k 0
i =1

T(n) = i k n k = n n k T(n) O(n k +1 )

i =1

i =1

i =1

i =1

2 T(n) = i k + (n - i + 1) k =
n

n
k
k
(i + (n - i + 1) ) ( )

i =1
i =1 2

n
(either i
2
T(n)
E.G.M. Petrakis

or

n
(n - i + 1) )
2

1
2

k +1
k +1
n

T
(
n)

(n
)
k +1

Algorithm Analysis

20

Recursive Relations (1)

1
if n = 1

T(n) =
T(n - 1) + n if n > 1

T(n) = T(n - 1) + n =
= T(n - 2) + (n - 1) + n =
= ......
= T(1) + 2 + .... + (n - 1) + n =
n(n + 1)
= i =
T(n) (n 2 )
2
i =1
n

E.G.M. Petrakis

Algorithm Analysis

21

Fibonacci Relation

F(0) = 0
F(1) = 1
F(n) = F(n - 1) + F(n - 2) if n 2

(a) F(n) = F(n - 1) + F(n - 2)

2F(n - 1) 4F(n - 2) ....
2 n -1 F(1) = 2 n -1 F(n) O(2 n )

(b) F(n) = F(n - 1) + F(n - 2) 2F(n - 2) 4F(n - 4)

F(n) 2 (n-1)/2 if n odd
n
n/2
F(n) (2 ) = ( 2 )

(n - 2)/2
if n even
F(n) 2
E.G.M. Petrakis

Algorithm Analysis

22

Fibonacci relation (cont.)

From (a), (b): F(n) ( 2 ) O(2 )
n

It can be proven that F(n)

where

(n)

1+ 5
=
= 1.6180
2

golden ratio
E.G.M. Petrakis

Algorithm Analysis

23

Binary Search
int BSearch(int table[a..b], key, n)
{
if (a > b) return ( 1) ;
int middle = (a + b)/2 ;
if (T[middle] == key) return (middle);
else if ( key < T[middle] )
return BSearch(T[a..middle 1], key);
else return BSearch(T[middle + 1..b], key);
}
E.G.M. Petrakis

Algorithm Analysis

24

Analysis of Binary Search

Let n = b a + 1: size of array and n = 2k 1
for some k (e.g., 1, 3, 5 etc)
The size of each sub-array is 2k-1 1

c k =0

T(n) =T(2 - 1) =
k 1
1) k > 0
d +T(2
k

c: execution time of first command

d: execution time outside recursion
T: execution time
E.G.M. Petrakis

Algorithm Analysis

25

Binary Search (cont.)

T(n) =T(2 k 1) = d +T(2 k -1 - 1) = 2d +T(2 k -2 - 1) = ...
= kd +T(0) = kd + c = d log(n + 1) + c
=>T(n) O(logn)

E.G.M. Petrakis

Algorithm Analysis

26

Binary Search for Random n

c
k =0

T(n) =
d + max{T((n 1)/2),T( (n 1)/2)} k > 0
T(n) <= T(n+1) <= ...T(u(n)) where u(n)=2k 1
For some k: n < u(n) < 2n
Then, T(n) <= T(u(n)) = d log(u(n) + 1) + c =>
d log(2n + 1) + c => T(n) O(log n)
E.G.M. Petrakis

Algorithm Analysis

27

Merge sort
list mergesort(list L, int n)
{
if (n == 1) return (L);
L1 = lower half of L;
L2 = upper half of L;
return merge (mergesort(L1,n/2),
mergesort(L2,n/2) );
}
n: size of array L (assume L some power of 2)
merge: merges the sorted L1, L2 in a sorted array
E.G.M. Petrakis

Algorithm Analysis

28

E.G.M. Petrakis

Algorithm Analysis

29

log2n merges

Analysis of Merge sort

[25]

[57]

[48]

[37]

[12]

[92]

[86]

[33]

[25

57]

[37

48]

[12

92]

[33

86]

[25

37

48

57]

[12

33

86

92]

[12

25

33

37

48

57

86

92]

N steps per merge, log2n steps/merge => O(n log2n)

E.G.M. Petrakis

Algorithm Analysis

30

Analysis of Merge sort

2T(n/2) + c2 n n > 1
T(n) =
c1
n =1

Two ways:
recursion analysis
induction

E.G.M. Petrakis

Algorithm Analysis

31

Induction
Let T(n) = O(n log n) then T(n) can be
written as T(n) <= a n log n + b, a, b > 0
Proof: (1) for n = 1, true if b >= c1

(2) let T(k) = a k log k + b for k=1,2,..n-1

(3) for k=n prove that T(n) <= a nlog n+b
For k = n/2 : T(n/2) <= a n/2 log n/2 + b
therefore, T(n) can be written as
T(n) = 2 {a n/2 log n/2 + b} + c2 n =
a n(log n 1)+ 2b + c2 n =

E.G.M. Petrakis

Algorithm Analysis

32

Induction (cont.)
T(n) = a n log n a n + 2 b + c2 n
Let b = c1 and a = c1 + c2 =>
T(n) <= (c1 + c2) n log n + c1 =>
T(n) <= C n log n + B

E.G.M. Petrakis

Algorithm Analysis

33

Recursion analysis of mergesort

T(n) = 2T(n/2) + c2 n <=
<= 4T(n/4) + 2 c2 n <=
<= 8T(n/8) + 3 c2 n <=

T(n) <= 2i T(n/2i) + ic2n where n = 2i

T(n) <= n T(1) + c2 n log n =>T(n) < = n c1 + c2
n log n => T(n) <= (c1 + c2) n log n =>
T(n) <= C n log n
E.G.M. Petrakis

Algorithm Analysis

34

Best, Worst, Average Cases

For some algorithms, different inputs
require different amounts of time
e.g., sequential search in an array of size n
n

...

x = pi xi = (n + 1)/2
comparisons

worst-case: element not in array => T(n) = n

best-case: element is first in array =>T(n) = 1
average-case: element between 1, n =>T(n) = n/2
E.G.M. Petrakis

Algorithm Analysis

35

Comparison
The best-case is not representative
happens only rarely
useful when it has high probability

The worst-case is very useful:

the algorithm performs at least that well

very important for real-time applications

The average-case represents the typical

behavior of the algorithm

not always possible

knowledge about data distribution is necessary

E.G.M. Petrakis

Algorithm Analysis

36