DataStructuresandAlgorithms

Week2

1. Complexity of algorithms 2. Correctness of algorithms 3. Asymptotic analysis 4. Special case analysis

AnalysisofAlgorithms

Efficiency:

Runningtime Spaceused

Efficiencyisdefinedasafunctionofthe inputsize:

DSA09

Numberofdataelements(numbers, points). Thenumberofbitsofaninputnumber.

M.Bhlen 2

TheRAMmodel

Itisimportanttochoosethelevelofdetail. TheRAMmodel:

Instructions(eachtakingconstanttime)we usuallychooseonetypeofinstructionasa characteristicoperationthatiscounted:

Arithmetic(add,subtract,multiply,etc.) Datamovement(assign) Controlflow(branch,subroutinecall,return) Comparison

Datatypesintegers,characters,andfloats
M.Bhlen 3

DSA09

AnalysisofInsertionSort/1

Timetocomputetherunningtimeasa functionoftheinputsize(exactanalysis).
cost

for j := 2 to n do key := A[j] // Insert A[j] into A[1..j-1] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-A[i+1]:= key
DSA09 M.Bhlen

c1 c2 0 c3 c4 c5 c6 c7

times

n n-1 n-1 n-1

n

j= 2 t j n j= 2 t j1 n j= 2 t j1

n-1
4

AnalysisofInsertionSort/2

Therunningtimeofanalgorithmisthe sumoftherunningtimesofeachstate ment. Astatementwithcostcthatisexecuted ntimescontributesc*ntotherunning time. ThetotalrunningtimeT(n)ofinsertion sortis

n n n j=2 j=2 j j=2 j

t T(n)=c1*n+c2(n1)+c3(n1)+c4 t 1 t 1 c5+c6+c7(n1)
j

DSA09

M.Bhlen

AnalysisofInsertionSort/3

Oftentheperformancedependsonthe detailsoftheinput(notonlylengthn). Thisismodeledbytj. Inthecaseofinsertionsortthetimetj dependsontheoriginalsortingofthe inputarray.

DSA09

M.Bhlen

PerformanceAnalysis

Oftenitissufficienttocountthenumberof iterationsofthecore(innermost)part.

Nodistinctionbetweencomparisons,assignments, etc(thatmeansroughlythesamecostforallof them). Givespreciseenoughresults.

Insomecasesthecostofselectedoperations dominatesallothercosts.

DiskI/OversusRAMoperations. Databasesystems.
M.Bhlen 7

DSA09

Best/Worst/AverageCase/1

Analyzinginsertionsorts

n j=2

t j 1

Bestcase:elementsalreadysorted,tj=1, innermostloopiszero,totalrunningtimeis linear(time=an+b). Worstcase:elementssortedininverse order,tj=j,totalrunningtimeisquadratic (time=an2+bn+c). Averagecase:tj=j/2,totalrunningtimeis quadratic(time=an2+bn+c).

M.Bhlen 8

DSA09

Best/Worst/AverageCase/2

Foraspecificsizeofinputsizen,investigate runningtimesfordifferentinputinstances:

6n

running time

5n 4n 3n 2n 1n

}
A B C D E F input instance G
M.Bhlen

worst case

=4321

average case = 3 1 2 4 =1234

best case

DSA09

Best/Worst/AverageCase/3

Forinputsofallsizes:
worst-case
6n

average-case best-case

Running time

5n 4n 3n 2n 1n 1 2 3 4 5 6 7 8 9 10 11 12 ..
10

Input instance size

DSA09 M.Bhlen

Best/Worst/AverageCase/4

Worstcaseisusuallyused:

DSA09

Itisanupperbound. Incertainapplicationdomains(e.g.,airtraffic control,surgery)knowingtheworstcase timecomplexityisofcrucialimportance. Forsomealgorithmsworstcaseoccursfairly often. Theaveragecaseisoftenasbadastheworst case. Findingtheaveragecasecanbeverydifficult.

M.Bhlen 11

AnalysisofLinearSearch
INPUT: A[1..n] a sorted array of integers, q an integer. OUTPUT: j s.t. A[j]=q. NIL if j(1jn): A[j]q

10

j := 1 while j <= n and A[j] != q do j++ if j <= n then return j else return NIL

Worst case running time: n Average case running time: n/2 Best case running time: 0
M.Bhlen 12

DSA09

BinarySearch
411

Idea:Leftandrightbound.Elementstotheright ofrarebiggerthansearchelement,... Ineachstephalftherangeofthesearchspace.

INPUT: A[1..n] sorted (increasing) array of integers, q integer. OUTPUT: an index j such that A[j] = q. NIL, if j (1 j n): A[j] q

DSA09

l := 1; r := n do m := (l+r)/2 if A[m] = q then return m else if A[m] > q then r := m-1 else l := m+1 while l <= r return NIL
M.Bhlen

13

AnalysisofBinarySearch

Howmanytimesistheloopexecuted?

Witheachexecutionthedifference betweenlandriscutinhalf.

Initiallythedifferenceisn. Theloopstopswhenthedifferencebecomes0 (lessthan1).

DSA09

Howmanytimesdoyouhavetocutnin halftoget0? lognbetterthanthebruteforce approachoflinearsearch(n).

M.Bhlen 14

LinearvsBinarySearch

Costsoflinearsearch:n Costsofbinarysearch:log(n) Shouldwecare? Phonebookwithnentries:

n=200000,logn=log200000=18 n=2M,log2M=21 n=20M,log20M=24

M.Bhlen 15

DSA09

CorrectnessofAlgorithms

Analgorithmiscorrectifforanylegalinput itterminatesandproducesthedesired output. Automaticproofofcorrectnessisnot possible. Therearepracticaltechniquesandrigorous formalismsthathelptoreasonaboutthe correctnessof(partsof)algorithms.

M.Bhlen 16

DSA09

PartialandTotalCorrectness

Partialcorrectness
IFthispointisreached, THENthisisthedesiredoutput

Anylegalinput

Algorithm

Output

Totalcorrectness
INDEEDthispointisreached, ANDthisisthedesiredoutput

Anylegalinput

Algorithm

Output

DSA09

M.Bhlen

17

Assertions

Toprovepartialcorrectnessweassociatea numberofassertions(statementsaboutthe stateoftheexecution)withspecific checkpointsinthealgorithm.

E.g.,A[1],,A[j]formanincreasingsequence

Preconditionsassertionsthatmustbevalid beforetheexecutionofanalgorithmora subroutine(INPUT). Postconditionsassertionsthatmustbevalid aftertheexecutionofanalgorithmora subroutine(OUTPUT).

M.Bhlen 18

DSA09

Pre/postconditions

Example:

Writeapseudocodealgorithmtofindthetwo smallestnumbersinasequenceofnumbers (givenasanarray).

INPUT:anarrayofintegersA[1..n],n>0 OUTPUT:(m1,m2)suchthateither

m1A<m2Aand foreachi[1..n]:m1 A[i]and,ifA[i] m1, thenm2 A[i],or m2=m1=A[1]ifj,i[1..n]:A[i]=A[j].

M.Bhlen 19

DSA09

LoopInvariants

Invariants:assertionsthatarevalidanytime theyarereached(manytimesduringthe executionofanalgorithm,e.g.,inloops) Wemustshowthreethingsaboutloop invariants:

Initialization:itistruepriortothefirstiteration. Maintenance:ifitistruebeforeaniteration,then itistrueaftertheiteration. Termination:whenaloopterminatesthe invariantgivesausefulpropertytoshowthe correctnessofthealgorithm

M.Bhlen 20

DSA09

Example:BinarySearch/1

Wewanttoshowthatqis l := 1; r := n; notinAifNILisreturned. do m := (l+r)/2 Invariant: if A[m]=q then return m i[1..l1]:A[i]<q(Ia) else if A[m]>q then r := else l := m+1 i[r+1..n]:A[i]>q(Ib) while l <= r Initialization:l=1,r=n return NIL theinvariantholdsbecause therearenoelementstotheleftoflortotherightofr. l=1yieldsj,i[1..0]:A[i]<q thisholdsbecause[1..0]isempty r=nyieldsj,i[n+1..n]:A[i]>q thisholdsbecause[n+1..n]isempty
DSA09 M.Bhlen

m-1

21

Example:BinarySearch/2

Invariant: i[1..l1]:A[i]<q(Ia) i[r+1..n]:A[i]>q(Ib)

l := 1; r := n; do m := (l+r)/2 if A[m]=q then return m else if A[m]>q then r := m-1 else l := m+1 while l <= r return NIL

Maintenance:l,r,m=(l+r)/2 A[m]!=q&A[m]>q,r=m1,Asortedimplies k[r+1..n]:A[k]>q(Ib) A[m]!=q&A[m]<q,l=m+1,Asortedimplies k[1..l1]:A[k]<q(Ia)

M.Bhlen 22

DSA09

Example:BinarySearch/3

116

Invariant: i[1..l1]:A[i]<q(Ia) i[r+1..n]:A[i]>q(Ib) Termination:l,r,l<=r Twocases:

l := 1; r := n; do m := (l+r)/2 if A[m]=q then return m else if A[m]>q then r := m-1 else l := m+1 while l <= r return NIL

l:=m+1weget(l+r)/2+1>l r:=m1weget(l+r)/21<r

Therangegetssmallerduringeachiterationandtheloop willterminatewhenl<=rnolongerholds.
M.Bhlen 23

DSA09

Example:InsertionSort/1

Invariant: outsidewhileloop

A[1...j1]issorted A[1...j1]Aorig

insidewhileloop:

for j := 2 to n do key := A[j] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-A[i+1] := key

A[1...i],key,A[i+1j] A[1...i]issorted A[i+1j]issorted A[k]>key,i+1<=k<=j

M.Bhlen 24

DSA09

Example:InsertionSort/2

outsidewhileloop

insidewhileloop:

A[1...j1]issorted A[1...j1]Aorig A[1...i],key,A[i+1j] A[1...i]issorted A[i+1j]issorted A[k]>key,i+1<=k<=j

for j := 2 to n do key := A[j] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-A[i+1] := key

Initialization:

j=2:theinvariantholds,A[11]istriviallysorted. i=j1:A[1...j1],key,A[jj]wherekey=A[j] A[1j1]issorted(invariantofouterloop) A[jj]issorted

M.Bhlen 25

DSA09

Example:InsertionSort/3

outsidewhileloop

insidewhileloop:

A[1j1]issorted A[1...j1]Aorig A[1...i],key,A[i+1j] A[1...i]issorted A[i+1j]issorted A[k]>key,i+1<=k<=j

for j := 2 to n do key := A[j] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-A[i+1] := key

Maintenance:

A[1j1]sorted+insertA[j]=>A[1j]sorted A[1...i],key,A[i+1j]satisfiesconditionsbecauseof conditionA[i]>keyandA[1...j1]beingsorted

M.Bhlen 26

DSA09

Example:InsertionSort/4

outsidewhileloop

insidewhileloop:

A[1j1]issorted A[1...j1]Aorig A[1...i],key,A[i+1j] A[1...i]issorted A[i+1j]issorted A[k]>key,i+1<=k<=j

for j := 2 to n do key := A[j] i := j-1 while i>0 and A[i]>key do A[i+1] := A[i] i-A[i+1] := key

Termination:

mainloop,j=n+1:A[1n]sorted. A[i]key:(A[1...i],key,A[i+1j])issorted i=0:(key,A[1j])issorted.

M.Bhlen 27

DSA09

AsymptoticAnalysis

Goal:tosimplifytheanalysisoftherunning timebygettingridofdetails,whichare affectedbyspecificimplementationand hardware

roundingofnumbers:1,000,001 1,000,000 roundingoffunctions:3n2 n2

Capturingtheessence:howtherunningtime ofanalgorithmincreaseswiththesizeofthe inputinthelimit.

Asymptoticallymoreefficientalgorithmsarebest forallbutsmallinputs
M.Bhlen 28

DSA09

AsymptoticNotation/1

ThebigOh ONotation

Usedforworstcase analysis

asymptoticupperbound f(n)=O(g(n))iffthere existsconstantsc>0and n0>0,s.t.f(n) cg(n) forn n0 f(n)andg(n)are functionsovernon negativeintegers

c g ( n)
Running Time

f n

n0

Input Size

DSA09

M.Bhlen

29

AsymptoticNotation/2

ThebigOmega Notation

Running Time

Usedtodescribebestcase runningtimesorlower boundsofalgorithmic problems.

asymptoticlowerbound f(n)= (g(n))iffthereexists constantsc>0andn0>0,s.t. cg(n) f(n)forn n0

f n

c g ( n)

n0

E.g.,lowerboundof searchinginanunsorted arrayis (n).

Input Size

DSA09

M.Bhlen

30

AsymptoticNotation/3

SimpleRule:Droplowerorderterms andconstantfactors.

50nlognisO(nlogn) 7n3isO(n) 8n2logn+5n2+nisO(n2logn)

Note:Although(50nlogn)isO(n5),it isexpectedthatanapproximationisof thesmallestpossibleorder.

M.Bhlen 31

DSA09

AsymptoticNotation/4

ThebigTheta Notation

asymptoticlytightbound f(n)= (g(n))ifthereexists constantsc1>0,c2>0,and n0>0,s.t.forn n0 c1g(n) f(n) c2g(n)

c 2g n

Running Time

f n
c1 n g

f(n)= (g(n))iff f(n)= (g(n))andf(n)= (g(n)) O(f(n))isoftenabusedinsteadof (f(n))

M.Bhlen

n0

Input Size

DSA09

32

AsymptoticNotation/5

Twomoreasymptoticnotations

"LittleOh"notationf(n)=o(g(n)) nontightanalogueofBigOh

Foreveryc>0,thereexistsn0>0,s.t. f(n)< cg(n)forn n0 Iff(n)=o(g(n)),itissaidthatg(n)dominates f(n).

"Littleomega"notationf(n)= (g(n)) nontightanalogueofBigOmega

M.Bhlen 33

DSA09

AsymptoticNotation/6

10

Analogywithrealnumbers

f(n)=O(g(n)) f(n)= (g(n)) f(n)= (g(n)) f(n)=o(g(n)) f(n)= (g(n))

f f f= f< f>

g g g g g

Abuseofnotation:f(n)=O(g(n)) actuallymeansf(n) O(g(n))

M.Bhlen 34

DSA09

ComparisonofRunningTimes

Determiningthemaximalproblemsize.
RunningTime 1second T(n)ins 400n 20nlogn 2n2 n4 2n 2500 4096 707 31 19 1minute 150000 166666 5477 88 25
M.Bhlen

1hour 9000000 7826087 42426 244 31

35

DSA09

SpecialCaseAnalysis

Howdoweproceedifwehavetocheck thecorrectnessofaprogram? Considerextremecasesandmakesure oursolutionworksinallcases. Theproblemisthenreducedto identifyingspecialcases. ThisisrelatedtoINPUTandOUTPUT specifications.

M.Bhlen 36

DSA09

SpecialCases

emptydatastructure (array,file,list,) singleelementdata structure completelyfilleddata structure enteringafunction terminationofa function

M.Bhlen

zero,emptystring negativenumber borderofdomain

startofloop endofloop firstiterationof loop

37

DSA09

Sortedness/1

Thefollowingalgorithmcheckswhether anarrayissorted.

INPUT: A[1..n] an array integers. OUTPUT: TRUE if A is sorted; FALSE otherwise for i := 1 to n if A[i] A[i+1] then return FALSE return TRUE

Analyzethealgorithmbyconsidering specialcases.
M.Bhlen 38

DSA09

Sortedness/2
INPUT: A[1..n] an array integers. OUTPUT: TRUE if A is sorted; FALSE otherwise

12

for i := 1 to n-1 if A[i] > A[i+1] then return FALSE return TRUE

Startofloop,i=1OK Endofloop,i=n1OK Firstiteration,fromi=1toi=2OK A=[1,1,1]OK Emptydatastructure,n=0?(forloop) A=[1,0,1,3]OK A=[3,2,1]?(ascending,descending)

M.Bhlen 39

DSA09

BinarySearch,Variant1

Analyzethefollowingalgorithmbyconsidering specialcases.
l := 1; r := n do m := (l+r)/2 if A[m] = q then return m else if A[m] > q then r := m-1 else l := m+1 while l < r return NIL

DSA09

M.Bhlen

40

BinarySearch,Variant2

Analyzethefollowingalgorithmbyconsidering specialcases.
l := 1; r := n while l < r do { m := (l+r)/2 if A[m] <= q then l := m+1 else r := m } if A[l-1] = q then return q else return NIL

DSA09

M.Bhlen

41

BinarySearch,Variant3

Analyzethefollowingalgorithmbyconsidering specialcases.
l := 1; r := n while l <= r do m := (l+r)/2 if A[m] <= q then l := m+1 else r := m if A[l-1] = q then return q else return NIL

DSA09

M.Bhlen

42

AQuickMathRefresher

Arithmeticprogression
n

n(1 + n) i = 1 + 2 + 3 + ... + n = 2 i= 0

Geometricprogression

givenanintegern0andarealnumber0<a 1
1 an +1 ai = 1 + a + a 2 + ... + a n = 1 a i= 0

geometricprogressionsexhibitexponential growth
M.Bhlen 43

DSA09

Miscellaneous/1

Manipulatingsummations:

c a j=c a j j j

j a j b j = j a j j b j

DSA09

M.Bhlen

44

Miscellaneous/2

Manipulatinglogarithmsandpowers:

a

logb=logb/loga

logab=bloga log(ab)=loga+logb a =(a )

a logb=b
M.Bhlen 45
a

mn

m n

aman=am+n

DSA09

Summations

Therunningtimeofinsertionsortis determinedbyanestedloop.
for j := 2 to n key := A[j] i := j-1 while i>0 and A[i]>key A[i+1] := A[i] i := i-1 A[i+1] := key

Nestedloopscorrespondto n summations: j=2 j1

M.Bhlen 46

DSA09

ProofbyInduction/1

WewanttoshowthatpropertyPistrueforall integersn n0. Basis:provethatPistrueforn0. Inductivestep:provethatifPistrueforallk suchthatn0 k n1thenPisalsotrueforn.

n( n + 1) Example S ( n) = i = for n 1 2 i =0
n

Basis

1(1 + 1) S (1) = i = 2 i= 0
1

DSA09

M.Bhlen

47

ProofbyInduction/2

InductiveStep
S ( k ) = i =
i =0 n k

k ( k +1) for 1 k n 1 2
n 1

S ( n) = i =i + n = (n 1) + n = S
i =0 i =0

( n 1 +1) ( n 2 n + 2 n) = (n 1) +n = = 2 2 n( n +1) = 2
DSA09 M.Bhlen 48

Summary

Algorithmiccomplexity Correctnessofalgorithms

Pre/Postconditions Invariants BigOnotation Growthoffunctionsandasymptoticnotation

M.Bhlen 49

Asymptoticanalysis

DSA09

Specialcaseanalysis

NextWeek

Divideandconquer Mergesort Writingrecurrencestoanalyzethe runningtimeofrecursivealgorithms.

DSA09

M.Bhlen

50