EE 232 Data Structures

Session-05 , Spring-07

Chapter 7
Sorting

Copyright © Kashif Javed Sorting 7-1

Chapter 7: Sorting
7.1 Preliminaries
7.2 Insertion Sort
7.3 A Lower Bound for Simple Sorting Algorithms
7.4 Shellsort
7.5 Heapsort
7.6 Mergesort
7.7 Quicksort

Copyright © Kashif Javed Sorting 7-2

Chapter 7: Sorting
Our goals: We will learn that
 There are several easy algorithms to sort in O(N2) ,
such as insertion sort
 There is an algorithm, Shellsort, that is very simple to
code, runs in o(N2) , and is efficient in practice

 There are slightly more complicated O(N logN) sorting

algorithms

 Any general-purpose sorting algorithm requires Ω(N

logN) comparisons

Copyright © Kashif Javed Sorting 7-3

Chapter 7: Sorting
7.1 Preliminaries
7.2 Insertion Sort
7.3 A Lower Bound for Simple Sorting Algorithms
7.4 Shellsort
7.5 Heapsort
7.6 Mergesort
7.7 Quicksort

Copyright © Kashif Javed Sorting 7-4

Preliminaries
 Each algorithm will be passed an array containing the
elements and an integer containing the number of
elements

 We will assume that N, the number of elements

passed to our sorting routines, has already been
checked and is legal

 Data will start at position 0 for all the sorts

Copyright © Kashif Javed Sorting 7-5

Chapter 7: Sorting
7.1 Preliminaries
7.2 Insertion Sort
7.3 A Lower Bound for Simple Sorting Algorithms
7.4 Shellsort
7.5 Heapsort
7.6 Mergesort
7.7 Quicksort

Copyright © Kashif Javed Sorting 7-6

Insertion Sort
 The Algorithm
 Insertion sort consists of N - 1 passes
 For pass P = 1 through N - 1 , insertion sort
ensures that the elements in positions 0 through P
are in sorted order
 Insertion sort makes use of the fact that elements
in positions 0 through P - 1 are already known to
be in sorted order

 In pass P, we move the element in position P left

until its correct place is found among the first P + 1
elements
Copyright © Kashif Javed Sorting 7-7
Insertion Sort…
 Example

Copyright © Kashif Javed Sorting 7-8

Insertion Sort…
 Implementation void
InsertionSort( ElementType A[ ], int N )
{
int j, P;
ElementType Tmp;

/* 1*/ for( P = 1; P < N; P++ )

{
 Analysis /* 2*/ Tmp = A[ P ];
 O(N2) /* 3*/ for( j = P; j > 0 && A[ j - 1 ] > Tmp; j-- )
/* 4*/ A[ j ] = A[ j - 1 ];
 Presorted, O ? /* 5*/ A[ j ] = Tmp;
}
O(N) }

Copyright © Kashif Javed Sorting 7-9

Chapter 7: Sorting
7.1 Preliminaries
7.2 Insertion Sort
7.3 A Lower Bound for Simple Sorting Algorithms
7.4 Shellsort
7.5 Heapsort
7.6 Mergesort
7.7 Quicksort

Copyright © Kashif Javed Sorting 7-10

A Lower Bound for Simple
Sorting Algorithms
 Inversion
 An ordered pair (i, j) such that i < j but a[i] > a[j]

 Example
• 34,8,64,51,32,21
– (34,8), (34, 32), (34, 21) , (64, 51), (64, 32), (64, 21),
(51, 32), (51, 21), and (32, 21)

 This is exactly the number of swaps that needed

to be performed by insertion sort
 For a sorted array, number of inversions ?

Copyright © Kashif Javed Sorting 7-11

A Lower Bound for Simple
Sorting Algorithms…
 Theorem 1
 The average number of inversions in an array of N
distinct numbers is N(N-1)/4

 List L, Reverse list Lr

 Consider any pair of two numbers in the list (x, y) ,
with y > x
 Clearly, in exactly one of L and Lr , this ordered pair
represents an inversion
 Total number of these pairs in a L and Lr , N(N-1)/2
 An average list has N(N-1)/4

Copyright © Kashif Javed Sorting 7-12

A Lower Bound for Simple
Sorting Algorithms…
 Theorem 2
 Any algorithm that sorts by exchanging adjacent
elements requires Ω(N²) average time

 The average number of inversions is N(N-1)/4 =

Ω(N²)

Copyright © Kashif Javed Sorting 7-13

Chapter 7: Sorting
7.1 Preliminaries
7.2 Insertion Sort
7.3 A Lower Bound for Simple Sorting Algorithms
7.4 Shellsort
7.5 Heapsort
7.6 Mergesort
7.7 Quicksort

Copyright © Kashif Javed Sorting 7-14

Shellsort
 Algorithm
 It compares elements that are distant; the distance
between comparisons decreases as the algorithm
runs until the last phase, in which adjacent
elements are compared
 Also called Diminishing increment sort
 It uses a sequence, h1, h2,…ht , called the
increment sequence
 h1 = 1

Copyright © Kashif Javed Sorting 7-15

Shellsort…
 Algorithm…
 After a phase, using some increment hk, for every
i, A[i]  A[i+hk] ; all elements spaced hk apart are
sorted

 hk – sorted

Copyright © Kashif Javed Sorting 7-16

Shellsort…
void
Shellsort( ElementType A[ ], int N )
{
int i, j, Increment;
ElementType Tmp;
/* 1*/ for( Increment = N / 2; Increment > 0; Increment /= 2 )
/* 2*/ for( i = Increment; i < N; i++ )
{
/* 3*/ Tmp = A[ i ];
/* 4*/ for( j = i; j >= Increment; j -= Increment )
/* 5*/ if( Tmp < A[ j - Increment ] )
/* 6*/ A[ j ] = A[ j - Increment ];
else
/* 7*/ break;
/* 8*/ A[ j ] = Tmp;
}
}
Copyright © Kashif Javed Sorting 7-17
Shellsort…
 Example
81, 94, 11, 96, 12, 35, 17, 95, 28, 58, 41, 75, 15

Copyright © Kashif Javed Sorting 7-18

Shellsort…
 Worst-case Analysis

 Theorem 1
• The worst case running time of Shellsort, using
Shell’s increments, is Q(N2)

 Theorem 2
• The worst case running time of Shellsort, using
Hibbard’s increments, is Q(N3/2)
– 1, 3, 7, …, 2k -1

Copyright © Kashif Javed Sorting 7-19

Chapter 7: Sorting
7.1 Preliminaries
7.2 Insertion Sort
7.3 A Lower Bound for Simple Sorting Algorithms
7.4 Shellsort
7.5 Heapsort
7.6 Mergesort
7.7 Quicksort

Copyright © Kashif Javed Sorting 7-20

Heapsort
 Priority queues can be used to sort in O(NlogN) time
 The algorithm based on this idea is known as
Heapsort

 Basic Strategy
• Build a binary heap of N elements
• Perform N DeletMin operations
• Record these elements in a second array and
then copy them back to the first array
• Since each DeletMin takes O(logN) time, so
total running time is O(NlogN)

Copyright © Kashif Javed Sorting 7-21

Heapsort…
 *Example
 We have studied how to convert the unsorted
array

into a max-heap:

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-22

Heapsort…
 *Perform the DeleteMax operation

 This leaves a gap at the back of the array:

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-23

Heapsort…
 * This is the last entry in the array, so why not fill it
with the largest element?

 Repeat this process: Perform the DeleteMax operation

and then insert it at the end of the array:

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-24

Heapsort…
 * Repeat this process
 DeleteMax and append 70

 DeleteMax and append 63

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-25

Heapsort…
 * We have the 4 largest elements in order
 DeleteMax and append 52

 DeleteMax and append 46

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-26

Heapsort…
 * Continuing...
 DeleteMax and append 34

 DeleteMax and append 28

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-27

Heapsort…
 * Finally, we can dequeue 17, insert it into the 2nd
location, and the resulting array is sorted

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-28

Chapter 7: Sorting
7.1 Preliminaries
7.2 Insertion Sort
7.3 A Lower Bound for Simple Sorting Algorithms
7.4 Shellsort
7.5 Heapsort
7.6 Mergesort
7.7 Quicksort

Copyright © Kashif Javed Sorting 7-29

Mergesort
Algorithm
 Based on divide-and-conquer strategy

 Divide the list into two smaller lists of about equal

sizes
 Sort each sub list recursively
 Merge the two sorted lists to get one sorted list

Copyright © Kashif Javed Sorting 7-30

Mergesort…
 *Merging Algorithm
 Consider the two sorted arrays and an empty array
 Define three indices at the start of each array

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-31

Mergesort…
 *Merging Algorithm…
 We compare 2 and 3: 2 < 3
 Copy 2 down
 Increment the corresponding indices

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-32

Mergesort…
 *Merging Algorithm…
 We compare 3 and 7
 Copy 3 down
 Increment the corresponding indices

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-33

Mergesort…
 *Merging Algorithm…
 We compare 5 and 7
 Copy 5 down
 Increment the appropriate indices

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-34

Mergesort…
 *Merging Algorithm…
 We compare 18 and 7
 Copy 7 down
 Increment...

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-35

Mergesort…
 *Merging Algorithm…
 We compare 18 and 12
 Copy 12 down
 Increment...

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-36

Mergesort…
 *Merging Algorithm…
 We compare 18 and 16
 Copy 16 down
 Increment...

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-37

Mergesort…
 *Merging Algorithm…
 We compare 18 and 33
 Copy 18 down
 Increment...

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-38

Mergesort…
 *Merging Algorithm…
 We compare 21 and 33
 Copy 21 down
 Increment...

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-39

Mergesort…
 *Merging Algorithm…
 We compare 24 and 33
 Copy 24 down
 Increment...

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-40

Mergesort…
 *Merging Algorithm…
 We would continue until we have
passed beyond the limit of one of
the two arrays

 After this, we simply

copy over all remaining
entries in the non-
empty array

 Running for the merging

algorithm is O(N)

Copyright © Kashif Javed *http://www.ece.uwaterloo.ca/~ece250/ Sorting 7-41

Mergesort…

Copyright © Kashif Javed Sorting 7-42

Mergesort…
 Analysis
 Let T(N) be running time of Mergesort to sort N
numbers

• Divide phase: O(1)

• Conquer phase: 2 T(N/2)
• Merging Algorithm: O(N)

 Recurrence equation
T(1) = 1
T(N) = 2T(N/2) + N

Copyright © Kashif Javed Sorting 7-43

Mergesort…
N
T ( N )  2T ( ) N
2 Since N = 2k, we have k = log2 N
N N
 2( 2T ( )  )  N N
4 2 T ( N )  2 T ( k )  kN
k

N 2
 4T ( )  2 N
4  N  N log N
N N
 4( 2T ( )  )  2 N  O( N log N )
8 4
N
 8T ( )  3 N  
8
N
 2 T ( k )  kN
k

2
Copyright © Kashif Javed Sorting 7-44
Chapter 7: Sorting
7.1 Preliminaries
7.2 Insertion Sort
7.3 A Lower Bound for Simple Sorting Algorithms
7.4 Shellsort
7.5 Heapsort
7.6 Mergesort
7.7 Quicksort

Copyright © Kashif Javed Sorting 7-45

Quicksort
 Divide and conquer approach

 Average case
• O(N log N)
 Worst case
• O(N2)
• But, the worst case seldom happens

Copyright © Kashif Javed Sorting 7-46

Quicksort…
 Basic Algorithm
 Given an array S to be sorted
1. If size of S < 1 then return;
2. Pick any element v in S as the pivot
3. Partition S-{v} (remaining elements in S) into two
groups
1. S1 = {x Є S-{v}| x  v}
2. S2 = {x Є S-{v} | x ≥ v}
4. Return {quicksort(S1) followed by v followed by
quicksort(S2) }

Copyright © Kashif Javed Sorting 7-47

Quicksort…
81 43 31 57 75
 Illustration 13 92 65 26 0

Select pivot

81 43 31 57 75
13 65 26 0
92

partition
13 31 57 81
26 0 65 92 75
43

Recursive call Recursive call

0 13 26 31 43 57 75 81 92

Merge

0 13 26 31 43 57 65 75 81 92

Copyright © Kashif Javed Sorting 7-48

Quicksort…
 Picking the Pivot
 A wrong way
• Use the first element as the pivot
• if the array is presorted or in reverse order, all elements
go to S1 or S2
• Quicksort will take N2 time to do nothing

 A safe Maneuver
• choose a random pivot
• but random number generation is expensive

Copyright © Kashif Javed Sorting 7-49

Quicksort…
 Median-of-three Partitioning
• Ideally, the pivot should be the median of input array S
– Median = element in the middle of the sorted
sequence

• Unfortunately, we cannot calculate median without

sorting first

• So find the approximate median

– Pivot = median of the left-most, right-most and center
element of the array S

Copyright © Kashif Javed Sorting 7-50

Quicksort…
 Example
 Let S = {6, 1, 4, 9, 0, 3, 5, 2, 7, 8}
 Left = 0 and S[left] = 6
 Right = 9 and S[right] = 8
 center = (left+right)/2 = 4 and S[center] = 0

 Pivot
• = Median of S[left], S[right], and S[center]
• = median of 6, 8, and 0
• = S[left] = 6

Copyright © Kashif Javed Sorting 7-51

Quicksort…
 Original input : S = {6, 1, 4, 9, 0, 3, 5, 2, 7, 8}

 Get the pivot out of the way by swapping it with the

last element

8 1 4 9 0 3 5 2 7 6
pivot
 Have two ‘indices’ – i and j
 i starts at first element and moves forward
 j starts at next-to-last element and moves backwards

8 1 4 9 0 3 5 2 7 6
i j pivot
Copyright © Kashif Javed Sorting 7-52
Quicksort…
 While (i < j)
 Move i to the right till we find a number greater
than pivot
 Move j to the left till we find a number smaller than
pivot
 If (i < j) swap(S[i], S[j])
 (The effect is to push larger elements to the right
and smaller elements to the left)

 Swap the pivot with S[i]

Copyright © Kashif Javed Sorting 7-53

Quicksort…
i j pivot
8 1 4 9 0 3 5 2 7 6
Move i j pivot
8 1 4 9 0 3 5 2 7 6
i j pivot
swap
2 1 4 9 0 3 5 8 7 6
i j pivot
move
2 1 4 9 0 3 5 8 7 6
i j pivot
swap
2 1 4 5 0 3 9 8 7 6
move j i pivot i and j
2 1 4 5 0 3 9 8 7 6 have crossed
Swap S[i]
2 1 4 5 0 3 6 8 7 9
with pivot
j i
Copyright © Kashif Javed pivot Sorting 7-54
Quicksort…
 For small arrays (N ≤ 20),
 Insertion sort is faster than Quicksort

Copyright © Kashif Javed Sorting 7-55

 THE END

For further discussions, and questions, email me :

kashifuet@gmail.com

Copyright © Kashif Javed Sorting 7-56