Introduction "Without loss of generality, may all your theorems be proved." - Rashid.

"If thy theorems fail deduction reasoning thou shalt apply induction reasoning with the most meticulous care." - Rashid.

An algorithm, named after the ninth century scholar Abu Jafar Muhammad Ibn Musu Al-Khowarizmi, is defined as follows: Roughly speaking:

An algorithm is a set of rules for carrying out calculation either by hand or on a machine. An algorithm is a finite step-by-step procedure to achieve a required result. An algorithm is a sequence of computational steps that transform the input into the output. An algorithm is a sequence of operations performed on data that have to be organized in data structures. An algorithm is an abstraction of a program to be executed on a physical machine (model of Computation).

The most famous algorithm in history dates well before the time of the ancient Greeks: this is the Euclid's algorithm for calculating the greatest common divisor of two integers. This theorem appeared as the solution to the Proposition II in the Book VII of Euclid's "Elements." Euclid's "Elements" consists

of thirteen books, which contain a total number of 465 propositions.

The Classic Multiplication Algorithm 1. Multiplication, the American way: Multiply the multiplicand one after another by each digit of the multiplier taken from right to left.

2. Multiplication, the English way: Multiply the multiplicand one after another by each digit of the multiplier taken from left to right.

Algorithmic is a branch of computer science that consists of designing and analyzing computer algorithms 1. The design pertain to i. The description of algorithm at an abstract level by means of a pseudo language, and ii. Proof of correctness that is, the algorithm solves the given problem in all cases. 2. The analysis deals with performance evaluation (complexity analysis).

We start with defining the model of computation, which is usually the Random Access Machine (RAM) model, but other models of computations can be use such as PRAM. Once the model of computation has been defined, an algorithm can be describe using a simple language (or pseudo language) whose syntax is close to programming language such as C or java.

Algorithm's Performance Two important ways to characterize the effectiveness of an algorithm are its space complexity and time complexity. Time complexity of an algorithm concerns determining an expression of the number of steps needed as a function of the problem size. Since the step count measure is somewhat coarse, one does not aim at obtaining an exact step count. Instead, one attempts only to get asymptotic bounds on the step count. Asymptotic analysis

makes use of the O (Big Oh) notation. Two other notational constructs used by computer scientists in the analysis of algorithms are (Big Theta) notation and (Big Omega) notation. The performance evaluation of an algorithm is obtained by totaling the number of occurrences of each operation when running the algorithm. The performance of an algorithm is evaluated as a function of the input size n and is to be considered modulo a multiplicative constant.

The following notations are commonly use notations in performance analysis and used to characterize the complexity of an algorithm. -Notation (Same order) This notation bounds a function to within constant factors. We say f(n) = (g(n)) if there exist positive constants n0, c1 and c2 such that to the right of n0 the value of f(n) always lies between c1 g(n) and c2 g(n) inclusive. In the set notation, we write as follows: (g(n)) = {f(n) : there exist positive constants c1, c1, and n0 such that 0 c1 g(n) f(n) c2 g(n) for all n n0} We say that is g(n) an asymptotically tight bound for f(n).

Graphically, for all values of n to the right of n0, the value of f(n) lies at or above c1 g(n) and at or below c2 g(n). In other words, for all n n0, the function f(n) is equal to g(n) to within a constant factor. We say that g(n) is an asymptotically tight bound for f(n). In the set terminology, f(n) is said to be a member of the set (g(n)) of functions. In other words, because O(g(n)) is a set, we could write f(n) (g(n)) to indicate that f(n) is a member of (g(n)). Instead, we write f(n) = (g(n)) to express the same notation. Historically, this notation is "f(n) = (g(n))" although the idea that f(n) is equal to something called (g(n)) is misleading. Example: n2/2 2n = (n2), with c1 = 1/4, c2 = 1/2, and n0 = 8.

-Notation (Upper Bound)

This notation gives an upper bound for a function to within a constant factor. We write f(n) = O(g(n)) if there are positive constants n0 and c such that to the right of n0, the value of f(n) always lies on or below c g(n). In the set notation, we write as follows: For a given function g(n), the set of functions (g(n)) = {f(n): there exist positive constants c and n0 such that 0 f(n) c g(n) for all n n0} We say that the function g(n) is an asymptotic upper bound for the function f(n). We use -notation to give an upper bound on a function, to within a constant factor.

Graphically, for all values of n to the right of n0, the value of the function f(n) is on or below g(n). We write f(n) = O(g(n)) to indicate that a function f(n) is a member of the set (g(n)) i.e. f(n) (g(n)) Note that f(n) = (g(n)) implies f(n) = (g(n)), since -notation is a stronger notation than -notation. Example: 2n2 = (n3), with c = 1 and n0 = 2.

Equivalently, we may also define f is of order g as follows: If f(n) and g(n) are functions defined on the positive integers, then f(n) is (g(n)) if and only if there is a c > 0 and an n0 > 0 such that | f(n) | | g(n) | for all n n0

Historical Note: The notation was introduced in 1892 by the German mathematician Paul Bachman.

-Notation (Lower Bound) This notation gives a lower bound for a function to within a constant factor. We write f(n) = (g(n)) if there are positive constants n0 and c such that to the right of n0, the value of f(n) always lies on or above c g(n). In the set notation, we write as follows: For a given function g(n), the set of functions (g(n)) = {f(n) : there exist positive constants c and n0 such that 0 c g(n) f(n) for all n n0} We say that the function g(n) is an asymptotic lower bound for the function f(n).

The intuition behind -notation is shown above. Example: n = (lg n), with c = 1 and n0 = 16.

Algorithm Analysis The complexity of an algorithm is a function g(n) that gives the upper bound of the number of operation (or running time) performed by an algorithm when the input size is n. There are two interpretations of upper bound. Worst-case Complexity The running time for any given size input will be lower than the upper bound except possibly for some values of the input where the maximum is reached. Average-case Complexity The running time for any given size input will be the average number of operations over all problem instances for a given size.

Because, it is quite difficult to estimate the statistical behavior of the input, most of the time we content ourselves to a worst case behavior. Most of the time, the complexity of g(n) is approximated by its family o(f(n)) where f(n) is one of the following functions. n (linear complexity), log n (logarithmic complexity), na where a 2 (polynomial complexity), an (exponential complexity).

Optimality Once the complexity of an algorithm has been estimated, the question arises whether this algorithm is optimal. An algorithm for a given problem is optimal if its complexity reaches the lower bound over all the algorithms solving this problem. For example, any algorithm solving the intersection of n segments problem will execute at least n2 operations in the worst case even if it does nothing but print the output. This is abbreviated by saying that the problem has (n2) complexity. If one finds an O(n2) algorithm that solve this problem, it will be optimal and of complexity (n2).

Reduction Another technique for estimating the complexity of a problem is the transformation of problems, also called problem reduction. As an example, suppose we know a lower bound for a problem A, and that we would like to estimate a lower bound for a problem B. If we can transform A into B by a transformation

step whose cost is less than that for solving A, then B has the same bound as A. The Convex hull problem nicely illustrates "reduction" technique. A lower bound of Convex-hull problem established by reducing the sorting problem (complexity: (n log n)) to the Convex hull problem.

Classifications of algorithms: Greedy Introduction

Greedy algorithms are simple and straightforward. They are shortsighted in their approach in the sense that they take decisions on the basis of information at hand without worrying about the effect these decisions may have in the future. They are easy to invent, easy to implement and most of the time quite efficient. Many problems cannot be solved correctly by greedy approach. Greedy algorithms are used to solve optimization problems

Greedy Approach Greedy Algorithm works by making the decision that seems most promising at any moment; it never reconsiders this decision, whatever situation may arise later. As an example consider the problem of "Making Change". Coins available are:

dollars (100 cents) quarters (25 cents) dimes (10 cents) nickels (5 cents) pennies (1 cent)

Problem Make a change of a given amount using the smallest possible number of coins.

Informal Algorithm

Start with nothing. at every stage without passing the given amount. o add the largest to the coins already chosen.

Formal Algorithm Make change for n units using the least possible number of coins. MAKE-CHANGE (n) C {100, 25, 10, 5, 1} // constant. Sol {}; // set that will hold the solution set. Sum 0 sum of item in solution set WHILE sum not = n x = largest item in set C such that sum + x n IF no such item THEN RETURN "No Solution" S S {value of x} sum sum + x RETURN S

Example Make a change for 2.89 (289 cents) here n = 2.89 and the solution contains 2 dollars, 3 quarters, 1 dime and 4 pennies. The algorithm is greedy because at every stage it chooses the largest coin without worrying about the consequences. Moreover, it never changes its mind in the sense that once a coin has been included in the solution set, it remains there.

Characteristics and Features of Problems solved by Greedy Algorithms

To construct the solution in an optimal way. Algorithm maintains two sets. One contains chosen items and the other contains rejected items. The greedy algorithm consists of four (4) function.
1.

2. 3.

4.

A function that checks whether chosen set of items provide a solution. A function that checks the feasibility of a set. The selection function tells which of the candidates is the most promising. An objective function, which does not appear explicitly, gives the value of a solution.

Structure Greedy Algorithm

Initially the set of chosen items is empty i.e., solution set. At each step o item will be added in a solution set by using selection function. o IF the set would no longer be feasible reject items under consideration (and is never consider again). o ELSE IF set is still feasible THEN add the current item.

Definitions of feasibility A feasible set (of candidates) is promising if it can be extended to produce not merely a solution, but an optimal solution to the problem. In particular, the empty set is always promising why? (because an optimal solution always exists) Unlike Dynamic Programming, which solves the subproblems bottom-up, a greedy strategy usually progresses in a top-down fashion, making one greedy choice after another, reducing each problem to a smaller one. Greedy-Choice Property The "greedy-choice property" and "optimal substructure" are two ingredients in the problem that lend to a greedy strategy. Greedy-Choice Property It says that a globally optimal solution can be arrived at by making a locally optimal choice. Divide-and-Conquer Algorithm Divide-and-conquer is a top-down technique for designing algorithms that consists of dividing the problem into smaller subproblems hoping that the solutions of the subproblems are easier to find and then composing the partial solutions into the solution of the original problem.

Little more formally, divide-and-conquer paradigm consists of following major phases:

Breaking the problem into several subproblems that are similar to the original problem but smaller in size, Solve the sub-problem recursively (successively and independently), and then Combine these solutions to subproblems to create a solution to the original problem.

Binary Search (simplest application of divide-andconquer) Binary Search is an extremely well-known instance of divideand-conquer paradigm. Given an ordered array of n elements, the basic idea of binary search is that for a given element we "probe" the middle element of the array. We continue in either the lower or upper segment of the array, depending on the outcome of the probe until we reached the required (given) element.

Problem Let A[1 . . . n] be an array of non-decreasing sorted order; that is A [i] A [j] whenever 1 i j n. Let 'q' be the query point. The problem consist of finding 'q' in the array A. If q is not in A, then find the position where 'q' might be inserted.

Formally, find the index i such that 1 i n+1 and A[i-1] < x A[i].

Sequential Search Look sequentially at each element of A until either we reach at the end of an array A or find an item no smaller than 'q'. Sequential search for 'q' in array A for i = 1 to n do if A [i] q then return index i return n + 1

Analysis This algorithm clearly takes a (r), where r is the index returned. This is (n) in the worst case and O(1) in the best case. If the elements of an array A are distinct and query point q is indeed in the array then loop executed (n + 1) / 2 average number of times. On average (as well as the worst case), sequential search takes (n) time.

Binary Search Look for 'q' either in the first half or in the second half of the array A. Compare 'q' to an element in the middle, n/2 , of the array. Let k = n/2 . If q A[k], then search in the A[1 . . . k]; otherwise search T[k+1 . . n] for 'q'. Binary search for q in subarray A[i . . j] with the promise that A[i-1] < x A[j] If i = j then return i (index) k= (i + j)/2 if q A [k] then return Binary Search [A [i-k], q] else return Binary Search [A[k+1 . . j], q]

Analysis Binary Search can be accomplished in logarithmic time in the worst case , i.e., T(n) = (log n). This version of the binary search takes logarithmic time in the best case. Iterative Version of Binary Search Interactive binary search for q, in array A[1 . . n] if q > A [n] then return n + 1 i = 1;

j = n; while i < j do k = (i + j)/2 if q A [k] then j = k else i = k + 1 return i (the index)

Analysis The analysis of Iterative algorithm is identical to that of its recursive counterpart.

Dynamic Programming Algorithms Dynamic programming is a fancy name for using divide-andconquer technique with a table. As compared to divide-andconquer, dynamic programming is more powerful and subtle design technique. Let me repeat , it is not a specific algorithm, but it is a meta-technique (like divide-and-conquer). This technique was developed back in the days when "programming" meant "tabular method" (like linear programming). It does not really refer to computer programming. Here in our advanced algorithm course, we'll also think of "programming" as a "tableau method" and certainly not writing code. Dynamic programming is a stage-wise search method suitable for optimization problems whose solutions may be viewed as the result of a sequence of decisions. The most attractive property of

this strategy is that during the search for a solution it avoids full enumeration by pruning early partial decision solutions that cannot possibly lead to optimal solution. In many practical situations, this strategy hits the optimal solution in a polynomial number of decision steps. However, in the worst case, such a strategy may end up performing full enumeration. Dynamic programming takes advantage of the duplication and arrange to solve each subproblem only once, saving the solution (in table or in a globally accessible place) for later use. The underlying idea of dynamic programming is: avoid calculating the same stuff twice, usually by keeping a table of known results of subproblems. Unlike divide-and-conquer, which solves the subproblems top-down, a dynamic programming is a bottom-up technique. The dynamic programming technique is related to divide-and-conquer, in the sense that it breaks problem down into smaller problems and it solves recursively. However, because of the somewhat different nature of dynamic programming problems, standard divide-and-conquer solutions are not usually efficient. The dynamic programming is among the most powerful for designing algorithms for optimization problem. This is true for two reasons. Firstly, dynamic programming solutions are based on few common elements. Secondly, dynamic programming problems are typical optimization problems i.e., find the minimum or maximum cost solution, subject to various constraints. In other words, this technique used for optimization problems:

Find a solution to the problem with the optimal value. Then perform minimization or maximization. (We'll see example of both in CLRS).

The dynamic programming is a paradigm of algorithm design in which an optimization problem is solved by a combination of caching subproblem solutions and appealing to the "principle of optimality."

There are three basic elements that characterize a dynamic programming algorithm: 1. Substructure Decompose the given problem into smaller (and hopefully simpler) subproblems. Express the solution of the original problem in terms of solutions for smaller problems. Note that unlike divide-and-conquer problems, it is not usually sufficient to consider one decomposition, but many different ones. 2. Table-Structure After solving the subproblems, store the answers (results) to the subproblems in a table. This is done because (typically) subproblem solutions are reused many times, and we do not want to repeatedly solve the same problem over and over again. 3. Bottom-up Computation

Using table (or something), combine solutions of smaller subproblems to solve larger subproblems, and eventually arrive at a solution to the complete problem. The idea of bottom-up computation is as follow: Bottom-up means i. Start with the smallest subproblems. ii. Combining theirs solutions obtain the solutions to subproblems of increasing size. iii. Until arrive at the solution of the original problem. Once we decided that we are going to attack the given problem with dynamic programming technique, the most important step is the formulation of the problem. In other words, the most important question in designing a dynamic programming solution to a problem is how to set up the subproblem structure.

If I can't apply dynamic programming to all optimization problem, then the question is what should I look for to apply this technique? Well! the answer is there are two important elements that a problem must have in order for dynamic programming technique to be applicable (look for those!). 1. Optimal Substructure Show that a solution to a problem consists of making a choice, which leaves one or sub-problems to solve. Now suppose that you are given this last choice to an optimal solution. [Students often have trouble understanding the relationship between

optimal substructure and determining which choice is made in an optimal solution. One way to understand optimal substructure is to imagine that "God" tells you what was the last choice made in an optimal solution.] Given this choice, determine which subproblems arise and how to characterize the resulting space of subproblems. Show that the solutions to the subproblems used within the optimal solution must themselves be optimal (optimality principle). You usually use cut-and-paste:

Suppose that one of the subproblem is not optimal. Cut it out. Paste in an optimal solution. Get a better solution to the original problem. Contradicts optimality of problem solution.

That was optimal substructure. You need to ensure that you consider a wide enough range of choices and subproblems that you get them all . ["God" is too busy to tell you what that last choice really was.] Try all the choices, solve all the subproblems resulting from each choice, and pick the choice whose solution, along the subproblem solutions, is best.

We have used "Optimality Principle" a couple of times. Now a word about this beast: The optimal solution to the problem contains within it optimal solutions to subproblems. This is some times called the principle of optimality.

The Principle of Optimality The dynamic programming relies on a principle of optimality. This principle states that in an optimal sequence of decisions or choices, each subsequence must also be optimal. For example, in matrix chain multiplication problem, not only the value we are interested in is optimal but all the other entries in the table are also represent optimal. The principle can be related as follows: the optimal solution to a problem is a combination of optimal solutions to some of its subproblems. The difficulty in turning the principle of optimally into an algorithm is that it is not usually obvious which subproblems are relevant to the problem under consideration.

Now the question is how to characterize the space of subproblems?

Keep the space as simple as possible. Expand it as necessary.

As an example, consider the assembly-line scheduling. In this problem, space of subproblems was fastest way from factory entry through stations S1, j and S2, j. Clearly, no need to try a more general space of subproblems. On the hand, in case of optimal binary search trees. Suppose we had tried to constrain space of subproblems to subtrees with keys k1, k2, . . . , kj. An optimal BST would have root kr , for some 1 r j. Get subproblems k1, . . . , kr 1 and kr + 1, . . . , kj. Unless we could guarantee that r = j, so that subproblem with kr + 1, . . . , kj is empty, then this subproblem is not of the form k1, k2, . . . , kj. Thus, needed to

allow the subproblems to vary at both ends, i.e., allow both i and j to vary.

Optimal substructure varies across problem domains:

1.

2.

How many subproblems are used in an optimal solution. How many choices in determining which subproblem(s) to use.

In Assembly-line Scheduling Problem: we have 1 subproblem and 2 choices (for Si, j use either S1, j 1 or S2, j 1). In the Longest Common Subsequence Problem: we have 1 subproblem but as far as choices are concern, we have either 1 choice (if xi = yj , LCS of Xi 1 and Yj 1), or 2 choices (if xi = yj , LCS of Xi 1 and Y , and LCS of X and Yj 1). Finally, in case of the Optimal Binary Search Tree Problem: we have 2 subproblems (ki , . . . , kr 1 and kr + 1, . . . , kj ) and j i + 1 choices for kr in ki, . . . , kj . Once we determine optimal solutions to subproblems, we choose from among the j i + 1 candidates for kr .

Informally, the running time of the dynamic programming algorithm depends on the overall number of subproblems times the number of choices. For example, in the assembly-line scheduling problem, there are (n) subproblems and 2 choices for each implying running time is (n). In case of longest common subsequence problem, there are (mn) subproblems and at least 2 choices for each implying (mn) running time. Finally, in case of optimal binary search tree problem, we have

(n2) sub-problems and (n) choices for each implying (n3) running time.

Dynamic programming uses optimal substructure bottom up fashion:

First find optimal solutions to subproblems. Then choose which to use in optimal solution to the problem.

When we look at greedy algorithms, we'll see that they work in top down fashion:

First make a choice that looks best. Then solve the resulting subproblem.

Warning! You'll surely make an ass out of yourself into thinking optimal substructure applies to all optimization problems. IT DOES NOT. Let me repeat, dynamic programming is not applicable to all optimization problems. To see this point clearly, go through pages 341 344 of CLRS where authors discussed two problems that look similar: Shortest Path Problem and Longest Simple Path Problem. In both problems, they gave us an unweighted, directed graph G = (V, E). And our job is to find finding a path (sequence of connected edges) from vertex u in V to vertex v in V.

Subproblems Dependencies It is easy to see that the subproblems, in our above examples, are independent subproblems: For example, in the assembly line problem, there is only 1 subproblem so it is trivially independent. Similarly, in the longest common subsequence problem, again we have only 1 subproblem thus it is automatically independent. On the other hand, in the optimal binary search tree problem, we have two subproblems, ki, . . . , kr 1 and kr + 1, . . . , kj, which are clearly independent.

2. Polynomially many (Overlapping) Subproblems An important aspect to the efficiency of dynamic programming is that the total number of distinct sub-problems to be solved should be at most a polynomial number. Overlapping subproblems occur when recursive algorithm revisits the same problem over and over. A good divide-and-conquer algorithm, for example the merge-sort algorithm, usually generate a brand new problem at each stage of recursion. Our Textbook CLRS has a good example for matrix-chain multiplication to depict this idea. The CLRS also talked about the alternative approach socalled memoization. It works as follows:

Store, don't recompute Make a table indexed by subproblem. When solving a subproblem: o Lookup in the table. o If answer is there, use it.

Otherwise, compute answer, then store it.

In dynamic programming, we go one step further. We determine in what order we would want to access the table, and fill it in that way.

Four-Step Method of CLRS Our Text suggested that the development of a dynamic programming algorithm can be broken into a sequence of following four steps. 1.Characterize the structure of an optimal solution. 2.Recursively defined the value of an optimal solution. 3.Compute the value of an optimal solution in a bottom-up fashion. 4.Construct an optimal solution from computed information.

Dynamic-Programming Algorithm for the Activity-Selection Problem

An activity-selection is the problem of scheduling a resource among several competing activity. Problem Statement Given a set S of n activities with and start time, Si and fi, finish time of an ith activity. Find the maximum size set of mutually compatible activities.

Compatible Activities Activities i and j are compatible if the half-open internal [si, fi) and [sj, fj) do not overlap, that is, i and j are compatible if si fj and sj fi

Dynamic-Programming Algorithm The finishing time are in a sorted array f[i] and the starting times are in array s[i]. The array m[i] will store the value mi, where mi is the size of the largest of mutually compatible activities among activities {1, 2, . . . , i}. Let BINARY-SEARCH(f, s) returns the index of a number i in the sorted array f such that f(i) s f[i + 1].

for i =1 to n do m[i] = max(m[i-1], 1+ m [BINARY-SEARCH(f, s[i])]) We have P(i] = 1 if activity i is in optimal selection, and P[i] = 0 otherwise

i=n while i > 0 do if m[i] = m[i-1] then P[i] = 0 i=i-1 else i = BINARY-SEARCH (f, s[i]) P[i] = 1

Analysis The running time of this algorithm is O(n lg n) because of the binary search which takes lg(n) time as opposed to the O(n) running time of the greedy algorithm. This greedy algorithm assumes that the activities already sorted by increasing time.