IS703:

Decision Support and Optimization

Week 10: Local Search

Lau Hoong Chuin
School of Information Systems
1

Local Search
Reference:
Johnson and McGeoch, The Traveling Salesman Problem - A
Case Study in Local Optimization. In Local Search in
Combinatorial Optimization, E. H. L. Aarts and J. K. Lenstra
(eds), John-Wiley and Sons, 1997
Objectives:
To learn the basic local search paradigm for solving (NPhard) optimization problems. Why?
To design LS-based meta-heuristics (tabu search, simulated
annealing, genetic algorithms, ACO, etc)

Heuristics

Optimization problem:
minxSf(x), where S=feasible solutions space
(find a feasible x such that f is minimized)
Heuristic algorithms do not guarantee to find an
optimal solution (as opposed to exact algorithms)
However, they are usually designed to find a
reasonably good solution quickly.
Solutions to optimization problems:
Global Optimum: Equal/better than all other solutions
Local Optimum: Equal/better than all solutions in a
certain neighborhood
3

Global vs. Local Optimum

Why Do We Need Heuristics?

Many real world applications are too large to be
solved by exact methods (e.g. Branch and Bound,
Cutting Planes, etc.)
Solutions are usually needed fast (sometimes even
in real time)
for hard problems, the time for finding an optimal
solution is usually much greater than finding a nearoptimal solution

Optimal solutions, although desirable, are often not

needed in the real world.

Local Search
Definitions:
N (neighborhood) : S2s
The size of a neighborhood is the number of solutions in
the neighborhood set
Algorithm:
1. Pick a starting solution s // by other algorithm
2. If f(s) < mins in N(s) f(s), stop // what is the time complexity?
3. Else set s = s s.t. f(s) = mins in N(s) f(s)
4. Goto Step 2

Local Search
An iteration in a local search

Current
Solution
Generate
Neighborhood

Choose
Neighbor

Evaluate
Neighborhood

Example: Traveling Salesman Problem

Given:
A set of n cities {c1, c2, , cn}
Cost matrix, containing cost to travel between cities
Find a minimum-cost tour.

Solving TSP Heuristically

3 Broad Steps:
1. Construction Heuristics
Constructing a tour from scratch
Nearest neighbor, insertion heuristic

2. Local Search Algorithms

Improving an existing tour

3. Extensions
Escaping from locally optimal tours
Meta-heuristics

Construction Heuristics (Nearest Neighbor)

Step 1: Choose starting city p at random
Step 2: Scan remaining cities for the city NNp
nearest to p
Step 3: Add edge (p,NNp) to the sequence
Step 4: Set p = NNp
Step 5: Repeat from Step 2 until all cities are added
Step 6: Add last edge to complete a tour

10

Some Variants of Nearest Neighbour

Double-Sided Nearest Neighbour
Grow the nearest neighbour sequence from both ends

Randomized Nearest Neighbour

Instead of using the nearest neighbour of a city, pick a
neighbour at random from the set of remaining k
nearest neighbours of the city and add it to the
sequence

11

Multiple Fragment Heuristic

Basic Idea:
Build a tour one edge at a time, by adding the
shortest remaining available edge to the tour edge
set.
Step 1: Order the set of edges in increasing length in a list
Step 2: Pick first available edge on the list
Step 3: Remove edge from list and add it to the tour edge set
Step 4: Repeat from step 2 until there are no more available
edges on the list
Note: similarity with Kruskals algorithm for MSTs
12

Multiple Fragment Tour:

13

Insertion Heuristics
Basic Idea:
Starting with a subtour made of 2 arbitrary chosen cities
grow a tour by inserting cities which fulfill some criteria.
The criteria usually depends on the cities which are already
in the tour
Step 1: Form a subtour by choosing 2 cities at random
Step 2: Scan remaining cities for the city which fulfills
insertion criteria
Step 3: Insert the city to the subtour
Step 4: Repeat Step 2 until all cities are in the tour

14

Some Insertion Criteria

Nearest Insertion:
Insert the city that has shortest distance to a tour city

Farthest Insertion:
Insert the city whose shortest distance to a tour city is
maximized

Cheapest Insertion:
Choose the city whose insertion causes the least
increase in length of the resulting subtour

Random Insertion
Select the city to be inserted at random

15

Nearest vs Cheapest Insertion

16

Analysis
Number of steps:
n-2 cities are to be inserted
For each insertion the set of remaining cities has to be
scanned : O(n)

Worst case run time: O(n2)

17

Local Search Algorithm for TSP

Create an initial tour // discussed previous lecture
Define a local search neighborhood
Two tours are neighbors if they share most of their edges

Edge exchange neighborhood:

Delete k edges in the current tour and then add k
edges which form a new feasible tour
This neighborhood is called k-opt.

18

2-opt
Delete 2 edges, add 2 edges to restore the tour

2-opt removes the crossings of edges in a tour

19

2-opt Analysis
How many possible ways?
Size of the 2-opt neighborhood:
Number of edge pairs = n(n-3)/2
Exactly 1 way to restore tour

At each step of Local Search, evaluate O(n2) new

tours and choose the best available.
Worst-case run time: O(n2)

20

3-opt
Delete 3 edges, add 3 edges to restore the tour

3-opt can remove subsequences of the tour

21

3-opt Analysis
Size of the 3-opt neighborhood:
n

3

Number of edge triples =

Each of which has 8 possible ways to be restored
Some neighbors may not be valid

At each step of Local Search, O(n3) new tours

worst case run time: O(n3)

(1)

(2)
22

k-opt
It may seem sensible to increase k even further.
However, once k edges of a tour have been
deleted there are O(nk) new tours. Expensive!
Observe that some 3-opt moves are equivalent to
applying two 2-opt moves

23

Lin-Kernighan Heuristic
First observed by Lin and Kernighan
One of most successful heuristics to solve
combinatorial optimization problems
Adaptive k-Opt
Build k-opt moves (with variable k) by a sequence of
2-opt moves
"Tabu lists for edges added and deleted
Multiple restarts

24

Vehicle Routing Problem with Time Windows

Given
- k vehicles, each with fixed capacity
- a set of customers, having location,
time window, service duration and demand
- cost matrix [cij]
Find min-cost vertex-disjoint feasible routes to
cover all customers.
Finding a feasible solution is NP-hard, even k=1!

25

Vehicle Routing Problem with Time Windows

11:00~11:30
30units
11:00~12:00
10units

11:30~12:30
20units

Depot

10:30~12:00
10units

26

Vehicle Routing Problem with Time Windows

Neighborhood
Relocate:

Distribute:

Exchange:

27

Vehicle Routing Problem with Time Windows

Neighborhood
2-Opt:

Question:
What is the time complexity of each neighborhood?

28

Iterated Local Search

Problem of Local Search:
may get stuck in a local optimum of poor quality
very dependent on the starting tour

Solution:
Apply Local Search to more than one tours

Search Diversification

29

Multi-start
Step 1: Create a feasible tour (randomly or by
using a construction heuristic)
Step 2: Apply Local Search until local optimality
Step 3: Repeat from Step 1 until some stopping
criteria is met
Step 4: Output best tour found during this process

30

Multi-start: Random vs Constructive Restarts

Random:
creating a random tour is fast O(n)
Local Search is likely to be slow
Local Search usually does not perform very well

Construction Heuristics
creating a new tour is slow (best case O(n2))
Local Search is usually quite fast
Local Search usually performs better when started from
good tours

31

Chained Local Search

Motivation:
Why abandon a tour that is already locally optimal?
Basic concept:
Instead of restarting local search from a new tour,
modify current tour (kick) and restart local search on
modified tour

32

Chained Local Search

Step 1: Create an initial tour
Step 2: Apply Local Search
Step 3: Modify the locally optimal tour in a way that Local
Search cannot repair
Step 4: Apply Local Search only to the modified parts of the
tour
Step 5: Repeat from step 3 until stopping criteria is met
Step 6: Output best tour found

33

Local Search for Satisfiability

3SAT:
Input: 3CNF, Output: Yes/No whether input can be satisfied
WalkSat( C)
Guess initial assignment
While unsatisfied or not time out do
Select from C an unsatisfied clause c = Xi v Xj v Xk
Select variable v in unsatisfied clause c
Flip v

GSat( C)
Guess initial assignment
While unsatisfied or not time out do
Flip value assigned to the variable that yields the greatest # of
satisfied clauses. (Note: Flip even if no improvement)
If sat assignment not found, repeat entire process, starting from a
different initial random assignment.
34

Example:

35

Mixing Random Walk with Greedy

Value of p determined empirically, by finding best

setting for problem class

36

Finding the best value of p

Q: What value for p?
Let Q[p, c] be quality of using MixedWalkSat[p] on
problem c.
Q[p, c] = Time to return answer, or
= 1 if MixedWalkSat[p] return (correct) answer within 5
minutes and 0 otherwise, or
= . . . perhaps some combination of both . . .

Then, find p that maximize the average performance of

MixedWalkSat[p] on a set of challenge problems.

37

Experimental Results: Hard Random 3CNF

Source: Selman and Kautz 1993

Time in seconds
Effectiveness: probability that random initial
assignment leads to a solution.
Test instances up to 400 variables
MixedWalkSat better than Simulated Annealing
MixedWalkSat better than Basic GSAT
38

Advanced Local Search Paradigms

(Meta-heurstics)
An iterative master process that guides and modifies the
operations of a subordinate heuristic to efficiently produce
high quality solutions (Voss et al, 2002)
Different philosophies and forms
Single-point
Tabu Search
Simulated Annealing
Greedy Randomized Adaptive Search Procedure

Population-based
Evolutionary (Genetic) Algorithms
Ant Colony Optimization
Particle Swarm

Hybrids, e.g. Hyper-Heuristics

39

What can go wrong with Local Search?

Caught in local optimality
Diversify : multiple (random restarts), etc

Cycling:

40

What can go wrong with Local Search?

Analogy

41

Tabu Search
Fred Glover (1986)
A tabu list is maintained for forbidden (tabu)
moves, to avoid cycling
Tabu moves are based on the short- and long-term
history of the search process.
Aspiration criteria is the condition which allows the
tabu status of a tabu move to be overwritten so that
the move can be considered at the iteration.
The next move is the best move among the
feasible moves from the neighborhood of the
current solution. A tabu move is taken if it satisfies
the aspiration criteria.
42

Tabu Search Algorithm

s: current solution
N(s): neighborhood set of s where N(s) S
T(s): tabu set of s where T(s) N(s)
A(s): aspiration set of s where A(s) T(s)
1. construct an initial solution s
2. while not finished
3. compute N(s), T(s), A(s)
4. choose s(N(s) - T(s)) A(s) s.t. cost(s) is min
5. s = s
6. endwhile
43

Tabu Search Algorithm

Step 1
Set Current
Solution

Step 2
Define the
Neighborhood

Stopping conditions
meet
Update the TabuList and
Tabu search
trigger any related events
Stopped

Step 6

Step 5
Move operate on the bestpicked neighbor to
generate new solution

Step 3
ObjectiveFunction used
to evaluate each of the
neighbors

Step 4
TabuList and
AspirationCriteria are
consulted

44

Example: Constrained MST Problem

6
x1

9 x2

x3

x4

x5

x6

x7

18

Constraints:
12

(1) x1 + x2 + x6 1
(2) x1 x3
Penalty of each constraint
violation= 50
45

Example
Neighborhood: standard edge swap
An edge is tabu if it was added within the last two
iterations
The aspiration criteria is satisfied if the tabu move
would create a tree that is better than the best tree
so far

46

Example
Iteration 1 (initial MST)
cost = 16 + 100
Drop

6
x1
2
x4
8

x6

9 x2

18
x3 Add
0
x5

Constraints:

x7

(1) x1 + x2 + x6 1

12

(2) x1 x3
Penalty of each constraint
violation= 50
47

Example
Iteration 2
cost = 28
6
x1

Drop

9 x2

x3

x4

x5

x6

x7

18
Tabu

12

Add

48

Example
Iteration 3

6
x1
2
x4
8

x6

Drop
Add
18
9 x2 x3
Tabu
0

cost = 32

x5
x7

Tabu
12
Edge x3 is aspired.
49

Example
Iteration 4
cost= 23
6
Tabu
x1 9 x
2

x3

2
x4

0
x5

x6

x7
Tabu

18

12

50

Tabu Search for TSP

Tabu List
Proposal 1: Tabu-ing the solution
Record the complete tour
Strictly prevent solution cycling
Verification required checking all rotations

Proposal 2: Tabu-ing the move

Record the recent made moves
Easy to verify
Less restrictive: Does not prevent cycling entirely

Proposal 3: Tabu-ing the objective value

Record the solutions objective value

Easy to verify
More restrictive: May accidentally miss out good solutions

51

TABU SEARCH Advanced Strategies

Reactive Tabu List

Tabu tenure varies according to search history

Lengthen tenure if a series of poor solutions is encountered
Shorten tenure if good (elite) solution is encountered

Intensification
Focus on solutions with good characteristics
Idea is to find better solutions around elite solutions
Usually applied on local optimal

Diversification
Forcing search to move away from specific characteristics
Idea is to explore new regions of the search space
Usually applied when potential of finding better solution is low

52

Simulated Annealing
Kirkpatrick et al. (1982); Metropolis et al. (1953)
Let T be non-negative
Loop:
pick random neighbor s in N(s)
let D = f(s) f(s) (improvement)
if D > 0 (better), s = s
else with probability eD/T , set s = s
As T tends to 0, hill climbing
As T tends to , random walk
In general, start with large T and decrease it slowly.
53

Intuition Behind Simulated Annealing

A stochastic variation on hill climbing in which
downhill moves can be made.
The probability of making a downhill move decreases
with time (length of the exploration path from a start
state).
Model after the physical process of annealing metals
(cooling molten metal to solid minimal-energy state)
solutions = states
cost of solution = energy of state

54

Intuition Behind Simulated Annealing

During the annealing process in metals, there is
a probability p that a transition to a higher energy
state (which is sub-optimal) occurs. This
probability is edE/T where
dE = (energy of previous state) (energy of current
state) (it is < 0)
T = temperature of the metal

p is higher when T is higher, and movement to

higher energy states become less likely as the
temperature comes down.
The rate at which system is cooled is called the
annealing schedule.
55

Effect of Varying dE for fixed T=10

dE
-43

e dE/10
0.01

-13

0.27

1.00

The smaller the

value of dE, the
bigger the step would
be downhill, and the
lower the probability
of taking this downhill
move.

56

Effect of Varying T for fixed dE = -13

T
1

e -13/T
0.000002

50

0.56

1010

0.9999

The greater the

value of T, the
smaller the relative
importance of dE
and the higher the
probability of
choosing a downhill
move.

57

Logistic (sigmoid) Function

select any move (uphill or downhill) with probability
1/(1+e dE/T)
1

T very high

0.5

-20

T near 0

20

dE

58

Annealing Schedule
After each step, how to update T?
Logarithmic: Ti = /log(i+2)
provable properties, slow
Geometric: Ti = T0 int(i/L)
lots of parameters to tune
Adaptive
change Ti dynamically

59

Properties of Simulated Annealing

If the annealing schedule lowers T slowly enough
(exponentially), algorithm can find global optimum.
If large running time is allowed, SA generally
outperforms all other meta-heurstics with respect to
quality of solutions.

60

Next Week

Population-Based Meta-Heuristics
Hybrid Meta-Heuristics

61