Beruflich Dokumente
Kultur Dokumente
Dynamic programming
Xiaolan Xie
Dynamic programming
Some applications
Xiaolan Xie
Dynamic programming
Some applications
Xiaolan Xie
Introduction
Dynamic programming (DP) is a general optimization
technique based on implicit enumeration of the
solution space.
The problems should have a particular sequential
structure, such that the set of unknowns can be made
sequentially.
It is based on the "principle of optimality"
A wide range of problems can be put in seqential form
and solved by dynamic programming
Xiaolan Xie
Introduction
Applications :
• Optimal control
• Most problems in graph theory
• Investment
• Deterministic and stochastic inventory control
• Project scheduling
• Production scheduling
Xiaolan Xie
Illustration of DP by shortest path problem
Problem : We are planning the construction of a
highway from city A to city K. Different construction
alternatives and their costs are given in the
following graph. The problem consists in determine
the highway with the minimum total cost.
14 D 3
B I 10
8 10 G 9
E 5 K
A 10 9
10 8
C 7 8 H J
F
15
Xiaolan Xie
BELLMAN's principle of optimality
General form:
if C belongs to an optimal path from A to B, then the sub-path A to
C and C to B are also optimal
or
all sub-path of an optimal path is optimal
A
B
C
optimal optimal
Corollary :
SP(xo, y) = min {SP(xo, z) + l(z, y) | z : predecessor of y}
Xiaolan Xie
Solving a problem by DP
1. Extension
Extend the problem to a family of problems of the same nature
2. Recursive Formulation (application of the principle of optimality)
Link optimal solutions of these problems by a recursive relation
3. Decomposition into steps or phases
Define the order of the resolution of the problems in such a way that,
when solving a problem P, optimal solutions of all other problems
needed for computation of P are already known.
4. Computation by steps
Xiaolan Xie
Solving a problem by DP
Xiaolan Xie
Shortest Path in an acyclic graph
• Problem setting : find a shortest path from x0 (root of the graph) to a given
node y0
• Extension : Find a shortest path from x0 to any node y, denoted SP(x0, y)
• Recursive formulation
SP(y) = min { SP(z) + l(z, y) : z predecessorr of y}
• Decomposition into steps : At each step k, consider only nodes y with
unknown SP(y) but for which the SP of all precedecssors are known.
• Compute SP(y) step by step
Remarks :
• It is a backward dynamic programming
• It is also possible to solve this problem by forward dynamic programming
Xiaolan Xie
DP from a control point of view
Consider the control of
(i) a discrete-time dynamic system, with
(ii) costs generated over time depending on the states and the
control actions
action action
Cost Cost
Xiaolan Xie
DP from a control point of view
System dynamics :
x t+1 = ft(xt, ut), t = 0, 1, ..., N-1
where
t : temps index
xt : state of the system
action action
ut = control action to decide at t
State t State t+1
Cost Cost
Xiaolan Xie
DP from a control point of view
Criterion to optimize
N 1
Minimize g N xN gt xt , ut
t 0
action action
Xiaolan Xie
DP from a control point of view
Value function or cost-to-go function:
N 1
J n x = Minimize g N xN gt xt , ut xn x
t n
action action
Cost Cost
Xiaolan Xie
DP from a control point of view
Optimality equation or Bellman equation
J n x = MIN g n x, un J n+1 f n x, un
un
action action
Cost Cost
Xiaolan Xie
Applications
Inventory control
Xiaolan Xie
Applications
Single machine scheduling (Knapsac)
Problem :
Consider a set of N production requests, each needing a
production time ti on a bottleneck machine and generating
a profit pi. The capacity of the bottleneck machine is C.
Question: determine the production requests to confirm in
order to maximize the total profit.
Formulation:
max pi Xi
subject to:
ti Xi C
Xiaolan Xie
Applications
Inventory control
See exercices
Xiaolan Xie
Applications
Traveling salesman problem
Problem :
Data: a graph with N nodes and a distance matrix
[dij] beteen any two nodes i and j.
Question: determine a circuit of minimum total
distance passing each node once.
Extensions:
C(y, S): shortest path from y to x0 passing once
each node in S.
Application: Machine scheduling with setups.
2007 Xiaolan Xie
Applications
Total tardiness minimization on a single machine
Xiaolan Xie
Stochastic dynamic programming
Model
Consider the control of
(i) a discrete-time stochastic dynamic system, with
(ii) costs generated over time
perturbation perturbation
action action
Xiaolan Xie
Stochastic dynamic programming
Model
System dynamics :
x t+1 = ft(xt, ut, wt), t = 0, 1, ..., N-1
where
t : time index
xt : state of the system
perturbation
ut = decision at time t action action
wt : random perturbations
State t State t+1
cost cost
Xiaolan Xie
Stochastic dynamic programming
Model
Criterion
N 1
Minimize E g N xN gt xt , ut , wt
t 0
perturbation
action action
cost cost
Xiaolan Xie
Stochastic dynamic programming
Model
Open-loop control:
Order quantities u1, u2, ..., uN-1 are determined once at time 0
Closed-loop control:
Xiaolan Xie
Stochastic dynamic programming
Control policy
The rule for selecting at each period t a control action ut
for each possible state xt.
N 1
Jp x0 E cmt xt r xt ut wt
t 0
Optimal control:
minimize Jp(x0) over all possible polciy p
pij(u, t) = P{xt+1 = j | xt = i, ut = u}
J * x0 MIN Jp x0
p P
Xiaolan Xie
Stochastic dynamic programming
Principle of optimality
Let p* = {m*0, ..., m*N-1} be an optimal policy for the basic
problem for the N time periods.
Then the truncated policy {m*i, ..., m*N-1} is optimal for the
following subproblem
• minimization of the following total cost (called cost-to-go
function) from time i to time N by starting with state xi at
time i
N 1
J i xi MIN E g N xN gt xt , mt xt , wt
t i
Xiaolan Xie
Stochastic dynamic programming
DP algorithm
Theorem: For every initial state x0, the optimal cost J*(x0) of
the basic problem is equal to J0(x0), given by the last step of
the following algorithm, which proceeds backward in time
from period N-1 to period 0
J N xN g N xN , ( A)
J t xt MIN
ut Ut xt
Ewt gt xt , ut , wt J t 1 ft xt , ut , wt , ( B)
Xiaolan Xie
Stochastic dynamic programming
Example
Consider the inventory control problem with the following:
Xiaolan Xie
Stochastic dynamic programming
DP algorithm
Optimal policy
Stock Stage 0 Stage 0 Stage 1 Stage 1 Stage 2 Stage 2
Cos-to-go Optimal Cos-to-go Optimal Cos-to-go Optimal
order order order
quantity quantity quantity
Xiaolan Xie
Sequential decision model
Key ingredients: Policy: Issues:
• A set of decision epochs
a sequence of Existence of opt.
• A set of system states decision rules in policy
• A set of available actions order to mini. the Form of the opt. policy
• A set of state/action cost function Computation of opt.
dependent immediate costs policy
• A set of state/action
dependent transition action action
probabilities
Present Next
state state
costs costs
Xiaolan Xie
Applications
Inventory management
Bus engine replacement
Highway pavement maintenance
Bed allocation in hospitals
Personal staffing in fire department
Traffic control in communication networks
…
Xiaolan Xie
Example
• Consider a with one machine producing one product. The
processing time of a part is exponentially distributed with rate
p. The demand arrive according to a Poisson process of rate d.
• state Xt = stock level, Action : at = make or rest
1 T hX , if X 0
Minimize lim
g X t dt with g
X
T T
t 0 bX , if X 0
0 1 2 3
d d d
d
Xiaolan Xie
Example
• Zero stock policy P(0) = 1-r, P(-n) = rnP(0), r = d/p
Xiaolan Xie
MDP Model formulation
Xiaolan Xie
Decision epochs
Times at which decisions are made.
Xiaolan Xie
State and action sets
At each decision epoch, the system occupies a state.
An Markov decision process is characterized by {T, S, As, pt(. |s, a), Ct(s, a)}
Xiaolan Xie
Exemple of inventory management
Consider the inventory control problem with the following:
Xiaolan Xie
Exemple of inventory management
Decision Epochs T = {0, 1, 2, …, N}
Set of states : S = {0, 1, 2} indicating the initial stock Xt
Action set As : indicating the possible order quantity Ut
A0 = {0, 1, 2}, A1 = {0, 1}, A2 = {0}
Cost function : Ct(s, a) = E[a + (s + a – wt)2]
Transition probability pt(. |s, a). :
p(j |s, a) a=0 a=1 a=2
s=0 (1, 0, 0) (0,9, 0,1, 0) (0,2, 0,7, 0,1)
s=1 (0,9, 0,1, 0) (0,2, 0,7, 0,1) Not allowed
s=2 (0,2, 0,7, 0,1) Not allowed Not allowed
Xiaolan Xie
Decision Rules
A decision rule prescribes a procedure for action selection in each
state at a specified decision epoch.
Xiaolan Xie
Decision Rules
A decision rule can also be either
Xiaolan Xie
Decision Rules
As a result, the decision rules can be:
Xiaolan Xie
Policies
A policy specifies the decision rule to be used at all decision epoch.
Xiaolan Xie
Example
Decision epochs: T = {1, 2, …, N}
State : S = {s1, s2}
Actions: As1 = {a11, a12}, As2 = {a21}
Costs: Ct(s1, a11) =5, Ct(s1, a12) =10, Ct(s2, a21) = -1, CN(s1) = rN(s2) 0
Transition probabilities: pt(s1 |s1, a11) = 0.5, pt(s2|s1, a11) = 0.5, pt(s1 |s1,
a12) = 0, pt(s2|s1, a12) = 1, pt(s1 |s2, a21) = 0, pt(s2 |s2, a21) = 1
Decision epoch 2:
P2, s1(a11) = 0.4, P2, s1(a12) = 0.6
P2, s2(a21) = 1
a11 a11 a21
{5, .5}
{5, .5}
S1 S2 {-1, 1}
a12
{10, 1}
Xiaolan Xie
Example
A deterministic history-dependent policy
Decision epoch 1: Decision epoch 2:
d1(s1) = a11 history h d2(h, s1) d2(h, s2)
d1(s2) = a21
(s1, a11) a13 a21
(s1, a12) infeasible a21
(s1, a13) a11 infeasible
Xiaolan Xie
Finite Horizon Markov Decision
Processes
Xiaolan Xie
Assumptions
Assumption 1: The decision epochs T = {1, 2, …, N}
Criterion:
N 1
infHR E Ct X t , at CN X N X1 s
p P t 1
Xiaolan Xie
Optimality of Markov deterministic policy
Theorem :
Assume S is finite or countable, and that As is finite for each
s S.
Xiaolan Xie
Optimality equations
Theorem : The following value functions
N 1
Vn s MIN E Ct X t , at CN X N X n s
p P HR
t n
satisfy the following optimality equation:
Vt s MIN Ct s, a pt j s, a Vt 1 j
aAs
jS
VN s rN s
and the action a that minimizes the above term defines the
optimal policy.
Xiaolan Xie
Optimality equations
The optimality equation can also be expressed as:
Vt s MIN Qt s, a
aAs
Qt s, a Ct s, a pt j s, a Vt 1 j
jS
Xiaolan Xie
Dynamic programming algorithm
•Set t = N and
VN sN rN sN for all sN S
•Substitute t-1 for t and compute the following for each st S
Vt s MIN Ct s, a pt j s, a Vt 1 j
aAs
jS
dt s arg min Ct s, a pt j s, a Vt 1 j
aAs jS
3. Repeat 2 till t = 1.
Xiaolan Xie
Infinite Horizon discounted
Markov decision processes
Xiaolan Xie
Assumptions
Assumption 1: The decision epochs T = {1, 2, …}
Assumption 2: The state space S is finite or countable
Assumption 3: The action space As is finite for each s
Assumption 4: Stationary costs and transition probabilities;
C(s, a) and p(j |s, a), do not vary from decision epoch to
decision epoch
Assumption 5: Bounded costs: | Ct(s, a) | M for all a As
and all s S (to be relaxed)
Xiaolan Xie
Assumptions
Criterion:
N
infHR lim E Ct X t , at l X1 s
t
p P N
t 1
where
0 < l < 1 is the discounting factor
PHR is the set of all possible policies.
Xiaolan Xie
Optimality equations
Theorem: Under assumptions 1-5, the following optimal cost
function V*(s) exists:
N
V * s infHR lim E Ct X t , at l X1 s
t
p P N
t 1
and satisfies the following optimality equation:
V * s MIN C s, a l p j s, a V * j
aAs
jS
Further, V*(.) is the unique solution of the optimality equation.
Moreover, a statonary policy p is optimal iff it gives the
minimum value in the optimality equation.
Xiaolan Xie
Computation of optimal policy
Value Iteration
Value iteration algorithm:
1.Select any bounded value function V0, let n =0
2. For each s S, compute
V s MIN C s, a l p j s, a V j
n 1 n
aAs
jS
3.Repeat 2 until convergence.
4. For each s S, compute
d s arg min C s, a l p j s, a V j
n 1
aAs jS
Xiaolan Xie
Computation of optimal policy
Value Iteration
Theorem: Under assumptions 1-5,
a.Vn converges to V*
b. The stationary policy defined in the value iteration
algorithm converges to an optimal policy.
Xiaolan Xie
Computation of optimal policy
Policy Iteration
Policy iteration algorithm:
1.Select arbitrary stationary policy p0, let n =0
2. (Policy evaluation) Obtain the value function Vn of policy pn.
3.(Policy improvement) Choose pn+1 = {dn+1, dn+1,…} such that
dn 1 s arg min C s, a l p j s, a V j
n
aAs jS
4.Repeat 2-3 till pn+1 = pn.
Xiaolan Xie
Computation of optimal policy
Policy Iteration
Policy evaluation:
For any stationary deterministic policy p = {d, d, …}, its
value function
V s E rt X t , at l X1 s
p t
t 1
is the unique solution of the following equation:
V p s C s, d s l p j s , d s V p j
jS
Xiaolan Xie
Computation of optimal policy
Policy Iteration
Theorem:
The value functions Vn generated by the policy iteration
algorithm is such that Vn+1 Vn.
Further, if Vn+1 Vn, Vn = V*.
Xiaolan Xie
Computation of optimal policy
Linear programming
Recall the optimality equation
V s MIN C s, a l p j s, a V j
aAs
jS
The optimal value function can be determine by the
following Linear programme:
Maximize V s
sS
subject to
V s r s, a l p j s, a V j , s, a
jS
Xiaolan Xie
Extensition to Unbounded Costs
Theorem 1. Under the condition C(s, a) ≥ 0 (or C(s, a) ≤0) for all
states i and control actions a, the optimal cost function V*(s) among
all stationary determinitic policies satisfies the optimality equation
V * s MIN C s, a l p j s, a V * j
aAs
jS
Theorem 2. Assume that the set of control actions is finite. Then, under
the condition C(s, a) ≥ 0 for all states i and control actions a, we have
lim V N s V * s
N
where VN(s) is the solution of the value iteration algorithm with V0(s) = 0.
Implication of Theorem 2 : The optimal cost can be obtained as the limit
of value iteration and the optimal stationary policy can also be obtained in
the limit.
Xiaolan Xie
Example
• Consider a computer system consisting of M different processors.
• Using processor i for a job incurs a finite cost Ci with C1 < C2 < ... < CM.
• When we submit a job to this system, processor i is assigned to our job with
probability pi.
• At this point we can (a) decide to go with this processor or (b) choose to hold the
job until a lower-cost processor is assigned.
• The system periodically return to our job and assign a processor in the same way.
• Waiting until the next processor assignment incurs a fixed finite cost c.
Question:
How do we decide to go with the processor currently assigned to our job versus
waiting for the next assignment?
Suggestions:
• The state definition should include all information useful for decision
• The problem belongs to the so-called stochastic shortest path problem.
Xiaolan Xie
Infinite Horizon average cost
Markov decision processes
Xiaolan Xie
Assumptions
Assumption 1: The decision epochs T = {1, 2, …}
Assumption 2: The state space S is finite
Assumption 3: The action space As is finite for each s
Assumption 4: Stationary costs and transition probabilities;
C(s, a) and p(j |s, a) do not vary from decision epoch to
decision epoch
Assumption 5: Bounded costs: | Ct(s, a) | M for all a As
and all s S
Assumption 6: The markov chain correponding to any
stationary deterministic policy contains a single recurrent
class. (Unichain)
Xiaolan Xie
Assumptions
Criterion:
1 N
infHR lim E
p P N
Ct X t , at X1 s
N t 1
where
PHR is the set of all possible policies.
Xiaolan Xie
Optimal policy
• Under Assumptions 1-6, there exists a optimal stationary
deterministic policy.
Xiaolan Xie
Relation between discounted and average cost MDP
g lim 1 l Vl s
l 1
h s lim Vl s Vl x0
differential
l 1 cost
for any given state x0.
Xiaolan Xie
Computation of the optimal policy by LP
Recall the optimality equation:
h s g MIN C s, a p j s, a h j
aAs
jS
h( x0 ) 0
Remarks: Value iteration and policy iteration can also be
extended to the average cost case.
Xiaolan Xie
Computation of optimal policy
Value Iteration
1.Select any bounded value function h0 with h0(s0) = 0, let n =0
2. For each s S, compute
U n 1
s h n 1
s g MIN
n
r s, a p j s, a h j
n
aAs
jS
h n 1 s U n 1 s U n 1 s0
g n U n 1 s0
3.Repeat 2 until convergence.
4. For each s S, compute
d s arg min C s, a p j s, a h j
n 1
aAs jS
Xiaolan Xie
Extensions to unbounded cost
h s lim Vl s Vl x0
l 1
Xiaolan Xie
Continuous time Markov decision
processes
Xiaolan Xie
Assumptions
Xiaolan Xie
Assumptions
Criterion:
infHR E C X t , a t e dt
t
p P t 0
1 T
infHR lim E C X t , a t dt
p P T T
t 0
Xiaolan Xie
Example
• Consider a system with one machine producing one product. The
processing time of a part is exponentially distributed with rate p. The
demand arrive according to a Poisson process of rate d.
• state Xt = stock level, Action : at = make or rest
hX , if X 0
g X t e dt with g X
t
Minimize
t 0 bX , if X 0
0 1 2 3
d d d
d
Xiaolan Xie
Uniformization
Any continuous-time Markov chain can be converted to a
discrete-time chain through a process called
« uniformization ».
Xiaolan Xie
Uniformization
In order to synchronize (uniformize) the transitions at the same
pace, we choose a uniformization rate
g MAX{m(i)}
« Uniformized » Markov chain with
• transitions occur only at instants generated by a common a
Poisson process of rate g (also called standard clock)
• state-transition probabilities
pij = mij / g
pii = 1 - m(i)/ g
where the self-loop transitions correspond to fictitious events.
Xiaolan Xie
Uniformization
CTMC
a Step1: Determine rate of the states
m(S1) = a, m(S2) = b
S1 S2
b
Step 2: Select an uniformization
Uniformized CTMC rate
g-a a
g-b g ≥ max{m(i)}
S1 S2
b
Step 3: Add self-loop transitions to
DTMC by uniformization states of CTMC.
1-a/g a/g 1-b/g
S1 S2 Step 4: Derive the corresponding
uniformized DTMC
b/g
Xiaolan Xie
Uniformization
m0,0 l1l2
m1,0 m1l2
m0,1 l1m2
m1,1 m1
Xiaolan Xie
Uniformization
Xiaolan Xie
Uniformization
0 1 2 3
d d d
d
0 1 2 3
d/g d/g d/g d/g d/g
(not make, p/g) (not make, p/g) (not make, p/g) (not make, p/g)
Xiaolan Xie
Uniformization
Under the uniformization,
• a sequence of discrete decision epochs T1, T2, … is generated
where Tk+1 – Tk = EXP(g).
• The discrete-time markov chain describes the state of the system at
these decision epochs.
• All criteria can be easily converted.
continuous cost C(s,a)
per unit time fixed cost
fixed cost k(s,a, j)
K(s,a)
(s,a) j
EXP(g) EXP(g) EXP(g)
Xiaolan Xie
Cost function convertion
for uniformized Markov chain
Discounted cost of a stationary policy p (only with continuous cost):
Tk 1
Ep C X t , a t e dt E C X t , a t e t dt
t
t 0 k 0 t Tk
Tk 1 State change & action taken only at Tk
E C X k , ak e dt
t
k 0 t Tk
Mutual independence of (Xk, ak) and
Tk 1
E C X k , ak E e dt
t (Tk, Tk+1)
k 0 t Tk
k
1 g Tk is a Poisson process at rate g
E C X k , ak
k 0 g g
g k C X , a
E
k k
k 0 g g
Xiaolan Xie
Cost function convertion
for uniformized Markov chain
Equivalent discrete time discounted MDP
• a discrete-time Markov chain with uniform transition rate g
• a discount factor l g/g
• a stage cost given by the sum of
─ continuous cost C(s, a)/(g),
─ K(s, a) for fixed cost incurred at T0
─ lk(s,a,j)p(j|s,a) for fixed cost incurred at T1
Optimality equation
C s, a
g
V s MIN
aAs g
K s, a
g
p j s , a k s , a , j V j
jS
Xiaolan Xie
Cost function convertion
for uniformized Markov chain
Equivalent discrete time average-cost MDP
• a discrete-time Markov chain with uniform transition rate g
• a stage cost given by C(s, a)/g whenever a state s is entered
and an action a is chosen.
Optimality equation :
C s, a
h s g MIN p j s, a h j
where g
aAs jS
• g = average cost per discretized time period
• gg = average cost per time unit (can also be obtained directly
from the optimality equation with stage cost C(s, a))
Xiaolan Xie
Example (continue)
Uniformize the Markov decision process with rate g = p+d
g s p d
l V s 1 V s 1 : producing
g pd pd
V s MIN
g s p d
g l V s V s 1 : not producing
p d p d
Xiaolan Xie
Example (continue)
From the optimality equation:
g s p
V s 1 MIN V s 1 V s , 0
d
V s V s
g pd pd
V(s+1) –V(s) > 0 and the decision is not producing, for all s >= K and
V(s+1) –V(s) <= 0 and the decision is producing, for all s < K
Xiaolan Xie
Example (continue)
Convexity proved by value iteration
g s p
V n 1 s
g
pd
MIN V n s 1 ,V n s
d
pd
V n s 1
V 0 s 0
Proof by induction.
V0 is convex.
MIN V n s 1 ,V n s is convex
If Vn is convex with minimum
at s = K, then Vn+1 is convex.
s
K-1 K
Xiaolan Xie