15.2.1.

1 The discrete case

Next: 15.2.1.2 The continuous case

Up: 15.2.1 Hamilton-Jacobi-Bellman Equation
Previous: 15.2.1 Hamilton-JacobiBellman Equation

15.2.1.1 The discrete case

Before entering the continuous realm, the concepts will first be

described for discrete planning, which is often easier to
understand.
Recall from Section 2.3 that if
, , and the stages
are discrete, then optimal planning can be performed
by using value
iteration or Dijkstra's algorithm on the search graph. The
stationary, optimal cost-to-go function
can
be used as a
navigation function that encodes the optimal feedback plan. This was
suggested in Section 8.2.2, and an
example was shown in
Figure 8.3.

Suppose that
denoted as

has been computed under Formulation

8.1 (or Formulation 2.3). Let the state
transition equation be

(15.5)

The dynamic programming recurrence for

is

(15.6)

which may already be considered as a discrete form of the

Hamilton-Jacobi-Bellman equation. To gain some insights
into the
coming concepts, however, some further manipulations will be
performed.

Let
denote the optimal action that is applied in the
of
(15.6). Imagine that

action but needs to be tested in (15.6) to make sure.

If it is truly optimal, then

is hypothesized as the optimal

(15.7)

This can already be considered as a discrete form of the Pontryagin

minimum principle, which will appear in Section
15.2.3. By
rearranging terms, a nice interpretation is obtained:

(15.8)

http://planning.cs.uiuc.edu/node814.html[5/3/2016 8:27:17 PM]

15.2.1.1 The discrete case

In a single stage, the optimal cost-to-go drops by

by

when

is used as a navigation function (multiply (15.8)

). The optimal single-stage cost is revealed precisely when

taking one step toward the goal along the optimal

path. This
incremental change in the cost-to-go function while moving in the best
direction forms the basis of both the
HJB equation and the minimum
principle.

Next: 15.2.1.2 The continuous case

Up: 15.2.1 Hamilton-Jacobi-Bellman Equation
Previous: 15.2.1 Hamilton-JacobiBellman Equation

Steven M LaValle
2012-04-20

http://planning.cs.uiuc.edu/node814.html[5/3/2016 8:27:17 PM]