OPTIMAL CONTROL OF CONTINUOUS

DYNAMIC SYSTEMS
General Continuous Optimal Control Problem
plant
u(t) (t ) f (x, u, t )
x x(t)

The optimal control problem is to find the input control signal

u*(t) on the time interval [to,T] that drives the plant (system)
x (t ) f (x, u, t ), x(to ) x o
along a trajectory x*(t) such that the cost function:

b g zLbx(t ), u(t ), t gdt

J (t o ) x(T ), T
T

to

is minimised, with terminal conditions:

x(T ), T 0
222
where x(t ) , u(t) and
n m p
 Calculus of Variations - Leibnizs Rule
If x(t ) n is a function of t and:
J ( x) zb
to
T
g
h x(t ), t dt
where J(x) and h(.) are scalar functionals (i.e.
functions of the function x(t)), then the variation:

b g b g
dJ h x(T ), T dT h x(t o ), t o dto

z[h b
x(t ), t g
T
x]dt
T
x
to
where
h T
hx x h
x
( a column vector (= gradient))
At a stationary point dJ = 0 223
 Solution of the Optimal Control Problem
Using Lagrange multipliers, we obtain the
augmented performance index:
J x(T ), T T x(T ), T
L(x,u, t ) T (t ) f (x,u, t ) x dt
T

to

where p is a constant multiplier and (t ) n

is a function of time.
Define the Hamiltonian function:
H (x,u, t ) L(x,u, t ) f (x,u, t )
T

giving:-
J x(T ), T x(T ), T + H ( x,u, t ) T 224
T
T
x dt
to
Then from Leibnizs rule:-
dJ (x x ) dx (t t )dt
T T T T
d
T T T

( H T x)dt ( H T x)dt
T to
T
H xT x H uT u T x ( H x)T dt
to

Now by integration by parts

T T
xdt x x T xdt
T T T
to T to to
T
dx xdt dx xdt
T T T T
xdt
T
T T to to to

noting that dx(T ) x(T ) x (T )dT 225

Hence:
dJ (x Tx )T dx (t Tt H T x T x)dt
T T

T
d ( H x x)dt dx
T T T
T to to
T
( H x )T x H uT u ( H x)T dt
to

For the particular case when to and x(to) are fixed,

dto = 0, dx(to) = 0 giving:-

dJ (x Tx )T dx (t Tt H )dt T d
T T T
T
( H x )T x H uT u ( H x)T dt
to
226
For a stationary point,
dJ 0 for all independent d ,
x, u and . Hence, the necessary conditions for a
minimum are:-

x H f x f (x,u, t ) (system model)

H x (costate equations)
H u 0 (stationarity condition)
(x Tx )T dx(T ) (t Tt H ) dT 0
T T

(boundary condition)
0 x(T ), T 0 (final-state constraint)

227
Summary of Necessary Optimality Conditions

System description

System model: x f (x, u, t ), t to , to fixed, x(to ) given

b g zL(x, u, t )dt
Performance index: J (to ) x(T ), T
T

to

Final state constraint: x(T ), T 0

228
Optimal controller
Hamiltonian: H ( x, u, t ) L( x, u, t ) T
f (x, u, t )

State equation: x H H f
T

Costate equation: H H
T

x
x
Stationarity: FH I
G J0
T

Hu
Hu K
Boundary conditions:
x(to ) given
(x Tx )T dx(T ) (t Tt H ) dT 0
T T

x(T ), T 0 229
Notes
1. The optimal control depends upon the solution of a
two- point boundary value problem
2. For fixed time horizon problems (i.e. to and T
fixed), dT = 0 and the boundary condition becomes:-
(x )
T
x
T
dx(T ) 0
T
3. If x(T) is fixed, dx(T) = 0 and (T) is free.
4. If x(T) is free, (T ) x Tx
and, furthermore, if (T ) x
does not exist,
and, even further, if (T ) 0.
does not exist

5. In the time invariant case, it can be shown that

the Hamiltonian is constant along an optimal
trajectory 230
Example

min J
u(t )
{ 1
2 z
T

0
( x 0.25u )dt
2 2
}
s.t. x u, x(0) 1

1
u(t) x(t)
s
plant

231
Application of the optimality conditions
Hamiltonian: H ( x 0.25u ) u
1
2
2 2

H x
Co-state equation: x

Stationarity: Hu 0 0.25u 0 u 4

Boundary conditions: x ( 0) 1
(T ) 0
Hence, the two point boundary value problem
is: x 4 , x(0) 1
x, (T ) 0 232
Solution of the two point boundary value problem
Write the two point boundary problem as:
LMO
x
P
x L
M O
P
A , x (0) 1, (T ) 0
NQ NQ
L
where A M
0 4O
P
N Q
1 0
The solution will be of the form:
LM
x (t ) O
P 1t
e e e e 2t

N
(t ) Q 1 1 2 2

where , are the eigenvalues of A

1 2
e1, e2 are the corresponding eigenvectors.
calculated as follows:
233
 L 4O
I A M P , I A 2
4 0 2
2
N1 Q 2

1 2 , 2 2
L
Adj I A M
4 O
P
,
L2O
e MP
L
2O
, e MP
2
N1 Q 1
N1Q N1Q2

Hence:

L
M
x (t ) O L2 O
P
MP
L
2O
e MP
2t
e ; x (0) 1, (T ) 0
2 t

N (t )Q N1Q N1Q
1 2
234
 where 1 and 2 are evaluated to satisfy the
boundary conditions: x(0) = 1, (T) = 0;
giving:
1 2 1 2 2 1 1
1 , 2
0 1e 2e
2T 2 T
2(1 e )
4T 4 T
2(1 e )
Hence:

L
x (t ) O
M P

1 L
M
2O
P
e
1 2t L
2O
MPe 2 t

N
(t ) Q2(1 e ) N
1Q 2(1 e ) N
4T
1Q 4 T

1 1
(t ) e
2t
4 T
e 2 t

2(1 e )
4T
2(1 e ) 235
and the optimal control signal and state response are:
2 2
u (t ) 4 (t )
*
e 2t
e 2 t

1 e4T 1 e 4 T
1 1 2 t
x (t )
*
e
2t
4 T
e
1 e 4T
1 e

u * ( t ) 2 e 2 t
Note: for large T
2 t
x (t ) e
*

-2e-2t 1 e-2t
s
u(t) x(t)
plant
Note: the determined optimal control is open loop
236
Feedback solution
If we assume: (t ) s(t ) x (t ) where s(T ) 0
Then: (t ) s(t ) x (t ) s(t ) x (t )
and substitution from the co-state and state
equations gives:
s( 4 ) sx
x sx 4 s x s 4 s 1
2 2

ds
2 4dt 2tanh (2 s) 4t c
-1

s 0.25
where the constant c needs to be evaluated to satisfy
the boundary condition s(T) = 0. This gives c = 4T
and:
b
tanh -1 (2 s) 2(T t ) s(t ) 21 tanh 2(T t )237 g
 tanh (2 s) 2(T t ) s(t )
-1 1
2 b
tanh 2(T t ) g
Hence, the feedback control is:
u(t ) 4 (t ) 4 s(t ) x (t ) k (t ) x (t )
where the time dependent feedback gain is:
b
k (t ) 2 tanh 2(T t ) g
u(t) 1 x(t)
s
plant

-k(t)
gain
Note as: T , k ( t ) 2
238
The closed-loop system can also be drawn in
the form:

ref = 0 u(t) 1 x(t)

+ k(t) s
-
gain plant

239
MATLAB results:
optimal control of a single integrator
1
0
u(t)
-1
-2
0 0.5 1 1.5 2 2.5 3 3.5 4
1

0.5
x(t)
0
0 0.5 1 1.5 2 2.5 3 3.5 4
2

1
k(t)
0
0 0.5 1 1.5 2 2.5 3 3.5 240 4
t
Sub-optimal control: Use the steady-state value:
k ( t ) 2, t 0T
comparison of optimal and sub-optimal control T = 4
1

0
u(t)
-1

-2
0 0.5 1 1.5 2 2.5 3 3.5 4

1
Jopt = 0.2533
Jsub_opt = 0.2533
0.5
x(t)

0
0 0.5 1 1.5 2 2.5 3 3.5 241 4
t
 comparison of optimal and sub-optimal control T = 1
0

-0.5

-1
u(t)
-1.5

-2
0 0.2 0.4 0.6 0.8 1

1
Jopt = 0.2442
Jsub_opt = 0.2487
0.5
x(t)

0
0 0.2 0.4 0.6 0.8 242 1
t
MATLAB Demo

243
 Linear Quadratic Regulator (LQR)
System model: x Ax Bu, x(0) xo , x(t ) n , u(t ) m
Performance index:
J 2 x (T )S(T )x(T ) 2
1 T 1
T
zc
xT Qx u T Ru dt
0
h
where Q and S(T) are +ve semi def, R is +ve def; Q, S(T) and R
are all symmetric, T is fixed. We wish to determine u*(t) on
[0,T] to minimise J.
Hamiltonian: H 12 (xT Qx uT Ru) T ( Ax Bu)
Co-state equation: H x Qx AT
Stationarity:
Hu 0 Ru B T 0 u R 1B T
Boundary conditions:
x(0) x o
( x )T dx(T ) 0 (T ) S(T ) x(T )
244
T
Two-point boundary value problem:
1
x Ax BR B , x(0) xo
T

Qx A ,
T
(T ) S(T )x(T )

which can be written as the hamiltonian

system:
x A BR B x
1 T

Q A
T

hamiltonian matrix 245

Feedback solution:
Assume: (t ) S(t )x(t ) where S(T ) is given
This gives closed-loop feedback control: u R 1B TSx
and the two-point boundary value problem becomes
x Ax BR 1BT Sx (A BR 1BT S)x
1
Sx Sx Sx S(A BR B S)x Qx A Sx
T T

giving the matrix Riccati equation:

1 T
S A S SA SBR B S Q, S(T ) given
T

which can be solved backwards in time. 246

The optimal control is: u(t ) R 1B TS(t )x(t )

That is: u(t ) K (t )x(t )

1 T
where K(t) (the Kalman gain) is: K ( t ) R B S( t )

r=0 -x(t) u(t) x(t)

K(t) x Ax Bu
+ -
controller plant
time-dependent

247
 Steady -state closed-loop control and
suboptimal feedback

As T the matrix Riccati equation converges to:

S(t ) S and we can then use a constant feedback
gain K

S is given by solving the algebraic Riccati equation:

1
A S SA SBR B S Q 0 S
T T

We then obtain the steady-state Kalman gain:

K R 1B T S
248
r=0 -x(t) x(t)
u(t)
K x Ax Bu
+ -

controller plant
time-invariant

249
Example min J
u( t )
{ 1
2 z
T

0
( x u )dt
2
1
2
}
1 x2 , x1 (0) 1
s. t. x
2 u,
x x2 (0) 0

u x2 x1
1 1
s s

Here:
L
0 1O L
AM P
0O
, BMP L
1 0O
, QM P, R 1
N
0 0Q N 1Q N
0 0Q

Substitution into the matrix Riccati equation:

1 T
S A S SA SBR B S Q, S(T ) 0
T

produces: 250
 L
M
s s OL
11
P
0 0O
M P
12 L
M
s s OL
P
s
M
11 12 11 O
L
s12 0 1
P
M O
P
Ns s QN1 0Q
12 22 Ns s QNs 12 22 12 Q
NQ
s22 0 0

M
Ls s OL
P
M
0O L
P
s
0 1M
11 12 11 s12 O
PL1 0O
M P
Ns s QN1Q Ns
12 22 12 s22 QN0 0Q
s11 s12 1,
2
s11 (T ) 0
s12 s11 s12 s22 , s12 (T ) 0 s11 (t ), s12 (t ), s22 (t )
s22 2 s12 s222 , s22 (T ) 0
The Kalman gain matrix: K R 1B T S then gives:

K 0
L
s
1M
s O
P
11

12
s12 (t ) s22 (t ) k1 (t ) k 2 (t )
N
s s Q
12 22 251
Feedback control structure
u Kx k1 (t ) x1 (t ) k2 (t ) x2 (t )

k2(t)

ref = 0 - 1 1
k1(t) x1
+ + u s x2 s
-

252
Numerical solution of the Riccati equations gives:
Matrix Riccati Solution, T = 8
1.5

s22
1
s12
s11
S(t)

0.5

0
0 1 2 3 4 5 6 7 8
t

Kalman Gain, T = 8
1.5

k2

1
k1
K(t)

0.5

0
253
0 1 2 3 4 5 6 7 8
t
For large T, the steady-state solution of the Riccati
equations gives:

s12
2
1 0
s11 s12 s22 0 s11 2 s12 1, s22 2
2 s12 s22 0
2

noting that we take positive values of s12 and s22 to

ensure that S is +ve def. This gives negative
feedback Kalman gains:

K s12 s22 k1 1, k2 2

254
and the sub-optimal control structure
(optimal ifT ):

ref = 0 -
1
1 1 x1
+ + u s x2 s
-

255
 comparison of optimal and sub-optimal responses, T = 8
1
x1(t)
0.5
x(t)
0
x2(t)
-0.5
0 1 2 3 4 5 6 7 8

0.5

0
u(t) J = 0.7119
opt
-0.5
J sub_opt = 0.7119

-1
0 1 2 3 4 5 6 7 256 8
t
 comparison of optimal and sub-optimal responses, T = 4
1
x1(t)
0.5
x(t)
0
x2(t)
-0.5
0 0.5 1 1.5 2 2.5 3 3.5 4

0.5

0
u(t) Jopt = 0.695
-0.5
Jsub_opt = 0.7095

-1
0 0.5 1 1.5 2 2.5 3 3.5 257 4
t
Let us look at the Hamiltonian matrix:
0 L
M
1 0 0 O
P
L
HM
A BR B1 T
O
M
P 0 0 0 1
P
N
Q AT QM1
M 0 0 0 P
P
N0 0 1 0 Q
The eigenvalues of H are:

R
S0.7071 j 0.7071
T0.7071 j0.7071
symplectic
258
 eigenvalues of Hamiltonian matrix
0.8

0.6

0.4

0.2

0
Imag Axis
-0.2

-0.4

-0.6

-0.8
-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8
Real Axis

259
The stable eigenvalues are those of the sub-optimal
closed-loop system.

Ac
L
0 1OL
A BK M P
0O
MP1
L
2 M
0 1 O
P
N
0 0QN
1Q N1 2 Q
2 j 2
I2 A c 2 1 0
2

260
 Linear Quadratic Regulator Design Using MATLAB

LQR Linear quadratic regulator design for continuous systems.

[K,S,E] = LQR(A,B,Q,R) calculates the optimal feedback gain
matrix K such that the feedback law u = -Kx minimizes the cost
function:
J = Integral {x'Qx + u'Ru} dt
subject to the constraint equation: .
x = Ax + Bu
Also returned is S, the steady-state solution to the associated
algebraic Riccati equation and the closed loop eigenvalues E:
-1
0 = SA + A'S - SBR B'S + Q E = EIG(A-B*K)

261
Example
A=[0 1;0 0];B=[0;1];
Q=[1 0;0 0];R=[1];
[K,S,E]=lqr(A,B,Q,R)

produces:
K= 1.0000 1.4142

S= 1.4142 1.0000
1.0000 1.4142

E = -0.7071 + 0.7071i
-0.7071 - 0.7071i

which agrees with our previous analysis.

262
MATLAB Demo

263
 Constrained Input Problems
The Minimum (Maximum) Principle
We will now consider the situation when the control
signal u(t) is constrained to lie in an admissible
region:
u(t ) U , e.g. ai ui (t ) bi (bounds)
Consider the general problem:

u (t )
min J (to ) x(T ), T L( x, u, t )dt
T

to
s.t. x f(x, u, t ), x(to ) given
u U
x(T ),T 0 264
The procedure is similar to before, but with a
significant alteration of the stationarity condition
which now becomes the optimality condition:

H (x , u , , t ) H (x , u, , t )
* * * * *

for all admissibleu(t ) U , where * denotes

an optimal value.
Note: The condition Hu =0 still applies if u is inside
the region, but not when any component of u is on a
bound.
This is the MINIMUM PRINCIPLE.

265
L.S. Pontryagin
Summary of Necessary Optimality Conditions
When the Input is Constrained

System description
System model:
x f (x, u, t ), u U , t to , to fixed, x(t o ) given
b gz T
Performance index: J (to ) x(T ), T L(x, u, t )dt
to

Final state constraint: x(T ), T 0

266
Optimal controller
Hamiltonian: H ( x, u, , t ) L ( x, u, t ) T
f (x, u, t )

state equation: x H H f
T

H
T

Costate equation: H x
x

Optimality: H (x* , u* , * , t ) H (x* , u, * , t ) , u* U

Boundary conditions:
x(to ) given
(x Tx )T dx(T ) (t Tt H ) dT 0
T T

x(T ), T 0
267
Example {
min J z} T
dt
0

s. t. x1 x2 , x1 ( 0) x10
x2 u , x2 ( 0) x20
u( t ) 1
x1 ( T ) 0
x2 ( T ) 0
+1 u 1 x2 1
x1
-1 s s

Hamiltonian: H 1 1 x2 2 u
H 0
Co-state equations: 1 x1

H
268
2 x 1 2
Boundary conditions:
x1 (0) x10 , x1 (T ) 0
x2 (0) x20 , x2 (T ) 0
( x Tx ) T dx(T ) ( t Tt H ) dT 0
T T

H ( T ) 0 1 1 ( T ) x2 ( T ) 2 ( T ) u ( T ) 0
2 ( T ) u ( T ) 1
Optimality condition: To minimise H we require 2u
1 uwhere:
to be as small as possible 1. Hence, if
2 is +ve then use u = -1; if 2 is -ve then use u = +1.
That is the optimal control is:

b g
u (t ) sgn 2 (t )
*

269
This is bang-bang control.
Solution of the co-state equations
1 0 1 (t ) constant = c1
2 1 2 (t ) c1t c2
Hence, 2(t) is linear and can change sign a
maximum of only once.
Optimal control: Since b g
u* (t ) sgn 2 (t ) the optimal
control is one of four choices, depending upon the
particular initial condition.
1) u(t) = -1 for all t
2) u(t) = -1 switching to +1
3) u(t) = +1 switching to -1
4) u(t) = +1 for all t 270
phase-plane trajectories
x1 x2 , x1 (0) x10
x2 u , x2 (0) x20 , u 1
Hence:

dx2 u x x 2 2
x2 dx2 u. dx1 u( x1 x10 )
2 20
dx1 x2 2

which is the equation of a parabola passing

through (x10,x20). Hence, the phase portrait will be
a family of parabolas, one for each initial
condition.
271
 u = +1
5

0
x2

-5
-5 0 5
u = -1
5

0
x2

-5
-5 0 272 5
x1
 switching curve
u=+1
5

0
x2 u=+1 x10,x20
-1
u=-1
-2
switching
curve
-3

-4
u=-1
-5
-5 0 273 5
x1
The only way to arrive at x1(T) = 0, x2(T) = 0, starting
at (x10,x20), at the particular point shown, is to first
apply u = -1. When the state arrives at the switching
curve, the control is then switched to u = +1 which
then drives the state to the origin. (Note that if
(x10,x20) happens to be on the switching curve the u(t)
would remain at +1 or -1 for all t)

The equation of the switching curve is:

x1 x2 x2
1
2

274
giving the feedback law:

R 1 if x x x
|| or if x x
1
1
2 2 2
1
x2 and x1 0
uS 1 2 2

||1 if x x x
1
1
2 2 2

T or if x x 1
1
2 2 x2 and x1 0

+1 u x1
feedback 1 1
law -1 s x2 s

275
 phase plane responses
5

1
T = 11.21
0 start 1
x2 T = 6.324
-1

-2

-3

-4 start 2

-5
-15 -10 -5 0 5 10 276 15
x1
 responses for trajectory 1
10

5
x1
0
0 1 2 3 4 5 6 7
2
0

x2 -2
-4
0 1 2 3 4 5 6 7
1

0
u

-1
0 1 2 3 4 5 6 277 7
t
 responses for trajectory 2
0
-5

x1 -10
-15
0 2 4 6 8 10 12
5

0
x2
-5
0 2 4 6 8 10 12
1

0
u

-1
0 2 4 6 8 10 278 12
t
MATLAB Demo

279