Beruflich Dokumente
Kultur Dokumente
VARIATIONAL METHODS
MORMON M. DENN
OPTIMIZATION BY
VARIATIONAL METHODS
MORTON M. DENN
Associate Professor of Chemical Engineering
University of Delaware
Preface
The American Chemical Society for Figures 4.2, 5.7, 5.8, 9.1, 9.2, 9.7 to
9.13, 10.5 to 10.15, 11.8 to 11.12, which appeared in Industrial and
Engineering Chemistry Monthly and Fundamentals Quarterly.
Taylor and Francis, Ltd., for Figures 11.1 to 11.7 and Sections 11.2 to
11.7, a paraphrase of material which appeared- in International
Journal of Control.
R. Aris, J. M. Douglas, E. S. Lee, and Pergamon Press for Figures 5.9,
5.15, 9.3, and Table 9.8, which appeared in Chemical Engineering
Science.
x PREFACE
MORTON M, DENN
Contents
Preface vii
Introduction
OPTIMIZATION AND ENGINEERING PRACTICE 1
BIBLIOGRAPHICAL NOTES 2
1.1 Introduction 4
1.2 The Simplest Problem 4
1.3 A Variational Derivation 7
1.4 An Optimal-control Problem: Formulation 10
1.5 Optimal Proportional Control 12
1.6 Discrete Optimal Proportional Control 13
1.7 Discrete Optimal Control 15
1.8 Lagrange Multipliers 18
1.9 A Geometrical Example 21
A
X11 CONTENTS
2.1 Introduction 44
2.2 Solution of Algebraic Equations 45
2.3 An Application of the Newton-Raphson Method 46
2.4 Fibonacci Search 49
2.5 Steep Descent 52
2.6 A Geometric Interpretation 53
2.7 An Application of Steep Descent 55
2.8 The Weighting Matrix 57
2.9 Approximation to Steep Descent 59
APPENDIX 2.1 Optimality of Fibonacci Search 62
APPENDIX 2.2 Linear Programming 65
BIBLIOGRAPHICAL NOTES 68
PROBLEMS 70
3.1 Introduction 73
3.2 Euler Equation -73
3.3 Brachistochrone 77
3.4 Optimal Linear Control 79
3.5 A Disjoint Policy 91
3.6 Integral Constraints 84
3.7 Maximum Area 85
3.8 An Inverse Problem 86
3.9 The Ritz-Galerkin Method 88
3.10 An Eigenvalue Problem 90
3.11 A Distributed System 91
3.12 Control of a Distributed Plant 93
BIBLIOGRAPHICAL NOTES 96
PROBLEMS 97
CONTENTS xUI
BIBLIOGRAPHICAL NOTES
An outstanding treatment of the logic of engineering design may be found in
D. F. Rudd and C. C. Watson: "Strategy of Process Engineering," John Wiley &
Sons, Inc., New York, 1968
Mathematical simulation and the formulation of system models is discussed in
A. E. Rogers and T. W. Connolly: "Analog Computation in Engineering Design,"
McGraw-Hill Book Company, New York, 1960
R. G. E. Franks: "Mathematical Modeling in Chemical Engineering," John Wiley &
Sons, Inc., New York, 1967
INTRODUCTION 3
1.1 INTRODUCTION
We shall solve this problem in several ways. Let us note first that
the minimum has the property that
g(xl,x2j . . . ,xn) - g(xl,x2. . . . ,xn) 0 (1)
ax,
n))<0 Sxl < 0 (2b)
dx=x2-1=0 (9)
or
x= ±1 (10)
OPTIMIZATION WITH DIFFERENTIAL CALCULUS 7
and so the point x = -3 also satisfies Eq. (7) and is in fact the global
minimum, since
g(-3) = -1s3 < a(+1) = 0 (12)
notation o(e) refers to a term which goes to zero faster than e; that is,
In Eq. (1),
zaxz
Ex, 6x2
o(max I6x,I,1Sx21) =
2 [ax? (axl)e + 2 ax az
+ 2
(Sxz)' (3)
where a is a small positive constant and the partial derivatives are again
evaluated at z,, x2. If these partial derivatives both vanish,. inequality
(4) is satisfied trivially; if not, we may write
But a sum of squares can be nonpositive if and only if each term in the
sum vanishes identically; thus, if 21'and x2 are the minimizing values,
it is necessary that
a3 = 38 =0 (8)
ax, ax2
+ 2 ax (ax,) 2 + 2
ax49
, ax2
z
ax, 5x2 + a
z
22 (0x2) 2
The terms with arrows through them vanish by virtue of Eq. (8). If we
now set
ax, = Ea, axz = Eat (10)
where e is an infinitesimal positive number and a,, az are arbitrary finite
numbers of any algebraic sign, Eq. (9) may be written, after dividing
by E2,
2 2
1 828 o(E2)
2 V a`a'
ax; az + E2
>0 (11)
s-1;1 ,
and letting c ---' 0, we find that for arbitrary a;, a; it is necessary that
2 2
828
a;a; > 0 (12)
; 1f
ax; ax;
I
dy = F(y,u) (1)
x -y - y w= -(u-u..)
and noting that i = y and 0, we have, finally,
Ax + w x(0) = xo (5)
min & = 2 to (x2 + c2w2) dt (6)
Letting
x = x(nQ)
w ua = A
eAA)
(10)
c2A 2
C2 (1 _. eA.A)2
a=eAo
12 OPTIMIZATION BY VARIATIONAL METHODS
min 6(M)
IA+ M 2
(8)
OPTIMIZATION WITH DIFFERENTIAL CALCULUS 13
Condition (7) requires the negative sign, but it will also follow from con-
sidering the second derivative.
For a minimum we require d2g/dM2 > 0. This reduces to
1 + A2c2
-2 (A+M)$-
.
>0 (11)
We shall now consider the same problem for the discrete analog described
by Eqs. (11) and (12) of Sec. 1.4,
xn = ax.-1 + u. (1)
N
min F. = I (xn2 + C2un2) (2)
n-1
We again seek the best control setting which is proportional to the state
at the beginning of the control interval
u = mxn-1 (3)
so that Eq. (1) becomes
xn = (a + (4)
which has a solution
xn = xo(a + m)n (5)
It is clear then that a requirement for stability is
ja+ml < 1 (6)
For simplicity in the later mathematics it is convenient to substi-
tute Eq. (4) into (3) and write
un =
a
+ m xn = Mxn (7)
14 OPTIMIZATION BY VARIATIONAL METHODS
and
x = 1 a M xn-1 (1 a M )nxo
= (8)
&(M) 2 xo2 (1
2!1 Ma
2C2 2
1- 1 a
M)21]
(10)
M)2 -
As in the previous section, we shall assume that the operation is suf-
ficiently long to allow N --p -, which reduces the problem to
1 1 + M2C2
min &(M) = (11)
2(1-M)2-a2
Setting d3/dM to zero leads to the equation
or
so that the negative sign should be used in Eq. (13) and M is negative.
The stability condition, Eq. (6), is equivalent to
1 - M > dal (15)
1-M>
2
+1a2>-y2(1+a2)> al (16)
2C
where the last inequality follows from
a2 - 21a1 + 1'= (Ial - 1)2 > 0 (17)
a --- 1 + A4 u- C2
Q2
C2
(19)
M o (21)
1 CA2c2 ) xn (22)
(2)
n-1
If we know all the values of x,,, n = 1, 2, . . . , N, we can calculate u,;
from Eq. (1), and so we may equivalently seek
Next,
a&
xN-1 + C2(XN-1 - axN-Y) - aC2(XN axN-1)
axN-1
= xN-1 + C'u,v-1 - aC2 UN = 0 (6)
and the procedure of Eqs. (4) to (8) may be applied ip order from n
to n = 1 to find the optimal control,settings ul, u2, . . . , UN.
We note from Eq. (9) that the optimal control will always be a
proportional controller of the form
-1aM1
xa-i (10)
or, if x.00,
M.M.+1+1 C2 M.+1-M.-I = 0 (13)
OPTIMIZATION WITH DIFFERENTIAL CALCULUS 17
+4
(a2+C2/z
g(xl,x2) = 0 (1)
(3a)
ax + ax
o (3b)
T1 +Xax
We shall now obtain this result somewhat more carefully and more
generally. As before, we suppose that we somehow know the optimal
values x,, 22, and we expand £ in a Taylor series to obtain
Now, however, we cannot choose axe and axe in any way that we wish,
for having chosen one we see that the constraint, Eq. (1), fixes the other.
We can, therefore, freely choose one variation, ax, or ax2, but not both.
Despite any variations in x, and x2, we must satisfy the constraint.
Thus, using a Taylor series expansion,
(+X)Oxt+(+X)Oxt
a& cig a&
ax, + Xa x = 0 (7)
ax +X a1g=0 (10)
which is Eq. (3a). Equations (1), (7), and (10) provide three equations
for determining the three unknowns x1, f2, and X. We note that by includ-
ing the constraint in this manner we have introduced an additional varia-
ble, the multiplier X, and an additional equation.
It is convenient to introduce the lagrangian 2, defined
£(x1,x2,X) = 3(xl,x2) + X9(x1,x2) (11)
Equations (7) and (10) may then be written as
a.e aye
8x1 = axe
=0
t We have not treated the most general situation, in which the lagrangian must be
written
Act + A,&
.1
So-called irregular cases do exist in which No - 0, but for all regular situations No may
be taken as unity without loss of generality.
OPTIMIZATION WITH DIFFERENTIAL CALCULUS 21
but that would take us into the area of nonlinear programming and away
from our goal of developing variational methods of optimization, and we
must simply refer the reader at this point to the specialized texts.
The curve g = 0 forms an ellipse, while for each value of 3 Eq. (1) defines
a straight line. We seek to minimize & subject to constraint (2); i.e., as
shown in Fig. 1.3, we seek the intersection of the straight line and ellipse
leading to the minimum intercept on the x2 axis.
As indicated by the multiplier rule, we form the lagrangian
E = ax, + bx: + Xa(x1)' + 2a$x1x: + Xy(x2)2 - XQ2 (4)
We then find the stationary points
at =
J7 a + 21%axi + 27$x2 = 0 (5a)
21 .
az
ax,
= b + 2\8x1 + 2a7xs - 0 (5b)
while 8.c/ax = 0 simply yields Eq. (2). Equations (5) are easily ^olved
for x1 and x2 as follows:
ay - bf
x1 = - 2X(ay-0') (6a)
ba -a#
X2= - 2X(ay - $2) (6b)
X can then be obtained by substitution into Eq. (2), and the final result is
ay - b$
X1 = f (7a)
(ory - $')(ya2 + ab2 - 2ab$B)
ba - a{3 (7b)
zs =
(a'y - $') (ya2 + ab2 - 2ab$)
OPTIMIZATION WITH DIFFERENTIAL CALCULUS 23
The ambiguity (±) sign in Eqs. (7) results from taking a square root, the
negative sign corresponding to the minimum intercept (point A in Fig.
1.3) and the positive sign to the maximum (point B).
Since we wish to find the minimizing 2N variables x1, x2, . . . , xN, U1,
U2, . . . , UN subject to the N constraints (2), we form the lagrangian
N N
a-
axN
= xN + aN = 0 (4b)
aL
axn
=xn+Xn-aXn+1=0 n=1,2, ...,N-1 (4c)
XN+1 = 0 (5)
We then obtain
xn+l - axn - Mn+lxn+l = 0 (8a)
x. + C2Mnxn - aC2Mn+lxn+1 = 0 (8b)
where presumably xo and yo are given. We shall suppose that the state
of the stream leaving the last stage is of interest, and we wish to choose
u1, u2, ... , uN in order to minimize some function S(xN,yN) of that
stream.
This problem can be formulated with the use of 2N Lagrange multi-
pliers, which we shall denote by Xi, X2, . . . , AN, A1, A2, . . . , AN. The
lagrangian is then
N
' - &(xN,yN) +
n-1
N
+ I An#n(xn,xn-l,yn,ya-l,un) (2)
n-l
and the optimum is found by setting the partial derivatives of £ with
respect to xn, yn, un, Xn, and An to zero, n = 1, 2, . . . , N. At stage N
we have
a +XNxN+AN-=o a
aN (3a)
+ XX ON + AN =0 (3b)
yN a yN ayN
XN
auN
+ AN =0 (3c)
aUN
28 OPTIMIZATION BY VARIATIONAL METHODS
or, eliminating XN and SAN from (3c) with (3a) and (3b),
as a#N a*N 4 N aON C18 aON 491PN 49ikN 4N =0 (4)
ayN {auN azN auN azN axN auN ayN UN ayN>
For all other n, n = 1, 2, . . . , N - 1, we obtain
an aon+1 + A. + 1 An+l a_n+l -0
n axn + Xn+1
aX n
a0ft
axn axn
(5a)
an
w ay. + Xn+1
aW
ayn
+ A. aOn
ay. + An+1 04'n+1
ayn
-o (5b)
Xn
ayn -1- An au" = 0 (5c)
au+l
n
+ °n+1 a,"unn+l = o
Xn+1 (5d)
we see that Eqs. (5a) to (5d) make up four linear and homogeneous
equations for the four variables X., X.+1, An, An+1, and by a well-known
result of linear algebra we shall have a nontrivial solution if and only if
the determinant of coefficients in Eqs. (5a) to (5d) vanishes; that is,
OPTIMIZATION WITH DIFFERENTIAL CALCULUS 27
or
9= (5)
4n
If Eqs. (6a) and (6b) are substituted into the generalized Euler
equation of the preceding section and k, and k2 are assumed to be of
Arrhenius form [Eq. (4)], then after some slight grouping of terms we
obtain
1+ v9 ks(u,.)G'(y,.)
1 + 9«ks(u,)G'(y,) E'ki(u»)F(x*) + 1 +
Ezks(u.+,)G(y+.+,)
(8)
ER i
ex
1)^ Jkio [ks(u,.) 1 +
kso ki(u,1) 1 +
El ks(uw)G'(yw)F(x*+,) llcsi-a
19
+ E2'11 + O ks(u+.)G'(y,)]G(y,.+,)]
[B(xn,x 1,y (g)
OPTIMIZATION WITH DIFFERENTIAL CALCULUS 21
+ p8Nk1(uN)ka(uN)F(xN)G'(yN) = 0 (11)
The computational procedure to obtain an optimal design is then
to assume u1 and solve Eqs. (6a) and (6b) for x1 and yl, after which
Eqs. (10a) and (10b) may be solved successively for n = 2, 3, . .. , N.
Equation (11) is then checked at n = N and the process repeated for a
new value of u1. Thus only 2N equations need be solved for the opti-
mal temperatures and concentrations.
This is perhaps an opportune point at which to interject a note of
caution. We have assumed in the derivation that any value of U. which
is calculated is available to us. In practice temperatures will be bounded,
and the procedure outlined above may lead to unfeasible design specifi-
cations. We shall have to put such considerations off until later chap-
ters, but it suffices to note here that some of the simplicity of the above
approach is lost.
8= 1 6tn(xn-1,u,,) (2)
ft-1
The choice of optimal conditions u1, u2, . . . , uN for such a process is
easily determined using the Lagrange multiplier rule, but we shall obtain
the design equations as a special case of the development of Sec. 1.11.
We define a new variable yn by
yn - yn-1 - 61n(xn_1)un) = 0 yo = 0 (3)
It follows then that
8= YN (4)
10 OPTIMIZATION BY VARIATIONAL METHODS
and substitution of Eqs. (1) and (3) into the generalized Euler equation
[Eq. (7), Sec. 1.111 yields
air*-1au,.1 a6i,.
of*_1/sun-1 + ax.-1- 0f/au,1
n= 1,2, . . . ,N (5)
with the boundary condition
acRrv
- 0 (6)
atN
The difference equations (1) and (5) are then solved by varying ul until
Eq. (6) is satisfied.
Fan and Wangt have collected a number of examples of processes
which can be modeled by Eqs. (1) and (2), including optimal distribution
of solvent in cross-current extraction, temperatures and holding times in
continuous-flow stirred-tank reactors with a single reaction, hot-air allo-
cation in a moving-bed grain dryer, heat-exchange and refrigeration sys-
tems, and multistage gas compressors.
We leave it as a problem to show that Eq. (5) can also be used
when a fixed value xN is required.
Our problem, then, is to find the functions A(u), B(u), C(u), and
D(u) such that Eq. (4) holds as an identity, and so we must require that
the coefficient of each power of x._2 vanish, which leads to three coupled
differential equations
AABC' + A'C - AC') = 0 (5a)
A'BC' + B'C' + A'2BC + AA'B'C + A'B'C
- AA'BC' - A2B'C' - 0 (5b)
A'D'B + B'D' + A'BB'C + B'2 - ABB'C' - AB'D' = Or (Sc)
If we assume that A (u) is not a constant, Eq. (5a) has the solution
C(u) = a(A - 1) (6)
axe +a ax =
0 (3b)
Let us denote the optimum by fi* and the optimal values of xl and
xs by x*, x2*. If we change the value of the constant b, we shall cer-
tainly change the value of the optimum, and so we may write S* as a
function of b
&* = E (b) (4a)
and
_xs
xi = xi (b) = xi (b) (4b)
T h us
d8' _ as dxi + as dx2
(5)
WY - \axl/=, __,. db ax2J 1... 1 db
t Recall that the lagrangian is minimized for a minimum of a convex objective func-
tion if the constraint is linear.
OPTIMIZATION WITH DIFFERENTIAL CALCULUS 33
or
1 - (499/49x2)1;_;; dx2 /db
dx*1 _ (7)
db - (499/ax,)=,-_,.
(9)
That is, the Lagrange multiplier represents the rate of change of the
optimal value of the objective with respect to the value of the constraint.
If E has units of return and b of production, then X represents the rate of
change of optimal return with production. Because of this economic
interpretation the multipliers are often referred to as shadow prices or
imputed values, and sometimes as sensitivity coefficients, since they repre-
sent the sensitivity of the objective to changes in constraint levels.
A second related. interpretation may be developed in the context of
the one-dimensional processes considered in Sec. 1.13. We seek the
minimum of
N
S= n(xn-1,un) (10)
n-1
by choice of ul, u2, . . . , UN, where
xn = f*(xn_1,un) (11)
Now we can find the optimum by differentiation, so that
aS
=0 n = 1, 2, . . . , N (12)
au,.
or, since x,. depends on all u;, i < n, by the chain rule,
aS _ afn of-+1 afe
aun aun + 49x,,
496tH+1
aun
+ 0x,+1 axn aun +
+ MIN afn am. +/ a'?-+I + of-+1 +
axN-1
. . .
au,. aun ( 49x axn+1 ax,.
+ MIN . at-+1 afn =0 (13)
T axN-1 49x49 aun
34 OPTIMIZATION BY VARIATIONAL METHODS
and similarly,
as a6t a6tn+1 af.
aun_1 = ax._, + axn ax._,
which, together with Eq. (16), is the result obtainable from the Lagrange
multiplier rule. The multiplier is then seen to be a consequence of the
chain rule and, in fact, may be interpreted in Eq. (17) a-
X, = (18)
the partial az
derivative of the objective with respect to the state at any
stage in the process. This interpretation is consistant with the notion
of a sensitivity coefficient.
(xi)2[a+2#b/3-ay+7(bfl 0 (10)
JJJJ
xi = ±A
ay -M (Ila)
qq q
(a7 - N2) (ya2 + ab2 - 2ab#)
and, from Eq. (8),
p ba - as
x2 = ±A (11b)
(a7 - 02) (ya2 + ab2 - 2abfi)
which are the results obtained from the Lagrange multiplier rule.
In practice we would consider a sequence of problems of the form of
31 OPTIMIZATION BY VARIATIONAL METHODS
The function [x1/(X1 + t)]2N will be vanishingly small for small e and
large N when the constraint is satisfied and exceedingly large when it is
violated. It is thus an excellent penalty function for this type of con-
straint. Other functions may be constructed to "smooth" hard con-
straints as needed.
1, a,,yP = 0 (4)
pe0
This algebraic equation will have n roots, y1, Y2, . , yn, which we
-
or
y= -a±1/a2_# (8)
The particular solution xk(r) can be found from the method of undeter-
mined coefficients by the choice
xk(v) = A + Bk (13)
38 OPTIMIZATION BY VARIATIONAL METHODS
(18b)
(12+2+$)2+ 1 +2a+$
BIBLIOGRAPHICAL NOTES
Sections 1.8 and 1.3: The elementary theory of maxima and minima is treated in all
books on advanced calculus. The fundamental reference on the subject is
H. Hancock: "Theory of Maxima and Minima," Dover Publications, Inc., New
York, 1960
Useful discussions in the context of modern optimization problems may be found in
T. N. Edelbaum: in G. Leitmann (ed.), "Optimization Techniques-with Applications
to Aerospace Systems," Academic Press, Inc., New York, 1962
G. Hadley: "Nonlinear and Dynamic Programming," Addison-Wesley Publishing
Company, Inc., Reading, Mass., 1964
D. J. Wilde and C. S. Beightler: "Foundations of Optimization," Prentice-Hall, Inc.,
Englewood Cliffs, N.J., 1967
Sections 1.4 to 1.7: We shall frequently use problems in control as examples of applica-
tions of the optimization theory, and complete references are given in later chapters.
A useful introduction to the elements of process dynamics and control is
OPTIMIZATION WITH DIFFERENTIAL CALCULUS 39
See also
Appendix 1.1: Good introductions to the calculus of finite differences and difference
equations may be found in
T. Fort: "Finite Differences and Difference Equations in the Real Domain," Oxford
University Press, Fair Lawn, N.J., 1948
V. G. Jenson and G. V. Jeffreys: "Mathematical Methods in Chemical Engineering,"
Academic Press, Inc., New York, 1963
W. R. Marshall, Jr., and R. L. Pigford: "The Application of Differential Equations
to Chemical Engineering Problems," University of Delaware Press, Newark,
Del., 1947
H. S. Mickley, T. K. Sherwood, and C. E. Reed: "Applied Mathematics in Chemical
Engineering," 2d ed., McGraw-Hill Book Company, New York, 1957
PROBLEMS
1.1. The chemical reaction X --+ Y - Z, carried out in an isothermal batch reactor,
is described by the equations
U-kix - k,y
z + y + z - coast
If the initial concentrations of X, Y, and Z are ze, 0, and 0, respectively, and the values
per unit mole of the species are cx, cyy, and cz, find the operating time 8 which maidmizes
the value of the mixture in the reactor
IP - Cx(x(8) - xe] + CYy(8) + czz(8)
12. For the system in Prob. 1.1 suppose that ki and k, depend upon the temperature
u in Arrhenius form
B,)
ki .N kie exp (
For fixed total operating time 8 find the optimal constant temperature. Note the
difference in results for the two cases E, < E, (exothermic) and E, < E, (endothermic).
U. The feed to a single.etage extractor contains a mass fraction xe of dissolved solute,
and the extracting solvent contains mass fraction ye. The mass fraction of solute in
the effluent is x and in the exit solvent stream is y. Performance is approximately
described by the equations
x + oy - xe + aye
y - Kx
where K is a constant (the distribution coefficient) and a in the solvent-to-feed ratio.
The cost of solvent purification may be taken approximately as
C, -dy - ye)
Ye
OPTIMIZATION WITH DIFFERENTIAL CALCULUS 41
and the net return for the process is the value of material extracted, P(xo - x), less
the cost of solvent purification. Find the degree of solvent purification yo which
maximizes the net return.
1.4. The reversible exothermic reaction X Y in a continuous-flow stirred-tank
reactor is described by the equations
0 - cf - c - er(c,cf,T)
0 = Tf - T + OJr(c,cf,T) - eQ
where c denotes the concentration of X, T the temperature, and the subscript f refers
to the feed stream. r is the reaction rate, a function of c, cf, and 7', a the residence
time, J a constant, and Q the normalized rate of heat removal through a cooling coil.
For fixed feed conditions find the design equations defining the heat-removal rate Q
which maximizes the conversion, cf - c. Do not use Lagrange multipliers. (Hint:
First consider c a function of T and find the optimal temperature by implicit differentia
tion of the first equation. Then find Q from the second equation.) Obtain an
explicit equation for Q for the first-order reaction
Obtain explicit equations for a and d in terms of the experimental data. Generalise
to relations of the form
N
y= akfk(x)
x 0 1 2 3 4 5 6
f. dx = 0 i0j
42 OPTIMIZATION BY VARIATIONAL METHODS
Find the coefficients c,, c2, ... , cN which are best in the sense of minimizing the
weighted integral of the square of the deviations
N
mine = f ab P(x)[y(x) - Z C.O.(x)dx
n-l
Show that the sequence sin x, sin 2x, sin 3x, ... is orthogonal over the interval
0 < z < r with weighting unity. Find the coefficients of the first four terms for
approximating the functions
(a) y=1 0<x<r
(b) y =x 0 <x <T
r
x 0 < x <
(c) y = -2
T-x 2<x<
Compare the approximate and exact functions graphically.
1.7 The cost in dollars per year of a horizontal vapor condenser may be approxi-
m4ated by
C- 0,N-36D-1L-ss + #:N_6.2D0.'L-' + S,NDL + j94N-1."D-4 L
where N is the number of tubes, D the average tube diameter in inches, and L the
tube length in feet. 01, Bt, 03, and 04 are coefficients that vary with fluids and con-
struttion costs. The first two terms represent coat of thermal energy; the third, fixed
charges on the heat exchanger; and the.fourth, pumping costs: Show that for all
values of the coefficients the optimal cost distribution is 43.3 percent thermal energy,
53.3 percent fixed charges, and 3.33 percent pumping cost.
Show that the optimal value of the cost can be written
r, \f,\f,
C=C
\f=) 0)
where f,, f:, f,, f4 are respectively the fractions of the total cost associated with the
first, second, third, and fourth terms in the cost. [Hint: If A - aC, B - OC, and
a + A - 1, then C - (A/a)-(B/p)s.) Thus obtain explicit results for N, D, and L in
terms of the 0;. Solve for 01 - 1.724 X 106, 02 - 9.779 X 104, P = 1.57, Y4
3.82 X 10-1, corresponding to a desalinatign plant using low-pressure steam. (These
results are equivalent to the formalism of geometric programming, but in this case
they require only the application of the vanishing of partial derivatives at a minimum.
The problem is due to Avriel and Wilde.)
1.8. For the minimization of a function of one variable, &(x), extend the analysis of
Sec. 1a3 to obtain necessary and sufficient conditions for a minimum when both the
first and second derivatives vanish. Prove that a point is a minimum if and only if
the lowest-order nonvanishing derivative is positive and of even order.
OPTIMIZATION WITH DIFFERENTIAL CALCULUS 43
13.. Prove the converse of Eqs. (13) and (14) of Sec. 1.3, namely, that a quadratic
form
ax' + 20xy + 'yy'
is positive definite if a > 0, a8 > y'.
1.10. For the system described in Prob. 1.4 suppose that the cost of cooling is equal to
pQ. Find the design equation for the rate of heat removal which maximizes conversion
less cost of cooling. Lagrange multipliers may be used.
UL Prove that when a is convex (the hessian is positive definite) the minimum of E
subject to the linear constraints
8w-
f _. dE
r(T,E)
where 8,, is the holding time, xw the conversion in the stream leaving the nth bed, and
r( T, the reaction rate. In an adiabatic bed the temperature is a linear function of
inlet temperature and of conversion. Thus the conversion can be expressed as
8w - _,,dE F(xw-1
xwf T,,)
1=. i R(T,,E)
where T. is the temperature of the stream entering the nth bed. Obtain design
equations and a computational procedure for choosing 8w and T. in order to maximize
conversion in N beds while maintaining a fixed total residence time
N
9- 8w
n-1
This problem has been considered by Horn and Huchler and Aria.)
2
Optimization with Differential
Calculus: Computation
2.1 INTRODUCTION
The previous chapter was concerned with developing algebraic con-
ditions which must be satisfied by the optimal variables in minimizing
an objective. The examples considered for detailed study were some-
what extraordinary in that the solutions presented could be obtained
without recourse to extensive numerical calculation, but clearly this will
rarely be the case in practice. In this chapter we shall consider several
methods for obtaining numerical solutions to optimization problems of
the type introduced in Chap. 1. An entire book could easily be devoted
to this subject, and we shall simply examine representative techniques,
both for the purpose of introducing the several possible viewpoints and
of laying the necessary foundation for our later study of more complex
situations. Two of the techniques which we wish to include for com-
pleteness are not conveniently derived from a variational point of view,
so that in order to maintain continuity of the development the details
are included as appendixes.
µ
OPTIMIZATION WITH DIFFERENTIAL CALCULUS: COMPUTATION 45
If our kth approximation to the solution is x(h) and we suppose that the
(k + 1)st trial will be the exact solution, we can write the Taylor series
expansion of f (x(k+I)) about f(x(k)) as
f(x(kFI)) = 0 = f(x(k)) -+' f(x(k))(x(k+i) - xu)) + (2)
Neglecting the higher-order terms and solving for x(k+i), we then obtain
the recursive equation
x(k+I) = X(k) ._. f (x(j (3)
As an example of the use of Eq. (3) let us find the square root of 2
by solving
f(x) = x2 - 2 = 0 '(4)
'(4
_ i a II
x(m, where f (x) is different in sign from f (f), will result in divergence
from the solution.
For an optimization problem the function f (x) in Eq. (1) is the
derivative 6'(x) of the function 8(x) -which is being minimized. Equa-
-tion (3) then has the form
(k)
z(k+l) = X (k) - e(z(k) (6)
At the minimum, 6" > 0, so that convergence is possible (but not guar-
anteed!) only if 8" is positive for each approximation. For the minimi-
sation. of a function of several variables, 3(x1,x2, x.), the iteration
formula analogous to Eq. (6) can be demonstrated by an equivalent
...
development to be
wi;826(x1(k),x!(k),
- . . . ,xn(k)) _ 1 i=p
J-
ax, x, a'' - { o i p (8)
Okloe-E''I"xo pxo
(1 + Bkloe-81`)°)(1 + Bk2oe81'J") - 1 + 0kioe-E,'1u (3)
(6)
E1?
We shall not write down here the lengthy explicit relations for 6' and V.
For purposes of computation the following numerical values were
used:
x0= 1 yo=0
k10 =5.4X 1010 k20=4.6X 1017
Ei = 9,000 Eq = 15,000
p=0.3 6= 10
The first estimate of u was taken as 300, with the first estimate of v then
calculatectfrom Eq. (4a). Table 2.1 contains the results of the iteration
4a OPTIMIZATION BY VARIATIONAL METHODS
-0.1
-0.4
- 0.5
-0.6
300 320 340 360 380 400 Fig. 2.2 Objective function versus
u temperature for consecutive reactions.
OPTIMIZATION WITH DIFFERENTIAL CALCULUS: COMPUTATION 4!
6(x)
6(x2)
so that the region 361.8 < u < 400 is excluded. The remaining point is
u = 338.2, which is at 61.8 percent of the new interval 300 < u < 361.8,
and the point which is symmetric at 38.2 percent is u = 323.6076. The
process is then repeated, leading this time to the elimination of the region
on the far left. The first several eliminations are shown graphically in
Fig. 2.5, and the full sequence of calculations is shown in Table 2.2,
where many extra significant figures have again been retained. The
Table 2.2 . Successive Iterations of the Fibonacci search method to the final
Interval of uncertainty of the optimal temperature for consecutive reactions
No. of
Computa-
tlona Int .1Val 140.888E g(u0.088L) u0.818L &(u0.818L)
Ox1
Ox2 -W2I aS
87X2)11.12
(3b)
which satisfies Eq. (2), so that if w1 and w2 are small enough, the new
value x1 + Sxl, x2 + u2 will be a better approximation-of the minimum
than the old.
An obvious generalization is to choose
e& as (5a)
axl - wl2 ax2
Ox1 = - w11
for small enough w;,. All we have done, of course, is to define a set of
directions
s
which will ensure a decrease in the function 6 for sufficiently small dis-
tances. We have at this point neither a means for choosing the proper
weighting matrix of components w;, nor one for determining how far to
travel in the chosen direction. We simply know that if we guess values
x1i x2, then by moving a small distance in the direction indicated by Eq.
(7) for any positive definite matrix, we shall get an improved value.
bS = [
>h.3,
a
ax2 2..i.
ax2 (1)
i= 1,2 (4)
54 OPTIMIZATION BY VARIATIONAL METHODS
where A is a step size and w; is the same for each variable but changes in
value at each position.
The ratios
as/ax;
[(as/ax,), + (as/ax,),)i
will be recognized as the set of direction cosines for the gradient at the
point and Eq. (3) is simply a statement that the most rapid
change in 6 will be obtained by moving in the direction of the negative
gradient vector. A potential computational scheme would then be as
follows:
mXnE,ax, ! - aa,
Call the minimizing point the new (f,,,) An approximate value
of a might be obtained by evaluating 8 at two or three points and
interpolating or fitting a cubic in X. A Fibonacci search could be
used, though for an approximate calculation of this nature the num-
ber of function evaluations would normally be excessive.
3. Repeat until no further improvement is possible. In place of step 2
it might sometimes be preferable to use some fixed value w; = w
and then recompute the gradient.. If the new value of S is not less
than s(x,,fi,), the linearity assumption has been violated and w is
too large and must be decreased.
It must be noted that this gradient method will find only a single
minimum, usually the one nearest the surfing point, despite the possible
existence of more than one. Thus, the process must be repeated several
times from different starting locations in order to attempt to find all the
places where a local minimum exists.
A far more serious reservation exists about the method derived
above, which might have been anticipated from the results of the previ-
ous section. We have, as is customary, defined distance by the usual
euclidean measure
A= = 0x,), + (ox,)' (5)
Since we shall frequently be dealing with variables such as temperatures,
'concentrations, valve settings, flow rates, etc., we must introduce nor-
malizing factors, which are, to, a certain extent, -arbitrary. The proper
OPTIMIZATION WITH DIFFERENTIAL CALCULUS: COMPUTATION 55
and the direction of steep descent, and hence the rate of convergence, will
depend on the scale factors a and y. Moreover, it is quite presumptu-
ous to assume that the natural geometry of, say, a concentration-valve-
setting space is even euclidean. A more general definition of distance is
A2 = a(axi)' + 2$ axl 6x2 + 'y(ax2)2 (7)
(8c)
where
(ay aFi ' a&)2
p aFi aFi
D
={ - #) [y
2
(.axl) + a (WX2-
2S
axl
ax2] c
(Sd)
I aubt,
11
l0
-j
iOj
k
Si OPTIMIZATION BY VARIATIONAL METHODS
where
k,oe a,-ru, i = 1, 2 (3)
and
E2
0 (8)
1
Though they are easily computed, we shall not write down the cumber-
some expressions for a6/avl and a8/av2.
The values of the parameters were the same as those used previ-
ously in this chapter, except that 0 was set equal to 5 in order to main-
tain comparable total residence times for the one- and two-reactor
problems. The simplest form of steep descent was used, in which the
correction is based on the relation [Eq. (3) of Sec. 2.5]
vlmw = vlold - W1 a&i
(9a)
av1
Since the relative effects of vi and v2 should be the same, no scale factor is
needed and wl and w2 were further taken to be the same value w. Based
OPTIMIZATION WITH DIFFERENTIAL CALCULUS: COMPUTATION 57
Initial 9.358 X 10-0 300.0 9.358 X 10'' 300.0 .3340 1.749 X 1011 1.749 X 1011
1 1.843 333.1 1.842 333.1 0.6360 2.877 X 1010 2.427 X 1010
2 2.131 334.6 2.085 334.6 0.6472 1.771 X 1010 1.354 X 1010
3 2.308 335.9 2.220 335.4 0.6512 1.192 X 1010 8.158 X 10,
4 2.427 336.5 2.302 335.9 0.6530 8.434 X 100 5.067 X 100
6 2.511 337.0 2.353 336.1 0.6538 6.159 X 100 3.168 X 100
6 2.573 337.3 2.384 336.3 0.6542 4.602 X 100 1.959 X 100
7 2.619 337.5 2.404 336.4 0.6544 3.500 X 100 1.175 X 100
8 2,654 337.6 2.416 336.5 0.6546 2.701 X 100 6.623 X 100
9 2.681 337.8 2.422 336.5 0.8546 2.111 X 100 3.286 X 100
10 2.702 337.9 2.426 336.5 0.6547 1 867 X 100 1.139 X 100
11 2.719 338.0 2.427 336.5 0.6547 1.329 X 100 -2.092 X 101
upon the values of derivatives computed in the example in Sec. 2.3, this
weighting w was initially taken as 10-27, and no adjustment was required
during the course of these particular calculations. The initial estimates
of u1 and u2 were both taken as 300, corresponding to vi and V2 of 9.358 X
10-14. The full sequence of calculations is shown in Table 2.3.
Examination of the value of the objective on successive iterations
shows that the approach to values of I; near the optimum is quite rapid,
while ultimate convergence to the optimizing values of v1 and v2 is rela-
tively slower. Such behavior is characteristic of steep descent. Some
trial and error might have been required to find an acceptable starting
value for w had a good a priori estimate not been available, and some
improvement in convergence might have been obtained by estimating
the optimal value of w at each iteration, but at the expense of more
calculation per iteration.
This is an appropriate point at which to interject a comment on
the usefulness of the consecutive-chemical-reaction problem as a compu-
tational example, for we shall use it frequently in that capacity. The
examples done thus far indicate that the objective is relatively insensi-
tive to temperature over a reasonably wide range about the optimum,
a fortunate result from the point of view of practical operation. This
insensitivity is also helpful in examining computational algorithms, for it
means that the optimum lies on a plateau of relatively small values of
derivatives, and computational schemes basing corrections upon calcu-
lated derivatives will tend to move slowly and have difficulty finding the
true optimum. Thus, codlpetitive algorithms may be compared under
difficult circumstances. '
+ cc
8x.axiax;+ ... (2)
i-i,-1 8x
or, for compactness of notation, we may denote the components of the
gradient 88/8x by Gi and the hessian 8'8/(8x; 8x;) by H1,, so that
angle ABC provides data for crudely estimating the gradient, and if A is
the worst of the three points, the line with an arrow from A through the
centroid of the triangle provides a reasonable first approximation. Thus,
we can simply reflect the triangle about the line BC to obtain a new tri-
angle B1C in a region of lower average value of &.With only a single new
calculation the worst point can again be found and the triangle reflected,
leading to the triangle B21. The process is continually repeated, as
shown.
There are several obvious difficulties with this overly simple pro-
cedure. First, continuous reflection is not adequate, for it will result in
too slow convergence far from, the optimum and too much correction and
oscillation near the optimum. Thus, the point reflected through the cen-
troid should be moved a fractional distance a > 1 on the other side, and
if the new point is also the worst, the distance moved should then be
reduced by a fractional factor of 1/r < 1. Hence, the triangle will be
distorted in shape on successive iterations. In some cases the distortion
will cause the triangle to degenerate to 'a line, so that more than three
starting points will usually be needed. For n variables the number of
points would then be greater than n + 1. (The coordinate of the cen-
troid of N points is simply the sum of the individual coordinates divided
by N.)
380 P_
370
360
350
U2
340
330
320
310
I I _ I I I 1
Itera-
tion It, [t2 It t U2
t2
-c
Initial 35:3.0 0.0657
:330.0 322.0 358.0 0.384-1 878.0 370.0 0.0222
1 383.0 330.0 0.0657 322.0 358.0 0.3844 338.9 330.1 0.6431
2 302.4 3-51.6 0.5018 322.0 358.0 0.3844 3:38.9 1330.1 0.6431
3 302.4 351.6 10.5018 319.9 331.7 0.5620 :3:35.9 33(1. 11 0.6431
4 343.5 319.9 110.6055 319.9 331.7 0.5620 338.9:330.1 1 0.6431
5 343.8 319.9 10.6055 336.4 :326.6 0.6229 335.9 3:30.1 1 0.6431
6 :334.3 3:3'2.9 0.6402 336.4 326.6 0.6229 33,`5.3) 330.1 0.6431
7 334.3 332.9 336.7 334.1 0.65(18 33,5.9 330.1 0.6431
S 339.7 331.7 0.6450 336.7 334.1 0.11508 338.9 330.1 10.6431
9 339.7 331.7 0.6480 336.7 334.1 0.6508 337.9 3:34.4 0.6529
10 336.0 i
335.6 0.6518 336.7 334.1 0.6608 :337.9 334.4 0.6529
11 336.0 335.6 0.6518 337.0 1 335.5 0.6534 :337.9. 334.4 0.6529
12 338.2 I :3:34.6 0.6534 337.0 335.5 0.6534 337.9 334.4 0.6529
13 1338. 2 3:34.6 0.6534 337.0 335.5 10.6534 337.5 335.4 10.6538
14 338.2 334.6 10.6534 338.3 334.7 0.6537 3:37.5 { 335.4 1 0.6.538
F. = sup L. (1)
then
The notation sup in Eq. (1) stands for supremum, or least upper bound.
We use this instead of "maximum" because while L. will be able to
approach the upper bound F. arbitrarily closely, it will never in fact be
able to take on this value. Our development follows that of Bellman
and Dreyfus quite closely.
Clearly if we have made no observations, we can place the mini-
mum within a unit interval only if we have started with a unit interval,
so that Fo = 1, and since a single observation is of no use whatsoever in
placing the minimum, one observation is no better.than none and F1 = 1.
For n = 2 the minimum may lie either in the interval [0,x2] or [x1,L2], and
so neither of these may exceed unity. It is obvious, however, that each
of these intervals may be set equal to unity and the value of L2 maxi-
mized by placing x1 and x2 equal distances from the center of the interval
and as close together as possible; that is,
in which case
L2=2- (5)
OPTIMIZATION WITH DIFFERENTIAL CALCULUS: COMPUTATION 43
L. = (1 - 2) F,._2) (13)
xs = ( 1 - 2} (14)
so that the interval remaining after these two trials is as close as possible
to the largest possible interval with n - 2 evaluations left. SucF a place-
ment of points is consistent with our induction hypothesis. It follows,
then, that
F. = sup L. > (15)
64 OPTIMIZATION BY VARIATIONAL METHODS
and combining the inequalities (12) and (15), we obtain the desired result
F. +1' +)
2 i/5 2 2 /5 \ 2
(17)
F.
2 -VV/ 5
` 2 J (18)
Thus
2
= 0.618
Fn .-1+V5 (19)
(2)
-1
x;>0 j= 1,2, . . . n (3)
where the coefficients c; and a;, are constants and the notation (<, =, > )
signifies that any of the three possibilities may hold in any constraint.
This is the standard linear programming problem. We note that if there
should be an inequality in any constraint equation (2), then by defining
a new variable y; > 0 we may put
a,,x; ± y; = b; (4)
-1
Thus, provided we are willing to expand the number of variables by
introducing "slack" and "surplus" variables, we can always work with
equality constraints, and without loss of generality the standard linear
programming problem can be written
n
(14)
\72 2
which corresponds to eliminating w. Thus, we now use the Gauss-Jordan
procedure again to obtain y in the third equation only, the result being
-3'zx +u-34v- -%W = 10 (15a)
Z + 34v - 34w 15 (15b)
2x + y + 32w = 75 (15c)
-x -%v-Y2w+g= -390 (15d)
The basic feasible solution is then x, v, w = 0, u = 10, z = 15, y = 75,
a OPTIMIZATION BY VARIATIONAL METHODS
and the corresponding value of & - -390. There are no positive coeffi-
cients in Eq. (15d), and so this is the minimum.. In terms of the original
variables only, then, x = 0, y = 75, z = 15, and only two of the three
original inequality constraints are at equality.
It should be clear from this example how a general computer code
using only simple algebraic operations and data comparison could be
constructed. The details of obtaining the required starting basic feasible
solution for the iterative process under general conditions, as well as other
facets of this extensive field, are left to the specialized texts on the subject.
The interested reader should establish for himself that the one-at-a-time
substitution used in the simplex method is the required result from steep
descent when, instead of the quadratic-form definition of distance,
42 (16)
a
a sum-of-absolute-value form is used,
(17)
BIBLIOGRAPHICAL NOTES
Section t..5: Discussions of the convergence properties of the Newton-Raphson and
related techniques may be found in such books on numeripal analysis as
_
Section 2.4 and Appendix t.1: The derivation of the Fibonacci search used here is
based on one in
R. E. Bellman and S. E. Dreyfus: "Applied Dynamic Programming," Princeton
University Press, Princeton, N.J., 1962
OPTIMIZATION WITH DIFFERENTIAL CALCULUS: COMPUTATION a
which contains references to earlier work of Kiefer and Johnson. An alternative approach
may be found in
D. J. Wilde: "Optimum Seeking Methods," Prentice-Hall, Inc., Englewood Cliffs,
N.J., 1964
and C. S. Beightler: "Foundations of Optimization," Prentice-Hall, Inc.,
Englewood Cliffs, N.J., 1967
Sections 2.5 and 2.6: The development of steep descent was by Cauchy, althougA his
priority was questioned by Sarru8:
A. Cauchy: Compt. Rend., 25:536 (1847)
F. Sarrus: Compt. Rend., 25:726 (1848)
Some discussion of the effect of geometry may be found in the books by Wilds cited above;
see also
Section 9.8: The most powerful of the procedures for computing the weighting matrix
is probably a modification of a method of Davridon in
Section 2.9: The references cited for Sec. 2.8 are pertinent for approximate procedures
as well, and the texts, in particular, contain extensive references to the periodical
literature. The simplex-complex procedure described here is a simplified version
for unconstrained problems of a powerful technique for constrained optimization
devised by
M. J. Box: Computer J., 8:42 (1965)
It is an outgrowth of the more elementary simplex procedure, first described in this sec-
tion, by
W. Spendley, G. R. Hext, and F. R. Himaworth: Technometrics, 4:441 (1962)
PR03LEMS
2.1. Solve Prob. 1.3 by both Newton-Raphson and Fibonacci search for the following
values of parameters:
2.2. Obtain the optimal heat-removal rate in Prob. 1.4 by the Fibonacci search method,
solving the nonlinear equation for T at each value of Q by Newton-Raphson for the
following rate and parameters:
p (_ 40,000
r = 2.5 X 101 exp '0"') 2.0 X 10Tex\ c
J = 10' B = 10-2
OPTIMIZATION WITH DIFFERENTIAL CALCULUS: COMPUTATION 71
2.3. Derive Eqs. (7) and (8) of Sec. 2.2 for the multidimensional Newton-Raphson
method.
2.4. The following function introduced by Roscnbrock is frequently used to test
computational methods because of its highly curved contours:
& = 100(x2 - y')2 + (1 - x)2
Compare the methods of this chapter and that of Fletcher and Powell (cited in the
bibliographical notes) for efficiency in obtaining the minimum with initial values
x = -1, y = -1.
Solve each of the following problems, when appropriate, by steep descent, Newton-
Raphson, and the complex method.
2.5. Solve Prob. 1.7 numerically and compare to the exact solution.
2.6. Using the data in Prob. 1.5, find the coefficients a and t4 which minimize both the
sum of squares of deviations and the sum of absolute values of deviations. Compare
the former to the exact solution.
2.7. The annual cost of a heavy-water plant per unit yield is given in terms of the flow
F, theoretical stages N, and temperature T, as
300F + 4,000NA + 80,000
18.3(B - 1)
where
A =2+3exp(16.875- T)
\ 14.4
4(1 - s)
B =
0.6(1 - $)(c 8 - 1) + 0.40
0 =a (as)N+t+0-1
F
0
1,400
(The problem is due to Rosenbrock and Storey, who give a minimum cost of S =
1.97 X 10'.)
2.8. Obtain the optimal heat-removal rate in Prob. 1.4 by including the system equa-
tion for temperature in the objective by means of a penalty function. The parameters
are given in Prob. 2.2.
2.9. Using the interpretation of Lagrange multipliers developed in See. 1.15, formulhte
a steep-descent algorithm for multistage processes such as those in See. 1.13 in which
it is not necessary to solve explicitly for the objective in terms of the stage decision
variables. Apply this algorithm to the example of Sec. 2.7.
72 OPTIMIZATION BY VARIATIONAL METHODS
2.10. Solve the following linear programming problem by the simplex method:
min E = 6x1 + 2x2 + 3x3
30x3 + 20x2 + 40x3 2!34
x1+x2+x2=1
10x, + 70x2 < 11
X1, x2, x3 > 0
Hint: Find a basic feasible solution by solving the linear programming problem
min & = z1 + z2
30x1 + 20x2 + 40x3 - w1 + 21 a 34
X1 + x7 + x$ + Z2 - 1
10x1 + 70x2 + w3 - 11
xl, x2, x3, w1, W3, zl, ZS > 0
You should be able to deduce from this a general procedure for obtaining basic feasible
solutions with which to begin the simplex method.
2.11. Formulate Prob. 1.4 for solution using a linear programming algorithm iteratively.
3
Calculus of Variations
3.1 INTRODUCTION
Until now we have been concerned with finding the optimal values of a
finite (or infinite) number of discrete variables, x1, x2, . . . , x,,, although
we have seen that we may use discrete variables to"approximate a func-
tion of a continuous variable, say time, as in the optimal-ebnitrol problem
considered in Sees. 1.4 and 1.6. In this chapter we shall begin consider-
ation of the problem of finding an optimal function, and much of the
remainder of the book will be devoted to this task. The determination
of optimal functions forms a part of the calculus of variations, and we
shall be concerned in this first chapter only with the simplest problems
of the subject, those which can be solved using the techniques of the
differential calculus developed in Chap. 1.
explicitly on the value of x(t), the derivative ±(t), and t; that is,
5 = a(x,x,t) (1)
we see that &[X(t) + s, (t)] depends only on the constant e, since 2(t) and
n(t) are (unknown) specified functions, and we may write
t`i(E) = 10 ff(x + En, ± + En, t) dt (4)
We can differentiate under the integral 'sign in Eq. (4), noting that
dT al; dx Off dx
de = T. de + ax ae
(6)
ax '1 + ax ''
and
dP-
f I dt = 0 (7)
de Jo [()x_5 n +
We integrate the second term by parts to obtaint
e 496 . 85 0 e d ag
(8)
fo azndt = axnlo - fo n dtaxdt
t Note that we are assuming that off /a± is continuously differentiable, which will not
be the case if t(t) is not continuous. We have thus excluded the possible cases of
interest which have cusps in the solution curve.
CALCULUS OF VARIATIONS 75
Jo ax it ax)t)dt = 0 (10)
n(t) = w(t)
aT - d af
(TX dt
where w(0) = w(0) = 0 and w(t) > 0, 0 < I < 0, we obtain
z
ax ax) dt = 0
w(t) (12)
Io ax
The only way in which an integral of a nonnegative function over a
positive region can be zero is for the integrand to vanish idlintically,
and so we obtain
of da9=0 0<t<0 (13)
ax dt ax
which is known as Euler's differential equation.
If we now return to Eq. (9), we are left with
axy (ax n1 =
,_o
0
(14)
If x(9) is not specified, ,,(B) need not be zero and we may choose a func-
tion such that
n(0) = E1 I- (15)
ax 1 + f= ax e_o - (17)
and it follows that a5/ax must vanish at an end at which x(t) is not
specified.
We have thus obtained a condition for the minimizing function
equivalent to the vanishing of the first derivative for the simple calculus
problem: the optimal function 2(t) must satisfy the Euler second-order differ-
ential equation
d a5 a25 a=5 a2S a5 ( )
x 18
at ax = at ax + ax ax x + ax= = ax
with the boundary conditions
Finally, if we seek not one but several functions xl(t), xs(t), ... ,
. . . ,x,,,il,tt, . . . ±.,t), then the Euler
x (t) and if is a function 5(xl,x=,
equations (13) and (18) are written
d a5 as
I ax; - ax; = 0 i = 1, 2, . . . , n (23)
3.3 BRACHISTOCHRONE
Dividing out the common factoi m and multiplying both sides by ds/dt,
we find
1d ds 2 dx
2 dt dt) - g dt (2)
or, integrating,
ds
dt
= v2g(x - c) (3)
X,
!-arc length
X2
where the constant c is determined from the initial height and velocity as
C
(d),
= x(0) - 2- o (4)
We note that
ds2 = dx2 + dz2 (5)
or
ds = 1+
()2 dz (6)
Thus, substituting Eq. (6) into (3) and integrating, we obtain the totq l
time T to travel the path
2g T = 1 + (dx/dz) 2
dz (7)
I-=
- V x-c
This is the integral to be minimized.
Identifying Eq. (7) with our notation f Sec. 3.2, we find
+ x2
1 (8)
x - c
1 = const = (1 (9)
(x-c)(1+x2) 2b
or
z = k+b(1-+sinr) (13)
CALCULUS OF VARIATIONS 79
r+sini'
z=zl'+(z2-zl) _f+sinl'f 0<<3'f (14b)
I1 IZ
X=C
If. we solve for w in Eq. (1) and substitute into Eq. (2), we have
g jo [x2 + c2(± - Ax)2] dt (3)
or
if = Y2[x2 + c2(± -- Ax)2] (4)
or
(5b)
c (1 + A2c2)x
x(O) is given. Since x(O) is not specified, the second boundary condition
is obtained. from a9/ft = 0 at t = B, or
x-Ax=O att=B (6)
Equation (5) may be solved quite easily to obtain x(t), and hence
i(t) and the optimal control policy w(t). For purposes of implemen-
tation, however, it is convenient to have a feedback controller, where
w is a function of the present state x, for such a result will be independ-
ent of the particular initial condition x(0). Motivated by the results
obtained in Chap; 1, we seek a solution in the form of a proportional
controller
w(t) = M(t)x (7)
and Eq. (1) becomes
x = (A + M) x (8)
Differentiating once, we have
.t = Az + Mt + Mx = Mx + (A + M) 2x (9)
and this must equal the right-hand side of Eq. (5b). For x 96 0, then,
M must satisfy the Riccati ordinary differential equation
M + 2AM + M2 - =0 (10)
M -(A+ 0
/1 +A2c2) (15)
which is the result obtained in Sec. 1.5. Here we do not satisfy M(B) = 0
but rather x(B) = 0. We have again found, then, that the optimal con-
trol for the linear system with quadratic objective function is a proportional
feedback control, with the controller gain the solution of a Riccati equation,
and a constant for an infinite control time.
0 = ax (2)
that is, A reacting to form B with' some reverse reaction, then since there
are no inflow and outflow streams in a batch reactor, the conservation of
mass requires that the rate of change of the concentration of A, denoted
by a(t), be equal to the net rate of formation of A. That is
d = -r(a,b,u) (8)
where b is the concentration of B and u is the temperature. Since the
amount of B present is simply the initial amount of B plus the amount
formed by reaction, we have
f a(o)
ace)
da =9
r(a,u) -
(11)
We wish to choose u(t) to minimize 9, and we note that since a(t) will be
a monotonic function of time, when the solution is available, it will be
sufficient to know a if we wish to know the time t. Thus, we may con-
sider u as a function of a and write
(13)
An internal maximum cannot occur when E,' > Eg (an endothermic reac-
tion), since r > 0, and the maximum is attained at the highest value of u
allowable. This, too, is perhaps intuitively obvious. The only case of
interest, then, is the exothermic reaction, E2 > E.
The maximum of Eq. (13) can be found by differentiation to give
U
Ei - E; (14)
In p,E' [b(0) + (n2/nI)a(0) - (n2/n,)alnt
(PlEf a n,
U*
U(f)
U*
(5- + XG) -
at ax (if +
AG)J 'n2(t) dt +
fo c3x az (if + XG)172 to = 0
(7b)
Since ill(t) is arbitrary, we obtain, as.in Sec. 3.2, the Euler equation
d (9 =0
dcai(g+XG) -a (if+XG)
(8)
The constant Lagrange multiplier X is found from Eq. (2): For n "func-
tions xl(t), x2(t), ... , x,,(t) and m constraints
If, in particular, we choose the area between the function x(t) and the
axis x = 0 between t = 0 and t = 0, as shown in Fig. 3.4, for a curve x(t)
of fixed are length L, we then seek to minimize
E = - area = - fa x(t) dt (1)
with
Are length = 0
(1 + ±2)3`' dt = L (2)
TX
= r(t)x + q(t)x + n(t) (7b)
where ko satisfies any nonzero boundary specifications and the l4. are
members of a complete set which vanish at any boundary where x(t) is
specified. The minimum of S is then found approximately by choosing
the coefficients C1, C29. . . , CN.
Substituting Eq. (2) into Eq. (1), we obtain
6 x f0 5(#o + FC.O., Oo + t) dt (3)
S is now a function of C1, Cs, . . . , CN, and upon differentiating K with
respect to each of the C. and setting the result to zero
aC
f ax + ax '*) dt = 0 n = 1, 2, ... , N (4)
Here the subscript ap refers to the fact that the partial derivatives are
evaluated for the approximate solution. Equation (4) leads to N alge-
braic equations for the N coefficients after the integration is carried out,
and is generally known as the Ritz method.
An alternative form which requires less computation when dealing
directly with the solution of Euler differential equations is obtained by
integrating Eq. (4) by parts. Thus
d
fo k az d:
ax) 4,.(t) dl + az no (5)
and since either 4.. is zero (x specified) or 8ff/81 vanishes (natural bound-
ary condition), we obtain, finally, the Galerkin form
0 BS.D dW.P n= 1,2, ... N
fo ax ' dt ea~ } ¢.(t)dt=0 (6)
(12b)
and the coefficients are found by solving the linear algebraic equations
fo
2
R(C,,C2, .. ,CN,t)tn-'(1 - t) dt = 0
n = 1, 2, . . . , N (13)
In particular, for a one-term approximation
N = 1: R(C,,t) = -C, + (C2 - 1)t - Cit2 (14)
and the solution of Eq. (13) is C, = -0.4, or
N = 1: x(t) ;; -0.4(1 --- t) (15)
OPTIMIZATION BY VARIATIONAL METHODS
- x(t)
where Jo is the Bessel function of zero order and first kind. Table 3.1
compares the two approximations to the exact solution.
g[x(t,z)l = f o1 Jo [
()2
+
(+ l
0(x) i dt dz (1)
a
at at f o n
dt (5)
Since n is arbitrary except for the boundary conditions and obvious differ-
entiability requirements, we may set
z- 2-
n(t,z) = w(t,x)
1
- ate azz + 0'(x)] (7)
CALCULUS OF VARIATIONS u
where
w(0,z) = w(l,z) = tv(t,O) = w(t,1) = 0
(8)
w(t,z) > 0 for t, z * 0, 1
Thus,
Jol z- 2- 2
fr
Jol
((
w(t,z)
at2 + az2 - '(z) dl dz = 0 (9)
and it follows that x must satisfy the Euler partial differential equation
491X
2
x specified at
t = 0, 1
ate + az2 -
(x) = 0
z=0,1 (10)
r dt + x2 = u(t)
2 x2(0) = 0 (3a)
where
a2 cos a(l - z)
K(t'z) = e IT
cos a - (a/p) sin a
a
COS (1 - Z)Pi
+ 2a2
it (a2 -
Oil)
1 + 1 + P
cos +i
(5)
Thus,
f01
t;[u] = [xl (z) - fo K(6 - t, z)u(t) dt,2 dz (7)
de I.so =
-2f g x, (r) fo K(8 - t, z)n(t) dt dz
+ 2 Jot [ fo K(0 - r, z)u(r) dr, [fo K(9 - t, z),7 (t) dt] dz = 0 (10)
-f o
fo' K(O - 1, z)K(8 - r, z)u(r) dz dr] dt = 0 (11a)
or, setting the arbitrary function n(t) proportional to the quantity in
brackets,
foI
K(8 - t, z)x, (z) dz
B I
fo [ fo
K(0 - t, z)K(8 - r, z) dz] a(r) dr = 0 (11b)
Equation (11) is an integral equation for the function u(i). It is
simplified somewhat by defining
101
ow = K(8 - t, z)x; (z) dz (12a)
I
G(t,r) = 0
K(9 - 1, z)K(O - r, z) dz (12b)
BIBLIOGRAPHICAL NOTES
Sections 3.2 and 3.3: An extremely good introduction to the calculus of vairations by
means of detailed study of several examples, including the brachistochrone, is
G. A. Bliss: "Calculus of Variations," Carus Mathematical Monograph, Mathematical
Association of America, Open Court Publishing Co., La Salle, Ill., 1925
Other good texts on the calculus of variations include
N. I. Akhiezer: "The Calculus of Variations," Blaisdell Publishing Company, Wal-
tham, Mass., 1962
G. A. Bliss: "Lectures on the Calculus of Variations," The University of Chicago
Press, Chicago, 1946
0. Bolza: "Lectures on the Calculus of Variations," Dover Publications, Inc., New
York, 1961
It. Courant and D. Hilbert: "Methods of Mathematical Physics," vol. 1, Interacience
Publishers, Inc., New York, 1953
L. A. Pars: "An Introduction to the Calculus of Variations," John Wiley & Sons,
Inc., New York, 1962
Applications specifically directed to a wide variety of engineering problems are found in
It. S. Schechter: "The Variational Method in Engineering," McGraw-Hill Book
Company, New York, 1967
Sections 3.6 and 3.7: The texts cited for Secs. 3.2 and 3.3 are pertinent here as well.
Section 3.8: The general inverse problem may be stated: When is a differential equation
an Euler equation? This is taken up in the text by Bolza cited above and in
J. Douglas: Trans. Am. Math. Soc., 50:71 (1941)
P. Funk: "V'ariationsrechnung and ihr Anwendung in Physik and Technik," Springer-
Verlag OHG, Berlin, 1962
F. B. Hildebrand: "Methods of Applied Mathematics," Prentice-Hall, Inc., Engle-
wood Cliffs, N.J., 1952
Sections 3.9 and 3.10: The approximate procedure outlined here is generally known as
the Ritz or Rayleigh-Ritz method and as Galerkin's method when expressed in terms
of the residual. Galerkin's method is one of a number of related procedures known
as methods of weighted residuals for obtaining approximate solutions to systems
of equations. A review of such methods with an extensive bibliography is
B. A. Finlayson and L. E. Scriven: Appl. Mech. Rev., 19:735 (1966)
See also the text by Schechter cited above and others, such as
W. F. Ames: "Nonlinear Partial Differential Equations in Engineering," Academic
Press, the., New York, 1965
L. Collatz: "The Numerical Treatment of Differential Equations," Springer-Verlag
OHG, Berlin, 1960
L. V. Kantorovich and V. I. Krylov: "Approximate Methods of Higher Analysis,"
Interscience Publishers, Inc., New York, 1938
Sections 3.11 and 3.12: Distributed-parameter systems are considered in some detail
in Chap.' 11, with particular attention to the control problem of Sec. 3.12. The
derivation of the Euler equation used here for that process follows
Y. Sakawa: IEEE Trans. Autom. Contr., AC9:420 (1964)
PROBLEMS
3.1. A system follows the equation'
x- -x + u
Find the function u(t) which takes x from an initial value xo to zero while minimizing
E= (K + us) dt
IU
!t OPTIMIZATION BY VARIATIONAL METHODS
(time plus cost of control) where 8 is unspecified. Hint: Solve for fixed 0; then deter-
mine the value of 0 which minimizes E.
3.2. A body of revolution with axis of symmetry in the x direction may be defined as
one which intersects all planes orthogonal to the x axis in a circle. Consider such a
body whose surface in any plane containing the x axis is described by the curve y(x),
y(0) -0 y(L) -R
The drag exerted by a gas stream of density p and velocity v flowing in the x direction
is approximately
J - 4rpvi JOL y dx
dx)
Find the function y(x) passing through the required end points which makes the drag
a minimum.
3.3. S is & function of t, x, and the first n derivatives of x with respect to t. Find the
Euler equation and boundary conditions for
min & - foe T(x,z, ... x(R),t) di
3.4. A second-order process described by the equation
I+ax+bx - u
is to be controlled to minimize the error integral
r oe
min E - (x' + c'u') dt
Show that the optimal control can be expressed in the feedback form
u -Mix +M:x
and find the equations for M, and M2.
3.5. Obtain the Euler equation and boundary conditions for minimization of
dz Dx2-0
D- - h(x - xo) atz-0
Ddx -0 atz -L
where k, D, h, and xo are constants. Find the parameters in a cubic approximation
to the solution.
3.7. Using a polynomial approximation, estimate the first two eigenvalues of
Y+ax-0
CALCULUS OF VARIATIONS
LI -! J aQ I as v(x,y) dx dy
(The form of the approximation is due to Sparrow and Siegal. Numerical values of
the exact solution are given in the book by Schechter.)
(b) Formulate the flow problem in terms of minimization of an integral and use
the complex method to estimate values for A, B, and C.
4
Continuous Systems:
4.1 INTRODUCTION
In the previous chapter we investigated the determination of an entire
function which would minimize an objective, and we were led to the
Euler differential equation for the minimizing function. It is rare that
a problem of interest can be formulated in so simple a fashion, and we
shall require a more general theory. Consider, for example, a chemical
reactor which we wish to control in an optimal manner by changing cer-
tain flow rates as functions of time. The laws of conservation of mass
and energy in this dynamic situation are represented by ordinary differ-
ential equations, and the optimizing function must be chosen consistent
with these constraints.
We shall assume that the state of our system can be adequately
represented by N variables, which we shall denote by xi, x2, . . . , xx.
In a chemical system these variables might be concentrations of the per-
tinent species and perhaps temperature or pressure, while for a space
vehicle they would represent coordinates and velocities. In addition,
100
CONTINUOUS SYSTEMS: I 101
On the other hand, if the controls were to be set to bring x1, x2, . . . ,
xN to fixed values in the minimum time 8, we would wish to minimize
& = 0 or, equivalently,
±1 = f1(x1,x2,u1,u:) (14)
i2 = f2(xl,x:,u1,u1.) (lb)
me OPTIMIZATION BY VARIATIONAL METHODS
at
(xl + axl) - at- xl = 6x1
= fl(xl + bxl, 22 + 6x2, ul + aul, u2 + but) - fl(21,22,41,u2)(5a)
The last term in Eq. (6) must be added in the event that the total time 0
is not specified, and a change in the decisions requires a change in the
total process time in order to meet some preset final condition. B then
represents the optimal duration and 60 the change.
If we expand the functions fl, ff, and T at every t about their.respec-
tive values at that t when the decisions ul, u2 are used, we obtain
ai; = fo
/ `
a bxl -} ax, +
a bui + a' bug dt
ax, axe aul 0U2
\ + 60 + o(E) (7)
CONTINUOUS SYSTEMS: I 103
Sz2
af26xl+ af2Sx2+ af2Sul+ af2Su2+0(e) (8b)
ax1 ax2 aul 8U2
aul
+ (aul + Xi aul + X2 of l)
+
(a + X l Al
au2 + X. au:
af2)
sue dt + I _a Sa + o(e) (9)
au2
Integrating the terms X, ft, and X2 6±2 by parts gives, finally,
fIM-5
SS= +Xlafl+X2azi+Sxl
+ (az + X1 ofy + X2 auz + ^2) 6x2
(T11l
+ + Xl aui + X2 aui1 Sul
(C of l
sue dt
+ au, + l au2 + Z au2
of t )
Now, just as in Sec. 1.8, we find ourselves with terms which are at
our disposal, namely, the decision changes Sul(t), 5u2(t), and terms which
are not, the state variations Sxl(t) and Sx2(t). We therefore eliminate
these latter terms from the expression for a& by removing some of the
arbitrariness from the functions al(t), X2(t) and requiring that they satisfy
the differential equations
a-
afl aft
^1 = - axl - a l xl ,2 (h a)
axl
2 ag afl aft
Xl aZ2 - X2 (11b)
ax2 49x2
Note that we have not yet specified boundary conditions. The variation
104 OPTIMIZATION BY VARIATIONAL METHODS
8& is now
b8
= f a + Xl auafl '} X2 auafsl` 8161
g
au1 1
For the problem with fixed 9, Eqs. (13) and (17) provide a total of four
boundary conditions for the four differential equations (1a), (1b) and
(11a), (11b). When B is variable, conditions (16) and (17) provide five
conditions, four boundary conditions and a stopping condition.t With
these conditions we finally obtain an expression for 68 which., depends
only on the variations in the decision variables,
We now introduce into the discussion, for the first time, the fact that
we are considering an optimization problem. Thus, if the choices ut(t),
til(t) are those which minimize ,8, then for all variations fti(t), 5u2(t) it is
necessary that 8 increase, or S8 > 0. We consider in this section only
situations in which the optimal decisions are unbounded and we are free
to make any (small) variations we wish. In doing so we exclude a large
class of problems of interest (indeed, most!), and we shall return shortly
to considerations of constraints on the allowable decisions.
As in Chap., 1, we choose a particular set of variations which makes
our task easy. We set .
where e' is a small positive constant. Thus Eq. (18) of the preceding
section becomes
S8 = -E' for
s
,
ta5 X Z
lim o(e) = 0
t
t For the case # unspecified and both x,(B) and x2(6) unspecified we require an addi-
tional condition. We shall find later that this condition is 5 - 0.i-e
106 OPTIMIZATION BY VARIATIONAL METHODS
and dividing by e' and taking the limit in Eq. (2), we obtain
of
I0
foe
of l afz = afl af, _
Kau, + l aul + = aui) + 1 8U2 + X1 au= + X= au=/ j dt -0
(4)
Since this is the integral over a positive region of a sum of squares, we
can satisfy the inequality only' if the integrand vanishes identically
(except, perhaps, on a set of discrete points). We therefol'e conclude
that if ui and u2 are the unconstrained functions which cause.& to take on its
minimum value, it is necessary that
aui + X, aui + a:
aui .0. (5a)
^1 aH x2= - OH (7b)
Together with the first boundary condition [Sec. 4.2, Eq. (16)] this givesf
H = Hl,_, - 0 (11)
On the other hand, if x1j x2, or both are specified, Eq. (11) follows directly
from the remaining boundary conditions [Sec. 4.2, Eq. (16)].
We may summarize the results of this section in the following
statements:
The unconstrained functions ul(t) and u2(t) which minimize 6 make
the hamiltonian stationary$ for all t, 0 < t < 8. The hamiltonian is
constant along the optimal path, and the constant has the value zero
when the stopping time B is not specified.
This is a very weak form of what has come to be called Pontryagin's
minimum principle.
For the more general problem of N state and R decision variables
we require N multipliers, and the hamiltonian has the form
N
H=5+ Xnfn(x1)x2, . . ,xv,ul,uz,' . ,us) (12)
n-l
with canonical equations
x;=aaH
°f ; (13a)
OH afn
X. (13b)
ax;
ax nil
and the boundary conditions
X. = 0 x; free (14a)
a; unspecified x; fixed (14b)
t This is the miming stopping condition which we noted previously.
We shall find later that the hamiltonian is in fact a minimum at these stationary
points.
109 OPTIMIZATION BY VARIATIONAL METHODS
First, we must put the problem in the form of Eqs. (1) of Sec. 4.2.
To do so, we observe that for a given (or optimal) initial condition, x(0),
the function x(t) is uniquely determined by its derivative ±(t). Thus,
t may be taken as the decision variable. ' Furthermore, the etplicit
dependence of 9 on t may be removed by the simple guise of defining a
new state variable. That is, if we let x1 denote x, the problem can be
reformulated as
e
S= %(xl,u,x2) dt (2)
o
with
i1 = U (3a)
x2 = 1 X2(0) = 0 (3b)
We generally do not need Eq. (5b), but we should note that it requires
that X2 be constant if 5 is independent of t.
The condition that the hamiltonian be stationary is
aU=au+a1= 01= 015
+X) (7)
CONTINUOUS SYSTEMS: I 109
The necessary boundary conditions follow from Eq. (13) of Sec. 4.2 as
The reader familiar with classical mechanics may wish to relate the
hamiltonian defined here with the function of the same name employed
in mechanics. It will be recalled that in a system of N particles, with
masses ml, m2i . . . , mN and positions in a three-dimensional space x11,
. . . , xNl, xN2, xN3, the physical system is the one
x12, x13, x21, x22, x23,
which minimizes (or makes stationary) the action integral
N 3
z L-.----V(x11,x12)x1`3,
fo [s-li-1
L , xN1,xN2,xN3..11 (1)
where the first term is kinetic and the second potential energy. Renum-
bering, we may use a single-subscript notation
3N M2 ,_+,2
S = fo - V (x1jx2, . . . ,x3N) dt (2)
i-1 IJ
with m1 = m2 = m3, m4 = m5 = m6, etc.
no OPTIMIZATION BY VARIATIONAL METHODS
' - axi
aH aV
ax;
while the stationary condition is
(6)
aH
=0=ma-us +X; (7a)
au,
or
X;
(7b)
m;
(8)
and
3N
H= pit -- V (x1,x2,
2m;
. . ,x3N) (9)
i-i
A=V(A, - A) - kA (2)
k = koexp( T l! (3)
Here the subscript c refers to the coolant stream, p is the density, Cp the
Coolant
Feed stream
91.
O T
nX
Product stream
A, T
specific heat, U the overall heat-transfer coefficient times the cooling area,
AH the heat of reaction, and K a constant, defined as
K= 2CPCPC
U (5)
CPpT
Z (-AH)A1 (6d)
Thus
Z1 = V (1 - Z1) - kZ1 (7a)
k = ko exp - (-AH)A,
E'CPp 1
Z2 (8)
Let us now assume that the reactor has been designed to operate
at a stable steady-state condition Z18, Z25 and that we shall control the
reactor in the neighborhood of the steady state by adjusting the flow
rates q and q,. Let x1 and x2 denote variations about the steady state
in (dimensionless) concentration and temperature, respectively, and ul
and u2 in feed and coolant flow rates; that is,
x1 = Z1 - Zis X2 = Z2 - Z2s (9a)
ul = q - qs u2 = qc - qcs (9b)
where the subscript S refers to the steady state. Substituting into Eqs.
(7) and expanding the right-hand.sides in Taylor series about the steady
state, exactly as in Sec. 1.4, we obtain the following equations for the
dynamic behavior of the reactor in a neighborhood of the steady state
x1=x2=u1=u2=0:
x1 = a11x1 + a12x2 + b11u1 (10a)
z2 = a21x1 + a22x2 + b21U1 + b22U2 (10b)
CONTINUOUS SYSTEMS: I 113
ail = V + ks (lla)
E'CppksZ,s
(11b)
a12 (- AH) A! Z zsz
a21 = -ks (11c)
qs UKq,s E'C,pksZls
azz = + VC,p(1 + Kq,s) (-AH)A,Z2s2
(lld)
It is clear that Eqs. (10) will apply to a large-lass of systems besides the
reactor problem considered here. Values of the parameters which are
used for subsequent numerical calculations are collected in Table 4.1 in
consistent (cgs) units.
In some cases we shall choose to hold ui at zero and control only
with the coolant flow rate. We can simplify the form of the equations by
defining new variables yl and Y2, as follows:
x1 = -yl
all _ 1
xz =
ail Y1 a12
Y2
Here,
231 = -(all + a22) + [(all + a22)2 - 4(alla22 - alzazl)]; (19a)
2S2 = -(all + a22) - [(all + a22) 2 - 4(alla22 -. al2a2i)j/ (19b)
For the parameters in Table 4.1, S1 and S2 are negative and real. Equa-
tions (10) then become
yl = S1Y1 - M11u1 - M12u2 (20a)
y2 = 527,2 + M21u1 + M12u2 (20b)
where
alibi, + alzbzl + S2b11
(21a)
S,-St
M1, = alzb22
(21b)
Sl - SZ
M21 = M11 + bl1 (21c)
giving
y, = Y2 (4a)
02 = - a2y, - a,y2 + u (4b)
and the objective
g
2
fo (ci,y,2 + 2c,2y,y2 + c22y22 + c::u=) dt (5)
where
a, = - (a + a22) (6a)
a2 = a,2a2, (6b)
u = -a,2b22u2 (6c)
C C1 + C2
a,22 (6d)
all
C,2 -Cla,2
2 (6e)
_ C2
C22 (6f)
a,22
Cs
C21
a,22b2:2
(69)
aH ay=
),2
e
-c12y1 - C22y2 - Xi + a1X2 (8b)
U= -- X2
C33
If a solution of the form of Eq. (12) is to exist, Eqs. (13),and (15) must
be identical for all yl and y2. That is, the coefficients of yl in Eqs. (13a)
and (15a) must be identical, as they must be in Eqs. (13b) and (15b),
and similarly for y2. Thus, equating coefficients, we obtain the three
CONTINUOUS SYSTEMS: I
differential equations
m11 = m122 -I- 2a2C3311112 - C33C11 (16a)
*22 m222 - 2c33m12 + 2a1c3391122 - Ca3C22 (16b)
m12 1n121n22 - C33nL11 + a1C33m12 + a2C33m22 - C3012 (16c)
Equations (16) are the multidimensional generalization of the
Riccati equation, which we have considered several times previously.
For finite B a solution can be obtained numerically with boundary con-
ditions m11(B) = m12(8) = m22(0) = 0, while as O--- ao, a stable constant
solution exists. We shall consider this case, and setting the time deriva-
tives in Eqs. (16) to zero, we obtain
m12 = -c3sa2 + (c$32a22 + C32C11)S4 (17a)
m22 = -C33a1 + [C332a12 + C33C22 - 2c332a2
+ c3s(C332a22 + (17b)
Combining Eqs. (11), (12), and (17), we obtain the optimal control
3.00
2.90
2.80
2.60
2.50
2.40
2.30
I I i
2.20 I I 1 1
Z1 = (1 - Z1) - U, (1)
V
and we shall suppose that the objective is again one of keeping flue tua-
tions small, that is,
3 = 3 (Z1 - Z1a)1 + 3c2(k - ks) 2 (2)
CONTINUOUS SYSTEMS: f U9
and so the system equation becomes, after subtracting out the steady-
state terms,
i = - (3+ ks) x - (Zis + x)u (4)
or
U = c (Zis + x) (9)
so that the problem is again one of finding the solution to the multiplier
equation.
For the linear system of the previous section we were able to obtain
a solution for X proportional to x. We might look upon this as the lead
term in a series expansion and seek a solution of the form
X = mx + px2 .+. . . . (10)
Again, for simplicity, we shall take 0 ---> so that it will be possible to
obtain the solution with ni, p, . . . as constants.
Differentiating Eq. (10), we `find, using Eqs. (4) and (9),
- (m + 2px)
1 (l la)
while from Eqs. (7), (9), and (10),
\1
_ -x + V + ks l (mx }px2) +
m2x2Z>. s
c2
+ ... (11b)
P
(q + ks + mZ1s2 ) + m2Z1s = 0 ( 13 )
``
V c2 c2
M --
Z1s2 I V
+ k,, - (1 ks) + ZC2 ] (14)
t There is no obvious physical reason why k should be any less meaningful in the
integral than Z1, since it is the reaction rate, rather than the temperature, which
affects conversion.
CONTINUOUS SYSTEMS: I 121
controller defined by Eq. (17). As we shall see in the next chapter, the
overall policy is the one which we intuitively expect, namely, full cooling
or no cooling until the line defined by Eq. (16) is reached and then, non-
linear control as defined above.
If we follow the procedure of Sec. 4.4 and remove the explicit time
dependence by definition of a new variable, we can write Eqs. (1) and (2)
in autonomous form as
a1 = Ax, + D(x2) + u x1(0)" = xo (3a)
x2 = 1 x2(0) = 0 (3b)
2
(x12 + c2u2) dt (4)
Ia
or
(7b)
M + 2AM + M2 - c = 0 (Ila)
and therefore L(t) must satisfy
L + [A + M(t)]L + M(t)D(t) = 0 (llb)
Equation (Ila) is, of course, simply the Riccati equation of Sec. 3.4.
In order to establish boundary conditions for Eqs. (Ila) and (llb) we
shall assume that at some time t < 0 the disturbance vanishes and
remains identically zero.During this final period we have the problem
we have already solved, namely, a system offset from x = 0 with no
disturbances, and we know that the solution requires
M(9) = 0 or x(0) = 0 (12a)
The solution for M(t) is thus given by Eqs. (13) and "(14) of Sec. 3.4.
It follows then, from Eqs. (8b) and (9), that the proper boundary con-
CONTINUOUS SYSTEMS: I 123
M = -A - 1 le' (13)
L` - J1 C2
/
[-
cA2c= f~ }
L(t) _ -tA+ exp 1 c2 (r - t)]
\ c2 D(r) dr (15)
For bounded disturbances the integrand will effectively vanish for future
times greater than several time constants, c(1 + A2c2)-, and it is neces-
sary only to know (or estimate) the disturbance for that time into the
future. Indeed, disturbances often have the form of step changes,
D(t) = D = const 0 < t < nc(.1 + A2c2)- (16)
in which case Eq. (15) yields
L - 1+ A 2C2
1+A2c2)D (17b)
where the term c2u2 is intended as a penalty function to keep the rate
of change of control action within bounds. Because D is a Constant, we
can differentiate Eq. (1) once to obtain
z = Ax + it x(0) = 0 z(0) = D (3)
or defining xl=x,x2=z,w=u,
21 = x2 x1(0) = 0 (4a)
x2 = Arxe + w x2(0) = D (4b)
Fi = 2 (x12 + c2w2) dt (5)
0
The problem defined by Eqs. (4) and '(5) is precisely the one con-
If we let 0 - ao, then, by relating coefficients. the
-sidered in Sec. 4.7.
CONTINUOUS SYSTEMS: 1 125
That is, the optimal control is proportional to both the offset and the
integral of the offset. The integral mode is often referred to as reset.
Most industrial controllers employ proportional and reset modes.
The importance of the reset mode can be clearly seen by substitut-
ing the control action into Eq. (3). The system response is determined
by the equation
2+!i+-1X=0 x(0)=0 x(0)=D (8)
That is, the response is overdamped and without oscillation when c < j,
underdamped with oscillations for c > Y4, but. always decaying exponen-
tially to zero after the initial rise. Thus, there can be no steady-state
offset.
We then have
o f 1 C1xj
(8a)
aft af=
da=
jf
IYL au + 1 (\8u 0x111
= =h
+ au ax=f= + ax=
a=f1
_
(11)
+ d= au (au ax,f1 + au ax=f= + au= u = 0
CONTINUOUS SYSTEMS: I 127
+ 412
(12)
Equations (10) and (12) are linear homogeneous algebraic equations for
4,t and 4#t, and the condition that they have a nontrivial solution (Jj and
1r2 both not identically zero) is the vanishing of the determinant of
coefficients. Thus
Of, Of, aft aft 012 a2f2 a2ft 02f2
T (- au 5Z ax, ax, 49u,
aft 0fl Of,02ft 02fl aft Of,
a2fl
-au(rauaxt-au0xs+auaxtft+auaxef2+autu> =0
(13)
or, solving for u,
aft 02f, afl afl aft Ofl
au'au axt fl + aua2ft
axe f 2 - U ax au x2)
Of, 02f2 aft aft aft aft1
is =
au (_O'ft
au axlfl + au ax2f2 au axl au axt
(14)
Of, 02f2 aft atfl
auau2 -auau2
Equation (14), which has been called a generalized Euler equation,
is an ordinary differential equation for the optimal decision function u(t),
which must be solved together with Eqs. (5a) and (5b) for xt and X2-
A boundary condition is still required, and it is obtained by evaluating
Eq. (10) at t = 0 with the values of ¢l, 02 obtained from Eqs. (7c) and
(7d) :
the algebraic conditions needed to pass from Eqs. (10) and (12) to (13)
require that the number of state variables exceed the number of decision
variables by no more than 1, so that there will be at least as many Itb'mo-
geneous equations as multipliers.
u
vF(x,)G'(x2) I - E'1J
E2'G(x2) k io exp u (5)
u
E2-E,' t=B (6 )
In E2kzoG(x2) 1
1(y - c)E;kioF(xi) J
Equation (5) requires that the optimal temperature always decrease
in time or with reactor length. When E2 > Ei, a high temperature
favors the second reaction with respect to the first, and a decreasing
temperature makes sound physical sense, for it suggests a high tempera-
ture initially to encourage the reaction X, -- X2 when there is little X2
to react and then a low temperature in the latter stages in order to prevent
the decomposition of the valuable product X2. On the other hand, if
E' > Ez, a decreasing temperature profile contradicts the physical intui-
tion that since the reaction X, --> X2 is favored with respect to the
decomposition of X2, the highest possible temperature is optimal at all
times. In Chap. 6 we shall develop a condition analogous to the second-
derivative test of Sec. 1.3 which verifies this physical reasoning and
demonstrates that Eq. (5) defines an optimum only when E2 > E.
The procedure for obtaining the optimal temperature profile and
optimal profit is as described at the end of Sec. 4.11. The feed composi-
tions x,(0) and x2(0) are presumed known, and a value is assumed for
u(0). Equations (1) and (5) are then integrated simultaneously until
Eq. (6) is satisfied, and u(0) is varied and the procedure repeated until
the solution of Eq. (6) occurs at t == 0. Amundson and Bilous have
carried out such solutions for several cases.
130 OPTIMIZATION BY VARIATIONAL METHODS
BIBLIOGRAPHICAL NOTES
Sections 4.2 and 4.3: The derivation follows
J. M. Douglas and M. M. Denn: Ind. Eng. Chem., 57(11): 18 (1965)
CONTINUOUS SYSTEMS: I 131
The results obtained here are a special case of much more general ones derived in subse-
quent chapters, and a complete list of references will be included later. A funda-
mental source for Chaps. 4 to 8 is
L. S. Pontryagin, V. G. Boltyanskii, R. V. Gamkrelidze, and E. F. Mishchenko:
"Mathematical Theory of Optimal Processes," John Wiley & Sons, Inc., New
York, 1962
Section 4.4: Any of the texts on the calculus of variations noted in the bibliographical
notes for Chap. 3 will contain a discussion of the corner condition.
Section 4.6: The model of a stirred-tank reactor and an analysis of its transient behavior
are contained in
R. Aris: "Introduction to the Analysis of Chemical Reactors," Prentice-Hall Inc.,
Englewood Cliffs, N.J., 1965
This is also an excellent source of details on other reactor models used as examples through-
out this book.
Section 4.7: This section follows the paper by Douglas and Denn cited above. The basic
work i
R. E. Kalman: Bol. Soc. Mat. Mex., 5:102 (1960)
A more general discussion is included in Chap. 8, and an extensive survey of optimal
linear control is contained in
M. Athans and P. Faib: "Optimal Control," McGraw-Hill Book Company, New
York, 1966
The reader unfamiliar with the conventional approach to process control may wish-to
consult a text such as
D. R. Coughanowr and L. B. Koppel: "Process Systems Analysis and Control,"
McGraw-Hill Book Company, New York, 1965
'D. D. Perlmutter: "Chemical Process Control," John Wiley & Sons, Inc., New York,
1965
J. Truxal: "Automatic Feedback Control System Synthesis," McGraw-Hill Book
Company, New York, 1955
Section 4.8: The expansion technique for obtaining nonlinear feedback controls is due
to Merriam; see
132 OPTIMIZATION BY VARIATIONAL METHODS
PROBLEMS
4.1. The pressure-controlled chemical reaction A = 2B, carried out in a tubular
reactor, is described by the equation for the concentration of A
4(xo - x)=
z = -k,u 2xox - x + k2u!
(2xo - x)'
where x0 is the initial value of x, u is the pressure, and k, and k2 are constants. x(8)
is to be minimized. Obtain an algebraic equation for the theoretical minimum value
of x(8) in terms of e, k,, k2, and x0. For comparison in ultimate design obtain the
equation for the best yield under constant pressure. (The problem is due to Van de
. Vusse and Voetter.)
4.2. Batch binary distillation is described by the equations
z1--u
u
i2 = [x2 - F(x2,u)]
x1
Formulate the problem so that it can be solved by the methods of this chapter and
obtain the complete set of equations describing the optimum. Describe a computa-
tional procedure for efficient solution. (The problem is due to Converse and Gross.)
4.3. Consider the linear system -
x=u
X(0) = xa i(0) = yo
x(8) = 0 x(8) = 0
and the objective to be minimized,
1
min 8 = a U2(t) dt
2
(a) Find the unconstrained function u which minimizes 8 for a fixed 8.
(b) Examine the nature of the minimum 8 in part (a) as a function of 8 graphi-
cally. Comment on the sensitivity of the solution to changes in 8.
(c) Find the unconstrained function u which minimizes 8 for unspecified. B.
Comment on the significance of the solution in terms of the results of part (b). (The
problem is due to Gottlieb.)
4.4. Solve the control problem of Sec. 3.4 by the methoc of Sec. 4.7.
4.5. Consider the nonlinear system
i = f(x) + b(x)u
x(0) - xa x(8) - 0
where f and b are continuous differentiable functions of x. Show that the optimum
unconstrained function u which minimizes
8 = fo [9(x) + c'u=] dt
with c a constant and g a nonnegative continuous differentiable function of x, has th
feedback form
Here n is the neutron density, c the precursor concentration, u the reactivity, and the
constants A, y, a, and 0, respectively, the neutron generation time, the decay constant,
the power coefficient of reactivity, and the fraction of neutrons given off but not
emitted instantaneously. Initial conditions no and co are given, and it is desired to
bring the neutron density from no to ono in time 8, with the further condition that n(8)
be zero and the effort be a minimum,
e
minc =21 fo u'dt
Obtain the equations needed for solution. (The problem is due to Rosztoczy, Sage,
and Weaver.)
4.8. Reformulate Prob. 4.7 to include the final constraints as penalty functions,
(What is Obtain the equations needed for solution. Normalize the equations
with respect to ono and obtain an approximate solution with the approach of Sec. 4.8,
utilizing the result of Prob. 4.6.
4.9. Let I be an inventory, P production rate, and S sales rate. Then
1*-P-S
Assuming quadratic marginal costs of manufacturing and holding inventories, the
excess cost of production for deviating from desired, values is
where the a; are constants. Find an approximate feedback solution for the "control"
u which regulates the CO2 level by minimizing
& -2f
1
o
e
(x' + c'u') de
5.1 INTRODUCTION
We now generalize our discussion of systems described by ordinary differ-
ential equations somewhat by relaxing the requirement that the optimal
decision functions be unconstrained. Complete generality must await
the next chapter, but most situations of interest will be included within
thescope of this one, in which we presume that the optimal decision
functions' may be bounded from above and below by constant values.
Typical bounds would be the open and shut settings on valves, safety
limitations on allowable temperatures and pressures, or conditions
describing the onset of unfavorable reaction products.
We again assume that the state of the system is described by the N
ordinary differential equations
with boundary conditions from Eq. (13) or (16) of Sec. 4.2, depending
upon whether or not 0 is specified.
CONTINUOUS SYSTEMS: II 137
for sufficiently small e' and obtain the same result as before:
When the optimal decision u; (i = 1 or 2) lies between the constraints
u;. and u* and the hamiltonian is differentiable with respect to u;,
it is necessary that the hamiltonian be stationary with respect to u,
(aH/au; = 0).
Let us now consider what happens when tit = u*. For convenience
we set Sue = 0 for all t and Sul = 0 whenever ul 96 u,*. Because of the
constraint all changes in ul must be negative (Fig. 5.1), and so we have
Sul < 0 (g)
Let us make the particular choice
Sul=elaH<0 (9)
where we cannot set the algebraic sign of El since we do not know the sign
u,
_T8
8u, may be
Su, moy only positive or
be negative negative when
when u,=u*
8u, may only
be positive but > 0
when u,=u,*
But 0
eui2
u, * -------------1 j- - bue20
J
r
(3u > dt + o(f) > 0 (10)
or, equivalently,
it=x: (2a)
2==u (2b)
We shall suppose that the system is initially at some state x1(0), x:(0)
and that we wish to"choose the function u(t), subject to the boundedness
constraints
u*=-1<u<+1=u* (3a)
or
Jul < 1 (3b)
in order to reach the origin (x1 = x= = 0) in the minimum time; that is,
6=fu 1dt=0 (4a)
=1 (4b)
and since 0 is unspecified, the constant value of H along the optimal path
is zero. The canonical equations for the multipliers are
8H = 0 (6a)
and since four boundary conditions are given on the -state variables, the
boundary conditions for the multipliers are unspecified. Equation (6a)
has the solution
Xi. = cl = const (7a)
that case 1\x is also zero [Eq. (7a)] and the hamiltonian, Eq. (5), has the
value unity. Since we have already noted that the optimal value of H
must be zero, it follows that the necessary conditions for a minimum
can never be satisfied by a control function which is in the interior of
the allowable region for any finite time interval.
The only possibilities for the optimum, then, are u = +1 and
u = -1. A control system of this type, which is always at one of its
extreme settings, is known as a bang-bang or relay controller. A typical
example is a thermostat-controlled heating system. We note that the,
question of when to use u = + 1 or u = -1 depends entirely upon the
algebraic sign of X2, for when X2 is positive, the hamiltonian is made a
minimum by using u = -1 (-11\2 < + 11\2, X2 > 0), while when 1\2 is
negative, the hamiltonian is minimized by u = +1 (+11\2 < -11\2,
X2 < 0). Thus, the optimal policy is
u = - sgn X2 = sgn (cxt + c2) (9)
Here the sgn (signum) function is defined as
_ (10)
sgn y Iyl = { ±1 y<0
and is undefined when y = 0.
We now have sufficient information to solve the system differential
equations, starting at xx = x2 = 0 and integrating in reverse time, i.e.,
calling the final time t = 0 and the initial time - 0. The condition
H = 0 establishes that C2 = 1, and for each value of cl, - co < cx cc,
we shall define a trajectory in the xxx2 plane, thus flooding the entire
plane.with optimal trajectories and de6ning a feedback control law.: In
this case, however; the entire, problem can -,be solved more siipply by
analytical methods.
We note first that the argument of the signum function in Eq. (9)
can change sign at most once. Thus the optimal solution may switch
from one extreme value to the other at most onee.t During an interval
in which the optimal control policy is u = +1 the system equations (2)
become
±1 = x2 (11a)
is = 1 (llb)
or
x2=t+C: (12a)
'/ + c3)2 + (C4 -
xx = %t2 + Cat + C4 = 72(t
2
(12b)
2
t It can be demonstrated that for the time-optima! control of an nth-order dynamical
system with all real characteristic roots the number of switches cannot exceed 1. less
than the order of the system.
CONTINUOUS SYSTEMS: 11 1a,
X2
and at most one switch is possible. The only way in which initial states
below the dashed line can be brought to the origin in this manner is to
x2
X,
XZ
XI
use the control u = +1 until the resulting trajectory intersects the line
of Eq. (15) and then to switch to u = -1 for the remainder of the
control time. Similarly, initial states above the dashed line are brought
to the origin by employing the control action u = -1 until intersection
with the dashed line [Eq. (15)] followed by u = -1. This defines the
optimal feedback control policy, and only the switching curve [Eq. (15)]
ief required for implementation. The optimal trajectories are then as
shown in Fig. 5:5.
X2
X1
Thus,
X2(t) = cl(O - t) (3)
which cannot change algebraic sign, and therefore the optimal control
function, defined by Eq. (9) of Sec. 5.3, must always be +1 or -1, with
no switching possible.
Figure 5.6 shows the trajectories in the right half-plane. For start-
ing values above the dashed line x1 + 112x2Ix2I = 0 the line x1 = 0 can
be reached without switching only by using the policy u = -1. For
starting points below the line x1 + t3 x2Ix21 = 0, however, the x2 axis
can be reached without switching by using either u = + 1 or u = -1.
Thus, even in this simplest of problems, the minimum principle does not
lead to a unique determination, and the true optimum must be dis-
tinguished between the two candidates by other considerations.
In this case the true optimum can be determined analytically.
Setting u = ± 1 and dividing Eq. (2a) of Sec. 5.3 by (2b), we obtain the
equation for the tangent to each trajectory passing through a point
Thus, referring to Fig. 5.6, the line segments Q,P2 and Q,P, are equal in
magnitude. But integrating Eq. (2a) of Sec. 5.3,
8 = x2(8) - x2(0) = QOQ2 u = +1 (5a)
B = x2(0) - x2(8) = QoQ1 u = - 1 (5b)
and, by inspection,
QOQ2 > QOP2 = QOP1 > Q0Q1 (6)
Thus, u = -1 leads to the shorter time in the entire right-hand plane..
By similar reasoning, when x, < 0, the optimal policy is u = +1.
where the parameters S1, S2, M11, M12, and M21 are defined in Sec. 4.6 by
Eqs. (19) and (21). In this section we shall consider the problem of
returning the system from some initial state y1(0), 1/2(0) to the steady
state y1 = Y2 = 0 in the minimum time by choice of the functions u1(t),
u2(t), subject to the operating constraints on the flow rates
u,. < u1 < u; (3a)
U2* < u2 < us (3b)
Substitution of Eqs. (6) and (7) demonstrates that this.equality can hold
for more than a single instant only if S1 = S2, a degenerate case which
we exclude. Thus, the optimal u1 must always lie at a bound, u1. or
ui , and the"same may easily be shown true for u2. The coefficient of u1
in Eq. (4) is -X1M11 + X2Mf21, so that the hamiltonian is minimized with
respect to u, by setting u1 equal to the smallest possible value when the
coefficient is positive and the largest possible value when the coefficient
is negative, and similarly for u2:
ui M21X2 < 0
U, = (8a)
u,. -M11A1 + M21A2 > 0
u2 1112(X2 - X1) < 0
U2 = (8b)
u2 M12(X2 - X1) > 0
For the parameters listed in Table 4.1 we have S1 < S2 < 0,
M21 > M11 > 0, M12 < 0. Using Eqs. (6)f we can then rewrite the
optimal control policy in Eqs. (8) as
u1 11> 0
u1 =
X20 (r A go
(9a)
u1. X20 (r ecs,-s,I: - 1 1 < 0
X 10
io
- X20 020 1 > 0
U2 = (9b)
U2* X20 (\X20
X10 ecs2-s,1t -11< 0
//
where
r=Mzi<1 (10)
146 OPTIMIZATION BY VARIATIONAL METHODS
are both always negative, and depending on the sign of X20, it folldws
from Eqs. (9) that the optimal control must always be either the pair
(u; ,u2.) or (u1*,u2 ), with no switching possible. These pairs may also
occur when A1o and A2o have the same algebraic sign with A1o/X20 > 1 but
only when t is sufficiently large for (A1o/1\20)e(s; s=)1 to be less than unity.
If 1 < A10/A20 < 1/r, the initial policy is (ul.,u2.) if A20 > 0, fol-
lowed by (u1.,ug) if the origin has not been reached after a time 't2 such
that
A1° 1=0 (11 a)
X20
t2 _ I Ago
(llb)
S,2 S11n A1o
0.02 \ a04
otherwise these trajectories could not reach the origin by an. optimal
sequence. Similarly, the line y_+, corresponding to (u1.,u2 ), must be
the switching curve for trajectories with control (ul+,u2*)
By choosing points on the y+_ curve and solving Eqs. (2) with the
constant control policy (ui ,u2) for a time interval 1/(S2 - S1) In r we
obtain the curve where the optimal control must have switched from the
policy (u1.,u2 ), the y++ switching curve, and similarly for the y__ curve.
We obtain in this way a line for the y++ (or y__) curve which stops short
a finite distance from the origin, for we have seen that we can reach the
origin along an optimal trajectory prior to switching from (us,u= ). We
obtain the remainder of the y++ curve by setting x1, = x2r = 0 and
(u1,u2) = (ui ,u=) in Eq. (14), and similarly for y_ _. These switching
curves may fail to be smooth at the intersection of the two segments.
We have now divided the x1x2 plane into four sections, in each of
which the control action is completely specified, with the change in con-
trol indicated by reaching a boundary. We have, therefore, by deter-
mining the switching curves, constructed the optimal feedback control for
the time-optimal problem. The curves in Fig. 5.7 are calculated for the
values of the parameters given in Table 4.1, together with the constraints
-8<u1<+10=ui (15a)
u2.= -5<u2<15=?42 (15b)
while Fig. 5.8 shows the trajectories after transformation to the original
dimensionless concentration (Z1) and temperature (Z2) coordinates.
Only one switching was required for most trajectories, and initial con-
ditions for trajectories requiring more than one switching generally fall
too far frgm the origin in the Z1Z2 plane for a linearized solution to be
useful, in some cases generating trajectories which, lead to negative con-
centrations. It is interesting to observe that many of the optimal tra-
jectories which approach the y++ and y__ curves do so with a common
tangent.
At this point it is useful to note again an alternative method of
solution which is well suited to automatic digital computation. We
make use of the fact that for problems with unspecified total operating
times the hamiltonian has the value zero. Thus, when the origin has
been reached, from Eq. (4),
X2(e)
_- X 1(6)[M11u1(8) + M12u2(B)} - 1 (16)
M21u1(e) + M12u2(e)
If we specify the final values u1(6) and u2(8), Eq. (16) definer a unique
relation between A1(8) and X2(8); for some range of values of A1(O) this
relation will be consistant with the requirements of Eq. (9) for the choice
of u1i u2. For example, the final policy (ui,us) requires, after some
CONTINUOUS SYSTEMS: (I 149
3.00
290
2.80
N2.70
2.60
a
E
r°- 2.50
2.40
2.30
I (
2.20 i 1 1 1
The constants are defined in Sec. 4.6, with numerical values given in
Table 4.1.
In order to avoid complication we shall assume that the flow rate q
is fixed at qs and that control is to be carried out only by varying the
coolant flow rate q. subject to the bounds
qes+u: <qc <_ qcs+U:. (3)
where the numerical values of U2. and u: are the same as those in the
previous section. The hamiltonian for time-optimal control is then
(1 +k) x1-ka,
aH E'C,pkZI
X: = - az:. (-AH)A fZ:=
[.1 UKga
E'C pkl, 1 X, (5b)
+ V + 1'Dp(1 + Kq.) (-LH)AfZ:=
CONTINUOUS SYSTEMS: II 151
Because of the nonlinear nature of Eqs. (1) and (5) analytical solutions
of the type employed in the previous section cannot be used to deter-
mine the maximum number of switches or the switching curves. It is
to be expected that in a region of-the steady state the behavior of the
nonlinear system will approximate that of the linearized system, so that
a first approximation to the switching curve can be obtained, by setting
q. equal, in turn, to its upper and lower limits and obtaining, respec-
tively, the curves y+ and y_, shown in Fig. 5.9. The optimal trajectories
can then be computed, using the policy us" above the switching curve,
us. below, and switching upon intersecting y+ or y_. Because of the
manner of construction of these curves no more than one switch will ever
be made. It should be noted that many of the trajectories approach the
y_ switching curve along a common tangent, as'in the linearized solution,
although this is not true of y+. No trajectories can enter the region of
negative concentration.
The verification of this solution must be carried out by the back-
ward tracing procedure described in the previous section. When steady
state has been reached, Eqs. (4) and (7) become
A choice of X2(8) uniquely determines q,(8) from Eq. (9), while Eq. (8)
determines X1(8). Thus, since x1(8) = x2(8) = 0, the four equations (1)
and (5) can be integrated simultaneously in the reverse time direction,
always monitoring the algebraic sign of (Z2S - Z,)X2. When this sign
changes, qe is switched to the other extreme of the range and the process
continued. This is done for a range of values of X2(e), and the locus of
switching points is then the switching curve for the feedback control sys-
tem. The trajectories and switching curves in Fig. 5.9 were verified by
Douglas in this way, except in the region where trajectories approach y_
along a common tangent, where extremely. small changes in X2(8) (of the
order of 10-6) are required to generate new trajectories.
It is important to recognize that the nonlinear differential equa-
tions (1) may admit more than one steady-state solution; in fact, the
parameters listed in Table 4.1 are such that three solutions are possible.
Thus, there is a separatrix in the x1x2 plane which is approximately the
line Z2 = 2.4, below which no trajectory can be forced to the desired
steady state with the given control parameters. The design steady state
is controllable only subject to certain bounded upsets, a fact which would
not be evident from a strictly linearized analysis.
3.00
E
r 2.60
2.40'
0 0.02 0.04 0.06 0.08 0.10
Composition r,
and seek the minimum time response to the origin. The hamiltonian is
then
H = 1+Xixz-X2x1+X2u (4)
with multiplier equations
aH = Xz (5a)
i
aH = -al (5b)
Eqs. (2) is
u= +1: (x,-1)2+(x2)4=R2 (9a)
a series of concentric circles centered at x, = 1, z2 = 0, while for u = -1
the integral is a series of circles centered at x, -1, x2 = 0:
u = -1: (x1 + 1)2 + (x2)2 = R2 (9b)
All trajectories must then lie along segments of the curves shown in Fig.
5.10, and since the control action changes after every v time units, a tra-
jectory can consist of at most semicireles. Thus, the approach to the
origin must be along one of the dashed arcs, which must also form part
of the switching curves, -y+ and -y-.
We can complete the construction of the switching curves by con-
sidering, for example, any point on y+. The trajectory leading to it
must be a semicircle with center at -1, and so the corresponding point
on the y_ curve can be constructed, as shown in Fig. 5.11. In this fashion
the switching curve shown in Fig. 5.12 is built up; with some typical tra-
jectories shown. Above the switching curve and on y+ the optimal con-
trol is u = +1, while below and on y_ the optimum is u = -1.
has complex roots. In that case a similar construction to the one above
leads to a switching curve of the type shown in Fig. 5.13 for the time-
optimal problem. We leave the details of the construction to the inter-
ested reader.
X2,
and we shall assume that the total operating time for control to the origin
is unspecified but that there is A. premium on both time and fuel. The
objective which we wish to minimize is then
l= -aH=0 (5a)
aH
xY
axY =
-al (5b)
or
Al = C1 = const (6a)
A2 = -clt + C2 (6b)
CONTINUOUS SYSTEMS: II 157
shown as the segments yo+ and yo- in Fig. 5.14. The switching curves
y-o from u = -1 to It = 0 and y+o from It = +1 tp u = 0 can be con-
structed analytically in the following way.
Let t2 denote the time that a trajectory intersects the switching
curve yo+ and t, the prior time of intersection with y_o. From Eqs.
(6b) and (9) we write
A201) = +1 = -c1t1 - c2 (Ila)
X2(t2) = -1 = -c1t2 - c2 (1 lb)
158 OPT`MIZATiON BY VARIATIONAL METHODS
X2
In the interval t,1 < t < t2 the solution of Eqs. (1), with u = 0, is
x2(12) = x2(tl) (15a)
x1(12) = x1(11) + x2(tl)(12 - tl) (15b)
But, from Eq. (10), the curve yo+ has the equation
x1(12) = 32x22(t2) (16)
and combining Eqs. (14) to (16), we obtain the equation for the switch-
ing curve y_o
We obtain the y+o curve in a similar way, with the entire switching curve
represented by
P 4
xl + p x2Ix21 = 0 (18)
CONTINUOUS SYSTEMS: II 15I
The two switching curves, together with typical trajectories, are shown
in Fig. 5.14.
Note that as p in which case only time is important, the
two switching curves coincide and the coast period vanishes, giving, as
expected, the time-optimal solution. As p -- 0, the y+o and y_o lines
tend to the x, axis, which we anticipate will be the fuel-optimal solution
(bang-coast). In fact the minimum fuel can be obtained by more than
one control action, and the one obtained here represents the solution
requiring the minimum time. Furthermore, the reader will note that
we are asking that the system move at zero velocity between states,
which is impossible, so that the limiting solution clearly does not exist,
but we can come arbitrarily close to implementing a fuel-optimal solu-
tion as p is allowed to become arbitrarily large.
5.9 A MINIMUM-INTEGRAL-SQUARE-ERROR
CRITERION AND SINGULAR SOLUTIONS
In Sec. 4.7 we studied the optimal control of the linear second-order
system, representing the stirred-tank chemical reactor, for an objective
in the form of an integral of squares of the state deviations and the
square of the control function, leading to a linear feedback control. The
u2 term in the objective may be rationalized as a penalty function to
keep the control action within bounds, but this goal can also be accom-
plished by the methods of this chapter. We thus return to the second-
order system with quadratic objective, but we shall now eliminate the
cost-of-control term from the objective and include bounds on the con-
trol action, the coolant flow rate.
After an appropriate change of variables the reactor equations
become Eqs. (4) of Sec. 4.7
yi = Y2 (la):
J2 = - a2y1 - a1y2 + u (lb)
where we have set ca, to zero, but we now seek the optimal function u
subject to the restriction
u* < u < u* (3)
The hamiltonian may then be written
H = i2(C11y12 + 2c12y1y2 + c22y22)
+ X1y2 - a2A2y1 - a1X2y2 + X2u (4)
160 OPTIMIZATION BY VARIATIONAL METHODS
It will now become clear why we; have always been careful to
examine each 'system for the possibility of an intermediate solution.
Setting all/au to zero, we obtain
aHasO (7)
au
which is the situation not covered by Eq. (6). But if X2 is zero for a
finite time interval, so must be its derivative and Eq. (5b) becomes
0=--aI -c1sy>-C::ys (8)
u = (as+Ciilyi+a1ys
\\ Call
(10)
where the positive sign corresponds to Eq. (12b). Combining Eqs. (13b)
and (14) and solving, we obtain
C ' (t - r)
yl = yl(r) exp ± Cft (15)
with the positive exponential corresponding to" Eq. (12b). Thus the con-
troller is unstable and clearly not a candidate for the optimum along line
(12b), since the objective will grow without bound, while the solution is
stable along the line (12a) and, hence, perhaps optimal. At all other
points in the y,y2 phase plane except this single line, however, the opti-
mal solution must be at one of the extremes, u* or u*. Indeed, only the
finite segment of the line (12a) satisfying
ZYYe
2c,20192 4- czsv22) dt > 0 (17)
vi
Integrating Eqs. (19a) and (19b) and subtracting, we make use of the
fact that of and yl are identical at both end points (we are comparing
the value of the objective along different paths between the same end
points) to establish Eq. (17):
y,-y," y1-yt11
y, - y," 2 y, ' y:"
(Cllyl + 2c12y,y2 + C22y 2) dt - 2 2
(Cllul + 201401472
Jy:y, _y,'
= yt
J
y,
Y,=yi
y,'
y, - y,"
y, - yi'
+ 022022) dl = f ( CSI yl + C22 Y2)2 dt > 0 (20)
y, - y,'
y=- y1
This proves that whenever it is possible to use the linear feedback con-
trol defined by Eq. (10)-i.e., whenever Eqs. (12a) and (16) are satisfied
-it is optimal to do so.
The remainder of the optimal policy can now be easily constructed
by the backward tracing procedure. At any point on the singular line
values are known for yl, y2, Xi, and A2
yl = yi (21a)
Y2 = Y2 (21b)
X1 = - cl2yi - c22y2 (21c)
_A2 = 0 (21d)
Equations (1) and (5) can then be integrated in reverse time for u = u*,
checking at all times to be sure that X2 < 0. When A2 again returns to
zero, we have reached a point on the y_+ switching curve and u is set to
u* and the process continued. In a similar way, setting u initially to u*
will generate a point on the y+_ switching curve when X2 returns to zero.
By carrying out this procedure for all points on the singular line the
entire switching curve can be generated, just as for the time-optimal
problems considered in previous sections.
A final word concerning practical implementation is perhaps in
CONTINUOUS SYSTEMS: II 1u
V
Z= = (Z, - Z2) - VCP(1 + Kq,) (Z2 - Z.) + kZ, (1b)
V
where k = k(Z2). Since q/(1 + Kq,) is a monotonic function of qc, we
may simply define a new decision variable w as the coefficient of Z2 - Z.
in Eq. (lb) and write
Z2 = (Z, - Z2) - w(Z2 - Z.) + kZ1 (lc)
y
_
where ks = k(Z2S). The choice of k, rather than Z2, in Eq. (3) is one of
convenience, though it can be rationalized by the observation that it is
deviations in reaction rate, not temperature, which adversely affect the
product composition. This is, of course, the objective chosen in Sec. 4.8.
The hamiltonian for this problem is `
X 2 = - 0 (k - ks) aZ2 +
ak
X1
ak q
Z 1 aZ2 + X2 V + X2w - X2Z1
ak
aZt
(5b)
These two equations, together with Eq. (1a),-are identical to Eqs. (1),
(7), and (8) of Sec. 4.8, and the approximate solution for the singular line
is given by Eq. (16) and the corresponding singular control by Eq.. (17)
of that section. Thus, the singular solution is equivalent. to choosing
the temperature Z2 which minimizes the objective, Eq. (3), provided that
the resulting flow rate is consistant with Eq. (2).
In the previous section we utilized a mathematical argument to
prove the optimality of the singular solution. Here, we simply rely on
physical reasoning. Clearly, if we are in fact free to specify the tem-
perature directly, we shall choose to do so, since this is our real physical
goal, and whenever the choice of flow rate coincides with the optimal
choice of temperature, that choice must also lead 'to the optimal flow
rate. Only when the optimal temperature is not accessible by choice of
CONTINUOUS SYSTEMS: it 16S
exp - T2 t (3)
where ao and bo are the initial values of reactant and product, respec-
tively. When E2 > Ei, there is a maximum value of r with respect to
1K OPTIMIZATION BY VARIATIONAL METHODS
The hamiltonian is
H = p - r(a,T) - Xir(a,T) + X,Jr(a,T) - a2u
= p - r(a,T)(1 + ai - JX2).- X2U (5)
with multiplier equations
as = a (1 + al:- At)
a,(e) = 0 (6a)
aH &
aT
= aT- (1 + X1 - JX2) x2(e) = 0 (6b)
11 =
u X2 > 0
(7)
0 1\ <0
with an intermediate solution possible only if X2 vanishes over a finite
interval.
O
Fig. 5.15 Reaction paths and switching
o° curve for optimal cooling in a batch or
tubular reactor. [After C. D. Siebenthal
and R. Aris, Chem. Eng. Sci., 19:747
(1964). Copyright 1964 by Pergamon
NM FG Press. Reprinted by permission of the
Temperature T copyright owner.]
CONTINUOUS SYSTEMS: II 167
r-o
and, after slight manipulation, the point C is defined by
ar/aa
u* > p J - (10)
ar/aT>
Since the integrand in Eq. (4b) is positive whenever the system is to the
right of r = p, an optimal policy cannot terminate at r = p to the right
of point C, for smaller values of S could be obtained by stopping earlier.
Thus, an optimal policy can terminate to the right of rmax only under the
policy u = u* on the segment AC of r = p.
In a similar way we find that the system can terminate to the left
of rmax only under adiabatic conditions (u = 0) on the pegment BA of
r = p, where B is defined by J > (ar/aa)/(a)-/aT). Thus, whenever the
system lies to the right of )'max and to the left of the full-cooling tra-
168 OPTIMIZATION BY VARIATIONAL METHODS
jectory AF, the optimal policy is full cooling until the intersection with
rmax, followed by the intermediate value of u necessary to remain on rmax,
for the system can never leave the region bounded by DAF, and this is
the policy which will minimize the term p6 in the objective, a(6) being
fixed. Similarly, to the left of rmax and below the adiabatic line DA the
policy is adiabatic, followed by the singular operation. Within the region
bounded by FACG the policy is only full cooling, and in EGAD only adi-
abatic, but even here some trajectories starting in regions r < p might
give positive values to the objective, indicating a loss, so that some initial
states will be completely excluded, as will all initial states outside the
envelope EBACG.
By combining Eq. (1) and Eq. (14) of Sec. 3.5 it easily follows that
the optimal policy on the singular line rmax is
BIBLIOGRAPHICAL NOTES
Section 5.2: The derivation follows
J. M. Douglas and M. M. Denn: Ind. Eng. Chem., 57 (11):18 (1965)
The results are a special case of more general ones derived in Chap. 6, where a complete
list of references will be included. A fundamental source is
L. S. Pontryagin, V. G. Boltyanskii, R. V. Gamkrelidze, and E. F. Mishchenko:
"Mathematical Theory of Optimal Processes," John Wiley & Sons, Inc., New
York, 1962
Sections 6.3 to 5.5: The time-optimal control problem for linear systems is dealt with in
the book by Pontryagin and coworkers and in great detail in
M. Athans and P. Falb: "Optimal Control," McGraw-Hill Book Company, New
York, 1966 it
R. Oldenburger: "Optimal Control," Holt, Rinehart & Winston, New York, 1966
Historically, the bang-bang result was obtained by Bushaw and further developed by
Bellman and coworkers and LaSalle:
R. Bellman, I. Glicksberg, and O. Gross: Quart. Appl. Math., 14:11 (1956)
D. W. Bushaw : "Differential Equations with a Discontinuous Forcing Term,"
Stevens Inst. Tech. Expt. Towing Tank Rept. 469, Hoboken, N.J., 1953; Ph.D.
thesis, Princeton University, Princeton, N.J., 1952; also in S. Lefschetz (ed).,
"Contributions to the Theory of Nonlinear Oscillations," vol. 4, Princeton
University Press, Princeton, N.J., 1958
170 OPTIMIZATION BY VARIATIONAL METHODS
J. P. LaSalle: Proc. Natl. Acad. Sci. U.S., 45:573 (1959); reprinted in R. Bellman
and R. Kalaba (eds.), "Mathematical Trends in Control Theory," Dover Publica-
tions, Inc., New York, 1964
The calculations shown here for the reactor problem are from the paper by Douglas and
Denn. Further considerations of the problem of nonuniqueness are found in
1. Coward and R. Jackson: Chem. Eng. Sci., 20:911 (1965)
A detailed discussion of all aspects of relay control is the subject of
I. Flugge-Lotz: "Discontinuous and Optimal Control," McGraw-Hill Book Company,
New York, 1968
Section 5.7: See the books by Pontryagin and coworkers and Athans and Falb.
Section 5.8: The book by Athans and Falb contains an extensive discussion of linear and
nonlinear fuel-optimal problems.
Section 5.1P The literature contains many examples of further applications of the prin-
ciples developed here, particularly such periodicals as A IA A Journal, Automatica,
Automation and Remote Control, Chemical Engineering Science, Industrial and
Engineering Chemistry Fundamentals Quarterly, IEEE Transactions on Auto-
matic Control, International Journal of Control, Journal of Basic Engineering
(Trans. ASME, Ser. D), and Journal of Optimization Theory and Applications.
The annual reviews of applied mathematics, control, and reactor analysis in Indus-
trial and Engineering Chemistry (monthly) list engineering applications. Some
other recent reviews are
M. Athans: IEEE Trans. Autom. Contr., AC11:580 (1966)
A. T. Fuller: J. Electron. Contr., 13:589 (1962); 15:513 (1963)
B. Paiewonsky : A IAA J., 3:1985 (1965)
Applications outside the area of optimal design and control are growing, particularly in
economics. A bibliography on recent applications of variational methods to economic
and business systems, management science, and operations research is
G. S. Tracz: Operations Res., 16:174 (1968)
PROBLEMS
5.1. Extend the minimum principle of Sec. 5.2 to nth-order systems
xi - fi(x1,x2, . . . xwrul,uir . . . 1 m 1, 2, . . . ,n
Show that the hamiltonian is defined by
H-5+ 1xif,
aH as " 8fi
axi axi 7
t ' axi
5.2. Solve the optimal-temperature-profile problem of Sec. 4.12 for the case of an
upper bound on the temperature. By considering the ratio of multipliers show how
the problem can be solved by a one-dimensional search over initial values of the ratio
of multipliers. For consecutive first-order reactions
F(x1) - x, G(x2) _= x,
obtain an algebraic expression for the time of switching from the upper bound to an
intermediate temperature in terms of the initial ratio of the multipliers. Discuss the
computational effect of including a lower bound on temperature as well.
5.3. Solve the optimal-pressure-profile problem of Sec. 4.13 for the case of an upper
bound on the pressure. By considering the ratio of multipliers show how the problem
172 OPTIMIZATION BY VARIATIONAL METHODS
can be solved by a one-dimensional search over initial values of the ratio of multipliers.
Discuss the computational effect of including a lower bound on the pressure as well.
5.4. Consider the system
i, - a,tx, + a12x2 + b,u
x2 = a21x, + a2222 + b2u
Iul < 1
Establish the number of switches possible in the minimum time control to the origin
for all values of the parameters and construct the switching curve for the case of
complex roots to the characteristic equation.
S.S. A system is termed controllable to the origin (xt - 0, z: - 0) if, for each initial
state, there exists a piecewise-continuous control u(t) such that the origin can be
attained in some finite time B. We have assumed throughout that the systems with
which we are dealing are controllable. For the linear system
21 - a12x2 + b,u
x2 = a2,21 + a22x2 + b2u
It is helpful to approximate arrival rates as constant during the latter phases of the
rush period in obtaining a completely analytical solution. (The problem is due to
Gazis.)
5.8. The chemical reaction X Y -+ Z is to be carried out isothermally in a catalytic
reactor with a blend of two catalysts, one specific to each of the reactions. The
describing equations are
i = u(ksy - kiz)
y = u(kix - k2y) - (1 - u)kay
x+y+z - const
Here u is the fraction of catalyst specific to the reaction between X and Y and is
bounded between zero and 1. Initial conditions are 1, 0, and 0 for x, y, and z, respec-
tively. The catalyst blend is to be specified along the reactor, 0 < 1 < 0, to maximize
conversion to Z,
max 61 - z(0) - 1 - x(8) - y(8)
(a) When k, = 0, show that the optimum is a section with u = 1 followed by
the remainder of the reactor with u = 0 and obtain an equation for the switch point.
(b)r When k2 $ 0, show that the reactor consists of at most three compartments,
the two described above, possibly separated by a section of constant intermediate
blend. Obtain the value of the intermediate blend and show that the time interval
(t1,t,) for intermediate operation is defined by
t1 = log
k1 + k2 \1 + k' + ksk,
k1
v k:
5.12. The chemical reactor described by Eqs. (7) of Sec. 4.6 is to be started up from
initial conditions Z,Q, Z20 and controlled to steady state Z,,, Zu by manipulation of
coolant flow rate q, and feed temperature Z, with bounds
0 < q. < q! Z,. < Z, < Z1
If heating costs are proportional to feed temperature, the loss in profit during start-up
is proportional to
&=Io,[(Z1.-Z.)+c(Z,-Z,.)1dd
6.1 INTRODUCTION
In the preceding chapters we have obtained and applied necessary con-
ditions for optimality in a wide variety of optimal-design and optimal-
control problems. Greater generality is required, and that is one aim
of this chapter. A more serious deficiency of the preceding work,
however, is that while the results are certainly correct, the motivation
for several important operations in the derivations is not at all obvious.
Thus we have little direction in attacking new problems by these methods,
and, indeed, we sense that our ability to devise efficient computational
algorithms is dependent upon our understanding of the logical steps in
a proof of necessary conditions.
There is, in fact, an underlying logic which can be applied to both
the theoretical and computational aspects of variational problems. The
logic is firmly grounded in the elementary theory of linear ordinary
differential equations, and the first considerations in this chapter will
of necessity be directed to a discussion of this theory. The resulting
175
176 OPTIMIZATION BY VARIATIONAL METHODS
I r = I (`)L
rkixi T
l kia. jx, +
(`n)
rkibi (10)
i-I i-Ii-i i-i
Integrating from t = 0 to any time and integrating the left-hand side
by parts, we obtain
inn
rki(r)bi(r) dr (11)
i -i
As for the first-order system, we remove most of the arbitrariness
from r by partially defining Fk,(t) as a solution of
n
where r,,(e,t) satisfies Eq. (12) with respect to t and the condition
rik(e,e) = bik (17)
We shall sometimes be interested in a linear combination of the
components x; of x at time 8. We define a vector y(t) with components
Ti(t) by
and
Yi = - 4 7,(t)a,i(t) (20)
i-1
As a simple illustrative example consider the linear second-order
system with constant coefficients
x + a1x + a2x = b(t) (21)
or, equivalently,
z1 = x2 (22a)
x2 = -a2x1 _ a1x2 + b2(t) (22b)
THE MINIMUM PRINCIPLE 179
The adjoint system, defined by Eq. (12), with boundary conditions from
Eq. (14), is
v11 = a2r12 T11 = 1 at t = 0 (23a)
F12 _ - r11 + a,r12 r12=0 att=0 (23b)
F21 = a2r22 r21 = 0 at t = 0 (23c)
V22 _ - r21 + a1r22 r22 = 1 at t = 0 (23d)
The solution is easily found to be
el4« 'u-e)
(ai - a,2 --4a2)
2 a12 - 4a2
exp[% a,2-4a2(t- 8))- (a1+ a12-4a2)
exp [-Y2 1/x12 - 4a2 (t - 0)) } (24a)
eSSa'(t-e)
r12(0,t) = - {exp [32 a12 - 4a2 (t - B))
a,2 - 4a2
- exp [-3z -\ a12 - 4a2 (t - 0))} (24b)
14«,cr-e)
a12 - 4a2 (t - B))
r21 (8,t) = ate { exp [Y2
1/a ,2 - 4az
-exp[-Y2 a12-4a2(t-0))} (24c)
exp (32
l
a12 - 4az 8) }
- fe eSS«,ce-e)
o ate - 4a2
(1111-
-l
af`axj+
8x;
'Yjaxi
k l
fauk+o(6)
auk
r
`.
n
181
(9
(10)
-ti = Ji(x,u)
0<t<8 (1)
i= 1, 2, . . . n
that the decision variables uk(t) satisfy inequality constraints
U,(u)>0 p = 1, 2, . . . P (2)
The constraint equations (2) to (4) are more general than those con-
sidered previously. We shall see later that the choice of the objective
&[x(e)] also includes the form of previous chapters as a special case.
We shall first assume that 0 is specified. If we specify u(t) and
5u(t) subject to the constraint equation (2) and to and Szo subject to
Eq. (3), we may write Eq. (11) of Sec. 6.3
i-1 i-1
(5)
Furthermore, q(8o +.Szo) and q(2o) must both equal zero, as must
g[!(0) + Sx(0)] and g[2(8)], so that
n
m = 1, 2, . .. , M (6)
9.[!(0) + &z(e)l - 9.[2(0)] 8x, axl(e) + o(E) = 0
s=1,2,...,S (7)
The change in & as a result of decision changes is reflected through a
change in x(9)
n
We have not yet specified the value of the vector y(9). Let us
write this as
71(e) = ax + .i (9)
49X,
at;
I
a-1
ag,
(12b)
and
M
where x; (0) and xw*,a are fixed numbers. In the former case
a&
Y.(0) = ax + V. (16a)
(17b)
and, finally,
n
as = I as
, axi
[axi(s) + fi as] + o(E) (17c)
Defining yi(B) by Eq. (12b), y;(0) by Eq. (13), and substituting Eqs. (5)
and (18) into (17c), we obtain
Finally, because of the extra degree of freedom resulting from the non-
specification of 0 we may set
with the boundary conditions Eqs. (3), (4), (12), and (13) of Sec. 6.4
and the additional condition
0 unspecified: H=0 at t = 8 11 (3)
If we now assume that u(t) is in fact the set of allowable functions which
minimize &, then, from Eq. (14) of the preceding section, we may write
e
rr
aS
= Jo 1 a u auk dt +
k-1
0 (4)
In the spirit of Sec. 5.2, we choose all variations but one to be zero and
that one as
a U, aH
Suk (8)
auk auk
From Eq. (7) auk and aUD/auk must have the same algebraic sign.
a
auk `auk
-)j
Substituting into Eq. (4), we obtain
all, k 0
(9a)
or
au°>0 (9b)
or, equivalently,
aukaH>0
auk - (10b)
.136 OPTIMIZATION BY VARIATIONAL METHODS
lY = 1 aHx;+
Z aHti.+ L aHuk (11)
ax,
i-1 ._1 a1 k-1 auk
The first two terms are equal with opposite signs as a result of the
canonical equations (2). When the optimal uk is unconstrained, all/auk
vanishes. When at a constraint for a finite interval, however, we might
move along the constraint surface with changing Uk if more than one
decision enters into the constraint. Thus, we cannot simply set &k to
zero at a constraint. However, when the constraint U, is equal to zero
(
for a finite time, its time derivative must vanish, so that
U' I
k-1 auk
uk = 0 (12)
H
i-1
THE MINIMUM PRINCIPLE 117
where
aH aH
z:=ayti ti:=-az;
with 2n boundary conditions
o) = 0 m = 1, 2, . . . , i if
9.[z(O)) =0 s 1, 2, . . . S
IV
aq-
7; (0) 1 7. i = 1, 2, n
m-t az,
as
Ti(0)=W S
Lg.
az,
i= 1, 2, n
ax,
we may write
&=foxodt=xo(9) (4)
aH
ax;
aT
-7o ax, - 4C7k ax,
afk
(6b)
k -1
rend
all
yo= -axo= 0 (6c)
as in Chaps. 4 and S.
It is also useful to consider cases .in which the independent varia-
ble t appears explicitly in the system equations or in the objective.
Here we consider systems
zi = f;(x,u,t) i= 1, 2, . . . ,n (9)
and objectives
8 = 6[x(9),91. (10)
with
n
aH
Yo=-axo -- i
7; It- (13a)
a&
0 unspecified
7o(B) = axo a8 (13b)
unspecified xo(O) = 0 specified
n
Clearly nothing is changed except that the sum yifi is not equal to
i-1
a constant when some fi depend on t, since H = const and 7o varies
with t according to Eq. (13a).
where we now seek the minimum time control not to the origin but to a
circle about the origin of radius R; that is, we impose the final condition
9[z(8)I = x12(8) + x22(0) - R2 = 0 t=9
Using the equivalent formulation of Sec. 6.6, we have 5 = 1 for
110 OPTIMIZATION BY VARIATIONAL METHODS
yI =
aH0 (3a)
ax,
aH
12 = -axs= -7i (3b)
Thus, as previously,
y, = c, = const (4a)
y2 = -c,t - C2 (4b)
a$
11(0) = + `' ax, = 2yx,(e) (7a)
0
10
When x2(8) and x1(8) have the same algebraic sign (the first and
-third quadrants), the algebraic sign of the argument in Eq. (11) never
changes (t < 8 and therefore t/[8 + x2(8)/x1(8)] < 1). Thus, all tra-
jectories ending on the circle of radius R in the first and third quadrant
must do so without switching between control extremes and must con-
sist of one of the parabolas defined by Eq. (6). Inspection of Fig. 5.4
clearly indicates that for approaches from outside the circle the optimum
is
u= -I x1(8)>0
x2(8) > 0
(12a)
x1(8) < 0
U = +1 (12b)
X2(0) < 0
prior to which the control must have been u = +1. The point on the
parabolic trajectory ending at x1(8), x2(8) corresponding to a time
0 - t. units earlier is
xl(t.) =
R2
x1(0)
- 1x2 2(8)
2 x12(8)
which then defines the switching curve as x1(8) runs from zero to -R,
x2(0) from R to zero. The switching curve in the second .
quadrant is
similarly constructed, giving the entire feedback policy.
it is possible to prove that the optimal function u(t) not only makes the
hamiltonian stationary but that it makes the hamiltonian an absolute
minimum. This latter result is of importance in some applications.
The essential part of the derivation of the weak minimum principle
was in obtaining an infinitesimal change in x(®) and relating this change
to the hamiltonian and to the infinitesimal change in u for all t. It is
possible to obtain an infinitesimal change in x(O) (and thus S) by making
a change in u of finite magnitude if that change is made over a suffi-
ciently small time interval, and the hamiltonian enters in a different way.
Let us suppose that the optimal decision function u(t) and optimal
state vector !(t) are available and that at time t we have effected a
change in the state such that
x(tl) = !(t1) + &x(tl) 1sx(tl)j < E (1)
If in the interval t1 < t < 8 we employ the optimal decision 6(t), we
may write
or
il 'y
(4)
ax,
-1
Next we presume that for all time earlier than t, - A we make
no changes at all.Thus, x(ti 1(tl 1A), or
5x(t, - A) = 0 (5)
During the interval ti - A < t < t1 we set
u(t) = u(t) + su t1 - A < t < t1 (6)
of
MX, u + &u) = M2, u + &u) + i Sx; (11)
ill 8x;
where the partial derivatives are evaluated somewhere between x and
x. Because Eq. (10) establishes Sx as being of the same order as °,
any integral of fix over an interval ° must be of order °2. Thus
n
((t,
Jt,-o I' ax; bx; dt = o(°)
1
Cifi
(12)
where tj is any point of continuity of ft. The sum in Eq. (18) is the
hamiltonian
n
I tirf;
;-i
(19)
+
1
2
/=
o I
j,k1
r;j(t's) a2f
azk azl
bxk dxt ds
rt a2fj
+ 0
n
j,k--I
R
r;j(t,s)
OXk aul
axk aul ds + o(e2) 0<t<9
1
which is an integral equation for &x. We next substitute 6x(t) from Eq.
(3) into the last two terms of Eq. (3). For simplicity we set &x(0) to
zero, either as a special variation or because of fixed initial values, and
we obtain the explicit representation at t = 0,
n R af
6X, (0) =
Jo
0
11 r;j(o,8) aL. auk ds
i-1 k-I
n R 2j
c
+ 2 Io j-1
11 kJ-1
r`j(B's).au Jau, auk 6u, ds
()R
n a2j'.
fo rl,,,(s,v) au, do I ds
+ jo 1 1 r;j(e,8)
1 k,r-1
auk ax, auk our
n R
+2 Jo I I
j,k,l,m,r-11 r,w-1
r,,(o,s) azk2alxl
where all products of variations higher than the second have been
included in o(e2).
We now make a special choice of the variation Bu(t). As we have
already established that the hamiltonian is a minimum for noninterior
values of the optimal decision, we set au; to zero whenever the corre-
sponding decision u; does not correspond to a stationary value of the
1% OPTIMIZATION IJY VARIATIONAL METHODS
+ 2 Jt
azf'au, ykyl ds + o(E2) + o(0)
rij(B,s) auk (6)
Je,! o j-2 k,l-1
If we multiply Eqs. (2) and (6) by a constant, y;(8), and sum over i = 1
to n, then, as in Sec. 6.2, we define a vector ?(t) satisfying
n
ax;yj 0<t<e
i-I
and
so that
as
bS = bxi (9) + O (A) + o (e2) (10)
ax:
i-1
THE MINIMUM PRINCIPLE -197
and introduce the hamiltonian, the combination of Eqs. (10), (11), and
(7) becomes
n
aH 1
a2H
as = f 1 y' ds + 21=,-o G au; au; ysY; ds + o(A)
:-1
(12)
aaH
'Y = y2k2G' 72(e) _ - 1 (4b)
It = P72 - 72 k 2
k'G
jp (8)
k' = k:oE;
u2
Thattis, the solution derived in Sec. 4.12 is optimal only when E1 < .E2.
When El > E2, the optimum is the highest possible temperature.
It has been found recently that certain reaction and separation processes
which are normally operated in the steady state give improved perfor-
mance when deliberately forced to operate in a time-varying manner, where
the measure of performance is taken to be a time-averaged quantity such
as conversion. Horn has shown a straightforward procedure for deter-
mining under i erhh.in conditions when improved performance mA.y he
expected in the unsteady state by the obvious but profound observation
that if the problem is posed as one of finding the time-varying operating
conditions to optimize a time-averaged quantity, the best steady state
can be optimal only if it satisfies the necessary conditions for the time-
varying problem. As we shall see, the application of this principle
requires the strong minimum principle.
We shall consider an example of Horn and Lin.of parallel chemical
reactions carried out in a continuous-flow stirred-tank reactor. The
reactions are
X2
X1
X,
where the reaction X1 -+ X2 is of order n and the reaction X1--' X,
is of first order. X2 is the desired product, and the goal is to choose
the operating temperature which will maximize the amount of X2. The
200 OPTIMIZATION BY VARIATIONAL METHODS
To establish that Eq. (4) does indeed lead to a maximum we take the
second derivative of Eq. (2) with respect to u to obtain, after using Eq.
(4) and the vanishing of ax2/au,
0
z /
dug t l + x21/n-lu l/n nr r 1) - x21)n u-(I+2n)/n(6)
Since u and x2 are both positive, it follows at once that (92x2/au2 is nega-
tive and that a maximum is obtained.
If we formulate the problem as a dynamical one, our goal is to
choose u(t) in the interval 0 < t < 0 in order to maximize the time-
average value of x2(t) or, equivalently, minimize the negative of the
time-average value. Thus, i
aJ = - e f x2(t) dt (7)
THE MINIMUM PRINCIPLE 201
(8)
8
aH 1
12
(IX 2
= B+ 72 (10b)
Since we are testing the steady state for optimality, it follows that
all time derivatives must vanish and the Green's functions y, and y2 must
also be constant. Thus, from Eqs. (10),
nuxln-'
y (13a)
y2=-e1 (13b)
(Clearly we are excluding small initial and final transients here.) Setting
aH/au to zero in Eq. (11) and using Eq. (13) leads immediately to the
solution
[anr'_ 1) (14)
the optimal steady state, so that the optimal steady state does satisfy the
first necessary condition for dynamic optimality. However, it follows
from Eq. (13a) that y, is always negative, so that a2H/au2 in Eq. (12)
has the algebraic sign of r - 1. Thus, when r > 1, the second deriva-
tive is positive and the hamiltonian is a minimum, as required. When
r < 1, however, the second derivative is negative and the hamiltonian is
a local maximum. Thus, for r < 1 the best steady-state operation can
always be improved upon by dynamic operation.
We shall return to this problem in a later chapter with regard to the
202 OPTIMIZATION BY VARIATIONAL METHODS
Hydrogen
product
72 - ax2
aH _ 0 as
y2(0) =axe
1
xa(e) + r (6b)
aH as _ Q - Ar - x2(9)
=0
ye
axe
73(9) = axe - Ix:(B) rJ 2
(6c)
It is convenient to define
Z = yi(O + r) (7)
R°I 8
30
40
50
90
5 10 15 20 25 30 35 40 45
t
Fig. 6.2 Optimum target octane schedule for 15 - 1.
(Courtesy of A. W. Pollock.)
x1 _
Nb2_-1 Z + B - bol(B + 2)u2 + 2Bu] ( 13)
b1[(B + 2)u 2 + 2Bu] b1[(B + 2)u 2 + 2Bu]
8
2
0 100 30
U
40
98
E 96
E 60
94
70
92 0
O 90
90
0 5 10 15 20'. '25 30 35 40 45
f
Fig. 6.4 Optimum target octane schedule for B a 4. (Cour-
tesy of A. W. Pollock.)
u B($-2)
B+2-$ (15)
(4)
or
which is known as the Legendre condition. Both these results are gener-
alized to situations with differential-equation side conditions.
Here we have assumed a single decision variable and made use of the
fact that all/au is zero for an optimum and a2H/au2 vanishes for singular
solutions. fix(O) was taken to be zero, and we have deleted the term
a2S
axj(9) axk(e)
axj axk
leaving to the reader the task of demonstrating that for the special vari-
ation which we shall choose this term is of negligible magnitude compared
to the terms retained. We now assume the symmetric special variation
+ au.- au t - to)+
(CA afjaf'eafk! (3a)
aH
axk au
axkau fe [axkaH. +d: aH (t-to)+
au dt axle au ] [ Eoafk
au
+1 eG1 axj au auk) (t - to) + d ED j i
clu
ft ) (t 21to) + (3b)
a
Here all derivatives are evaluated at to, and ax has been calculated from
the equation
dik =
41I
akkax;+au
f
6U (4)
which is, in fact, a special case of the more general relation which we
shall not derive
(d2k OH)
(-1) a VU d12k au ?0 k = 0, 1, 2, . . . (7)
But X2 = 0, a2r/aT2 < 0, and since the singular are has been shown. to
be a final arc with \,(O) = 0, it easily follows from Eq. (6a) of Sec. 5.11
that a, > 0. Thus Eq. (10), is satisfied, and the singular are does
satisfy the further necessary condition.
If the state is specified, Eq. (2) determines the decision. Any change
in x or u must satisfy, to first order,
bQ = IG ax:ay ax` + au au = 0
:-1
a6l
(3)
f 1 aQ
ax;, bx;
lax; - au (au) (5)
Thus, using the results of Sec. 6.2, along this section of the optimal
path the Green's vector must satisfy the equation
af, af a- Q 1 aQ
Although the canonical equations are not valid over the constrained
part of the trajectory we define the hamiltoniai as before, and it is
easily shown that H is constant over the constrained section. If con-
tinuity of H is required over the entrance to the constrained section,
the multipliers are continuous and H retains the same constant value
throughout and, in particular, H = 0 when 0 is unspecified.
In the general case of K1 independent constraints at equality
involving R1 >_ K1 components of the decision vector we solve the K1
THE MINIMUM PRINCIPLE 211
equations
aQkk aui
sui + I aQk axi = 0 k = 1, 2, . . . , Kl (7)
for the first K1 components Sui. (It is assumed that aQk/aui is of rank
K1.) The equation for the Green's vector is then
Ki
- axi -
af; 8J aQk
is = Spk - 1'i (8)
P.I au, Ox,
t. -
I yi ax: (5)
i-1 i-1
with constraints
and if neither constraints (2b) nor (2c) are in effect, the multiplier equa-
tions are
_ aH Or
7r '(1+Yr-J72) (5a)
as aa
_ aH Or
tie
aT aT
(l + yt - Jy2) (5b)
aH
Ya = - = y2 (5c)
aq
We assume that xl must he within the region bounded by the straight lines
Q1=xi-(ai--$it)>0 (2a)
We then have
g = x2(0) (6)
Q1 = x1 + $1x3 - al 1 0 (7a)
Q2 = -x1 - $2x3 + a2 0 (7b)
The hamiltonian is
H = Y1u + i2Y2(x12 + c2u2) + 73 (g)
tit = - Y2x1
y2 = 0 Y2(8)
'Y3=0
Furthermore, along unconstrained sections the optimal path is defined by
c2x1-xl=0 (12)
From the time t1, when xl intersects the constraint Q1 = 0, until leav-
ing at t2, we have
1=0=u+$1 t1 <t<t2 (13)
and together with the continuity of the hamiltonian it follows that the
control is continuous
u(tl) = -$1 = u(t1+) (15)
216 OPTIMIZATION BY VARIATIONAL METHODS
In the interval 0 < t < t, Eq. (12) must be satisfied, and the solution with
x,(t,) = al - Iglti (16)
is
The contribution to the objective from this final section is made as small
as possible by choosing 0 as large as allowable, 0 = a2/02 Evaluating
Eq. (19b) at t2, the continuity of u(t2) requires that t2 be the solution of
at $2t2
(a, - Nlt2) coth
- - CO, = 0 (20)
Yl = -C2u (21)
with
U1iu2>0 (2)
Q1 = x2 - ul - U2 >0 (3a)
Q2 =xl-u2>0 (3b)
We shall carry out our analysis for x2(9) > x1(9), in which case
u2(9) = x1(9) (8)
Then Q2 = 0 is in force, and just prior to t = 0 the state satisfies the
equations
x1 = 0 (9a)
z2 = a2x1 (9b)
or
x1(t) = x1(8) (103)
x2(t) = a2x1(8)(t - 0) + x2(9) (104)
The Green's functions satisfy Eq. (5) of Sec. 6.15 when Q2 = 0 is in effect,
which leads to
or
72 = 1
(12a)
71 = a2(8 - t) (12b)
The former condition ends at time t1, when 72 equals a171, which is, from
Eq. (12),
tl = 9 _ 1
alas
(14a)
THE MINIMUM PRINCIPLE
it
__
0
_ I' - 11
x,(8) at
1
(14b)
The simpler case encompasses final states for which it > t,. I or times
just preceding t2i then, the optimal policy is
u1 = 0 (15a)
X1(t)
U2 =min x2 (15b)
1X2(t)) =
The state equations are
z,=0 (16a)
x2 = a2x2 (16b)
in which case
x1(t < t2) = x,(t2) (17a)
x2(t < t2) < x2(t2) (17b)
so that x2 < x, for all t < it during which this policy is in effect. But
now Q, = 0 is in effect, leading to multiplier equations
7', = 0 71 = 71(12) = a2(0 - t2) (18a)
72(t2)e-°,(e-y) = e-%(-y)
7'2 = -a272 72 = (18b)
and
6.20 SUFFICIENCY
In Sec. 1.3 we found that only .a slight strengthening of the necessary
conditions for optimality was required in order to obtain sufficient con-
ditions for a policy that was at least locally optimal. Sufficient con-
ditions in the calculus of variations are far more difficult to establish,
and, in general, only limited results can be obtained.
Our starting- point is Eq. (4) of Sec. 6.9, which, after multiplication
by -yi(6) and summation from i = 1 to n yields
n R
e aH 82H
/i
1
yi(9) 8xi(B) = Ia Si6k + R` Su; but
auk .41 49% auk
i-1 k 1
IOa
y7(3) = auk(t) dt (2)
i-1 k-1 r"(s,t)
L.l L.l auk
of minimizing
e
fo ff(x,u) dt (3)
xi Aijxj + I "ikuk DD
(4)
+ + Y L riHikuk (5)
=1 k=1
and for a function satisfyh: the minimum principle Eq. (1) becomes
R 25
R
SS-1J\1
r //
0 au auk 444
a2
axj auk 6%yi
If we integrate Eq. (1) for the two choices of xo and u and subtract, we
obtain
fo'
. fix (t) = 5xo + [f(2 + &x, u + &u) - f(2,u)I ds 0 < t < B (4)
Making use of Eqs. (2) and (3), this leads to
Isx(t)I < e + Lp fog (ISxI + 16u() ds 0<t<e (5)
If ISzI. is equal to the least upper bound of ISxI in 0 < t < 0, then
[fix (t)( <- e(1 + Lpt) + (6)
or
Isz(e)I < Ke (9)
where
K = 2eLD° - 1
BIBLIOGRAPHICAL NOTES
Section 6.2: Solution proxdures for linear ordinary differential equations are discussed
in any good text, such as
E. A. Coddington and N. Levinson: "Theory of Ordinary Differential Equations,"
McGraw-Hill Book Company, New York, 1955
THE MINIMUM PRINCIPLE
Section 8.5: The properties of the adjoint system of the variational equations were exploited
in pioneering studies of Bliss in 1919 on problems of exterior ballistics and subse-
quently in control studies of Laning and Battin; see
G. A. Bliss: "Mathematics for Exterior Ballistics," John Wiley & Saps, Inc., New
York, 1944
J. H. Laning, Jr., and R. H. Battin: "Random Processes in Automatic Control,"
McGraw-Hill Book Company, New York, 1956
Section 6.8: The earliest derivation of a result equivalent to the strong minimum principle
was by Valentine, using the Weierstrass condition of the classical calculus of varia-
tions and slack variables to account for inequality constraints; see
F. A. Valentine: in "Contributions to the Theory of Calculus of Variation, 1933-37,"
The University of Chicago Press, Chicago, 1937
The result was later obtained independently by Pontryagrin and coworkers under some-
what weaker assumptions and is generally known as the Pontryagin maximum (or
minimum) principle; see
V. G. Boltyanskii, R. V. Gamkrelidze, and L. S. Pontryagin: Rept. Acad. Sci. USSR,
110:7 (1956); reprinted in translation in R. Bellman and It Kalaba (eds.),
"Mathematical Trends in Control Theory," Dover Publications, Inc., New York,
1964
L. S. Pontryagin, V. A. Boltyanskii, R. V. Gamkrelidze, and E. F. Mishchenko:
"The Mathematical Theory of Optimal Processes," John Wiley & Sons, Inc.,
New York, 1962
A number of differ.mt approaches can be taken to the derivation of the minimum principle.
These are typified in the following references, some of which duplicate each other in
approach:
M. Athans and P. L. Falb : "Optimal Control," McGraw-Hill Book Company, New
York, 1966
L. D. Berkovitz: J. Math. Anal. Appl., 3:145 (1961)
A. Blaquiere and G. Leitmann: in G. Leitmann (ed.), "Topics in Optimization,"
Academic Press, Inc., New York, 1967
224 OPTIMIZATION BY VARIATIONAL METHODS
F. Horn and R. C. Lin: Ind. Eng. Chem. Process Design Develop., 6:21 (1967)
A good introduction to the notion of process improvement by unsteady operation is
Applications to other problems with decaying catalysts have been carried out by
A. Chou, W. H. Ray, and R. Aris: Trans. Inst. ('hem. Engre. (London), 45:T153 (1967)
S. Szepe and O. Levenspiel: unpublished research, Illinois Institute of Technology,
Chicago
Section 6.13: The classical Weierstrass and Legendre conditions are treated in any of the
texts on calculus of variations cited for Sec. 3.2.
THE MINIMUM PRINCIPLE 225
Section 8.20: Sufficiency for more general situations is considered in several of the refer-
ences cited for Sec. 6.8.
PROBLEMS
6.1. Show that the equation
Here h is the altitude, v the velocity, m the mass, g the acceleration due to gravity,
and c the exhaust velocity. D is the drag, a f unction of v and h. The control variable
is the thrust T, to be chosen subject to
B<T <T'
to maximize the terminal value of h for fixed m. Show that intermediate thrust is
possible only for drag laws satisfying
aD 2a
c= 0
av! c 0,V
6.6. Given
=u
X(0) - z(1) 0
x(0) - vo > 0 z(l) + -vl < 0
Jx(t) I < L
THE MINIMUM PRINCIPLE 227
u!(t) dt
2 Jo
(The problem is due to Bryson, Denham, and Dreyfus.)
6.7. Consider the problem of determining the curve of minimum length between point
(x,o,x20) and the origin which cannot pass through a closed circular region. The
equations are
i, = sin u
it = Cog U
(xi-a):+xs:>_R'
B
min 6 = (o dt
7.1 INTRODUCTION
In Chap. 1 we briefly examined some properties of optimal systems in
which the state is described by finite difference equations, while the pre-
ceding four chapters have involved the detailed study of continuous sys-
tems. We shall now return to the consideration of staged systems in
order to generalize some of the results of the first chapter and to place
the optimization of discrete systems within the mathematical framework
developed in Chap. 6 for continuous systems. We shall find that the
conditions we develop are conveniently represented in a hamiltonian for-
mulation and that many analogies to continuous variational problems
exist, as well as significant differences.
In the study of continuous systems it has often been possible to
suppress the explicit dependence of variables on the independent varia-
ble t so that inconvenient notational problems have rarely arisen. This
is not the case with staged processes, where it is essential to state the'
location in space or discretized time precisely. For typographical rea-
sons we shall denote the location at which a particular variable is con-
sidered by a superscript. Thus, if x is the vector representing the state
xn
STAGED SYSTEMS 229
of the system, the state at position n is x", and the ith component of
The systems under consideration are represented by the block
x" is xi".
diagram in Fig. 7.1, and the relation between the input and output of
any stage is
x" = f"(x"-1,u") n = 1, 2, . . . , N (la)
or
1 2
xi" = fi"(x"-1,un)
(lb)
n= 1, 2,, N
S is the number of variables needed to represent the state. The func-
tions f" need not be the same for each stage in the process.
denoting both the stage number n and the final stage N, with which we
multiply Eq. (1) and sum over n from 1 to N and over i from 1 to S.
Thus,
N S N S N S
I I rki Nnxin= 4 r ) kiNnAijxjn-1 + I 4 rki Nnbi n
n-1 i-1 n-1 i.j-1 n-1 i-1
k= 1, 2, . . . S (2)
I I rkiN"x," = InI-IrkjN."-'xjn-1
n-lei-1
j-1 + I rkiNNxiN
ILLL
i-I
s
- I rkiNOXie (3)
{-1
X'=f1(XO, u1)
=f0(X"-1
U") + XN=fN(IN-1, UN)
-
X11
Xa X"
XN-1 IN
Sta g e n .. . j. St age N
Decision u'
i
Decisi on u" Decision uN
1,kiNNxiN - I rkiNOxi0
i-1 i-1
N S S N S
(`j ``J
1N
n-1 j-1
I \I t/
i-1
rkiNnAijn '- 1'A.ji"n-1
/ xjn-1 + n-1i-1
rkiN nb,n
k = 1, 2, . . , S (4)
Since the goal of summing the equations is to express the output x"'
in terms of the input x° and the forcing functions bn, we define the Green's
functions by the adjoint difference equations
rkjN.n-1 =
(S
\ rkiNnAijn k,j=1,2,...,5
i-L I n=1,2,...,N (5)
rkjNN
= akj =
1 k - `6)
o k
xin = fin(xn-I,un)
i=1,2,...,5
n= 1,2, . . . N (1)
with the objective the minimization of a function of the final state S(xN).
The decision functions u" may be restricted by inequality constraints
Upn(un) > 0 p = 1, 21 P
n=1,2,...,N (2)
where the partial derivatives are evaluated at xn, nn. Equation (6) is
linear and has Green's functions defined by Eq. (10) of the preceding
section as
2=C($N)+ II
n-1 i-1
Xi"[-xi" + fn(t"-1,u;./J
L K
+ I v,gi(x'') - I 1kgk(x°) (4)
1-1 k-1
Here we have introduced a multiplier Xe" for each of the constraint Eqs.
(1), a multiplier vi for each Eq. (2), and 'hk for each Eq. (3).
Setting partial derivatives with respect to u;" to zero, we obtain
s of, n
Yin au n= 0 j = 1, 2, , R (5)
I3
j-l
J1 ou'
ax;
- K
k-1
a
u ax;° = (8a)
By defining X,° through Eq. (6) this last relation may be written
K
Xc°- 1 7,, C9
a=0 (8b)
k-1
Equations (5) to (8) are identical to the equations for the weak
minimum principle when the decisions are unconstrained if the Lagrange
multipliers Xi" are identified with the Green's functions y;". Surprisingly
some authors have failed to recognize this simple relation and have
devoted considerable space in the literature to the complicated deri-
vation (or rederivation) through a "minimum principle" of results in
unconstrained systems more easily (and frequently, previously) obtained
through application of lagrangian methods. Indeed, the interpretation
of the multipliers in Sec. 1.15 in terms of partial derivatives of the objec-
tive leads directly to the weak minimum principle with constraints, and
it is only to simplify and unify later considerations of computation that
we have adopted the form of presentation in this chapter.
Equations (1) are not of the form for which we have developed the
theory, which would be
Since we need only partial derivatives of f" and g" in order to apply the
theory, however, this causes no difficulty. For example, partially differ-
entiating Eq. (la) with respect to x"-' yields
a f"
1 - ax"''
af" - O"k1(u")F'(x")
ax"-3
=0 (5a)
or
af" __ 1
(5b)
ax"'' 1 + 8"kl(un)F'(xn)
The other partial derivatives are similarly obtained, and we can write
the multiplier equations
af" ag" yl"
yl"-i = 'Y1' ax"-1 + ax"-' 1 + 8"kl(u")F'(x")
'Y 2 "pO"k l (u")F'(x")
+ [1 + O"kl(u")F'(x"))[1 + 8"k2(u")G'(y")]
afn ay "
72"-' - 71" ayn_1 + 72" 49yn_1
y2 "
1 + 9"k2(u*)G'(y")
(6b)
y2"'=ayN= -1 (7b)
Z36 OPTIMIZATION BY VARIATIONAL METHODS
Also,
all- _ af" yl"9"k1(un)F(x")
T--
` 7i" au- + 72 nag-
au" 1 -}- 9"kl(un)F'(x")
72np8"kl(u")F(x")
+ [1 + 9"k2(u")G'(y"))[1 + 9"ki(u")F'(x°)]
72"9"k2(u")G(y")
1 + 9"k2(u")G'(y") (8)
we obtain
1 + 9"kl(u")F'(x") p9"kl(u")F'(x")
(10)
1+ 9"k2(u")G'(y") 1 + 9"k2(u")G'(y")
N=P (11)
with solutions of aHn/au" = 0 at solutions of
9"k1(u")F(x") p9"k; (u")F(x")
1 + 9"ki(u")F'(x") + [1 + 9"k2(u")G'(y")][1 + 9"kl(u")F'(x"))
9"k2(u")G(y") =0 (12)
1 + 9"k2(u")G'(y")
The required computational procedure is then as follows:
1. Assume r'.
2. Solve Eqs. (1) and (12) simultaneously for x', y', u'.
3. If u' exceeds a bound, set u' to the nearest bound and recompute
x', y' from Eqs. (1).
4. Compute x2, y:, us, r simultaneously from Eqs. (1), (10), and (12).
5. If u2 exceeds a bound, set u2 to the nearest bound and recompute
x2, y2, ?.2 from Eqs. (1) and (10).
6. Repeat steps 4 and 5 for n = 3, 4, . . . , N.
in order to minimize
S = _x,2 (3)
By direct substitution
S= + (u1)21(2 - u') + 12(U2)2 (4)
But
s
'rll = axil = -(I + x21) (-2 + 'u') (8a)
72'
axaH2
=1=l
= -xl1 7'l 11 + (499
9(499 (8b)
Thus, all'/au' does indeed vanish at u' = 1. But from Eqs. (7) and
(8a),
a'H' -1
2 = -1 < 0 (9)
a(u1)2
S 2f n if
2n S R
Sukn axjn-1
+21 axjna' axkn-1 axn-1 axkn-1 +
j-1 k=1
axja
I c`) 3If.n
Sujn bUk n +UO(J)
O/( (1)
+ . L1
J,k - 1
all n
./ j,k=1 p-1
+ m=1 I rijnm 32",m axk,
au,- Sxkm-1 + O(E2) (3)
Evaluating Eq. (3) for n = m - 1 and substituting back into the right-
hand side, we obtain, finally, an explicit representation for Sxv
N S R R
2 Ln
SxiN =
rJ rijNn
(Li auk~ + `I S2lkn aupn
pal aua . P
LI n
n-1j+1k-1
N S n -1 S R
+ n-I j.k,pI
n-i S R
Nn _ J7
axkn al dxpn-1
2f.. n
111
q1 v1
rk n-l,m
Q
{{
a2lvm
a2cvm
{r \ N S R
2 fn
X rp r n
n
axk
n-1 j,kml allp
n-1 S R a{19m
X rjqn-l.m
auvm) + O(E2) (4)
allvm
,n-1 q =1 v-1
SS = bx,N N ax;N
a.Z,N (5)
and multiplying Eqs. (2) and (3) by y,,' and summing we obtain, with
the usual definition of the stage hamiltonian,
I 2 n
SF = I an aukn +
2 LI 3unfaukn bu7n aukn
n=1 k=1 j,k=1 7
N S n n-1 S R
fi{m
II
2
rkj n-l,m
v
Suym/J1
+ n=1 k,p=1
8X"al axPn-1 (m=1 7-1 v=1 U
n-1 S
/`'1
/C T
R
, t
LL
N S' CR auPn 2 n
X (I I V 1
r=l q -1 s=1
P4n-l r
euzr} + n =1 k-l p= I
axka Suyn
n-1 S P
m
where the A,," are constants. In that case both the second partial deriv-
atives of Hn with respect to components only of xn-1 and xn-1 and un
vanish identically, and Eq. (6) reduces to
N` R
II
n 2 n
SS
= n-1 k=1 aukn
sukn +
2 n-1 I
L j,k-1 8u n aukn Sujn 7
Sukn + 0(E2) > o
(8)
The first term is zero by virtue of the weak minimum principle and the
assumption concerning admissable bun. It follows, then, that the hessian
matrix of second derivatives of Hn must he positive definite, correspond-
ing to a minimum. Furthermore, since the Sun are arbitrary, Eq. (8)
STAGED SYSTEMS 241
establishes the sufficiency of the strong minimum principle for the linear
separable case as well.
Linear nonseparable equations are of the form
s
x," = I A;;" (u") x"-' + b;" (u") (9)
j-1
Here the mixed second partial derivatives do not vanish. By choosing
the special variation, however,
bu* m = n*
bum _ (10)
0 m n*
for some specified n* the last term in Eq. (6) also vanishes, and we obtain,
using the weak minimum principle,
SE = 2
I
j R
I
asg"*
au "* auk"*
Su* Su,* + o(EZ) > 0
which establishes the necessity of the strong minimum principle for this
case also. That it is not sufficient is apparent from Eq. (6).
If there is a single state variable, with the scalar equation
then, since y" will be nonzero, the vanishing of all"/auk" implies the van-
ishing of 49f"/auk". It follows from Eq. (6), then, that the strong mini-
mum principle is both necessary and sufficient. Furthermore, f" always
takes on an extreme value, and if of"/ax"-' is positive for all n, then y". is
always of one sign. If, in addition, x" must always be positive for physi-
cal reasons, the policy is disjoint, which means that minimizing H" in
order to minimize or maximize x" is equivalent to minimizing or maxi-
mizing x" at each stage by choice of u". This should recall the discussion
of optimal temperature profiles for single reactions in Sec. 3.5, and, indeed,
an identical result holds for the choice of temperatures for a single reaction
occurring in a sequence of staged reactors.
We emphasize that the results obtained here are applicable only for
a linear objective.t If S is nonlinear, an additional term is required in
Eq. (6). In particular, a linear system with a nonlinear objective is
formally equivalent to a nonlinear system with a linear objective, for
which the strong minimum principle does not apply. A counterexample
t With the exception of those for a single state variable.
242 OPTIMIZATION BY VARIATIONAL METHODS
1
The decision un is determined by Eq. (4), and the Green's function corre-
STAGED SYSTEMS 243
I Sp/c
au; = atp i,p= 1, 2, . . . ,K1 (10)
k-1
BIBLIOGRAPHICAL NOTES
Section 7.1: Finite difference representations might arise either because of time or space
discretization of continuous processes or because of a natural staging. For the for-
mer see, for example,
Section 7.2: The linear analysis for difference equations used here is discussed in
M. M. Denn and R. Aria: Ind. Eng. Chem. Fundamentals, 4:7 (1965)
T. Fort: "Finite Differences and Difference Equations in the Real Domain," Oxford
University Press, Fair Lawn, N.J., 1948
L. A. Zadeh and C. A. Desoer: "Linear System Theory," McGraw-Hill Book Com-
pany, New York, 1963
244 OPTIMIZATION BY VARIATIONAL METHODS
Sections 7.3 and 7.4: The analysis follows the paper by Denn and Aria cited above.
Similar variational developments may be found in
Section 7.5: The lagrangian development was used by Horn prior to the work cited above:
F. Horn: Chem. Eng. Sci., 15:176 (1961)
Section 7.6: This reactor problem has been studied in the papers cited above by Horn and
Denn and Aris and in the book ,
R. Aria: "The Optimal Design of Chemical Reactors," Academic Press, Inc., New
York, 1961
A number of other applications are collected in these references and in the book by Fan
and'Wang.
Section 7.7: The observation that a strong minimum principle is not available in general
for discrete systems is due to Rozenoer:
Several workers, notably Holtzman and Halkin, have examined the set-theoretic foun-
dations of the necessary conditions for optimal discrete systems with care. Some of
their work and further references are contained in
Section 7.9: These results are obtained in the paper by Denn and Aria cited for Sec. 7.8.
STAGED SYSTEMS 245
PROBLEMS
7.1. Consider the necessity and sufficiency of a strong minimum principle for
S
xi" - j-1I aijxjw-1 + biu"
N jS
min
I[
[ La xi"Qijxj" + P(u")']
2 n11 i,j-1
7.2. Denbigh has introduced the pseudo-first-order reaction sequence
ki k,
X, X, -, X,
Lk, J,k,
Yl Y2
Here, ul is the product of reactor residence time and k1, and u2 is the ratio k1/k1.
Taken together, they uniquely define temperature and residence time. The following
values of Denbigh's have been used:
3, T 0
xw+: = F(xw+l,xw,uw)
7.4. Repeat Prob. 1.13 using the formalism of the discrete minimum principle. Extend
to the case in which T. is bounded from above and below.
7.5. Consider the reaction X Y -, Z in a sequence of stirred tanks. This is defined
by Eqs. (1) of Sec. 7.6 with the added equation -
X0 + y* + z* - coast
Derive the procedure for maximizing z"'. Specialize to the case
F(x*) - x* G(y*) - y*
and compare your procedure for computational complexity with the one used in the
book by Fan and Wang for studying an enzymatic reaction.
8
Optimal and Feedback Control
8.1 INTRODUCTION
Many of the examples considered in the preceding chapters have been
problems in process control, where we have attempted to use the opti-
mization theory in order to derive a feedback control system. Even in
some of the simple examples which we have studied the derivation of a
feedback law has been extremely difficult or possible only as an approxi-
mation; in more complicated systems it is generally impossible. Further-
more, we have seen that the optimal feedback control for the same sys-
tem under different objectives will have a significantly different form,
although the objectives might appear physically equivalent.
In this chapter we shall briefly touch upon some practical attacks
on these difficulties. We first consider the linear servomechanism prob-
lem and show how, for a linear system, the optimization theory com-
pletely determines the feedback and feedforward gains in a linear control
system and how classical three-mode feedback control may be viewed as
a natural consequence of optimal control. The problem of ambiguity of
247'
248 OPTIMIZATION BY VARIATIONAL METHODS
a2H=1>0 (5b)
au2
Equation (5a) defines the optimum when set to zero
OPTIMAL AND FEEDBACK CONTROL
(6)
M;jxj + MijAjkzk
I Mijbj Q Muxkb,
j k.!
+ kjI Dudb,) + IjMijdj + Ej Aidj , (9a)
k
1 ((
i
(11)
the objective
or, integrating,
where
K = 12Ma3(Maa + 2a) (14a)
M33(M33 + 2a)
TI = (14b)
2 VC1
2
TD = (14c)
M33 + 2a
These equations can be rearranged to yield
K=
2(1 2
- aTD) (15)
TD
aTD<1 (16)
The governing equation for the controlled system subject to step
disturbances is
and substituting Eq. (15) into (17), the governing equation becomes
+
[2
TD rD] x + [b+_(1 - aTD)] x
+L D3(1 amD)] x = 0 (19)
For a stable uncontrolled system (a, b > 0) the necessary and sufficient
condition that Eq. (19) have characteristic roots with negative real parts
and that the controlled system be stable is
(coefficient of t) (coefficient of t) > (coefficient of x) (20a)
or
2-arDrb+ 2a(1-aTD)] (20b)
TD L TD Q7D
-2 rp3+(a2+b)rp2+(a3{arn+i2-aJ=0 (22)
It will often happen in practice that a2 > b. For example, in the reactor-
control problem used in Chaps. 4 and 5 and again in a subsequent section
of this chapter a = 0.295, b = 0.005. In that case Eq. (22) approxi-
mately factors to
\\ // \\
rD2+arp-1arn-2+a)=0 (23)
C
and the unique root satisfying the inequality arD is
arp=2-- (24)
,
The nonnegativity of TD :iliu the upper-
tuoullua vu.cll
.. .V..., n
..u 1,
:.. '111LUtS-
al1Ow-
able values of u/a to
2 > -° > 1 (25)
-a
Equation (15) for the gain can now be rewritten in terms of the single
parameter o/a as
K _ 2a(o - a)
(26)
P (2a - 0')2
Further restrictions on the allowable controller design parameters
are obtained by substituting Eqs. (14), (18), (24), and (26) into the
quartic equation (9) and rearranging to solve for C2, yielding
C2 = 4K 2 [_a2 - c + av + (2a 0)2] (27)
a J
A meaningful optimum requires that the objective be positive definite,
or that C2 be nonnegative. Using the approximation b >> a2 for con-
sistancy, a and c are then further related by the inequality
a2-ao+a, <0 (28a)
dition! It further follows, then, from Eq. (25) that a is bounded from
below
a2i2o>2 (30)
a- j2v<2j1 -
2<<rp<211-+ -(32)
Thus, the ratio rr/rn isstrictly bounded by the one-parameter inequality
(31)
XiQrsx; (1)
E=2
i.J
(Linearity is essential, but the single control is not.) This form is typi-
cal of many processes. The criterion of driving E to zero as rapidly as
OPTIMAL AND FEEDBACK CONTROL 23
(We have made use of the symmetry of Q.) Since Eq. (4) is linear in u,
the minimum will always occur at an extreme when the coefficient of u;
does not vanish identically
u* I x.Qiibi > 0
u= `'' (5)
u* x;Q;ibi < 0
.
As T will always exceed the coolant temperature T,, the coefficient of the
bracketed term in Eq. (9) is always negative and Eq. (5). for the control
law reduces to
U _ 8 (T - T.) + a(A - A,) > 0 (10)
0 (T-T,)+a(A-A,)<0
where a = Q12/Q22 - This, is a linear switching law for the nonlinear proc-
ess. An even simpler result is obtained when cross terms are excluded
from the objective, in which case a = 0. Then the control is
u-_ 8 T > T,
0 T<T,
256 OPTIMIZATION BY VARIA' ONAL METHODS
That is, adiabatic operation when the temperature is below the steady-
state value, full cooling when above, irrespective of the relative weight-
ing placed on concentration and temperature deviations. It is interest-
ing to compare this result with the time-optimal control shown in Fig.
5.9, in which the switching curve does not differ significantly from the
steady-state temperature.
Paradis and Perlmutter have computed the response of this system
under the control equation (11) with an initial offset of T - T. = -20,
A - A. = 2 X 10-4. The phase plane in Fig. 5.9 indicates that away
from equilibrium the temperature should approach the steady-state value
immediately, first slowly and then quite rapidly, while the concentration
deviation should first grow and then approach zero. Figures 8.1 and 8.2
show that this is precisely what happens, where curve a is the controlled
response and curve bthe uncontrolled. The first two switches occur at
23.70 and 25.65 sec, after which the controller switches between extremes
rapidly. Such "chattering" near the steady state is a common charac-
teristic of relay controllers. It should be noted that the system is asymp-
totically stable and returns to steady state eventually even in the absence
of control.
In order to avoid chatteringt some criterion must be introduced for
t Chattering may sometimes be desirable, as -discussed in detail in the book by
Flilgge-Lots.
0
3
Q
1
Desire
operating
level
I I I I I I 1 I I 1
10 20 30 40 50 60 70 80 90 100
Time t
460
455
-450
K
445
440
10 20 30 40 50 60 70 80 90
Time t
bility of the system to complete the control. These results are shown as
curve c in Figs. 8.1 and 8.2 and indicate quite satisfactory performance.
unity
Jul < 1 (2)
If the bounds on u are not symmetric, the subsequent algebra is slightly
more cumbersome iut the essential conclusions are unchanged. Under
these assumptions the feedback control law of Eq. (5) of the preceding
section is
u = - sgn Q b;Q;;x;) (3a)
or
where
5(x) > 9(0) > 0 x 96 0 (5)
The inverse problem which we wish to solve is for a function F(x) satis-
fying Eq. (5) such that a control of the form of Eq. (3) minimizes 6.
The hamiltonian for the linear stationary system described by Eq.
(1) and the objective equation (4) is
H = F(x) + I y;A;;x; + y;b;u (6)
Equations (3) and (7) will define the same control if (but not only if!)
we take
yi = Q;;x; (8)
where we have used Eqs. (1), (7), and (8). The right-hand sides of
Eqs. (10) and (11) must be identical, leading to a family of partial differ-
ential equations for t(x):
ax; -
- (QriAikxk
+ xkQkiAii) + Qiibj sgn (I xkQktbt} (12)
j.k 7 k.t
F(x)
- 11 xj(Q;iAjk + Qk,Ai;)xk + I x,Q;;bj + const
I (13)
c.j.k
(x)
=2 x;C;ixi +
I x;Qi;b; (14)
i., i.i
where
u
- - (A21 Q222) xl - A22x2 (19)
On the other hand, the vanishing of the switching function means that
the integrand of the objective is simply 3' (Cllxl2 + 2C12x1x2 + C22x22y,
and the criterion for singular control was found in Sec. 5.9 for this case
to be
u = - (A ell
\ 21-C22)xl-Az2xz (20)
and
Together with Eq. (15) this yields only discrete values of the ratio
C matrix is at the disposal of the designer, only infinitesimal
changes are needed to avoid the possibility of intermediate control. The
generalization to higher dimensions is straightforward and yields the
same result.
Finally, it remains to be shown that there cannot be another con-
trol policy which satisfies the minimum principle. Here, for the first
time, we make use of our wish to avoid chattering and presume that
within some neighborhood of the origin we want to switch from the relay
controller to some other form of control, perhaps linear. We shall choose
that region to be an ellipsoidal surface such that the control effort is to
terminate upon some manifold
g(x) = I x;(0)Q;;x;(6) - const = 0 (23)
:.,
where the final time 0 is unspecified. The boundary condition for the
Green's functions is then
y;(0) = v ax = 2v I Qi,x, (24)
and
T; = fa
(b x sinh bQx + Isinh bQxl) dt (30)
The coefficients Aij and bi are taken as constants, though the extension
to functions of n is direct. We have not included a disturbance term,
though this, too, causes no difficulties, and we seek only to find the
sequence of controls Jun 1 which regulates the system following an initial
upset or change in desired operating point. The minimum-square-error
criterion is again used
N`
S R(u")2]1
`l l (2)
=2 [I R(u")2]
Setting partial derivatives with respect to xi" and u" to zero, respectively,
we obtain
xinCij - xjn + X;"+'Aii = 0 (4)
Since X '+' is zero, Mk"" must be zero, except for N -- oo, in which case
xkN - 0. ' Substitution of Eq. (8) into Eq. (7) then leads to the required
264 OPTIMIZATION BY VARIATIONAL METHODS
feedback form
u" = I K1"x "-I (9)
where
I CikbkAij + I blAklAlki"A,;
K." i.l.k
(10)
R + I b,Cikbk + b1Ak1:llk,"bi
i,k
The feedback dependence must be on x"-', since the state at the begin-
ning of the control interval is the quantity that can be measured.
We still need a means of calculating 211,," in order to compute the
feedback gains Kj". This is done by first substituting Eq. (9) into Eq. (1)
+ Ai;
[1 .1f, " I (AIk + b1Kk")] xk"-' = 0 (12)
1 k
If Eq. (12) is to hold for all values of x"-1, the coefficient of xk"-1 must
be identically zero, in which case i1ij" must be a solution to the difference
equation
:11jk"-' _ CijAik +N' Aijifil"AIk + biCij + Aijhlil"bzKk"
1l1;kN = 0 (13)
Equation (13), with Kj" defined by Eq. (10), is the generalization of the
Riccati difference equation first encountered in Sec. 1.7. For the impor-
tant case that N ---* co, a constant solution is obtained. Clearly there is
no difficulty here in allowing R to go to zero.
The instantaneously optimal approach of Sec. 8.5 may be applied
to the discrete system to obtain an interesting result. We define a posi-
tive definite quadratic error over the next control interval
E (14)
= l xi"Q11xj" +
2 P(u")2
The control which makes E as small as possible over the following inter-
val is found by setting the derivative of E with respect to u" to zero
O1' _ ax" (15)
fail" - I
x,nQ'j
au."
+ Pu" = 0
OPTIMAL AND FEEDBACK CONTROL 265
The required feedback form for u" is obtained by substituting Eq. (1) for
x;" into Eq. (17) and solving
u" = I kx;"-' (18)
s
I / biQikAki
k
i
k, c/` (19)
/-I I biQikbk+P
k i
It can be shown from Eqs. (10) and (13) that µik is symmetric(µik = µki).
Equation (10) for the regulator feedback gain can then be written
I I bi(Cik + Uik)Aki
k i (21)
K1=-
11 1 bi(Cik + Iik)bJ + R
k i
BIBLIOGRAPHICAL NOTES
Section 8.1: The conventional approach to the design of feedback control systems is treated
extensively in texts such as
P. S. Buckley: "Techniques of Process Control," John Wiley & Sons, inc., New
York, 1964
D. R. Coughanowr and L. B. Koppel: "Process Systems Analysis and Control,"
McGraw-Hill Book Company, New York, 1965
D. D. Perlmutter: "Chemical Process Control," John Wiley & Sons, Inc., New York,
1965
J. Truxal: "Automatic Feedback Control System Synthesis," McGraw-Hill Book
Company, New York, 1957
The design of optimal control systems based on a classical frequency-domain analysis
is treated in, for example,
S. S. L. Chang: "Synthesis of Optimum Control Systems," McGraw-Hill Book
Company, New York, 1961
A modern point of view somewhat different from that adopted here is utilized in
C. W. Merriam: "Optimization Theory and the Design of Feedback Control Systems,"
McGraw-Hill Book Company, New York, 1964
1
The two approaches are reconciled in our Chap. 12; see also
L. B. Koppel: "Introduction to Control Theory with Applications to Process Control,"
Prentice-Hall, Inc., Englewood Cliffs, N.J., 1968
i
OPTIMAL AND FEEDBACK CONTROL 267
Section 8.8: The properties of the linear system with quadratic-error criterion have been
investigated extensively by Kalman, with particular attention to the asymptotic
properties of the Riccati equation. In particular see
It. E. Kalman: Bol. Soc. Mat. Mex., 5:102 (1960)
in It. Bellman (ed.), "Mathematical Optimization Techniques," University
of California Press, Berkeley, 1963
: J. Basic Eng., 86:51 (1964)
A computer code for the solution of the Riccati equation, as well as an excellent and
detailed discussion of much of the basic theory of linear control, is contained in
It. E. Kalman and T. S. Englar: "A User's Manual for the Automatic Synthesis
Program," NASA Contractor Rept. NASA CR-475, June, 1966, available from
Clearinghouse for Federal Scientific and Technical Information, Springfield,
Va. 22151
N. N. Puri and W. A. Gruver: Preprints 1967 Joint Autorn. Contr. Conf., Philadelphia,
p. 335
Though not readily apparent, the procedure used in this paper is equivalent to that dis-
cussed in Sec. 9.6 for the numerical solution of nonlinear differential equations.
Section 8.3: The general relationship between the linear servomechanism problem with
a us cost-of-control term and classical control is part of a research program being
carried out in collaboration with G. E. O'Connor. See
Similar procedures, generally coupled with Liapunov stability theory (Appendix 8.1),
268 OPTIMIZATION BY VARIATIONAL METHODS
Several authors have recently studied the related problem of comparing performance of
simple feedback controllers to the optimal control for specified performance indices.
See
A. T. Fuller: Intern. J. Contr., 5:197 (1967)
M. G. Millman and S. Katz: Ind. Eng. Chem. Proc. Des. Develop., 6477 (1967)
Section 8.6: The Riccati equation for the feedback gains is obtained in a different manner
in the monograph by Kalman and Englar cited for Sec. 8.2, together with a computer
code for solution. See also the books by Koppel and Lapidus and Luus and
W. G. Tuel, Jr.: Preprints 1967 Joint Autom. Contr. Conf., Philadelphia, p. 549
J. Tou: "Optimum Design of Digital Control Systems," Academic Press, Inc., New
York, 1963
"Modern Control Theory," McGraw-Hill Book Company, New York, 1964
The book by Roberts contains further references. Instantaneously optimal methods have
been applied to discrete systems by
R. Koepcke and L. Lapidus: Chem. Eng. Sci., 16:252 (1961)
W. F. Stevens and L. A. Wanniger: Can. J. Chem. Eng., 44:158 (1966)
Appendix 8.1: A good introductory. treatment of Liapunov stability theory can be found
in most of the texts on control noted above and in
J. P. LaSalle and S. Lefschetz: "Stability by Liapunov's Direct Method with Applica-
tions," Academic Press, Inc., New York, 1961
For an alternative approach see
M. M. Denn: "A Macroscopic Condition for Stability," AIChE J, in press
PROBLEMS
8.1. For systems described by the equations
2i - f,(x) + b,(x)u
use the methods of Secs. 8.2 and 4.8 to obtain the linear and quadratic terms in the
nonlinear feedback control which minimizes
s
e (j' xiC,;z; + u' dt
3 2 fo \4
Extend to the feedback-feedforward control for a step disturbance d which enters as
z+at+bx-u+eii.+d
8.3. Extend the control approach of Sec. 8.4 and the optimization analysis to the case
when E(x) is an arbitrary convex positive definite function.
M. The unconstrained control problem
x = f(x,u,P)
x(0) = xo
s
min E a fo 5(x,u,c) di
where p and c are parameters, has been solved for a given set of values of xo, p, and c.
Obtain equations for the change du(t) in the optimal control when xo, p, and c are
changed by small amounts Sxo, bp, and Be, respectively. In particular, show that au
276 OPTIMIZATION BY VARIATIONAL METHODS
may be expressed as
where ax is the change in x and Ki(t) and g ,(t) are solutions of initial-value problems.
(Hint: The method of Sec. 8.2 can be used to solve the coupled equations for changes
in state and Green's functions.) Comment on the application of this result to the
following control problems:
(a) Feedback control when the system state is to be maintained near an optimal
trajectory.
(b) Feedback-feedforward control when small, relatively constant disturbances
can enter the system and the optimal feedback control for the case of no disturbance
can be obtained.
8.6. The system
x(t) + at(t) + bx(t) - u(t)
is to be regulated by piecewise constant controls with changes in u every r time units
to minimize
1
S- Ioe (z' + ci') dt
2
Obtain the equivalent form
xl" - 2="-1
Obtain explicit values for the parameters in the optimal control for N -- co
W. - -K1z1"-1 - Ktx:"-1
9.1 INTRODUCTION
The optimization problems studied in the preceding six chapters are
prototypes which, because they are amenable to analytical solution or
simple computation, help to elucidate the structure to be anticipated in
certain classes of variational problems. As in Chaps. I and 2, however,
where we dealt with optimization problems involving only differential
calculus, we must recognize that the necessary conditions for optimality
will lead to serious computational difficulties if rational procedures for
numerical solution are not developed. In this chapter we.shall consider
several methods of computation which are analogs of the techniques
introduced in Chap. 2. In most cases we shall rely heavily, upon the,
Green's function treatment of linear differential and difference equations
developed in Sees. 6.2 and 7.2.
k-1
where fix = x - 2, &u = u - n, and partial derivatives are evaluated
along the trajectory determined by u. Defining the Green's functions
rij(B,t) by
s
I
k-1
k
ax. (5)
1r"
(12)
L/d
k-1
NN
_
-a" ij0
j (13)
x'N = x'N + r
rijNO(xjo - xio) (17)
j-1
where r > 1 is a parameter controlling the step size. r must be taken
large initially and allowed to approach unity during the final stages of
convergence.
reaction sequence
X, --> X2 - > products
We have examined this system previously in Sees. 4.12 and 6.10 and have
some appreciation of the type of behavior to be anticipated. Taking
v = 1, F(x1) = x12 (second-order reaction), and G(x2) = x2 (first-order
reaction), the state is described by the two equations
z, = -k1oe-E,'""x,2 x1(0) = x10 (1a)
x2 = k10e-E"'l"x12 - k20e-E"/' x2 x2(0) = X20 (lb)
with u(t) bounded from above and below
U. < U < u* (2)
-- -
af,
- r,, axl r,2
af2
ax,
= 2x,k,oe-E,'I"(r - r,2)
r11(9,9) = I
t12 = - r , ,
axt
-
af-t
- r 12
af2
ax2
= k20e-Ei',"r,2 r12(8,0) = 0
(4a)
(4b)
af,
1'21 = -r21ax, - r22 af2
ax,
r,,(9,9) = 0 (4c)
It easily follows that r12(9,t) =_ 0 and will not enter the computation,
although we shall carry it along in the discussion for the sake of generality.
The hamiltonian for the optimization problem is
H= -Y,ktoe-E-'i"xt2
+ Y2(ktoe-E,'1"x12 - k2pe-Ei'i"x2)
(5)
where the Y; are computed from the r;; by
a6 as
71 = W r,, + ax2 r2, _ -crl, - r21 (6a)
as as
ax, r,2 + ax2 r22 = -cr,2 - r22
Y2 = (6b)
276 OPTIMIZATION BY VARIATIONAL METHODS
Minimization of the hamiltonian leads to the equation for the optimal u(t)
u* v(t) > u*
u(t) = v(t) u* < v(t) < u* (7)
U (t) =
E' - E'
2 1
(8)
y2x2k2o
In
('Y2 - yi)xi2kio
The computational procedure is now to choose trial values x1(6),
22(6) and integrate the six equations (1) and (4) numerically from
t = 6 to t = 0, evaluating u at each step of the integration by Eq. (7)
in conjunction with Eqs. (8) and (6). At t = 0 the computed values
21(0), 22(0) are compared with the desired values x10, x20 and a new trial
carried out with values
r (9a)
(9b)
x1(9) = 0.254, 0.647 and x2(0) = 0, 0.746, calculated from the limiting
isothermal policies as defining the extreme values.
The approach to the values of x1(9) and x2(9) which satisfy the
two-point boundary-value problem (x1 = 0.421, x2 = 0.497) are shown
in Fig. 9.1, where the necessity of maintaining r > 1 during the early
stages to prevent serious overshoot or divergence is evident. Successive
values of the optimal temperature profile for the iterations starting from
the point xl = 0.647, x2 = 0 are shown in Fig. 9.2, where the ultimate
profile, shown as a broken line, is approached with some oscillation. In
all four cases convergence to within 0.1 in u(t) at all t was obtained with
between 12 and 20 iterations.
As described in this section and the preceding one, the algorithm
is restricted to problems with unconstrained final values. This restric-
tion can be removed by the use of penalty functions, although it is found
that the sensitivity is too great to obtain convergence for large values
of the penalty constant, so that only approximate solutions ,can be
ests in part
realized. The usefulness of the Newton-Raphson method rests'
upon the ability to obtain an explicit representation of the optimal
decision, such as Eq. (7), for if the hamiltonian had to be minimized by
use of the search methods of Chap. 2 at each integration step of every
iteration to find the proper u(t), the computing time would be excessive.
0.7
0.6
0.5
m 0.4
0.3
aH _ 2k,u CM
yl _ - 5x, A + X2
(y1 - 2yz) 7j(e) = ax, = 0 (4a)
aH 2k,,ux, 8k2uzyzAx2
12 = - ax2 (A +x2)2 (yl
2X:} + (A '+ x2)3
'at = -1
'12(8) _ xz
(4b)
0.8000 0.4500 5.2 X 10' 1.130 0.5000 1.0000 3.9 X 1010 0.759 0.2000 0.9000 6.4 X 1010 0.732
1 0.8000 0.4500 5.2 X 10' 1.130 0.6000 1.0000 3.9 X 1010 0.759 0.8900 0.6317 2.3 X 100 0.955
2 0.8000 0.4500 5.2 X 10' 1.130 0.8879 0.4837 1.4 X 10' 1.123 0.8900 0.8317 2.3 X 10' 0.965
3 0.8000 0.4500 5.2 X 10' 1.130 0.8879 0.4837 1.4 X 10' 1.123 0.8470 0.4778 1.1 X 10' 1.129
4 0.8000 0.4500 5.2 X 10' 1.130 0.8385 0.4685 1.2 X 10' 1.132 0.8470 0.4778 1.1 X 10' 1.129
5 0.8000 0.4500 6.2 X 10' 1.150 0,8385 0.4685 1.2 X 10' 1.132 0.8045 0.4494 6.3 X 10' 1.132
6 0.8330 0.4637 7.6 X 10' 1.132 0.8386 0.4885 1.2 X 10' 1.132 0.8045 0.4494 6.3 X 10' 1.132
7 0.8330 0.4637 7.6 X 102 1.132 0.8082 0.4502 6.0 X 10' 1.132 0.8045 0.4494 6.3 X 10' 1.132
8 0.8330 0.4637 7.6 X 10' 1.132 0.8082 0.4502 6.0 X 10' 1.132 0.8291 0.4610 133 1.132
9 0.8110 0.4512 95.6 1.132 0.8082 0.4502 6.0 X 10' 1.132 0.8291 0.4810 133 1.132
10 0.8110 0.4512 95.6 1.132 0.8167 0.4544 22.3 1.132 0.8291 0.4610 188 1.132
11 0.8110 0.4512 95.6 1.132 0.8167 0.4544 22.3 1.132 0:8174 0.4547 8.6 1.132
12 0.8203 0.4563 6.5 1.132 0.4187 0.4544 22.3 1.132 0.8174 0.4547 8.6 1.132
13 0.8203 0.4563 6.5 1.132 0.8200 0:4561 0.14 1.132 0.8174 0.4547 8.6 1.132
14 0.8203 0.4563 6.6 1.132 0.8200 (r:4561 ' 0,14 .1.132 0.8194 0.4558 0.78 1.132
15 0.8199 0.4561 0.63 1.132 0.8200 0.4561 0.14 1.132 0.8194 0.4568 0.78 1.132
16 0.8199 0.4561 0.63 1.132 0.8200 0.4561 0.14 1.132 b.8202 0.4563 0.04 1.132
17 0.8200 0.4562 0.01 1.132 0.8200 0.4611 0.14 .1.132 0.8202 0.4563 0.04 1.132
Table 9.2 Successive approximations to the Initial values of multipliers using the complex
method for steep-descent boundary iteration to minimize final error in boundary conditions
0.7000 0.4000 2.5 X 106 1.123 0.6000 0.5000 6.1 X 1010 0.882 1.0000 1.0000 6.8 X 101, 0.834
1 0.7000 0.4000 2.5 X 105 1.123 0.6000 0.5000 6.1 X 1010 0.882 0,4633 0:1567 7.9 X 106 - 1.916
2 0.7000 0.4000 2.5 X 10' 1.123 0.5869 0.3414 5.8 X 106 1.111 0.4633 0.1667 7.9 X 10, -1.916
3 0.7000 0.4000 2.5 X 105 1.123 0.5869 0.3414 5.8 X 1011, 1.111 0.6014 0.3208 4.4 X 106 1.087
4 0.7000 0.4000 2-5 X 105 1.123 0.6847 0.3705 2.5 X 105 1.116 0.6014 0.3208 4.4 X 105 1.087
5 0.7000 0.4000 2.5 X 106 1_123 0.6847 0.3706 2.6 X 106 1.116 0.7409 0.4197 1.2 X 106 1.127
6 0.7900 0.4000 2.6 X 105 1.123 0.7121 0.4007,. 3,7 X 10' 1.130 0.7409 0.4197 1.2 X 106. 1.127
7 0.7406 0.4156 2.2 X 10' 1.131 0.7121 0.4007 3.7 X 10' 1.130 0.7409 0.4197 1.2 X 105 1.127
8 0.4156 2.2 X 10' 1.131 0.7121 0.4007 8.7 X 10' 1.130 0.7186 0.4020 1.4 X 10' 1.131
9 0.7406 0.4156 2.2 X 10' 1.131 0.7390 0.4131 9.7 X 10' 1.131 0.7186 0.4020 1.4 X 10' 1.131
10 0.7225 0.4032 1.1 X 10' 1.132 0.7390 0.4131 9.7 X 105 1.131 0.7186 0.4020 1.4 X 10' 1.131
11 0.7225 0.4082 1.1 X 10' 1.132 0.7390 0.4131 9.7 X 10' 1.131 0.7372 0.4115 8.1 X 10' 1.132
12 0.7464 0.4171 8.3 X 10' 1.132 0.7390 0.4181 9.7 X 10' 1.131. 0.7372 0.4115 8.1 X 10' 1.132
13 0.7464 0.4171 8.3 X 10' 1.132 0.7433 0.4149 7.2 X 10' 1.132 0.7372 0.4115 8.1 X 10' 1.132
14 0.7370 0.4111 7.9 X 10' 1.132 0.7433 0.4149 7.2 X 10' 1.132 0.7372 0.4115 8.1 X 10' 1.132
15 0.7370 0.4111 7.9 X 10' 1.182 0.7433 0.4149 7.2 X 10' 1.132 0.7417 0.4138 7.2 X 10' 1.132
16 0.7454 0.4161 6.9 X 10' 1.132 0.7433 0.4149 7.2 X 10' 1.132 0.7417 0.4138 7.2 X 10' 1.132
17 0.7454 0.4161 6.9 X 10' 1.132 .0.7437.. 0.415p. 6.9 X 10' 11.132 0.7417 0.4138 7.2 X 10' 1.132
Table 9.3 Successive approximations to the initial values of multipliers using the complex
method for steep-descent boundary iteration to minimize final error in boundary conditions
=zX10(6) xxX10(6)
Iteration -yt(0) - y,(0) E X 10' E X 108 x2 (8)
-y'(0) -ys(U) --`(U) - y,(0) E X 106
= = X102
0.4000 0.6000 8.4 X 10' 0.780 0.5000 0.8000 1.7 X 1010 0.774 0.7000 0.7000 3.3 X 10' 0.834
1 0.4000 0.6000 8.4 X 10' 0.780 0.5767 0.5700 2.1 X 10' 0.836 0.7000 0.7000 3.3 X 10' 0.834
2 0.7654 0.6537 1.2 X 10' 0.874 0.5767 0.5700 2.1 X 10' 0.836 0.7000 0.7000 3.3 X 10' 0.834
3 0.7654 0.6537 1.2 X 10' 0.874 0.5767 0.5700 2.1 X 10' 0.836 0.6566 0.5648 9.8 X 108 0.871
4 0.7654 0.6537 1.2 X 10' 0.874 0.7819 0.6302 7.7 X 108 0.894 0. 6-566 0.5648 9.8 X 108 0.871
5 .0.6939 0.5675 6.9 X 108 0.889 0.7819 0.6302 7.7.x 108 0.894 0.6566 0.5648 9.8 X 108 0.871
6 0.6939 0.5675 6.9 X 108 0.889 0.7819 0.6302 7.7 X 108 0.894 0.7818 0.6170 6.2 X 108 0.902
7 0.6939 0.5676 6.9 X 108 0.889 0.7143 0.5720 6.0 X 108 0.897 0.7818 0.6170 6.2 X 108 0.902
8 0.7769 0.6089 5.7 X 108 0.905 0.7143 0.5720 6.0 X 108 0.897 0.7818 0.6170 6.2 )( 108 0.902
9 0.7769 0.6089 5.7 X 108 0.903 0.7143 0.5720 6.0 X 108 0.897 0.7263 0.5763 5.7 X 10' 0.900
10 0.7769 0.6089 5,.7 X 108 0.906 0.7716 0.6035 5.5 X 108 0.906 0.7263 0.5763 5.7 X 108 0.900
11 0.7340 0.5799 5.5 X 10' 0.902 0.7716 0.6035 5.5 X 108 0.906 0.7263 0.5763 5.7 X 108 0.900
12 0.7340 0.5799 5.5 X 108 0.902 0.7716 0.6086 6.6 X 10' 0.906 0.7669 0.6000 5.5 X 108 0.906
13 0.7340 0.5799 6.6 X 108 0.902 0.7392 0.5826 5.5 X 108 0.903 0.7669 0.6000 5.5 X 108 0.906
14 0.7632 0.5974 5.4 X 108 0.906 0.7392 0.5826 6.5 X 108 0.903 0.7669 0.6000 5.5 x 108 0.906
15 0.7632 0.5974 5.4 X 108 0.906 0.7788 0.6072 5.4 X 108 0.908 0.7669 0.6000 6.6 X 108 0.906
16 0.7632 0.5974 5.4 X 108 0.906 0.7788 0.6072 5.4 X 10' 0.908 0.7732 0.6035 5.4 X 108 0.907
17 0,7632 0.5974 5.4 X 10' 0.906 0.7625 0.5968 5:4 X 10' 0.906 0.7732 0.6035 5.4 X 108 0.907
NUMERICAL COMPUTATION 283
For problems of the specific type considered here, where all initial
values of the state variables are known and final values unspecified, we
can use a more direct approach to steep-descent boundary iteration.. The
value of the objective 6 depends only upon the choice of y(0), for every-
thing else is determined from the minimum-principle equations if all initial
conditions are specified. Instead of minimizing E, the error in final con-
ditions, it is reasonable simply to seek the minimum of S directly by steep-
descent iteration on the initial conditions. Tables 9.4 and 9.5 show
the results of such a calculation using the simplex-complex procedure.
In neither case are the values of yi(O) = -0.8201, yz(0) _ -0.4563
approached, although the same ratio 1.80 of these values is obtained.
It is evident from Eq. (5) that only the ratio,-yl/,y2 is required for defining
the optimum, and Eqs. (4) can be combined to give a single equation for
this ratio. Hence the optimum is obtained for any initial pair in the
ratio 1.80 and, had we so desired, we might have reduced this particular
problem to a one-dimensional search.
where the function x(t) is to be chosen in the interval 1 < t < 2 subject
to boundary conditions x(1) = 1, x(2) = 2. This is a special case of
Fermat's minimum-time principle for the path of a light ray through an
optically inhomogeneous medium. In the notation of Sec. 3.2 we write
ff(x,x,t) = x-I[1 + (±)2J 4 (2)
(0)(0) ys(0)(0)
Iteration -y,(0) -,y2(0) 'Y-Y12
-h X 10' -71(0) -Y:(0) (0) --E X 10' -y,(0) -y:(0) -S X 10'
Yt (0) W(O)
0.5000 0.5000 1.00 0.834 0.5000 1.0000 0.50 0.759 0.2000 0.9000 0.22 0.732
1 0.5000 0.5000 1.00 0.834 0.5000 1.0000 0.50 0.759 0.6600 0.6700 0.99 0.831
2 0.5000 0.5000 1.00 0.834 0.6227 0.3637 1.71 1.107 0.6600 0.6700 0.99 0.831
3 0.5000 0.5000 1.00 0.834 0.6227 0.3637 1.71 1.107 "0.5087 0.3048 1.67 1.086
4 0.5739 0.3136 1.83 1.126 0.6227 0.3637 1.71 1.107 0.5087 0.3048 1.67 1.086
5 0.5739 0.3136 1.83 1.126 0.8227 0.8637 1.71 1.107 0.6460 0.3567 1.81 1.131
6 0.5739 0.3138 1.83 1.126 0.6129 0.3418 1.79 1.132 0.8468 0.3567 1.81 1.131
.
7 0.6591 0.3682 1.79 1.132 0.6129 0.3418 1.79 1.132 0.6460 0.3587 1.81 1.131
8 0.6591 0.8682 1.79 1.132 0.6129 0.3418 1.79 1.132 0.6384 0.3544 1.80 1.132
9 0.6078 0.3381 1.80 -1.132 0.8129 0.3418 1.79 1.132 0.6384 .0.3544 1.80 1.132
10 0.6078 0.3381 1.80 1.132 0.6243 0.3474 1.80 1.132 0.6384 0.3544 1.80 1.132
Table 9.5Successive approximations to the initial values of multipliers using the
complex method for steep-descent boundary iteration to minimize the objective
0.9000 0.5500 1.64 1.068 0.5000 1.0000 0.50 0.759 0.2000 0.9000 0.22 0.732
1 0.9000 0.5500 1.64 1.068 0.5000 1.0000 0.50 0.759 0.9667 0.7083 1.36 0.937
2 0.9000 0.5500 1.64 1.068 0.9873 0.5830 1.69 1.098 0.9667 0.7083 1.36 0.937
3 0.9000 0.5500 1.64 1.068 0.9873 0.5830 1.69 1.098 0.9408 0.5489 1.71 1.108
4 0.9982 0.5744 1.74 1.118 0.9873 0.6830 1.69 1.098 0.9408 0.5489 1.71 1.108
5 0.9982 0.5744 1.74 1.118 0.9600 0.5503 1.75 1.121 0.9408 0.6489 1.71 1.108
6 .0.9982 0.5744 1.74 1.118 0.9600 0.5503 1.75 1.121 0.9995 0.5695 1.75 1.124
7 0.9699 0.5521 1.76 1.125 0.9600 0'.6603 1.76 1.121 0.9995 0.5695 1.75 1.124
8 0.9699 0.5521 1.76 1.125 0.9979 0.5665 1.76 1.126 0.9996 0.5696 1.75 1.124
9 0.9699 0.6521 1.76 1.126 0.9979 0.5665 1.76 1.126 0.9756 0.5539 1.76 1.126
10 0.9957 0.5645 1.76 1.127 0.9979 0.5665 1.76 1.126 0.9766 0.5689 1.76 1.126
I
2116 OPTIMIZATION BY VARIATIONAL METHODS
Eq. (4) can be integrated directly to obtain the solution satisfying the
boundary conditions
x(t) = (6t - t= - 4))s (6)
We wish now to solve Eq. (4) iteratively by a Newton-Raphson
expansion analogous to that developed for nonlinear algebraic equations
in Sec. 2.2. We suppose that we have an estimate of the solution,
x(*) (t), and that the solution is the result of the (k + 1)st iteration,
x(k+')(t). The nonlinear terms in Eq. (4) may be written
x(k+1)x(k+1) = x(k)x(k) + x(k) \\(x(k+l) - x(k)) + x(k) (i(k+1) - x(k))
+ higher-order terms (7a)
(Z(k+1))2 = (±(k))2 + 21(k)(t(k+1) _, t(k))
+ higher-order terms (7b)
With some rearranging and the dropping of higher-order terms Eq. (5)
then becomes a linear ordinary differential equation in x(k+1)
This linear homogeneous equation has two solutions, x(') = 1/t and
x(1) = 1. Using the principle of superposition, the general solution is
a linear combination of the two
x(1) = c1 + C2t-1 (ii)
and the constants are evaluated from the boundary conditions at t 1
NUMERICAL COMPUTATION Z$7
andt=2
x(1)(1) = 1 = C, + C2 (12a.)
x(1)(2) = 2 = C1 + %C2 (12b)
The solution is then
x(') (t) = 3 - 2t-' (13)
Table 9.6 shows the agreement between the exact solution and these
first two Newton-Raphson approximations.
The starting approximation need hot satisfy all or any of the
boundary conditions, though by use of superposition all subsequent
approximations will. For example, with the constant starting value
x(0) = 1.5 Eq. (8) for x(1) becomes
x(u = -23 (14)
The homogeneous solutions are x(') = 1, x(') = t, while the particular
solution obtained from the method of undetermined coefficients is -%t2.
By superposition, then, the general solution is
x(1) = C1 + C2t - /St2 (15)
t x(°)(t) xttl
Table 9.7 shows the start of convergence for this sequence of -Newton-
Raphson approximations, and considering the crude starting value, the
agreement on the first iteration is excellent.
It is helpful to observe here that the principle of superposition can
be used in such a way as to reduce the subsequent algebra. We can
construct a particular solution satisfying the boundary condition at t = 1,
x('D) = 3 - %J2 (1S)
To this we add a multiple of a nontrivial homogeneous solution which
vanishes at t = I
x(Ih) = t -1 (19)
and the right-hand side may be expanded in a Taylor series about the
function y(n) as
s
zC
aFi(Y(n)) - yi(n)) +
0i(n+1) = Fi(Y(n)) + / (y1(n+l)
(4)
i_1 ayi
2S zs
C aFi(y(n)) aF,(y(n))
a yj(n+n + [F() _ ay y,(n)1 (5)
is1 y, 1
I
F,(y(n))
cn+n,n = --- y1(n+1).h (g)
yi
2" OPTIMIZATION BY VARIATIONAL METHODS
Substitution of Eq. (4) into Eqs. (1) and (3) leads to the four equations
in four variables
x1 =
k12x12 _1 +272
-
71
(5a)
k2x2 2
k12x12 712
z2 = (5b)
ksx22 1- 7 2
k12x1
tit = - 2ksysx22 (71 - 272)2 (5c)
k12x12
72 (71 - 272) s (5d)
= 2k27 2x2a
In the notation of the previous section x1; x2, 71i and 72 correspond,
respectively, to y1, y2, y,, y4 Values at t = 0 are given for x1 and z2
(y1 and y2) and at t = 0 for 71 and 72 (y: and y4). The linearized equa-
tions for the iteration become, after some simplification,
k12(x1("))2 2 71(n)
±2('1+0 = k12(x1("))2 2
1-
(71(n))2
] x1(.+,)
k2(x2(n))2
x100 4(72("))2
2
x2%) 1
- (71(n) 2 (n)
x2(n+l) - 2(y2(n))2 y1(n+1)
(71(n))2
+ k12(x1(n))2 _ (71(n)))
-
k12x1(n)
2k2(x2(n))27s(n)
(7 i(n) - 27z("))2 (6c)
27x(")
- (71(n))2
72(n+1)
k12(xl(n))2
(71(n) - 272("))2 (6d)
2k2(x2(n))i72(n)
72(n+l).P, is the solution of Eqs. (6) with initial conditions xlo, x20, 0, 1.
(The latter two are arbitrary.) The homogeneous equations are obtained
by deleting the last term in each of Eqs. (6), and the homogeneous solu-
tions, with components xl(n+l).hk, x2(n+l).Ak, 71(n+1).h,, 7:(n+1),hk, k = 1, 2,
are solutions of the homogeneous equations with initial conditions 0, 0,
1, 0 and 0, 0, 1, 1. (The latter two are arbitrary but must be inde-
pendent for the two solutions and not identically zero.) The general
solution can then be written
xl("+1)(t) = x1('+I).P(t) + c1x1(n+l).A,(1) + C2x1(n+1).k(t) (74)
x2(n+1)(t) = x2(n+1).P(t) + Cax2(n+l).A,(t) -4- c2x2(n+l).A,(t) (7b)
= 7t(n+1),P(J) +
71(n+1)(t) C171(n+1).A,(t) + c271(n+l).k0)
(7c)
72(ntll(t) = .2(n+1).P(t) + C172(n+1).A,(t) + C272(n+1).)y(t)
(7d)
Convergence of xi and x2 is shown in Fig. 9.3, and the results of the first
five iterations are listed in Table 9.8.. The rather poor starting values
result in rapid convergence, though with some oscillation, and it can,
in fact, be established that when convergence occurs for this Newton-
Raphson procedure, it is quadratic.
The computational techniques discussed thus far make use of the neces-
sary conditiops for the optimal decision function at each iteration and
are often called indirect methods. Reasonable starting estimates can
often be obtained by first assuming a function u(t) and` then solving
state and multiplier equations to obtain functions or boundary con-
ditions, as required, or by the use of the direct methods to be developed
in the following sections. .
i x((0) X 10' XI(" X 10' x;(') X 10' X 102 Xs(') X 10' x,(') X 10'
0 1.0000 1.0000 1.0000 .1.0000 1.0000 1.0000
2 1.0000 0.6434 0.6372 0.6620 0.6634 0.6634
4 1.0000 0.6370 0.5119 0.5332 0.5342 0.5343
6 1.0000 0.6607 0.4393 0.4486 0.4493 0.4493
8 1.0000 0.6602 0.3927 0.3860 0.3864 0.3864
i x:(0) X 10' x:(') X 10' x:(') X 10' x2(') X 102 xs(`) X 10' x:(') X 10'
t From E. S. Lee, Chem. Eng. Sci., 21:183 (1966). Copyright 1966 by Pergamon
Press. Reprinted by permission of the copyright owner.
z=ax+u (1)
y = -ay (2)
NUMERICAL COMPUTATION 2!S
The goal is to minimize 8[x(0)1, and for the present we shall assume.that
the values of x(0) are entirely unconstrained.
If we choose any function u(t) which satisfies the upper- and lower-
bound constraint, we may integrate Eq. (1) and obtain a value of 8.
The effect of a small change ou(t) in the entire decision function is then
described to first order by the linear variational equations
ax,(0) (4)
8x;
ti
2!6 OPTIMIZATION BY VARIATIONAL' METHODS
Until this point the approach does not differ from the development
of the weak minimum principle in Sec. 6.5. Now, as in Sec. 2.5, we make
use of the fact that u(t) is not the optimum, and we seek au(t) so that t
is made smaller, or S8 < 0. An obvious choice is
SS = - fo w(t)
af;z dt < 0 (10)
y` au
(au)
y d ii au + xg(su)'] = 0 (13)
G(t) y; af;/au
au = -A (;
af,/au)2 (16)
[J0 G(r)
fr
dr
\ y'
where G(t) is the inverse of g(t) and the positive square root has been
used. For later purposes it is helpful to introduce the notation
\2
rEE = f G(r) y; a') dr (17)
in which case
G(t) I y; af;/au
au = -A _
(18)
1.",
EEE
Thus,
w(t)
G( (19)
1881,
t
H= y;f: (20)
298 OPTIMIZATION BY VARIATIONAL METHODS
for any decision function, not just the optimum, the improvements in
u may be conveniently written
Su = -zv(t) (21)
au
In the case of more than one decision it easily follows that
61H
5u, wi; (22)
au
aw - win C1Hft
au;n (24)
with
Hn = \ 1tinlia (25)
Ls
af" as
y'n-1 = yin axtp;-1 7tiN = aXN
(26)
it = -2k,u A X1 xl (1a)
xz = 4k,u A
+ - 4kzu2 2
(lb)
x2 (A + x2)2
S = -X2(0) (2)
2k,u
A+x2 (Y1 - 2Y2) Y1(0) = 0 (3a)
2k,ux1 8k2u2Y2Ax2
'Y2 = - (A + x2) 2 (y I - 272) + (A + X2)' 72(0) _ -1 (3b)
aH x22
The values of the parameters are those used for previous calcu-
lations. Following a choice of u, a(t), the new decision function is caleu-
lated from the equation
unew(t) = u - W (5)
au
where Eq. (4) for all/au is evaluated for u and the solutions of Eqs. (1)
and (3) are calculated using u. For these calculations w was initially
set equal to the constant value of 25. The program was written so as
to reduce w by a factor of 2 whenever improvement was not obtained,
but effective convergence was obtained before this feature was needed.
Figures 9.4 and 9.5 show successive pressure profiles computed from con-
0.7
0.6
0
0.5
1
0.4 2
0.3
3
0.2
Fig. 9.4 Successive approximations to
0.1
the optimal pressure profile using steep 5
descenti, starting from the constant 00 1 2 3 4 5 6 7 8
policy u = 0.5. Residence time t
300 OPTIMIZATION BY VARIATIONAL METHODS
I
I
2
i
3 4
I
5
function on successive iterations using
Iteration number steep descent.
NUMERICAL COMPUTATION 301
au - u2 (71 - 72) u2
y2 (5)
The parameters for computation are the same as those in Sec. 9.3. The
new decision is calculated from the equation
) IZ At
In', [ a
or, combining (At)3h into the step size,
S
w(t) = (9)
(tn)]T
In I1 [",
302 OPTIMIZATION BY VARIATIONAL METHODS
or
S N34 (11)
For simplicity b is taken as an integer. Equation (6) is written for
computation as
unew = u - b (t)
ell/au
N lI(in)ls}1
(12)
[_5_U
{ 1,
where b(t) is a constant b, unless Eq. (12) would cause violation of one
of the constraints, in which case b(t) is taken as the largest value which
does not violate the constraint. The constant b is taken initially as the
smallest integer greater than N4 and halved each time a step does not
lead to improvement in the value of S. For the calculations shown here
N = 60, and the initial value of b is 8.
Figure 9.7 shows 'successive iterations startijig from an initial con-
stant policy of u = u* except over the first integration step, where u is
linear between u* and u.. This initial segment is motivated by the
10
6II
4
2
i
I 2
i
3 4 5
i
6
O
Residence time f
355
a
E
340
3351 I l I 1 I
0 1 2 3 4 5 6
Residence time f
Fig. 9.8 Successive approximations to the optimal tem-
perature profile using steep descent, starting from the con-
stant policy u = 355. [From J. M. Douglas and M. M.
Denn, Ind. Eng. Chem., 57(11):18 (1965). Copyright 1965
by the American Chemical Society. Reprinted by permission
of the copyright owner.]
0
' 0.30
00.25
Fig. 9.9 Improvement of the objec-
tive function on successive iterations
using steep descent. [From J. M.
Douglas and M. M. Denn, Ind. Eng.
Chem., 57(11):18 (1965). Copyright
1965 by the American Chemical Society.
Reprinted by permission of the copy- 2 4 6 8 10 12 14
right owner. ] Iteration number
3" OPTIMIZATION BY VARIATIONAL METHODS
y "
ye-1 = 1 + 8k2oe- J,/4-* 72N = -1 (4b)
8 - y2"E=k2oe-a,'l""x2"(1 + 28k1oe-8-'lu"xi%)
(un)2 (1 + 8k2oe-e-'/u")(1 + 28kioe'80"x1")
(5)
Following the specification of a temperature sequence {in I and the
successive solution of Eqs. (1) and Eqs. (4), the new decision is obtained
NUMERICAL COMPUTATION 305
355
350
U,
u2
E U3
340
335
0 5 10 15
Iteration number
Fig. 9.10 Successive approximations to the optimal tem-
perature sequence in three reactors using steep descent.
[From M. M. Denn and R. Aria, Ind. Eng. Chem. Funda-
mentals, 4:213 (1965). Copyright 1965 by the American
Chemical Society. Reprinted by permission of the copyright
owner.]
306 OPTIMIZATION BY VARIATIONAL METHODS
355
335 ___ 1 I I I I I
1 5 10 20 30 40 50 60
Stoge numbern
For these calculations the tolerance in the constraint was set at 10-1.
Figure 9.12 shows the result of a sequence of penalty-function
calculations for N = 60 starting with K = 1. Convergence was obtained
in 17 iterations from the linearly decreasing starting-temperature sequence
to the curve shown as K = 1, with an error in the constraint x1N = 0.4
of 1.53 X 10-1. Following Eq. (11), the new value of K was set at
1.53 X 102, and 107 iterations were required for convergence, with a
constraint error of 8.3 X 10-1. A value of K of 1.27 X 10' was then
used for 24 further iterations to the dashed line in Fig. 9.12 and an error
of 10-1. This extremely slow convergence appears to be typical of the
penalty-function approach and is due in part to the sensitivity of the
boundary conditions for y to small changes in the constraint error, as
may be seen from Eq. (10a).
355
K'I.53 X102
340
5 t0 20 30 40 50 60
Stage number n
with bounds on u,
u* <u<u* (2)
For any function u(t) which lies between the upper and lower bound
we can integrate Eqs. (1) to obtain a value of & and of 0 which will
generally not be zero. The first-order effect of a small change is then
described by the equations
ax,(9) (6)
ax;
We now define two sets of Green's functions for Eq. (4), y and 4, satis-
fying the adjoint differential equation but different boundary conditions
(7)
axi
af, a11
(8)
ax; ax;
Using Green's identity, Eqs. (5) and (6) will then become, respectively,
SS = fo y; au au dt (9)
a= Io
E ,y; af' su dt (10)
NUMERICAL COMPUTATION
and ask for the function Su which minimizes M in Eq. (9) for a fixed
distance 0' and a specified correction to the constraint 34. This is an iso-
perimetric problem with two integral constraints, and hence two constant
Lagrange multipliers, denoted by X and is. The Euler equation is
or
au - -TX-('Y.+i4)au
where G(t) is the inverse of g(t).
We need to use the constraint equations (10) and (11) to evaluate
X and P. By substituting Eq. (13) for Su into Eq. (10) for the fixed
value 6.0 we obtain
a
-y, T dr
- fo G(r)
aui
2
dr (14)
P = - Icm + A so
Ioe
(16)
If Eqs. (13) and (16) are now substituted into Eq. (11) for A2, we obtain,
after some algebraic manipulation,
1
TX-
- ± 1IImmlL2 - (6.0)2
1
(17)
L ee - J 2J
310 OPTIMIZATION BY VARIATIONAL METHODS
The presence of the square root imposes an upper limit on the correction
30 which can be sought, and a full correction cannot generally be obtained
in a single iteration. The procedure outlined here is geometrically equiv-
alent to projecting the gradient onto the subspace defined by the con-
straint. Identical equations are obtained for systems with difference
equations if integrals are replaced by sums and continuous variables by
their discrete analogs.
If the gradient-projection procedure is applied to the reactor-tem-
perature problem of the preceding section with ¢(xN) = x1N - 0.4, the
only additional computations required are the recalculation of Eqs. (4)
of that section, substituting 4'1", 412" for yl", y2", with boundary conditions
1, 0 on *N and 0, -1 on yN, followed by calculation of the sums Is., Is*f
and Ii,. Computations were carried out using a maximum correction
60 of 0.01 in magnitude for each iteration. Lines I and II in Fig. 9.13
0.25
0.20
0.15
0
00.10
9.14 MIN H
Steep descent provides a rapidly convergent method of obtaining approxi-
mate solutions to variational problems, but it is evident from the examples
that ultimate convergence to the solution of the minimum-principle
necessary conditions is slow and computationally unfeasible. The min-H
procedure is one which can be used only in the latter stages of solution
but which will, with only minor modifications of the computer code,
lead to a solution satisfying the necessary conditions.
The system and multiplier equations are
x; = f;(x,u) x;(0) = x;o (1)
y; - - Yi
ax,
y;(0)
ax;
(2)
For a given decision function f4(t) the solutions to Eqs. (1) and (2) are
denoted as Z(t), Y(t). Writing the hamiltonian in terms of these latter
variables, we have
(3)
(4)
r
A value of r = 2 has generally been found to be satisfactory. Modi-
fications similar to those in Sec. 9.13 can be made to accommodate
end-point constraints.
The relation to steep descent may be observed by considering the.
special case for which the minimum of H occurs at an interior value.
312 OPTIMIZATION BY VARIATIONAL METHODS
5
y' au
This can be expanded about u for a Newton-Raphson solution as
`Jt
at;(31,a
au + L. ry`
a2};(2,u)
au2
0 - u) + .. . =0 (6)
a
That is, we get nearly the same result as a steep-descent procedure with
the inverse of a2H/au2 as the weighting factor w. At the minimum the
hessian of H is positive, and so Eq. (7) will allow convergence, and this
must be true by continuity arguments in a neighborhood of the minimum.
For staged systems we have no strong minimum principle in general, and
a2H/au2 need not be positive near the optimum, so that this procedure
might lead to divergence arbitrarily close to the minimum in a staged
system.
As an example of the convergence properties of the min-H algorithm
we consider the problem of the optimal tubular-reactor temperature
profile for the consecutive reactions.The state and multiplier equations
(1) and (4) of Sec. 9.11 are solved sequentially to obtain xl, 22, tit, tit
for the specified u. Using the necessary conditions in Sec. 9.3, we then
find il from the relation
u* u(t) > u*
tZ = v(t) u* < v(t) < u* (8)
U* u(t) < u*
with
u (t)
E_ - E; (9)
12 x 2 k2o
In
(tit - 'Y1)xl2klo
For these computations the initial profile was taken as the constant
u = 345 and r was set at 2. The results are shown in Table 9.9, where e
is the integral of the absolute value of the difference between old and
new temperature. The stabilizing factor r = 2 clearly slows convergence
for the small values of t. Convergence is not uniform and, indeed, S is
not uniformly decreasing, but very rapid convergence to a solution
satisfying the necessary conditions is obtained in this way. Numerical
Table 9.9 Successive approximations to optimal temperature profile using the min-H method
Iteration Number
t 0 1 2 4 6 8 10 12 14
x,(6) 0.4147 0.4088 0.4186 0.4196 0.4196 0.4196 0,4196 0.4196 0.4196
x,(6) 0.4952 0.4999 0.4977 0.4973 0.4974 0.4974 0.4974 0.4974 0.4974
-s; 0.3196 0.3225 0.3233 0.3232 0.3233 0.3233 0.3233 0.3233 0.3233
29.7 18.1 4.3 0.70 0.29 0,086 0.022 0.0054 0.0013
314 OPTIMIZATION BY VARIATIONAL METHODS
and we seek to minimize 6[x(9)]. If we choose u(t) and expand Eq. (1)
to second order, we have
bx-axbx+aubu+ax2bx2+axaubxbu+au bu2
bx(0) = 0 (2)
The corresponding second-order change in E is
Sa; = g' bx(9) + 3E" 5x(9)2 (3)
would correspond to Eq. (5) without the term a'f/(ax au) Ox, so that
NUMERICAL COMPUTATION 31S
Bu(t) + (L\
auz_I ax a 1o exp Cllr ax (o)] au (r) bu(r) dr
__ a2f -1 of (7)
au2> au
There is no computational advantage in doing so, however, for the solu-
tion of an integral equation is not, an easy task and Eq. (5), although it
contains the unknown ox, can be used directly. Noting that ax is simply
x - x, it follows that the best second-order correction bu is a known
explicit function of x, which we denote as bu(x). Equation (1) can then
be written
x = fix, a + bu(x)] (8)
H= Yf, (2)
(3)
316 OPTIMIZATION BY VARIATIONAL METHODS
Now, if we expand Eq. (1) about some choice u(t) and retain terms
to second order, we obtain
a2 =
64 = ax;
aui 6U + ,-,
OXJ OX
ax; axk +
8x,
t9u ox, Ou
).k
1 2f,
+ a au= azi(o) = 0 (4)
C
_ - a(ax;)
A _
Y'`
af ;
ax; - /C, f axk
k Y' ax; axk
a ,
.
4'i
aft
2
ax; au
alb
(7)
a(as) aE
7( ) a(ax;) = ax; + k ax, axk
12-'8- axk (8)
or
a=ft 4'i a=f
su = - auz I au
aft + id ax au s
ax; j (10)
i
5Yi=4',-Y; (11)
NUMERICAL COMPUTATION 9vi
a2H a2f
ax; au
au - ay;
ax; au
au (12)
while the boundary condition is obtained from Eqs. (3) and (8)
I a2& I a2&
Sy;(B) = axk - ax; (13)
ax; + ax; axk k ax, axk 6xk(9)
+G ax
uLfi
au
Lfi ) au ay` auSx'
1
'
aax;2fau ax;) (14)
+ I.1ay`
We require a solution to the system of Eqs. (1) and (12) to (14) which
can be implemented for computation without the necessity of solving a
boundary-value problem.
The fact that we have only quadratic nonlinearities suggests the
use of the approximation technique developed in Sec. 4.8. We shall
seek a solution of the form
ay;(t) = g;(t) + I M;k(t) axk(t) + higher-order terms (15)
k
The functions g; and M;k must be obtained, but it is evident from Eq. (13)
that they must satisfy the final conditions
g,(8) = 0 (16a)
2
M'k(B) (16b)
= ax, axk
We can also express Eqs. (4), (12), and (14) to comparable order as
I
:
a; a2
S' au
y' ay` ax; - ax; axk axk - ax; au
+ higher-order terms (18)
au
(a2H)-l
aH + ayt aJ; + a2H
au au au ax; au
-- higher-order terms (19)
311 OPTIMIZATION BY VARIATIONAL METHODS
On the other hand it must be possible to differentiate Eq. (15) with respect
to t and write
aii=#;+I M;kaxk+I mo,61k (21)
k k
- (!)'
a=H Ma=H 1
of
au=au au ax; -( k 'k au) (au=> ( au'i
M
aH
a2 H aH aH _ -1 aH
2
=0 Mi;(9) =
a8
:
(23)
gi
+ azi ax;
_ (
L
k
axi au au=
afk (a=H\-' at; _ af,
Mik au au= J au ax; + au=
axi au
af, a=H 1
axi ax;
au au axi J 9i
of 0H
H 82H -1 aH
a=H-1 aH
0 gi(9) = 0
au (au=) au - axi au au=) au -
(24)
Equation (23) for the "feedback gain" is simply the Riccati equation
obtained for the optimal control problem in Sec. 8.2, and Mi; is symmetric
NUMERICAL COMPUTATION 31!
(M;; = M;;). As convergence occurs and 8H/au goes to zero, the forcing
term in Eq. (24) also goes to zero and the "feedforward gain" vanishes.
The correction to the decision, bu(t), is then obtained from Eqs. (19)
and (15) as
a2 /-1 M
bu(t) = au / + g' aut
\
+ au' M,:1 (x: _f:) (25)
where x; is the (as yet unknown) value of the state corresponding to the
new decision function.
The computational algorithm which arises from these equations
is then as follows:
1. Choose u(t), solve Eqs. (1) for x and then (3) for 7 in succession, and
evaluate all partial derivatives of H.
2. Solve Eqs. (23) and (24), where all coefficients depend upon 2(t),
!(t), and Y(t).
3. Compute u(t) from the expression
unew(x,t) = t(t) + T
bu(t) (26)
3 4 5 6
Residence time t
Fig. 9.14 Successive, values of feedback gains using the
second-variation method.
Iteration number
t 0 1 2 4 5
x (8) 1 3-3338 X 10-1 1 3.6683 X 10-1 3.&309 X 10-1 3.8825 X 10-1 3.8624 X 10-+ 3.8622 X 10-1
z 1(8) 1.0823 X 10-1` 1.1115 X 10-1 1.1315 X 10-1 1.1319 X 10-1 1.1319 X 10-1 1.1319 X 10-1
NUMERICAL COMPUTATION 321
BIBLIOGRAPHICAL NOTES
Section 9.1: We shall include pertinent references for individual techniques as they are
discussed. We list here, however, several studies which parallel all or large parts
of this chapter in that they develop and compare several computational procedures:
R. E. Kopp and H. G. Moyer: in C. T. Leondes (ed.), "Advances in Control Sys-
tems," vol. 4, Academic Press, Inc., New York, 1966
L. Lapidus: Chem. Eng. Progr., 63(12):64 (1967)
L. Lapidus and R. Luus: "Optimal Control of Engineering Processes," Blaisdell
Publishing Company, Waltham, Mass., 1967
322 OPTIMIZATION BY VARIATIONAL METHODS
Section 9.4: Little computational experience is available for th4, rather obvious approach.
Some pertinent remarks are contained in the reviews of Noton and .Storey and
Rosenbrock cited above; see also
J. W. Sutherland and E. V. Bohn: Preprints 1966 Joint Autom. Contr. Conf., Seattle,
p. 177
Neustadt has developed a different steep-descent boundary-interation procedure for
solving linear time-optimal and similar problems. See papers by Fndden and
Gilbert, Gilbert, and Paiewonsky and coworkers in the collection edited by Bala-
krishnan and Neustadt and a review by Paiewonsky,
B. Paiewonsky: in G. Leitmann (ed.), "Topics in Optimization," Academic Press,
Inc., New York, 1967
Sections 9.5 to 9.7: The Newton-Raphson (quasilinearization) procedure for the solution
of boundary-value problems is developed in detail in
R. E. Bellman and R. E. Kalaba: "Quasilinearization and Nonlinear Boundary-
value Problems," American Elsevier Publishing Company, New York, 1965
NUMERICAL COMPUTATION 323
Sections 9.9 to 9.13: The idea of using steep descent to solve variational problems origi-
nated with Hadamard; see
R. Courant: Bull. Am. Math. Soc., 49:1 (1943)
The first practical application to a variational problem appears to be in
J. H. Laning, Jr., and R. H. Battin: "Random Processes in Automatic Control,"
McGraw-Hill Book Company, New York, 1956
Subsequently, practical implementation was accomplished, independently about 1960 by
Bryson, Horn, and Kelley; see
Section 9.14: The min-H approach was suggested by Kelley, in the article cited above,
and by
S. Katz: Ind. Eng. Chem. Fundamentals, 1:226 (1962)
For implementation, including the extension to systems with final constraints, see
R. G. Gottlieb: AIAA J., 5:(1967)
R. T. Stancil: AIAA J., 2:I365 (1964)
The examination of convergence was-in
M. M. Denn: Ind. Eng. Chem. Fundamentals, 4:231 (1965)
The example is from
R. D. Megee, III: Computational Techniques in the Theory of Optimal Processes,
B. S. thesis, University of Delaware, Newark, Deli, 1965
PROBLEMS
9.1. Solve the following problem by each of the, methods of this chapter.If a computer
is not available, carry the formulation to the point where a skilled programmer with
no knowledge of optimization theory could code the program. Include a detailed
logical flow sheet.
2
i1 = - arctan u - x1
x
is = xI - x:
is = x2 - xa
x1(0) = 0 x2(0) _ -0.4 xa(0) = 1.5
min & = 103 1(2x:)sa + xa= + 0.01u'1 dt
Take a = 2 and 10. The latter case represents a penalty function approximation
for Ix,! < 5J. The arctangent is an approximation to a term linear in u with the
constraint Jul < 1. (This problem is due to Noton, and numerical results for some
cases may be found in his book.) Repeat for the same system but with
i1 -U-x1
Ju l < 1
9.2. Solve Prob. 5.8 numerically for parameter values
k1=ka=1 k2 -10
9-0.4 and 9-1.0
Compare with the analytical solution.
9.3. Solve Prob. 7.2 using appropriate methods of this chapter. (Boundary iteration
is difficult for this problem. Why?)
9.6. Develop an extension of each of the algorithms of t%is chapter to the case in
which a is specified only implicitly by a final constraint of the form
#(x(e)l = 0
Note that it might sometimes be helpful to employ a duality of the type described in
Sec. 3.5.
10
Nonserial Processes
10.1 INTRODUCTION
A large number of industrial processes have a structure in which the
flow of material and energy does not occur in a single direction because
of the presence of bypass and recycle streams. Hence, decisions made
at one point in a process can affect the behavior at a previous point.
Though we have included the spatial dependence of decisions in processes
such as the plug-flow reactor, our analyses thus far have been couched
in the language and concepts of systems which evolve in time. For
such systems the principle of causality prevents future actions from
influencing the present, and in order to deal with the effect of feedback
interactions in spatially complex systems we shall have to modify our
previous analyses slightly.
The optimization of systems with complex structure involves only
a single minor generalization over the treatment of simple systems in
that the Green's functions satisfy a different set of boundary conditions.
There has been some confusion in the engineering literature over this
326
NONSERIAL PROCESSES 327
problem, however, and we shall proceed slowly and from several different
points of view. The results of the analysis will include not only the
optimization of systems with spatial interactions but also, because of
mathematical similarities, the optimal operation of certain unsteady-state
processes.
x(0) x(8)
L> x(0)-G[x,,x(8) x=f(x,u)
t
Fig. 10.1 Schematic of a continuous recycle process.
32i OPTIMIZATION BY VARIATIONAL METHODS
0<t<0 (6)
y`
ill ax;
NO) ax;(8) = 1 NO
[I aG, ax;(B) ] +fp y: a ' au dt
0 (0)
(7)
or
aG ; to aH
I y:(B) - l 70) axi (e)] ax; (9) =
fo au
su do (8)
afi
- fo au au dt (12)
where the input to the first stage is related to the effluent x' and feed
NONSERIAL PROCESSES 32!
2
N=fN(xN-t,uN)
N
N
x, by the equation
xie = G,(xi,xN) (14)
, Q,, _ aS
(17)
ysN ^fj ax;N ax:N
i-t
Unreocted X
or, equivalently,
= fo [k(u)F(x1) - g(u)] dt (4)
The hamiltonian is
H = -y,k(u)F(x,) + y2[-k( u)F(x1) + 9(u)] (7)
aH
y2° -ax2=0 (8b)
be interior, then
aH = 0 = - (y, + 1)F(xi)k'(u) + g'(u) (10)
au
or
g'(u) = (y, + 1)F(xi) (11)
k' (u)
It is convenient to differentiate both sides of Eq. (11) with respect
to t to obtain
d
g(u) = 1,F(x,) + (71 + 1)F'(x,)x,
Wt k,( u )
_ (y, + 1)k(u)F'(x1)F(x,) + (y, + 1)F'(x,)[-k(u)F(x,)]
=0 (12)
y;(B) = (8)
ax; - 4 vi
where the 'n; and v; are undetermined multipliers. The functions r;
satisfy the following equations and transversality conditions:
t'; = 0 r, = const (9)
r;(e) = v; (11)
Combining Eqs. (7) to (11), we are led to the mixed boundary condition
for the y; obtained previously
S
d(i d6
W ax,
(12)
i-1
Setting partial derivatives with respect to each of the variables u', u2,
. . UN, zi°, xi', . . . , x;N to zero, we obtain
a. _ S
VU_
:n afin
aun
= o (4)
i-1
S
a.C
ax'°
X,n 1 T, Xn+1
ax n
=O n=0,1,2,...,N-1
i-1
(5)
aae as X.N + ° aG; _o
aX7°5-ZW - ill I ax:N
(6)
where S(k) is the number of state variables required in the kth subsystem
and a single decision is assumed for convenience. The subsystems are
related to one another (by mixing equations of the form
x(4)0 = G(k)({z,1,fz(1)N1) (2)
a8 = axj(i)N (6)
axj(1'N
S(k)
0f.(k)
ax/i(k) 0<t<0(k)
- 1 77(k) i = 1, 2, . .
S(k) (7b)
i-1
subsystems as
8(k) S(k)
)
)i(k).V bxi(k)N I .yi(k)O Sxi(k)o + s(k)
L.r au(k) bu(k) (11
i-1 i-1
We now substitute Eq. (5) for bx,(k)0 into Eq. (11), which is written
as
S(k) s(k) S,)) aGi(k>
7i(k)N bxi(k)N - p aH(k)
axi(I)N = cl(k)
4 ,,i(k)O j..1
II axj(!)N
u
a(k) 50)
i-1 i-I I
(12)
Equation (12) is then summed over all subsystems
3(k) S(k) S(,)
In the first term the dummy indices for summation, i and k, may be
replaced by j and 1, respectively, and order of finite summation inter-
changed in the second such that Eq. (13) is rewritten
S(n Su) S(k)
a(y,
yj (i)N axi(!)N yf(k)U 11 axj (1)N
1 j-1 1 j-1 ( k i-1
ax(I)N/
OH(k)
c7 (k) au(k) a2E(k) (14)
A;
or, finally,
Su) S(k )
yi(k)o aGi(k)1
all(k)
1 j-1
(ywv
- k i-1
ax (I)N J/
bx.(°N =
1
k
eS<k)
au(k)
au(k) (15)
Comparison with Eq. (6) for & dictates the boundary conditions for
the Green's functions
So)
Yi(1)N - .ri(k)0 6Cii
(R,
azi(pN
_ dE
ax.(I)N
(16)
k i-1
This is a generalization of Eqs. (11) and (17) of Sec. 10.2. The first-
order variation in the objective can then be written explicitly in terms
of the variations in decisions by combining Eqs. (6), (15), and (16)
aH(k)
59 _ (k) bulk) au(k) (17)
k
NONSERIAL PROCESSES 337
We may now adopt either of the two points of view which we have
developed for variational problems. If we are examining variations
about a presumed optimum, SS must be nonnegative and analyses identi-
cal to those in Sees. 6.5 and 7.4 lead to the weak minimum principle:
The Hamiltonian H(k)n or H(k) in each subprocess is made stationary
by interior optimal decisions and a minimum (or stationary) by
optimal decisions at a boundary or nondifferentiable point.
zG(k)(t) aH(k)/au(k)
5.'(k)(t) (19b)
all(k)
S(k)G(k)
k
au(k)
)7'
where A is a step size and G(k) n, G(k) (t) are nonnegative weighting functions.
X1 --" Xz -- products
338 OPTIMIZATION BY VARIATIONAL METHODS
Solvent
E7
Solvent
I II
Prod ct
Extraction of x,
Solvent (choose amount of solvent)
removal
I 2 3
Feed
pure x, Mixing Reaction x,-x2--x3
(choose temperatures)
The material-balance equations about the first and second stage of the
extractor are, respectively,
x11 -I- u11[4'(x11)
- 0111)1 - x18 = 0 (2)
X111 + 4111(x111) - x11 = 0 (3)
Here u11 is the ratio of volumetric flow rates of solvent to product stream
and #(x1) is the equilibrium distribution between the concentration of
x1 in solvent and reactant stream. Because the solvent extracts X1
only, there is no material-balance equation needed for x2 about the
extractor, for x23 = x211. The external feed is pure X1i so that the
NONSERIAL PROCESSES
feed to reactor 1 is
x1° = xu + u"I '(xi') (4a)
x2° = 0 (4b)
We wish to maximize the production of X2 while allowing for costs of
raw material and extraction, and hence we seek to minimize
8 = -x23 - CxIII + OuII (p)
The temperatures u', u2, u3 and solvent ratio u11 are to be chosen subject
to constraints
U* < u', U2, u3 < u* (6a)
0<u"I (6b)
Equations (1) to (5) are not in the form required for application
of the theory, for, though we need not have done so, we have restricted
the analysis to situations in which a decision appears only in one stage
transformation and not in mixing conditions or objective. This is
easily rectified by defining a variablet x3 with
x3' = x3' (7)
x31I = x3I + uII (8)
Equations (2) and (3) are then rewritten, after some manipulation,
4,(xII) - xI` = 0 (9)
xIII + uII4,(xi"1) - X11
=0 (10)
23' = 0 (lld)
The system is then defined by Eqs. (1) and (7) to (11), and the objective
is rewritten
8 = --:X1 a - X1 11 + Ox3II (12)
r- II
1 Hi 2
1 3 I
2e,Y2nk1pe E,"/u"xin
+ (1 + 28k1oe-E''lu'x1")(1 + 8k20e_E='!
n = 1,2,3 (13a)
72"-1 = 1 + k620e-E="'U'
ry2" n = 1, 2, 3 (13b)
yI = ,(xll)
?1
(14a)
171I1
i
= 1 + ull#'(xi') (14b)
y23 = -1 (15c)
.I
7Y31I + 3t1 , (X19 ' or (15d)
- Y1"(1 +
aHn _ 0 - y2nE2k2oe-E="'"X2"(1 + 20kjoe-E'''""x1")
au". (u")2 (1 + 9k2oe-E! I"")(1 + 20k10e-E"''""xl")
n = 1, 2, 3 (16a)
OHII 11 0111 11 0311
aull - 71 aull + 13 auli
7i ll_(x 1 i1) II (16b)
1 + u' I.l" (x i11) + '1' 3
NONSERIAL PROCESSES -341
It is evident that the artificial variables x3', x3II and -y3I, 73 11 are never
needed in actual computation.
The simultaneous solution of the material-balance relations, Eqs.
(1) to (4), and Green's function, Eqs. (13) to (15), with the optimal tem-
peratures and solvent ratio determined from the weak minimum principle
by means of Eqs. (5) and (16), is a difficult task. Application of the
indirect methods developed in Chap. 9 would require iterative solution
of both the state and multiplier equations for any given set of boundary
conditions because of the mixing conditions for both sets of variables, and
even with stable computational methods.this would be a time-consuming
operation. Steep descent, on the other hand, is quite attractive, for
although the material-balance equations must be solved iteratively for
each assumed set of decisions, the multiplier equations are then linear
with fixed coefficients and can be solved by superposition.
To illustrate this last point let us suppose that the decisions and
resulting state variables have been obtained. Addition of Eqs. (15a) and
(15b) leads to
ay1° = b YI (20)
1 + 29xt'"k3oe-E,'l"
-
a
8
by1' uII - b7'I° -113 + uII + '11° (21)
342 OPTIMIZATION BY VARIATIONAL METHODS
and combining Eqs. (19) to (21), it follows that the correct value of yl' is
11 - "T IC + ION'
yl = lla + 3
1+
u"I"(x,')[1 + u"IL"(xi)] X111 1 + 28xI kloe-1W1u°
(22)
Thus, once the temperatures and solvent ratio have been specified and
the material-balance equations solved iteratively, the equations for the
Green's functions need be solved only twice with one application of Eq.
(22).
The corrections to the values of u', us, u', u"I are calculated using
Eqs. (16a) and (16c) from the relations
bun = -A
= -A[ 3
3
()2 Gn aHn/aun
GII aH"/aull
u'Ibull (23a)
(23b)
G. aI (OH-)2
+G II au"'CauII)J
The physical parameters used in the calculations in Secs. 9.3 and 9.12
were used here, with 0 = 2 and a total reactor residence time of 6. The
function 4,(x,) is shown. in Fig. 10.7 and has the analytical form
0.9
0.8
0.7
0.6
0.5
0.4
0.3
Fig. 10.7 Equilibrium distribution func-
tion of feed between solvent and reactant
0.2 streams. [From M. M. Denn and R.
0.1 Aris, Ind. Eng. Chem. Fundamentals,
4:248 (1965). Copyright 1965 by the
0 0.2 0.4 0.6 0.8
American Chemical Society. Reprinted
X1 by permission of the copyright owner.]
NONSERIAL PROCESSES 343
tation, G" was taken as 0.005. As in Sec. 9.12 the initial step size a
was set equal to 2, with the step size halved for each move not resulting
in a decrease in the value of S. The criterion for convergence was taken
to be a step size less than 10-1. In all calculations the material-balance
equations were solved by a one-dimensional direct search. Figure 10.8
shows a typical convergence sequence, in this case for o = 0.30.
The profit -S is plotted in Fig. 10.9 as a function of extraction cost.
The horizontal line corresponds to the optimal nonrecycle solution found
in Sec. 9.12, and in the neighborhood of the intersection two locally opti-
mal solutions were found. Figure 10.10, for example, shows another set
of calculations for a = 0.30 starting at the same initial temperature policy
0.4 344
0.3 343
0.2 342
0.1 341
0 340
11 I 1 1
0 5 10 15
Iteration number
as in Fig. 10.8 but at a different solvent ratio, and the resulting tempera-
tures are those for the optimal three-stage serial process. The optimal
temperature and solvent policies as functions of o are shown in Figs.
10.11 and 10.12. For sufficiently inexpensive separation the tempera-
tures go to u1., indicating low conversion and large recycle, while for suf-
ficiently costly separation the optimal policy is nonrecycle. Multiple
solutions are shown by dashed lines. The discontinuous nature of the
extraction process with separation cost has obvious economic implications
if capital investment costs have not yet been taken into account in deriv-
ing the cost factors for the process.
A mixed continuous staged process with the structure shown in Figs.
10.5 and 10.6 is obtained by replacing the three stirred-tank reactors
with a plug-flow tubular reactor of residence time 0 = 6. The material-
balance equations for the continuous serial process are then
z1 = -kloe-E,'I"(x,)2 0<t<6 (25a)
x2 =
ki0e_E,11u(x1)2
- kgpe_E''1ux2
0<t<6 (25b)
0.43
0.41
0.39
0.37
0.35
0
a
0.33
0.75 345
0.50 344
0.25 343
0 342
I l I I I 1 I I
0 1 3 4 5 6 7
Iteration number
345
340
Fig. 10.11 Optimal temperature se-
E
quence as a function of cost of extrac- ,!
tion. From M. M. Denn and R. Aris,
Ind. Eng. Chem. Fundamentals, 4:248
(1965). 'Copyright 1965 by the American 335
Chemical Society. Reprinted by permis- 0.10 0.15 0.20 0.25 0.30
sion of the copyright owner.) Cost of extraction c
346 OPTIMIZATION BY VARIATIONAL METHODS
1.1
1.0
0.9
0.8
0.7
0:6
0N
-0.5
0
0.4
c
00 .0.3
E Fig. 10.12 Optimal solvent ratio as a
a 0.2 function of cost of extraction. [From
0.1 M. M. Denn and R. Aris, Ind. Eng.
Chem. Fundamentals, 4:248 (1965).
0
Copyright 1965 by the American Chemi-
0.15 0.20 0.25 0.30 0.35 cal Society. Reprinted by permission of
Cost of extraction o the copyright owner.]
Residence time t
Fig. 10.13 Successive approximations to the optimal temperature
profile in a continuous reactor using steep descent. (From M. M.
Denn and R. Aris, Ind. Eng. Chem. Fundamentals, 4:248 (1965).
Copyright 1965 by the American Chemical Society. Reprinted by
permission of the copyright owner.)
cal convergence sequence is shown in rigs. 10.13 and 10.14 for o = 0.25.
The solid starting-temperature profile is the optimal nonrecycle solution
found in Sec. 9.3, while the shape of the dashed starring curve was dic-
tated by the fact that the unconstrained solution for the temperature
profile can be shown from the necessary conditions to require an infinite
355
Z 350
v
c 345
CL
E
340
335
The observation that the efficiency of some separation and reaction sys-
tems can be enhanced by requiring the system to operate in the unsteady
state, as demonstrated, for example, in Sec. 6.11, has led to substantial
interest in the properties of periodic processes. These are processes in
which a regular cyclic behavior is established and therefore, in terms of
time-aver6ged behavior, allows the overall operation to 'be considered
from a production point of view in steady-state terms. As we shall see,
the fact that a decision made during one cycle has an effect on an
NONSERIAL PROCESSES 349
S= fo 5(x,u) dt (3)
x,+1(0) = 0 (5)
In that case
&[x(o)J = x,+1(e) (6)
1 as: aj
'Yj i=1,2, ...,5 (8a)
(8b)
1 05 aft
e axi axi i = 1, 2, ... 'S
1
j-1
'y (0) _ 'yi(B) i = 1, 2, .,S (12)
'
S
T au + i-1I aui
0 (17)
= ax; + jsl
S
afax;
i=1,2,...,5
x; = o (18)
The hamiltonian is
1
H=- B
x2 - yl(uxl" + au'x1 + x1 - 1) + 7'2(uxln - x2) (24)
l
e
X2(0)
1 f
e-0 o
dr (27b)
71(0) =
f o exp [f' (nuxln-1 + au' + 1) dv] u(T)xin(r) dr
e (29b)
1 - exp [f (nuxln-1 + au' + 1) da]
The adjustment in u(t) is then computed from Eq. (14) using all/au in
Eq. (25).
The numerical values used are as follows:
n =2 r =0.75
a=1 a =0.1
u*=1 u*=5
The optimal steady state, computed from Eq. (15) of Sec. 6.11, is
u = 2.5198, with corresponding values
x1 = 0.27144
x2(= -&) = 0.18567
71 = X3.1319
The starting value of u(t) for the steep-descent calculation was taken as
u(t) = 2.5198 + 0.40 sin 20irt (30)
The weighting factor w(t) for steep descent was based on the normalized
form and taken as
w(t) = G(t) _
(31)
dtj34
[f0 au}
with G(t) equal to 0.1 or the maximum required to reach a bound and
halved in case of no improvement in 8.
NONSERIAL PROCESSES 353
0 3
V
Q
v
0
0
0.18 5'
Fig.10.17 Time-averaged conversion on
successive iterations.
0 1
I
2 3 4 55 6
I 4
7 8 9 10 11
Iterations number
354 OPTIMIZATION BY VARIATIONAL METHODS
0 280
0270
0 260
0 001 0.02 0 03 0.04 0.05 006 0 07 0.08 0.09 0.10
I
Fig. 10.18 Dynamic response of x, on successive iterations to peri-
odic forcing.
and 10.19, respectively. Note that following the first iteration the entire
time response of x2(t) lies above the optimal steady-state value. The
course of the iterations is clearly influenced by the sinusoidal starting
function and appears to be approaching a bang-bang policy, though this
was not obtained for these calculations. Horn and Lin report that the
optimum forcing function for this problem is in fact one of irJiinitely rapid
switching between bounds or, equivalently, one for which the length of
the period goes to zero. The steep-descent calculation evidently cannot
suggest this result for a fixed value of 0.
In realistic applications the decision function will normally not
influence the system response directly, as the temperature does in this
0 195
X20)0.190
0
t
0
0.01 0.02 003 004 005 0.06 007 008 0.09 0...
10.9 DECOMPOSITION
The computational procedure which we have developed for the solution
of variational problems in complex structures has depended upon adjust-
ing the boundary conditions of the Green's function in order to incorpo-
rate the mixing boundary conditions. There is an alternative approach
that has sometimes been suggested which we can briefly illustrate by
means of the examples used in this chapter.
We consider first the reactor problem from Sec. 10.3. The problem
is formulated in terms of two variables
it = -k(u)F(xl) (1a)
x2 = -k(u)F(x1) + g(u) (lb)
If we knew the value of the input x1(0), this would specify the output
x1(8). For any fixed input, say x*, we can find the function u(1) which
minimizes E subject to Eqs. (1) and (2b) and
x1(0) = xl (4a)
x 1 (0) = P x1 - 1 P xf ( 4b )
P
Furthermore, Eqs. (6) and (7) can be solved for ull in terms of x13* and
x111# Call this value uuI*(x13*,xlll*). After carrying out these two
operations we then seek the proper values of x13 and xlll by the operation
min [-x23*(X18*,xlIi*) - Cxi11* + vu1I*(xl3*,xlli*)] (11)
x1*a.z,.n
BIBLIOGRAPHICAL NOTES
Section 10.4: The reformulation to permit use of the straight-chain process result is
pointed out in
F. J. M. Horn and R. C. Lin: Ind. Eng. Chem. Process Design Develop., 6:21 (1967)
Section 10.5: The lagrangian approach was taken by Jackson to obtain results applicable
to the more general problem of the following section in
R. Jackson: Chem. Eng. Sci., 19:19, 253 (1964)
Sections 10.6 and 10.7: The general development and the example follow
M. M. 1)enn and R. Aris: Ind. Eng. Chem. Fundamentals, 4:248 (1965)
Further examples of the optimization of nonserial processes may be found in
L. T. Fan and C. S. Wang: "The Discrete Maximum Principle," John Wiley & Sons,
Inc., New York, 1964
-- and associates: "The Continuous Maximum Principle," John Wiley & Sons,
Inc., New York, 1966
J. D. Paynter and S. G. Bankoff: Can. J. Chem. Eng., 44:340 (1966); 45:226 (1967)
A particularly interesting use of the Green's functions to investigate the effects on reaction
systems of local mixing and global mixing by recycle and bypass is in
F. J. M. Horn and M. J. Tsai: J. Optimization Theory Appl., 1:131 (1967)
R. Jackson: J. Optimization Theory Appl., 2:240 (1968)
Section 10.8: This section is based on the paper by Horn and Lin cited above. The paper
was presented at a symposium on periodic processes at the 151st meeting of the
American Chemical Society in Pittsburgh, March, 1966, and many of the other
papers were also published in the same issue of Industrial and Engineering Chemis-
try Process Design and Development. Another symposium was held at the 60th
31$ OPTIMIZATION BY VARIATIONAL METHODS
PROBLEMS
10.1. The autocatalytic reaction X, + X, = 2X2 in a tubular reactor is described
by the equations
C_Et\
.t,i - kio exp C-EIJ xIxt - kto exp x,2
uJ u
x, + xt - coast
The reaction is to be carried out in a reactor of length L with product recycle such that
x,(0) = (1 - p)xi + px,(L)
x2(0) - (1 - p)xi + px,(L)
Develop an equation for the maximum conversion to x*(L) when
E,
=2 xj = 1 - E Zq = E
E,
(The problem is due to Fan and associates.)
10.2. A chemical reaction system in a tubular reactor is described by the differential
equations
z, = f: (z)
x;(0) - xio
Obtain the equivalent result for recycle of a stream from t, to ti. (The problem is
du: to Jackson and Horn and Tsai.)
11
21.1 INTRODUCTION
A great many physical processes must be described by functional equa
tions of a more general nature than the ordinary differential and differ-
ence equations we have considered thus far. The two final examples in
Chap. 3 are illustrative of such situations. The procedures we have
developed in Chaps. 6, 7, and 9 for obtaining necessary conditions and
computational algorithms, however, based upon the construction of
Green's function for the linear variational equations, may be readily
extended to these more general processes. We shall illustrate the essen-
tially straightforward nature of the extension in this chapter, where we
focus attention on distributed-parameter systems describable by partial
differential equation.; in two independent variables.
3.12, for the results which we shall obtain will be applicable not only to a
more realistic version of that problem but to problems in chemical
reaction analysis as well. The state of the system is describedt by the
(generally nonlinear) diffusion-type partial differential equation
ax ax a2x 0<z<1
at = I x,
az' az2
t z,tl
0<t<e (1)
r dt = h(v,u) (5)
The former case (lagless control) is included by taking the time constant r
to be zero and h as u - v. It is assumed that the purpose of control
is to minimize some function of the state at time B, together with the
possibility of a cost-of-control term; i.e., we seek an admissible function
u(t), 0 < t < 0, in order to minimize
s[u] = f01 E[x(0,z);z] dz + fo C[u(t)] dt (6)
t Note that the state at any time t" is represented by an entire function of z, z(t',z),
0 < z < 1.
DISTRIBUTED-PARAMETER SYSTEMS 361
Integrating with >;espect tot from 0 to B and using Eqs. (2) to (4), we
then obtain
foI y(B,z) bx(9,z) dz = fo fo' [ye + 'Yf= - (yf=.)=
+ (yJ=,.)=_] ax dz dt + fo ax[-(f., - dt 1', - 1
t In accordance with common practice for partial differential equations and in order
to avoid some cumbersome notation we shall use subscripts to denote partial dif-
ferentiation in Sees. 11.3 to 11.7. Thus,
ax
X.
ax of
x` at - as f=" - a0=x/M)
and so forth. Since we are dealing here with a single state variable and a single
decision variable, and hence have no other need for subscripts, there should be no
confusion.
362 OPTIMIZATION BY VARIATIONAL METHODS
Finally, we note that the variation in the first term of the objective
[Eq. (6), Sec. 11.2] as a.result of the infinitesimal change Sv(t) is
fo E2[x(O,z);z} Ex(9,z) dz
Su(t) = 0
0 < t < t,
t1+A<t<tl (1)
Su(t) = finite t, < t < t, + 0
where A is infinitesimal. Thus, during the interval 0 < t < t1 the value
of v(t) is unchanged, and Sv(t1) = 0.
During the interval t1 < t < t1 + A it follows from Eq. (5) of
Sec. 11.2 that Sv must satisfy the equation
T(bv), = h(v + Sv, u + du) - h(f,u) (2)
or
(e
> C(u) Je
y(s,0)f._(s,0)g.(s) exp T had) dE] ds (9)
T
i
everywhere except possibly at a set of discrete points. If the final time 9
is not specified, we obtain a further necessary condition by differentiating
Eq. (6) of Sec. 11.2 with respect to 0 and equating the result to zero to
obtain
0 unspecified: foI y(B,z)f(B,z) dz + C[u(9)] = 0 (11)
-
xt _ x :`
0<z<1 (1)
0<t<0
X. = 0 at z = 1 for all t (2)
xz = p(x - v) at z = 0 for all t (3)
x=0 att=0forallz (4)
rv,u - V 0 <t<0 (5)
fol
min E = [x-(z) - x(B,z)]2 dz + fo C[u(t)] dt (6)
From Eqs. (6) and (15) of Sees. 11.2 ajid 11.3, respectively, we have,
for any small change bu, the first-order expression for the corresponding
change in Is,
Thus, by substituting for by in Eq. (1) and changing the order of inte-
gration we obtain the expression for the variation in the objective
SE=fo {C
hu f ° y(s,O)f...(s,O)g.(s) exp [T f ' h,(E) dt] I bu(t) dt (4)
au(t) = -w(t) I C.
- hfT
9
'(s,0)f=..(s,0)g.(s) exp [
f. d>:] ds} (5)
where w(t) > 0 reflects the geometry of the space and is chosen suf-
ficiently small to ensure that a constraint is not violated. For r = Of
with u equal to v, we obtain
bu(t) = -w(t)[C - y(t,0)f (t,0)g.(t)J (6)
The necessary conditions for an optimum are given in Sec. 11.5 by Eqs.
(17) and (18), while the direction of steep descent is determined by
e
ou(t) _ -w(t) p f y(8,0)e(t )!T ds (13)
1.0
0.8
00.6
CL
0.4
0
U
0.2
2 3
2 4 6 8
Iteration number of the copyright owner.]
DISTRIBUTED-PARAMETER SYSTEMS 36!
1.0
0
a
60.4
c
0
U
0.2
0.04 0.08 0.12 0.16 0.20 0.24 0.28 0.32 0.36 0.40
Time t
10-
to-
10-
Fig. 11.5 Reduction in the objective on
successive iterations, 0 = 0.4. [From
M. M. Denn, Intern. J. Control, 4:167
10- (1966). Copyright 1966 by Taylor and
4 6 8 10 12
Francis, Ltd. Reprinted by permission
Iteration number of the copyright owner.)
ti
m 0.8
0.6
0
w 0.4
E
0.2
0.04 0.08 0.12 0.16 0.20 0.24 0.28 0.32 0.36 0.40
Time f
Fig. 11.7 Unsmoothed intermediate optimal control policies for three
starting functions, 0 - 0.4. [From M. M. Denn, Intern. J. Control,
4:167 (1966). Copyright 1966 by Taylor and Francis, Ltd. Reprinted
by permission of the copyright owner.]
Here A is the ratio of length to radius, Pe and Pe' the radial Peclet num-
bers for mass and heat transfer, respectively, Dal and Datii the first and.
third Damkohler numbers, and r(x,y) the dimensionless reaction rate.
372 OPTIMIZATION BY VARIATIONAL METHODS
ay
az = - u(t) at z = I for all t (3f)
while the boundary conditions corresponding to Eqs. (11), (12), and (14)
are
yl = -2z at t = 1 for all z (8a)
72 = 0 at t = 1 for all z (8b)
y,=y20 atz=0foralf t (8c)
z2
y, azl = y2 - =0 at z = l for all t (8d)
0.16
0.14
0
U
0 10
0
Fig. 11.9 Improvement in outlet con-
version on successive iterations. [From
0.08
M. M. Denn, R. D. Gray, Jr., and
J. R. Ferron, Ind. Eng. Chem. Funda-
mentals, 5:59 (1966). Copyright 1966
by the American Chemical Society. Re- 0.06
1 1
0 1 2
owner.] Iteration number
15
0 0.5 I.0
ed by permission of the copyright owner.] Axial distance t
376 OPTIMIZATION BY VARIATIONAL METHODS
error associated with solving Eqs. (1) and (2) followed by Eqs. (6) and (7)
was sufficiently large to give an incorrect sign to the small quantity
y2(t,1) in the region of the optimum. Such numerical difficulties must
be expected in the neighborhood of,the optimum in the. solution of varia-
tional problems for nonlinear distributed systems, and precise results
will not be obtainable.
From an engineering point of view the results obtained thus far
represent a goal to be sought by a practical heat-exchanger design. The
coolant temperature required to produce the calculated beat-removal
rate u(t) can be determined by the relation
ay(t_1)
aZ
_ -u(t) = n[yJt) - y(t,l)l (14)
calculated from Eq. (14), is plotted as the dashed line in Fig. 11.12,
together with the results from the steep-descent calculations. The cor-
respondence with the results from the rigorous optimization procedure
is striking, indicating for this case the true optimality of the conventional
heat-exchanger design, and, in fact, as a consequence of the numerical
difficulties in the more complicated steep-descent procedure, the con-
version for the best constant coolant is slightly above that obtained from
the steep-descent computation.
This example is a useful demonstration of several of the practical
engineering aspects of the use of optimization theory. An optimal design
based on an incomplete physical model, such as neglect of important
radial effects, might give extremely poor results in practice. A careful
optimization study, however, can be used to justify a practical engineer-
ing design by providing a theoretical guideline for required performance.
Finally, in complicated systems the application of the theory is limited
by the sophistication and accuracy of the numerical analysis, and all
results in such'situations must be taken as approximate.
ay+VayaHCx-Ay
at az ax
y(O,z) = y(t,1) = 0
min 0
H de
u(t)
DISTRIBUTED-PARAMETER SYSTEMS 379
It follows from the results for the lumped system (and will be established
independently below) that M is symmetric in its spatial arguments
M(t,z,) = M(t,E,z) (13)
I
+ v(/ a+ a3f J+ 2AM
B2 [ fo' M(z,o) do] [ fo' df] + C(t)S(z - E) = 0 (17)
OD
+ V aZ + AD - B2 [fo' M(z,v) do]
[Joy D(Z) dE]
+ VM(z,0) = 0 (18)
Any solution of Eq. (17) clearly satisfies the symmetry condition, Eq.
(13). In the special case V = 0 Eq. (17) reduces with two integrations
to the usual Riccati equation for lumped systems, Eq. (10) of Sec. 8.2.
For the industrially important case of 0 - oo Eqs. (17) and (18), as in
the lumped-parameter analogs, tend to solutions which are independent
of time. We shall consider regulation only of the exit stream, so that
C(E) = C6(1 - E), and we let 0 - oo. The optimal control is computed
from Eqs. (10) and (11) as
u(t) = fo' GFB(z)x(t,z) dz + GFF d(t) (19)
where the feedback (FB) and feedforward (FF) gains are written in
terms of the solutions of Eqs. (17) and (18) as
GFB(z) = -B f0' M(z,E) dE (20)
GFF = -B fO' D(t) dt (21)
Because of the term 5(1 - ) the steady-state solution to Eq. (17)
is discontinuous at the values r = 1, t = 1. By using the method of
characteristics a solution valid everywhere except at these points can be
obtained in the implicit form
S(Z (23)
- z) = {1 >z
Integration of Eq. (22) then leads to an etuation for the feedback gain
CB e(2AIV)(1-0
GFB(z)
V
+ B j' GFB(,j)e-(2A1V)(t-0 do rL' GFB(t) dt (24)
Equation (24) is in a form which is amenable to iterative solution for the
optimal gain. Equation (18) in the steady state is simply a linear first-
order equation which can be solved explicitly for the feedforward gain
GFF _ B Jo' dz e-(AI')( t) d Jfl e-(2A1v)E-,)GF 1(,?)G1B(n - t) do
V - B f ' dz 1',;e-(AIV)(:-e'GFB(t) dl;
(25)
This result can be easily generalized to include nonconstant dis-
turbances which are random with known statistical properties, in which
case the feedforward gain GFF depends upon.the statistics.of the dis-
turbance. In particular, if d(t) is uncorrelated ("white") noise, GFF is
zero and the optimal control is entirely feedback. The feedback section
of the optimal control requires the value of x at every position z, which is
clearly impossible. The practical design problem then becomes one of
locating a minimum number of sensing elements in order to obtain an
adequate approximation to the integral in Eq. (19) for a wide range of
disturbance states x(t,t).
Feed
Valve open
during r,
during r3
solute is removed from the top. This physical process leads to an inter-
esting variational problem when we seek the distribution of hot and cold
feed over a cycle of periodic operation which maximizes the separation.
Denoting normalized temperatures by T and concentrations by c,
with subscripts f and s referring to fluid and solid phases, respectively,
the mass and energy balances inside the column (0 < z < 1) are
82T, aT, aT, aT.
+ OK =o (la)
az2 + u(t) a + at
aat' + -t (T, - T,) =0 (lb)
a2c, ac, ac, ac,
u(t) 0 (lc)
'' aZ2 + a + at + X
at
ar+a(c*-c.)=0
u(t) is the fluid velocity, which is negative during TI (downflow, hot feed),
positive during Ta (upflow, cold feed), and zero during 72 and 7 4 (periods
DISTRIBUTED-PARAMETER SYSTEMS 393
of no flow). The values used for the other physical parameters are
#K 1.10 K=1.38
y=200 a=0.3
,,n - 10'
In order to simplify the model for this preliminary study two
approximations were made based on these values. First, y was taken
as approaching infinity, implying instantaneous temperature equilibra-
tion. In that case T. = T1 = T. Secondly, dispersion of heat and mass
were neglected compared to convection and ' and 71 set equal to zero.
This latter approximation changes the order of the equations and was
retained only during the development of the optimization conditions.
All calculations were carried out including the dispersion terms. The
simplified equations describing the process behavior are then
OT
(1 + RK) = -u aT (3a)
act ac, ac, (3b)
at az at
a at- = X (C!
- C*) (3c
fe
o
u(r) dr = 0 (6)
aat2
+ u a22 + Xy3 = 0 (11b)
All three functions are periodic over 8. The direction of steep descent
'for maximizing the separation (P is
1 aT ac
w(t) I y 4 1 + c/(t,0) - 10 (y1 az + 72 a J) dz in r1
r 1 aT + aCf)
2U(t) 743 + Cf(0) - f y1 y2 az J dz in ra
o az
(13)
Here w(t) > 0, and 741 and 743 are constants chosen to satisfy the restric-
tions of Eq. (10) on au.
The initial choice for the feed distribution was. taken to be sinusoidal
u(t) = -0.5 sin t (14)
First Eqs. (llb) and (lle) were combined to give a single equation in -Y2,
and the following form was assumed for application of the Galerkin
method introduced in Sec. 3.9:
72 = (2z - 1) + Clz(1 - z) cos t + C2[$(1 - z) + (1 - $)z] sin t
+ C3z(1 - z) cos 2t + C4[$(1 - z) + (1 - $)z] sin 2t (15)
where $ = +I when u > 0, j6 = 0 when u < 0. Evaluating the con-
stants in the manner described in Sec. 3.9 shows them to be
C, = - 5.54 C2 = 0.461
C3 = 0.056 C4 = -0.046
indicating rapid convergence, and, in fact, the C3 and C4 terms were
neglected for computation. y3 was then found by analytically integrating
Eq. (llc), treating the equation as a linear first-order equation with vari-
able coefficients. -y, was also estimated using the Galerkin method,
leading to the solution
y, = 0.0245[z(1 - z) cos t + [0(1 - z) + (1 - Q)z} sin t} (16)
The function Su(t) was computed from Eq. (13) for a constant
weighting function w(t) = k, and a function w(t) = k1sin t!. These are
shown normalized with respect to k in Fig. 11.14 as a dotted and solid
line, respectively. The dashed line in the figure is the function 0.5 u(t).
Both weighting functions indicate the same essential features when the
0.2
0.5
k
sa
0 0
.0.1
-0.5
-0.2
V
-1.0
a*
0 2 2 2
Fig. 11.14 Normalized correction to velocity. Solid line w(t) = klsint1; dotted line
w(t) = k; dashed line 0.5u(t) _ -0.25 sin t.
386 OPTIMIZATION BY VARIATIONAL METHODS
0.8 r-
0.6
04
0.2
u (f) 0
-0.2
-0.4
-0.6
0
I
3w
J
2
2w
Fig. 11.15Velocity functions. Dashed line u(t) = -0.5 sin t; solid line sinusoidal
weighting with k = 0.8; rectangular wave final estimate of optimum.
DISTRIBUTED-PARAMETER SYSTEMS 387
of the model did not appear to justify the additional computational effort.
The preliminary results obtained here do suggest a fruitful area of appli-
cation-of optimization theory when physical principles of the type con-
sidered here have been carried to the point of process implementation.
BIBLIOGRAPHICAL NOTES
Sections 111 to 11.7: These sections follow quite closely
M. M. Denn: Intern. J. Contr., 4:167 (1966)
ThePartieular problem of temperature control has been considered in terms of a minimum
principle for integral formulations by
A. G. Butkovskii: Proc. sd Intern. Congr. IFAC, paper' 513 (1963)
and by classical variational methods by
Y. Sakawa: IEEE Trans. Aulom. Cont., AC9:420 (1964); AC11:35 (1966)
Modal analysis, approximation by a lumped system, and a procedure for approximating
the solution curves are used by
1. McCausland: J. Electron. Contr., 14:635 (1963)
Proc. Inst. Elec. Engrs. (London), 112:543 (1965)
Sit OPTIMIZATION BY VARIATIONAL METHODS
M. M. Denn, R. D. Gray, Jr., and J. It. Ferron: Ind. Eng. Chem. Fundamentals,
6:59 (1966)
Details of the construction of the model and numerical analysis are in
R. D. Gray, Jr.: Two-dimensional Effects in Optimal Tubular Reactor Design, Ph.D.
thesis, University of Delaware, Newark, 1)el., 1965
The usefulness of an isothermal coolant and the effect of design parameters are investigated in
A. R. Hoge: Some Aspects of the Optimization of a Two-dimensional Tubular Reactor,
B.S. thesis, University of Delaware, Newark, Del., 1965
J. D. Robinson: A Parametric Study of an Optimal Two-dimensional Tubular Reactor
Design, M.Ch.E. thesis, University of Delaware, Newark, Del., I966
Section 11.10: This section follows a paper with A. K. Wagle presented at the 1989
Joint Automatic Control Conference and published in the preprints of the meeting.
More details are contained in
A. K. Wagle: Optimal Periodic Separation Processes, M.Ch.E. Thesis, University of
Delaware, Newark, Del., 1969
Separation by parametric pumping was developed by Wilhelm and coworkirs and reported
in
R. H. Wilhelm, A. W. Rice, and A. R. Bendelius: Ind. Eng. Chem. Fundamentals,
5:141 (1966)
R. W. Rolke, and N. H. Sweed: Ind. Eng. Chem. Fundamentals,
7:337 (1968)
The basic equations are developed and compared with experiment in the thesis of Rice,
from which the parameters used here were taken:
DISTRIBUTED-PARAMETER SYSTEMS 389
E. B. Lee and. L. Markus: "Foundations of Optimal Control Theory," John Wiley &
Sons, Inc., New York, 1967
P. K. C. Wang: Intern. J. Contr., 7:101 (1968)
An excellent survey of Soviet publications in this area is
A. G. Butkovsky, A. I. Egorov, and K. A. Lurie: SIAM J. Contr., 6:437 (1968)
Many pertinent recent papers can be found in the SIAM Journal on Control; IEEE
Transactions on Automatic Control; Automation and Remote Control; and Auto-
matica; in the annual preprints of the Joint Automatic Control Conference; and in the
University of Southern California conference proceedings:
R. Jackson: Proc. Inst. Chem. Engrs. AIChE Joint Meeting, 4:32 (1965)
Intern. J. Contr., 4:127, 585 (1966)
Trans. Inst. Chem. Engrs. (London), 46:T160 (1967)
K. A. Lurie: in G. Leitmann (ed.), "Topics in Optimization," Academic Press, Inc.,
New York, 1967
A minimum principle fyir systems described by integral equations is outlined in the paper
by Butkovskii li4fd above. Results for difference-differential equations, together
with further refeonces, are in
D. H. Chyung: Pr: prints 1967 Joint Autom. Contr. Conf., p. 470,
M. M. Denn and R. Aris: Ind. Eng. Chem. Fundamentals, 4:213 (1965)
M. N. Ogflztoreli: "Time-lag Control Systems," Academic Press, Inc., New York, 1966
H. R. Sebesta and L. G. Clark: Preprints 1967 Joint Autom. Contr. Conf., p. 326
Results due to. Kharatishvili can be found in the book by Pontryagin and coworkers and
in the collection above edited by Balakrishnan and Neustadt.
PROBLEMS
11.1. Obtain the minimum-principle and steep-descent direction used in See. 11.8.
11.2. Obtain necessary conditions and the direction of steep descent for the problem in
Sec. 11.10. What changes result when the objective is maximization of the separation
390 OPTIMIZATION BY VARIATIONAL METHODS
factor,
I u(t)c1(t,l) dt
f u(i)cj(t,0) dt
11.3. Optimal control of a heat exchanger to a new set point by adjustment of wall
temperature is approximated by the equation
ax ax
at + az = P(u - x)
x(0,t) = 0
Obtain the optimal feedback gain as a function of P and C for the objective
K =2±K[1 -exp(-2P-PK)]
11.4. The reversible exothermic reaction X = Y in a tubular catalytic reactor is
approximated by the equations
ax,
az
x:rkoe [(1 - yo - (yo + x01 0<z<1
ax,
at
-«ux: 0<r<1
where x, is the extent of reaction, x2 the catalyst efficiency, u the temperature, r the
reactor residence time, and yo the fraction of reaction product in the feed. Boundary
and initial conditions are
x, =0 atz =0forall t
x==1 att -Oforallz
The temperature profile u(z) is to be chosen at each time t to maximize average
conversion
fo,
tP = dt
Obtain all the relevant equations for solution. Carry out a numerical solution if
necessary. Parameters are
rko3X10' Ko-2.3X10-6 yo=0.06
E_ = 10,000 E; - 5,000 « = 4 X 10'6
(Th(s problem is due to Jackson.)
DISTRIBUTED-PARAMETER SYSTEMS
11.5. Discuss the minimum time control of the function x(z,t) to zero by adjusting
surroundings temperature u(t) in the following radiation problem:
ax a'x 0<z<1
at - aZ2 0<t<a
x=0 att=0forall z
-=0
ax
aZ
atz=Oforallt
ax
=k[u4(t) -x'] atz = 1for all t
az
(The problem is due to Uzgiris apd D'Souza.)
11.6. Obtain necessary conditions and the direction of steep descent for a system
described by the difference-differential equations
ii(t) =l,fx(t),:(t - r), u(t), u(t - r)]
x,(t) = x,o(t) -r < t < 0
min E = fo 5[x(t),u(t)] dt
12
Dynamic Programming and
Hamilton-Jacobi Theory
12.1 INTRODUCTION
The major part of this book has been concerned with the application of
classical variational methods to sequential decision making, where the
sequence was either continuous in a timelike variable or, as in Chap. 7,
over discrete stages. Simultaneously with the refinement of these meth-
ods over the past two decades an alternative approach to such problems
has been developed and studied, primarily by Bellman and his coworkers.
Known as dynamic programming, this approach has strong similarities to
variational methods and, indeed, for the types of problems we have
studied often leads to the same set of equations for ultimate solution.
In this brief introduction to dynamic programming we shall first examine
the computational aspects which differ from those previously developed
and then demonstrate the essential equivalence of the two approaches to
sequential optimization for many problems.
DYNAMIC PROGRAMMING AND HAMILTON-JACOBI THEORY 393
Since we do not know what the proper value of xN`1 is, however, we must
do this for all values of xN-' or, more realistically, for a representative
selection of values. We can then tabulate for, each XN-1 the minimizing
value of uN and the corresponding minimum value of E.
We now move back one stage and suppose that we have available
xN-2, for which we must find the decisions uN-1 and uN that minimize
S(XN). A specification of uN-1 will determine xN-1, and for any given
xN-1 we already have tabulated the optimal uN and value of C. Thus,
we need simply search over UN-1 to find the tabulatedt xN-1 that results
in the minimum value of S. That is, we search over uN-1 only, and not
simultaneously over UN-1 and uN. The dimensionality of the problem is
thus reduced, but again we must carry out this procedure and tabulate
the results for all (representative) values of xN-2, since we do not. know
which value is the optimal one.
We now repeat the process for xN-1, choosing uN-2 by means of the
table for xN-2, etc., until finally we reach x°. Since x° is known, we can
then choose u' by means of the table ford This gives the optimal
value for x', so that u2 can then be found from-Ug table for x2, etc.
In this way the optimal sequence u1, U2, . . .. , UN is cdtp d for a
given x° by means of a sequence of minimizations over a single variable.
Note that we have made no assumptions concerning differentiability of
the functions or bounds on the decisions. Thus, this algorithm can be
used when those outlined in Chap. 9 might be difficult or inapplicable.
We shall comment further on the computational efficiency following an
example.
+ xn-')
yn = (1 + 8kjoeE,"/°")(1 + 02oe-E:'I"") (1b)
XN-1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
1/N-1
0 340 340 340 340 340 340 340 340 340 340
0.056 0.112 0.168 0.224 0.281 0.337 0.393 0.449 0.505 0.561
330 334 338 340 340 340 340 340 340 340 340
0.1
0.096 0.145 0.199 0.255 0.310 0.367 0.423 0.479 0.535 0.591 0.647
330 330 334 336 338 338 340 340 340 340 340
0.2
0.192 0.240 0.291 0.344 0.399 0.454 0.509 0.565 0.621 0.677 0.733
330 330 332 334 336 338 338 338 338 340 340
0.3
0.288 0.336 0.385 0.436 0.489 0.543 0.598 0.654 0.709 0.764 0.820
330 330 330 332 334 336 336 338 338 338 338
0.4
0.384 0.432 0.481 0.530 0.582 .0.634 0.688 0.742 0.798 0.853 0.908
330 330 330 332 334 336 336 336 338 338
0.5 / 330
0.480 0.528 0.577 0.625 0.675 0.727 0.779 0.833 0.887 0.942 0.997
330 330 330 330 332 334 334 334 336 336 338
0.6
0.576 0.624 0.673 0.721 0.770 0.820 0.872 0.925 0.978 1.032 1.086
330 330 330 330 330 332 332 334 334 336 336
0.7
0.672 0.720 0.769 0.817 0.866 0.913 0.966 1.018 1.070 1.123 1.177
330 330 330 330 330 330 332 334 334 334 336
0.8
0.768 0.816 0.865 0.913 0.962 1.010 1.060 1.111 1.163 1.215 1.268
330 330 330 330 330 330 332 332 332 334 334
0.9
0.864 0.912 0.961 1.009 1.058 1.106 1.155 1.205 1.25. .309 1.361
330 330 330 330 330 330 330 330 332 3341E
1.0
0.960 1.008 1.057 1.105 1.154 1.202 1.250 1.329 1.354 1.11 1.44
assumed that increments of 0.1 in xN-' and yN`' are sufficient for purposes
of interpolation.
Next we consider the penultimate [(N - 1)stj stage. For example,
for xN-2 = 0.7, yN-2 = 0.2, we obtain:
t For this case (No = 10) we found in Chap. 2 that u' = 338, u2 - 336.5, and
l; - -0.655 when there were no restrictions on u', u2. Since the restricted optimum
cannot be better than the unrestricted, we see some small error resulting from the
interpolation.
DYNAMIC PROGRAMMING AND HAMILTON-JACOBI THEORY 3!7
0.3
334 334 336 334 " 336 334
0.555 0.648 0.713 0.776 0.840 0.905
334 334 336 334 334 334
0.4
0.649 0.735 0.799 0.863 0.926 0.992
330 332 334 334 334 334
0.5
0.743 0.826 0.887 0.951 1.014 1.078
not a major limitation on procedures derivable from the calculus, but the
latter effectively eliminates their use. Thus, if such a restriction is
physically meaningful, an algorithm such as this one is essential, while
if the restriction is used simply as a means of obtaining first estimates by
use of a coarse grid, we might expect to obtain first estimates as easily
using steep descent as outlined in Chap. 9. In reducing the N-dimen-
sional search problem to a sequence of one-dimensional searches we have
traded a single difficult problem for a large number of simpler ones, and
indeed, we have simultaneously solved not only the problem of interest
but related problems for a range of initial values. If we are truly inter-
ested in a range of initial values, this is quite useful, but if we care only
about a single initial condition, the additional information is of little use
and seemingly wasteful. Dynamic programming does not. eliminate
the ty o-point boundary-value problem which has occupied us to such a
great extent, but rather solves it by considering a complete range of final
values.
The fact that the optimum is automatically computed as a feedback
policy, depending only upon the present state and number of remaining
decisions, suggests utility in control applications. The storage prob-
lem, however, is a serious restriction, for with two state variables we have
been able to tabulate data in a two-dimensional array, but three variables
Us OPTIMIZATION BY VARIATIONAL METHODS
that is, as the rate of change of the optimum with respect to the state
at the nth stage. Then Eqs. (9) and (10) may be written
,y,n-I Yin af;" (12)
(( ax'n-1
aS
?iN o (13)
axiN
400 OPTIMIZATION BY VARIATIONAL METHODS
These are the equations for the weak minimum principle derived in
Sec. 7.4, and they relate the Green's functions to the sensitivity interpre-
tation of the Lagrange multiplier in Sec. 1.15.
Even if u" is bounded, the minimization indicated in Eq. (4) can
be carried out and the minimizing value an found as a function of z"-'.
Equation (8) is still valid, and if the optimum occurs at an interior value,
the quantity in parentheses will vanish. If, on the other hand, the
optimal u" lies at a constraint, small variations in the stage input will
not cause any change and au"/ax;"'' will be zero. Thus, Eqs. (9) and
(10), or equivalently (12) and (13), are unchanged by constraints.
The minimum in Eq. (4) at a bound is defined by
()S"+' _ aS"+' af;^ >0 u" at lower bound (15a)
au" -L ti
ax,." au" <0 u" at upper bound
or
aS^+1
ax;
f ;^ y;"f" =min u^ at bound (15b)
(2)
preceding section, is
Sn(x"-1) = min S"+I[f"(x"-Ir"u")] (3)
U.
SN+1(xN) _ & _
I CjxjN
(4)
where _y n-1 and t" are to be determined. Substituting into Eq. (3), then,
j
(yjn-1
- i
yinA,jn) xi"-I = min [G yi"bin(u")J + n+t - t n (7)
u
i
The left-hand side depends upon x"-1 and the right does not, so that the
solution must satisfy the difference equation
yj"-I
yinA ,n (8)
Equations (8), (9), and (11) are, of course, the strong. form of the
minimum principle, which we established in Sec. 7.8 as both necessary
and sufficient for the linear separable system with linear objective. We
have not required here the differentiability of b" assumed in the earlier
proof. This special situation is of interest in that the optimal policy is
completely independent of the state as a consequence of the uncoupling
of multiplier and state equations, so that Eqs. (8), (9), and (11) may be
solved once and for all for any value of x" and an optimum defined only
in terms of the number of stages remaining.
402 OPTIMIZATION BY VARIATIONAL METHODS
+
"
S[x"-' + f At + o(At), n At + At] = S(x"-1, n At)
ax fi At +
1 a(n At)
At + o(At) (5)
or
As At gets very small, the distinction between x"-' and x", u"-' and u"
gets correspondingly small and both x"-1 and x" approach x(t) and simi-
DYNAMIC PROGRAMMING AND HAMILTON-JACOBI THEORY 403
larly for u^. Again letting t = n At, dividing by At, and taking the limit
as At -. 0, we obtain the Hamilton-Jacobi-Bellman partial differential
equation for the optimum objective S(x,t)
aSfi(x,u) asl
0 = min U(t) LL. ax,
+ at J (8)
or, defining
__ as
axi
ys (17)
7ifi(a,u) = 0 (20)
Equations (18) to (20) are, of course, the strong minimum principle for
this system, and Eq. (17) establishes the Green's function y as a sensi-
tivity variable.
It is apparent from these- results and those of Sec. 12.4 that if we
so wished, we could derive the results of the preceding chapters from a
dynamic programming point of view. We have not followed this course
for several reasons. First, the variational approach which we have gen-
erally adopted is closer to the usual engineering experience of successive
estimation of solutions. Second, computational procedures are more
naturally and easily obtained within the same mathematical framework
by variational methods. Finally (and though much emphasized in the
literature, of lesser importance), the dynamic programming derivation
of the minimum principle is not internally consistent, for it must be
assumed in the derivation that the partial derivatives aS/axi are con-
tinuous in x when in fact the solution then obtained for even the ele-
mentary minimum-time problem of Sec. 5.3 has derivatives which are
discontinuous at the switching surface.
DYNAMIC PROGRAMMING AND THEORY 405
or, defining
1 1
i.+1 = x;C;ixi + u2 x,+1(0) = 0 (3)
2
we have
C[a(e)) = x,+1(e) (4)
0 = min
+1 asl; asl
u axi + at) (5)
i-1
or, from Eqs. (1) and (3),
0 = min
' as as 1 as
A;ixt + i-1
L, ax; b'u + 2 ax.+ 1 x;C,ixi
Y axi
i,7-1
1 as 2 Ls) (6)
+ 2 ax__ u + at
The function S(x,t) satisfying Eqs. (8) and (9) will be of the form
in which case
k = 1, 2,
C'xk 2 \i-1 x,Mik + ,-1 Mt;x;) ,s (11)
2 xix1(MikAkj + AkiMkf)
i,,.k-I
2
+ x;x; (Mik + Mki)bk] 12 b1(Mii + M11)]
+.i. -I L
The optimal feedback control is then found directly from Eqs. (7) and
(10) as
,
u = - I b;M,,x, (15)
i.,-1
Since this result was obtained in the form of a solution to the Hamilton-
Jacobi-Bellman equation, we know that it is sufficient for a minimum,
a result also established by other means in Sec. 6.20.
DYNAMIC PROGRAMMING AND THEORY 407
BIBLIOGRAPHICAL NOTES
We shall not attempt a survey of the extensive periodical literature on dynamic program-
ming but content ourselves with citing several texts. Excellent introductions may be
found in
R. Aris: "Discrete Dynamic Programming," Blaisdell Publishing Ccmpany, Wal-
tham, Mass., 1964
R. E. Bellman and S. E. Dreyfus: "Applied Dynamic Programming," Princeton Uni-
versity Press, Princeton, N.J., 1962
G. Hadley: "Nonlinear and Dynamic Programming," Addison-Wesley Publishing
Company, Inc., Reading, Mass., 1964
G. L. Nemhauser: "Introduction to Dynamic Programming," John Wiley & Sons,
Inc., New York, 1966
Problems of the type considered in Sec. 12.6 are treated in
S. E. Dreyfus: "Dynamic Programming and the Calculus of Variations," Academic
Press, Inc., New York, 1965
and numerous topics of fundamental interest are treated in the original book on the subject:
R. Bellman: "Dynamic Programming," Princeton University Press, Princeton, N.J.,
1957
Extensive applications are examined in
R. Aris: "The Optimal Design of Chemical Reactors: A Study in Dynamic Program-
ming," Academic Press, Inc., New York, 1961
R. Bellman: "Adaptive Control Processes: A Guided Tour," Princeton University
Press, Princeton, N.J., 1961
S. M. Roberts: "Dynamic Programming in Chemical Engineering and Process Con-
trol," Academic Press, Inc., New York, 1964
D. F. Rudd and C. C. Watson: "Strategy of Process Engineering," John Wiley &
Sons Inc., New York, 1968
J. Tou: "Optimum Design of Digital Control via Dynamic Programming," Academic
Press, Inc., New York, 1963
PROBLEMS
12.1. Develop the Hamilton-Jacobi-Bellman equation for the system
z" = f^(z"-' u")
N
min s = E 61"(x"-',u") + F(1N)
n-1
Apply this directly to the linear one-dimensional case
x" = A(u^)x"-' + 14(A(u") - 11
N
t = a(xN - x°) + E M(u")
n-1
and establish that the optimal decision u" is identical at each stage.
OPTIMIZATION BY VARIATIONAL METHODS
12.2. Formulate Prob. 7.2 for direct solution by dynamic programming. Draw a
complete logical flow diagram and if a computer is available, solve and compare
the effort to previous methods used. Suppose holding times u," are restricted to
certain discrete values?
12.3. The system
x=u lul <1
is to be taken from initial conditions x(O) = 1, i(0) = 1 to the origin to minimize
E - 5(x,i,u) dt
0
Show how the optimum can be computed by direct application of the dynamic pro-
gramming approach for discrete systems. (Hint: Write
ij - x: x1(t - A) = xi(t) - x3(t)A
is = u x:(t - A) = X2(t) - u(t)A
and obtain the Hamilton-Jacobi-Bellman difference equation for recursive calculation.)
Assume that u can take on only nine evenly spaced values and obtain the solution for
(a)5
(b) if = %(xs= + u=)
(c) if - 3Vx' + V)
Compare the value of the objective with the exact solution for continuously variable
u(t) obtained with the minimum principle.
1L4. Using the dynamic programming approach, derive the minimum principle for
the distributed system in Sec. 11.4.
Indexes
Name Index
Saaty, T. L., 69
Nemhauser, G. L., 358, 407 Sakawa, Y., 76, 97, 387
Neustadt, L. W., 224, 322, 389 Sarrus, F., 69
Newman, A. K., 268 Scharmack, D. K., 322
Nieman, R. A., 324 Schechter, R. S., 96, 357
Noton, A. R. M., 132, 267, 322, 325 Schley, C. H., 323
OPTIMIZATION BY VARIATIONAL METHODS
414
Calculus of variations:
isoperimetric problem, 84, 297, 309 Data fitting, 41
"simplest" problem, 73, 108 Decomposition, 355
(See also Euler equation) Definite function, 9, 21, 43, 259
Canonical equations, 106, 136, 184, Design, 1, 2, 24, 128, 130, 165, 235,
232, 378 300, 304, 371, 376, 386
Catalyst, 173, 224, 390 Difference-differential equation, 389
-Catalytic reformer, 202 Difference equation, 36, 64, 228, 236,
Chatter, 256 262, 398
Chemical reaction, 27, 40, 41, 98, Differential calculus, 4, 34
111, 199, 202, 329, 337, 351, 371 Differential equation, 176, 221
415 '
416 OPTIMIZATION BY VARIATIONAL METHODS
This book was set in Modern by The Maple Press Company, and printed on
-permanent paper and bound by The Maple Press Company. The designer was
J. E. O'Connor; the drawings were done by J. & R. Technical Services, Inc.
The editors were B. J. Clark and Maureen McMahon. William P. Weiss
supervised the production.