ELEC 403/503 Engineering Design by Optimization Summer 2013

Course Outline

Instructor
Dr. Wu-Sheng Lu
Phone: 721-8692
E-mail: wslu@ece.uvic.ca
URL: http://www.ece.uvic.ca/~wslu

Office Hours
Days: Wednesdays
Time: 14:00 16:00
Location: EOW 427

Lectures
Days: Tuesday, Wednesday, Friday
Time: 12:30 1:20 p.m.
Location: CLE A206
Labs
Location: ELW B326
Section Days Time
B01 (30274) Wednesday 14:30 17:30
B04 (30277) Tuesday 14:30 17:30
ELEC 403 Lab starts on May 22, Wednesday.

Required Text
Title: Practical Optimization: Algorithms and Engineering Applications
http://www.ece.uvic.ca/~optimization/
Author: A. Antoniou and W.-S. Lu
Publisher: Springer
Year: 2007

Assessment
Assignments: 10%
Labs (ELEC403) 15%
Labs & Project (ELEC503) 15%
Mid-term exam 20% Date: July 9 (Tuesday)
Final exam 55%

2
Other Things to Note:
Course web site:
http://www.ece.uvic.ca/~wslu/403/index.html
You need the user name and password to access Course Material.

About assignments:
Only paper copies will be accepted.
Submit your assignment on the due day by 4:00 pm.
You may submit your work to the instructor at class; or leave
it in his mailbox in EOW 448; or slide it into his office EOW 427.
ELEC403/503 does not use drop box for submissions.
Please clearly print your name and the course number on the
cover page.
Due day for each assignment will be announced in class and
posted on the course website.

There will be no classes on June 4 (Tuesday), June 5 (Wednesday),

June 7 (Friday), and June 11 (Tuesday) due to the Congress of the
Social and Humanities.

3
ELEC 403/503 classes on May 21 (Tuesday) and May 22 (Wednesday)
are cancelled. A make-up class of 100 minutes will be scheduled
in the second week.

To ELEC 503 students:

You must also enroll ELEC 403 Labs.
Do get a copy of the Additional Course Outline for ELEC 503
from the course web site for important information about
course project, lab experiments, and due days of your lab
reports, etc.

4
 ELEC 403/503
Engineering Design by Optimization

Dept. of Electrical and Computer Engineering

University of Victoria

Course Instructor: Wu-Sheng Lu

 What Is Optimization?

People Optimize
Investors seek to create portfolios that avoid excessive risk while
achieving high rate of return;
Manufacturers aim for maximum efficiency in the design and
operation of their production processes;
Engineers adjust parameters to optimize the performance of their
designs.

Nature Optimizes
Physical systems tend to a state of minimum energy;
Molecules in an isolated chemical system react with each other
until the total potential energy of their electrons is minimized;
Rays of light follow paths that minimize their travel time.

a
 Optimization is a tool for decision science and analysis of physical
systems.

To use this tool, we must identify some objective, a quantitative

measure of the performance of the system under study. Examples of
objective: profit, time, potential energy, or any quantity that can be
represented by a single real number.

Typically the objective depends on certain characteristics of the

system, called variables or unknowns.

Our goal is to find values of the variables that optimize the

objective.

Often the variables are constrained. Example: the length of a

robotic link has to be in a certain range. As a result, optimization is
usually carried out subject to certain constraints.
b
 There are two steps to follow in an optimization task:

A step of identifying objective, variables, and constraints for a

given problem (this step is known as modeling);

Once the model has been formulated, one finds or develops an

algorithm to solve the problem, typically with the help of a
computer.

There are no optimization algorithms that are universally

applicable, rather there is a collection of algorithms, each tailored
for a particular class of problems. It is often the responsibility of the
user to identify an algorithm that is most appropriate for a specific
application.

c
1.2 The Basic Optimization Problem
Before optimization is attempted, the problem at hand must be
properly formulated. A performance criterion F must be derived
in terms of n parameters x1 , x2 , . . . , xn as

F = f (x1 , x2 , . . . , xn )

F is a scalar quantity which can assume numerous forms.

The most basic optimization problem is to adjust variables
x1 , x2 , . . . , xn in such a way as to minimize quantity F . This
problem can be stated mathematically as

minimize F = f (x1 , x2 , . . . , xn ) (1.1)

Quantity F is usually referred to as the objective or cost

function.

Chap. 1
To simplify the notation, matrix notation is usually employed. If
x is a column vector with elements x1 , x2 , . . . , xn , the transpose
of x, namely, xT , can be expressed as the row vector

xT = [x1 x2 xn ]

In this notation, the basic optimization problem of Eq. (1.1) can

be expressed as

minimize F = f (x) for x E n

where E n represents the n-dimensional Euclidean space.

Chap. 1
On many occasions, the optimization problem consists of
finding the maximum of the objective function. Since

max[f (x)] = min[f (x)]

the maximum of F can be readily obtained by finding the

minimum of the negative of F and then changing the sign of the
minimum. Consequently, in this and subsequent chapters we
focus our attention on minimization without loss of generality.

Chap. 1
1.3 General Structure of Optimization Algorithms
Most of the available optimization algorithms entail a series of
steps which are executed sequentially. A typical pattern is as
follows:
Algorithm 1.1 General optimization algorithm
Step 1
(a) Set k = 0 and initialize x0 .
(b) Compute F0 = f (x0 ).
Step 2
(a) Set k = k + 1.
(b) Compute the changes in xk given by column vector x k where

xTk = [x1 x2 xn ]

by using an appropriate procedure.

(c) Set xk = xk 1 + xk
(d ) Compute Fk = f (xk ) and Fk = Fk 1 Fk .

Chap. 1
Step 3
Check if convergence has been achieved by using an appropriate
criterion, e.g., by checking Fk and/or xk . If this is the case,
continue to Step 4; otherwise, go to Step 2.
Step 4
(a) Output x = xk and F = f (x ).
(b) Stop.

The column vector x is said to be the optimum, minimum,

solution point , or simply the minimizer , and F is said to be the
optimum or minimum value of the objective function. The pair
x and F constitute the solution of the optimization problem.

Chap. 1
Convergence can be checked in several ways, depending on
the optimization problem and the optimization technique used.
For example, one might decide to stop the algorithm when the
reduction in Fk between any two iterations has become
insignificant, that is,

|Fk | = |Fk 1 Fk | < F (1.2)

where F is an optimization tolerance for the objective function.

Alternatively, one might decide to stop the algorithm when the
changes in all variables have become insignificant, that is,

|xi | < x for i = 1, 2, . . . , n (1.3)

where x is an optimization tolerance for variables

x1 , x2 , . . . , xn . A third possibility might be to check if both
criteria given by Eqs. (1.2) and (1.3) are satisfied
simultaneously.

Chap. 1
1.4 Constraints
In many optimization problems, the variables are interrelated by
physical laws like the conservation of mass or energy,
Kirchhoffs voltage and current laws, and other system
equalities that must be satisfied. In effect, in these problems
certain equality constraints of the form

ai (x) = 0 for x E n

where i = 1, 2, . . . , p must be satisfied before the problem can

be considered solved.

Chap. 1
In other optimization problems a collection of inequality
constraints might be imposed on the variables or parameters to
ensure physical realizability, reliability, compatibility, or even to
simplify the modeling of the problem.
In these problems, a collection of inequality constraints of the
form
cj (x) 0 for x E n
where j = 1, 2, . . . , q must be satisfied before the optimization
problem can be considered solved.

Chap. 1
An optimization problem may entail a set of equality constraints
and possibly a set of inequality constraints. If this is the case,
the problem is said to be a constrained optimization problem.
The most general constrained optimization problem can be
expressed mathematically as

minimize f (x) for x E n (1.4a)

subject to: ai (x) = 0 for i = 1, 2, . . . , p (1.4b)
cj (x) 0 for j = 1, 2, . . . , q (1.4c)

A problem that does not entail any equality or inequality

constraints is said to be an unconstrained optimization problem.

Chap. 1
Example 1.2 Consider a control system that comprises a
double inverted pendulum as depicted in Fig. 1.3. The objective
of the system is to maintain the pendulum in the upright position
using the minimum amount of energy. This is achieved by
applying an appropriate control force to the car to damp out any
displacements 1 (t ) and 2 (t ). Formulate the problem as an
optimization problem.

u(t)

Figure: The double inverted pendulum.

Chap. 1
Solution The dynamic equations of the system are nonlinear
and the standard practice is to apply a linearization technique to
these equations to obtain a small-signal linear model of the
system as [6]
x(t
) = Ax(t ) + fu(t ) (1.5)
where
1 (t ) 0 1 0 0 0

1 (t ) 0 0 1
x(t ) =
2 (t ) , A=
0
, f=
0 0 1 0

2 (t ) 0 0 0

Chap. 1
with > 0, > 0, and 6= . In the above equations, x(t ),
1 (t ), and 2 (t ) represent the first derivatives of x(t ), 1 (t ), and
2 (t ), respectively, with respect to time, 1 (t ) and 2 (t ) would be
the second derivatives of 1 (t ) and 2 (t ), and parameters and
depend on system parameters such as the length and weight
of each pendulum, the mass of the car, etc. Suppose that at
instant t = 0 small nonzero displacements 1 (t ) and 2 (t )
occur, which would call for immediate control action in order to
steer the system back to the equilibrium state x(t ) = 0 at time
t = T0 . In order to develop a digital controller, the system model
in (1.5) is discretized to become

x(k + 1) = x(k ) + gu(k ) (1.6)

where = I + t A, g = t f, t is the sampling interval, and I

is the identity matrix.

Chap. 1
Let x(0) 6= 0 be given and assume that T 0 is a multiple of t ,
i.e., T0 = K t where K is an integer. We seek to find a
sequence of control actions u(k ) for k = 0, 1, . . . , K 1 such
that the zero equilibrium state is achieved at t = T 0 , i.e.,
x(T0 ) = 0.
Let us assume that the energy consumed by these control
actions, namely,
K 1
u 2 (k )
X
J=
k =0

needs to be minimized.

Chap. 1
This optimal control problem can be formulated analytically as
K 1
u 2 (k )
X
minimize J = (1.7a)
k =0
subject to: x(K ) = 0 (1.7b)

From Eq. (1.6), we know that the state of the system at

t = K t is determined by the initial value of the state and
system model in Eq. (1.6) as
K 1
K
K k 1 gu(k )
X
x(K ) = x(0) +
k =0
K
X 1
h + gk u(k )
k =0

where h = K x(0) and gk = K k 1 g.

Chap. 1
Hence constraint (1.7b) is equivalent to
K
X 1
gk u(k ) = h (1.8)
k =0

If we define u = [u(0) u(1) u(K 1)]T and

G = [g0 g1 gK 1 ], then the constraint in Eq. (1.8) can be
expressed as Gu = h, and the optimal control problem at hand
can be formulated as the problem of finding a u that solves the
minimization problem

minimize uT u (1.9a)

subject to: a(u) = 0 (1.9b)

where a(u) = Gu h.

Chap. 1
In practice, the control actions cannot be made arbitrarily large
in magnitude. Consequently, additional constraints are often
imposed on |u(i )|, for instance,

|u(i )| m for i = 0, 1, . . . , K 1

These constraints are equivalent to

m + u(i ) 0
m u(i ) 0

Hence if we define
m + u(0)

m u(0)

..

c(u) = .


m + u(K 1)

m u(K 1)

Chap. 1
then the magnitude constraints can be expressed as

c(u) 0 (1.9c)

Obviously, the problem in Eq. (1.9) fits nicely into the standard
form of optimization problems given by Eq. (1.4).

1.5 The Feasible Region

Any point x that satisfies both the equality as well as the
inequality constraints is said to be a feasible point of the
optimization problem. The set of all points that satisfy the
constraints constitutes the feasible domain region of f (x).
Evidently, the constraints define a subset of E n . Therefore, the
feasible region can be defined as a set R given by

R = {x : ai (x) = 0, i = 1, 2, . . . , p and cj (x) 0, j = 1, 2, . . . , q}

where R E n .
Chap. 1
One More Example: The Portfolio Optimization Problem

Harry Markowitz received the 1990 Nobel Prize in

Economics for his work in the 1950s that offered the first
systematic treatment of a dilemma that every investor faces:
the conflicting objectives of high profit versus low risk.

In dealing with this fundamental issue, Markowitz came up

with a parameteric optimization model that was both
sufficiently general for a significant range of practical
situations and simple enough for theoretical analysis and
numerical solution.

22a
 What is a portfolio?
Todays investors can choose from a wide range of possible
securities bills and notes, bonds, individual stocks from
exchanges around the world, mutual funds, and so on each
has its own potential risks and rewards. A portfolio is a list of
investments held by a person.

Some Observations

The returns on some securities are unspectacular but steady,

while the returns on others are higher over the long term but
are subject to wild fluctuations.

22b
 The returns on different securities are often correlated,
positively or negatively. For example, the stock prices of
gold mining companies and of gold itself tend to increase or
decrease more or less synchronously, i.e., they are positively
correlated. On the other hand, returns on oil investments may
be negatively correlated with returns on the stocks of fuel-
cell manufacturers.

To see how a portfolio can reduce risk, consider the case of

two securities A and B, for which the fluctuations on return
are negatively correlated. Assume both investments tend to
be profitable over time, but return on A usually increases
whenever the return on B decreases and vice versa. Its easy
to perceive that the returns on a portfolio that mixes A and B
will tend to fluctuate less than the returns on an investment in
A or B alone.
22c
A Quantitative Analysis

Suppose the portfolio contains n securities. Let xi be a

random variable that represents the return on the ith security
during the next month.

We assume that each xi is normally distributed and the

values of the expected return i = E[ xi ] and the variance
i2 = E[( xi i ) 2 ] are available. We also assume that values
of the correlations between each pair of securities, defined by

E[( xi i )( x j j )]
i , j = i, j = 1,, n
i j

22d
 We try to construct a portfolio by allocating a fraction wi of
the available resources to security i, for each i. So we have
the simple constraints on wi as
n
0 wi 1, w =1
i =1
i

The expected return on this portfolio is

n n
E[ wi xi ] = wi i
i =1 i =1

which is the quantity we want to maximize.

The variance for the portfolio can be evaluated as follows:

22e
 n n
E[ wi xi E ( wi xi )]2
i =1 i =1
n n n n
= E{[ wi xi E ( wi xi )][ w j x j E ( w j x j )]}
i =1 i =1 j =1 j =1

= E{[ wi ( xi i )][ w j ( x j j )]}

n n

i =1 j =1
n n
= E[ wi w j ( xi i )( x j j )]
i =1 j =1
n n
= wi w j E[( xi i )( x j j )]
i =1 j =1
n n
= wi w j i j i , j
i =1 j =1
Qij

n n
= Qij wi w j
i =1 j =1
22f
Portfolio Optimization Problem 1: Minimize risk for a
given level of return:

n n
minimize
{ wi }
Q w w
i =1 j =1
ij i j

n
subject to: w
i =1
i i

wi 0 for 1 i n
n

w =1
i =1
i

22g
Portfolio Optimization Problem 2: Maximize return for a
given level of risk:

n
maximize
{ wi }
w
i =1
i i

n n
subject to: Q w w
i =1 j =1
ij i j 2

wi 0 for 1 i n
n

w =1
i =1
i

22h
Portfolio Optimization Problem 3: Keep the expected
return large and the variance small:

n n n
minimize Qij wi w j i wi
{ wi }
i =1 j =1 i =1

subject to: wi 0 for 1 i n

n

w =1
i =1
i

where constant > 0 is a risk tolerance parameter that

quantifies the relative weighting of the two aims.

22i
Example Consider a portfolio optimization problem with
only two stocks on the investors list. The statistical data for
these stocks on a per $100 scale are as follows:

Stock # 1: 1 = 5, 12 = 9
Stock # 2: 2 = 12, 22 = 196

with a negative correlation between them: 12 = 21 = 0.5

The matrix Q in this case is given by

12 1 2 12 9 21
Q= =

1 2 21 2
2
21 196

22j
and the portfolio optimization problem that minimizes the
risk while having an acceptable minimum expected return
is explicitly given by

2 2
minimize Q w w
i =1 j =1
ij i j = 9 w 42 w1 w2 + 196 w
2
1
2
2

subject to: 1w1 + 2 w2 = 5w1 + 12 w2

w1 + w2 = 1
w1 0, w2 0

22k
Solution Below we consider two cases.
Case 1: The minimum expected return is set to = 7.
The feasible region in this case is a portion of the segment of
the straight line w1 + w2 = 1 in the first quadrant of the w1w2
plane with endpoints P1 and P2, where P2 = [0 1]T and P1 is
determined by the equations

w1 + w2 = 1
5w1 + 12 w2 = = 7
whose solution is given by w1 = 5/7 = 0.7143 and w2 = 2/7 =
0.2857, i.e.,
P1 = [0.7143 0.2857]T

22l
The solution in this case can readily be identified as point P1
because as soon as the candidate solution point starts to leave
P1 towards P2 (see Figure A below), it suggests a buying
policy that invests more money on the risky stock # 2
therefore increases the objective function.

22m
 Feasible Region
2

1.5

P2
1
w2

w1 + w2 = 1

0.5
P
1

5*w1 + 12*w2 = 7

0.5
0.5 0 0.5 1 1.5 2
w1

Figure A: Feasible region for Case 1

22n
An alternative and more quantitative way to explain why P1
is the solution point is to evaluate polynomial
f ( w1 , w2 ) = 9 w12 42 w1w2 + 196 w22 along the line from P2 to
P1.To do this we substitute w2 = 1 w1 into f(w1, w2) which
yields

f ( w1 ) = 9 w12 42 w1 (1 w1 ) + 196(1 w1 ) 2 = 247 w12 434 w1 + 196

where w1 varies from 0 to 0.7143. The plot of f ( w1 ) over

interval [0, 0.7143] is depicted in Figure B below. It is
observed that the minimum value of the function is achieved
at the end point w1 = 0.7143 which together with w2 = 1 w1
= 0.2857 corresponds to point P1.

22o
 Polynomial 247*w12 434*w1 + 196 on [0 0.7143]
250

200

150

100

50

0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
w1

Figure B: Polynomial f(w1, w2) over line P2P1

22p
The minimized risk at the solution point P1 is found to be

2 2

Q w w
i =1 j =1
ij i j = 12.0204

indicating a fairly moderate risk increase while having the

expected return increased from 5 (buying Stock # 1only) to 7
(using 71.43% of the money to buy Stock # 1 and remaining
28.57% to purchase Stock # 2).

Case 2: The minimum expected return is set to = 9.

Similar to the fires case, the feasible region here is also a

portion of the line w1 + w2 = 1 in the first quadrant with the

22q
same endpoint P2 = [0 1]T, but endpoint P1 is now
determined by the equations

w1 + w2 = 1
5w1 + 12 w2 = = 9

whose solution is given by w1 = 3/7 = 0.4286 and w2 = 4/7 =

0.5714, i.e.,

P1 = [0.4286 0.5714]T

With an argument similar to that given in Case 1, we

conclude that point P1 represents the solution of the
minimization problem. Namely, the minimum expected
return is increased almost twice from 5 to 9 if the investor
22r
uses 42.86% of the money to but Stock # 1 and the rest
57.14% to purchase Stock #2. In doing so, however, he takes
a considerably higher risk:
2 2

Q w w
i =1 j =1
ij i j = 55.3673

22s
1.5 The Feasible Region

Any point x that satisfies both the equality as well the

inequality constraints is said to be a feasible point of the
optimization problem. The set of all points that satisfy the
constraints constitute the feasible region of f(x). Evidently,
the constraints define a subset of En. Therefore, the feasible
region can be defined as set R given by

R = { x : ai ( x ) = 0, i = 1,2,, p and c j ( x ) 0, j = 1,2,, q}

where R E n .

22t
The optimum point x must be located in the feasible region,
and so the general constrained optimization problem can be
stated as
minimize f (x) for x R
Any point x not in R is said to be a nonfeasible point .
If the constraints in an optimization problem are all inequalities,
the constraints divide the points in the E n space into three
types of points, as follows:
1. Interior points
2. Boundary points
3. Exterior points
Aninterior point is a point for which c j (x) > 0 for all j . A
boundary point is a point for which at least one c j (x) = 0, and
an exterior point is a point for which at least one c j (x) < 0.
Interior points are feasible points, boundary points may or may
not be feasible points, whereas exterior points are nonfeasible
points.
Chap. 1
Example 1.5 By using a graphical method, solve the following
optimization problem

minimize f (x) = x12 + x22 4x1 + 4

subject to: c1 (x) = x1 2x2 + 6 0
c2 (x) = x12 + x2 1 0
c3 (x) = x1 0
c4 (x) = x2 0

Chap. 1
Solution The objective function can be expressed as

(x1 2)2 + x22 = f (x)

Hence the contours pof f (x) in the (x1 , x2 ) plane are concentric
circles with radius f (x) centered at x1 = 2, x2 = 0.
Constraints c1 (x) and c2 (x) dictate that

x2 12 x1 + 3

and
x2 x12 + 1
respectively, while constraints c 3 (x) and c4 (x) dictate that x1
and x2 be positive. The contours of f (x) and the boundaries of
the constraints can be constructed as shown in Fig. 1.5.

Chap. 1
 5
c 2 (x) = - x12 + x 2 -1
x2 c 3 (x)
4
c 1 (x) = x1 -2 x 2 +6

3
Feasible 9
region
2
4

A
1 1

c 4 (x)
B

-2 -1 0 1 2 3 4 5
x1

-1

-2

Figure: Graphical construction for Example 1.5.

Chap. 1
The feasible region for this problem is the shaded region in
Fig. 1.5. The solution is located at point A on the boundary of
constraint c2 (x). In effect, the solution is a constrained optimum
point. Consequently, if this problem is solved by means of
mathematical programming, constraint c 2 (x) will be active when
the solution is reached.
In the absence of constraints, the minimization of f (x) would
yield point B as the solution.

Chap. 1
From Fig. 1.5, we see that the minimum point is point A at
which the parabola intersects with one of the contour circles
with one intersection point.

One way to determine the coordinates of point A is to identify

the radius r of the circle that tangents with the parabola. This
can be done by requiring the two real-valued solutions (that
correspond to the two intersection points) of

x12 + x2 1 = 0 (a)

(
1x 2) 2
+ x2
2
= r 2
(b)

to be identical (in such a case, in the figure the number of

intersection points appears to be one, but actually there are two
(but identical) intersection points.)
27a
From Eqs. (a) and (b), we obtain

x14 + 3 x12 4 x1 + (5 r 2 ) = 0 (c)

Again from Fig. 1.5, we see that r = 1and r = 2 correspond to

two contour circles with no intersection and 2 intersection
points, respectively.
Hence the right value of r lies on the interval [1, 2]. A bisection
search on [1, 2] can be carried out to find this r. To this end, the
MATLAB command roots can be used to find the roots of the
equation in (c) because no intersection and two intersection
points simply mean no real roots and two real roots for Eq.
(c), respectively. The bisection process is illustrated in the
figure below.
27b
 1 1.5 1.75 2

0 r.r. 2 r.r.
0 r.r.

0 r.r.

The search for r was implemented using MATLAB (see the code
below). It was found that r* = 1.9491 gives two equal real roots.
With this value of r, Eq. (c) yields x1 = 0.5536 and Eq. (a) then
gives x2 = 1.3064. Therefore, the minimizer of the constrained
optimization problem is given by

x = [ 0.5536 1.3064]
T

27c
% ex1_5.m
% This function uses a bisection method to identify radius r
% with which the contour circle tangents to the parabola,
% see the analysis in Example 1.5 of the class notes.
% Using this r, the function also computes the minimizer of
% the constrained optimization problem.
% Input: epsi -- tolerance for the uncertainty range of radius r.
% Outputs: x -- the minimizer
% r -- value of the radius
% k -- number of iterations used.

function [x,r,k] = ex1_5(epsi)

rl = 1;
ru = 2;
d = ru - rl;
k = 0;
while d > epsi,
27d
 r = 0.5*(rl+ru);
rts = roots([1 0 3 -4 5-r^2]);
im1 = imag(rts(1));
im2 = imag(rts(3));
if im1*im2 ~= 0,
rl = r;
else
ru = r;
end
d=d/2;
k = k+1;
end

disp('the value of r is:')

r = (ru+rl)/2
disp('the roots with this r are:')
rts = roots([1 0 3 -4 5-r^2])
im1 = imag(rts(1));
27e
im2 = imag(rts(3));
if abs(im1) < 1e-5,
x1 = real(rts(1));
else
x1 = real(rts(3));
end
x2 = 1+x1^2;
disp('the minimizer is:')
x = [x1 x2]'

27f