Optimization Techniques

Multi-variable Unconstrained Optimization:

Unidirectional Search

Dr. Nasir M Mirza

Email: nasirmm@yahoo.com

In this Lecture
In this lecture we will discuss two important methods
that deal with Multi-variable Unconstrained
Optimization:
Multi Dimensional Plots and
Uni-directional Search

Example 4.2
The following function of two variables has distinct local minimum points and
inflection point:

f = (x - 2)2 + (x - 22)2
Let us draw a mesh grid
graph in three dimensions
using MATLAB.
% matlab function for 3D plot
clear all
[X,Y] = meshgrid(-2:.1:2);
Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ;
mesh(X, Y, Z) %interpolated
axis tight; hold on
plot3(x,y,z,'.','MarkerSize',15)

SURFACE
PLOT using
MATLAB

% matlab function for 3D plot

clear all
[X,Y] = meshgrid(-2:.1:2);
Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ;
surf(X,Y,Z,'FaceColor','red','EdgeColor','none')
camlight left; lighting phong

Example 4.2 continued

Object function is: f = (x - 2)2+ (x - 22)2
f
= 2( x 2) + 2( x 2 y 2 ) = 4 + 4 x 4 y 2
x
f
= 8 y ( x 2 y 2 ) = 8 xy + 16 y 3
y

gradient vector =
f

2 f
2
x
H= 2
f
yx

4
4
x
4
y
=

8 xy + 16 y

2 f

8y

xy 4

=
2
2

f 8 y 8 x + 48 y
y 2

CONTOUR
PLOT using
MATLAB
Next step is to
draw a contour
graph and identify
points of interest.
(Here we see (2, 1)T and (2, 1)T as
important points.
% matlab function for 3D plot
clear all
[X,Y] = meshgrid(-2:.05:5);
Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ;
contour(X,Y,Z, 3000)

SURFACE
PLOT using
MATLAB

% matlab function for 3D plot

clear all
[X,Y] = meshgrid(-2:.05:5);
Z = ( X - 2.0).^2 + (X - 2.*Y.*Y).^2 ;
surf(X,Y,Z,'FaceColor','red','EdgeColor','none')
camlight left; lighting phong
hold on
contour(X,Y,Z, 1000)

Example 4.2 continued

Two types of contour plots showing possible maximum or minimum points

Example 4.2
Optimality condition (first order):
f
= 2( x 2) + 2( x 2 y 2 ) = 4 + 4 x 4 y 2 0
x
f
= 8 y ( x 2 y 2 ) = 8 xy + 16 y 3 0
y

Possible solutions (x, y) are following:

(1, 0) ; (2, -1); (2, 1)
Let us check with second optimality condition for each point:
For point (1, 0):

4 0

H =
0 8

Principal minors are:

A1 = a11= 4, A2 = det(A)= -32
It is then inflection point and Function value = 2.

Example 4.2
Let us check with second optimality condition for point (2, -1)

4 8

H =
8 32

Principal minors are:

A1 = a11= 4, A2 = det(A)= 128 64 = 64
It is then minimum point and Function value = 0.

Let us check with second optimality condition for point (2, +1)

4 8

H =
8 32

Principal minors are:

A1 = a11= 4, A2 = det(A)= 128 64 = 64
It is then minimum point and Function value = 0.

Example 4.3

The following function of two variables has only one minimum point
but two distinct local maximum points and two inflection points:
f(x) = 25x2 12x4 6xy + 25y2 24x2y2 12y4.

Let us first draw 3D graph:

% matlab function for 3D plot
clear all
[X,Y] = meshgrid(-1.5:.05:1.5);
Z = 25.*X.^2-12.*X.^4-6.*X.*Y
+25.*Y.^2 -24.*(X.^2).*(Y.^2) 12.*Y.^4.
surf(X,Y,Z,'FaceColor','red','Edge
Color','none')
camlight left; lighting phong
hold on
contour(X,Y,Z, 1000)

Example 4.3

% matlab function for this graph

clear all
[X,Y] = meshgrid(-1.5:.1:1.5);
Z = 25.*X.^2-12.*X.^4-6.*X.*Y +25.*Y.^2 -24.*(X.^2).*(Y.^2) - 12.*Y.^4.
surf(X, Y, Z);
colormap(jet);

Example 4.3

% matlab function for 3D plot

clear all
[X,Y] = meshgrid(1.5:.01:1.5);
Z = 25.*X.^2-12.*X.^4-6.*X.*Y
+25.*Y.^2 -24.*(X.^2).*(Y.^2) 12.*Y.^4.
contour(X, Y, Z, 200);

Example 4.3

Object function is: f(x) = 25x2 12x4 6xy + 25y2 24x2y2 12y4.

f
= 50 x 48 x 3 6 y 48 xy 2
x
f
= 6 x + 50 y 48 x 2 y 48 y 3
y

2 f
2
x
H= 2
f
yx

2 f
2
2

xy 50 144 x 48 y
=
2
f
6 96 xy
y 2

2
2
50 48 x 144 y
6 96 xy

Example 4.3
Optimality condition (first order):
f
= 50 x 48 x 3 6 y 48 xy 2 = 0
x
f
= 6 x + 50 y 48 x 2 y 48 y 3 = 0
y

Possible solutions (x, y) are following:

(-0.764, 0.764) ; (-0.677, -0.677); (0, 0); (0.764, -0.764) ; (0.677, 0.677);
Let us check with second optimality condition for each point:
For point (-0.764, 0.764):

62 50

H =
50 62

Principal minors are:

A1 = a11= -62, A2 = det(A)= 1344
It is then maximum point and
Function value = 16.33

Example 4.3
Let us check with second optimality condition for point (-0.677, -0.677)

38 50

H =
50 38

Principal minors are:

A1 = a11= -38, A2 = det(A)= -1056
It is then inflection point and Function value = 10.08

Let us check with second optimality condition for point (0, 0)

50 6

H =
6 50

Principal minors are:

A1 = a11= 50, A2 = det(A)= 2464
It is then minimum point and Function value = 0.

Example 4.3
Let us check with second optimality condition for point (+0.677, +0.677)

38 50

H =
50 38

Principal minors are:

A1 = a11= -38, A2 = det(A)= -1056
It is then inflection point and Function value = 10.08

Let us check with second optimality condition for point (0.764, -0.764)

62 50

H =
50 62

Principal minors are:

A1 = a11= -62, A2 = det(A)= 1364
It is then maximum point and
Function value = 16.3

Example 4.3
The graph shows
these points on
contour plot as
maximum,
minimum and
inflection points;

Maximum

Inflection

Minimum

Inflection

Maximum

Unidirectional Search Method

A unidirectional search is a one-dimensional search
performed by comparing function values only along a
specific direction.
Usually a unidirectional search is performed from a
point x(t) in a specific direction s(t).
Only points that lie on a line passing through x(t) in
direction of s(t) are allowed.
Any arbitrary point on that line can be expressed as
x(a) = x(t) + as(t).
The parameter a is a scalar quantity and it is measure
of distance of point x(a) from x(t).

Unidirectional Search Method

The above equation is a vector equation in multidimensional space.
Hence when a is known the x(a) can be estimated.
Any positive or negative value of a will create x(a) on
the line for search.
When a is zero, the current point x(t) is obtained.
When we want minimum point, we convert objective
function in terms of single variable a by putting all xi as
x(a) = x(t) + as(t).
Once optimal a is found the corresponding point is found.

Example 3.2.1
Consider the objective function:
Minimize: f(x, y) = (x 10)2 + (y 10)2

Step 1: First draw the contour plot of this function. The contour
plot has lines where any two points on the line have the same
function value.

The optimal point is a minimum at (10, 10)T. The function value

is zero at this point.

Step 2: Let us say that a current point selected is x(t) = (2, 1)T.
and the function value in a search direction s(t) = (2, 5)T from
the current point.
x(a) = x(t) + as(t) ; vector equation

x(a) = 2 + a2 ; y(a) = 1 + a5; scalar equations

Example 3.2.1
Now we find the equation for the straight line passing
through (2, 1)T and direction (2, 5)T as:
(x 2)/2 = (y 1)/5 ;
y 1 = (5/2)(x 2).

Now let us find a point on this line as x = 3 then y = 3.5

and we can estimate a from above equations as a = 0.5
If the point on line is with x = 9 and y = 18.5,
then a = 3.5
The function value for (3, 3.5)T is 91.25
And at point (9, 18.5)T the value is 73.25.
Then at a = (3.5 + 0.5)/2 = 2, then x = 6, y = 11 and
function value is 17. So, approx. opt. is (6, 11)T.

Example 3.3.1
Consider the Himmelblau function:
Minimize:
f(x, y) = (x2 + y 11)2 + (x + y2
7)2
in the interval 0 < x < 5;

Step 1: This function is plotted as

Figure. It appears from the plot that

a minimum point is at (3, 2)T.

% matlab function for 3D plot

clear all
[X,Y] = meshgrid(0:.05:5);
Z = ( X.*X + Y - 11.0).^2 + (X + Y.*Y - 7.0).^2 ;
surf(X, Y, Z);
colormap(jet);

Example 3.3.1: contour lines

A contour line is a collection of
points having identical function
values.
The contour plot for the object
function is shown here.
The continual decrease in the
function value of the successive
contour lines as they approach a
point means that the point is
minimum point.
% matlab function for contour plot
clear all
axis([0 5 0 5])
hold on
[X,Y] = meshgrid(-2:.05:5);
Z = ( X.*X + Y - 11.0).^2 + (X + Y.*Y - 7.0).^2 ;
contour(X, Y, Z, 200)

Example 3.3.1
Step 1: We now choose an initial
point x(0) = (1, 1)T
Size reduction parameter = (2, 2)T
Also choose e = 0.001
Step 2: We create a two
dimensional hypercube (a square)
around x(0) such that
X(1) = (0, 0)T , x(2) = (2, 0)T ,
X(3) = (0, 2)T , x(4) = (2, 2)T
Step 3: The function values at these
points are:
f(0) = 106 ; f(1) = 170 ; f(2) = 74
f(3) = 90 ; f(4) = 26
The minimum is at (2, 2)T. Let us say this as x* =(2, 2)T

Example 3.3.1
Step 4: Since x* is not x(0), we
now set x(0) = (2, 2)T and got to
step 2.
Step 2: We create a new square
around this x(0) with = 1 such
that
X(1) = (1, 1)T , x(2) = (3, 1)T ,
X(3) = (1, 3)T , x(4) = (3, 3)T
Step 3: The function values at
these points are:
f(0) = 26 ; f(1) = 106 ; f(2) = 10
f(3) = 58 ; f(4) = 26

The minimum is at (3, 1)T. The function

value is 10.
Let us say this as x* =(3, 1)T

Example 3.3.1
Step 4: Since x* is not x(0), we
now set x(0) = (3, 1)T and go to
step 2.
Step 2: We create a new square
around this x(0) with = 1 such
that
X(1) = (2, 0)T , x(2) = (4, 0)T ,
X(3) = (2, 2)T , x(4) = (4, 2)T
Step 3: The function values at
these points are:
f(0) = 10 ; f(1) = 74 ; f(2) = 34
f(3) = 26 ; f(4) = 50

The minimum is at (3, 1)T where function

value is 10. It is same as x(0).
Let us say this as x* =(3, 1)T

Example 3.3.1
Step 4: Since x* is same as x(0),
we now set x(0) = (3, 1)T and go
to step 2 after reducing step size =
0.5
Step 2: We create a new square
around this x(0)=(3, 1)T with = 0.5
such that
X(1) = (2.5, 0.5)T , x(2) = (3.5, 0.5)T
,
X(3) = (2.5, 1.5)T , x(4) = (3.5, 1.5)T
Step 3: The function values at
these points are computed and
minimum is at x(4) = (3.5, 1.5)T

The minimum is at (3.5, 1.5)T

And function value is 9.125.

Let us say this as x* =(3.5, 1.5)T

Example 3.3.1
Step 4: Since x* is not same as
x(0), we now set x(0) = (3.5, 1.5)T
and go to step 2 with step size =
0.5
Step 2: We create a new square
around this x(0)=(3, 1)T with = 0.5
such that
X(1) = (3, 1)T , x(2) = (4, 1)T ,
X(3) = (3, 2)T , x(4) = (4, 2)T
Step 3: The function values at
these points are
f(0) =9.125; f(1) = 10 ;

The minimum is at (3, 2)T

f(2) = 40; f(3) = 0; f(4) = 50

This is the best minimum so far as f=0.

the minimum is f(3) = 0.0

Example 3.3.1
It is important to see that although
we have found a minimum point but
the algorithm does not terminate at
this step. We may continue.
It is clear that the convergence
depends upon the initial cube size
and location and the chosen size
reduction parameter.
Small size may lead to premature
convergence and large may never
lead to convergence.

The minimum is at (3, 2)T

This is the best minimum so far as f=0.

Example: Suppose f(x, y) = 2xy + 2x x2 2y2

Using the steepest ascent method to find the next point if we are
moving from point (-1, 1).

f
f
= 2 y + 2 2x
= 2x 4 y
x
y
f

1
,
1
)
x
2(1) + 2 2(1) 6
At (1,1), f = f
=
=

(1,1) 2(1) 4(1) 6

y

Let g (h) = f (-1 + 6h,1 6h)

Next step is to find h that maximize g(h)

f ( x) = 2 xy + 2 x x 2 2 y 2
g (h) = f (-1 + 6h,1 6h) = ... = 7 + 72h 180h 2
Setting g ' (h) = 0 yields
72 360h = 0 h = 0.2
If h = 0.2 maximizes g(h), then x = -1+6(0.2) = 0.2 and y = 16(0.2) = -0.2 would maximize f(x, y).
So moving along the direction of gradient from point (-1,
1), we would reach the optimum point (which is our next
point) at (0.2, -0.2).