Sie sind auf Seite 1von 41

MBE2036 Engineering Computing, part 8, version 2

Multidimensional Unconstrained Optimization

Part 8

Jia Pan

Office: AC-1 Y6720 Email: jiapan@cityu.edu.hk

Department of Mechanical and Biomedical Engineering

City University of Hong Kong

MBE2036 Engineering Computing, part 8, version 2

Analytical Method

For one dimensional unconstraint optimization problem, the optimal point is the point where the curve is flat

In mathematical terms, the first derivative f’(x)=0

If second derivative f”(x)<0, the point is a maximum

If second derivative f”(x)>0, the point is a minimum

MBE2036 Engineering Computing, part 8, version 2

Analytical Method (cont’)

For two dimensional problems, the optimal point for a function f(x,y)is:

First partial derivatives should be zero i.e.

is: – First partial derivatives should be zero i.e. and – If |H| >0 and then
is: – First partial derivatives should be zero i.e. and – If |H| >0 and then

and

If |H| >0 and

derivatives should be zero i.e. and – If |H| >0 and then f(x,y) has a local

then f(x,y) has a local minimum.

If |H| >0 and then f(x,y) has a local maximum.

– If |H| >0 and then f(x,y) has a local maximum. – If |H| <0, then

If |H| <0, then f(x,y) has a saddle point.

MBE2036 Engineering Computing, part 8, version 2

Analytical Method (cont’)

Hessian of f(x,y): H =
Hessian of f(x,y):
H =

Eq 1

The determinant of Hessian:

Hessian of f(x,y): H = Eq 1 The determinant of Hessian: = Eq2 The analytical approach

=

Hessian of f(x,y): H = Eq 1 The determinant of Hessian: = Eq2 The analytical approach

Eq2

The analytical approach for finding optimal points for two dimensional functions can be extended to multi-dimensional functions.

MBE2036 Engineering Computing, part 8, version 2

Example 1

Find the optimal point of f(x,y)=x 2 y 2

2 Example 1 • Find the optimal point of f(x,y)=x 2 – y 2 Similarly, 0

2 Example 1 • Find the optimal point of f(x,y)=x 2 – y 2 Similarly, 0

Similarly,

2 Example 1 • Find the optimal point of f(x,y)=x 2 – y 2 Similarly, 0
2 Example 1 • Find the optimal point of f(x,y)=x 2 – y 2 Similarly, 0
2 Example 1 • Find the optimal point of f(x,y)=x 2 – y 2 Similarly, 0
2 Example 1 • Find the optimal point of f(x,y)=x 2 – y 2 Similarly, 0

0 as x and y are independent variables

MBE2036 Engineering Computing, part 8, version 2

f(x,y)=x 2 – y 2 0.3 0.2 0.1 0 -0.1 -0.2 -0.3 -0.4 0.5 0
f(x,y)=x 2 – y 2
0.3
0.2
0.1
0
-0.1
-0.2
-0.3
-0.4
0.5
0
0
y
-0.5
-0.5
z

x

0.5

MBE2036 Engineering Computing, part 8, version 2

MBE2036 Engineering Computing, part 8, version 2 Example 1(cont’) = 2 From Eq 2: = Therefore

Example 1(cont’)

Example 1(cont’)
Example 1(cont’)
Example 1(cont’)

= 2

Computing, part 8, version 2 Example 1(cont’) = 2 From Eq 2: = Therefore , (0,0)
Computing, part 8, version 2 Example 1(cont’) = 2 From Eq 2: = Therefore , (0,0)
Computing, part 8, version 2 Example 1(cont’) = 2 From Eq 2: = Therefore , (0,0)
Computing, part 8, version 2 Example 1(cont’) = 2 From Eq 2: = Therefore , (0,0)
Computing, part 8, version 2 Example 1(cont’) = 2 From Eq 2: = Therefore , (0,0)

From Eq 2:

part 8, version 2 Example 1(cont’) = 2 From Eq 2: = Therefore , (0,0) is

=

part 8, version 2 Example 1(cont’) = 2 From Eq 2: = Therefore , (0,0) is
part 8, version 2 Example 1(cont’) = 2 From Eq 2: = Therefore , (0,0) is
part 8, version 2 Example 1(cont’) = 2 From Eq 2: = Therefore , (0,0) is

Therefore , (0,0) is a saddle point which is neither a maximum or minimum

MBE2036 Engineering Computing, part 8, version 2

Example 2

Find the optimal point of f(x,y)=2xy +2x – x 2 –2y 2

2xy
2xy
2 Example 2 Find the optimal point of f(x,y)=2xy +2x – x 2 –2y 2 2xy

x 2

2 Example 2 Find the optimal point of f(x,y)=2xy +2x – x 2 –2y 2 2xy

2y 2

= 2xy x 2 2y 2 = y Eq 4
=
2xy
x 2
2y 2
=
y
Eq 4
2 Example 2 Find the optimal point of f(x,y)=2xy +2x – x 2 –2y 2 2xy

MBE2036 Engineering Computing, part 8, version 2

f(x,y)=2xy +2x – x 2 –2y 2 10 0 -10 -20 -30 -40 -50 -60
f(x,y)=2xy +2x – x 2 –2y 2
10
0
-10
-20
-30
-40
-50
-60
-2
-1
0
1
-1
0
2
1
2
3
3
Y
4 4
X
Z

-2

MBE2036 Engineering Computing, part 8, version 2

f(x,y)=2xy +2x – x 2 –2y 2 100 0 -100 -2 -1 -2 0 -1
f(x,y)=2xy +2x – x 2 –2y 2
100
0
-100
-2
-1
-2
0
-1
0
1
1
2
2
3
3
Y
X
Z

4

4

0

-10

-20

-30

-40

-50

MBE2036 Engineering Computing, part 8, version 2

Example 2 (cont’)

To find the optimal point:

8, version 2 Example 2 (cont’) To find the optimal point: 4y =0 Eq 6 Solving
8, version 2 Example 2 (cont’) To find the optimal point: 4y =0 Eq 6 Solving

4y =0

Eq 6

Solving Eq 5 and 6, we obtain the optimal point x=2, y =1

MBE2036 Engineering Computing, part 8, version 2

Example 2 (cont’)

MBE2036 Engineering Computing, part 8, version 2 Example 2 (cont’) = - 2 4y =0 4
MBE2036 Engineering Computing, part 8, version 2 Example 2 (cont’) = - 2 4y =0 4
MBE2036 Engineering Computing, part 8, version 2 Example 2 (cont’) = - 2 4y =0 4

= - 2MBE2036 Engineering Computing, part 8, version 2 Example 2 (cont’) 4y =0 4 y 4y 2

MBE2036 Engineering Computing, part 8, version 2 Example 2 (cont’) = - 2 4y =0 4
MBE2036 Engineering Computing, part 8, version 2 Example 2 (cont’) = - 2 4y =0 4

4y =0

MBE2036 Engineering Computing, part 8, version 2 Example 2 (cont’) = - 2 4y =0 4

4y

MBE2036 Engineering Computing, part 8, version 2 Example 2 (cont’) = - 2 4y =0 4
MBE2036 Engineering Computing, part 8, version 2 Example 2 (cont’) = - 2 4y =0 4

4y

MBE2036 Engineering Computing, part 8, version 2 Example 2 (cont’) = - 2 4y =0 4

2

MBE2036 Engineering Computing, part 8, version 2 Example 2 (cont’) = - 2 4y =0 4
MBE2036 Engineering Computing, part 8, version 2 Example 2 (cont’) = - 2 4y =0 4

MBE2036 Engineering Computing, part 8, version 2

Example 2 (cont’)

Engineering Computing, part 8, version 2 Example 2 (cont’) = As |H| > 0 and maximum.

=

Computing, part 8, version 2 Example 2 (cont’) = As |H| > 0 and maximum. ,
Computing, part 8, version 2 Example 2 (cont’) = As |H| > 0 and maximum. ,
Computing, part 8, version 2 Example 2 (cont’) = As |H| > 0 and maximum. ,

As |H| > 0 and maximum.

version 2 Example 2 (cont’) = As |H| > 0 and maximum. , function value f(2,1)

, function value f(2,1) is a

The above example is pretty straight forward to find the optimal point using first partial derivatives and Hessian. However, this is not always that straight forward. So, numerical method would be used instead!

MBE2036 Engineering Computing, part 8, version 2

Finite Difference Approximation

MBE2036 Engineering Computing, part 8, version 2 Finite Difference Approximation

MBE2036 Engineering Computing, part 8, version 2

Gradient Methods

In any multi-dimensional optimization searching algorithm, there are two key questions to be asked when searching for an optimal solution.

1. The first one is about the direction of travel.

2. The second question is how long one should pursue a solution in this direction before re-evaluating an alternative direction.

question is how long one should pursue a solution in this direction before re-evaluating an alternative
question is how long one should pursue a solution in this direction before re-evaluating an alternative

MBE2036 Engineering Computing, part 8, version 2

Gradient Methods (cont’)

The concept of gradients tell you what direction to go, at least in the short term

One dimensional case:

Ideal but not practical method:

One dimensional case: • Ideal but not practical method: – Start from any selected point of

Start from any selected point of (x0, y0). Evaluate gradient at (x0, y0).

Walk for a tiny distance along the gradient direction till (x1, y1).

Re-evaluate gradient at (x1, y1) and repeat the process.

MBE2036 Engineering Computing, part 8, version 2

Gradient Methods (cont’)

Pros: always keep steepest direction and walk shortest distance

Cons: not practical due to continuous re- evaluation of the gradient.

Question: How to find the best (x1, y1)

MBE2036 Engineering Computing, part 8, version 2

Gradient Methods (cont’)

The concept of gradients tells you what direction to go, at least in the short term.

For the second question, if a re-evaluation strategy is executed too often, it increases computational costs. If it is executed too infrequently, one may end up pursuing a direction that may take the search away from the optimal solution.

The steepest ascent algorithm proposes a simple solution to the second question.

MBE2036 Engineering Computing, part 8, version 2

Steepest Ascent Method (or Steepest Descent)

Start at an initial point (x o ,y o ), determine the direction of steepest ascend, that is, the gradient.

Then search along the direction of the gradient until f(x,y) stops increasing i.e. becomes level along your direction of travel.

Maximum point (x o ,y o )
Maximum point
(x o ,y o )

MBE2036 Engineering Computing, part 8, version 2

Steepest Ascent Method (cont’)

This stopping point becomes the starting point where the gradient is re- evaluated and a new direction followed.

This process is repeated until the peak is reached.

Maximum point (x o ,y o )
Maximum point
(x o ,y o )

The inner the contour line the higher the value

MBE2036 Engineering Computing, part 8, version 2

Steepest Ascent Method (cont’)

Steepest ascent method uses the gradient approach as its choice for the “best” direction.

If f(x,y) is a two dimensional function, the gradient vector tells us

What direction is the steepest ascent?

How much we will gain by taking that step?

MBE2036 Engineering Computing, part 8, version 2

Steepest Ascent Method (cont’)

The gradient is defined as:

f

x

f

y

del f

f

 

i

j

Eq7

 f  y del f f   i  j Eq 7 Directional derivative

Directional derivative of f(x,y) at point x=a and y=b

 f  y del f f   i  j Eq 7 Directional derivative

MBE2036 Engineering Computing, part 8, version 2

Steepest Ascent Method (cont’)

•For n dimensions

  f  (x)  f        f
  f
(x)
 f
 
 f


x

1

(x)

f

(

x

)

x

Eq8

2

(x)

x

n

MBE2036 Engineering Computing, part 8, version 2

Steepest Ascent Method (cont’)

To transform a function of x and y into a function of h along the gradient section:

y y

f

x

f

y

x x

o

h

Eq 9

 x  f  y  x x o  h Eq 9  o

o

h

Eq 10

x  f  y  x x o  h Eq 9  o 
x  f  y  x x o  h Eq 9  o 

h is distance along the h axis

MBE2036 Engineering Computing, part 8, version 2

Steepest Ascent Method (cont’)

MBE2036 Engineering Computing, part 8, version 2 Steepest Ascent Method (cont’) • Assume:

Assume:

MBE2036 Engineering Computing, part 8, version 2 Steepest Ascent Method (cont’) • Assume:
MBE2036 Engineering Computing, part 8, version 2 Steepest Ascent Method (cont’) • Assume:

MBE2036 Engineering Computing, part 8, version 2

Example 3

Using Steepest Ascent method for finding the maximum point of f(x,y)=2xy +2x – x 2 –2y 2

Using initial guesses, x=-1 and y=1

+2x – x 2 –2y 2 Using initial guesses, x=-1 and y=1 2xy x 2 2y

2xy
2xy
– x 2 –2y 2 Using initial guesses, x=-1 and y=1 2xy x 2 2y 2

x 2

2 –2y 2 Using initial guesses, x=-1 and y=1 2xy x 2 2y 2 = 2xy

2y 2

= 2xy x 2 2y 2 Recall: = y Eq 4 is the max point
=
2xy
x 2
2y 2
Recall:
=
y
Eq 4
is the
max point

MBE2036 Engineering Computing, part 8, version 2

Example 3 (cont’)

Using initial guesses, x=-1 and y=1

2 Example 3 (cont’) Using initial guesses, x=-1 and y=1 y= 2(−1) − 4(1)=−6 Therefore the
2 Example 3 (cont’) Using initial guesses, x=-1 and y=1 y= 2(−1) − 4(1)=−6 Therefore the
2 Example 3 (cont’) Using initial guesses, x=-1 and y=1 y= 2(−1) − 4(1)=−6 Therefore the

y= 2(−1) − 4(1)=−6 Therefore the gradient vector is:

2 Example 3 (cont’) Using initial guesses, x=-1 and y=1 y= 2(−1) − 4(1)=−6 Therefore the
2 Example 3 (cont’) Using initial guesses, x=-1 and y=1 y= 2(−1) − 4(1)=−6 Therefore the
2 Example 3 (cont’) Using initial guesses, x=-1 and y=1 y= 2(−1) − 4(1)=−6 Therefore the
2 Example 3 (cont’) Using initial guesses, x=-1 and y=1 y= 2(−1) − 4(1)=−6 Therefore the
2 Example 3 (cont’) Using initial guesses, x=-1 and y=1 y= 2(−1) − 4(1)=−6 Therefore the

MBE2036 Engineering Computing, part 8, version 2

Example 3(cont’)

To find the maximum, we could search along the gradient direction, that is, along an h axis running along the direction of the above gradient vector.

The function can be expressed along this axis as:

vector. The function can be expressed along this axis as: Using initial guesses, x=-1 and y=1

Using initial guesses, x=-1 and y=1

The function can be expressed along this axis as: Using initial guesses, x=-1 and y=1 f(x,y)=2xy
The function can be expressed along this axis as: Using initial guesses, x=-1 and y=1 f(x,y)=2xy
The function can be expressed along this axis as: Using initial guesses, x=-1 and y=1 f(x,y)=2xy

f(x,y)=2xy +2x – x 2 –2y 2

MBE2036 Engineering Computing, part 8, version 2

Example 3(cont’)

Multiplying the terms, we develop a one-dimensional function along the gradient as:

develop a one-dimensional function along the gradient as: For finding the maximum point along the h

For finding the maximum point along the h axis:

Therefore
Therefore

This means that If we travel along the h 0 -axis, g(h) reaches maximum value when h=0.2

MBE2036 Engineering Computing, part 8, version 2

Example 3(cont’)

Substitute h=0.2 into Eq 9 and Eq 10

2 Example 3(cont’) Substitute h=0.2 into Eq 9 and Eq 10 The search moves from point
2 Example 3(cont’) Substitute h=0.2 into Eq 9 and Eq 10 The search moves from point
2 Example 3(cont’) Substitute h=0.2 into Eq 9 and Eq 10 The search moves from point

The search moves from point 0 to point 1

The second iteration repeat the previous steps with the new starting point of (0.2, -0.2)

MBE2036 Engineering Computing, part 8, version 2

Example 3(cont’)

MBE2036 Engineering Computing, part 8, version 2 Example 3(cont’)

MBE2036 Engineering Computing, part 8, version 2

Example 3(cont’)

MBE2036 Engineering Computing, part 8, version 2 Example 3(cont’) y= 2(0.2) − 4(−0.2)=1.2
MBE2036 Engineering Computing, part 8, version 2 Example 3(cont’) y= 2(0.2) − 4(−0.2)=1.2
MBE2036 Engineering Computing, part 8, version 2 Example 3(cont’) y= 2(0.2) − 4(−0.2)=1.2

y= 2(0.2) − 4(−0.2)=1.2MBE2036 Engineering Computing, part 8, version 2 Example 3(cont’)

MBE2036 Engineering Computing, part 8, version 2 Example 3(cont’) y= 2(0.2) − 4(−0.2)=1.2
MBE2036 Engineering Computing, part 8, version 2 Example 3(cont’) y= 2(0.2) − 4(−0.2)=1.2
MBE2036 Engineering Computing, part 8, version 2 Example 3(cont’) y= 2(0.2) − 4(−0.2)=1.2

MBE2036 Engineering Computing, part 8, version 2

Example 3(cont’)

f(x,y)=2xy +2x – x 2 –2y 2 Substituting x2, y2 values into the functions yields:

2 Substituting x2, y2 values into the functions yields: For finding the maximum point along the

For finding the maximum point along the h 1 axis:

Therefore
Therefore

This means that If we travel along the h 1 -axis, g(h) reaches maximum value when h=1

MBE2036 Engineering Computing, part 8, version 2

Example 3(cont’)

Engineering Computing, part 8, version 2 Example 3(cont’) = 0.2+1.2(1)=1.4 The search moves from point 1
Engineering Computing, part 8, version 2 Example 3(cont’) = 0.2+1.2(1)=1.4 The search moves from point 1

= 0.2+1.2(1)=1.4Engineering Computing, part 8, version 2 Example 3(cont’) The search moves from point 1 to point

part 8, version 2 Example 3(cont’) = 0.2+1.2(1)=1.4 The search moves from point 1 to point
part 8, version 2 Example 3(cont’) = 0.2+1.2(1)=1.4 The search moves from point 1 to point

The search moves from point 1 to point 2 The third iteration repeat the previous steps with the new starting point of (1.4, 1), and in so doing move closer to the maximum. The approach can be repeated

until

starting point.

so doing move closer to the maximum. The approach can be repeated until starting point. approach

approach to zero for the any new starting point of (1.4, 1), and in so doing move closer to the maximum. The approach
approach to zero for the any new

MBE2036 Engineering Computing, part 8, version 2

Example 3(cont’)

MBE2036 Engineering Computing, part 8, version 2 Example 3(cont’)

MBE2036 Engineering Computing, part 8, version 2

Example 3(cont’)

MBE2036 Engineering Computing, part 8, version 2 Example 3(cont’) y= 2(1.4) − 4(1)=-1.2
MBE2036 Engineering Computing, part 8, version 2 Example 3(cont’) y= 2(1.4) − 4(1)=-1.2
MBE2036 Engineering Computing, part 8, version 2 Example 3(cont’) y= 2(1.4) − 4(1)=-1.2
MBE2036 Engineering Computing, part 8, version 2 Example 3(cont’) y= 2(1.4) − 4(1)=-1.2

y= 2(1.4) − 4(1)=-1.2

MBE2036 Engineering Computing, part 8, version 2 Example 3(cont’) y= 2(1.4) − 4(1)=-1.2

MBE2036 Engineering Computing, part 8, version 2

Example 3(cont’)

MBE2036 Engineering Computing, part 8, version 2 Example 3(cont’) What can you observe?

What can you observe?

MBE2036 Engineering Computing, part 8, version 2

Seems

MBE2036 Engineering Computing, part 8, version 2 Seems • Is this a coincidence?

Is this a coincidence?

MBE2036 Engineering Computing, part 8, version 2

Algorithm of Steepest Ascent

Step 0: Initial guess

Step 1: compute then stop

Step 2: define

Step 3: solve

Step 4:

, set k = 0 , if

then stop • Step 2: define • Step 3: solve • Step 4: , set k
then stop • Step 2: define • Step 3: solve • Step 4: , set k
then stop • Step 2: define • Step 3: solve • Step 4: , set k
then stop • Step 2: define • Step 3: solve • Step 4: , set k
then stop • Step 2: define • Step 3: solve • Step 4: , set k

,

then stop • Step 2: define • Step 3: solve • Step 4: , set k
then stop • Step 2: define • Step 3: solve • Step 4: , set k
then stop • Step 2: define • Step 3: solve • Step 4: , set k

and find , k -> k+1 go to
, k -> k+1 go toand find

then stop • Step 2: define • Step 3: solve • Step 4: , set k
then stop • Step 2: define • Step 3: solve • Step 4: , set k
then stop • Step 2: define • Step 3: solve • Step 4: , set k
then stop • Step 2: define • Step 3: solve • Step 4: , set k
then stop • Step 2: define • Step 3: solve • Step 4: , set k
then stop • Step 2: define • Step 3: solve • Step 4: , set k

step 1

MBE2036 Engineering Computing, part 8, version 2

Other Methods

Engineering Computing, part 8, version 2 Other Methods • Newton’s method • Taylor’s series • At

Newton’s method

Taylor’s series

Methods • Newton’s method • Taylor’s series • At the optimal point, , which means •

At the optimal point,

method • Taylor’s series • At the optimal point, , which means • Newton’s method in
method • Taylor’s series • At the optimal point, , which means • Newton’s method in

, which means

Taylor’s series • At the optimal point, , which means • Newton’s method in high dimensional

Newton’s method in high dimensional space:

means • Newton’s method in high dimensional space: • Does not guarantee • Steepest descent: simple
means • Newton’s method in high dimensional space: • Does not guarantee • Steepest descent: simple

Does not guarantee

Steepest descent: simple but slow; Newton’s method:

dimensional space: • Does not guarantee • Steepest descent: simple but slow; Newton’s method: complex but

complex but fast

MBE2036 Engineering Computing, part 8, version 2

Q&A

Thank you

Department of Mechanical and Biomedical Engineering

City University of Hong Kong