Jussi Hakanen, Markus Hartikainen &

Karthik Sindhya
Dept. of Mathematical Information
Technology, University of Jyvskyl,
Finland

Dept. of Mathematical Information Technology

Nonlinear Multiobjective
Optimization

Syllabus
Dept. of Mathematical Information Technology

http://users.jyu.fi/~jhaka/uppsala/syllabus.p
df

Where we come from?

January 23-27, 2012

Dagstuhl Seminar on Learning in

Multiobjective Optimization

Dept. of Mathematical Information Technology

Jyvskyl
University of Jyvskyl
Dept. of Mathematical Information
Technology
Industrial Optimization Group

Jussi Hakanen
jussi.hakanen@jyu.fi

Dept. of Mathematical Information Technology

A quick look at single objective

optimization

Contents
Dept. of Mathematical Information Technology

Introduction to single objective optimization

What is optimization?
Examples of optimization problems
Elements of an optimization problem
Optimality conditions
Solving an optimization problem
Optimization software

What is optimization?
Dept. of Mathematical Information Technology

Scientific approach to decision making

Prof. Saul I. Gass
Searching for the best solution with respect
to given constraints
Enables systematic search of the best
solution (cf. trial and error)

Examples of practical optimization

Dept. of Mathematical Information Technology

Process design and optimization

Optimal shape design
Portfolio optimization
Route optimization in logistics
Supply chain management
etc.

Optimization problem
Dept. of Mathematical Information Technology

Objective function (cost function)

= measure for the goodness of the solution
Variables (decision, design, ...)
= values change the solution
Constraints (equality, inequality)
= define feasible solutions
Feasible region = all the constraints are
satisfied
Parameters = values dont change during
optimization (cf. variables)

Mathematical formulation

Optimality: find such that

Note: solutions of the optimization
problems max () and min ()
are the same

Dept. of Mathematical Information Technology

Feasible region

Example1: mixing problem

Dept. of Mathematical Information Technology

Refinery produces 3 types of gasoline by mixing 3

different grude oil. Each grude oil can be
purchased maximum of 5000 barrels per day. Let
us assume that octane values and lead
concentrations behave linearly in mixing. Refining
costs are 4$ per barrel and the capacity of the
refinery is 14000 barrels per day. Demand of
gasoline can be increased by advertizing (demand
grows 10 barrels per day for each $ used for
advertizing).
Determine the production quantities of each
type of gasoline, mixing ratios of different
grude oil and the advertizing budget so that the
daily profit is maximized.

Mixing problem
Sale price
Lower limit for octane
Upper limit for lead
Demand
Refining costs

70
10
0.01
3000
4
Grude oil 1

Purchase price
Octane value
Lead concentration
Availability

45
12
0.005
5000

Gasoline2
60
8
0.02
2000
4
Grude oil 2

Gasoline3
50
6
0.01
1000
4

Grude oil3
35
25
6
8
0.02
0.03
5000
5000

Dept. of Mathematical Information Technology

Gasoline1

Mixing problem
Dept. of Mathematical Information Technology

Variables:
= amount of grude oil used for producing gasoline
= the amount of money used for advertizing gasoline
Net income:
11 : 70 45 4 = 21
12 : 60 45 4 = 11
13 : 50 45 4 = 1
21 : 70 35 4 = 31
22 : 60 35 4 = 21
23 : 50 35 4 = 11
31 : 70 25 4 = 41
32 : 60 25 4 = 31
33 : 50 25 4 = 21

Mixing problem

Capacity:

3
=1

3
=1

14000

Dept. of Mathematical Information Technology

Objective function:
max 21 11 + 11 12 + 13 + 3121 +
2122 + 1123 + 4131 + 3132 +
2133 1 2 3
Nonnegativity:
0 , 0

Mixing problem
Dept. of Mathematical Information Technology

Demands:
Gasoline 1: 11 + 21 + 31 = 3000 + 101
Gasoline 2: 12 + 22 + 32 = 2000 + 102
Gasoline 3: 13 + 23 + 33 = 1000 + 103
Availabilities:
Grude oil 1: 11 + 21 + 31 5000
Grude oil 2: 12 + 22 + 32 5000
Grude oil 3: 13 + 23 + 33 5000

Mixing problem
Gasoline 1:
Gasoline 2:

Gasoline 3:

1211 +
11 +
1212 +
12 +
1213 +
13 +

621 +831
21 +31
622 +832
22 +32
623 +833
23 +33

10
8

Lead concentrations:
Gasoline 1:
Gasoline 2:
Gasoline 3:

0.00511 + 0.0221 +0.0331

11 + 21 +31
0.00512 + 0.0222 +0.0332
12 + 22 +32
0.00513 + 0.0223 +0.0333
13 + 23 +33

0.01
0.02
0.01

Dept. of Mathematical Information Technology

Octane values:

Example 2: Water allocation

Dept. of Mathematical Information Technology

Water allocation
Dept. of Mathematical Information Technology

Papermaking process consumes lots of

water
Water can be circulated and reused in
different parts of the process as long as it
remains fresh enough
Fresh water costs
The aim is to minimize the amount of fresh
water required by the process
How to formulate the optimization
problem?

Water allocation
Dept. of Mathematical Information Technology

Objective function: minimize the amount of

fresh water used
Constraints:
water used should be fresh enough
Energy and mass balances between the
different unit processes (requires a
process model)
Can not (usually) be formulated explicitly
but requires e.g. the use of a process
simulation software

Linear = all functions are linear

Nonlinear = at least one function is nonlinear
Continuous = variables real-valued
Discrete = only finite (or countable) number of
possible values for the variables
Stochastic = problem contains uncertainties
Multiobjective = multiple objective functions

Dept. of Mathematical Information Technology

Different types of optimization

problems

Unconstraint = all values of the variables

are feasible
Box constraints = variables have upper and
lower bounds
Linear constraints = feasible region is
convex polyhedron
Nonlinear constraints = feasible region can
be anything

Dept. of Mathematical Information Technology

Different types of optimization

problems

Local vs. global optima

2

Dept. of Mathematical Information Technology

1.5

0.5

local minima
-0.5

-1

global minimum
-1.5
-5

-4

minimize
Adopted from Prof. Janos D. Pinter

-3

-2
-1
2
sin(x +x)+cos(3x)

-5x2

Local vs. global optima

1
0

-1
0
-4

-2

-2
0
-4

2
4

Only two variables curse of dimensionality

Adopted from Prof. Janos D. Pinter

Dept. of Mathematical Information Technology

How to find optimal solutions?

Dept. of Mathematical Information Technology

Trial and error widely used in practice, not

efficient and high possibility to miss good
solutions
Better to use a systematic way to find optimal
solution
Typically we know only
function value(s) at the current trial point
possibly gradients at the current trial point
How can we know when an optimal solution
has been reached?

Optimality conditions
Dept. of Mathematical Information Technology

How can we know that a solution is optimal?

One way is to utilize optimality conditions
Necessary optimality conditions = conditions
that an optimal solution has to satisfy (does
not guarantee optimality)
Sufficient optimality conditions = conditions
that guarantee optimality when satisfied
1. order conditions (1. order derivatives) and 2.
order conditions (2. order derivatives)

Unconstraint problem
Necessary: Let be differentiable in . If is a
local minimizer, then

= 0
Sufficient: Let be twice differentiable in . If
= 0 and the Hessian ( ) is positive
definite, then is a strict local minimizer.
Note: An element of

is =

2 ( )

Dept. of Mathematical Information Technology

min , . .

Info: Definite Matrices

0.
A symmetric matrix is positive
definite if
> 0 0 .

Dept. of Mathematical Information Technology

A symmetric matrix is positive

semidefinite if

Example of optimality conditions

2

Dept. of Mathematical Information Technology

satisfies only necessary conditions

1.5

0.5

-0.5

-1

satisfies both necessary and sufficient conditions

-1.5
-5

-4

minimize
Adopted from Prof. Janos D. Pinter

-3

-2
-1
2
sin(x +x)+cos(3x)

Unconstraint problem
Dept. of Mathematical Information Technology
Adopted from Prof. L.T. Biegler (Carnegie Mellon
University)

Inequality constraints
Dept. of Mathematical Information Technology
Adopted from Prof. L.T. Biegler (Carnegie Mellon
University)

Inequality and equality constraints

Dept. of Mathematical Information Technology
Adopted from Prof. L.T. Biegler (Carnegie Mellon
University)

Optimality conditions: necessary

Dept. of Mathematical Information Technology

Optimality conditions: sufficient

Dept. of Mathematical Information Technology

Solving an optimization problem

Dept. of Mathematical Information Technology

Find optimal values for the variables

Problems that can be solved analytically
min 2 , 3 = 3
Usually impossible to solve analytically
Must be solved numerically
approximation of the solution

Numerical solution
Dept. of Mathematical Information Technology

Modelling mathematical model of the

problem
Numerical methods numerical simulation
model for the mathematical model
Optimization method solve the problem
utilizing the numerical simulation model
SO
modelling simulation optimization

Optimization method
1.

2.

3.

Choose a stopping parameter > 0, starting point 1 and a

symmetric positive definite matrix 1 (e.g. 1 = ). Set 1 = 1
and = = 1.
If ( ) < , stop. Otherwise, set = ( ). Let be a
solution of
min ( + ), s.t. 0.
Set +1 = + . If = , set 1 = +1 = +1 , = + 1, = 1
and repeat (2).
Compute +1 . Set = + 1 and go to (2).

Method: numerical methods included

Software: a method implemented as a
computer programme

Dept. of Mathematical Information Technology

Algorithm: a mathematical description

Local optimization methods

Dept. of Mathematical Information Technology

Find a (closest) local optimum

Fast
Usually utilize derivatives
Mathematical convergence
For example
Direct search methods (pattern search,
Hooke & Jeeves, Nelder & Mead, )
Gradient based methods (steepest descent,
Newtons method, quasi-Newton method,
conjugate gradient, SQP, interior point
methods)

Global optimization methods

Dept. of Mathematical Information Technology

Try to get as close to global optimum as

possible
No mathematical convergence
Do not assume much of the problem
Slow, use lots of function evaluations
Heuristic, contain randomness
Most well known are evolutionary methods
based on improving a population of
solutions at a time instead of a single
solution

Hybrid methods
Dept. of Mathematical Information Technology

Combination of global and local methods

Try to combine the benefits of both
rough estimate with a global method,
fine tune with a local method
Challenge: how the methods should be
combined?
e.g. when to switch from global to local?
(speed vs. accuracy)

On selecting a software
Dept. of Mathematical Information Technology

Lots of software available (also open

source)
Decision Tree for Optimization Software,
http://plato.asu.edu/guide.html
NEOS Optimization Software Guide,
http://www.neos-guide.org/OptimizationGuide

Optimization platforms
Dept. of Mathematical Information Technology

AIMMS, http://www.aimms.com/
AMPL, http://www.ampl.com/
GAMS, http://www.gams.com/
Matlab,
http://www.mathworks.com/products/matlab
MPL, http://www.maximalsoftware.com/

Matlab example (by yourself)

Dept. of Mathematical Information Technology

Solve the mixing problem related to oil

refinery introduced earlier with Matlab
Matlab Optimization toolbox
Routine linprog is an optimization method
for linear problems

Examples of optimization
literature
Dept. of Mathematical Information Technology

P.E. Gill et al., Practical Optimization, 1981

M.S. Bazaraa et al., Nonlinear Programming: Theory
and Algorithms, 1993
D.P. Bertsekas, Nonlinear Programming, 1995
S.S. Rao, Engineering Optimization: Theory and
Practice, 1996
J. Nocedal, Numerical Optimization, 1999
A.R. Conn et al., Introduction to Derivative-Free
Optimization, 2009
M. Hinze et al., Optimization with PDE Constraints,
2009
L.T. Biegler, Nonlinear Programming Concepts,
Algorithms, and Applications to Chemical Processes,
2010

Journals in optimization
Dept. of Mathematical Information Technology

Applied Mathematics and Optimization

Computational Optimization and Applications
European Journal of Operational Research
Decision Support Systems
Journal of Global Optimization
Journal of Multi-Criteria Decision Analysis
Journal of Optimization Theory and Applications
Mathematical Programming
Omega
Operations Research
Optimization Letters
Optimization Methods and Software
Optimization
SIAM Journal on Control and Optimization
SIAM Journal on Optimization
Structural and Multidisciplinary Optimization

AIChE Journal
American Institute of Aeronautics and Astronautics
Applied Thermal Engineering
Computers & Chemical Engineering
Engineering Optimization
Engineering with Computers
Environmental Modelling & Software
Industrial & Engineering Chemistry Research
Journal of Environmental Engineering and Science
Optimization and Engineering
Water Science and Technology

Dept. of Mathematical Information Technology

Examples of journals in
application areas