1.

Continuous Optimization

Differentiable
Characteristics Related Solution algorithms
Linear programming, The general form of a LP is to determine the optimal To enhance the clarifying and understanding of algorithms,
LP solution from the linear objective function which the example problem is represented below:
limitation by the differentiable linear equality and
inequality constraints. An example of linear function and
constraint as showed below:
f (x) = 75 X1 + 50 X2 + 35 X3
4 X1 + 5 X2 20/3
8 X1 + 7 X3 25
X1, X2, X3 0
Where, Where,
F (x) = A linear objective function X1, X2, Xn = A vector of decision variables
X1, X2, X3 = A vector of decision variables C1, C2, Cn, a1, a2, an = constant variables or matrix/ linear
constraints limiting the possible combinations of value for
The problem of LP can be described using relationship of decision variables
linear line in graph. The limitation of LP is only
applicable in linearity of relations of decision variables. Optimization problem:
There is not all the function is linear objective function Minimize non-linear objective function subject to
and constraints in the surrounding environment. constraints
 Comparison:
In LP problem, it is constructed with linear objective In LP problem, it is used to matrix method to determine the
function which means the decision variables of LP only maximum or minimum point from the linear objective
can be solved using the linear objective function and function and constraint. Since the matrix is generally not
constraints. As other way around, the decision variables square form, which mean matrix normally perform in more
of NLP only can be solved using the linear objective columns than rows. Therefore, solving the LP problem is
function and constraints. not as simple as just inverting the matrix constraint. There
is great latitude to determine the maximum or minimum
point due to the matrix is likely to be under-determination.

Nonlinear Nonlinear programming, NLP also called as a smooth To enhance the clarifying and understanding of algorithms,
programming, NPL nonlinear programming, which is to determine the the example problem is represented below:
optimal solution from the non-linear objective function
which limitation by the differentiable linear or non-linear
equality and inequality constraints. An example of
nonlinear function and constraint as showed below:
F (x) = 2 X12 + 13 X22 + log X3
10 X1 + 8 X3 30
where,
X2 + 7 X3 19
f (x) = Nonlinear objective function
X2 8 X1 X3 0
gi(x)= Nonlinear constraints limiting the possible
Where,
combinations of value in {1, , m}
F (x) = A nonlinear objective function
hj(x) = Nonlinear constraints limiting the possible
 X12, X23, log X3 = A vector of decision variables combinations of value in {1, , p}
x = Vector of decision variables
Based on above equation example as we can see that
some of the decision variables used to raise a power or Optimization problem:
multiplied with other logarithms in NPL problems. Minimize non-linear objective function subject to
Sometimes, NLP problems can be either performed in constraints
convex or non-convex condition. NLP is applicable in
linearity or non-linearity of relations of decision In NLP, it is used to matrix method to determine the
variables. maximum or minimum point from the linear or non-linear
constraint and non-linear objective function. Matrix
Comparisons: normally performs in more columns than rows as well as
Nonlinear functions are difference with the linear not in square from of matrix. Therefore, solving the NLP
functions, the decisions variables of nonlinear problem is not as simple as just inverting the matrix
programming problem which is raised to a power, constraint. There is great latitude to determine the
multiplied or divided by other variables. The efficiency maximum or minimum point due to the matrix is likely to
of NLP to solve the problem is lower than the LP be under-determination. The constraint of NLP problem
problem. The reason is because sometimes NLP will be defined by a system of equalities and inequalities.
performed in non-convex condition but convex problems
easier to be solved compare to non-convex problems due
to the power of decision variables in NLP problem.
Quadratic A quadratic programming, QP is special case of NPL. The To enhance the clarifying and understanding of algorithms,
programming, QP objective function of QP is a linear objective which is to the example problem is represented below:
determine the optimal solutions from the linear objective
function which limitation by differentiable quadratic or
linear or non-linear equality and inequality constraints.
An example of quadratic objective function and
constraint as showed below:

f(X) = 0.5x13 + x22 x1x3 2x1 -6x2 where,

3 X1 + 3 X2 8 f (x) = A quadratic or linear function
4 X2 + 5 X3 21 Ax, b, Aeq, beq = linear constraints limiting the possible
X1 X2 0 X3 5 combinations of value in {Ib, , ub}
Where, x = vector of unknown variables
F (x) = A quadratic or linear function
X1, X2, X3 = A vector of numerical or integer decision Optimization problem:
variables Minimize objective function, f (x) subject to constraints,
Ax and Aeqx
Based on above equation example as we can see that
some of the decision variables used to raise a degree QP problem likely same with smooth non-linear problem
power of 3, 2, and 1 in a non-linear objective function. and but it is used to solve by specialized, more efficient
Sometimes, QP problems can be either performed in method. However, others non-linear optimization method
convex or non-convex condition. such as the method of SQP or GRG can also be used to
Comparison: solve the QP problem. The constraint of QP problem
QP problems are something likely to the LP problems. Due to defined by a system of equalities and inequalities.
 the linear constraints as represented in QP problem, the
optimal point can be found anywhere within the feasible
region or on its surface. However, LP only has linear objective
function and constraints but the QP can have non-linear
quadratic objective function and linear constraint. The
efficiency of QP to solve the problem is lower than the LP
problem. The reason is because sometimes NLP problem
will be performed in non-convex condition but convex
problems easier to be solved compare to non-convex
problems due to the power of decision variables in QP
problem.
Non-differentiable
Direct Search In direct search method, it is do not use derivative To enhance the clarifying and understanding of
information but tend to converge slowly and tolerant to algorithms, the example problem is represented below.
present of noise in function and constraints. Direct search Hooke and Jeevess Pattern search algorithm provided
method is derivative free because they have neither f C
proof those assumptions of use to establish
compute nor approximately derivatives. In unconstrained
convergence.
minimization:
Minimize f (x),
n F is convex and f ( x ) + because |x|+
Where f : R R
To reduce k ,

This method also can be called as zero order method where

it does construct approximations of function f(x). f ( x k ) f (x k k e i)
Numerical function value is not required when
where i = {1,,n}
differentiating characteristics of direct search method. It is
ei = ith unit coordinate vector
depends on objective functions and only new iterates which
k very small, so f ( x k +e i ) < f ( x k ) f ( x k ei ) < f ( x k )
will decrease in objective function that can be accepted.

for atleast i {1, ., n }

Comparison:
In direct search method, it is does not require any
gradients information of the objective function. It is unlike
with others optimization methods that require information
of gradient to find the optimal solutions. It is one of the
method that is very easy and fastest for those non-
differentiable or discontinuous objective function to obtain
the optimal solutions.
Stochastic Search In stochastic search, it is to solve many hard combinatorial To enhance the clarifying and understanding of
problems. It can generate random variables that appear algorithms, the example problem is represented below:
itself in formulation of optimization problem. There are L() and g() can be measured as:
two properties of stochastic search method. One is random
y ( ) L ( ) + ( )
noise in measurement of L() or g() . Another one
Y ( ) g ( )+ e ( )
is random choice made in search direction as algorithm
iterates toward a solution. Where and e is noise terms.

Noise term show independent at .

Where,
Measurement of L or g will be at =^ k , so the
k = random vector for both the properties
equation will be:
k = at kth iterations and k =1, 2, 3,..
( y k ) y k ( ^ k )=L( ^ k )+ k ( ^ k )

Stochastic search method can apply to solve the global ( Y k ) Y k ( ^ k )=g ( ^ k )+ e k ( ^ k )

optimization problem. In global optimization problem,

Where,
min f (x ) dependent on k
xS
y ( ) and yk as measurement of L
Where,
x = vector of decisions variables Y ( ) and Y k as measurement of g
S = n-dimensional feasible region
If noise not depend on , then
Then the problem denote by x* and y*
 Discrete Optimization

Characteristics Solutions
Integer programming, Integer programming problem, IP also called as integer To enhance the clarifying and understanding of
IP linear programming, which is to determine an optimal algorithms, the example problem is represented below:
solution from linear objective function of discrete real
decision variable which limitation by linear equality and
inequality constraints.

As its name implies, the integer programming problem, IP

which all the variables are limited to take integer or
discrete values only. The zero and 1 as commonly used of
each integer variables in IP problems. An example of
integer objective function and constraint as showed below: where,
f (x) = 7 X1 + 5 X2 c and b = vectors
6 X2 + 3 X1 3 A = Matrix with having integer values
4 X2 + 9 X1 20 x = Vector of unknown integer decision variables (-2,
X1 X2 0 and integer -1, 0, 1, 2)
T and n = power of vector
Where,
f (x) = A linear function Optimization problem:
X1, X2, X3 = A vector of integer decision variables Minimize objective function, f (x) subject to constraints,
 Comparison: Ax and Aeqx
Linear programming, LP is difference with the integer
programming, IP. Integer programming only allowed On the other hand, IP is a technique which to conduct a
integer quantities to be used instead of fractions number. qualitative assessment of the linear objective function and
Therefore, the difficulty of solving integer programming constraint. IP as a polyhedron in feasible set. The
problems is getting higher compare to linear programming constraint of QP problem defined by a system of
problem. equalities and inequalities.

Mixed integer linear IMILP is to determine the optimal solutions from linear To enhance the clarifying and understanding of
programming, MILP objective function of discrete real decision variable which algorithms, the example problem is represented below:
limitation by linear equality and inequality constraints to
obtain a global optimum can be identified. An example of
mixed integer linear function and constraint as showed
below:

f (x) = 7 X1 + 5 X2 + 3 X3
6 X1 + 3 X2 -10 Where,
4 X1 + 9 X3 20 f and b = vectors
0 X1, X2, X3 5 A and Aeq = Linear matrix with having integer values
X1 0 Ib and ub = lower and upper vector of decision variables
T and n = power of vector
Where, Xi = Vector of integer decision variables (-2, -1, 0, 1,
 f (x) = A linear function 2)
X1, X2, X3 = A vector of numerical and real integer decision Xj = decision variables which must be a binary number
variables (0,1)

The special case of MILP problem is a decision variable Optimization problem:

X1 of linear equation which the decision variable x1 must Minimize objective function, f (x) subject to constraints,
be either binary number 0 or 1 in the optimal solution. The Ax and Aeqx
decision in MILP problem can be defined by binary integer
variables. However, the difficulty to solve the MILP
problem is presented due to optimization problem of non-
convex caused by integer variables.

Comparisons:
The difference between MILP problem and IP problem is
IP allowed only integer values meanwhile MILP allowed to
take numerical/ real value and integer value in decision
variables.Therefore, the MILP problem easier to solve
compare to integer programming, IP problem.

Mixed integer In MINLP, is to determine the optimal solutions from non- To enhance the clarifying and understanding of
nonlinear linear objective function of discrete real decision variable algorithms, the example problem is represented below:
programming which limitation by non-linear equality and inequality
problem (MINLP) constraints to obtain a global optimum can be identified.
In MINLP, some of the decision variables are numerical or
other variables integers to get a global optimum can be
identified. An example of integer function and constraint as
showed below:
f (x) = 4 X12 + 5 X2
6 X12 + 3 X2 -10
5 X12 + 9 X2 17
0 X12, X2, X3 6
X1 0

Where,
f (x) = A nonlinear function Where,

X1, X2, X3 = A vector of numerical and real integer decision f (x) = Objective function
variables b and beq = vectors
A an Aeq = Linear vectors of matrix with having integer

The special case of MNILP problem is a decision variable values

X1 of non-linear equation which the decision variable x1 Ib and ub = lower and upper vector of decision variables
must be either binary number 0 or 1 in the optimal C = Nonlinear vector vectors of matrix with having
solution. The decision in MINLP problem can be defined integer values
 by binary integer variables. Xi = Integer decision variables ( -1, 0, 1)
Xj = Binary decision variables (0,1)
Comparison:
The similarities between MINLP and MILP is both allowed
only to solve the linear objective function integer values in Optimization problem:
which some of the decision variables are numerical or Minimize objective function, f (x) subject to constraints,
other variables integers. MINLP problem is allowed only to Ax and Aeqx
solve the nonlinear objective function with integer values
in which some of the decision variables are numerical or Generally, the applications of MINLP are widely used in
other variables integers. Therefore, the Mixed integer process plant design, manufacturing, chemical
linear programming, MILP problem easier to solve engineering and finance field to find the best value of the
compare to Mixed integer linear programming, MINLP performance criterion. The use of integer variables greatly
problem. Since, all Linear programming, LP problem, LP expands the scope of useful optimization problems that
are convex which mean it is intrinsically easier to solve you can define and solve. The constraint of QP problem
than nonlinear programming problems, NLP which may be defined by a system of equalities and inequalities.
non-convex.

The comparison tables of the optimization models between similarities and differences of 8 sub-types of optimization models are represented:
 Table 1.1: Comparison table of the optimization models between similarities and differences of LP, NLP and QP.

Optimization models Linear Programming, LP Non-linear Programming, NLP Quadratic Programming, QP

Objective Function Linear objective function Non-linear objective function Non-linear objective function
Constraints Linear constraints Linear or non-linear constraints Linear constraints
Gradient preformed Convex Convex and non-convex Convex and non-convex
Vector of decision Decision variables only raised Decision variables raised to a power Decision variables raised to a power
variable to a power with 1. or multiplied by other variables or or multiplied by other variables.
logarithms
Advantages - Most suitable to solve - Applicable in linearity or non- - Applicable in linearity or non-
complex problems linearity of relations of decision linearity of relations of decision
- Ease of use or solve variables variables
- Efficiency - Lower efficiency compare to LP - Likely with the LP due to linear
- Fast and reliable to solve constraints
- Lower efficiency compare to LP

Disadvantages - Only applicable in linearity of - Sometimes will be performed in - Sometimes will be performed in
relations of variable non-convex condition non-convex condition

Table 1.2: Comparison table of the optimization models between similarities and differences of direct search and stochastic search.

Optimization models Direct Search Stochastic Search

Objective Function Non-linear objective function Random objective function
Constraints Non-linear or linear constraints Non-linear or linear constraints
Gradient preformed - No require any information of gradient Convex and non-convex

Advantages - Suitable to solve non-differentiable - Suitable to solve non-differentiable objective function or

objective function or non-even continuous
- To solve difficult global optimization
problems
- Ease to solve, high efficiency
Disadvantages - Need to do few times of iteration
non-even continuous
- Random iteration to be used
- Uneasy to solve
- Not always can find the global optimum point
- Needed higher computational effort

Table 1.3: Comparison table of the optimization models between similarities and differences of LP, NLP and QP.

Optimization Integer programming, LP Mixed integer linear programming, Mixed integer non-linear programming,
models MILP MINLP
Objective Function Linear objective function Linear objective function Non-linear objective function
Constraints Linear constraints Linear or non-linear constraints Linear or non-linear constraints

Gradient preformed Convex Convex Convex and non-convex

Vector of decision Decision variables only Decision variables raised to a real Decision variables raised to a real
variable raised to a integer value integer and numerical mathematical integer and numerical mathematical
Advantages - Suitable to solve - More easier to solve compare to - Applicable in linearity or non-linearity
discrete problem MINP of relations of decision variables
- Applicable for linear
objective function and
constraints
Disadvantages - Only applicable in - Only applicable in linearity of - Sometimes will be performed in non-
linearity of relations of relations of variable convex condition
variable - The vector of decision variables - The vector of decision variables
- The vector of decision include an integers value include an integers value
variables must be an
integers