Basics on Optimization

Topics Linear programming, Graph & SIMPLEX,

lpSolveAPI in R, GRC nonlinear, Evolutionary
(Genetic Algorithm)

Linear versus Non-linear Models

Linear function
When the function is linear (without any power
function, logarithmic function, inverse function or
trigonometric function)
ax + by + cz + dw = e (a, b, c, d, e = real valued; x, y,
z, w = variables)
Linear programming
All of the Objective function and Constraints are linear
combination of variables
Non-linear programming
At least one of the objective function or constraints will
be non-linear
Dipankar Bose - XLRI

Excel Solver Main Solution

Procedures

Simplex LP
Linear problem
Variable can be real, binary or integer
For binary or integer variable
Branch-and-bound technique is used
GRG (Generalized Reduced Gradient) Nonlinear
Used for non-linear problem
Concept based on derivative
Generates local optimal solution
Evolutionary
Used for non-linear problem
Compares across the ranges of variables
Dipankar Bose - XLRI

Example of a Linear Maximization

Problem

A carpenter can produce Chairs and Tables. Profits are as

follows:
Chair = 1 per unit; Table = 5 per unit
The carpenter can decide the number of chairs and tables
he will produce
These are known as variables
In other words, variables are controllable in real life
Assume he will prepare x chairs and y tables
Then his total profit = x + 5y
This is linear function of x and y
Carpenters objective is to maximize the profit
In absence of constraints The profit can take any
value
Dipankar Bose - XLRI

Constraints Makes us to act within

certain boundary

Constraints Creates boundary to my objective

Suppose there are following constraints:
Number of Tables should be
At the max (No. of Chairs + 10)/3 or less
x + 3y <= 10
He can produce maximum 6 items in a day
x + y <= 6
Maximum difference between Chairs and Tables
Cannot exceed 2 units
x y <= 2
Can we create these inequality constraints?
Chairs = x, Tables = y
Dipankar Bose - XLRI

The Maximization Problem

Maximize:
Subject to:

z = x + 5y
x + 3y <= 10
x + y <= 6
x y <= 2
Sign constraints: x, y >= 0
The constraints can be presented as equality constraints:
Subject to:
x + 3y + S1 = 10
x + y + S2 = 6
x y + S3 = 2
S1, S2 and S3 are also variables as they depend on the
values of x and y Total variables = 5
Lets open Excel file on graph
Dipankar Bose - XLRI

Graphical Interpretation of Linear

Optimization Problem

Dipankar Bose - XLRI

Simplex We get following vectors

Objective z = x + 5y
and matrices:
Constraints

x + 3y + S1 = 10
x + y + S2 = 6
x y + S3 = 2

List of variables:
x, y, S1, S2, S3
Note: All variables must be at left side of equations
The objective function:
z x 5y 0S1 0S2 0S3 = 0
Objective function coefficient vector
[ 1 5 0 0 0]
Co-efficient Matrix
Constraint co-efficient vector
1 3 1 0 0

1
1 0 1 0
Identity Matrix
1 1 0 0 1
Right hand side vector:
[10 6 2]

Dipankar Bose - XLRI

Simplex method How it works

Meaning of simplex table

Standard form
Basic variable
Basic solution (may not be feasible)
Feasible solution satisfies all constraints
Leaving and entering variable
Pivot row, column and number
Basic feasible solution (may not be optimal)
Optimal solution provides best result
How to check along with reason
Infeasibility, degeneracy, unbound and multiple
optimal
Dipankar Bose - XLRI

Standard form of Simplex Table

Objective z = x + 5y
Constraints
x + 3y + S1 = 10
x + y + S2 = 6
x y + S3 = 2

Reduced cost of
Basic Variables
Objective function row
BV

S1

S2

S3

Soln.

S1

10

S2

S3

Current Solution of Objective function

Variables in basic solution

Basic variable solution

Dipankar Bose - XLRI

Operation on Simplex Table Rules

BV

S1

S2

S3

Soln.

S1

10

S2

S3

Current solution:
[x y S1 S2 S3] = [0 0 10 6 2] at (x, y) = (0, 0)
Objective function value:
sumproduct([ 1 5 0 0 0] ,[0 0 10 6 2]) = 0
This is not optimal solution
All the values in z-row must be >= 0 (non-negative)
Objective function improves maximum
If we remove maximum negative co-efficient from the
objective (z-row)
Dipankar Bose - XLRI

SIMPLEX Justification of the

algorithm

Why the z-row values must be NON-NEGATIVE?

In maximization problem, any negative coefficient
Will make corresponding variable = 0
z-row values are known as relative cost
Variables which are part of the solution
Relative cost =0 and
Variables which are NOT part of the solution
Relative cost >=0 and
Why maximum negative in z-row enters?
In previous example
If x becomes BV, objective is increased = 1/unit
If y becomes BV, objective is increased = 5/unit
Dipankar Bose - XLRI

Simplex Continued:

BV

S1

S2

S3

Soln.

Ratio

S1

10

10/3

S2

6/1

S3

NA

Then y will be part of basic solution

Which of S1, S2 or S3 will exit from basic solution?
Use Minimum Ratio Rule
(Solution column value)/(entering variable column
POSITIVE value)
In this case Minimum{10/3, 6/1} = 10/3
S1 will leave basic solution S1 becomes Binding
Constraint
Intersection (black rectangle) = PIVOT
Dipankar Bose - XLRI

Why Ratio test is done?

Revisit the Excel file

A negative coefficient will cut variable axis at

Negative value (violates sign constraints)
Hence, they are not considered
The lowest ratio moves the point on the axis of the
entering basic
To the next point
In this case The point moves on y-axis
From (0, 0) to (0, 10/3)
Why row operations are done?
It brings the table with respect to point (x, S1) = (0, 0)

Dipankar Bose - XLRI

Simplex Iteration Continued

BV

S1

S2

S3

Soln.

S1

10

S2

S3

First, divide all leaving row values by PIVOT

BV

Z
y

S1

0
1/3

3/3

S2

S3

1/3
After Row
Operation

S2

S3

0/3

0/3

Soln.
10/3

Second, New row = Old row Leaving row /PIVOT

(entering column CELL)
Dipankar Bose - XLRI

BV

S1

S2

S3

Soln.

S1

10

S2

S3

S1
0 + 5/3
1/3
0 1/3
0 + 1/3

S2
0
0
1
0

S3
0
0
0
1

Soln.
0 + 50/3
10/3
6 10/3
2 + 10/3

S1
5/3
1/3
1/3
1/3

S2
0
0
1
0

S3
0
0
0
1

Soln.
50/3
10/3
8/3
16/3

BV
Z
y
S2
S3

x
y
1 5/3 5 + 5
1
1/3
1 + 1/3
11
1 1/3 1 + 1

Finally:
BV
Z
y
S2
S3

x
8/3
1/3
4/3
2/3

y
0
1
0
0

Dipankar Bose - XLRI

Simplex Iteration Continued

BV
Z
y
x
S3

x
0
0
1
0

y
0
1
0
0

S1

S2

1/4

3/4

S3
0
0
0
1

Soln.

Finally
BV
Z
y
x
S3

x
8/3 + 8/3
1/3 + 1/3
1
2/3 2/3

y
0
1
0
0

S1
S2
0+2
5/3 2/3
1/3 1/12 0 + 1/4
3/4
1/4
1/3 + 1/6 0 1/2

S3
0
0
0
1

Soln.
50/3 + 16/3
10/3 + 2/3
2
16/3 4/3

Dipankar Bose - XLRI

Final Simplex Table Interpretation

BV
Z
y
x
S3

x
0
0
1
0

y
0
1
0
0

S1
1
1/4
1/4
1/2

S2
2
1/4
3/4
1/2

S3
0
0
0
1

Soln.
22
4
2
4

Optimal value of objective

= 22, where x = 2 and y = 4
Constraint related to S3 (x y <= 2) is non-binding
S3 = 4
Both Constraint 1 and Constraint 2 are binding
Solution is optimal where Constraint 1 and 2 intersects
(the constraints related to two leaving variables)
Revisit Graph Solution
Dipankar Bose - XLRI

Lets Move to Excel Solver

Dipankar Bose - XLRI

Excel Solver Main Components

Objective z = x + 5y
Constraints
x + 3y 10
x+y 6
xy 2

Must be function of
Variable cells

Dipankar Bose - XLRI

Excel Solver Dialog Boxes

Dipankar Bose - XLRI

Solving the problem using Solver

Objective z = x + 5y
Sensitivity Analysis
Constraints

x + 3y 10
x+y 6
xy 2

After solving in Excel Solver

We also get sensitivity analysis
BV
Z
y
x
S3

x
0
0
1
0

y
0
1
0
0

S1
1
1/4
1/4
1/2

S2
2
1/4
3/4
1/2

S3
0
0
0
1

Soln.
22
4
2
4

Variable Cells
Cell
Name
Variable 1
x
Variable 2
y
Constraints
Constraint
Cell
LHS
Constraint 1 x + 3y
Constraint 2
x+y
Constraint 3
xy

Final
Value
2
4

Reduced
Cost
0
0

Objective
Coefficient
1
5

Allowable
Increase
4
1E+30

Allowable
Decrease
2.666666667
4

Final
Value
10
6
-2

Shadow
Price
1
2
0

Constraint
R.H. Side
10
6
2

Allowable
Allowable
Increase
Decrease
8
8
8
2.666666667
Dipankar
Bose
- XLRI
1E+30
4

Excel Solver Interpretation of

Shadow Price

If the Shadow price is ZERO

The constraint is non-binding
Increase in RHS of the constraint has no effect
If the Shadow price is POSITIVE
The constraint is binding
One unit increase in RHS of the constraint
Will increase optimal value of model by shadow
price
Positive shadow priced constraints are also called
Bottleneck
Those are the investment points
Dipankar Bose - XLRI

Lets Move to R and lpSolveAPI

(Refer material on R-Optimization)
Objective Maximize z =
Constraints

x + 5y
x + 3y
x+y
xy

<= 10
<= 6
<= 2

Dipankar Bose - XLRI

Working with lpSolveAPI

library(lpSolveAPI)

lptest = make.lp(0, 2)
lp.control(lptest, sense="max")

set.objfn(lptest, c(1, 5), indices = 1:2)

add.constraint(lptest, c( 1, 3), "<=", 10, indices = 1:2)

add.constraint(lptest, c(1, 1), "<=", 6, indices = 1:2)
add.constraint(lptest, c(1, 1), "<=", 2, indices = 1:2)

solve(lptest)

Dipankar Bose - XLRI

lpSolve positions all variables in a

vector How to position a matrix cell

Assume a variable = Vij where,

In Excel variable is at ith row and jth column
Consider 43 matrix is presented as OBJECTIVE vector
If we want to select position of Row 3 values only
indices = (desired Row number 1)(No. of columns)
+ c(1: No. of columns) = 7 8 9
R1

R1

R1

R2

R2

R2

R3

R3

R3

R4

R4

R4

C1

C2

C3

C1

C2

C3

C1

C2

C3

C1

C2

C3

If we want to select position of Column 2 values only

indices = seq(desired column number, (No. of rows
No. Of columns), No. of columns) = 2 5 8 11
R1

R1

R1

R2

R2

R2

R3

R3

R3

R4

R4

R4

C1

C2

C3

C1

C2

C3

C1

C2

C3

C1

C2

C3

Dipankar Bose - XLRI

Introduction to Non-linearity

Dipankar Bose - XLRI

Excel Non-smooth and Discontinuous

functions

Common non-smooth functions in Excel

ABS/MIN and MAX/INT and ROUND/CEILING and
FLOOR
These functions can also be converted into linear
function and SIMPLEX LP can be applied (by adding
integer variables)
Common discontinuous function in Excel
IF/CHOOSE/VLOOKUP/COUNT
Use Evolutionary Solving method to find a good
solution

Dipankar Bose - XLRI

Non-linear model Local versus

Global Solution

Consider the inventory problem in Excel file shared

Solve the problem in Excel Solver using
GRG Nonlinear
Starting point
x = 40
x = 45
x = 50
Evolutionary
0 x 70

Dipankar Bose - XLRI

Solver Controls Common Points

Different from
Objective function
convergence

Dipankar Bose - XLRI

Solver Controls Common Points

Continued

Max Sub-problems box

Type the maximum number of sub-problems that you
want to allow
Max Feasible Solutions box
Type the maximum number of feasible solutions that
you want to allow

Dipankar Bose - XLRI

GRG Nonlinear Solver Control Points

Convergence
Maximum percentage difference in
objective values in last five iterations
Derivatives
Central differencing yields more accurate
derivatives compared to forward
differentiating
Requires twice as many calculations of
the worksheet at each new trial solution

Dipankar Bose - XLRI

Multistart Option GRG Nonlinear

Used for Global Optimization

GRG Nonlinear method will be run
repeatedly
Starting from different starting values for
the decision variables
Automatically chosen based on
population size, random seed and
required bounds
Require Bounds on Variables
Can check or uncheck
Revisit Inventory problem in Excel
Dipankar Bose - XLRI

GRG Nonlinear Comments

Solver found a solution

A locally optimal solution is found

Solver has converged to the current solution

Objective function value is changing very slowly for the
last few iterations or trial solutions

Solver cannot improve the current solution

Model is degenerate and Solver is probably cycling
Some of your constraints are redundant

Dipankar Bose - XLRI

Terms used in Evolutionary

Fitness function is Constrained objective function

Crossover
Based on point of break
1-point crossover
1 0 0 1 0 1 1 0 0 1

1 0 0 1 0 1 1 1 1 0

0 1 0 1 0 0 0 1 1 0

0 1 0 1 0 0 0 0 0 1

N-point

crossover

0 1 0 1 0 1 1 0 1 1

1 0 0 1 0 0 1 0 1 0

1 0 0 1 0 0 0 1 1 0

0 1 0 1 0 1 0 1 1 1

There

are more procedures

Dipankar Bose - XLRI

Terms used in Evolutionary

Continued

Mutation
1 0 0 1 0 1 1 0 1 1

1 0 1 1 0 0 1 0 1 1

0 1 0 1 0 0 0 1 1 1

0 1 0 1 0 0 0 1 0 1

Mutation rate
Selection
Example of one technique
Give random values to each member of population
Stochastic sampling with replacement
Spin the wheel n times to select n values
Corresponding members are considered

Dipankar Bose - XLRI

Revisit Non-linear Inventory Problem

in Shared Excel File

Take initial guess of variable = 42

Make following changes
Convergence = 0.1
Mutation rate = 0.0000001
Random Seed = 1234

Dipankar Bose - XLRI

Solver Controls Evolutionary

Continued

Sub-problem
A new set of population
after crossover and
mutation
Same as iteration

Convergence
Maximum allowable percentage
difference in objective values for
the top 99% of the population
Mutation Rate
The relative frequency with which
some member of the population
will be altered or mutated to
create a new trial solution
During each generation or subproblem
Typically between 1/(population
size) and 1/(chromosome length)
Dipankar Bose - XLRI

Solver Controls Evolutionary

Continued

If convergence,
mutation rate or
population size is
increased Time
should be increased

Population Size
Number to maintain at any given time
in its population of candidate solutions
Solver uses a population size of 10
times the number of decision variables
in the problem, but no more than 200
Random seed
Change in random seed will change
the solution
Maximum Time without Improvement
Maximum number of seconds for
running model
Dipankar Bose - XLRI

Non-linear Inventory Problem in

Shared Excel File

Take initial guess of variable = 42

Make only one change at a time
Convergence 0.1 to 0.00001
Mutation rate 0.0000001 to 0.0075
Population rate 100 to 10000

In any of the cases, we will again get optimal solution

Dipankar Bose - XLRI

Rules for Evolutionary

Evolutionary outcome
Good Versus Optimal Solutions
Solver has Converged to the Current Solution
Solver Cannot Improve the Current Solution

All variable must have lower and Upper bounds specified

Dipankar Bose - XLRI

Evaluating a Solution Found by the

Evolutionary Solving Method

First, do the followings

Tighten the Convergence value
Increase the Population Size
Increase the Mutation Rate
Increase the time limit without improvement
Then, keep the resulting solution
Run evolutionary method again
Finally, keep the resulting solution
Switch to the GRG Solving method
Start it from that solution
The local optimal in GRC Nonlinear will be Global
Optimal
Dipankar Bose - XLRI

Some additional concepts on Linear

Optimization Models

Dipankar Bose - XLRI

Special cases

Unbounded solution
Multiple optimal solution
Degeneracy

Dipankar Bose - XLRI

Unbounded Solution

Maximize:
Subject to:

Sign constraints:

The initial simplex table:

BV
Z
S1
S2

x
4
1
2

z = 4x + y
x y <= 1
2x + y <= 2
x, y >= 0

y
1
1
1

S1
0
1
0

S2
0
0
1

Soln.
0
1
2
Dipankar Bose - XLRI

Unbounded Solution Continued

After 1st iteration we get:

BV
Z
x
S2

x
0
1
0

y
5
1
1

S1
4
1
2

S2
0
0
1

Soln.
4
1
4

Now y is the entering basic variable

However, there is no positive value in that column

y cannot enter

The problem is UNBOUNDED

Dipankar Bose - XLRI

Alternative Optimal Solutions

Maximize:
Subject to:

z = 4x + 10y
2x + y <= 10
2x + 5y <= 20
2x + 3y <= 18
Sign constraints: x, y >= 0
The initial simplex table:
BV

S1

S2

S3

Soln.

10

S1

10

S2

20

S3

18

Dipankar Bose - XLRI

Alternative Optimal Solutions

Continued

The final table (after some iterations):

BV

S1

S2

S3

Soln.

40

S1

8/5

1/5

2/5

1/5

S3

4/5

3/5

For the optimal table we find:

x = 0, y = 4 and Optimal value = 40
Here, S1-y-S3 are BV and others are Non-BV
But among Non-BV, x also has relative cost = 0
So it can become BV by replacing one of the current BV
Dipankar Bose - XLRI

Alternative Optimal Solutions

Continued

After one iteration

We find alternative optimal table:
BV

S1

S2

S3

Soln.

40

5/8

1/8

15/4

1/4

1/4

5/2

S3

1/2

1/2

Here, x = 15/4, y = 5/2 and optimal value = 40 (again)

Dipankar Bose - XLRI

Degeneracy

Maximize:
Subject to:

Sign constraints:

z = 2x + y
4x + 3y <= 12
4x + y <= 8
4x y <= 8
x, y >= 0

BV

S1

S2

S3

Soln.

Ratio

S1

12

12/4

S2

8/4

S3

8/4

Clearly, x enters, but which one will leave between S1 and

S2? Ratio test fails here.
Dipankar Bose - XLRI

Another Maximization problem: Use

of Big-M method

Maximize:
Subject to:

z = x + 3y
x + 2y >= 2
3x + y <= 3
x <= 4
Sign constraints: x, y >= 0
The constraints can be presented as:
Subject to:
x + 2y S1 = 2
3x + y + S2 = 3
x + 0y + S3 = 4
Co-efficient of S1 is negative
We need to penalize the function with artificial
variable R1
Dipankar Bose - XLRI

Primal dual problem

Canonical form
Conversion to Dual problem
Dual SIMPLEX method (Difference with Primal)
Comparison Final SIMPLEX table in primal and dual

Economic interpretation of duality

Complementary slackness Theorem

Dipankar Bose - XLRI

Some additional concepts on Nonlinear Optimization Models

Dipankar Bose - XLRI

Slopes or Derivatives for functions

Smooth functions
y = f(x) = x2

y = f(x) = - x2

Non-smooth function/discontinuous derivative function

y = f(x)
= |x| = ABS(x)

Negative slope

Positive slope

y = f(x) = x3

Derivative does not exist

Discontinuous function
Discontinuous function, Derivative
does not exist
Dipankar Bose - XLRI

Use of derivative in GRC Non-linear

Candidates for local optimal point When derivative

changes its sign while moving from LHS to RHS
Derivative will be either ZERO or
The function is discontinuous and the Derivative does
not exist or
The function is non-smooth and the Derivative does not
exist
Central differencing
Uses two points in opposite directions to check the sign
change rule
Forward differencing
Uses a single point, slightly different from the current
point
Dipankar Bose - XLRI

Main differences between GRG and

Evolutionary

Random versus Deterministic Operation

May generate different results for the same problem
Depends on the random seed chosen
A zero value specifies that Solver should use a different
seed for the random number generator each time it
runs
Population versus Single Best Solution
A population of solutions is created
One of these is best
Other members are sample points with the hope of
finding better solution in next iterations
Dipankar Bose - XLRI

Main differences between GRG and

Evolutionary Continued

Creating New Solutions Through Mutation

Periodically makes random changes or mutations in
one or more members of the current population
Evolutionary Solver uses five different mutation
strategies
Combining Solutions Through Crossover
Attempts to combine elements of existing solutions in
order to create a new solution, with some of the
features of each parent
Evolutionary Solver uses multiple variations of two
different crossover strategies
Dipankar Bose - XLRI

Main differences between GRG and

Evolutionary Continued

Selecting Solutions through Survival of the Fittest

The most fit members of the population survive
The least fit members are eliminated
Fitness depends
Partly on whether a solution is feasible
Partly on its objective function value

Dipankar Bose - XLRI

Evolutionary Solving Method

Limitations

Does not rely on

Derivative or gradient information
Different from GRG Nonlinear
It cannot determine
Whether a given solution is optimal
It never really knows
When to stop
It is not necessary that
With further iteration or sub-problem
The solution will improve
It may also deteriorate
But the best result is considered among iterations
Dipankar Bose - XLRI