Optimization with Scilab

June 29th 2011

Michal BAUDIN & Vincent COUVERT

Scilab Consortium

The free and open source software for numerical computation

Outline
Part 1 - What's new in Scilab 5 ?
Focus on the Nelder-Mead component

Part 2 - Optimization in Scilab: Matlab compatibility

Part 3 - OMD2 project: Scilab Platform Development

Part 4 - Conclusion
What is missing in Scilab ?

The
Thefree
freeand
software
open for
source
numerical
software
computation
for numerical computation
Part 1 What's new in Scilab 5 ?

1. Introduction 2. The Nelder-Mead Component

1.1 What's in Scilab ? 2.1 Introduction
1.2 What's new in Scilab v5 ? 2.2 The algorithm
1.3 What's new on Atoms ? 2.3 Test cases
2.4 Conclusions

The
Thefree
freeand
software
open for
source
numerical
software
computation
for numerical computation
1.1 What's in Scilab ?
Objective Bounds Equality Inequalities Size Gradient Solver
Needed
Linear yes linear linear medium - linpro
Quadradic yes linear linear medium - quapro
Quadratic yes linear linear large - qpsolve
Quadratic yes linear linear medium - qld
Non-Linear yes large yes optim
Non-Linear small no fminsearch
Non-Linear yes small no neldermead
Non-Linear yes small no optim_ga
Non-Linear small no optim_sa
N.Li.Lea.Sq. large optional lsqrsolve
N.Li.Lea.Sq. large optional leastsq
Min-Max yes medium yes optim/nd
Multi-Obj yes small no optim_moga
Semi-Def. lin. (spectral) large no semidef
L.M.I. lin. (spectral) lin. (spectral) large no lmisolve

The free and open source software for numerical computation

1.2 What's new in Scilab 5 ?
Genetic Algorithms: nonlinear objective, bounds, global optimization

Simulated Annealing: nonlinear objective, global optimization

The Nelder-Mead component: nonlinear objective, unconstrained,

derivative-free, local optimization

fminsearch: Matlab compatible

The free and open source software for numerical computation

1.3 What's new on ATOMS ?
Optimization Solvers:
Quapro: linear or quadratic objective, linear constraints (full
matrices),
SciIpopt: an interface to Ipopt. Nonlinear objective, nonlinear
constraints (beta version),
Fmincon: nonlinear objective, nonlinear constraints (alpha
version) Matlab compatible,
Other modules: Cobyla, Particle Swarm Optimization,
Optkelley,
Test Problems:
Uncprb: 35 unconstrained optimization problems,
AMPL: load AMPL problems into Scilab,
And also: CUTEr.

The free and open source software for numerical computation

Outline

1. Introduction 2. The Nelder-Mead Component

1.1 What's in Scilab ? 2.1 Introduction
1.2 What's new in Scilab v5 ? 2.2 The algorithm
1.3 What's new on Atoms ? 2.3 Test cases
2.4 Conclusions

Thefree
The freeand
software
open for numerical
source computation
software for numerical computation
2. The Nelder-Mead Component
2.1 Introduction

John Ashworth Nelder (8 October 1924 7 August 2010)

Source: http://www.guardian.co.uk/technology/2010/sep/23/john-nelder-obituary

The free and open source software for numerical computation

2. The Nelder-Mead Algorithm
2.1 Introduction

We are interested in solving the unconstrained continuous

optimization problem:

Minimize f(x)

with unbounded, real, multidimensional, variable x.

A direct search algorithm:
Uses only function values (no gradient needed),
Does not approximate the gradient.
A simplex method for function minimization , John Nelder, Roger
Mead, Computer Journal, vol. 7, no 4, 1965, p. 308-313

The free and open source software for numerical computation

2. The Nelder-Mead Algorithm
2.1 Introduction

Virginia Torczon (1989) writes:

"Margaret Wright has stated that over fifty percent of the calls
received by the support group for the NAG software library
concerned the version of the Nelder-Mead simplex algorithm to be
found in that library."

The free and open source software for numerical computation

2. The Nelder-Mead Algorithm
2.2 The algorithm

A simplex: a set of n+1 vertices, in n dimensions.

In 2 dimensions.

In 3 dimensions.

The free and open source software for numerical computation

2. The Nelder-Mead Algorithm
2.2 The algorithm

Steps in the Nelder-Mead algorithm

Inputs:
the n+1 vertices v(1), v(2), ..., v(n+1) of a nondegenerate
simplex in n dimensions,
the associated function values f(1),...,f(n+1),
the coefficients (reflection), (expansion), (contraction),
and (shrinkage).
Standard Nelder-Mead: =1, =2, =1/2, and =1/2.

The free and open source software for numerical computation

2. The Nelder-Mead Algorithm
2.2 The algorithm

The free and open source software for numerical computation

2. The Nelder-Mead Algorithm
2.3 Test cases

2 2
f x 1, x 2 =x 1 x 2 x1 x 2

function [ y , index ] = quadratic ( x , index )

y = x(1)^2 + x(2)^2 - x(1) * x(2);
endfunction
nm = neldermead_new ();
nm = neldermead_configure(nm,"-numberofvariables",2);
nm = neldermead_configure(nm,"-function",quadratic);
nm = neldermead_configure(nm,"-x0",[2 2]');
nm = neldermead_search(nm);
xopt = neldermead_get(nm,"-xopt");
nm = neldermead_destroy(nm);

The free and open source software for numerical computation

2. The Nelder-Mead Algorithm
2.3 Test cases

The free and open source software for numerical computation

2. The Nelder-Mead Algorithm
2.3 Test cases
Mc Kinnon, Convergence of the neldermead simplex method to a nonstationary
point . SIAM J. on Optimization, 1998
Failure by repeated inside contraction

The free and open source software for numerical computation

2. The Nelder-Mead Algorithm
2.3 Test cases
C. T. Kelley. Detection and remediation of stagnation in the neldermead algorithm
using a sufficient decrease condition SIAM J. on Optimization, 1999
Restart the algorithm...

The free and open source software for numerical computation

2. The Nelder-Mead Algorithm
2.4 Conclusions
Some general facts:
Memory requirement is O(n)
Shrink steps are rare
Generally 1 or 2 function evaluations by iteration
Convergence is slow. Typical number of iterations is 100n,
where n is the number of dimensions
Hundreds of iterations are not rare
Convergence can be even slower when n > 10 (Han &
Neumann, 2006)
Restart the algorithm when in doubt for convergence (Kelley,
1999)
Convergence is guaranteed in 1 dimension (Lagarias et al.,
1999)

The free and open source software for numerical computation

2. The Nelder-Mead Algorithm
2.4 Conclusions

We should not use this algorithm just because the gradient is not
required:
For example, if f is smooth, Quasi-Newton methods (optim)
with numerical derivatives converge much faster.
We may use this algorithm when:
No other property of the problem can be used (e.g. non linear
least squares can be solved by lsqrsolve),
The objective function is nonsmooth or "noisy" (Kelley, 1999),
We do not need too much accuracy (Torzcon, 1989),
The number of parameters is moderate (Han & Neumann,
2006).

The free and open source software for numerical computation

Part 3
Optimization in Scilab: Matlab compatibility

1. Introduction
2. Scilab Coverage
3. Overview

The
Thefree
freeand
software
open for
source
numerical
software
computation
for numerical computation
1. Introduction

Matlab has many functions for optimization:

Minimization,
Equation solving,
Datafitting and nonlinear least squares,
Global optimization.

Scilab has often similar functions: let's see which ones.

Matlab is a registered trademark of The Mathworks, Inc.

The free and open source software for numerical computation

1. Introduction
For each Matlab function, we search:
Scilab function, if available,
Differences of features, differences of algorithms.
(*) : Function will be reviewed at the end of the talk,
For most functions, the match is not 100% identical,
But some other functions can do it : which ones ?
We consider only Scilab Industrial Grade solvers:
Scilab internal modules,
ATOMS modules,
Portables on all OS,
Well documented,
Tested.

The free and open source software for numerical computation

1. Introduction
Main differences

Design:
Matlab: problem oriented (may be with several solvers),
Scilab: solver oriented (may be several solvers).
Function arguments:
Matlab nearly always provides common options,
Scilab is less homogeneous.
Management of the callbacks/extra-arguments:
Matlab: M-file or @
Scilab: list

The free and open source software for numerical computation

2. Scilab Coverage
Minimization:
fminbnd Not 100% identical,
But optim can do it.
fmincon ATOMS/fmincon (alpha version)
fminimax Not 100% identical,
but optim/''nd'' is designed for it.
fminsearch fminsearch 90% identical in Scilab 5.3.2.
fminsearch 99% identical in 5.4.0

The free and open source software for numerical computation

2. Scilab Coverage
fminunc Not 100% identical,
but optim/''qn'' or optim/''gc'' are designed for it.
No sparsity pattern of Hessian in Scilab.
No PCG in optim: L-BFGS instead.
linprog 100% for full matrices: karmarkar
ATOMS/quapro: linpro
No known solver for sparse matrices (*).
quadprog 100% for full matrices: qpsolve, qp_solve
ATOMS/quapro: quapro
No known solver for sparse matrices.

The free and open source software for numerical computation

2. Scilab Coverage
Equation Solving:
fsolve fsolve 100% for full matrices.
No known solver with sparse Jacobian (*).
fzero No identical function.
But fsolve can do it.
Least Squares (Curve Fitting):
lsqcurvefit datafit
lsqnonlin lsqrsolve (leastsq)

The free and open source software for numerical computation

2. Scilab Coverage
Global Optimization Toolbox:
Genetic Algorithm Not 100% identical,
But optim_ga is built-in Scilab.
No linear equality and inequality in Scilab,
but bounds are managed.
Simulated Annealing Not 100% identical,
But optim_sa is built-in Scilab
No bounds in Scilab SA, but user can
customize the neighbour function.

The free and open source software for numerical computation

3. Overview
Matlab Problem Scilab

bintprog Binary Integer Programming -

fgoalattain Multiobjective goal attainment -
fminbd Single-variable, on interval optim
fmincon Constrained, nonlinear, multivariable ATOMS/fmincon
fminimax Minimax, constrained optim/''nd''
fminsearch Unconstrained, multivariable, derivative-free fminsearch (100%)
fminunc Unconstrained, multivariable optim/''qn'',''gc''
fseminf Semi-infinitely constrained, multivariable, -
nonlinear
ktrlink Constrained or unconstrained, nonlinear, -
multivariable using Knitro
linprog Linear programming karmarkar,
ATOMS/quapro
quadprog Quadratic programming qpsolve,
ATOMS/quapro

The free and open source software for numerical computation

3. Overview
Matlab Problem Scilab

fsolve Solve systems of nonlinear equations fsolve

fzero Root of continuous function of one variable -
lsqcurvefi t Nonlinear least squares curve fitting datafit
lsqlin Constrained linear least squares -
lsqnonlin Nonlinear least-squares (nonlinear data-fitting) lsqrsolve, leastsq
lsqnonneg Nonnegative least squares -
optimtool GUI to select solvers, options and run problems -
Global Search Solve GlobalSearch problems -
Multi Start Solve MultiStart problems -
Genetic Algorithm Genetic Algorithms optim_ga
Direct Search Pattern Search -
Simulated Annealing Simulated Annealing optim_sa

The free and open source software for numerical computation

Part 4 -
OMD2 project: Scilab Platform Development

1. Overview
2. Modules
2.1 Data Management
2.2 Modeling
2.3 Optimization

Thefree
The freeand
software
open for numerical
source computation
software for numerical computation
Overview
OMD2 / CSDL projects collaboration

Will be available on Scilab forge:

http://forge.scilab.org/index.php/p/omd2/
Private project up to first release.
Scilab Optimization Platform:

Batch mode (script edition, large scale execution),

GUI mode (interactive edition, prototyping).
Future Scilab external module available through ATOMS.

The free and open source software for numerical computation

Main functionalities
Project management (Save & Load working data as HDF5 files)
Wrappers:
Scilab algorithms,
External tools,
Proactive.
Mask complexity for users
Modules:
Data Management,
Modeling,
Optimization,
Visualization.

The free and open source software for numerical computation

Data Management Module (1/2)
Factors / Parameters:
Load existing Design Of Experiments (Isight .db files, )
Generate Design Of Experiments:
DoE generator wrappers (LHS, ),

DoE generator settings.

Responses simulation using:

External tool (openFOAM, Catia, CCM+, ),
Scilab function.

2-D visualization:
Factor / Factor,
Response / Factor.

The free and open source software for numerical computation

Data Management Module (2/2)

The free and open source software for numerical computation

Modeling module (1/2)
Point selection:
Learning points used for modeling,
Validation points used to validate model,
Bad points (simulation issue, ).
Modeler:
Selected among modeler wrappers (DACE, Lolimot, ),
Parameters configuration,
Multiple model management with best model user selection.
Visualization:
2-D models,
Cross correlation,
Sensibility analysis.

The free and open source software for numerical computation

Modeling module (2/2)

The free and open source software for numerical computation

Optimization Module (1/2)

Responses coefficients values setting

Optimizer:
Selection among generic wrappers (optim, fmincon, genetic
algorithms, ),
Optimizer configuration,
Enable two chained optimizers.

Visualization:
Optimal point,
Paretos,
Robustness.

The free and open source software for numerical computation

Optimization Module (2/2)

The free and open source software for numerical computation

Part 4 - Conclusion

1. What is missing ?
2. Bibliography

Thefree
The freeand
software
open for numerical
source computation
software for numerical computation
Conclusion
1. What is missing ?

High Performance Optimization:

Use BLAS/LAPACK within optim ?
Sparse Linear Programming:
Update LIPSOL ?
Non Linear Programming:
Improve fmincon ?
Non Linear Programming Test Cases:
CUTEr requires a compiler on the test machine,
Connect the Hock-Schittkowski collection ?

The free and open source software for numerical computation

Conclusion
2. Bibliography

Nelder-Mead User's Manual , Michal Baudin, Consortium Scilab

DIGITEO, 2010
Optimization in Scilab , Baudin, Couvert, Steer, Consortium Scilab
- DIGITEO INRIA, 2010
Optimization with scilab, present and future , Michal Baudin and
Serge Steer, 2009 IEEE International Workshop on Open Source
Software for Scientific Computation, pp.99-106, 18-20 Sept. 2009
Introduction to Optimization with Scilab , Michal Baudin,
Consortium Scilab DIGITEO, 2010
Unconstrained Optimality Conditions with Scilab , Michal Baudin,
Consortium Scilab DIGITEO, 2010

The free and open source software for numerical computation

 Thanks for your attention

www.scilab.org

The free and open source software for numerical computation

Extra-Slides

Some slides you won't see, unless you ask...

The free and open source software for numerical computation

The Nelder-Mead Algorithm
Some Historical References:
Spendley, Hext, Himsworth (1962): fixed shape simplex algorithm
Nelder, Mead (1965): variable shape algorithm
Box (1965): simplex algo., with constraints
O'Neill (1971): Fortran 77 implementation.
Torczon (1989): Multi Directional Search.
Mc Kinnon (1998): Counter examples of N-M.
Lagarias, Reeds, Wright, Wright (1998): Proof of convergence in
dimensions 1 and 2 for strictly convex functions.
Han, Neumann (2006): More counter examples of N-M.

The free and open source software for numerical computation

The Nelder-Mead Algorithm

In what softwares N-M can be found ?

Matlab (fminsearch)
NAG (E04CBF)
Numerical Recipes (amoeba)
IMSL (UMPOL)
and Scilab since v5.2.0 in 2009
and R after Sbastien Bihorel's port of Scilab's source code.

The free and open source software for numerical computation

The Nelder-Mead Algorithm
1. Sort by function value. Order the vertices: f(1) f(n) f(n+1)
2. Calculate centroid. B = (v(1)+...+v(n))/n
3. Reflection. Compute R = (1+)B v(n+1) and evaluate f(R).
4. Expansion. If f(R)<f(1), compute E=(1+)B v(n+1) and evaluate f(E).
If f(E)<f(R), accept E, else accept R and goto 1.
5. Accept R. If f(1) f(R) < f(n), accept R and goto 1.
6. Outside Contraction. If f(n)f(R)<f(n+1), compute Co=(1+)B v(n+1)
and evaluate f(Co). If f(Co)<f(R), then accept Co and goto 1 else, goto 8.
7. Inside Contraction. If f(n+1)f(R), compute Ci=(1-)B +v(n+1) and
evaluate f(Ci). If f(Ci)<f(n+1), then accept Ci and goto 1 else, goto 8.
8. Shrink. Compute the points v(i)=v(1)+(v(i)-v(1)) and evaluate
f(i)=f(x(i)), for i=2,3,...,n+1. Goto 1.

The free and open source software for numerical computation

The Nelder-Mead Algorithm
Lagarias, Reeds, Wright, Wright (1998)
1. In dimension 1, the Nelder-Mead method converges to a
minimizer, and convergence is eventually M-step linear, when
the reflection parameter = 1.
2. In dimension 2, the function values at all simplex vertices in the
standard Nelder-Mead algorithm converge to the same value.
3. In dimension 2, the simplices in the standard Nelder-Mead
algorithm have diameters converging to zero.
Note that Result 3 does not implies that the simplices converge to a
single point x*.

The free and open source software for numerical computation

What's in Matlab ?
Minimization:
bintprog Solve binary integer programming problems
fgoalattain Solve multiobjective goal attainment problems
fminbnd Find minimum of single-variable function on
fixed interval
fmincon Find minimum of constrained nonlinear
multivariable function
fminimax Solve minimax constraint problem
fminsearch Find minimum of unconstrained multivariable
function using derivative-free method

The free and open source software for numerical computation

What's in Matlab ?
fminunc Find minimum of unconstrained multivariable
function
fseminf Find minimum of semi-infinitely constrained
multivariable nonlinear function
ktrlink Find minimum of constrained or unconstrained
nonlinear multivariable function using KNITRO
third-party libraries
linprog Solve linear programming problems
quadprog Quadratic programming

The free and open source software for numerical computation

What's in Matlab ?
Equation Solving:
fsolve Solve system of nonlinear equations
fzero Find root of continuous function of one variable

Least Squares (Curve Fitting):

lsqcurvefit Solve nonlinear curve-fitting (data-fitting)
problems in least-squares sense
lsqlin Solve constrained linear least-squares problems
lsqnonlin Solve nonlinear least-squares problems
lsqnonneg Solve nonnegative least-squares constraint
problem

The free and open source software for numerical computation

What's in Matlab ?
Utilities:
optimtool GUI to select solver, optimization options, and
run problems
optimget Optimization options values
optimset Create or edit optimization options structure

The free and open source software for numerical computation

What's in Matlab toolboxes ?
Global Optimization Toolbox:
GlobalSearch Create and solve GlobalSearch
problems
MultiStart Create and solve MultiStart problems
Genetic Algorithm Use genetic algorithm and Optimization
Tool, and modify genetic algorithm
options
Direct Search Use direct search and Optimization Tool,
and modify pattern search options
Simulated Annealing Use simulated annealing and
Optimization Tool, and modify simulated
annealing options

The free and open source software for numerical computation