CLASSICAL OPTIMIZATION THEORY

Quadratic forms

Let

x1
x
2

X .

xn

be a n-vector.

Let A = ( aij) be a nn symmetric matrix.

We define the kth order principal minor as
the kk determinant
a11 a12 ... a1k
a21 a22 ... a2 k
.
.
ak 1 ak 2 ... akk

Then the quadratic form

Q( X ) X A X
T

aii xi2 2
i

a x x

1i j n

ij i

(or the matrix A) is called

Positive semi definite if XTAX 0 for all X.
Positive definite if XTAX > 0 for all X 0.
Negative semi definite if XTAX 0 for all X.
Negative definite if XTAX < 0 for all X 0.

A necessary and sufficient condition for A (or

Q(X)) to be :
Positive definite (positive semi definite) is
that all the n principal minors of A are > 0 (
0).
Negative definite ( negative semi definite) if
kth principal minor of A has the sign of (-1)k ,
k=1,2,,n (kth principal minor of A is zero or
has the sign of (-1)k , k=1,2,,n )

Let f (X)=f (x1 ,x2 ,,xn ) be a real-valued

function of the n variables x1 ,x2 ,,xn (we
assume that f (X) is at least twice
differentiable ).
A point X0 is said to be a local maximum of
f (X) if there exists an > 0 such that
r
f ( X 0 h ) f ( X 0 ) for all h j
Here
and

h (h1 , h2 ,.., hn )

X 0 ( x10 , x20 ,.., xn0 )

r
0
0
0
X 0 h ( x1 h1 , x2 h2 ,.., xn hn )

X0 is said to be a local minimum of f (X) if

there exists an > 0 such that

f ( X 0 h ) f ( X 0 ) for all h
X0 is called an absolute maximum or global
maximum of f (X) if
j

f (X ) f (X0) X
X0 is called an absolute minimum or global
minimum of f (X) if

f (X ) f (X0) X

Theorem
A necessary condition for X0 to be an
optimum point of f (X) is that f ( X 0 ) 0
(that is all the first order partial derivatives f
xi
are zero at X0.)
Definition
A point X0 for which f ( X 0 ) 0
is called a stationary point of f (X) (potential
candidate for local maximum or local
minimum).

Theorem
Let X0 be a stationary point of f (X).
A sufficient condition for X0 to be a local
minimum of f (X) is that the Hessian matrix
H(X0) is positive definite; local maximum of
f (X) is that the Hessian matrix H(X0) is
negative definite.

Here H(X) is the nn matrix whose ith row are

the partial derivates of

f
x j

(j =1,2,..n)

with respect to xi. (i =1,2,..n)

i.e. H(X) =

2 f

x
1

2 f

x1x2

2 f
x1x2
2 f
x22

.
.

.
.

2 f

x1xn

2 f

x2 xn

f
xn x1
2

f
.
xn x2
2

f
xn2
2

Problem 3 set 20.1A page 705

Find the stationary points of the function
f (x1,x2,x3) = 2x1x2x3 4x1x3 2x2x3 + x12 + x22 +
x32 2x1 - 4x2 + 4x3
And hence find the extrema of f (X)
f
2 x 2 x3 4 x3 2 x1 2
x1

f
2 x1 x3 2 x3 2 x 2 4
x 2
f
2 x1 x 2 4 x1 2 x 2 2 x3 4
x3

All these above equations = 0 give

x2 x3 2 x3 x1 1

(1)

x1 x3 x3 x2 2

(2)

x1 x2 2 x1 x2 x3 2

(3)

(2) x2 2 x3 x1 x3

Substituting in (3) for x2, we get

2x1+ x1x3 x12x3 2x1 2 x3 + x1x3 + x3 =
-2
Or

x1x3 (2 x1) = 0

Therefore x1= 0 or x3 = 0 or x1 = 2

Case (i)

x1 = 0

(1) implies x2x3 2x3 = 1

(4)

(2) implies x2 x3 = 2

(5)

(3) implies -x2+x3 = -2

same as (5)

(4) using (5) gives

x3(2 + x3) 2x3 = 1
i.e. x3=
or x32 = 1

Sub case (i) x3 = 1 gives (using (5) ) x2 = 3

(ii) x3= -1 gives (using (5) ) x2 = 1
Therefore, we get 2 stationary points
(0,3,1), (0,1,-1).

Case (ii) x3 = 0
(1) gives x1 = 1
(2) gives x2 = 2
For these x1, x2
LHS of (3) = x1x2 - 2x1 - x2 = -2 = RHS
Therefore, we get the stationary point (1,2,0)

Case (iii) x1= 2

(1) gives x2x3 2x3 = -1

(6)

(2) gives x2 + x3 = 2

(7)

(3) gives x2 + x3 = 2
(6) Using (7) gives

same as (7)

x3(2 x3) 2x3 = -1

i.e.

x32 = 1, therefore x3 = 1

sub case (i) x3 = 1 gives (using (5)) x2 = 1

(ii) x3 = -1, gives (using (5) x2 = 3
Therefore, we get two stationary points
(2,1,1), (2,3,-1).
Hessian matrix =

2
2 x3 2 x2 4

2 x3
2
2 x1 2
2 x2 4 2 x1 2
2

Point

Principal
minors

Nature

(0,3,1)

2,0,-32

Saddle pt

(0,1,-1)

2,0,-32

Saddle pt

(1,2,0)

2,4,8

Local min

(2,1,1)

2,0,-32

Saddle pt

(2,3,-1)

2,0,-32

Saddle pt

Definition:
A function f(X)=f(x1,x2,xn) of n variables is
said to be convex if for each pair of points X,Y
on the graph, the line segment joining these
two points lies entirely above or on the graph.
i.e. f((1-)X + Y) (1-)f(X)+ f(Y)
for all such that 0 < < 1.

f is said to be strictly convex if for each pair

of points X,Y on the graph,
f ((1-) X + Y) < (1-) f(X)+ f(Y)
for all such that 0 < < 1.
f is called concave ( strictly concave) if
f is convex ( strictly convex).

Convexity test for function of one variable

A function of one variable f(x) is
2

convex if

d f
0
2
dx

concave if

d2 f
0
2
dx

Convexity test for functions of 2 variables

quantity

convex Strictly
convex

concave Strictly
concave

fxx fyy -(fxy)2

>0

>0

fxx

>0

<0

fyy

>0

<0

Constrained optimization
KarushKuhnTucker (KKT) conditions:
Consider the problem
maximize z = f(X) = f(x1, x2,, xn)
subject to g(X) 0 [g1(X) 0
g2(X) 0
.
gm(X) 0]
(the non-negativity restrictions, if any, are

Let S ( s , s ,..., s )
(where s , s22,..,sm2 are the non negative
slack variables added to g1(X) 0 , g2(X) 0,
, gm(X) 0 to make them into equalities).
2

2
1
2
1

2
2

2
m

We define the Lagrangean function

L(X, S, ) = f(X) [g(X) + S2]
= f(X) [1{g1(X) + s12} +
2{g2(X)+s22)}+ +m{gm(X) + sm2}]

KKT necessary conditions for optimality are

given by
ur
0
L
f ( X ) g ( X ) 0
X
L
2i Si 0, i 1, 2,..., m
Si
L
ur ( g ( X ) S 2 ) 0

These are equivalent to the following

ur
conditions: 0
ur
f ( X ) g ( X ) 0

i gi ( X ) 0, i 1, 2,..., m
g(X ) 0
(Complementary
slackness)

We denote the Lagrangean L(X, ) by

m

L( X , ) f ( X ) i gi ( X )
i 1

In scalar notation, this is given by

I.

i` 0, i=1,2,.m

II.

f
g1
g 2
g m
1
2
.. m
0
x j
x j
x j
x j
j 1, 2, . . . , n

III. i gi ( X ) 0, i 1,2,..., m
IV. g ( X ) 0, i 1,2,..., m
i

The same conditions apply for a

minimization problem also except that now
r
we have u

Also in both maximization and minimization

problems, the Lagrange multipliers i
corresponding to equality constraints
gi(X) = 0 must be URS (unrestricted in sign).

Sufficiency of the KKT conditions:

Required conditions
Sense of
optimization

Objective
function

maximization

concave

Convex set

minimization

convex

Convex set

Solution
space

It is simpler to verify whether a function is

concave or convex than to prove that the
solution space is a convex set.
We thus give a set of sufficient conditions
that are easier to check that the solution
space is a convex set in terms of the
convexity or concavity of the constraint
functions.

Consider the general non-linear problem:

Maximize or minimize z = f(X)
Subject to gi(X) 0 i = 1,2,.., p
gi(X) 0 i = p+1, p+2,.., q
gi(X) = 0 i = q+1, q+2,.., r

Sufficiency of the KKT conditions:

Sense of
optimization

Required conditions

f(X)

gi(X)

convex
maximization concave concave
linear
minimization convex

convex
concave
linear

i
1ip
0
p+1 i q
0
URS q+1 i r
0
0
URS

1ip
p+1 i q
q+1 i r

The conditions in the above table

represent only a subset of the conditions
given in the earlier table.
The reason is that a solution space may be
convex without satisfying the conditions
of the constraint functions given in the
second table.

Problem Use the KKT conditions to derive

an optimal solution for the following
problem:
maximize f(x1, x2) = x1+ 2x2 x23
subject to x1 + x2 1
x1

0
x2 0

Solution: Here there are three constraints

namely,
g1(x1,x2) = x1+x2 - 1 0
g2(x1,x2) = - x1

g3(x1,x2) =
- x2 0
Hence the KKT conditions become
10, 2 0, 3 0
g 3
g1
g 2
f
1
2
3
0
x1
x1
x1
x1

g 3
g1
g 2
f
1
2
3
0
x 2
x 2
x 2
x 2

1g1(x1,x2) = 0
2g2(x1,x2) = 0
3g3(x1,x2) = 0
g1(x1,x2) 0
g2(x1,x2) 0

Note:
f is concave
gi are convex,
maximization
problem
these KKT
conditions are
sufficient at the

i.e.

1 1 + 2 = 0

(1)

2 3x22 1 + 3 = 0 (2)
1(x1 + x2 1) = 0

(3)

2 x1

=0

(4)

3 x2

=0

(5)

x1 + x2 1 0

(6)

x1

(7)

x2

(8)

1 0

(10) and 3 0

(9)
(11)

(1) gives 1 = 1 + 2 1 >0 (using 10)

Hence (3) gives x1 + x2 = 1 (12)
Thus both x1, x2 cannot be zero.
So let x1>0 (4) gives 2 = 0. therefore 1 = 1
if now x2 = 0, then (2) gives 2 0 1 + 3 = 0 or 3 < 0
not possible
Therefore x2 > 0
hence (5) gives 3 = 0 and then (2) gives x22 = 1/3 so x2 =1/3
And so x1 = 1- 1/3

Maximize f(x) = 20x1 + 10 x2

(0,1)

Subject to x12 + x22 1

x1 + 2x2 2
x1 0, x2 0
KKT conditions become
20 - 21 x1 2 + 3 = 0
10 - 21 x2 22 + 4 = 0
1 (x12 + x22 1) =0
1 (x1 + 2x2 2) =0

(4/5,3/5
)

4 x2

=0

x12 + x22 1
x1 + 2x2 2
x1

x2

From the figure it is clear that max f occurs at (x1, x2) where
x1, x2 >0.
3 = 0, 4 = 0
suppose x1 + 2x2 2 0
2 = 0 , therefore we get 20 - 21 x1=0
10 - 21 x2=0

1x1=10 and 1x2=5, squaring and adding we get

12 = 125

1 = 55

therefore x1 = 2/5, x2 = 1/5, f= 50/5 >22

2 0

x1 + x2 2 = 0

Therefore x1=0, x2=1, f =10

Or

x1= 4/5, x2= 3/5, f=22

Therefore max f occurs at x1 = 2/5, x2 = 1/5

Problem Use the KKT conditions to derive

an optimal solution for the following
problem:
minimize f(x1, x2) = x12+ x2
subject to x12 + x22 9
x1 + x2 1
Solution: Here there are two constraints,
namely, g1(x1,x2) = x12+x22 - 9 0
g2(x1,x2) = x1 + x2 -1 0

Thus the KKT conditions are:

1 : 1 0, 2 0

as it is a minimization problem

2: 2x1 21 x1 2 0
1 -21 x2 2 0
3 : 1 ( x x 9) 0
2
1

2
2

2 ( x1 x2 1) 0
4: x x 9
2
1

2
2

x1 x2 1

Now 1 0

(from 2) gives 2 1 Not possible.

Hence 1 0 and so

x x 9
2
1

2
2

(5)

Assume 2 0. So (1st equation of ) (2) gives

2 x1 (1 1 ) 0

Since 1 0

we get x1= 0

From (5), we get x2 3

2nd equation of (2) says (with 1 0, 2 0 ) x2 = -3
Thus the optimal solution is:
1
x1 0, x2 3, 1 , 2 0
6

The optimal value is :

z 3

Basic

r
1
R1 0
R2 0
S1 0
S2 0
r
x1

x1
22
0
40
-18
1
1
0

R2 0

S1

S2

x2 1
-8
2
0
0
-18
1
10
1
1
0
4
0
19
10
9

20
19
10
29
20
89
20

2
5
0
1
4
0
0

1
-1
0
-1
0
0
0

2 R1
-1
0
0
-1
0
1
-1
0
0
0
0
0

29 89
9

20 20
20
1
1
1

40 40
40
29 89
9

20 20
20
1
1
1

40
40
40
1
1
1

40
40
40

11

20
1
40
9
20
1

40
1

40

1
0
1
0
0

R2 S1 S2
0
0
0
-1
0
0
0
0
0
1
0
0
0
1
0
0
0
1

Sol
70
0
20
50
6
18

59

1
2

59

11
2
35
2

Basic

r
x1

r
1

R2 0
x2
S2
r
x1

x2

917 2657 337

580 580
580

0
0

0
0

0
0

R2 0
x2

x1

0
0

1
0
0
0

7
7

1160 1160

327
1160

1740
580

2
29

327
1160

0
0

2
29

327
1160

1
0
0

R1

243
580

7
1160

337
580
2

29
327
1160

2320
2900

1 580
580
1
1

0 40
40

20
1
40

1 1

7
1160

917 2657 337

580 580 580

2
29

1
0
0

9
20
1

40

R2

S1

9
29

0
1

38
29
20
29

0
0

38
29

89
29

S2

Sol

1502
29

64
29

1502
29

110
29

18
29

0
0

1
2

59

11
2

3560 1160
327 327

720
327

0
0

Using Excel Solver, the optimum solution is:

x1 = 1.604651, x2 = 4.395349 and
the maximum z = 230.7209