1

Chapter 11
Optimization with Equality Constraints
Harold William Kuhn (1925) Albert William Tucker (1905-1995)
Joseph-Louis (Giuseppe Lodovico), comte de
Lagrange (1736-1813)
2
11.1 General Problem
Now, a constraint is added to the optimization problem:
max
x,y
u(x,y) s.t x p
x
+ y p
y
= I, where p
x
, p
y
are
exogenous prices and I is exogenous
income.
Different methods to solve this problem:
Substitution
Total differential approach
Lagrange Multiplier
3
( )
128 : for U(.) Value maximum Calculate ) 4
maximum 0 4
: s.o.c. Check
; 14 and ; 8 ; 0 4 32
: F.o.c. 3)
2 32 2 2 30
) x , U(x into ng Substituti ) 2
2 30
for x Solve 1)
2 4 60 s.t. 2 ) , (
*
2
1
2
*
2
*
1 1 1
2
1 1 1 1 1
2 1
1 2
2
2 1 1 2 1 2 1
=
< =
= = = =
= + =
=
+ = + = =
U
dx U d
x x x dx dU
x x x x x U
x x
x x x x x x x U U
11.1 Substitution Approach
Easy to use in simple 2x2 systems. Using the constraint, substitute
into objective function and optimize as usual.
Example:
4
( ) ( )
x x y y x y x y
x y x y
y
x
y
x
y x y x
g f g f g g f f
g g dy dx f f dy dx
dy
dx
g
g
d
dy
dx
f
f
d
dy g dx g dB dy f dx f dU
y x g B y x f U
= =
= =
(

=
(

=
= + = = + =
= =
; ) 10 9
; ) 8 7
0
0
B ;
0
0
U ) 6 5
0 ; 0 ) 4 3
, s.t. ; , 2) - 1
11.2 Total Differential Approach
Total differentiation of objective function and constraints:
Equation (10) along with the restriction (2) form the basis to solve
this optimization problem.
5
( )
( ) ( )
( )
( ) ( ) 128 8 2 14 8 ) 14 , 8 (
8 ; 14 2 2 1 1 4 60
2 2 2 1 2
2 1 ; 2
0 2 4 ; 0 2
*
1
*
2 2 2
2 1 2 1
2 1 2 1 2 1
2 1 1 2 1 1 2
= + =
= = + + =
+ = = +
= + =
= = = + + =
*
U
x x x x
x x x x
dx dx x x dx dx
dx dx dB dx dx x dx x dU
11.2 Total Differential Approach
Example: U=x
1
x
2
+ 2x
1
s.t. 60=4x
1
+ 2 x
2
Taking first-order differentials of U and budget constraint (B):
6
11.2 Total-differential approach
Graph for Utility function and budget constraint:
7
) , ( ) 2
) ' ' 10
) ' 10
y x g B
g f
g f
y y
x x
=
=
=

11.3 Lagrange-multiplier Approach

To avoid working with (possibly) zero denominators, let denote
the common value in (10). Rewriting (10) and adding the budget
constraint we are left with a 3x3 system of equations:
There is a convenient function that produces (10), (10) and (2) as
a set of f.o.c.: The Lagrangian function, which includes the objective
function and the constraint:
The constraint is multiplied by a variable, , called the Lagrange
multiplier (LM).
| | ) , ( ) , (
2 1 2 1
x x g B x x f L + =
8
11.3 Lagrange-multiplier Approach
| |
(
(
(

=
(
(
(

(
(
(

= =
= =
= =
+ =
0
0
L L
) 5 (
0 g ) 4 (
0 g ) 3 (
0 ) , ( ) 2 (
) , ( ) , ( ) 1 (
2
1
x x x x x
x x x x x
x x
x
x
2 1
2 1 2 1
2 2 1 2 2
2 1 1 1 1
2 1
2 2 2
1 1 1
B
x
x
L L L
L L L
L
f L
f L
x x g B L
x x g B x x f L
x x
x x

Once we form the Lagrangian function, the Lagrange function

becomes the new objective function.
9
11.3 LM Approach
(
(
(

=
(
(
(

(
(
(

0
0
g g 0
) 5 (
2
1
x x x x x
x x x x x
x x
2 2 1 2 2
2 1 1 1 1
2 1
B
x
x
L L g
L L g

Note that
L

= 0
L
x1
= g
x1
L
x2
= g
x2
.
Then
If the constraints are linear, the Hessian of the Lagrangian can be
seen as the Hessian of the original objective function, bordered by
the first derivatives of the constraints. This new Hessian is called
bordered Hessian.
10
( )
( ) ( )
yy x xx y xy y x
yy yx y
xy xx x
y x
yy yx y
xy xx x
y x
y
y
x
x
B y y y
x x x y x
y x
y x
y x
U P U P U P P
U U P
U U P
P P
Z Z Z
Z Z Z
Z Z Z
H
P
U
P
U
Z P U Z
P U Z yP xP Z
yP xP B y x U Z
yP xP B
U U x,y U U
2 2
2
0
0
0 0
,
constraint budget the Subject to
0 , where Utility Maximize
=

= =
= = = = =
= = = =
+ =
+ =
> =

|

11.3 LM Approach: Example
11
( )
( ) ( )
y
x
y
x
y
y
x
x
y x
y
y
y y
x
x
x x
y x
y x
U
U
P
P
B
U
P
U
P
U
yP xP B
L
P
U
P U
y
L
P
U
P U
x
L
yP xP B y x U L
P P B y x L L
=
c
c
= = =
= =
c
c
= = =
c
c
= = =
c
c
+ =
=
,
f.o.c. From
0 ) 5 (
0 ) 4 (
0 ) 3 (
, ) 2 (
, , ; , , ) 1 (
*

11.3 LM Approach: Interpretation

Note: From the f.o.c., at the optimal point:
- = marginal utility of good i, scaled by its price.
- Scaled marginal utilities are equal across goods.
- The ratio of marginal utilities (MRS) equals relative prices.
12
( )
y
x
y
x
y
x
y
y x
y
x
y x
U
U
P
P
dx
dy
x
P
P
P
B
y
yP xP B
U
U
dx
dy
dy U dx U dU
y x U U

= =
+ =
=

=
= + =
=
,
0
,
line budget and constraint Budget
(MRS) on substituti of ate marginal r
11.3 LM Approach: Interpretation
Note: Again, at the optimal point, the ratio of marginal utilities
(MRS) is equal to the ratio of prices.
Now, we total differentiate U(x,y). Well get similar results
13
11.3 LM Approach: Second-order conditions
has no effect on the value of L
*
because the constraint equals
zero but
A new set of second-order conditions are needed
The constraint changes the criterion for a relative max. or min.
( )
0 h 2 iff 0 ) 5 (
h 2 ) 4 (
2 (3)
y for constraint the solve

(2)
0 y x . . 2 ) 1 (
2 2
2
2 2
2
2
2 2
> + >
|
|
.
|

\
|
+ =
|
|
.
|

\
|
+
|
|
.
|

\
|
+ =
=
= + + + =
o o |
o o |
|
| o
b a H

x
b a H
x

b x

hx ax H
x y
t s by hxy ax H
14
min 0 a
0
iff
0 y x s.t. definite positive is ) 3 (
b - h 2 a - a
0
) 2 (
0 b h 2 a ) 1 (
2 2
2 2
<
= +
+ = =
> + =
b h
h
H
b h
h H
H
|
o
| o
| o
|
o
| o
11.3 LM Approach: Second-order conditions
15
( )
( )
( )
( ) ( )
( )
( )
( )( ) ( ) 4 ; 128 ; 8 2 14 8 ) 17 15
14 ; 2 8 4 60 ) 14 13
8 ; 2 2 2 4 60 ) 12 11
2 2 ; 2 1 2 1 4 1 ) 10 9
2 1 ; 0 2 ) 8 7
2 1 4 1 ; 0 4 2 ) 6 5
0 2 4 60 ) 4
2 4 60 2 3)
function Lagrangian the Form
0 2 4 60 B s.t. 2 2) - 1
* *
*
2 2
*
1 1 1
1 2 1 2
1 1
2 2
2 1
2 1 1 2 1
2 1 1 2 1
2
1
= = + =
= =
= + =
= = +
= = =
+ = = + =
= =
+ + =
= = + =

U U
x x
x x x
x x x x
x x L
x x L
x x L
FOC
x x x x x L
x x x x x U
x
x

11.3 LM Approach: Example

16
( )
maximum is L definite, negative is ; 0 16 ) 8
;
0
2
60
0 1 2
1 0 4
2 4 0
) 7
0 2 ) 6
0 4 2 ) 5
conditions order 1 0 2 4 60 ) 4
function Lagrangian 2 4 60 2 3)
constraint Budget 0 2 4 60 B 2)
function Utility 2 1)
* 2
2
1
1
2
2 1
2 1 1 2 1
2 1
1 2 1
2
1
L d H
x
x
x L
x L
x x L
x x x x x L
x x
x x x U
x
x
st

> =
(
(
(

=
(
(
(

(
(
(

= =
= + =
= =
+ + =
= =
+ =

11.3 LM Approach: Example - Cramers rule

17
( ) ( ) 128 8 2 14 8 2 ) 16
14 16 224 8 16 128 4 16 64 ) 15 13
224 240 16
0 1 2
2 0 4
60 4 0
) 12
128 8 120
0 0 2
1 2 4
2 60 0
) 11
64 60 4
0 1 0
1 0 2
2 4 60
) 10
16 8 8
0 1 2
1 0 4
2 4 0
) 9
*
1
*
2
*
1
*
*
2
*
1
*
2
1
= + = + =
= = = = = =
= + =

=
= + =

=
= + =

=
= + =

=
x x x U
x x
J
J
J
J
x
x

11.3 LM Approach: Example - Cramers rule

18
;
0
H ;
0
H
Where
min :) definite positive 0 ,..., 0 , 0 , 0 H
max ( : definite negative 0 ) 1 ,...( 0 , 0 , 0 H
361) (p. soc, case variable n d)
min :) definite positive 0 , 0 H
max ( : definite negative 0 , 0 H
soc of test variable 3 c)
min :) definite positive 0 H
max ( : definite negative 0 H
soc of test variable 2 b)
constraint than variable more one be must there because test variable - one No a)
33 3
22 2
11 1
3 2 1
3
22 21 2
12 11 1
2 1
2
4 3 2
4 3 2
3 2
3 2
2
2
(
(
(
(

=
(
(
(

=
< < < <
> > < >

< <
< >

<
>

Z Z Z g
Z Z Z g
Z Z Z g
g g g
Z Z g
Z Z g
g g
H H H
H H H
H
H
n
n
n
11.3 LM Approach: n-variable case
11.5 Optimality Conditions Unconstrained
Case
Let x* be the point that we think is the minimum for f(x).
Necessary condition (for optimality):
df(x*) = 0
A point that satisfies the necessary condition is a stationary
point
It can be a minimum, maximum, or saddle point
Q: How do we know that we have a minimum?
Answer: Sufficiency Condition:
The sufficient conditions for x* to be a strict local minimum
are:
df(x*) = 0
d
2
f(x*) is positive definite
11.5 Constrained Case KKT Conditions
To proof a claim of optimality in constrained minimization (or
maximization), we have to check the found point (x*) with respect
to the (Karesh) Kuhn Tucker (KKT) conditions.
Kuhn and Tucker extended the Lagrangian theory to include the
general classical single-objective nonlinear programming problem:
minimize f(x)
Subject to g
j
(x) > 0 for j = 1, 2, ..., J
h
k
(x) = 0 for k = 1, 2, ..., K
x = (x
1
, x
2
, ..., x
N
)
11.5 Interior versus Exterior Solutions
Interior: If no constraints are active and (thus) the solution lies at
the interior of the feasible space, then the necessary condition for
optimality is same as for unconstrained case:
Vf(x*) = 0 (V difference operator for matrices --del )
Exterior: If solution lies at the exterior, then the condition
Vf(x*) = 0 does not apply because some constraints will block
movement to this minimum.
Some constraints will (thus) be active.
We cannot get any more improvement (in this case) if for x* there
does not exist a vector d that is both a descent direction and a
feasible direction.
In other words: the possible feasible directions do not intersect
the possible descent directions at all.
11.5 Mathematical Form
A vector d that is both descending and feasible cannot exist if
-Vf = E
i
(Vg
i
) (with
i
> 0) for all active constraints ieI.
This can be rewritten as 0 = Vf + E
i
(Vg
i
)
This condition is correct IF feasibility is defined as g(x) s 0.
If feasibility is defined as g(x) > 0, then this becomes -Vf = E

i
(-Vg
i
)
Again, this only applies for the active constraints.
Usually the inactive constraints are included as well, but the
condition
j
g
j
= 0 (with
j
> 0) is added for all inactive constraints
jeJ.
This is referred to as the complimentary slackness condition.
Note that this condition is equivalent to stating that
j
= 0 for
inactive constraints -i.e., zero price for non-binding constraints!
Note that I+J = m, the total number of (inequality) constraints.
11.5 Necessary KKT Conditions
For the problem:
Min f(x)
s.t. g(x) s 0
(n variables, m constraints)
The necessary conditions are:
Vf(x) + E
i
Vg
i
(x) = 0 (optimality)
g
i
(x) s 0 for i = 1, 2, ..., m (feasibility)

i
g
i
(x) = 0 for i = 1, 2, ..., m (complementary slackness)

i
> 0 for i = 1, 2, ..., m (non-negativity)
Note that the first condition gives n equations.
11.5 Necessary KKT Conditions: General Case
For general case (n variables, M Inequalities, L equalities):
Min f(x) s.t.
g
i
(x) s 0 for i = 1, 2, ..., M
h
j
(x) = 0 for J = 1, 2, ..., L
In all this, the assumption is that Vg
j
(x*) for j belonging to active
constraints and Vh
k
(x*) for k = 1, ...,K are linearly independent
This is referred to as constraint qualification
The necessary conditions are:
Vf(x) + E
i
Vg
i
(x) + E
j
Vh
j
(x) = 0 (optimality)
g
i
(x) s 0 for i = 1, 2, ..., M (feasibility)
h
j
(x) = 0 for j = 1, 2, ..., L (feasibility)

i
g
i
(x) = 0 for i = 1, 2, ..., M (complementary slackness)

i
> 0 for i = 1, 2, ..., M (non-negativity)
(Note:
j
is unrestricted in sign)
11.5 Necessary KKT Conditions (if g(x)>0)
If the definition of feasibility changes, the optimality and
feasibility conditions change.
The necessary conditions become:
Vf(x) - E
i
Vg
i
(x) + E
j
Vh
j
(x) = 0 (optimality)
g
i
(x) > 0 for i = 1, 2, ..., M (feasibility)
h
j
(x) = 0 for j = 1, 2, ..., L (feasibility)

i
g
i
(x) = 0 for i = 1, 2, ..., M (complementary slackness)

i
> 0 for i = 1, 2, ..., M (non-negativity)
11.5 Restating the Optimization Problem
Kuhn Tucker Optimization Problem: Find vectors x
(Nx1)
,
(1xM)
and
(1xK)
that satisfy:
Vf(x) + E
i
Vg
i
(x) + E
j
Vh
j
(x) = 0 (optimality)
g
i
(x) s 0 for i = 1, 2, ..., M (feasibility)
h
j
(x) = 0 for j = 1, 2, ..., L (feasibility)

i
g
i
(x) = 0 for i = 1, 2, ..., M (complementary
slackness
condition)

i
> 0 for i = 1, 2, ..., M (non-negativity)
If x* is an optimal solution to NLP, then there exists a (*, *)
such that (x*, *, *) solves the KuhnTucker problem.
The above equations not only give the necessary conditions for
optimality, but also provide a way of finding the optimal point.
11.5 KKT Conditions: Limitations
Necessity theorem helps identify points that are not optimal. A
point is not optimal if it does not satisfy the KuhnTucker
conditions.
On the other hand, not all points that satisfy the Kuhn-Tucker
conditions are optimal points.
The KuhnTucker sufficiency theorem gives conditions under which
a point becomes an optimal solution to a single-objective NLP.
11.5 KKT Conditions: Sufficiency Condition
Sufficient conditions that a point x* is a strict local minimum of
the classical single objective NLP problem, where f, g
j
, and h
k
are
twice differentiable functions are that
1) The necessary KKT conditions are met.
2) The Hessian matrix V
2
L(x*) = V
2
f(x*) + E
i
V
2
g
i
(x*) +
E
j
V
2
h
j
(x*) is positive definite on a subspace of R
n
as defined
by the condition:
y
T
V
2
L(x*) y > 0 is met for every vector y
(1xN)
satisfying:
Vg
j
(x*)y = 0 for j belonging to I
1
= { j | g
j
(x*) = 0, u
j
* > 0}
(active constraints)
Vh
k
(x*)y = 0 for k = 1, ..., K
y = 0
KKT Sufficiency Theorem (Special Case)
Consider the classical single objective NLP problem.
minimize f(x)
Subject to g
j
(x) s 0 for j = 1, 2, ..., J
h
k
(x) = 0 for k = 1, 2, ..., K
x = (x
1
, x
2
, ..., x
N
)
Let the objective function f(x) be convex, the inequality
constraints g
j
(x) be all convex functions for j = 1, ..., J, and the
equality constraints h
k
(x) for k = 1, ..., K be linear.
If this is true, then the necessary KKT conditions are also
sufficient.
Therefore, in this case, if there exists a solution x* that satisfies
the KKT necessary conditions, then x* is an optimal solution to
the NLP problem.
In fact, it is a global optimum.
11.5 KKT Conditions: Closing Remarks
Kuhn-Tucker Conditions are an extension of Lagrangian
function and method.
They provide powerful means to verify solutions
But there are limitations
Sufficiency conditions are difficult to verify.
Practical problems do not have required nice properties.
For example, you will have a problems if you do not know
the explicit constraint equations.
If you have a multi-objective (lexicographic) formulation, then I
would suggest testing each priority level separately.
38
( ) ( )
( ) ( ) ( ) ( )
( )
( )
( )
( ) ( )
( )( ) ( ) ( )
( ) ( )
) conditions KKT (Meets 0 13 / 16 , 13 / 28 39 / 84 , 13 36
6 2 3 3 2 ; 4 4 2 / 3 4 3 2
4 3 2 4) From
4 4 2 3 4 4 3 2 2) and 1) From
: inactive) Constraint (1st 0 , 0 Let : 2 Case
) conditions KKT (Violates 0 13 / 28 , 13 / 24 , 13 / 10 39 / 30
13 6 * 3 5 4 3 2 4 2 3 3
; 2 3 3 3) From
; 4 3 2 4 2) and 1) From
: inactive) Constraint (2nd 0 , 0 Let : 1 Case
; 0 2 3 12 ) 4
; 0 3x 2x - 6 ) 3
; 0 2 3 4 2 ) 2
; 0 3 2 4 2 ) 1
: F.o.c.
2 3 12 3x 2x - 6 4 4 : Lagrangian Form
0 x , ; 0 2 3 - 12 6; 3x 2x .t. 4 4 Minimize
2
*
1
*
2
*
2 2 2
2 1
2 1 2 1 2
1
1
*
1
*
2
*
2 2 2
2 1
2 1 1
2
2 1
2 1
2 1 2
2 1 1
2 1 2 2 1 1
2
2
2
1
2 1 2 1 2 1
2
2
2
1
1
2
2
1
2
1
> = = = =
= + = +
+ =
+ = = =
< =
< = = = =
= =
=
= =
< =
s + + =
s =
= + =
= + =
+ + + + + =
> > > + + =

x x
x x x
x x
x x x x
L
x x
x x x
x x
x x
L
x x L
L
x L
x L
x x x x L
x x x s x x C
x
x
11.5 KKT Conditions: Example
39
( ) minimum. a for conditions KKT meet the not do , , ,
0
13
10
26
20
0
13
24
26
48
0
13
28
26
56
20 48 , 56 , 26
8 0 3
8 2 2
6 2 0
,
2 8 3
0 8 2
3 6 0
,
2 0 8
0 2 8
3 2 6
,
8
8
6
2 0 3
0 2 2
3 2 0
x and , x , for solve to (3) & (2), (1), eq.s Use
0 3 8 2 ) 3
0 2 8 2 ) 2
0 3x 2x - 6 ) 1
. 0 , 0 ,& x , x therefore , 0 L 0, L L L Assume
2 1 2 1
2 1 1
2
1
1
2 1 1
1 2
1 1
2 1
2 1 2 1 x x
2 1 1
2 1 1
2
1
1
2 2 1 1
x x
x x
J J J J
J J J
x
x
x L
x L
L
x x
x x
x
x

> = = > = = < =

=
= = = =

=

=
(
(
(

=
(
(
(

(
(
(

> =
> =
s =
= > < = = =
11.5 KKT Conditions: Using Cramers rule
40
( ) minimum. a for conditions KKT meet the , , ,
0
13
36
26
72
0
13
28
26
56
0
13
16
26
32
72 56 , 32 , 26
8 0 2
8 2 3
12 3 0
,
2 8 2
0 8 3
2 12 0
,
2 0 8
0 2 8
2 3 12
,
8
8
12
2 0 2
0 2 3
2 3 0
x and , x , for solve to (3) & (2), (1), eq.s Use
0 2 8 2 ) 3 (
0 3 8 2 ) 2 (
0 2 3 12 ) 1 (
. 0 , 0 ,& x , x therefore , 0 L 0, L ,& L , L Assume
2 1 2 1
2 1 2
2
1
2
2 1 2
2 2
2 1
2 1
1 2 2 1 x x
2 1 2
2 1 2
2
1
2
1 2 2 1
x x
x x
J J J J
J J J
x
x
x L
x L
x x L
x x
x x
x
x

> = = > = = > = =
= = = =
= = =
(
(
(

=
(
(
(

(
(
(

> + =
> + =
s + + =
= > < =
11.5 KKT Conditions: Example