ECEN 629 Shuguang Cui

Lecture 5: Duality and KKT Conditions

Lagrange dual function
Lagrange dual problem
strong duality and Slaters condition
KKT optimality conditions
sensitivity analysis
generalized inequalities
1
ECEN 629 Shuguang Cui
Lagrangian
standard form problem, (for now) we dont assume convexity
minimize f
0
(x)
subject to f
i
(x) 0, i = 1, . . . , m
h
i
(x) = 0, i = 1, . . . , p
optimal value p

, domain D
called primal problem (in context of duality)
Lagrangian L : R
n+m
R
L(x, , ) = f
0
(x) +
m

i=1

i
f
i
(x) +
p

i=1

i
h
i
(x)

i
0 and
i
called Lagrange multipliers or dual variables
objective is augmented with weighted sum of constraint functions
2
ECEN 629 Shuguang Cui
Lagrange dual function
(Lagrange) dual function g : R
m
R ]
g(, ) = inf
x
L(x, , ) = inf
x
_
f
0
(x) +
m

i=1

i
f
i
(x) +
p

i=1

i
h
i
(x)
_
minimum of augmented cost as function of weights
can be for some and
g is concave (even if f
i
not convex!)
example: LP
minimize c
T
x
subject to a
T
i
x b
i
0, i = 1, . . . , m
Note that L(x, ) = c
T
x+
m

i=1

i
(a
T
i
xb
i
) = b
T
+(A
T
+c)
T
x
hence g() =
_
b
T
if A
T
+ c = 0
otherwise
3
ECEN 629 Shuguang Cui
Lower bound property
if x is primal feasible, then
g(, ) f
0
(x)
proof: if f
i
(x) 0 and
i
0,
f
0
(x) f
0
(x)+

i
f
i
(x)+

i
h
i
(x) inf
z
_
f
0
(z) +

i
f
i
(z) +

i
h
i
(z)
_
= g(, )
f
0
(x) g(, ) is called the duality gap
minimize over primal feasible x to get, for any _ 0 and ,
g(, ) p

R
m
and R
p
are dual feasible if _ 0 and g(, ) >
dual feasible points yield lower bounds on optimal value!
4
ECEN 629 Shuguang Cui
Lagrange dual problem
lets nd best lower bound on p

:
maximize g(, )
subject to _ 0
called (Lagrange) dual problem
(associated with primal problem)
always a convex problem, even if primal isnt!
optimal value denoted d

we always have d

(called weak duality)

p

is optimal duality gap

5
ECEN 629 Shuguang Cui
Strong duality
for convex problems, we (usually) have strong duality:
d

= p

when strong duality holds, dual optimal

serves as certicate of optimality for primal

optimal point x

many conditions or constraint qualications guarantee strong duality for convex problems
Slaters condition: if primal problem is strictly feasible (and convex), i.e., there exists
x relint D with
f
i
(x) < 0, i = 1, . . . , m
h
i
(x) = 0, i = 1, . . . , p
then we have p

= d

6
ECEN 629 Shuguang Cui
Dual of linear program
(primal) LP
minimize c
T
x
subject to Ax _ b
n variables, m inequality constraints
dual of LP is (after making implicit equality constraints explicit)
maximize b
T

subject to A
T
+ c = 0
_ 0
dual of LP is also an LP (indeed, in std LP format)
m variables, n equality constraints, m nonnegativity contraints
for LP we have strong duality except in one (pathological) case: primal and dual both
infeasible (p

= +, d

= )
7
ECEN 629 Shuguang Cui
Dual of quadratic program
(primal) QP
minimize x
T
Px
subject to Ax _ b
we assume P 0 for simplicity Lagrangian is L(x, ) = x
T
Px +
T
(Ax b)

x
L(x, ) = 0 yields x = (1/2)P
1
A
T
, hence dual function is
g() = (1/4)
T
AP
1
A
T
b
T

concave quadratic function

all _ 0 are dual feasible
dual of QP is
maximize (1/4)
T
AP
1
A
T
b
T

subject to _ 0
. . . another QP
8
ECEN 629 Shuguang Cui
Equality constrained least-squares
minimize x
T
x
subject to Ax = b
A is fat, full rank (solution is x

= A
T
(AA
T
)
1
b)
dual function is
g() = inf
x
_
x
T
x +
T
(Ax b)
_
=
1
4

T
AA
T
b
T

dual problem is
maximize
1
4

T
AA
T
b
T

solution:

= 2(AA
T
)
1
b
can check d

= p

9
ECEN 629 Shuguang Cui
Introducing equality constraints
idea: simple transformation of primal problem can lead to very dierent dual
example: unconstrained geometric programming
primal problem:
minimize log
m

i=1
exp(a
T
i
x b
i
)
dual function is constant g = p

(we have strong duality, but its useless)

now rewrite primal problem as
minimize log
m

i=1
exp y
i
subject to y = Ax b
10
ECEN 629 Shuguang Cui
let us introduce
m new variables y
1
, . . . , y
m
m new equality constraints y = Ax b
dual function
g() = inf
x,y
_
log
m

i=1
exp y
i
+
T
(Ax b y)
_
inmum is if A
T
,= 0
assuming A
T
= 0, lets minimize over y:
e
y
i

m
j=1
e
y
j
=
i
solvable i
i
> 0, 1
T
= 1
g() =

i
log
i
b
T

11
ECEN 629 Shuguang Cui
same expression if _ 0, 1
T
= 1 (0 log 0 = 0)
dual problem
maximize b
T

i
log
i
subject to 1
T
= 1, ( _ 0)
A
T
= 0
moral: trivial reformulation can yield dierent dual
12
ECEN 629 Shuguang Cui
Duality in algorithms
many algorithms produce at iteration k
a primal feasible x
(k)
a dual feasible
(k)
and
(k)
with f
0
(x
(k)
) g(
(k)
,
(k)
) 0 as k
hence at iteration k we know p

_
g(
(k)
,
(k)
), f
0
(x
(k)
)
_
useful for stopping criteria
algorithms that use dual solution are often more ecient (e.g., LP)
13
ECEN 629 Shuguang Cui
Nonheuristic stopping criteria
absolute error = f
0
(x
(k)
) p

stopping criterion: until
_
f
0
(x
(k)
) g(
(k)
,
(k)
)
_
relative error =
f
0
(x
(k)
) p

]p

]

stopping criterion:
until
_
g(
(k)
,
(k)
) > 0 &
f
0
(x
(k)
)g(
(k)
,
(k)
)
g(
(k)
,
(k)
)

_
or
_
f
0
(x
(k)
) < 0 &
f
0
(x
(k)
)g(
(k)
,
(k)
)
f
0
(x
(k)
)

_
achieve target value or, prove is unachievable
(i.e., determine either p

or p

> )
stopping criterion: until
_
f
0
(x
(k)
) or g(
(k)
,
(k)
) >
_
14
ECEN 629 Shuguang Cui
Complementary slackness
suppose x

, and

are primal, dual feasible with zero duality gap (hence, they are
primal, dual optimal)
f
0
(x

) = g(

) = inf
x
_
f
0
(x) +
m

i=1

i
f
i
(x) +
p

i=1

i
h
i
(x)
_
f
0
(x

)+
m

i=1

i
f
i
(x

)
hence we have

m
i=1

i
f
i
(x

) = 0, and so

i
f
i
(x

) = 0, i = 1, . . . , m
called complementary slackness condition
ith constraint inactive at optimum =
i
= 0

i
> 0 at optimum =ith constraint active at optimum
15
ECEN 629 Shuguang Cui
KKT optimality conditions
suppose
f
i
are dierentiable
x

are (primal, dual) optimal, with zero duality gap

by complementary slackness we have
f
0
(x

) +

i
f
i
(x

) = inf
x
_
f
0
(x) +

i
f
i
(x) +

i
f
i
(x)
_
i.e., x

minimizes L(x,

)
therefore
f
0
(x

) +

i
f
i
(x

) +

i
h
i
(x

) = 0
16
ECEN 629 Shuguang Cui
so if x

, and

are (primal, dual) optimal, with zero duality gap, they satisfy
f
i
(x

) 0
h
i
(x

) = 0

i
0

i
f
i
(x

) = 0
f
0
(x

) +

i
f
i
(x

) +

i
h
i
(x

) = 0
the Karush-Kuhn-Tucker (KKT) optimality conditions
conversely, if the problem is convex and x

satisfy KKT, then they are (primal, dual)

optimal
17
ECEN 629 Shuguang Cui
Geometric interpretation of duality
consider set
, = (u, t) R
m+1
] x f
i
(x) u
i
, f
0
(x) t ]
, is convex if f
i
are
for _ 0,
g() = inf
_
_

1
_
T
_
u
t
_

_
u
t
_
,
_
u
t
,
t +
T
u = g()
g()
_

1
_
18
ECEN 629 Shuguang Cui
(Idea of) proof of Slaters theorem
problem convex, strictly feasible =strong duality
u
t
,
_

1
_
p

(0, p

) , supporting hyperplane at (0, p

):
(u, t) , =
0
(t p

) +
T
u 0

0
0, _ 0, (,
0
) ,= 0
strong duality supporting hyperplane with
0
> 0: for

= /
0
, we have
p

t +

T
u (t, u) ,, p

g(

)
Slaters condition: there exists (u, t) , with u 0; implies that all supporting
hyperplanes at (0, p

) are non-vertical (
0
> 0)
19
ECEN 629 Shuguang Cui
Sensitivity analysis via duality
dene p

(u) as the optimal value of

minimize f
0
(x), subject to f
i
(x) u
i
, i = 1, . . . , m
0
0
u
p

(
u
)
epi p

(0)
T
u

gives lower bound on p

(u): p

(u) p

m
i=1

i
u
i
if

i
large: u
i
< 0 greatly increases p

if

i
small: u
i
> 0 does not decrease p

too much
if p

(u) is dierentiable,

i
=
p

(0)
u
i
,

i
is sensitivity of p

w.r.t. ith constraint

20
ECEN 629 Shuguang Cui
Generalized inequalities
minimize f
0
(x)
subject to f
i
(x) _
K
i
0, i = 1, . . . , L
_
K
i
are generalized inequalities on R
m
i
f
i
: R
n
R
m
i
are K
i
-convex
Lagrangian L : R
n
R
m
1
R
m
L
R,
L(x,
1
, . . . ,
L
) = f
0
(x) +
T
1
f
1
(x) + +
T
L
f
L
(x)
dual function
g(
1
, . . . ,
L
) = inf
x
_
f
0
(x) +
T
1
f
1
(x) + +
T
L
f
L
(x)
_

i
dual feasible if
i
_
K

i
0, g(
1
, . . . ,
L
) >
21
ECEN 629 Shuguang Cui
lower bound property: if x primal feasible and
(
1
, . . . ,
L
) is dual feasible, then
g(
1
, . . . ,
L
) f
0
(x)
(hence, g(
1
, . . . ,
L
) p

)
dual problem
maximize g(
1
, . . . ,
L
)
subject to
i
_
K

i
0, i = 1, . . . , L
weak duality: d

always
strong duality: d

= p

usually
Slater condition: if primal is strictly feasible, i.e.,
x relint D : f
i
(x)
K
i
0, i = 1, . . . , L
then d

= p

22
ECEN 629 Shuguang Cui
Example: semidenite programming
minimize c
T
x
subject to F
0
+ x
1
F
1
+ + x
n
F
n
_ 0
Lagrangian (multiplier Z _ 0)
L(x, Z) = c
T
x + Tr Z(F
0
+ x
1
F
1
+ + x
n
F
n
)
dual function
g(Z) = inf
x
_
c
T
x + Tr Z(F
0
+ x
1
F
1
+ + x
n
F
n
)
_
=
_
Tr F
0
Z if Tr F
i
Z + c
i
= 0, i = 1, . . . , n
otherwise
dual problem
maximize Tr F
0
Z
subject to Tr F
i
Z + c
i
= 0, i = 1, . . . , n
Z = Z
T
_ 0
strong duality holds if there exists x with F
0
+ x
1
F
1
+ + x
n
F
n
0
23