Dynamic Optimization

Institut fr Theoretische Volkswirtschaftslehre
Makrokonomik
Prof. Dr. Thomas Steger
Advanced Macroeconomics | Lecture| SS 2014
Dynamic Optimization
N P i G diti
(June 05, 2014)
No-Ponzi-Game condition
Method of Lagrange Multipliers
Dynamic Programming
Control Theory
Makrokonomik
Th N P i G diti The No-Ponzi-Game condition
Finite life
( ) i e ( ) excluded a T a T < 0 0
Everyone must repay his/her debt, i.e. leave the scene without debt at terminal point in time.
No-Ponzi-Game condition (NPGC) represents an equilibrium constraint that is imposed on every agent.
Infinite life
( ) , i.e. ( ) excluded a T a T < 0 0
Infinite life
Assume Mr. Ponzi (and his dynasty) wishes to increase consumption today by x. Consumption
lim ( ) , i.e. lim ( ) excluded
rt rt
t t
e a t e a t

< 0 0
Charles Ponzi (photo:
1910) became known in
the 1920s as a swindler
for his money making
scheme. He promised
clients huge profits by
buying assets in other
countries and redeeming
expenditures are being financed by borrowing money. Debt repayment as well as interest
payments are being financed by increasing indebtedness further. Debt then evolves according to
t 0 1 2
debt x x(1+r) x(1+r)
2
discrete time: ( )
t
t
d x r = + 1
countries and redeeming
them at face value in the
US. In reality, Ponzi was
paying early investors
using the investments of
later investors. This type
of scheme is now known
as a "Ponzi scheme".
(Wikipedia June 3
rd
2013)
Noting that d(t)=-a(t) the above NPGC may be stated as
debt x x(1+r) x(1+r)
2

consumption debt repayment debt repayment
continuous time: ( )
rt
d t e x =
(Wikipedia, June 3 2013)
If M P i i ti b fi d b l i hi i ti fi i h d bt l di t
[ ]
lim ( )
lim ( ) , i.e. lim exclu d ( ) de
rt
t
rt rt
t t
e a t
e t d e d t
>
0
0 0
2
If Mr. Ponzi increases consumption by x, financed by employing his innovative financing scheme, debt evolves according to
d(t)=e
rt
x such that the present value of debt would remain positive, which is excluded
lim
t
rt rt
e e x x
= > 0
Makrokonomik
Th M th d f L M lti li (1) The Method of Lagrange Multipliers (1)
Consider the problem of maximizing an intertemporal objective function extending over
three periods p
( ) ( ) ( ) ( ) (*)
s t ( ) with given (**)
t
t
t
u c u c u c u c
x f x c x

=
= + +
=
2
0 1 2
0 1 2
0
s.t. ( , ) with given (
(real economy)
(NPGC, finan
)
cial economy)
t t t
t
x f x c x
x
x
+
1 0
2
0
0
where c
t
denotes the control variable, x
t
the state variable, and 0<<1 is the discount factor.
The problem is to maximize the objective function (*) with respect to c, c, and c subject to
the constraint (**).
This problem can easily be solved by the Method of Lagrange multipliers. This requires to
formthe Lagrangian function (We focus on interior solutions ) form the Lagrangian function. (We focus on interior solutions.)
[ ] [ ]
2
0 1 2 0
1 2
1 1 2 1 2 0 1
( ) ( ) ( ) ( , ) ( , ) x u c u c u c f x c f x c x = + + L
3
where we treat x as given and the variables c, c, c, x and x as unknowns.
Makrokonomik
Differentiating w.r.t. the unknowns yields
( ) ( )
( ) ( )
u c u c
c
f x c f x c
= = =

L
L
2
2 2
2
0 0
( , ) ( , )
( ) ( )
( , )
( )
f x c f x c
u c u c
c c c
f x c
u c
c c

= + = + =

= + =

L
L
2
1 1 1 1
1 2 1 2
1 1 1
0 0
0 1
0 0
0
( ) ( )
c c
x
f x c f x c

= = =

L
L
0 0
2
2 2
2
0 0
T th ith f( ) thi t f ti d fi ll d
( , ) ( , ) f x c f x c
x x x

= + = + =

L
2
1 1 1 1
1 2 1 2
1 1 1
0 0
Together with x
t+1
=f(x
t
,c
t
) this system of equation defines c, c, c, x, x as well as and
2
.
The FOC u(c)=0, assuming a concave utility function with u(c)>0 for all c, has the following
interpretation. It basically says that consumption should be increased as far as possible.
4
p y y p p
That is, the entire wealth is to be consumed in the terminal period. This implication can be
made explicit by taking border conditions into account.
Makrokonomik
Consider now the following stochastic intertemporal optimization problem
( )
t
E

`
{ }
s.t. , given
(real economy)
max ( )
( , )
t
t
t
c
t
t t t t
t
E u c
x f x c
x
x
=
+ +
`
)
=
= +
0
0
0
1 1
0
E denotes the expected value given information at t=0 and
t
is an i.i.d. random variable with E(
t
)=0 and V(
t
)=
.
No Ponzi Game condition (NPGC) says that a Ponzi type financing scheme of consumption (increase consumption today by
(NPGC, financial econom l ) im y
t
t
t
r
x
| |

|
+
\ .
1
0
1
No-Ponzi Game condition (NPGC) says that a Ponzi-type financing scheme of consumption (increase consumption today by
running into debt and finance repayment and interest payments permanently by further increasing indebtedness) is excluded.
At t=0 the agent decides on c, taking x as given. The unknowns at this stage are c and x. The
associated Lagrangian function reads associated Lagrangian function reads
[ ]
1 0 1
0 0
: ( ) ( , )
t
t t
t
t t t t
t t
E u c f x c x

+
= =
+

=
`
)

L
This formulation is in line with the RBC examples (Heer and Maussner, 2004,
p. 37)
It leads to a FOC of the form u(c
t
)=
t
. (this formulation is quite usual).
If consumption today reduces the capital stock today (consumption at the
Remark: The definition of differs from Chow, where it reads

This formulation is in line with Chow (Chapter 2)
beginning of period), this formulation is appropriate.
5
[ ]
1
0 1
0 0
1 1
: ( ) ( , )
t
t
t t t
t
t
t
t
t
c E u c x f x
+
+
=
+ +
=

=
`
)

L
This formulation is in line with Chow (Chapter 2).
It leads to a FOC of the form u(c
t
)=
t+1
(this formulation is less usual).
If consumption today reduces the capital stock tomorrow (consumption at the
end of period), this formulation is appropriate.
Makrokonomik
For convenience the Lagrangian for the problem at t=0 is restated
[ ]
1 0 1
0 0
: ( ) ( , )
t
t t
t
t t t t
t t
E u c f x c x

+
= =
+

=
`
)

L
Let us write the relevant parts of the Lagrangian function (i.e. those including c and x)
[ ] [ ]
{ ( ) ( ) ( , ) ( , ) ...} x c x E u c u c c f x f x = + + L
0 1
0 0 1 0 1 1 2
1
1 2 0 1 1
0
0
The FOCs /c
0
=0 and /x
1
=0 read as follows
( ) f x c L
0 0
0 0 0
0 0
1
1 1
( , )
{ ( ) } 0
( , )
{ } 0
f x c
E u c
c c
f x c
E
= + =

+
L
L
6
1
1 1
0 0 1
1 1
( , )
{ } 0
f
E
x x
= + =

Makrokonomik
At t=1 the agent decides on c
1
, taking x
1
as given. The unknowns at this stage are c
1
and x
2
.
The associated Lagrangian function reads The associated Lagrangian function reads
[ ]
1 1
1 1 1
1 1
: ( ) ( , )
t t
t t t t t t
t t
E u c x f x c

+ +
= =

=
`
)

L
Let us write the relevant parts of the Lagrangian function (i.e. those including c
1
and x
2
)
[ ] [ ]
{ ( ) ( ) ( ) ( ) } E u c u c x f x c x f x c = + + L
0 1 0 1
1 1 t t )
The FOCs /c
1
=0 and /x
2
=0 read as follows
[ ] [ ]
{ ( ) ( ) ( , ) ( , ) ...} E u c u c x f x c x f x c + + L
1 1 2 1 2 1 1 2 2 3 2 2 3
generalizing
1 1
1 1 1
1 1
( , ) ( , )
{ ( ) } 0 { ( ) } 0
( ) ( )
t t
t t t
t
f x c f x c
E u c E u c
c c c
f x c f x c

= + = + =

L
L
Solution strategy: {c,c,c,...} were chosen sequentially, given the information x
t
at time t (closed-loop policy), rather
generalizing 1
1 1 2 2
1 1 2 1
2 2 1
( , ) ( , )
{ } 0 { } 0
t
t t
t t t
t
f x c f x c
E E
x x x

+ +
+
+

= + = + =

L
7
than choosing {c,c,c,...} all at once at t=0 (open loop policy).
The dynamic constraint x
t+1
=f(x
t
, c
t
)+
t+1
and the TVC lim
t
t
E
t
(
t
x
t
)=0 belong to the set of FOCs.
Because x
t
is in the information set when c
t
is to be determined, the expectations are E
t
and not E
0
.
Makrokonomik
D i P i i l l (1) Dynamic Programming: a simple example (1)
Consider an individual living for T periods. The intertemporal utility function is given by
T
The intertemporal budget constraint reads a
t+1
=(a
t
-c
t
)R
t
with R
t
=1+r
t
.
( ,..., ) ( )
T
t
T t
t
U c c u c
=
=
1
0
The optimal consumption plan {c, c,..., c
T
} can be determined by backward induction. At T-1 the
individual has wealth a
T-1
and the maximum utility may be described as follows
[ ]
( ) : max{ ( ) ( ) } (*)
T
T T
T T T T T T
c
c a
V a u c u a c R

=
= +
1
1 1 1
0
1 1 1
Th fi t d diti f th bl th RHS i
Value function V
T-1
(a
T-1
) shows an indirect utility, i.e. the maximum
attainable utility given wealth a
T-1
.
The first-order condition for the problem on the RHS is
[ ]
( ) ( ) ( )
T T T T T T T T
u c u a c R R c c a

= =
1 1 1 1 1 1 1 1
( )
1
Example: ( ) ln
1 1 1
t t
a c
u c c
=
=
8
Substituting c
T-1
=c
T-1
(a
T-1
) back into equ. (*) gives V
T-1
=V
T-1
(a
T-1
).
1 1
1 1 1 1
1 1
1
1
1
1
1 1
2
T T
T T T T
T T
a c
c a c
c a
c

=
+

= =
=
+
+
+
Makrokonomik
D i P i i l l (2) Dynamic Programming: a simple example (2)
At T-2 the consumer's maximization problem may be expressed as
This is just like equ (*) but with "second-period" utility replaced by indirect utility V (a )
[ ]
( ) max{ ( ) ( ) }
T
T
T T T T T T T
c
a
V a u c V a c R

= +
2
1
2 2 2 1 2 2 2
This is just like equ. ( ) but with second-period utility replaced by indirect utility V
T-1
(a
T-1
).
This equation is often referred to as the Bellman equation.
The first-order condition for the problem on the RHS is
[ ]
( ) ( ) ( )
T T T T T T T T T
u c V a c R R c c a

= =
2 1 2 2 2 2 2 2 2
Substituting c
T-2
=c
T-2
(a
T-2
) back into the Bellman equation gives V
T-2
=V
T-2
(a
T-2
).
Hence, the entire sequence V
t
=V
t
(a
t
) for all t{0,...,T} can be traced out.
Once we know V
t
=V
t
(a
t
), optimal c
t
for all t{0,...,T} follows from the first-order conditions
( ) ( )
t t t t
u c V a R
+ +

=
1 1
9
Finally, the time path a
t
for all t{0,...,T} results from a
t+1
=(a
t
-c
t
)R
t
with a given.
Makrokonomik
D i P i i l l (2 ) Dynamic Programming: a simple example (2a)
Consider an agent with u(c)=ln(c) living for 10 periods who is endowed with initial wealth a=1. The subsequent figure
displays the time paths for wealth (a
t
and c
t
) assuming different constellations with regard to R and .
10
Makrokonomik
D i P i l t t t (1) Dynamic Programming: a more general treatment (1)
The setup
d f l l d h i l lf b ( ) Consider an infinitely-lived, representative agent with intertemporal welfare given by (0<<1)
( )
t
t
U u c
The agent is assumed to solve the following problem

t =
{ }
max s.t. ( , ),
t t t t t
c
U x f x c x
+
=
1
0
Notice that the state variable x
t
cannot be controlled directly but is under indirect control by choosing c
t
.
An optimal program is a sequence {c
t
, c
t+1
,...} which solves the above stated problem. The value of
an optimal program is denoted by
{ }
( ) max s.t. ( , )
t t t t t
c
V x U x f x c
+
= =
1
11
The value function V(x
t
) shows the maximum level of intertemporal welfare U
t
given the amount of x
t
.
Makrokonomik
Step #1: Bellman equation and first-order conditions
Since the objective function U is additively separable it may be written as U =u(c )+U and Since the objective function U
t
is additively separable, it may be written as U
t
=u(c
t
)+U
t+1
and
hence the maximization problem may be stated as
( ) max{ ( ) ( )}
t t t
V x u c V x
+
= +
1
This is the Bellman equation. The problem with potentially infinitely many control variables was broken down into
many problems with one control variable.
( ) { ( ) ( )}
t
t t t
c

+1
Notice that, compared to U
t
=u(c
t
)+U
t+1
, U
t+1
is replaced by V(x
t
). This says that we assume that the optimal program
for tomorrow is solved and we worry about the maximization problem for today only.
The first-order condition for the problem on the RHS of Bellman equation, noting x
t+1
=f(x
t
,c
t
), is
The benefit of an increase in c
t
consist in higher utility today, which is reflected here by marginal utility u(c
t
).
( ) ( ** ) ( )
t c t
V x f u c
+
+ =

1
0
The cost consists in lower overall utility - the value function V(.) - tomorrow.
The reduction in overall utility amounts to the change in x
t+1
, i.e. the derivative f
c
, times the marginal value of x
t+1
, i.e. V(x
t+1
).
As the disadvantage arises only tomorrow, this is discounted at the rate .
12
Notice that this FOC, together with x
t+1
=f(x
t
,c
t
), implicitly defines c
t
= c
t
(x
t
). As we know very little about the properties
of V(.) at this stage, however, we need to go through two further steps in order to eliminate V(.) from this FOC and
obtain a condition that uses only functions of which properties like signs of first and second derivatives are known.
Makrokonomik
Step #2: Evolution of the costate variable V(x
t
)
At first we set up the maximized Bellman equation This is obtained by replacing the control variable in At first we set up the maximized Bellman equation. This is obtained by replacing the control variable in
the Bellman equation by the optimal level of the control variable according to the FOC, i.e. c
t
=c
t
(x
t
)
( ) ( ( )) [ ( , ( ))]
t t t t t t
V x u c x V f x c x = +

The derivative of V(x
t
) w.r.t. x
t
gives
t
x
+1
( ) ( )
( ) ( ( )) [ ( ( ))]
t t t t
dc x dc x
V x u c x V f x c x f f
(
= + +
(
Inserting the first-order condition u(c
t
)=-V(x
t+1
)f
c
gives
( ) ( ( )) [ ( , ( ))]
t
t t t t t t x c
t t
V x u c x V f x c x f f
dx dx
= + +
(

( ) ( )
( ) ( ) [ ( , ( ))] ( ( ***) ) ( )
t t t
t t t t
t t c t t t x c t t x
t t
dc x dc x
V x V x f V f x c x f f V x V x f
dx dx

+ +
(
= + + =
(

1 1
This equation is a difference equation in the costate variable V(x
t
). The costate variable is also called the shadow price
of the state variable x
t
.
V(x ) says how much an additional unit of the x is valued: As V(x ) gives the value of optimal behavior between t and
13
V (x
t
) says how much an additional unit of the x
t
is valued: As V(x
t
) gives the value of optimal behavior between t and
the end of the planning horizon, V(x
t
) says by how much V(x
t
) changes when x
t
is changed marginally.
Equ. (***) describes how the shadow price of the state variable changes over time when the agent behaves optimally.
Makrokonomik
Step #3: Euler equation
i (**) i
( ) ( ) (**
(
)
** ) * ( ) ( )
t
t t c
t t x
u c V x f
V x V x f
+
+
+ =
=
1
1
0
Noting (**) one may write
;
( ) ( )
( ) ( )
t t
t
c
t
c
u c
V x
f
u c
V x
f
= =

1
1
Inserting into equ. (***) gives
( )
(
( )
( ) )
t t
x x
t
c
t
t
t
c
u c u c
V x f f
f
V x
f

+
= =

1
1
This is the famous Euler equation
( )
( )
t
t
x
t
u c
u
f
c

1
This is the famous Euler equation.
It represents a difference equation (DE) in marginal utility, which can be transformed into a DE in c
t
.
If, for instance, the utility function is of the usual CIES shape, then the Euler equation implies
( )
t
t
x
t
c
f
c

=
1
1
14
The dynamic evolution is then determined by this equation together with x
t+1
=f(x
t
,c
t
)
Boundary conditions read: x=given and a terminal condition c
=c.
Makrokonomik
C t l Th M i P i i l (1) Control Theory: Maximum Principle (1)
The dynamic problem under consideration can be stated as follows
{ ( )}
max ( , , ) ( , )
T
T
u t
t
J I x u t dt F x T = +
}
0
. . ( , , )
( ) ; ( ) ; ( )
T
s t x f x u t
x t x x T x u t U
=
= =
0 0
I(.), F(.), and f(.) are continuously differentiable functions. x is an n-dimensional vector of state variables
and u is an r-dimensional vector of control variables (nr).
The control trajectory u(t) t[t,T] must belong to a given control set U, and must be a piecewise
continuous function of time.
The maximum principle can be considered as an extension of the method of Lagrange multipliers
to dynamic optimization problems. To keep the exposition simple, it is assumed that the control
trajectory u(t) is unconstrained.
15
Makrokonomik
Recall that the Method of Lagrange Multipliers requires to first introduce Lagrange multipliers and then
to set up the Lagrangian function. Subsequently, the saddle point of this function (maximizing w.r.t. to set up the Lagrangian function. Subsequently, the saddle point of this function (maximizing w.r.t.
choice variables and minimizing w.r.t. the Lagrange multipliers) is determined.
At first, we set up a (row) vector of so-called costate variables, which are the dynamic equivalent of the
Lagrange multipliers
Next we set up the Lagrangian function
( ) ( ( ), ( ),..., ( ))
n
t t t t =
1 2
Next we set up the Lagrangian function
[ ] [ ] { } ( )
( , , ) ( , ) ( , , ) ( , , ) ( , , ) , (*)
T T T
T T
L I x u t dt F x T f x u t x dt I x u t f x u t x dt F x T = + + = + +
} } }

Lagrange multipliers times constraints
objective function
t t t

0 0 0
By analogy to the static case, a saddle point of the Lagrangian would yield the solution. Here, however,
the saddle point is in the space of functions, where (u*(t),*(t)) represent a saddle point if
( ( (
16
( ), ( ) ( ), ( ) ( ), ( ) L u t t L u t t L u t t

( ( (

Makrokonomik
We now consider the necessary conditions for a saddle point of (*). A change in the costate variable trajectory
from (t) to (t)+(t), where (t) is any continuous function of time, would change the Lagrangian to ( ) ( ) ( ) ( ) y g g g
H one set of first order necessar conditions lti f L 0 d f ll
( , , )
T
t
L f x u t x dt

( =
}
0
Hence, one set of first-order necessary conditions, resulting from L=0, reads as follows
( , , ) x f x u t

=
To develop the remaining necessary first-order conditions, we first rewrite (*) as follows
{ } ( ) ( , , ) ( , , ) ,
T T
T
L I x u t f x u t dt xdt F x T = + +
} }

( ) ( ) ( ) ( ) | ( ) ( )
(i t ti b t )
b b
b
a
a a
f x g x dx f x g x f x g x dx

=
} }
The preceding equation may be expressed as
t t
} }
0 0
(integration by parts)
|
T T
T
t
t t
xdt x xdt =
} } 0
0 0
(
{ } ( )
( , , ) ( , , ) | ,
T T
T
t T
t t
T
L I x u t f x u t dt x xdt F x T
(
= + +
(
(

} } 0
0 0
17
{ } [ ] ( ) ( , , ) ( , , ) ( ) ( ) ( ) ( ) , (**)
T
T
t
I x u t f x u t x dt T x T t x t F x T = + + +
}
0
0 0

Makrokonomik
We now define the following function, labeled the Hamiltonian function
Equ. (**) may then be written as
( , , , ) : ( , , ) ( , , ) H x u t I x u t f x u t = +
{ } [ ] ( )
( , , , ) ( ) ( ) ( ) ( ) ,
T
T
t
L H x u t x dt T x T t x t F x T = + +
}
0
0 0
Now consider the effect of a change in the control trajectory from u(t) to u(t)+u(t), associated by a
corresponding change in the state trajectory from x(t) to x(t)+x(t), which yields
( )
T
T
T t
H H F
L u x dt T x
u x x

(
| |
= + + +
`
|
(

\ .
)

}
0
For a maximum it is necessary that L=0 implying that

[ ]
; ; ( )
H H F
T t t T

0

18
[ ]
; ; ( ) ,
T
T t t T
u x x
= = =

0
0
Makrokonomik
Some comments
T h k f i ffi i diti t b id d I thi t t th f ll i iti To check for a maximum, sufficiency conditions must be considered. In this context the following propositions
are helpful:
Provided that the Hamiltonian is jointly concave in the control and the state variable (Mangasarian
sufficiency conditions) or
Provided that the maximized Hamiltonian is concave in the state variable (Arrow sufficiency conditions),
the necessary conditions are also sufficient (Kamien and Schwartz, 1981, part II, sections 3 and 15).
Interpretation of the costate variables : J*/x(t)=*(t). That is, the (initial) costate variables give the
change in the optimal value of the objective functional due to changes in the corresponding initial state
variables. This is analogous to the interpretation of the static Langrange multipliers.
The (current-value) Hamiltonian can be viewed as net national product in utility terms (Solow, 2000, p. 127).
This can be seen more clearly by writing the Hamiltonian as H=u(C)+I(C) where I denotes investment and This can be seen more clearly by writing the Hamiltonian as H=u(C)+ I(C), where I denotes investment and
I(C) indicates that investment depends negatively on consumption. Note that the shadow price has the
dimension "utility per unit of numeraire good. Hence, maximization of H w.r.t. C requires the following first-
order necessary condition to hold: H/C=0.
The costate equation, =-H/K, can be viewed as arbitrage condition or Fisher equation (Solow, 2001, p.
160). For this purpose rewrite the costate equation as: +H/K=. LHS shows the (marginal) benefit that can
be obtained by devoting one unit of output for investment. This comprises an increase in "income in utility
terms" H/K, accounting for a change in the value of capital. RHS gives the opportunity costs of devoting one
19
te s / , accou t g o a c a ge t e a ue o cap ta . S g es t e oppo tu ty costs o de ot g o e
unit of output to investment rather than consumption. Notice that =-H/K implies (0)=
0
H/Kexp(-
t)dt (provided that appropriate boundary conditions hold).
Makrokonomik
Th M th d f L M lti li (5 ) fi i h d The Method of Lagrange Multipliers (5a): unfinished
Consider the investment problem of a firm
The FOC read
( ) ( )
1
{ , }
0
1
max 1
1 t t
t
t
t t t t t
I L
t
t
I
A K L w L I
r K
=

(
| |

| |
+ (
`
| |
+
\ .
\ . (

)
The FOC read

( ) ( )
( )
t t t
t
A K L w
L

= =
L
1 0
( )
1
1
1 1
1 0
1
t t t
t t t t
t t t t
I I q
I q I K
I K K K

(
| |
| | | |
= + + = = (
|
| |
+
( \ . \ .
\ .

L
\ .

20

Dynamic Optimization

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Dynamic Optimization

Hochgeladen von

Copyright:

Verfügbare Formate

Institut fr Theoretische Volkswirtschaftslehre

The agent is assumed to solve the following problem

Institut fr Theoretische Volkswirtschaftslehre

For a maximum it is necessary that L=0 implying that

The FOC read

Das könnte Ihnen auch gefallen