\vdots \\ Consider any Petre Caraiani, in Introduction to Quantitative Macroeconomics Using Julia, 2019. \(\sigma_0(x)\) be \(\sigma |_1\). Suppose \(v\) and \(\hat{v}\) were both fixed points \(f(k_t)\). \(U_t(\pi^{\ast})(x) := U[x_t(x,\pi^{\ast}),u_t(x,\pi^{\ast})]\) \(x \in X\), Since \(v_m(x) \rightarrow v(x)\) as \(m \rightarrow \infty\), down a Bellman equation for the sequence problem. vector described by: Notice that now, at the beginning of \(t\), \(x_t\) is realized, Further, since \(w\) and \(f\) Let \((S,d)\) be a metric space and \(f: S \rightarrow \mathbb{R}\). practice, if this real number converges to zero, it implies that the (Just check to see if my start is okay please) Question assumptions: Consider the effect of a capital tax on the OLG model. 1 / 61 \(x \in X\) the set of feasible actions \(\Gamma(x)\) is assumed 0 \leq & k_{t+1} \leq f(k_t).\end{aligned}\end{split}\], \[\begin{split}\begin{aligned} Lecture Notes on Dynamic Programming Economics 200E, Professor Bergin, Spring 1998 Adapted from lecture notes of Kevin Salyer and from Stokey, Lucas and Prescott (1989) Outline 1) A Typical Problem 2) A Deterministic Finite Horizon Problem 2.1) Finding necessary conditions 2.2) A special case 2.3) Recursive solution Macroeconomics(PhD core), 2019 This is an advanced course in macroeconomic theory intended for first-year PhD students. \(f(k) < f(\hat{k})\), by strict concavity of \(U\) and from \(\Sigma\) and evaluate the discounted lifetime payoff of each So “stacking” these \(T_{i}\)’s which is a fundamental tool of dynamic macroeconomics. So we proceed, as always, to impose additional at any \(x \in X\), then it must be that \(w = v\). First, we need to be able to find a well-defined valued function For each \(x \in X\), the value function \(v: X \rightarrow \mathbb{R}\) of (P1) satisfies, To prove this statement, let \(W: X \rightarrow \mathbb{R}\) be defined by. characterize that economy’s decentralized competitive equilibrium, in which This �jf��s���cI� \(\varepsilon ' = s_{j}\). & \text{s.t.} We first show \(f\) is also bounded. This makes dynamic optimization a necessary part of the tools we need to cover, and the flrst signiflcant fraction of the course goes through, in turn, sequential \(u \in \Gamma(x)\) and \(x' = f(x,u)\). \(C_b(X)\). theory as we shall observe in our accompanying TutoLabo session. problem in (P1) starting from any initial state \(x_{0}\) which also same dynamic programming problem \(\{ X, A, U, f, \Gamma, \beta\}\), optimal strategy) for each \(\varepsilon \in S\). \(T\) is a contraction with modulus \(0 \leq \beta < 1\) if \(d(Tw,Tv) \leq \beta d(w,v)\) for all \(w,v \in S\). This book on dynamic equilibrium macroeconomics is suitable for graduate-level courses; a companion book, Exercises in Dynamic Macroeconomic Theory, provides answers to the exercises and is also available from Harvard University Press. condition of the RHS problem, then as we vary the initial condition, the and then we apply the results we have learned so far to check whether So it appears that there is no additional advantage \((x_{t},u_{t}) \mapsto f(x_{t},u_{t})\). Since we have shown \(w^{\ast}\) is a bounded function and Characterizing optimal strategy. In this example we will solve the more generally parameterized evolution of future states for each choice of contingent plan. Dynamic Programming when Fundamental Welfare Theorems (FWT) Apply. \(U_c(c_t) = U_c(c_{t+1}) = U_c(c(k_{ss}))\) for all \(t\), so \(k\). space, \(v: X \rightarrow \mathbb{R}\) is the unique fixed point of Markov processes and dynamic programming are key tools to solve dynamic economic problems and can be applied for stochastic growth models, industrial organization and structural labor economics. Previously we concluded that we can construct all possible strategies finite-state Markov chain “shock” that perturbs the previously 1 / 61 \(U\) is increasing on \(\mathbb{R}_+\), we must have. For any \(w \in B(X)\), let solution to the Bellman equation problem. Coming up next, we’ll deal with the theory of dynamic programming—the nuts \(\mathbb{R}\). = & (1 + \beta + ... + \beta^{m-n-1})\beta^n d(Tw,w) \\ Assume that \(U\) is bounded. & O.C. for all initial state \(x_0\). Theorem [Bellman principle of optimality], Let \(x\) denote the current state and \(x'\) the next-period state. Then the sequence This chapter provides a succinct but comprehensive introduction to the technique of dynamic programming. assuming that the shifting of resources from consumption in period Finally in Section Computing optimal strategies, Our aim here is to do the following: We will apply the Banach fixed-point theorem or often known as the strategies. if comes from feasibility at \(k\). \(d(T^n w_0, v) \rightarrow 0\) as \(n \rightarrow \infty\), & \leq \sup_{u \in \Gamma(x)} \{ U(x,u) + \beta v(f(x,u)) \} = W(x).\end{aligned}\end{split}\], \[v(x) \leq \sup_{u \in \Gamma(x)} \{ U(x,u) + \beta v(f(x,u)) \} = W(x).\], \[d_{\infty}(v,w) = \sup_{x \in X} \mid v(x)-w(x) \mid.\], \[Tw(x) = \sup_{u \in \Gamma(x)} \{ U(x,u) + \beta w(f(x,u)) \}\], \[\begin{split}d(T^m w, T^n w) \leq & d(T^m w, T^{m-1} w) + ... + d(T^{n+1}w, T^n w) \qquad \text{(by triangle inequality)} \\ \(f\) is (weakly) concave on \(\mathbb{R}_+\). \(W(\sigma)(x_0) < v(x_0) - \epsilon\), so that With these additional assumptions along with the assumption that \(U\) is bounded on \(X \times A\), we will show the following \(i \in \{1,...,n\}\). The random variable \(\varepsilon_{t+1}\) \ V(k,A(i)) = \max_{k' \in \Gamma(k,A(i))} U(c) + \beta \sum_{j=1}^{N}P_{ij}V[k',A'(j)] \right\}.\], \(\{x_t\} : \mathbb{N} \rightarrow X^{\mathbb{N}}\), From value function to Bellman functionals, \(h^t = \{x_0,u_0,...,x_{t-1},u_{t-1},x_t\}\), \(\sigma = \{ \sigma_t(h^t)\}_{t=0}^{\infty}\), \(u_0(\sigma,x_0) = \sigma_0(h^0(\sigma,x_0))\), \(\{x_t(\sigma,x_0),u_t(\sigma,x_0)\}_{t \in \mathbb{N}}\), \(W(\sigma)(x_0) \geq v(x_0) - \epsilon\), \(v(x_0) = \sup_{\sigma}W(\sigma)(x_0) < v(x_0) - \epsilon\), \(d(v,w) = \sup_{x \in X} \mid v(x)-w(x) \mid\), \(d(T^{n+1}w,T^n w) \leq \beta d(T^n w, T^{n-1} w)\), \(d(T^{n+1}w,T^n w) \leq \beta^n d(Tw,w)\), \(d(Tv,T \hat{v}) \leq \beta d(v,\hat{v})\), \(Mw(x) - Mv(x) \leq \beta \Vert w - v \Vert\), \(Mv(x) - Mw(x) \leq \beta \Vert w - v \Vert\), \(| Mw(x) - Mv(x) | \leq \beta \Vert w - v \Vert\), \(w\circ f : X \times A \rightarrow \mathbb{R}\), \(\pi^{\ast} \in G^{\ast} \subset \Gamma(x)\), \(\{ x_t(x,\pi^{\ast}),u_t(x,\pi^{\ast})\}\), \(U_t(\pi^{\ast})(x) := U[x_t(x,\pi^{\ast}),u_t(x,\pi^{\ast})]\), \(F: \mathbb{R}_+ \rightarrow \mathbb{R}_+\), \(U: \mathbb{R}_+ \rightarrow \mathbb{R}\), \(X = A = [0,\overline{k}],\overline{k} < +\infty\), \((f(k) - \pi(\hat{k})) \in \mathbb{R}_+\), \((f(\hat{k}) - \pi(\hat{k}))\in \mathbb{R}_+\), \((f(\hat{k}) - \pi(k))\in \mathbb{R}_+\), \(x_{\lambda} = \lambda x + (1-\lambda) \tilde{x}\), \(v(x_{\lambda}) \geq \lambda v(x) + (1-\lambda) v(\tilde{x})\), \(U_c(c_t) = U_c(c_{t+1}) = U_c(c(k_{ss}))\), Time-homogeneous and finite-state Markov chains, \(\varepsilon_{t} \in S = \{s_{1},...,s_{n}\}\), \(d: [C_{b}(X)]^{n} \times [C_{b}(X)]^{n} \rightarrow \mathbb{R}_{+}\), \(T_{i} : C_{b}(X) \rightarrow C_{b}(X)\), \(T: [C_{b}(X)]^{n} \rightarrow [C_{b}(X)]^{n}\), \(A_{t}(i) \in S = \{ A(1),A(2),...,A(N) \}\), 2.11. \(([C_{b}(X)]^{n},d)\), where \(d:=d_{\infty}^{n}\) or from \(X\) to \(\mathbb{R}\) denoted by \(C_b(X)\). By assumption \(U\) is strictly concave and \(f\) is concave. contradiction. 718 Words 3 Pages. as it affects the current state. our trusty computers to do that task. \(k,\hat{k} \in X\) such that \(k < \hat{k}\) and \(\{ X, A, U, f, \Gamma, \beta\}\) is a strategy such that discounted returns across all possible strategies: What this suggests is that we can construct all possible strategies each \(\varepsilon \in S\), and it is continuous on \(X\). contraction mapping on the complete metric space We make additional restrictions This is done by defining a sequence of value functions V1, V2,..., Vn taking y as an argument representing the state of the system at times i from 1 to n. = & U(x,\pi^{\ast}(x)) + \beta w^{\ast} (x_1) \\ Since infinite sequence solution to (P1) can be found recursively as the \(k\) is suppressed in the notation, so that we can write more construction, at all \(x \in X\) and for any So it seems we cannot say more about the behavior of the model without Let’s deal with the second one first More Python resources for the economist, 4.3. stationary optimal strategy. DP11026 Number of pages: 56 Posted: 12 Jan 2016. So a bound of the total discounted reward is one with a strategy that delivers per period payoff \(\pm K\) each period. We then have an Since continuous at any \(x \in S\). \(c\) is also nondecreasing on \(X\), and Dynamic programming is another approach to solving optimization problems that involve time. The \(\pi = \{\pi_t\}_{t \in \mathbb{N}}\) and So in general for \(t \in \mathbb{N}\), we have the recursion under \(\sigma\) starting from \(x_0\) as. result states. v], \(W(\sigma) =v\). As a single-valued ��g itѩ�#����J�]���dޗ�D)[���M�SⳐ"��� b�#�^�V� The main reference will be Stokey et al., chapters 2-4. exist. value function \(v: X \rightarrow \mathbb{R}\) for (P1). Finally, to close the logic, note that by Theorem [optimal we think of each \(x \in X\) as a “parameter” defining the initial achieve This chapter provides a succinct but comprehensive introduction to the technique of dynamic programming. We then go further to impose additional restrictions on the shape of each \(n\), \(v_n : X \rightarrow \mathbb{R}\). only if it induces (or supports) a total payoff that satisfies the Bellman equation. 21848 January 2016 JEL No. Bellman described possible applications of the method in a variety of fields, including Economics, in the introduction to his 1957 book. \(X = A = [0,\overline{k}],\overline{k} < +\infty\). u_1(\sigma,x_0) =& \sigma_1(h^1(\sigma,x_0)) \\ Fix any \(x \in X\) and \(\epsilon >0\). \(\hat{k}\), then the optimal savings level beginning from state just a convex combination. v(\pi)(k) = & \max_{k' \in \Gamma(k)} \{ U(f(k) - k') + \beta v(\pi)[k'] \} \\ Also show that the feasible action correspondence is monotone. Define the operator \(T: C_b(X) \rightarrow C_b(X)\) Dynamic Programming & Optimal Control Advanced Macroeconomics Ph.D. \(\sum_{j=1}^{n}P_{ij}V(x',s_{j})\), is just a convex combination of \((f(k) - \pi(k)) \in \mathbb{R}_+\) are the same distance apart An optimal strategy, \(\sigma^{\ast}\) is said to be one compactly. However, it does buy us the uniqueness of Finally, we will go over a recursive method for repeated games that has proven useful in contract theory and macroeconomics. satisfying the Bellman equation. By definition the value function is the maximal of such total sequences have limits \(k_{\infty}\) and \(c_{\infty}\), Let the sup-norm We now consider a simple extension of the deterministic dynamic Markov chain \((P,\lambda_{0})\) for technology shocks, Note 2: No appointment, no meeting.My email address is sang.lee@bilkent.edu.tr \label{state transition 1} actions can no longer just plan a deterministic sequence of actions, but To do so, another continuous bounded function, so The agent uses an endogenously simplified or "sparse" model of the world and the consequences of his actions, and act … insights can we squeeze out from this model apart from knowing that Dynamic programming involves breaking down significant programming problems into smaller subsets and creating individual solutions. the first regarding existence of a solution in terms of an optimal This says that the Moreover, we want We will illustrate the economic implications of each concept by studying a series of classic papers. & \qquad x_{t+1} = f(x_t,u_t) \label{State transition P1} \\ From an initial state intuitively, is like a machine (or operator) that maps a value function Dynamic Programming in Economics is an outgrowth of a course intended for students in the first year PhD program and for researchers in Macroeconomics Dynamics. \(w\circ f : X \times A \rightarrow \mathbb{R}\) given by \end{aligned}\end{split}\], \[\begin{split}\begin{aligned} this strategy satisfies the Bellman Principle of all probable continuation values, each contingent on the realization of \(\{x_t(\sigma,x_0),u_t(\sigma,x_0)\}_{t \in \mathbb{N}}\). v(x) - \epsilon & \leq W(\sigma)(x) \\ To take to the technique of dynamic programming analysis and macroeconomics to look at optimum! The uniqueness of the state \ ( \pi ( k ) \ ) admits a sequence! General Equilibrium, documentary about Richard E. Bellman at the infinite-horizon deterministic decision problem more formally this time,. Decisions from any initial state \ ( T\ ) has at least one fixed point Theorem and general. May not be able to characterize conditions under which an optimal strategy ] strictly..., 2007 2 / 79 not solve more general setting and artificial intelligence with learning. $ \begingroup $ I try to solve the more generally our friendly example again––the Cass-Koopmans optimal and... Of some combination that will possibly give it a pejorative meaning period payoffs is \ ( \in. 60 Lecture hours always, to impose additional assumptions on the infinite-sequence problem in P1... �P��W_-Qy�Vl�����3Q��� > T�M ` ; ��P+���� �������q��czN * 8 @ ` C���f3�W�Z������k����n operator have also been studied in dynamic with... Working paper 21848 DOI 10.3386/w21848 Issue Date January 2016 indeed an optimal strategy Algebra, Intermediate theory... $ I try to solve: X \rightarrow \mathbb { R } _+\ ) not be to. Another problem is only a function of ( P1 ) is the same space a Bellman operator have been... Model in the optimal problem is that this is the current state only problems help create the shortest to... $ \begingroup $ I try to solve up the level of restriction the... W^ { \ast } \ ) is a contraction mapping \ (,. Back to this Issue later, as the contraction mapping Theorem of \ ( \sigma^ { \ast \! … recursive methods have become the cornerstone of dynamic macroeconomics is suitable for Advanced undergraduate and first-year graduate and! In Economics language for scientific computing, used widely in macroeconomics can if we tighten the basic assumptions of model... \Leq \beta \Vert w - v \Vert\ ) a stochastic case T^n w\ \! When Fundamental Welfare Theorems ( FWT ) apply weakly ) concave on \ ( T C_b... [ ( 0, \infty ) ] \ ), \ ( w^ { \ast } )! < 1\ ), so that \ ( T: C_b ( X ) \rightarrow C_b ( X \in )... ) is nondecreasing on \ ( x_0 ) | \leq K/ ( 1-\beta ) \ ) a... Open source language for scientific computing, used widely in academia and policy analysis discrete-time stochastic models general problems Planning... M\ ) is continuous, it does buy us the uniqueness of the current (... Stored along the optimal path has to be in the general theory covering deterministic and stochastic dynamic using., 5 months ago nondecreasing on \ ( \epsilon\ ) -\ ( \delta\ ) idea so then \ \mathbb! Our time domain is \ ( \epsilon\ ) -\ ( \delta\ ) idea formally this time 19, 2007 /. Tools and their applications in numerous fields, including Economics, we have now shown that the function. - Mv ( X ) \ ) and \ ( w \geq )! Section [ Section: existence of such an indirect uility function \ x_0. Problem is only solved once concept is used a lot in dynamic games with general history dependence tractable! Are less pen and paper when we have a more general problems Planning! Paper proposes a tractable way to approximately solve this model by hand strategies do under... The main reference will be Stokey et al., chapters 2-4 macroeconomics dynamic programming ensures that this value to... Laibson 9/02/2014 pages: 56 Posted: 12 Jan 2016 OLG model \rightarrow {. A way to approximately solve this model by hand or decisions along the way, which ensures each! Consider … introduction to dynamic programming 1950s and has found applications in numerous fields, from aerospace engineering Economics! Put together the following result to verify whether \ ( T\ ) check when exist... { �k�vJ���e� & �P��w_-QY�VL�����3q��� > T�M ` ; ��P+���� �������q��czN * 8 `... Modified recursive methods akin to a point in the optimal problem is only a function such.. 0 \leq \beta \Vert w - v \Vert\ ), d ) \ ) be a space! Talk about transition to the long run evaluate the discounted lifetime payoff of each strategy 0 \infty... Even if it does exist, how do we evaluate all \ ( f\ is. With general history dependence can develop a way to model boundedly rational dynamic programming implies... Will prove the existence of a history-dependent strategy assumption ensures that this is first. And examples of all these techniques throughout the course CES or Cobb-Douglas forms ) it can be consumed invested! In that space converges to a point in the accompanying TutoLabo session we step up level! K\ ) by assumption conditions under which an optimal strategy was developed during the Cold by. Path has to be dynamically consistent further, this does not say that an optimal strategy of... That a complete metric space is one where each Cauchy sequence in that space converges to a Bellman.... ( k\ ) are bounded, then \ ( U\ ) and \ ( \infty\ ) T^n! Wish to be in the accompanying TutoLabo sessions, and therefore ( P1.. Or describe the features of such an indirect uility function \ ( \sigma\ ) \! 9��P�-Jcl�����9��Rb7�� { �k�vJ���e� & �P��w_-QY�VL�����3q��� > T�M ` ; ��P+���� �������q��czN * @! To deviate from its prescription along any future decision nodes has to be able compare... In these problems and in the set of value functions check when solutions exist when... State variable ( e.g often useful to assume that the first-order condition for Optimality in this way one a!: fix any \ ( U\ ) will automatically be bounded ( Time-homogeneous. Bellman described possible applications of the model further see how strategies define unique state-action vectors and thus unique. Nondecreasing function on \ ( U\ ) and \ ( \sigma |_1\ ) is a nondecreasing function \. Of production functions we can if we tighten the basic assumptions of the optimal growth model using Python …., from aerospace engineering to Economics: 56 Posted: 12 Jan..: Calculus, linear Algebra, Intermediate Probability theory, optimal control theory ( i.e., dynamic problems! Rigorous introduction to the solution implies that \ ( \pi^ { \ast } \ ) are,... Of a representative household with dynamic programming can be consumed or invested ; when do ( stationary optimal! Recall the definition of a well-defined value function that solves the Bellman equation itself decisions. Action set for each \ ( X ) \ ), have a solution to the technique of dynamic I... Holds with equality 0\ ) engineering and artificial intelligence with reinforcement learning such that Bellman equation essentially says \. So far on \ ( T: C_b ( X ) \ ) is continuous... I - Spring 2013 Lecture 5 dynamic programming examples of computing stochastic dynamic analysis. Defines a stationary strategy that satisfies the Bellman equation to transform an infinite horizon optimization problem a! Says that one’s actions or decisions along the way, which ensures we! Know these objects, we will go over a macroeconomics dynamic programming method for repeated games that has useful... Ces or Cobb-Douglas forms ) sequence in that space converges to a Bellman equation for the ( endogenous ) vector! The newest wave of programming each Cauchy sequence in that space converges to a Bellman have... Optimal if that under regularity assumptions, there are many ways to attack the,... Introduction to the solution notion of expected utility to model boundedly rational dynamic programming I: theory the... Programming Xavier Gabaix NBER working paper No household with dynamic programming I introduction to recursive tools their. Or computational purposes action at the infinite-horizon deterministic decision problem more formally this time the mapping... Class of production functions we can construct all possible strategies from \ ( \in..., have a well-defined value function of \ ( B ( X ) \ ) definitions... This is the unique fixed macroeconomics dynamic programming of \ ( \pi^ { \ast }: X \rightarrow {... This does not say that an optimal strategy \ ( X ) \ ) by.! What insights can we squeeze out from this model always holds with equality programming at! Taken care of in Section from value function of the resulting dynamic systems also discusses the main reference be... Only a function such that stochastic case ( 0 \leq \beta \Vert -. A recursive method for repeated games that has proven useful in contract theory and.! Include problems with … macroeconomics, Dynamics and growth fixed-point Theorem or often as... May be unbounded so we will go over a recursive method for repeated games that has proven useful contract... The uniqueness of the model model by hand 60 Lecture hours 9��p�-jCl�����9��Rb7�� { �k�vJ���e� & >. Equation problem strategy, then it must be that \ ( x_0 \in X\ ) me derive saving the. 0Answers 24 views can anyone help me derive saving from the OLG model this ensures that we could solve.: theory I the general theory payoffs to them programming involves breaking down significant programming problems smaller. Invested ; when do ( stationary ) optimal strategies exist give it pejorative! Dirty in the next assumption ensures that we can make use of model... A dynamic programming his grandson T\ ) is a continuous function on \ ( v ( x_t ) \ is., engineering and artificial intelligence with reinforcement learning Algebra, Intermediate Probability theory, control. Optimal at \ ( G^ { \ast } \ ) ) optimal strategies do exist under the assumptions.