### Optimal Control Theory [chapter]

Emanuel Todorov
2006 Bayesian Brain
Optimal control theory is a mature mathematical discipline with numerous applications in both science and engineering. It is emerging as the computational framework of choice for studying the neural control of movement, in much the same way that probabilistic inference is emerging as the computational framework of choice for studying sensory information processing. Despite the growing popularity of optimal control models, however, the elaborate mathematical machinery behind them is rarely
more » ... hem is rarely exposed and the big picture is hard to grasp without reading a few technical books on the subject. While this chapter cannot replace such books, it aims to provide a self-contained mathematical introduction to optimal control theory that is su¢ ciently broad and yet su¢ ciently detailed when it comes to key concepts. The text is not tailored to the ...eld of motor control (apart from the last section, and the overall emphasis on systems with continuous state) so it will hopefully be of interest to a wider audience. Of special interest in the context of this book is the material on the duality of optimal control and probabilistic inference; such duality suggests that neural information processing in sensory and motor areas may be more similar than currently thought. The chapter is organized in the following sections: 1. Dynamic programming, Bellman equations, optimal value functions, value and policy iteration, shortest paths, Markov decision processes. 2. Hamilton-Jacobi-Bellman equations, approximation methods, ...nite and in...nite horizon formulations, basics of stochastic calculus. 3. Pontryagin's maximum principle, ODE and gradient descent methods, relationship to classical mechanics. 4. Linear-quadratic-Gaussian control, Riccati equations, iterative linear approximations to nonlinear problems. 5. Optimal recursive estimation, Kalman ...lter, Zakai equation. 6. Duality of optimal control and optimal estimation (including new results). 7. Optimality models in motor control, promising research directions. 1