Efficient Optimistic Exploration in Linear-Quadratic Regulators via Lagrangian Relaxation [article]

Marc Abeille, Alessandro Lazaric
<span title="2020-07-13">2020</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
We study the exploration-exploitation dilemma in the linear quadratic regulator (LQR) setting. Inspired by the extended value iteration algorithm used in optimistic algorithms for finite MDPs, we propose to relax the optimistic optimization of and cast it into a constrained extended LQR problem, where an additional control variable implicitly selects the system dynamics within a confidence interval. We then move to the corresponding Lagrangian formulation for which we prove strong duality. As a
more &raquo; ... result, we show that an ϵ-optimistic controller can be computed efficiently by solving at most O(log(1/ϵ)) Riccati equations. Finally, we prove that relaxing the original problem does not impact the learning performance, thus recovering the Õ(√(T)) regret of . To the best of our knowledge, this is the first computationally efficient confidence-based algorithm for LQR with worst-case optimal regret guarantees.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2007.06482v1">arXiv:2007.06482v1</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/kurm6k3x2feu5dtmwccgiwk3by">fatcat:kurm6k3x2feu5dtmwccgiwk3by</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20200925095433/https://arxiv.org/pdf/2007.06482v1.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/17/1a/171a46ea0c20631327f0347fecb8224c874d6c4d.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2007.06482v1" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>