A semiparametric statistical approach to model-free policy evaluation

Tsuyoshi Ueno, Motoaki Kawanabe, Takeshi Mori, Shin-ichi Maeda, Shin Ishii
2008 Proceedings of the 25th international conference on Machine learning - ICML '08  
Reinforcement learning (RL) methods based on least-squares temporal difference (LSTD) have been developed recently and have shown good practical performance. However, the quality of their estimation has not been well elucidated. In this article, we discuss LSTDbased policy evaluation from the new viewpoint of semiparametric statistical inference. In fact, the estimator can be obtained from a particular estimating function which guarantees its convergence to the true value asymptotically,
more » ... mptotically, without specifying a model of the environment. Based on these observations, we 1) analyze the asymptotic variance of an LSTD-based estimator, 2) derive the optimal estimating function with the minimum asymptotic estimation variance, and 3) derive a suboptimal estimator to reduce the computational burden in obtaining the optimal estimating function.
doi:10.1145/1390156.1390291 dblp:conf/icml/UenoKMMI08 fatcat:tvrubmij7bd6nlnjurqb5ppie4