Learning Fast And Slow: Deviations From The Matching Law Can Reflect An Optimal Strategy Under Uncertainty [article]

Kiyohito Iigaya, Yashar Ahmadian, Leo Sugrue, Greg Corrado, Yonatan Loewenstein, William T. Newsome, Stefano Fusi
2017 bioRxiv   pre-print
Behavior which deviates from our normative expectations often appears irrational. A classic example concerns the question of how choice should be distributed among multiple alternatives. The so called matching law predicts that the fraction of choices made to any option should match the fraction of total rewards earned from the option. This choice strategy can maximize reward in a stationary reward schedule. Empirically, however, behavior often deviates from this ideal. While such deviations
more » ... e often been interpreted as reflecting noisy, sub-optimal, decision-making, here we instead suggest that they reflect a strategy which is adaptive in non-stationary and uncertain environments. We analyze the results of a dynamic foraging task. Animals exhibited significant deviations from matching, and animals turned out to be able to collect more rewards when deviation was larger. We show that this behavior can be understood if one considers that animals had incomplete information about the environment's dynamics. In particular, using computational models, we show that in such non-stationary environments, learning on both fast and slow timescales is beneficial. Learning on fast timescales means that an animal can react to sudden changes in the environment, though this inevitably introduces large fluctuations variance in value estimates. Concurrently, learning on slow timescales reduces the amplitude of these fluctuations at the price of introducing a bias that causes systematic deviations. We confirm this prediction in data -- animals indeed solved the bias-variance tradeoff by combining learning on both fast and slow timescales. Our work suggests that multi-timescale learning could be a biologically plausible mechanism for optimizing decisions under uncertainty.
doi:10.1101/141309 fatcat:esu7dvhviffdplv7yadvsp5kxq