Predictive representations can link model-based reinforcement learning to model-free mechanisms

Evan M. Russek, Ida Momennejad, Matthew M. Botvinick, Samuel J. Gershman, Nathaniel D. Daw, Jean Daunizeau
<span title="2017-09-25">2017</span> <i title="Public Library of Science (PLoS)"> <a target="_blank" rel="noopener" href="https://fatcat.wiki/container/ch57atmlprauhhbqdf7x4ytejm" style="color: black;">PLoS Computational Biology</a> </i> &nbsp;
2017. "Predictive representations can link model-based reinforcement learning to model-free mechanisms." PLoS Computational Biology 13 (9): e1005768. Abstract Humans and animals are capable of evaluating actions by considering their long-run future rewards through a process described using model-based reinforcement learning (RL) algorithms. The mechanisms by which neural circuits perform the computations prescribed by model-based RL remain largely unknown; however, multiple lines of evidence
more &raquo; ... gest that neural circuits supporting model-based behavior are structurally homologous to and overlapping with those thought to carry out model-free temporal difference (TD) learning. Here, we lay out a family of approaches by which model-based computation may be built upon a core of TD learning. The foundation of this framework is the successor representation, a predictive state representation that, when combined with TD learning of value predictions, can produce a subset of the behaviors associated with model-based learning, while requiring less decision-time computation than dynamic programming. Using simulations, we delineate the precise behavioral capabilities enabled by evaluating actions using this approach, and compare them to those demonstrated by biological organisms. We then introduce two new algorithms that build upon the successor representation while progressively mitigating its limitations. Because this framework can account for the full range of observed putatively model-based behaviors while still utilizing a core TD framework, we suggest that it represents a neurally plausible family of mechanisms for model-based evaluation.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1371/journal.pcbi.1005768">doi:10.1371/journal.pcbi.1005768</a> <a target="_blank" rel="external noopener" href="https://www.ncbi.nlm.nih.gov/pubmed/28945743">pmid:28945743</a> <a target="_blank" rel="external noopener" href="https://pubmed.ncbi.nlm.nih.gov/PMC5628940/">pmcid:PMC5628940</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/p35rio5a5bbv7dzfbqyru5vdxe">fatcat:p35rio5a5bbv7dzfbqyru5vdxe</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20180719191814/https://dash.harvard.edu/bitstream/handle/1/34492498/5628940.pdf?sequence=1" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/a3/65/a365690310c84a36bb1ba6c5950d11fb3c057161.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href="https://doi.org/10.1371/journal.pcbi.1005768"> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="unlock alternate icon" style="background-color: #fb971f;"></i> plos.org </button> </a> <a target="_blank" rel="external noopener" href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5628940" title="pubmed link"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> pubmed.gov </button> </a>