Reward Shaping for Model-Based Bayesian Reinforcement Learning.

Reinforcement learning models capture many behavioral and neural effects but do not explain recent findings showing that structure in the environment influences learning. ... In parallel, Bayesian cognitive models predict how humans learn structured knowledge but do not have a clear neurobiological implementation. ... Highlights • Recent advances have refined our understanding of reinforcement learning by emphasizing roles for attention and for structured knowledge in shaping ongoing learning. • Bayesian cognitive ...

doi:10.1016/j.tics.2019.01.010 pmid:30824227 pmcid:PMC6472955 fatcat:n3dvwxnh5vggzoaqkemfkgzxpe

These value systems have been effectively used in robotic systems to shape behavior. For example, many robots have used models of the dopaminergic system to reinforce behavior that leads to rewards. ... A recent Research Topic in Frontiers of Neurorobotics explored value and reward based learning. ... These value systems have been effectively used in robotic systems to shape behavior. For example, many robots have used models of the dopaminergic system to reinforce behavior that leads to rewards. ...

doi:10.3389/fnbot.2013.00013 pmid:24062683 pmcid:PMC3772325 fatcat:hbajnwwf2fb2jexnmgtdbtqqva

DOAJ

In reinforcement learning, a decision maker searching for the most rewarding option is often faced with the question: what is the value of an option that has never been tried before? ... as models of reinforcement learning in humans and animals. ... Acknowledgments We thank Nathaniel Daw for many fruitful discussions and Quentin Huys for comments on an earlier version of the manuscript. ...

doi:10.1111/tops.12138 pmid:25808176 pmcid:PMC4537661 fatcat:f626mbxvjfa2zp7u2nbwvofsha

Computational modeling provides a framework for delineating specific processes underlying social cognition and relating them to neural activity and behavior. ... We provide a primer on the computational modeling approach and describe how it has been used to elucidate psychological and neural mechanisms of impression formation, social learning, moral decision making ... . ** This study used computational modeling of reinforcement learning to examine trait-based impression formation, dissociating it from reward-based learning during social interaction. ...

doi:10.1016/j.copsyc.2018.09.001 fatcat:kwqmwyfdpfefbi3ppsb5ynigsq

We make these properties explicit by showing two scenarios in which active inference agents can infer behaviors in reward-free environments compared to both Q-learning and Bayesian model-based RL agents ... This problem is also considered in reinforcement learning (RL), but limited work exists on comparing the two approaches on the same discrete-state environments. ... We would like to thank the anonymous reviewers for their suggestions and insightful comments on the manuscript. Disclosure statement The authors have no disclosures or conflict of interest. ...

arXiv:1909.10863v3 fatcat:x5bwlzhyzvde3bh24er3zyvewm

Multiple Versions

We make these properties explicit by showing two scenarios in which active inference agents can infer behaviors in reward-free environments compared to both Q-learning and Bayesian model-based reinforcement ... We show that by operating in a pure belief-based setting, active inference agents can carry out epistemic exploration—and account for uncertainty about their environment—in a Bayes-optimal fashion. ... We thank the anonymous reviewers for their suggestions and insightful comments on the manuscript. Disclosure Statement We have no disclosures or conflict of interest. ...

doi:10.1162/neco_a_01357 pmid:33400903 fatcat:hwx6njdemjhhjelveg4uef2cge

In the model-based approach, the agent learns a model of the dynamics of the world and its rewards. Given the model, it tries to solve for the optimal control policy. ... The agent's goal is to find, based on its experience with the environment, a strategy or an optimal policy for choosing actions that will yield as much reward as possible. s Reinforcement learning has ...

doi:10.1609/aimag.v17i4.1244 dblp:journals/aim/MahadevanK96 fatcat:vrz3h6o2cnb6njmtnohflsfksa

A Bayesian credit-assignment model with built-in forgetting accurately predicts their trial-by-trial learning. ... Based on the history of action-reward pairs, participants quickly solved the credit assignment problem and learned the implicit payoff function. ... Our Bayesian model is conceptually similar to other reinforcement learning models where the decision for the next attempt is made on the basis of the history of rewardmovement pairs. ...

doi:10.1371/journal.pone.0055352 pmid:23408972 pmcid:PMC3568147 fatcat:t4kk7xdcfnd3lgpuyilji5xeme

DOAJ

Using just two programmer-provided functions Map() and Eval(), the Fleet Computer compiles and deploys swarms and continuously updates the reinforcement learning models that govern actions. ... The Fleet Computer provides a programming paradigm that simplifies multi-agent reinforcement learning (MARL) -- an emerging class of algorithms that coordinate swarms of agents. ... Singh et. al [54] demonstrates a novel reward-training mechanism for reinforcement learning to eliminate the need for reward shaping. ...

arXiv:2105.10605v1 fatcat:xddio3i75bg73ng4p2u4vqirei

Open Access

We also develop a hybrid Bayesian-reinforcement learning mixture of experts (MoE) model, which can estimate the most likely hypothesis state of individual participants based on their observed sequence ... Learning at all of these levels is accomplished via dopaminergic reward prediction error signals in each corticostriatal circuit. ... Notes We thank Andy Kaiser, Anne Collins, and Thomas Wiecki for helpful comments and discussion. Conflict of Interest: None declared. ...

doi:10.1093/cercor/bhr114 pmid:21693490 pmcid:PMC3278315 fatcat:lnirkrnsu5guvjj5sz4kn4a3fy

We present an approach that uses a model to derive reward bonuses as a means of intrinsic motivation to improve model-free reinforcement learning. ... All reinforcement learning algorithms must handle the trade-off between exploration and exploitation. ... A relationship between model-based information gain-based intrinsic motivation and Bayesian Q-Learning in the Linear Case We now discuss a relationship between intrinsic rewards based on information gain ...

arXiv:1804.01238v1 fatcat:wsba324bgfglha57e6bahwc2aa

In a series of papers, we have formalized an active Bayesian perception approach for robotics based on recent progress in understanding animal perception. ... Here we propose that this tuning should be learnt by reinforcement from a reward signal evaluating the decision outcome. ... Acknowledgements: We thank Kevin Gurney, Ashvin Shah and Alberto Testolin for discussions, and the organizers of the 2012 FIAS school on Intrinsic Motivations for hosting NL and GP while some of this work ...

doi:10.1109/iros.2013.6697038 dblp:conf/iros/LeporaMPP13 fatcat:je2hn7g5ebdk5lfbz67vvnine4

Bayesian inverse reinforcement learning (IRL) methods are ideal for safe imitation learning, as they allow a learning agent to reason about reward uncertainty and the safety of a learned policy. ... We propose Bayesian Reward Extrapolation (B-REX), a highly efficient, preference-based Bayesian reward learning algorithm that scales to high-dimensional, visual control tasks. ... Bayesian Inverse Reinforcement Learning In inverse reinforcement learning, the environment is modeled as an MDP\R where the reward function R is internal to the demonstrator and is unknown and unobserved ...

arXiv:1912.04472v1 fatcat:c2ouhzmearhupopywchlvp7ckq

Our proposal is to cast the active learning task as a utility maximization problem using Bayesian reinforcement learning with belief-dependent rewards. ... We consider the active learning problem of inferring the transition model of a Markov Decision Process by acting and observing transitions. ... Model-based Bayesian Reinforcement Learning We consider here model-based Bayesian Reinforcement Learning [17] (BRL), i.e., model-based RL where the knowledge about the model-now a random vector b-is ...

doi:10.1007/978-3-642-29946-9_8 fatcat:atu564zq2fdhvin3vwtstsq73e

The sampleefficiency and safety of learning were improved by adding two novel modifications to standard Bayesian optimization. ... Concentrating on tasks requiring constant impedance parameters throughout operation, a model-free learning framework is proposed to autonomously find the suitable parameters values. ... Unlike supervised approaches, reinforcement learning is more suited for general autonomous learning in unknown environments. ...

doi:10.1109/ssrr.2019.8848967 dblp:conf/ssrr/KhattabRES19 fatcat:sp7jgkoelndylfpjymscbij65u

Holistic Reinforcement Learning: The Role of Structure and Attention

Preserved Fulltext

Value and reward based learning in neurorobots

Preserved Fulltext

Novelty and Inductive Generalization in Human Reinforcement Learning

Preserved Fulltext

Computational neuroscience approaches to social cognition

Preserved Fulltext

Active inference: demystified and compared [article]

Preserved Fulltext

Other Versions

Active Inference: Demystified and Compared

Preserved Fulltext

The National Science Foundation Workshop on Reinforcement Learning

Preserved Fulltext

Credit Assignment during Movement Reinforcement Learning

Preserved Fulltext

Programming and Deployment of Autonomous Swarms using Multi-Agent Reinforcement Learning [article]

Preserved Fulltext

Mechanisms of Hierarchical Reinforcement Learning in Corticostriatal Circuits 1: Computational Analysis

Preserved Fulltext

Information Maximizing Exploration with a Latent Dynamics Model [article]

Preserved Fulltext

Active Bayesian perception and reinforcement learning

Preserved Fulltext

Deep Bayesian Reward Learning from Preferences [article]

Preserved Fulltext

Active Learning of MDP Models [chapter]

Preserved Fulltext

Bayesian-Optimized Impedance Control of an Aerial Robot for Safe Physical Interaction with the Environment

Preserved Fulltext