Active inference and epistemic value

Karl Friston, Francesco Rigoli, Dimitri Ognibene, Christoph Mathys, Thomas Fitzgerald, Giovanni Pezzulo
2015 Cognitive Neuroscience  
We offer a formal treatment of choice behavior based on the premise that agents minimize the expected free energy of future outcomes. Crucially, the negative free energy or quality of a policy can be decomposed into extrinsic and epistemic (or intrinsic) value. Minimizing expected free energy is therefore equivalent to maximizing extrinsic value or expected utility (defined in terms of prior preferences or goals), while maximizing information gain or intrinsic value (or reducing uncertainty
more » ... t the causes of valuable outcomes). The resulting scheme resolves the exploration-exploitation dilemma: Epistemic value is maximized until there is no further information gain, after which exploitation is assured through maximization of extrinsic value. This is formally consistent with the Infomax principle, generalizing formulations of active vision based upon salience (Bayesian surprise) and optimal decisions based on expected utility and risk-sensitive (Kullback-Leibler) control. Furthermore, as with previous active inference formulations of discrete (Markovian) problems, ad hoc softmax parameters become the expected (Bayes-optimal) precision of beliefs about, or confidence in, policies. This article focuses on the basic theory, illustrating the ideas with simulations. A key aspect of these simulations is the similarity between precision updates and dopaminergic discharges observed in conditioning paradigms. This article introduces a variational (free energy) formulation of explorative behavior and the (epistemic) value of knowing one's environment. This formulation tries to unite a number of perspectives on behavioral imperatives; namely, the explorationexploitation dilemma and the distinction between the explicit (extrinsic) value of controlled outcomes and their epistemic (intrinsic) value in reducing uncertainty about environmental contingencies Abstract: Contrary to Friston's previous work, this paper describes free energy minimization using categorical probability distributions over discrete states. This alternative mathematical framework exposes a fundamental, yet unnoticed challenge for the free energy principle. When considering discrete state spaces one must specify their granularity, as the amount of information gain is defined over this state space. The more detailed this state space, the lower the precision of the predictions will be, and consequently, the higher the prediction errors. Hence, an optimal trade-off between precision and detail is needed, and we call for incorporating this aspect in the free energy principle.
doi:10.1080/17588928.2015.1020053 pmid:25689102 fatcat:xl333x6oi5d5xkumzen624splm