Neural and neurochemical basis of reinforcement-guided decision making

Abbas Khani, Gregor Rainer
<span title="">2016</span> <i title="American Physiological Society"> <a target="_blank" rel="noopener" href="" style="color: black;">Journal of Neurophysiology</a> </i> &nbsp;
Khani A, Rainer G. Neural and neurochemical basis of reinforcement-guided decision making. Decision making is an adaptive behavior that takes into account several internal and external input variables and leads to the choice of a course of action over other available and often competing alternatives. While it has been studied in diverse fields ranging from mathematics, economics, ecology, and ethology to psychology and neuroscience, recent cross talk among perspectives from different fields has
more &raquo; ... yielded novel descriptions of decision processes. Reinforcement-guided decision making models are based on economic and reinforcement learning theories, and their focus is on the maximization of acquired benefit over a defined period of time. Studies based on reinforcementguided decision making have implicated a large network of neural circuits across the brain. This network includes a wide range of cortical (e.g., orbitofrontal cortex and anterior cingulate cortex) and subcortical (e.g., nucleus accumbens and subthalamic nucleus) brain areas and uses several neurotransmitter systems (e.g., dopaminergic and serotonergic systems) to communicate and process decisionrelated information. This review discusses distinct as well as overlapping contributions of these networks and neurotransmitter systems to the processing of decision making. We end the review by touching on neural circuitry and neuromodulatory regulation of exploratory decision making. decision making; reinforcement-guided decision Laurent A, Gentil M, Perret J. Acute and long-term effects of subthalamic nucleus stimulation in Parkinson's disease. Stereotact Funct Neurosurg 62: 76 -84, 1994. Berendse HW, Galis-de Graaf Y, Groenewegen HJ. Topographical organization and relationship with ventral striatal compartments of prefrontal corticostriatal projections in the rat. J Comp Neurol 316: 314 -347, 1992. Bermudez MA, Gobel C, Schultz W. Sensitivity to temporal reward structure in amygdala neurons. Curr Biol 22: 1839 -1844, 2012. Bermudez MA, Schultz W. Reward magnitude coding in primate amygdala neurons. J Neurophysiol 104: 3424 -3432, 2010. Bezzina G, Body S, Cheung TH, Hampson CL, Bradshaw CM, Szabadi E, Anderson IM, Deakin JF. Effect of disconnecting the orbital prefrontal cortex from the nucleus accumbens core on inter-temporal choice behaviour: a quantitative analysis. Behav Brain Res 191: 272-279, 2008. Bezzina G, Cheung TH, Body S, Deakin JF, Anderson IM, Bradshaw CM, Szabadi E. Quantitative analysis of the effect of lesions of the subthalamic nucleus on intertemporal choice: further evidence for enhancement of the incentive value of food reinforcers. Behav Pharmacol 20: 437-446, 2009. Bizot J, Le Bihan C, Puech AJ, Hamon M, Thiebot M. Serotonin and tolerance to delay of reward in rats. Psychopharmacology 146: 400 -412, 1999. Bloomfield MA, Morgan CJ, Kapur S, Curran HV, Howes OD. The link between dopamine function and apathy in cannabis users: an [ 18 F]-DOPA PET imaging study. Psychopharmacology 231: 2251-2259, 2014. Bolla KI, Eldreth DA, Matochik JA, Cadet JL. Neural substrates of faulty decision-making in abstinent marijuana users. Neuroimage 26: 480 -492, 2005. Boomhower SR, Rasmussen EB, Doherty TS. Impulsive-choice patterns for food in genetically lean and obese Zucker rats. Behav Brain Res 241: 214 -221, 2013. Boorman ED, Behrens TE, Rushworth MF. Counterfactual choice and learning in a neural network centered on human lateral frontopolar cortex. PLoS Biol 9: e1001093, 2011. Boorman ED, Behrens TE, Woolrich MW, Rushworth MF. How green is the grass on the other side? Frontopolar cortex and the evidence in favor of alternative courses of action. Neuron 62: 733-743, 2009. Boorman ED, Rushworth MF, Behrens TE. Ventromedial prefrontal and anterior cingulate cortex adopt choice and default reference frames during sequential multi-alternative choice. J Neurosci 33: 2242-2253, 2013. Botvinick M, Weinstein A. Model-based hierarchical reinforcement learning and human action control. Philos Trans R Soc Lond B Biol Sci 369: 20130480, 2014. Botvinick MM. Hierarchical reinforcement learning and decision making. Curr Opin Neurobiol 22: 956 -962, 2012. Bouret S, Richmond BJ. Ventromedial and orbital prefrontal neurons differentially encode internally and externally driven motivational values in monkeys. J Neurosci 30: 8591-8601, 2010. Bouret S, Sara SJ. Network reset: a simplified overarching theory of locus coeruleus noradrenaline function. Trends Neurosci 28: 574 -582, 2005. Brown E, Gao J, Holmes P, Bogacz R, Gilzenrat M, Cohen JD. Simple neural networks that optimize decisions. Int J Bifurcat Chaos 15: 803-826, 2005. Bunge SA, Wendelken C. Comparing the bird in the hand with the ones in the bush. Neuron 62: 609 -611, 2009. Burton AC, Kashtelyan V, Bryden DW, Roesch MR. Increased firing to cues that predict low-value reward in the medial orbitofrontal cortex. Cereb Cortex 24: 3310 -3321, 2014. Burton AC, Nakamura K, Roesch MR. From ventral-medial to dorsal-lateral striatum: neural correlates of reward-guided decision-making. Neurobiol Learn Mem 117: 51-59, 2015. Calaminus C, Hauber W. Modulation of behavior by expected reward magnitude depends on dopamine in the dorsomedial striatum. Neurotox Res 15: 97-110, 2009. Camille N, Tsuchida A, Fellows LK. Double dissociation of stimulus-value and action-value learning in humans with orbitofrontal or anterior cingulate cortex damage. J Neurosci 31: 15048 -15052, 2011. Cardinal RN, Pennicott DR, Sugathapala CL, Robbins TW, Everitt BJ. Impulsive choice induced in rats by lesions of the nucleus accumbens core. Science 292: 2499 -2501, 2001. Castillo PE, Younts TJ, Chavez AE, Hashimotodani Y. Endocannabinoid signaling and synaptic function. Neuron 76: 70 -81, 2012. Cavanagh JF, Wiecki TV, Cohen MX, Figueroa CM, Samanta J, Sherman SJ, Frank MJ. Subthalamic nucleus stimulation reverses mediofrontal influence over decision threshold. Nat Neurosci 14: 1462-1467, 2011. Chowdhury R, Guitart-Masip M, Lambert C, Dayan P, Huys Q, Duzel E, Dolan RJ. Dopamine restores reward prediction errors in old age. Nat Neurosci 16: 648 -653, 2013. Churchwell JC, Morris AM, Heurtelou NM, Kesner RP. Interactions between the prefrontal cortex and amygdala during delay discounting and reversal. Behav Neurosci 123: 1185-1196, 2009. Clayton EC, Rajkowski J, Cohen JD, Aston-Jones G. Phasic activation of monkey locus ceruleus neurons by simple decisions in a forced-choice task. J Neurosci 24: 9914 -9920, 2004. Cools R, Roberts AC, Robbins TW. Serotoninergic regulation of emotional and behavioural control processes. Trends Cogn Sci 12: 31-40, 2008. Cousijn J, Wiers RW, Ridderinkhof KR, van den Brink W, Veltman DJ, Porrino LJ, Goudriaan AE. Individual differences in decision making and reward processing predict changes in cannabis use: a prospective functional magnetic resonance imaging study. Addict Biol 18: 1013-1023, 2013. Crane NA, Schuster RM, Gonzalez R. Preliminary evidence for a sexspecific relationship between amount of cannabis use and neurocognitive performance in young adult cannabis users. J Int Neuropsychol Soc 19: 1009 -1015, 2013. Croxson PL, Walton ME, O'Reilly JX, Behrens TE, Rushworth MF. Effort-based cost-benefit valuation and the human brain. J Neurosci 29: 4531-4541, 2009. da Rocha FF, Malloy-Diniz L, Lage NV, Romano-Silva MA, de Marco LA, Correa H. Decision-making impairment is related to serotonin transporter promoter polymorphism in a sample of patients with obsessive-compulsive disorder. Behav Brain Res 195: 159 -163, 2008. Daw ND, O'Doherty JP, Dayan P, Seymour B, Dolan RJ. Cortical substrates for exploratory decisions in humans. Nature 441: 876 -879, 2006. Day JJ, Jones JL, Wightman RM, Carelli RM. Phasic nucleus accumbens dopamine release encodes effort-and delay-related costs. Fuso A, Laviola G. Pharmacological stimulation of the brain serotonin receptor 7 as a novel therapeutic approach for Rett syndrome. Neuropsychopharmacology 39: 2506 -2518, 2014. Delazer M, Hogl B, Zamarian L, Wenter J, Gschliesser V, Ehrmann L, Brandauer E, Cevikkol Z, Frauscher B. Executive functions, information sampling, and decision making in narcolepsy with cataplexy. Neuropsychology 25: 477-487, 2011. Denk F, Walton ME, Jennings KA, Sharp T, Rushworth MF, Bannerman DM. Differential involvement of serotonin and dopamine systems in costbenefit decisions about delay or effort. Psychopharmacology 179: 587-596, 2005. Deshmukh RR, Sharma PL. Stimulation of accumbens shell cannabinoid CB 1 receptors by noladin ether, a putative endocannabinoid, modulates food intake and dietary selection in rats. Pharmacol Res 66: 276 -282, 2012. Donahue CH, Seo H, Lee D. Cortical signals for rewarded actions and strategic exploration. . Ventral tegmental area cannabinoid type-1 receptors control voluntary exercise performance. Biol Psychiatry 73: 895-903, 2013. Egelman DM, Person C, Montague PR. A computational role for dopamine delivery in human decision-making. J Cogn Neurosci 10: 623-630, 1998. Eggan SM, Lewis DA. Immunocytochemical distribution of the cannabinoid CB1 receptor in the primate neocortex: a regional and laminar analysis. Ennaceur A, Delacour J. A new one-trial test for neurobiological studies of memory in rats. 1. Behavioral data. Behav Brain Res 31: 47-59, 1988. Eppinger B, Hammerer D, Li SC. Neuromodulation of reward-based learning and decision making in human aging. Ann NY Acad Sci 1235: 1-17, 2011. Fellows LK. The cognitive neuroscience of human decision making: a review and conceptual framework. Behav Cogn Neurosci Rev 3: 159 -172, 2004. Fetsch CR, Kiani R, Newsome WT, Shadlen MN. Effects of cortical microstimulation on confidence in a perceptual decision. Neuron 83: 797-804, 2014. Filbey FM, Schacht JP, Myers US, Chavez RS, Hutchison KE. Individual and additive effects of the CNR1 and FAAH genes on brain response to marijuana cues. Neuropsychopharmacology 35: 967-975, 2010. Fiorillo CD, Tobler PN, Schultz W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299: 1898 -1902, 2003. Fischer BA, McMahon RP, Kelly DL, Wehring HJ, Meyer WA, Feldman S, Carpenter WT, Gorelick DA. Risk-taking in schizophrenia and controls with and without cannabis dependence. Schizophr Res 161: 471-477, 2015. Flagel SB, Clark JJ, Robinson TE, Mayo L, Czuj A, Willuhn I, Akers CA, Clinton SM, Phillips PE, Akil H. A selective role for dopamine in stimulus-reward learning. Nature 469: 53-57, 2011. Floresco SB, Ghods-Sharifi S. Amygdala-prefrontal cortical circuitry regulates effort-based decision making. Cereb Cortex 17: 251-260, 2007. Forstmann BU, Brown S, Dutilh G, Neumann J, Wagenmakers EJ. The neural substrate of prior information in perceptual decision making: a model-based analysis. Front Hum Neurosci 4: 40, 2010. Frank MJ, Doll BB, Oas-Terpstra J, Moreno F. Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nat Neurosci 12: 1062-1068, 2009. Frank MJ, Samanta J, Moustafa AA, Sherman SJ. Hold your horses: impulsivity, deep brain stimulation, and medication in parkinsonism. Science 318: 1309 -1312, 2007. Fridberg DJ, Queller S, Ahn WY, Kim W, Bishara AJ, Busemeyer JR, Porrino L, Stout JC. Cognitive mechanisms underlying risky decisionmaking in chronic cannabis users. J Math Psychol 54: 28 -38, 2010. Friston K, Schwartenbeck P, Fitzgerald T, Moutoussis M, Behrens T, Dolan RJ. The anatomy of choice: active inference and agency. Front Hum Neurosci 7: 598, 2013. Friston K, Schwartenbeck P, FitzGerald T, Moutoussis M, Behrens T, Dolan RJ. The anatomy of choice: dopamine and decision-making. Philos Trans R Soc Lond B Biol Sci 369: 20130481, 2014. Fusi S, Asaad WF, Miller EK, Wang XJ. A neural circuit model of flexible sensorimotor mapping: learning and forgetting on multiple timescales. Neuron 54: 319 -333, 2007. Gan JO, Walton ME, Phillips PE. Dissociable cost and benefit encoding of future rewards by mesolimbic dopamine. Nat Neurosci 13: 25-27, 2010. Ganon-Elazar E, Akirav I. Cannabinoid receptor activation in the basolateral amygdala blocks the effects of stress on the conditioning and extinction of inhibitory avoidance. X. COMT Val158Met polymorphism influences the susceptibility to framing in decision-making: OFC-amygdala functional connectivity as a mediator. Hum Brain Mapp 37: 1880 -1892, 2016. Ghods-Sharifi S, Floresco SB. Differential effects on effort discounting induced by inactivations of the nucleus accumbens core or shell. Behav Neurosci 124: 179 -191, 2010. Ghods-Sharifi S, St Onge JR, Floresco SB. Fundamental contribution by the basolateral amygdala to different forms of decision making. J Neurosci 29: 5251-5259, 2009. Glimcher P. Decisions, decisions, decisions: choosing a biological science of choice. Neuron 36: 323-332, 2002. Glimcher PW, Dorris MC, Bayer HM. Physiological utility theory and the neuroeconomics of choice. Games Econ Behav 52: 213-256, 2005. Gold JI, Shadlen MN. The neural basis of decision making. Annu Rev Neurosci 30: 535-574, 2007. Gonzalez R, Schuster RM, Mermelstein RJ, Vassileva J, Martin EM, Diviak KR. Performance of young adult cannabis users on neurocognitive measures of impulsive behavior and their relationship to symptoms of cannabis use disorders. Griffith-Lendering MF, Huijbregts SC, Vollebergh WA, Swaab H. Motivational and cognitive inhibitory control in recreational cannabis users. J Clin Exp Neuropsychol 34: 688 -697, 2012. Gupta R, Koscik TR, Bechara A, Tranel D. The amygdala and decisionmaking. Neuropsychologia 49: 760 -766, 2011. Hadamitzky M, Feja M, Becker T, Koch M. Effects of acute systemic administration of serotonin2A/C receptor ligands in a delay-based decisionmaking task in rats. Kahnt T, Park SQ, Cohen MX, Beck A, Heinz A, Wrase J. Dorsal striatal-midbrain connectivity in humans predicts how reinforcements are used to guide decisions. J Cogn Neurosci 21: 1332-1345, 2009. Kamil AC, Roitblat HL. The ecology of foraging behavior: implications for animal learning and memory. Annu Rev Psychol 36: 141-169, 1985. Kano M, Ohno-Shosaku T, Hashimotodani Y, Uchigashima M, Watanabe M. Endocannabinoid-mediated control of synaptic transmission. Physiol Rev 89: 309 -380, 2009.
<span class="external-identifiers"> <a target="_blank" rel="external noopener noreferrer" href="">doi:10.1152/jn.01113.2015</a> <a target="_blank" rel="external noopener" href="">pmid:27226454</a> <a target="_blank" rel="external noopener" href="">fatcat:wpqjzj2sxnh2dh5lfqw5pszuhq</a> </span>
<a target="_blank" rel="noopener" href="" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener noreferrer" href=""> <button class="ui left aligned compact blue labeled icon button serp-button"> <i class="external alternate icon"></i> Publisher / </button> </a>