Filters








7,947 Hits in 5.7 sec

Stability of Gradient Learning Dynamics in Continuous Games: Vector Action Spaces [article]

Benjamin J. Chasnov, Daniel Calderone, Behçet Açıkmeşe, Samuel A. Burden, Lillian J. Ratliff
2021 arXiv   pre-print
Towards characterizing the optimization landscape of games, this paper analyzes the stability of gradient-based dynamics near fixed points of two-player continuous games.  ...  We introduce the quadratic numerical range as a method to characterize the spectrum of game dynamics and prove the robustness of equilibria to variations in learning rates.  ...  STABILITY OF 2-PLAYER CONTINUOUS GAMES In this section, we give stability results for 2-player continuous games on vector action spaces. Consider a game (f 1 , f 2 ).  ... 
arXiv:2011.05562v2 fatcat:b3pv7nhdbjdqbpfyfw4gmpo4ey

Stability of Gradient Learning Dynamics in Continuous Games: Scalar Action Spaces [article]

Benjamin J. Chasnov, Daniel Calderone, Behçet Açıkmeşe, Samuel A. Burden, Lillian J. Ratliff
2020 arXiv   pre-print
Learning processes in games explain how players grapple with one another in seeking an equilibrium. We study a natural model of learning based on individual gradients in two-player continuous games.  ...  To characterize this gap, we provide formal guarantees for the stability or instability of such fixed points by leveraging the spectrum of the linearized game dynamics.  ...  In the sequel, we characterize continuous games defined on vector action spaces. B. Chasnov, S. Burden, and L.  ... 
arXiv:2011.03650v1 fatcat:ogdi7ik23ba5xnuhitrapnl3hq

Policy-Gradient Algorithms Have No Guarantees of Convergence in Linear Quadratic Games [article]

Eric Mazumdar, Lillian J. Ratliff, Michael I. Jordan, S. Shankar Sastry
2019 arXiv   pre-print
We show by counterexample that policy-gradient algorithms have no guarantees of even local convergence to Nash equilibria in continuous action and state space multi-agent settings.  ...  In such games the state and action spaces are continuous and global Nash equilibria can be found be solving coupled Ricatti equations.  ...  However, we believe that such phenomena have not yet been shown to occur in the dynamics of multi-agent reinforcement learning algorithms in continuous action and state spaces.  ... 
arXiv:1907.03712v2 fatcat:givojocp2jf67amh2ld7fn44nu

Approachability in population games

Dario Bauso, ,Jan C. Willems Center for Systems and Control, ENTEG, Fac. Science and Engineering University of Groningen, Thomas W.L. Norman, ,Dip. di Ingengneria, Università di Palermo, IT, ,Magdalen College, Oxford, UK
2019 Journal of Dynamics & Games  
choosing actions in a similar manner.  ...  Second, since the endogenous evolution of the population's play is then important, we develop a model of two coupled partial differential equations (PDEs) in the spirit of mean-field game theory: one describing  ...  We are thus interested in a population continuously matched to play a one-shot Bayesian game, rather than a dynamic repeated game with learning.  ... 
doi:10.3934/jdg.2020019 fatcat:3bzpeydsgjaftbzavj7jplunjq

Approachability in Population Games [article]

Dario Bauso, Thomas W L Norman
2014 arXiv   pre-print
Second, we develop a model of two coupled partial differential equations (PDEs) in the spirit of mean-field game theory: one describing the best-response of every player given the population distribution  ...  actions.  ...  This idea of adapting the new action to the current state of the game is common to adaptive learning and evolutionary games as well, but in regret-based dynamics the state is in payoff (rather than strategy  ... 
arXiv:1407.3910v1 fatcat:jc5wmnfusbg5folz5w3xpmedgm

Differentiable Game Mechanics [article]

Alistair Letcher and David Balduzzi and Sebastien Racaniere and James Martens and Jakob Foerster and Karl Tuyls and Thore Graepel
2019 arXiv   pre-print
The behavior of gradient-based methods in games is not well understood -- and is becoming increasingly important as adversarial and multi-objective architectures proliferate.  ...  In this paper, we develop new tools to understand and control the dynamics in n-player differentiable games. The key result is to decompose the game Jacobian into two components.  ...  Nash Convergence of Gradient Dynamics in General-Sum Games. In UAI, 2000. G Stoltz and G Lugosi. Learning correlated equilibria in games with compact sets of strategies.  ... 
arXiv:1905.04926v1 fatcat:lao2jsl7f5ewffknywz3qegd5q

Riemannian game dynamics

Panayotis Mertikopoulos, William H. Sandholm
2018 Journal of Economic Theory  
We examine the close connections between Hessian game dynamics and reinforcement learning in normal form games, extending and elucidating a well-known link between the replicator dynamics and exponential  ...  We study a class of evolutionary game dynamics under which the population state moves in the direction that agrees most closely with current payoffs.  ...  Equivalence of continuous Hessian dynamics and reinforcement learning. We now describe a common derivation of the reinforcement learning dynamics (RLD) and (HD) in the continuous regime.  ... 
doi:10.1016/j.jet.2018.06.002 fatcat:mxkvxyxcjvdidicdabdohdno7i

Inverse reinforcement learning for video games [article]

Aaron Tucker and Adam Gleave and Stuart Russell
2018 arXiv   pre-print
Deep reinforcement learning achieves superhuman performance in a range of video game environments, but requires that a designer manually specify a reward function.  ...  Inverse reinforcement learning (IRL) algorithms can infer a reward from demonstrations in low-dimensional continuous control environments, but there has been little work on applying IRL to high-dimensional  ...  Acknowledgments This work was supported by the Center for Human-Compatible AI and the Open Philanthropy Project, the Future of Life Institute and the Leverhulme Trust.  ... 
arXiv:1810.10593v1 fatcat:t6co2wtxtfa6jfgoyipt6jhcn4

Effects of noise on convergent game-learning dynamics

James B T Sanders, Tobias Galla, Jonathan L Shapiro
2012 Journal of Physics A: Mathematical and Theoretical  
We study stochastic effects on the lagging anchor dynamics, a reinforcement learning algorithm used to learn successful strategies in iterated games, which is known to converge to Nash points in the absence  ...  The effects of this noise are studied analytically in the case where it is small but finite, and we show that the statistics and correlation properties of fluctuations can be computed to a high accuracy  ...  The authors of [17] concentrated on the continuous limit of the deterministic lagging anchor dynamics, and determined the region of the parameter space for which the dynamics is stable.  ... 
doi:10.1088/1751-8113/45/10/105001 fatcat:b2busqcbc5cqpgeqa7roukgca4

Learning in Games with Lossy Feedback

Zhengyuan Zhou, Panayotis Mertikopoulos, Susan Athey, Nicholas Bambos, Peter W. Glynn, Yinyu Ye
2018 Neural Information Processing Systems  
We propose a simple variant of the classical online gradient descent algorithm, called reweighted online gradient descent (ROGD) and show that in variationally stable games, if each agent adopts ROGD,  ...  We consider a game-theoretical multi-agent learning problem where the feedback information can be lost during the learning process and rewards are given by a broad class of games known as variationally  ...  In the context of mixing in games with continuous action spaces, the authors of [37] provide a convergence analysis for a perturbed version of the multiplicative weights algorithm in potential games.  ... 
dblp:conf/nips/ZhouMABGY18 fatcat:o2dcwanbonbmflzoh2ck6sicnq

On the Convergence of Gradient-Based Learning in Continuous Games [article]

Eric Mazumdar, Lillian J. Ratliff
2018 arXiv   pre-print
This is a strongly negative result for gradient-based learning in games.  ...  We study the limiting behavior of competitive agents employing gradient-based learning algorithms through the lens of dynamical systems theory.  ...  The actions of agent i in the continuous game framework described in previous sections are the parameters of their policy, and thus their action space is X i ⊂ R m i .  ... 
arXiv:1804.05464v2 fatcat:7d3epr6gzzbb3ocfle734z5bke

Convergence of Learning Dynamics in Stackelberg Games [article]

Tanner Fiez, Benjamin Chasnov, Lillian J. Ratliff
2019 arXiv   pre-print
In the class of games we consider, there is a hierarchical game being played between a leader and a follower with continuous action spaces.  ...  This paper investigates the convergence of learning dynamics in Stackelberg games.  ...  In fact, for games on scalar action spaces, it turns out that non-Nash attracting critical points of the simultaneous gradient play dynamics at which −D 2 2 f (x * ) > 0 must be differential Stackelberg  ... 
arXiv:1906.01217v3 fatcat:awbusd3qlbebvnhwnf2emjqtdu

Stability of learning dynamics in two-agent, imperfect-information games

John M. Butterworth, Jonathan L. Shapiro
2009 Proceedings of the tenth ACM SIGEVO workshop on Foundations of genetic algorithms - FOGA '09  
Use of an estimated gradient, either by opponent modelling or stochastic gradient ascent, destabilises the algorithm in a region of parameter space. There are two phases of behaviour.  ...  One issue in multi-agent co-adaptive learning concerns convergence.  ...  A property of the mean dynamics is that if the variables start in the space spanned by the two real vectors e r and e i , it will stay in that subspace.  ... 
doi:10.1145/1527125.1527143 dblp:conf/foga/ButterworthS09 fatcat:vai6axeyifd6vcv5td3eeoyy7m

Equilibrium Tracking and Convergence in Dynamic Games

Panayotis Mertikopoulos, Mathias Staudigl
2021 2021 60th IEEE Conference on Decision and Control (CDC)  
In this paper, we examine the equilibrium tracking and convergence properties of no-regret learning algorithms in continuous games that evolve over time.  ...  to it if the game stabilizes to a strictly monotone limit.  ...  Starting with the seminal work of [14] , much of the literature on continuous games has focused on problems where the vector field v(x) of individual payoff gradients satisfies the monotonicity condition  ... 
doi:10.1109/cdc45484.2021.9683224 fatcat:gg32qhasufhhbgchziz2bcg5na

Riemannian game dynamics [article]

Panayotis Mertikopoulos, WIlliam H. Sandholm
2018 arXiv   pre-print
We examine the close connections between Hessian game dynamics and reinforcement learning in normal form games, extending and elucidating a well-known link between the replicator dynamics and exponential  ...  Like these representative dynamics, all Riemannian game dynamics satisfy certain basic desiderata, including positive correlation and global convergence in potential games.  ...  There is a more surprising connection between Hessian dynamics and models of reinforcement learning in normal form games.  ... 
arXiv:1603.09173v3 fatcat:ic7ginobw5adlc36ad2qcj5fjm
« Previous Showing results 1 — 15 out of 7,947 results