Filters








9,470 Hits in 5.3 sec

Fast Variational Inference in the Conjugate Exponential Family [article]

James Hensman, Magnus Rattray, Neil D. Lawrence
2012 arXiv   pre-print
We present a general method for deriving collapsed variational inference algo- rithms for probabilistic models in the conjugate exponential family.  ...  Our method unifies many existing approaches to collapsed variational inference. Our collapsed variational inference leads to a new lower bound on the marginal likelihood.  ...  Acknowledgements The authors would like to thank Michalis Titsias for helpful commentary on a previous draft and Peter Glaus for help with a C++ implementation of the RNAseq alignment algorithm.  ... 
arXiv:1206.5162v2 fatcat:vhbymryvdncizodoa2nifvibsi

Composing graphical models with neural networks for structured representations and fast inference [article]

Matthew J. Johnson and David Duvenaud and Alexander B. Wiltschko and Sandeep R. Datta and Ryan P. Adams
2017 arXiv   pre-print
Our model family augments graphical structure in latent variables with neural network observation models.  ...  For inference, we extend variational autoencoders to use graphical model approximating distributions with recognition networks that output conjugate potentials.  ...  [7, 8] ) or optimizing out the local variational factors using fast conjugate updates (as in conjugate SVI [10] ) can be advantageous because in both cases local variational parameters for the entire  ... 
arXiv:1603.06277v5 fatcat:kkackdtwkrcenhj36csyvorlwq

DAPPER: Scaling Dynamic Author Persona Topic Model to Billion Word Corpora [article]

Robert Giaquinto, Arindam Banerjee
2018 arXiv   pre-print
Specifically, we develop Conjugate-Computation Variational Inference (CVI) based variational Expectation-Maximization (EM) for learning the model, yielding fast, closed form updates for each document,  ...  Our results show significant improvements in model fit and training time without needing to compromise the model's temporal structure or the application of Regularized Variation Inference (RVI).  ...  Khan and Lin [8] demonstrate that this approach leads to inference in a conjugate model, where non-conjugate terms have been replaced by exponential family approximations.  ... 
arXiv:1811.01931v1 fatcat:3fcnsrxaqjebbe4niqqz3jkqu4

Fast yet Simple Natural-Gradient Descent for Variational Inference in Complex Models [article]

Mohammad Emtiyaz Khan, Didrik Nielsen
2018 arXiv   pre-print
We show how to derive fast yet simple natural-gradient updates by using a duality associated with exponential-family distributions.  ...  Variational inference circumvents these challenges by formulating Bayesian inference as an optimization problem and solving it using gradient-based optimization.  ...  ACKNOWLEDGMENT We would like to thank the following people at RIKEN, AIP for discussions and feedback: Aaron Mishkin, Frederik Kunstner, Voot Tangkaratt, and Wu Lin.  ... 
arXiv:1807.04489v2 fatcat:tcmtkg3ldjatfdzco5qkpriv5e

Conjugate-Computation Variational Inference : Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models [article]

Mohammad Emtiyaz Khan, Wu Lin
2017 arXiv   pre-print
Variational inference is computationally challenging in models that contain both conjugate and non-conjugate terms.  ...  In this paper, we propose a new algorithm called Conjugate-computation Variational Inference (CVI) which brings the best of the two worlds together -- it uses conjugate computations for the conjugate terms  ...  Acknowledgments: We would like to thank the anonymous reviewers for their feedback.  ... 
arXiv:1703.04265v2 fatcat:xz2rk7yeibbx7l2hxn6nkp2dky

Estimating Diagnostic Error without a Gold Standard: A Mixed Membership Approach [chapter]

2014 Handbook of Mixed Membership Models and Their Applications  
Though the matrix in Equation (11.17) is often very complicated, it is superfluous to batch variational inference for conjugate exponential family models.  ...  The key to stochastic variational inference for conjugate exponential models is in selecting G.  ... 
doi:10.1201/b17520-15 fatcat:iqoz7k5w65aa3m7d4faxvsftx4

Variational methods for the Dirichlet process

David M. Blei, Michael I. Jordan
2004 Twenty-first international conference on Machine learning - ICML '04  
In this paper, we develop a meanfield variational approach to approximate inference for the Dirichlet process, where the approximate posterior is based on the truncated stick-breaking construction (Ishwaran  ...  Variational inference methods, including mean field methods and loopy belief propagation, have been widely used for approximate probabilistic inference in graphical models.  ...  Blei is supported by a fellowship from the Microsoft Corporation. We also would like to acknowledge a grant from DARPA for the CALO project.  ... 
doi:10.1145/1015330.1015439 dblp:conf/icml/BleiJ04 fatcat:gjmaqaojtvdczatgzcyo3zndue

Overdispersed Black-Box Variational Inference [article]

Francisco J. R. Ruiz, Michalis K. Titsias, David M. Blei
2016 arXiv   pre-print
Instead of taking samples from the variational distribution, we use importance sampling to take samples from an overdispersed distribution in the same exponential family as the variational approximation  ...  We introduce overdispersed black-box variational inference, a method to reduce the variance of the Monte Carlo estimator of the gradient in black-box variational inference.  ...  In detail, we first assume that the variational distribution is in the exponential family.  ... 
arXiv:1603.01140v1 fatcat:ogekryssuvay3p5px6txjnvvha

Discussion of "Fast Approximate Inference for Arbitrarily Large Semiparametric Regression Models via Message Passing" [article]

Dustin Tran, David M. Blei
2016 arXiv   pre-print
Discussion paper on "Fast Approximate Inference for Arbitrarily Large Semiparametric Regression Models via Message Passing" by Wand [arXiv:1602.07412].  ...  This exponential family structure implies that, conditionally, the posterior factors are also in the same exponential families as the prior factors (Diaconis and Ylvisaker, 1979) , p(φ | y, α) = h(φ)  ...  From our own work, we apply the idea in order to access fast natural gradients in variational inference, which accounts for the information geometry of the parameter space (Hoffman et al., 2013) .  ... 
arXiv:1609.05615v1 fatcat:wrxzim5lcnasjgxaagw4lvofl4

Automated Augmented Conjugate Inference for Non-conjugate Gaussian Process Models [article]

Théo Galy-Fajou, Florian Wenzel, Manfred Opper
2020 arXiv   pre-print
First, a fast and scalable stochastic variational inference method that uses efficient block coordinate ascent updates, which are computed in closed form.  ...  Building on the conjugate structure of the augmented model, we develop two inference methods.  ...  The distribution family π ϕ (ω|c) is derived by an exponential tilting of the prior distribution p(ω) and is discussed in Section 3.2. (3a) Augmented variational inference.  ... 
arXiv:2002.11451v1 fatcat:2btbs625pzabtows3zwvwuu3u4

Variational Message Passing with Structured Inference Networks [article]

Wu Lin, Nicolas Hubacher, Mohammad Emtiyaz Khan
2018 arXiv   pre-print
First, we propose structured inference networks that incorporate the structure of the graphical model in the inference network of variational auto-encoders (VAE).  ...  We propose a variational message-passing algorithm for variational inference in such models. We make three contributions.  ...  Johnson (Google Brain) and David Duvenaud (University of Toronto) for providing the SVAE code.  ... 
arXiv:1803.05589v2 fatcat:2y4gbetaufhgfbdvwqh3pfbcnu

Variational inference for Dirichlet process mixtures

David M. Blei, Michael I. Jordan
2006 Bayesian Analysis  
Thus far, variational methods have mainly been explored in the parametric setting, in particular within the formalism of the exponential family (Attias, 2000; Ghahramani and Beal, 2001; Blei et al., 2003  ...  In this paper, we present a variational inference algorithm for DP mixtures.  ...  When G 0 is not conjugate, a simple coordinate ascent update for τ i may not be available if p(η * i | z, x, λ) is not in the exponential family.  ... 
doi:10.1214/06-ba104 fatcat:u3utwh7t3bdttl6nksk4g3km5m

Variational Inference: A Review for Statisticians [article]

David M. Blei, Alp Kucukelbir, Jon D. McAuliffe
2018 arXiv   pre-print
We review the ideas behind mean-field variational inference, discuss the special case of VI applied to exponential family models, present a full example with a Bayesian mixture of Gaussians, and derive  ...  In this paper, we review variational inference (VI), a method from machine learning that approximates probability densities through optimization.  ...  The posterior density of the coefficients is not in an exponential family and we cannot apply the variational inference methods we discussed above.  ... 
arXiv:1601.00670v8 fatcat:opqknuo6t5cezfvluwhmu4cg7e

Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models [article]

Hugh Salimbeni, Stefanos Eleftheriadis, James Hensman
2018 arXiv   pre-print
The natural gradient method has been used effectively in conjugate Gaussian process models, but the non-conjugate case has been largely unexplored.  ...  We examine how natural gradients can be used in non-conjugate stochastic settings, together with hyperparameter learning.  ...  Acknowledgements We have greatly appreciated valuable discussions with Mark van der Wilk in the preparation of this work.  ... 
arXiv:1803.09151v1 fatcat:z3bowbbszfhyzbibxgbn7jp5sy

Probabilistic Models with Deep Neural Networks

Andrés R. Masegosa, Rafael Cabañas, Helge Langseth, Thomas D. Nielsen, Antonio Salmerón
2021 Entropy  
However, developments in variational inference, a general form of approximate probabilistic inference that originated in statistical physics, have enabled probabilistic modeling to overcome these limitations  ...  Recent advances in statistical inference have significantly expanded the toolbox of probabilistic modeling.  ...  conjugate exponential family because, in this case, inference is feasible (and scalable) as we showed in Section 2.  ... 
doi:10.3390/e23010117 pmid:33477544 pmcid:PMC7831091 fatcat:wzitmmruvjbehgdie2wgtk7dtq
« Previous Showing results 1 — 15 out of 9,470 results