A New Class of Upper Bounds on the Log Partition Function

M.J. Wainwright, T.S. Jaakkola, A.S. Willsky
2005 IEEE Transactions on Information Theory  
We introduce a new class of upper bounds on the log partition function of a Markov random field (MRF). This quantity plays an important role in various contexts, including approximating marginal distributions, parameter estimation, combinatorial enumeration, statistical decision theory, and large-deviations bounds. Our derivation is based on concepts from convex duality and information geometry: in particular, it exploits mixtures of distributions in the exponential domain, and the Legendre
more » ... ing between exponential and mean parameters. In the special case of convex combinations of tree-structured distributions, we obtain a family of variational problems, similar to the Bethe variational problem, but distinguished by the following desirable properties: i) they are convex, and have a unique global optimum; and ii) the optimum gives an upper bound on the log partition function. This optimum is defined by stationary conditions very similar to those defining fixed points of the sum-product algorithm, or more generally, any local optimum of the Bethe variational problem. As with sum-product fixed points, the elements of the optimizing argument can be used as approximations to the marginals of the original model. The analysis extends naturally to convex combinations of hypertree-structured distributions, thereby establishing links to Kikuchi approximations and variants. Index Terms-Approximate inference, belief propagation, Bethe/Kikuchi free energy, factor graphs, graphical models, information geometry, Markov random field (MRF), partition function, sum-product algorithm, variational method. 1 The Ising model [4] is a pairwise MRF on a binary random vector.
doi:10.1109/tit.2005.850091 fatcat:gj3suuqbcnawdlaipub25a7jt4