On Bayesian bounds

Arindam Banerjee
2006 Proceedings of the 23rd international conference on Machine learning - ICML '06  
We show that several important Bayesian bounds studied in machine learning, both in the batch as well as the online setting, arise by an application of a simple compression lemma. In particular, we derive (i) PAC-Bayesian bounds in the batch setting, (ii) Bayesian log-loss bounds and (iii) Bayesian bounded-loss bounds in the online setting using the compression lemma. Although every setting has different semantics for prior, posterior and loss, we show that the core bound argument is the same.
more » ... he paper simplifies our understanding of several important and apparently disparate results, as well as brings to light a powerful tool for developing similar arguments for other methods.
doi:10.1145/1143844.1143855 dblp:conf/icml/Banerjee06 fatcat:jxyka3aymzczthybvkboqebjvy