Bayesian Bin Distribution Inference and Mutual Information

D. Endres, P. Foldiak
2005 IEEE Transactions on Information Theory  
We present an exact Bayesian treatment of a simple, yet sufficiently general probability distribution model. We consider piecewise-constant distributions ( ) with uniform (second-order) prior over location of discontinuity points and assigned chances. The predictive distribution and the model complexity can be determined completely from the data in a computational time that is linear in the number of degrees of freedom and quadratic in the number of possible values of . Furthermore, exact
more » ... of the expectations of entropies and their variances can be computed with polynomial effort. The expectation of the mutual information becomes thus available, too, and a strict upper bound on its variance. The resulting algorithm is particularly useful in experimental research areas where the number of available samples is severely limited (e.g., neurophysiology). Estimates on a simulated data set provide more accurate results than using a previously proposed method.
doi:10.1109/tit.2005.856954 fatcat:3xr44pzdjbfwzn33lfbvrwapna