Bounds for Predictive Errors in the Statistical Mechanics of Supervised Learning

Manfred Opper, David Haussler
1995 Physical Review Letters  
Within a Bayesian framework, by generalizing inequalities known from statistical mechanics, we calculate general upper and lower bounds for a cumulative entropic error, which measures the success in the supervised learning of an unknown rule from examples. Both bounds match asymptotically, when the number m of observed data grows large. We find that the information gain from observing a new example decreases universally like d/m. Here d is a dimension that is defined from the scaling of small
more » ... scaling of small volumes with respect to a distance in the space of rules.
doi:10.1103/physrevlett.75.3772 pmid:10059723 fatcat:g7yofh4tfbadfj6dxbn6m2xday