Hyperparameters Evidence and Generalisation for an Unrealisable Rule

Glenn Marion, David Saad
1994 Neural Information Processing Systems  
Using a statistical mechanical formalism we calculate the evidence, generalisation error and consistency measure for a linear perceptron trained and tested on a set of examples generated by a non linear teacher. The teacher is said to be unrealisable because the student can never model it without error. Our model allows us to interpolate between the known case of a linear teacher, and an unrealisable, nonlinear teacher. A comparison of the hyperparameters which maximise the evidence with those
more » ... hat optimise the performance measures reveals that, in the non-linear case, the evidence procedure is a misleading guide to optimising performance. Finally, we explore the extent to which the evidence procedure is unreliable and find that, despite being sub-optimal, in some circumstances it might be a useful method for fixing the hyperparameters.
dblp:conf/nips/MarionS94 fatcat:nl2aza4gb5ccrnwgfcbmcrmd3e