Local asymptotic coding and the minimum description length

D.P. Foster, R.A. Stine
1999 IEEE Transactions on Information Theory  
Common approximations for the minimum description length (MDL) criterion imply that the cost of adding a parameter to a model fit to n observations is about (1/2) log n bits. While effective for parameters which are large on a standardized scale, this approximation overstates the parameter cost near zero. A uniform approximation and local asymptotic argument show that the addition of a small parameter which is about two standard errors away from zero produces a model whose description length is
more » ... shorter than that of the comparable model which sets this parameter to zero. This result implies that the decision rule for adding a model parameter is comparable to a traditional statistical hypothesis test. Encoding the parameter produces a shorter description length when the corresponding estimator is about two standard errors away from zero, unlike a model selection criterion like BIC whose threshold increases logarithmically in n. Abstract Common approximations for the minimum description length (MDL) criterion imply that the cost of adding a parameter to a model fit to n observations is about (1/2) log n bits. While effective for parameters which are large on a standardized scale, this approximation overstates the parameter cost near zero. A uniform approximation and local asymptotic argument show that the addition of a small parameter which is about two standard errors away from zero produces a model whose description length is shorter than that of the comparable model which sets this parameter to zero. This result implies that the decision rule for adding a model parameter is comparable to a traditional statistical hypothesis test. Encoding the parameter produces a shorter description length when the corresponding estimator is about two standard errors away from zero, unlike a model selection criterion like BIC whose threshold increases logarithmically in n. Local Asymptotics and MDL 2
doi:10.1109/18.761287 fatcat:7ndhb3yslnetdhbr4gsjhmwcpy