JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org. . American Statistical Association is collaborating with JSTOR to digitize, preserve and extend access to Journal of the American Statistical Association. We

more »

... estigate the compatibility of a null model Ho with the data by calculating a p value; that is, the probability, under Ho, that a given test statistic T exceeds its observed value. When the null model consists of a single distribution, the p value is readily obtained, and it has a uniform distribution under Ho. On the other hand, when the null model depends on an unknown nuisance parameter 0, one must somehow get rid of 0, (e.g., by estimating it) to calculate a p value. Various proposals have been suggested to "remove" 0, each yielding a different candidate p value. But unlike the simple case, these p values typically are not uniformly distributed under the null model. In this article we investigate their asymptotic distribution under Ho. We show that when the asymptotic mean of the test statistic T depends on 0, the posterior predictive p value of Guttman and Rubin, and the plug-in p value are conservative (i.e., their asymptotic distributions are more concentrated around 1/2 than a uniform), with the posterior predictive p value being the more conservative. In contrast, the partial posterior predictive and conditional predictive p values of Bayarri and Berger are asymptotically uniform. Furthermore, we show that the discrepancy p value of Meng and Gelman and colleagues can be conservative, even when the discrepancy measure has mean 0 under the null model. We also describe ways to modify the conservative p values to make their distributions asymptotically uniform.