Statistical Reviewing forHeadache

Timothy T. Houle, Donald B. Penzien
2009 Headache  
The process of reviewing the statistical analyses in a manuscript is a daunting one. There is an everchanging list of statistical tests, procedures, and best practices. The available statistical software to conduct analyses is similarly changing and many reviewers are probably finding it more and more difficult to interpret the presented data in a manuscript. Perhaps it is no wonder, then, that reviewers often neglect to comment on the statistical analyses of a manuscript, or provide comments
more » ... at do not serve to improve the scientific product. The difficulties in statistical reviewing have been well documented, 1 but a looming question remains: What can our research community do about improving the statistical aspects of peer review? Examination of publication practices reveal that changes in the behavior of reviewers can be difficult to initiate and maintain. Nevertheless, efforts at enhancing statistical peer review must be made if we are to see improvements in this important reviewing issue. The editorial staff of Headache has developed a new set of reviewer guidelines to assist reviewers in formulating constructive criticisms of submitted manuscripts. These guidelines also can provide a valuable resource for authors as they work to prepare manuscripts for submission. The purpose of this editorial is to introduce a statistical reviewing checklist that is embedded within these guidelines. The statistical checklist is intended to serve as an initial step in assisting reviewers for Headache to formulate basic criticisms of the statistical reporting and design of submitted manuscripts. REVIEWING STATISTICAL REPORTING In assessing the statistical quality of a manuscript, a reviewer can prudently focus on one overarching review question: Have the statistical methods been presented in sufficient detail such that they could be replicated? Too often, statistical methods are presented in insufficient detail. This leads to a scenario wherein a reader cannot focus on the presented data because of uncertainty that the methods were properly conducted. Good statistical reporting allows the actual data to be the focus of the manuscript. Table 1 presents basic guidelines that, if satisfied, will lead to improved statistical reporting. Reviewers are encouraged to submit inquiries to authors if any of these reporting issues have not been adequately addressed. Reporting the results of statistical tests is a crucial element of statistical reporting. Too often, authors rely on reporting P values outside the context of information concerning the effect sizes of the observed differences. 2,3 Relying too heavily on significance testing may not allow a manuscript to reach its full potential, for to properly interpret a P value many other statistical reporting elements must also be presented. Figure 1 provides a simple guiding principle regarding the information value of statistical results based on the question: What do these differences (effects) actually convey? Addressing this question through good reporting is crucial as a statistically significant effect can accrue in the absence of a large
doi:10.1111/j.1526-4610.2008.01322.x fatcat:337gkxmupjblxfceq557dyjzla