Permutation P-values for Inter-rater Agreement Measures

Yonghwan Um
2015 Journal of the Korea Society of Computer and Information  
Permutation p-values are provided for the agreement measures for multivariate interval data among many raters. Three agreement measures, Berry and Mielke's measure, Janson and Olsson's measure, and Um's measure are described and compared. Exact and resampling permutation methods are utilized to compute p-values and empirical quantile limits for three measures. Comparisons of p-values demonstrate that resampling permutation methods provide close approximations to exact p-values, and Berry and
more » ... s, and Berry and Mielke's measure and Um's measure show similar performance in terms of measuring agreement.
doi:10.9708/jksci.2015.20.12.169 fatcat:r4oytfz46bcc7aopz4oaxyrkka