Making Heads and Tails of Models with Marginal Calibration for Sparse Tagsets [article]

Michael Kranzlein, Nelson F. Liu, Nathan Schneider
2021 arXiv   pre-print
For interpreting the behavior of a probabilistic model, it is useful to measure a model's calibration--the extent to which it produces reliable confidence scores. We address the open problem of calibration for tagging models with sparse tagsets, and recommend strategies to measure and reduce calibration error (CE) in such models. We show that several post-hoc recalibration techniques all reduce calibration error across the marginal distribution for two existing sequence taggers. Moreover, we
more » ... pose tag frequency grouping (TFG) as a way to measure calibration error in different frequency bands. Further, recalibrating each group separately promotes a more equitable reduction of calibration error across the tag frequency spectrum.
arXiv:2109.07494v1 fatcat:kpxu5vpdfne5loqwja6nij3l3a