Algorithms for Active Classifier Selection

Paul N. Bennett, David M. Chickering, Christopher Meek, Xiaojin Zhu
2017 Proceedings of the Tenth ACM International Conference on Web Search and Data Mining - WSDM '17  
Software applications often use classification models to trigger specialized experiences for users. Search engines, for example, use query classifiers to trigger specialized "instant answer" experiences where information satisfying the user query is shown directly on the result page, and email applications use classification models to automatically move messages to a spam folder. When such applications have acceptable default (i.e., non-specialized) behavior, users are often more sensitive to
more » ... ilures in model precision than failures in model recall. In this paper, we consider model-selection algorithms for these precision-constrained scenarios. We develop adaptive model-selection algorithms to identify, using as few samples as possible, the best classifier from among a set of (precision) qualifying classifiers. We provide statistical correctness and sample complexity guarantees for our algorithms. We show with an empirical validation that our algorithms work well in practice.
doi:10.1145/3018661.3018730 dblp:conf/wsdm/BennettCMZ17 fatcat:ged2kupzc5bhhdmyqezeh5s6ii