Towards anytime active learning

Maria E. Ramirez-Loaiza, Aron Culotta, Mustafa Bilgic
2013 Proceedings of the ACM SIGKDD Workshop on Interactive Data Exploration and Analytics - IDEA '13  
Many active learning methods use annotation cost or expert quality as part of their framework to select the best data for annotation. While these methods model expert quality, availability, or expertise, they have no direct influence on any of these elements. We present a novel framework built upon decision-theoretic active learning that allows the learner to directly control label quality by allocating a time budget to each annotation. We show that our method is able to improve performance
more » ... ciency of the active learner through an interruption mechanism trading off the induced error with the cost of annotation. Our simulation experiments on three document classification tasks show that some interruption is almost always better than none, but that the optimal interruption time varies by dataset.
doi:10.1145/2501511.2501524 dblp:conf/kdd/Ramirez-LoaizaC13 fatcat:nvcanpnmynb7vivyc65rd6qpt4