Challenges of Reliable, Realistic and Comparable Active Learning Evaluation

Daniel Kottke, Adrian Calma, Denis Huseljic, Georg Krempl, Bernhard Sick
2017 European Conference on Principles of Data Mining and Knowledge Discovery  
Active learning has the potential to save costs by intelligent use of resources in form of some expert's knowledge. Nevertheless, these methods are still not established in real-world applications as they can not be evaluated properly in the specific scenario because evaluation data is missing. In this article, we provide a summary of different evaluation methodologies by discussing them in terms of being reproducible, comparable, and realistic. A pilot study which compares the results of
more » ... ent exhaustive evaluations suggests a lack in repetitions in many articles. Furthermore, we aim to start a discussion on a gold standard evaluation setup for active learning that ensures comparability without reimplementing algorithms.
dblp:conf/pkdd/KottkeCHKS17 fatcat:gb33hjl625gjdg6wa6rivfwhwy