A General Framework for Mining Concept-Drifting Data Streams with Skewed Distributions [chapter]

Jing Gao, Wei Fan, Jiawei Han, Philip S. Yu
2007 Proceedings of the 2007 SIAM International Conference on Data Mining  
In recent years, there have been some interesting studies on predictive modeling in data streams. However, most such studies assume relatively balanced and stable data streams but cannot handle well rather skewed (e.g., few positives but lots of negatives) and stochastic distributions, which are typical in many data stream applications. In this paper, we propose a new approach to mine data streams by estimating reliable posterior probabilities using an ensemble of models to match the
more » ... atch the distribution over under-samples of negatives and repeated samples of positives. We formally show some interesting and important properties of the proposed framework, e.g., reliability of estimated probabilities on skewed positive class, accuracy of estimated probabilities, efficiency and scalability. Experiments are performed on several synthetic as well as real-world datasets with skewed distributions, and they demonstrate that our framework has substantial advantages over existing approaches in estimation reliability and predication accuracy.
doi:10.1137/1.9781611972771.1 dblp:conf/sdm/GaoFHY07 fatcat:6ips4ugs2nfwjncopqk27tgn34