A framework for diagnosing changes in evolving data streams

Charu C. Aggarwal
2003 Proceedings of the 2003 ACM SIGMOD international conference on on Management of data - SIGMOD '03  
In recent years, the progress in hardware technology has made it possible for organizations to store and record large streams of transactional data. This results in databases which grow without limit at a rapid rate. This data can often show important c hanges in trends over time. In such cases, it is useful to understand, visualize and diagnose the evolution of these trends. When the data streams are fast and continuous, it becomes important to analyze and predict the trends quickly in online
more » ... quickly in online fashion. In this paper, we discuss the concept of velocity density estimation, a technique used to understand, visualize and determine trends in the evolution of fast data streams. We show h o w to use velocity density estimation in order to create both temporal velocity pro les and spatial velocity pro les at periodic instants in time. These pro les are then used in order to predict three kinds of data evolution: dissolution, coagulation and shift. Methods are proposed to visualize the changing data trends in a single online scan of the data stream, and a computational requirement which is linear in the number of data points. In addition, batch processing techniques are proposed in order to identify combinations of dimensions which show the greatest amount o f global evolution. The techniques discussed in this paper can be easily extended to spatio-temporal data, changes in data snapshots at xed instances in time, or any other data which has a temporal component during its evolution.
doi:10.1145/872824.872826 fatcat:jca5gfgxzvddbgtowfnhadgis4