Online Reconstruction of Structural Information from Datacenter Logs

Zaheer Chothia, John Liagouris, Desislava Dimitrova, Timothy Roscoe
2017 Proceedings of the Twelfth European Conference on Computer Systems - EuroSys '17  
Well-run datacenter application architectures are heavily instrumented to provide detailed traces of messages and remote invocations. Reconstructing user sessions, call graphs, transaction trees, and other structural information from these messages, a process known as sessionization, is the foundation for a variety of diagnostic, profiling, and monitoring tasks essential to the operation of the datacenter. We present the design and implementation of a system which processes log streams at
more » ... ts per second and reconstructs user sessions comprising millions of transactions per second in real time with modest compute resources, while dealing with clock skew, message loss, and other real-world phenomena that make such a task challenging. Our system is based on the Timely Dataflow framework for low latency, data-parallel computation, and we demonstrate its utility with a number of use-cases and traces from a large, operational, mission-critical enterprise data center.
doi:10.1145/3064176.3064195 dblp:conf/eurosys/ChothiaLDR17 fatcat:ohlqwf3bgbg3nbqrtnwlutudpe