Inference and analysis of formal models of botnet command and control protocols

Chia Yuan Cho, Domagoj Babi ć, Eui Chul Richard Shin, Dawn Song
2010 Proceedings of the 17th ACM conference on Computer and communications security - CCS '10  
We propose a novel approach to infer protocol state machines in the realistic high-latency network setting, and apply it to the analysis of botnet Command and Control (C&C) protocols. Our proposed techniques enable an order of magnitude reduction in the number of queries and time needed to learn a botnet C&C protocol compared to classic algorithms (from days to hours for inferring the MegaD C&C protocol). We also show that the computed protocol state machines enable formal analysis for botnet
more » ... fense, including finding the weakest links in a protocol, uncovering protocol design flaws, inferring the existence of unobservable communication back-channels among botnet servers, and finding deviations of protocol implementations which can be used for fingerprinting. We validate our technique by inferring the protocol state-machine from Postfix's SMTP implementation and comparing the inferred statemachine to the SMTP standard. Further, our experimental results offer new insights into MegaD's C&C, showing our technique can be used as a powerful tool for defense against botnets.
doi:10.1145/1866307.1866355 dblp:conf/ccs/ChocSS10 fatcat:sj4uh3bjfrdl3oeh4jlgk6ev3e