Detecting Communication Protocol Security Flaws by Formal Fuzz Testing and Machine Learning [chapter]

Guoqiang Shu, Yating Hsu, David Lee
2008 Lecture Notes in Computer Science  
Network-based fuzz testing has become an effective mechanism to ensure the security and reliability of communication protocol systems. However, fuzz testing is still conducted in an ad-hoc manner with considerable manual effort, which is mainly due to the unavailability of protocol model. In this paper we present our ongoing work of developing an automated and measurable protocol fuzz testing approach that uses a formally synthesized approximate formal protocol specification to guide the
more » ... process. We adopt the Finite State Machine protocol model and study two formal methods for protocol synthesis: an active black-box checking algorithm that has provable optimality and a passive trace minimization algorithm that is less accurate but much more efficient. We also present our preliminary results of using this method to implementations of the MSN instant messaging protocol: MSN clients Gaim (pidgin) and aMSN. Our testing reveals some serious reliability and security flaws by automatically crashing both of them.
doi:10.1007/978-3-540-68855-6_19 fatcat:rwlvzbbxwfeulk4qrrlc373rqm