On the infeasibility of modeling polymorphic shellcode

Yingbo Song, Michael E. Locasto, Angelos Stavrou, Angelos D. Keromytis, Salvatore J. Stolfo
2007 Proceedings of the 14th ACM conference on Computer and communications security - CCS '07  
Current trends demonstrate an increasing use of polymorphism by attackers to disguise their exploits. The ability for malicious code to be easily, and automatically, transformed into semantically equivalent variants frustrates attempts to construct simple, easily verifiable representations for use in security sensors. In this paper, we present a quantitative analysis of the strengths and limitations of shellcode polymorphism, and describe the impact that these techniques have in the context of
more » ... earning-based IDS systems. Our examination focuses on dual problems: shellcode encryption-based evasion methods and targeted "blending" attacks. Both techniques are currently being used in the wild, allowing real exploits to evade IDS sensors. This paper provides metrics to measure the effectiveness of modern polymorphic engines and provide insights into their designs. We describe methods to evade statistics-based IDS sensors and present suggestions on how to defend against them. Our experimental results illustrate that the challenge of modeling self-modifying shellcode Editors: Pavel Laskov and Richard Lippmann.
doi:10.1145/1315245.1315312 dblp:conf/ccs/SongLSKS07 fatcat:ebnrlydq6ncttbajrrz3icuroe