All rights reserved. iii DEDICATION I dedicate this dissertation to my family, loved ones and friends for giving me strength, patience, emotional balance and fortitude during the process of achieving this milestone in my life. iv ACKNOWLEDGMENTS First and foremost I wish to thank Dr. Yi Deng for taking the risk of believing in me and supporting me throughout my time as a graduate student. Special thanks to Dr. Peter J. Clarke for giving endless hours of time to my work providing invaluable

more »

... ack, comments, support and most importantly balance, strength and several cups of coffee. An oversees thanks to Dr. Eric Filiol for his countless reviews of my work and support of my research. A huge thanks and deep appreciation to Mario Consuegra for giving selflessly of his busy schedule to take the lead role in creating my implementation prototypes. A great thanks to the CIS faculty, especially Dr. Fast spreading unknown viruses have caused major damage on computer systems upon their initial release. Current detection methods have lacked capabilities to detect unknown viruses quickly enough to avoid mass spreading and damage. This dissertation has presented a behavior based approach to detecting known and unknown viruses based on their attempt to replicate. Replication is the qualifying fundamental characteristic of a virus and is consistently present in all viruses making this approach applicable to viruses belonging to many classes and executing under several conditions. A form of replication called self-reference replication, (SRreplication), has been formalized as one main type of replication which specifically replicates by modifying or creating other files on a system to include the virus itself. This replication type was used to detect viruses attempting replication by referencing themselves which is a necessary step to successfully replicate files. The approach does not require a priori knowledge about known viruses. Detection was accomplished at runtime by monitoring currently executing processes attempting to replicate. Two implementation prototypes of the detection approach called SRRAT were created and tested on the Microsoft Windows operating systems focusing on the tracking of user mode Win32 API system calls and Kernel mode system services. The research results showed SR-replication capable of distinguishing between file infecting viruses and benign processes with little or no false positives and false negatives. vi TABLE OF CONTENTS CHAPTER PAGE