Software Abnormal Behavior Detection Based on Function Semantic Tree

Yingxu LAI, Wenwen ZHANG, Zhen YANG
2015 IEICE transactions on information and systems  
Current software behavior models lack the ability to conduct semantic analysis. We propose a new model to detect abnormal behaviors based on a function semantic tree. First, a software behavior model in terms of state graph and software function is developed. Next, anomaly detection based on the model is conducted in two main steps: calculating deviation density of suspicious behaviors by comparison with state graph and detecting function sequence by function semantic rules. Deviation density
more » ... Deviation density can well detect control flow attacks by a deviation factor and a period division. In addition, with the help of semantic analysis, function semantic rules can accurately detect application layer attacks that fail in traditional approaches. Finally, a case study of RSS software illustrates how our approach works. Case study and a contrast experiment have shown that our model has strong expressivity and detection ability, which outperforms traditional behavior models. key words: software behavior, system call, state graph, semantic analysis, deviation density, function semantic rules Yingxu Lai received Ph.D from Chinese Academy of Sciences in 2003, M.S. degree from Beijing University of Chemical and Technology in 1997, B.S. degree from Shenyang Institute of Chemical and Technology in 1994. Now she is an associate professor at the College of Computer Science, Beijing University of Technology. Her research interest covers information network security and trusted computing. Wenwen Zhang received her M.S. degree from Beijing University of Technology in 2013 and B.S. degree from Nanjing University of Posts and Telecommunications in 2010. Her research interests include information security, trusted computing and software behavior analysis. Zhen Yang is an associate professor at the College of Computer Science, Beijing University of Technology. His research interest covers information content security, public opinion analysis, and trusted computing.
doi:10.1587/transinf.2015edp7098 fatcat:zxtmf7dpe5cm7plptcvkszlos4