Experimental program analysis

Joseph R. Ruthruff, Sebastian Elbaum, Gregg Rothermel
2006 Proceedings of the 2006 international symposium on Software testing and analysis - ISSTA'06  
Program analysis techniques are used by software engineers to deduce and infer characteristics of software systems. Recent research has suggested that certain program analysis techniques can be formulated as formal experiments. This article reports the results of research exploring this suggestion. Building on principles and methodologies underlying the use of experimentation in other fields, we provide descriptive and operational definitions of experimental program analysis, illustrate them by
more » ... example, and describe several differences between experimental program analysis and experimentation in other fields. We also explore the applicability of experimental program analysis to three software engineering problems: program transformation, program debugging, and program understanding. Our findings indicate that experimental program analysis techniques can provide new and potentially improved solutions to these problems, and suggest that experimental program analysis offers a promising new direction for program analysis research. Program analysis techniques analyze software systems to collect, deduce, or infer specific information about those systems. The resulting information typically involves system properties and attributes such as data dependencies, control dependencies, invariants, anomalous behavior, reliability, or conformance to specifications. This information supports various software engineering activities such as testing, fault localization, impact analysis, and program understanding. Researchers who are investigating these and other activities continue to seek new program analysis techniques that can address software engineering problems costeffectively, and continue to seek ways to improve existing techniques. Researchers for some time have harnessed principles of experimentation to aid program analysis techniques (e.g., [6, 15, 35, 49] .) Zeller recognized that such program analysis techniques might be able to establish causality relationships between system variables of interest in cases in which standard analyses have not succeeded [50] . We further argued that such techniques might also be able to more cost-effectively draw inferences about properties of software systems to characterize them [35] . Anyone who has spent time debugging a program will recognize characteristics that are experimental in nature. Debuggers routinely form hypotheses about the causes of failures, conduct program runs (in * This article is a revised and expanded version of a paper presented at 1 which factors that might affect the run other than the effect being investigated are controlled) to confirm or reject these hypotheses, and based on the results of these "experiments," draw conclusions or create new hypotheses about the cause of the fault. The "experimental" nature of this approach is reflected (in whole or in part) in existing program analysis techniques aimed at fault localization and debugging. For example, Howcome [49] is a tool intended to help engineers localize the cause of an observed program failure f in a failing execution e f . Howcome attempts to isolate the minimal relevant variable value differences in program states in order to create "cause-effect chains" describing why f occurred. To do this, Howcome conducts an experiment where a subset of e f 's variable values are applied to the corresponding variables in a passing execution e p to "test" an hypothesis regarding whether the applied changes reproduce f . If e p with the applied value subset "fails" this test (by not reproducing f ), then a different value subset treatment is tested. If the subset "passes" the test (by reproducing f ), then a subset of those incriminating variable values is considered. This process continues exploring different hypotheses until no further subsets can be formed or can reproduce the failure. While the use of principles of experimentation by program analysis techniques has increased in recent years, there remain many approaches by which techniques could draw on research in the statistical and experiment design literature to improve their efficiency or effectiveness. These approaches include (1) methodologies for manipulating independent variables of interest to test their effects on dependent variables, (2) procedures for conducting and adjusting hypothesis tests in program analysis contexts, (3) strategies for systematically controlling sources of variation during these tests, (4) experiment designs and sampling techniques to reduce the costs of experimentation, and (5) mechanisms to generate confidence measures in the reliability and validity of the results. To date, however, the opportunities offered by such approaches have not been pursued rigorously, with the support of a theory of experimental program analysis, by the program analysis research community. As a result, we believe that many program analysis techniques have not advanced to the degree that they could have. This article takes a first step in formalizing an experimental program analysis paradigm, and demonstrating the potential utility of experimental program analysis, through three contributions. First, we argue that a class of program analysis approaches exists whose members are inherently experimental in nature. By this, we mean that these techniques can be characterized in terms of guidelines and methodologies defined and practiced within the long-established paradigm of experimentation. Building on this characterization, we present an operational definition of a paradigm for experimental program analysis, and we show how analysis techniques can be characterized in terms of this paradigm. Second, we demonstrate how our formalization of the experimental program analysis paradigm can help researchers identify limitations of analysis techniques, improve existing program analysis techniques, and create new experimental program analysis techniques. We also show that techniques following this paradigm
doi:10.1145/1146238.1146245 dblp:conf/issta/RuthruffER06 fatcat:fq7fh7pjwvfftkxmkprvikrsue