Practical pairwise testing for software product lines

Dusica Marijan, Arnaud Gotlieb, Sagar Sen, Aymeric Hervieu
2013 Proceedings of the 17th International Software Product Line Conference on - SPLC '13  
One key challenge for software product lines is efficiently managing variability throughout their lifecycle. In this paper, we address the problem of variability in software product lines testing. We (1) identify a set of issues that must be addressed to make software product line testing work in practice and (2) provide a framework that combines a set of techniques to solve these issues. The framework integrates feature modelling, combinatorial interaction testing and constraint programming
more » ... hniques. First, we extract variability in a software product line as a feature model with specified feature interdependencies. We then employ an algorithm that generates a minimal set of valid test cases covering all 2-way feature interactions for a given time interval. Furthermore, we evaluate the framework on an industrial SPL and show that using the framework saves time and provides better test coverage. In particular, our experiments show that the framework improves industrial testing practice in terms of (i) 17% smaller set of test cases that are (a) valid and (b) guarantee all 2-way feature coverage (as opposite to 19.2% 2-way feature coverage in the hand made test set), and (ii) full flexibility and adjustment of test generation to available testing time. Software product lines (SPLs) allow companies to efficiently increase the range of products diversity imposed by a diverse user base. SPL represents a set of similar products developed from a common core components and with some variations in functionality. Therefore, instead of developing a collection of single customized (but similar) products, we can mass-customize products by exploiting their commonalities and maximizing reusable variation through a product line. This approach brings benefits in terms of higher productivity, shorter time to market and cost reduction, but also challenges in managing variability throughout the whole product line lifecycle. This paper focuses on managing variability in test case generation for SPL. Some companies have structured parts of their software development following the SPL concept, with software testing activities still mainly based on traditional techniques that are suited for single product development. When employed in testing SPL, such techniques and tools reveal scalability problems and high inefficiency beyond a certain number of product variants. Experienced test engineers manually create test configurations of mutually consistent software components leading to a suite of functional software tailored for different user bases. While such manual process is highly time consuming, projects are normally planned with strict schedules, imposing a strict time bound for testing activities. Any test case exceeding this bound is missed. Therefore, the objective of test engineers is to specify the smallest number of test cases in available amount of time such that specific coverage criteria are satisfied (for example, all two software feature interactions are tested) and that all test configurations are valid (all dependencies between features are satisfied). Given a very large configuration space, this manual task is extremely tedious and unsystematic, leading often to missed deadlines, insufficient test coverage and redundancy in test cases. In our industrial setting we have talked to engineers and have identified three main challenges for making SPL testing practical. First, the time that is provided for testing is inflexible -testing must be completed on time. Second, tests often contain invalid software configurations that cause failures in execution. Third, there is a lack of measurable test coverage criteria. In this work we provide a framework that addresses each of these limitations and is effective in practice. The framework (1) utilizes an anytime algorithm to balance the tradeoff between a test generation time and a number of test cases, (2) allows automated checking for validity of test configurations, (3) leverages combinatorial testing to increase test coverage, and (4) provides systematic training for engineers to make modeling easier and more accurate. The first step is manual extraction of variability in SPL. We present a methodology to identify the variation points in the SPL and construct the model of variability called a feature model. An instance of the feature model corresponds to a configuration of the SPL. We use our algorithm to automatically generate the minimal set of configurations that cover all pairwise interactions and satisfy all interdependencies between features of the feature model. The algorithm utilizes constraint optimization techniques to generate the set of test configurations of minimal size for defined time interval. We experimentally evaluate the framework on the industrial SPL and demonstrate the improvement over industrial testing practice. The contributions of the paper can be summarized as follows: • We identify a set of issues that must be addressed to make SPL testing work in practice. • We provide a SPL modeling methodology and training that helps practitioners to extract and abstract variability in a SPL and specify it formally as a feature model with
doi:10.1145/2491627.2491646 dblp:conf/splc/MarijanGSH13 fatcat:27xmygpuc5d4zjc7oe7vavqbqa