An evaluation framework for software test processes
Existing assessment and improvement models of software testing process mainly intend to raise maturity of an organization with reference to testing activities. Such process assessments are based on what testing activities are being carried out, and thus implicitly evaluate process quality. Other test process measurement techniques attempt to directly assess some partial quality attribute such as efficiency or effectiveness using few test measurements. There exists a potential for a formalized
... thod of evaluating test process quality that addresses both implicity and partially of these current evaluations. The first part of this dissertation surveys and analyzes breadth and depth of existing evaluation approaches in the area of software testing. Strengths and weaknesses of these approaches are highlighted to arrive at a set of requirements for an improved solution. On the other hand, to investigate the scientific and philosophical foundations of evaluation, a short study and analysis into the discipline of evaluation is also made in this part. This research helps identify salient features of an improved test evaluation approach. Building upon the first phase of research, the second part of the dissertation presents an evaluation framework for specifying and evaluating diverse quality aspects of software test processes. The framework comprises two types of components. Five informative components describe evaluation aspects of testing process, a quality model, a yardstick test process, test metrics, and synthesis techniques. These informative components support a sixth (core) component called evaluation process which details all the steps of test evaluations. The informative components are founded on the theory of evaluation and research into test measurement and evaluation, while the core component derives its concepts from relevant international standards. Combination of this theory, research and best practices helps create a comprehensive test evaluation approach. Finally, the presented approach is exemplified for a domain specific testing approach, i.e., testing process for service-oriented systems. The main contribution of this dissertation lies in its ability to combine several quality aspects of test processes and, furthermore, in providing an explicit evaluation approach. The developed framework complements the existing maturity models of software testing process by providing a solution that fulfils a subset of requirements at maturity level 4 of TMM/CMMI models.