Towards a hybrid testing process unifying exploratory testing and scripted testing

Syed Muhammad Ali Shah, Cigdem Gencel, Usman Sattar Alvi, Kai Petersen
2013 Journal of Software: Evolution and Process  
Given the current state of the art in research, practitioners are faced with the challenge of choosing scripted testing (ST) or exploratory testing (ET). This study aims at systematically incorporating strengths of ET and ST in a hybrid testing process to overcome the weaknesses of each. We utilized systematic review and practitioner interviews to identify strengths and weaknesses of ET and ST. Strengths of ET were mapped to weaknesses of ST and vice versa. Noblit and Hare's lines-ofargument
more » ... hod was used for data analysis. The results of the mapping were used as input to codesign a hybrid process with experienced practitioners. We found a clear need to create a hybrid process as follows: (i) both ST and ET provide strengths and weaknesses, and these depend on some particular conditions, which prevents preference of one approach to another; and (ii) the mapping showed that it is possible to address the weaknesses in one process by the strengths of the other in a hybrid form. With the input from literature and industry experts, a flexible and iterative hybrid process was designed. Practitioners can clearly benefit from using a hybrid process given the mapping of advantages and disadvantages. On the other hand, in ET, the tests are not defined in advance in an established test plan but are dynamically designed, executed, and modified [9] . Exploratory testing is also referred to as ad hoc testing [1] as it relies on the implicit and informal understanding of the testers. Because the literal meaning of ad hoc may correspond to sloppy and careless work, the term 'exploratory' was introduced by a group of experts instead of 'ad hoc' [6] . As the testers can freely explore an application by utilizing human intuition and experience [6, 40] , and it is not explicit how they make this exploration, the tasks are performed manually rather than with an automation support. Scripted testing and ET provide various benefits and weaknesses (Section 2). A few studies [12, 24, 25] mentioned that ET makes better use of testers' creativity and skills to discover the bugs that prescriptive testing may not uncover because of its mechanical nature. Agruss and Johnson [1] and Bach [6] claimed that software testing might benefit through using these approaches in combination. In general, there is a general interest in industry for a hybrid testing (HT) approach unifying the two approaches, which is, for example, visible in lively discussions in industry oriented blogs (see e.g., [43] ). In this study, our aim is to address the need for a systematic and repeatable investigation of such a hybrid process. To this end, we first explored the weaknesses and strengths of ST and ET by reviewing the literature and getting feedback from industry. Then, based on the signified findings by comparing the two approaches, we propose an HT process that unifies ET and ST in a way that some major weaknesses of ET and ST are minimized in a compromise form. With these objectives, we formulated the research questions (RQs) for this study as follows: • RQ1: What are the strengths of ST and ET? • RQ2: What are the weaknesses of ST and ET? • RQ3: What are the improvement opportunities for testing process by addressing some major weaknesses of ST and ET through unifying their processes in a hybrid form? It is important to point out that this paper does not focus on individual testing techniques that can be used within the testing process. For example, common testing techniques in ST for black-box testing include, boundary value analysis [32], equivalence partitioning [32], and decision tables [39]. For ST, the commonly used white-box testing techniques include decision coverage [3], path coverage [3], multiple condition/decision coverage (MC/DC) [15], and data flow coverage [10]. One example of a technique in ET is smoke testing [25] . However, instead, our focus here is on the overall 'testing process' that fulfills the characteristics of ST and ET mentioned previously. In order to answer our RQs, we used systematic literature review (SLR) ([31]) and interviews as the main research methods. Our research process is shown in Figure 1 and was inspired by the technology transfer model proposed by Gorschek et al. [18]. Our work starts off with the clear contrast between ET and ST. Consequently, companies could make conscious decisions on which process to choose based on evidence. This implies understanding the strengths and weaknesses of the approaches that are reported in the literature (see problems and issues in Figure 1) . Hence, the first phase (P1) of this exploratory research was investigating the strengths and weaknesses of ST and ET (P1 in Figure 1 ). Furthermore, we interviewed practitioners with extensive experience of ET and ST in order to identify their perspective on strengths and weaknesses and then compared the outcomes of the interviews to those of the literature review. Through interviews, we also could identify the connections between the strengths and weaknesses of ST and ET that later on helped in identifying the improvement opportunities for an HT process. The details of this step are given in Section 2. After having identified strengths and weaknesses, we mapped the strengths of one process to the weaknesses of the other and vice versa (P2). Practitioners with extensive experience in both HT and ST were involved in this mapping. They also reviewed the final mapping to improve the reliability of the results. The outcome of P1 and P2 provided two major results that are helpful in working towards an HT: (i) clearly establishing the need for an HT; and (ii) knowing how the strengths and weaknesses of ET and ST relate to each others' help in (i) connecting them to the activities of the HT process to check whether weaknesses are addressed and strengths are supported; and in (ii) providing input to questions to be asked when evaluating an HT. The details of this step are given in Section 3. • Test environment setup: For HT, there is a freedom for the selection of test environment. Based on the test case design and implementation, the test environment in which the test will be executed is established and maintained. • Test execution: Both RBTC and the test missions are executed, which were designed in test design phase. First, a tester has given the freedom to freely explore the application in order to learn and obtain knowledge about it. After that, RBTC and then the test missions are executed, and the execution artifacts are recorded. A session is a particular time slot assigned to a specific test Figure 3 . Process of hybrid testing.
doi:10.1002/smr.1621 fatcat:g7fm6cjv75cl7jet6ynhbshw6m