Tool support for the Test Template Framework

Maximiliano Cristiá, Pablo Albertengo, Claudia Frydman, Brian Plüss, Pablo Rodríguez Monetti
2012 Software testing, verification & reliability  
This paper describes tool support that has been implemented for the Test Template Framework (TTF). The TTF is a model-based testing (MBT) method specially well-suited for unit testing from Z specifications. Although the TTF is a sound MBT method and it has been widely referenced since its first publication, attention in recent years has decayed. In fact, some have argued that generating abstract test cases following the TTF is a manual task requiring that its users perform complex predicate
more » ... pulations. This paper shows that these observations are dubious by describing Fastest, a tool that implements solutions for all these issues and, according to many experiments, produces abstract test cases for more than 80% of the satisfiable test specifications. Furthermore, it is claimed that Fastest fulfils the needs of the Z user community regarding MBT tools, which is supported with a range of case studies. Abstract output Verification Refinement Abstraction Programming Figure 1. A general description of the MBT process. On the other hand, Z users find the TTF appealing and sound. Particularly, they find a great advantage in the TTF since all the MBT main artefacts-test specifications, abstract test cases, etc.-can be expressed in the same notation of the model, i.e. the Z formal notation. After studying the TTF, the authors concluded that it might be automated roughly to the same level of any other MBT method. Furthermore, providing tool support for it would contribute to fill two gaps: (a) MBT tools approaching unit testing, and (b) a MBT tool for the Z user community. The result of this work is a tool, called Fastest, that implements the TTF, allowing users to semiautomatically produce abstract test cases from Z specifications, which, in turn, enables the semiautomation of other tasks, like translating test cases into natural language. In this paper, an integral presentation of previous work [10, 11, 12] is made and new results are introduced as well. According to the TTF: (i) test specifications are assembled into so-called testing trees, generated by applying testing tactics; (ii) testing trees should be pruned to avoid unsatisfiable test specifications; and (iii) abstract test cases are derived from the remaining leaves of these testing trees. These three main features of the TTF required the implementation of complex predicate manipulation functions dealing with Z constructs such as schema boxes and types [5] . Obviously, test cases generated by Fastest are paragraphs of formal text. While this description is suitable for the automatic tasks involved in testing, it requires that stakeholders be able to read Z specifications in order to understand what is being tested. Hence, since it contributes to provide tool support for the TTF, this paper includes a description of a natural language generation (NLG) [13] template-based method to automatically translate abstract test cases into natural language [12] . The main contribution of the paper is the introduction of a tool supporting key tasks of the TTF which, as far as it has been investigated, did not exist before. More precisely, the tool gives support for the following tasks: (a) testing tactics can be applied automatically; (b) inconsistent test specifications can be semi-automatically eliminated from testing trees; (c) abstract test case generation can be performed semi-automatically; and (d) abstract test cases can be semiautomatically translated into natural languages. It is important to remark that Stocks and Carrington did not propose how to automate (b) and (c)-and they never mentioned (d). The results of running Fastest on eleven experiments, two of which are industrial-strength case studies, are also shown. The paper is structured as follows. The next section presents an introduction to the TTF by means of an example. In Section 3 the functional and architectural features of Fastest are introduced. Sections 4, 5 and 6 describe, respectively, the three main features of the TTF requiring predicate manipulation. The NLG method proposed to translate test cases into natural language is introduced in Section 7. Then, in Section 8, the degree of automation and limitations of the tool are discussed. Section 9 presents the results of applying Fastest to eleven case studies. This work is compared to similar approaches in Section 10. Finally, Section 11 describes the conclusions and future work. TOOL SUPPORT FOR THE TTF 3 THE TEST TEMPLATE FRAMEWORK This section introduces the TTF by means of an example that will be used throughout this article. The TTF is described without considering any particular implementation or tool. It is assumed that the reader is fluent in the Z notation. As it has been said, the TTF is a particular method for the "Generation" step of the MBT process (Figure 1) , specially well-suited for unit testing from Z specifications. Each operation within the specification is analysed to derive or generate abstract test cases. This analysis consists of the following steps (that will be detailed in the next subsections): 1. Consider the valid input space (VIS) of each Z operation. 2. Apply one or more testing tactics in order to partition the input space. 3. Build a tree of test specifications. 4. Prune inconsistent test specifications. 5. Find one abstract test case from each remaining test specification. One of the main advantages of the TTF is that all of these concepts are expressed in the same notation of the specification, as it will be shown shortly. Hence, the engineer has to know only one notation to perform the analysis down to the generation of abstract test cases. A Z Specification of a Simple Savings Accounts System Think about the savings accounts of a bank. Each account is identified by a so-called account number. Clients can share an account and each client can own many accounts-some of which might be shared with other clients. The bank only needs to keep record of the balance of each account, and the ID and name of each client. Any person can open an account at the bank, becoming the account's first owner. Account owners can add and remove other owners to their accounts, withdraw money and check the balance. Any person can deposit money in any accounts. To keep the example manageable, only one of these operations will be formalized. 2.1.1. Basic Types. As expressed in the previous section, each savings account is identified by an account number. Since account numbers are used just as identifiers, they can be abstracted away, ignoring their internal structure. Therefore, account numbers are introduced as a basic type: [ACCNUM] Along the same lines, basic types for the ID's of clients and their names are also introduced: [UID, NAME] The money that clients can deposit and withdraw, and the balance of savings accounts are represented as natural numbers. Specifying them as real numbers does not add any significant detail to the model, but makes it truly complicated since Z does not provides a native type for real numbers. If the decimal positions are really needed, then think that each natural number used in the specification is the result of multiplying the corresponding real number by a convenient power of 10-for instance, 100. Then: In other words, two synonyms are introduced for the set of natural numbers to make the specification more readable. Finally, an enumeration of different outputs that operations will perform in order to communicate whether they were successful or not is defined. MSG ::= ok | clientAlreadyExists | accountAlreadyExists The meaning of these constants will become clear when the operation is formalized. 1 == [NewClient VIS | u? ∈ dom clients ∧ n? ∈ dom balances] NewClient DNF 2 == [NewClient VIS | u? ∈ dom clients] NewClient DNF 3 == [NewClient VIS | n? ∈ dom balances] Now, SP is applied to ∪ in clients ∪ {u? → name?} in order to partition NewClient DNF 1 , yielding the following test specifications (note how test specifications are linked by schema inclusion):
doi:10.1002/stvr.1477 fatcat:6gb5br5ywrd5zhtgy77ny7xxny