Datamorphic Testing: A Method for Testing Intelligent Applications
2019 IEEE International Conference On Artificial Intelligence Testing (AITest)
Adequate testing of AI applications is essential to ensure their quality. However, it is often prohibitively difficult to generate realistic test cases or to check software correctness. This paper proposes a new method called datamorphic testing, which consists of three components: a set of seed test cases, a set of datamorphisms for transforming test cases, and a set of metamorphisms for checking test results. With an example of face recognition application, the paper demonstrates how to
... p datamorphic test frameworks, and illustrates how to perform testing in various strategies, and validates the approach using an experiment with four real industrial applications of face recognition. Index Terms-software testing, intelligent applications, machine learning, test case generation, test oracle, datamorphism, metamorphism. I. MOTIVATION We have seen a rapid growth in the application of machine learning (ML), data mining and other artificial intelligence (AI) techniques in recent years. Typical examples of such applications include driverless vehicles, face recognition and finger print recognition in security control, workload pattern learning in computer cluster operation, personalisation of social media networking for business intelligence, situation recognition and action rule learning in healthcare, smart homes and smart cities, etc. Many such applications are also closely integrated with robotics, the Internet-of-Things, Big Data, and Edge, Fog and Cloud computing. These all require automated and optimised data collection and processing. AI techniques, especially ML, have been widely regarded as a promising solution to the underlying hard computational problems. All such applications must be thoroughly tested to ensure their quality  . However, the current practice of testing AI applications lags far behind the maturity of testing traditional software. Testers have been confronted with grave challenges because the distinctive features of ML applications disqualify existing software testing methods, techniques and tools. As a result, generating realistic test cases and checking the results are prohibitively difficult and expensive  . To address these difficulties, we propose a novel approach called datamorphic testing for AI applications based on our previous work  on integration of data mutation testing  and metamorphic testing ,  . The paper is organized as follows. Section II introduces the basic concepts of datamorphic testing method. Section III discusses the process and various strategies of datamorphic testing. Section IV reports an experiment with four real industrial ML applications of face recognition. Section V discusses related work and future work.