Coverage Guided Testing for Recurrent Neural Networks [article]

Wei Huang, Youcheng Sun, James Sharp, Wenjie Ruan, Jie Meng, and Xiaowei Huang
2020 arXiv   pre-print
Recurrent neural networks (RNNs) have been applied to a broad range of applications such as natural language processing, drug discovery, and video recognition. This paper develops a coverage-guided testing approach for a major class of RNNs -- long short-term memory networks (LSTMs). We start from defining a family of three test metrics that are designed to quantify not only the values but also the temporal relations (including both step-wise and bounded-length) learned through LSTM's internal
more » ... tructures. While testing, random mutation enhanced with the coverage knowledge, i.e., targeted mutation, is designed to generate test cases. Based on these, we develop the coverage-guided testing tool testRNN. To our knowledge, this is the first time structural coverage metrics are used to test LSTMs. We extensively evaluate testRNN with a variety of LSTM benchmarks. Experiments confirm that there is a positive correlation between adversary rate and coverage rate, evidence showing that the test metrics are valid indicators of robustness evaluation. Also, we show that testRNN effectively captures erroneous behaviours in RNNs. Furthermore, meaningful information can be collected from testRNN for users to understand what the testing results represent. This is in contrast to most neural network testing works, and we believe testRNN is an important step towards interpretable neural network testing.
arXiv:1911.01952v2 fatcat:hm6emtrxs5bbraa7qs7jn66idy