Scenario-based testing is a common approach to verify and validate Advanced Driving Assistance System / Autonomous Driving (ADAS/AD) of motor vehicles. The main challenge in scenario-based testing is the selection of a finite number of scenarios to represent an infinite amount of possible scenarios. Beyond that, there is no metric to evaluate scenarios thus the quality of the testing process. We introduce a generic process chain to ensure traceability and reproducibility of scenario selection,

more »

... y generating scenarios automatically. A Feature Model (FM) builds the input data for our process chain. We identify three concepts to represent a scenario using a FM. We create a tool to transfer a configuration of the FM into a concrete scenario. A sample represents a set of scenarios, we define them as scenario suite. We evaluate the quality of a scenario suite by applying its scenarios to various mutants of driving functions in a simulation tool. The quality of the scenario suites is then determined by the number of discovered mutants. We evaluate the influence of various FMs in combination with common sampling algorithms such as ICPL, Chvatal, and IncLing, using an Autonomous Emergency Braking (AEB) as subject system. We discover a correlation between FM and mutation score as well as between mutation score and sampling algorithm. Within a scenario suite, we identify a strict separation between scenarios that are good to kill a mutant and those which are not. We discover, that sampling algorithms that aim for feature interaction coverage produce stronger scenario suites than feature-wise sampling algorithms. An evaluation of the relevance of single features on the mutation score provides features that are frequently involved in scenarios that are good to kill mutants. Beyond that, we discover a correlation between scenario suite and mutants that affects the mutant detection.