Effectiveness assessment of an early testing technique using model-level mutants

While modern software development technologies enhance the capabilities of model-based/driven development, they introduce challenges for testers such as how to perform early testing at model level to ensure the quality of the model. In this context, we have developed an early testing technique supported by the CoSTest tool to validate requirements at model level. In this paper we describe an empirical evaluation of CoSTest with respect to its effectiveness in terms of its fault detection and test suite adequacy. This evaluation is carried out by model-level mutation testing using first order mutants (created by injection of a single fault) and high order mutants (containing more than one fault) with seven conceptual schemas (of different sizes) that represent the functionality of different software systems in different domains. Our findings show that the tests generated by CoSTest are effective at killing a large number of mutants. However, there are also some fault types (e.g. delete the references to a class attribute or an operation call in a constraint) that our test suites were not able to detect. CoSTest was more effective in terms of detecting fault types using high order mutants that first order mutants. Thus, CoSTest's effectiveness is affected by the mutant type tested.