Assessing the Test Suite of a Large System Based on Code Coverage, Efficiency and Uniqueness

Regression test suites of evolving software systems play a key role in maintaining software quality throughout continuous changes. They need to be effective (in terms of detecting faults and helping their localization) and efficient (optimally sized and without redundancy) at the same time. However, test suite quality attributes are usually difficult to formalize and measure. In this paper, we rely on a recent approach for test suite assessment and improvement that utilizes code coverage information, but at a more detailed level, hence it adds further evaluation aspects derived from the coverage. The basic idea of the method is to decompose the test suite and the program code into coherent logical groups which are easier to analyze and understand. Several metrics are then computed from code coverage information to characterize the test suite and its constituents. We extend our previous study and employ derived coverage metrics (which express efficiency and uniqueness) to analyze the test suite of a large scale industrial open source system containing 27 000 test cases.

Keywords: code coverage, regression testing, test suite evaluation, test suite quality, test efficiency, test metrics.