Test Suite Evaluation using Code Coverage Based Metrics
Ferenc Horváth, Béla Vancsics, László Vidács, Árpád
Beszédes, Dávid Tengeri, Tamás Gergely and Tibor
Regression test suites of evolving software systems are
often crucial to maintaining software quality in the long term.
They have to be effective in terms of detecting faults and helping
their localization. However, to gain knowledge of such
capabilities of test suites is usually difficult. We propose a
method for deeper understanding of a test suite and its relation
to the program code it is intended to test. The basic idea is to
decompose the test suite and the program code into coherent
logical groups which are easier to analyze and understand.
Coverage and partition metrics are then extracted directly from
code coverage information to characterize a test suite and its
constituents. We also use heat-map tables for test suite
assessment both at the system level and at the level of logical
groups. We employ these metrics to analyze and evaluate the
regression test suite of the WebKit system, an industrial size
browser engine with an extensive set of 27,000 tests. In this
paper, we introduce this pilot implementation and, as a
proof-of-concept, present how the coverage results were used for
Keywords: code coverage,
regression testing, test suite evaluation, test metrics.