Effect of Test Completeness and Redundancy Measurement on Post
Release Failures - an Industrial Experience Report
Tamás Gergely, Árpád
Gyimóthy and Milán Imre Gyalai
In risk-based testing, compromises are often made to
release a system in spite of knowing that it has outstanding defects.
In an industrial setting, time and cost are often the "exit criteria"
and - unfortunately - not the technical aspects like coverage or defect
ratio. In such situations, the stakeholders accept that the remaining
defects will be found after release, so sufficient resources are
allocated to the "stabilization" phases following the release. It is
hard for many organizations to see that such an approach is
significantly costlier than trying to locate the defects earlier. We
performed an empirical investigation of this for one of our industrial
partners (a financial company). In this project, significant perfective
maintenance was performed on the large information system. Based on
changes made to the system, we carried out procedure level code
coverage measurements with code level change impact analysis, and a
similarity-based comparison of test cases in order to quantitatively
check the completeness and redundancy of the tests performed. In
addition, we logged and compared the number of defects found during
testing and live operation. The data obtained were surprising for both
the developers and the customer as well, leading to a major
reorganization of their development, testing, and operation processes.
After the reorganization, a significant improvement in these indicators
for testing efficiency was observed.
Keywords: Code coverage, White-box testing, Impact analysis,
Test redundancy, Test similarity, Test efficiency, Risk-based testing.