Simulating the Effect of Test Flakiness on Fault Localization
Béla Vancsics, Tamás
Gergely and Árpád Beszédes
Test flakiness (non-deterministic behavior of test
cases) is an increasingly serious concern in industrial practice.
However, there are relatively little research results available
that systematically address the analysis and mitigation of this
phenomena. The dominant approach to handle flaky tests is still
detecting and removing them from automated test executions.
However, some reports showed that the amount of flaky test is in
many cases so high that we should rather start working on
approaches that operate in the presence of flaky tests. In this
work, we investigate how test flakiness affects the effectiveness
of Spectrum Based Fault Localization (SBFL), a popular class of
software Fault Localization (FL), which heavily relies on test
case execution outcomes. We performed a simulation based
experiment to find out what is the relationship between the level
of test flakiness and fault localization effectiveness. Our
results could help the users of automated FL methods to understand
the implications of flaky tests in this area and to design novel
FL algorithms that take into account test flakiness.
Keywords: Test flakiness,
flaky tests, fault localization, Spectrum-Based Fault
Localization, testing and debugging.