Code Coverage Differences of Java Bytecode and Source Code
Instrumentation Tools
Ferenc Horváth, Tamás
Gergely, Árpád
Beszédes, Dávid
Tengeri, Gergő
Balogh and Tibor Gyimóthy
Many software testing fields, like white-box testing,
test case generation, test prioritization and fault localization,
depend on code coverage measurement. If used as an overall
completeness measure, the minor inaccuracies of coverage data
reported by a tool do not matter that much; however, in certain
situations they can lead to serious confusion. For example, a code
element that is falsely reported as covered can introduce false
confidence in the test. This work investigates code coverage
measurement issues for the Java programming language. For Java,
the prevalent approach to code coverage measurement is using
bytecode instrumentation due to its various benefits over source
code instrumentation. As we have experienced, bytecode
instrumentation-based code coverage tools produce different
results than source code instrumentation-based ones in terms of
the reported items as covered. We report on an empirical study to
compare the code coverage results provided by tools using the
different instrumentation types for Java coverage measurement on
the method level. In particular, we want to find out how much a
bytecode instrumentation approach is inaccurate compared to a
source code instrumentation method. The differences are
systematically investigated both in quantitative (how much the
outputs differ) and in qualitative terms (what the causes for the
differences are). In addition, the impact on test prioritization
and test suite reduction - a possible application of coverage
measurement - is investigated in more detail as well.
Keywords: Code coverage,
white-box testing, Java bytecode instrumentation, source code
instrumentation, coverage tools, empirical study.
Back