DyStaCCo - Dynamic and STatic Analysis Combined for code COverage
Code coverage is an important technique to assess the adequacy of software testing, and support decision making over a quality process. While most testing managers would probably agree on these considerations, despite the research effort that produced many code coverage criteria to address several testing goals, code coverage has not made its journey to become consolidated best practice yet.
The main reasons for the limited success of code coverage are traceable to the high (often unaffordable) costs of designing test suites with high coverage indicators. Manually inspecting programs to find test cases that exercise non-trivial portions of uncovered code can be extremely expensive. Even worse, relevant amount of effort might be wasted in the attempt of covering infeasible code elements, generally induced in non-negligible amounts by many relevant coverage domains. The many techniques to automatically generate test cases to increase code coverage work fairly well for simple metrics, like statement and branch coverage, but do not address well complex metrics, like data flow coverage criteria.
In the AVATAR project, we have started investigating a new generation of structural testing techniques that can automatically generate test suites with high structural coverage, and correctly account for relevant portions of infeasible elements of the coverage domain of choice. In particular, in AVATAR, we have defined a technique called ARC-B (short for Abstraction Refinement and Coarsening for Branch testing) that combines static and dynamic analysis techniques within an incrementally refined runtime model to automatically compute test suites that results in branch coverage. Through the lifespan of AVATAR, we have designed and implemented a prototype tool of ARC-B that addresses branch testing for programs written in C, and we have experimented this prototype against a preliminary set of benchmark programs. The empirical data collected so far on simple programs testify that ARC-B can successfully generate test suites with high branch coverage indicators, improving over the state of the art in the field. The research findings of the AVATAR project have been presented at the 5th International Workshop on Automation of Software Test and published in the Software Quality Journal.
Based on the success of AVATAR, the DyStaCCo project aims to progress over the current achievements and findings along two major research directions. First, this project will extend the ARC-B paradigm to deal with large programs. This requires extending ARC-B to cope with inter-procedural program flows, study alternative dynamic and static analysis techniques to solve performance problems, define models and procedures to efficiently integrate the different analysis techniques to overcome the limitation of the current technique.
Second, DyStaCCo aims to investigate qualitatively and quantitatively automatic dataflow testing. Dataflow testing techniques have not been automated beyond the computation of testing requirements, and have not been studied extensively due to the difficulty of achieving reasonable data flow coverage indicators even for simple programs. Grounded on the AVATAR experience with automating branch testing, this project aims to automate the generation of test suites that can achieve high data flow coverage and thus enable a quantitative study of these metrics. To extend the use of analysis for data flow testing we will need to experiment with different analysis techniques, and define new models and techniques to suitably combine the techniques and compute infeasible data flow elements.