Software Testing: Coverage Metrics, Path Testing, and Fault-Based Testing
Test Coverage Metrics
The motivation for using DD-paths is that they enable very precise descriptions of test coverage. In our quest to identify gaps and redundancy in test cases as these are used to exercise (test) different aspects of a program, we use formal models of the program structure to reason about testing effectiveness. Test coverage metrics are a device to measure the extent to which a set of test cases covers a program. Several widely accepted test coverage metrics are used; most of those are in the table below (Miller, 1977). Having an organized view of the extent to which a program is tested makes it possible to sensibly manage the testing process. Most quality organizations now expect the C1 metric (DD-Path coverage) as the minimum acceptable level of test coverage. Less adequate, the statement coverage metric (C0) is still widely accepted.
Path Testing
Definition: Given a program written in an imperative programming language, the program graph is a directed graph in which nodes are statement fragments, and edges represent the flow of control. If i and j are nodes in the program graph, an edge exists from node i to node j if and only if the statement fragment corresponding to node j can be executed immediately after the statement fragment corresponding to node i.
- The groups of statements that make up a node in the Program Graph are called a basic block.
- There is a straightforward algorithm to segment a code fragment into basic blocks and create the corresponding Program Graph.
- Construction of a program graph from a given program is illustrated here with the pseudo-code implementation of the triangle program and the maximum of 3 numbers.
- Line numbers refer to statements and statement fragments. The importance of the program graph is that program executions correspond to paths from the source to the sink nodes.
- We also need to decide whether to associate nodes with non-executable statements such as variable and type declarations; here we do not.
Assumptions in Fault-Based Testing
The effectiveness of fault-based testing depends on the quality of the fault model and on some basic assumptions about the relation of the seeded faults to faults that might actually be present. In practice, the seeded faults are small syntactic changes, like replacing one variable reference by another in an expression or changing a comparison from < to <=. We may hypothesize that these are representative of faults actually present in the program. If the program under test has an actual fault, we may hypothesize (assume) that it differs from another corrected program by only a small textual change. If so, then we need merely distinguish the program from all such small variants to ensure the detection of all such faults. This is known as the competent programmer hypothesis, an assumption that the program under test is close to a correct program.
Generic versus Specific Scaffolding
The simplest form of scaffolding is a driver program that runs a single, specific test case. If, for example, a test case specification calls for executing method calls in a particular sequence, this is easy to accomplish by writing the code to make the method calls in that sequence. Writing hundreds or thousands of such test-specific drivers may be cumbersome to thorough testing. At least we want to factor out some of the common driver code into reusable modules. It is worthwhile to write more generic test drivers that essentially interpret test case specifications. At least some level of generic scaffolding support can be used across a fairly wide class of applications. Such support typically includes basic support for logging test execution and results in addition to a standard interface for executing a set of test cases. Figure 1 illustrates the use of generic test scaffolding in the JFlex lexical analyzer generator.