Coverage Criteria - PowerPoint Presentation

Slide1

Coverage Criteria

Drawn mostly from

Ammann&Offutt

and

Pezze&YooungSlide2

IEEE definition of V&V

Validation:

The process of evaluating software at the end of software development to ensure compliance with intended

usageVerification: The process of determining whether the products of a given phase of the software development process fulfill the requirements established during the previous phase

2Slide3

Static vs dynamic testing

Static

testing

: Testing without executing the programInclude software inspections and some forms of analysesVery effective at finding certain kinds of problems – especially “potential” faults, that is, problems that could lead to faults when the program is

modified

Dynamic

testing : Testing by executing the program with real inputs

3Slide4

Testing vs debugging

Testing : Finding inputs that cause the software to

fail

Debugging : The process of finding a fault given a failure4Slide5

A test case

Test

case

values: The values that directly satisfy one test requirementExpected results: The result that will be produced when executing the test if the program satisfies it intended behavior

5Slide6

Top-down vs bottom up testing

Top-down testing

: Test the main procedure, then go down through procedures it calls, and so

onBottom-up testing : Test the leaves in the tree (procedures that make no calls), and move up to the root.Each procedure is not tested until all of its children have been tested

6Slide7

Test requirement

Test r

equirements:

Specific things (software artifacts) that must be satisfied or covered during testingNon-software example: test a bag of jelly beansCome up with ways to testSuppose the following flavors: Lemon (yellow), pistachio (green), cantaloupe (orange), pear (white), tangerine (orange), apricot (yellow)

One requirement: test each flavor (six test requirements)

7Slide8

Test criterion

Test criterion: A collection of rules and a process that define test

requirements

Flavor criterion: TR = {flavor=lemon, flavor=pistachio, flavor=cantaloupe, flavor=pear, flavor=tangerine, flavor=apricot}

8Slide9

Test coverage

Given a set of test requirements TR for coverage criterion C, a test set T satisfies C coverage if and only if for every test requirement

tr

in TR, there is at least one test t in T such that t satisfies trA test case with 12 beans: 3 lemons, 1 pistachio, 2 cantaloupe, 1 pear, 1 tangerine, 4 apricot OK to satisfy a test requirement with more than one test

9Slide10

Coverage level

Given a set of test requirements TR and a test set T, the coverage level is simply the ratio of the number of test requirements satisfied by T to the size of TR

Why? Sometime test requirements may be infeasible

Example: suppose tangerine jelly beans are rare and some bags may not contain anyFlavor criteria cannot be 100% satisfiedMaximum coverage level: 5/6 or 83%

10Slide11

Criteria vs subsumption

Criteria

subsumption

: A test criterion C1 subsumes C2 if and only if every set of test cases that satisfies criterion C1 also satisfies C2Must be true for every set of test casesExample: color criteria for the jelly bean: {yellow, green, orange, white}

If we satisfy flavor criteria, we’ll satisfy color criteria

Example : If a test set has covered every branch in a program (satisfied the branch criterion), then the test set is guaranteed to also have covered every statement

11Slide12

Coverage is not size

Coverage does not depend on the number of test cases

T0 , T1 : T1 >coverage T0

T1 <cardinality T0 T1 , T2 : T2 =coverage T1 T2 >cardinality T1

Small

test cases make failure diagnosis easier

A failing test case in T2 gives more information for fault localization than a failing test case in

T

1

12Slide13

Question 1

Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose

that test

set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.Does

T1 necessarily satisfy C2? Explain.

13Slide14

Question 2

Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose

that test

set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.Does

T1 necessarily satisfy C2?

Explain.

Yes. This follows directly from the definition of

subsumption.

14Slide15

Question 2

Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose

that test

set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.Does

T2 necessarily satisfy C1? Explain.

15Slide16

Question 2

Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose

that test

set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.Does

T2 necessarily satisfy C1? Explain

.

No. There is no reason to expect test requirements generated by C1 to be satisfied by T2.

16Slide17

Question 3

Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose

that test

set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.If P contains a fault, and T2 reveals the fault, T1 does not necessarily also reveal

the fault

. Explain.

17Slide18

Question 3

Suppose that coverage criterion C1 subsumes coverage criterion C2. Further suppose

that test

set T1 satisfies C1 on program P and test set T2 satisfies C2, also on P.

If P contains a fault, and T2 reveals the fault, T1 does not necessarily also reveal

the fault

. Explain.No. This is the hard question.

Testers often

think that test sets for strong criteria

are at

least as good at

finding faults

as test sets for weaker criteria. But subsumption

is about

criteria, not about test sets. In particular, there is no requirement that test set

T2 be

a subset of test set T1. So, it could happen that T2 contains that one test that

reveals the

fault, and T1 doesn't.

18Slide19

Statements vs branches

Traversing all edges of a graph causes all nodes to be visited

So test suites that satisfy the branch adequacy criterion for a program P also satisfy the statement adequacy criterion for the same program

The converse is not true: A statement-adequate (or node-adequate) test suite may not be branch-adequate (edge-adequate)

19Slide20

“

All branches

”

can still miss conditionsSample fault: missing operator (negation)

digit_high

==

-

1 ||

digit_low

== -1

Branch adequacy criterion can be satisfied by varying only

digit_low

The faulty sub-expression might never determine the result

We might never really test the faulty condition, even though we tested both outcomes of the branch

20Slide21

Condition testing

Branch coverage exposes faults in how a computation has been decomposed into cases

I

ntuitively attractive: check the programmer’

s case analysis

B

ut only roughly: groups cases with the same outcome Condition coverage considers case analysis in more detail

A

lso

individual conditions

in a compound Boolean expression

e.g., both parts of

digit_high

== 1 ||

digit_low

== -1

21Slide22

Basic conditions vs branches

Basic condition adequacy criterion can be satisfied without satisfying branch coverage

Branch and basic condition are not comparable (neither implies the other)

22Slide23

Covering branches and conditions

Branch and condition adequacy: cover all conditions and all decisions

Compound condition adequacy

Cover all possible evaluations of compound conditions

Cover all branches of a decision tree

Ch 12, slide

23Slide24

Compound conditions: Exponential complexity

(((a || b) && c) || d) && e

Test a b c d e

Case

(1)

T — T — T

(2)

F T T — T

(3)

T — F T T

(4)

F T F T T

(5)

F F — T T

(6)

T — T — F

(7)

F T T — F

(8)

T — F T F

(9)

F T F T F

(10)

F F — T F

(11)

T — F F —

(12)

F T F F —

(13)

F F — F —

S

hort-circuit

evaluation often reduces this to a more manageable number, but not always

24Slide25

Multiple (compound) condition

coverage

The

multiple condition coverage of T is computed as Cc/(Ce -Ci) , where Cc is the number of combinations covered, Ci is the number of infeasible simple combinations, and Ce is the total number of combinations in the

program

Potentially a large number of test cases

25Slide26

Modified condition/decision (MC/DC)

Motivation: Effectively test

important

combinations

of conditions, without exponential blowup in test suite size

“

Important

”

combinations means: Each basic condition shown to independently affect the outcome of each decision

Compound condition as a whole evaluates to

true

for one and

false

for the

other

26Slide27

MC/DC: linear complexity

N+1 test cases for N basic conditions

(((a || b) && c) || d) && e

Test a b c d e outcome

Case

(1)

true

--

true

--

true

true

(2) false

true

true -- true true

(3) true -- false

true

true true

(6) true -- true --

false

false

(11) true --

false

false

-- false

(13)

false

false

-- false -- false

Underlined values independently affect the output of the decision

Required by the

RTCA/DO-178B standard

27Slide28

Comments on MC/DC

MC/DC is

B

asic condition coverage (C)

B

ranch coverage (DC)

Plus one additional condition (M): every condition must

independently affect

the decision

’

s output

It is subsumed by compound conditions and subsumes all other criteria discussed so far

S

tronger than statement and branch coverage

A good balance of thoroughness and test size (and therefore widely used

)

Federal Aviation Administration’s requirement that test suites be MC/DC adequate

28Slide29

Paths? (Beyond individual branches)

Should we explore sequences of branches (paths) in the control flow?

Many more paths than branches

A pragmatic compromise will be needed

Ch 12, slide

29Slide30

(c) 2007 Mauro Pezzè & Michal Young

Ch 12, slide

30

Path adequacy

Decision and condition adequacy criteria consider individual program decisions

Path testing focuses consider combinations of decisions along paths

Adequacy criterion: each path must be executed at least once

Coverage:

# executed paths

# pathsSlide31

(c) 2007 Mauro Pezzè & Michal Young

Ch 12, slide

31

Practical path coverage criteria

The number of paths in a program with loops is unbounded

the simple criterion is usually impossible to satisfy

For a feasible criterion: Partition infinite set of paths into a finite number of classes

Useful criteria can be obtained by limiting

the number of traversals of loops

the length of the paths to be traversed

the dependencies among selected pathsSlide32

(c) 2007 Mauro Pezzè & Michal Young

Ch 12, slide

32

Cyclomatic adequacy

Cyclomatic

number: number of independent paths in the CFG

If e = #edges, n = #nodes, c = #connected components of a graph, it is:

e - n + c for an arbitrary graph

e - n + 2 for a CFG

Cyclomatic

coverage counts the number of independent paths that have been exercised, relative to

cyclomatic

complexity