Testing Your Program: - PowerPoint Presentation

Slide1

Testing Your Program: Input Space Partitioning

CS

1110

Introduction to Programming

Spring 2017Slide2

Software is Everywhere2

CS 1110Slide3

Software Failures3

(1997

)

Ariane 5 explosion

: Exception

-handling bug forced self-destruct on maiden flight (64-bit to 16-bit conversion), causing $370 millions(1999)

NASA’s Mars lander: Crashed due to a unit integration

fault(2003) Northeast blackout: The alarm system in the energy management system failed due to a software error and operators were not informed of the power overload in the system – affected 40 million people in 8 US states, 10 million people in Ontario, Canada(2012) Knight Capital’s trading software: Faults in a new trading algorithm causes $440 millions

(2014) Dropbox’s outage was due to a fault in a maintenance script(2016) Information lost after clicking the back button while using TurboTax web softwareCS 1110Slide4

Terminologies4

Software failures

: external, incorrect

behaviors

with respect to the requirements or other descriptions of the expected

behaviorExample: A patient gives a doctor a list of symptoms (

failures) Three kinds of failures we’ve seenSyntax failureRuntime failureLogical failure

Software faults: static defects in the softwareExample: The doctor tries to diagnose the

root cause (fault)Faults in software ≈ design mistakes in hardware.

CS 1110Slide5

An Example in Python

#

Return index

of the first occurrence of

a letter

in string,

#

Otherwise

, return -1

def get_index_of(string, letter): index = -1

for i in range(1, len

(string)): if string[i] == letter: index =

i return index

# Test1: inputs “python”, “z”

print(get_index_of

("python", "z

")) # expected: -1, actual: -1

#

Test2: inputs “python”, “z

”

print(

get_index_of

("python", "p"))

# expected: 0

,

actual: -1

Fault: Should start at 0, not 1

5

For simplicity, this example assumes a function accept a letter of size

1

Test1

Error:

i

is 1, not 0, on the first iteration

Failure: none

Test2

Error: i is 1, not 0, on the first iteration -- Error propagates to the variable indexFailure: index is -1 at the return statement

CS 1110Slide6

Overview: Testing6

What is software testing?

Why do we test software?

When should we test software?

Who should test software?

How do we test software?

Input space partitioningWhen do we stop testing? Good enough? CS 1110Slide7

Here! Test This!7

My first “testing” assignment

A stack of computer printouts – and no documentation

MicroSteff

– big

software system

for the mac

V.1.5.1 Jan/2007

Verdatim

DataLife

MF2-HD

1.44 MB

Big software program

Jan/2011

CS 1110Slide8

What is Software Testing? 8

Testing

= process of finding input values to check against a software

(today’s discussion)

Debugging = process of finding a fault given a failure

(out of today’s scope)

Testing is fundamentally about choosing finite sets of values from the input domain of the software being tested

Give the test inputs, compare the actual results with the expected results

Test case consists of test values and expected results

Test

values (inputs)Actual results

Program

Expected results

vs

CS 1110Slide9

Why do We Test Software?9

Goal

of testing

Not to prove correctness, but to increase our confidence in correctness

Improve quality

Reduce overall software development costPreserve customer satisfactionGet good grades in this course (and any other courses)

What fact does each test try to verify?BenefitsYou are not biased that your code worksYou will better understand what you need to buildYou will get insights on how to built it

CS 1110Slide10

When should We Test Software?10

60

50

40

30

20

10

0

Requirements

Fault origin (%)

Fault detection (%)

Unit cost (X)

Assume $1000 unit cost, per fault, 100 faults

$6K

$13K

$20K

$360K

$250K

$100K

Integration

Test

System Test

Post-

Deployment

Design

Program/Unit

test

Software Engineering Institute; Carnegie Mellon University; Handbook CMU/SEI-96-HB-002

CS 1110

Testing is the most time consuming and expensive part of software

development. Not

testing is even more

expensive.

If we have too little testing effort early, the cost of testing

increases

Planning for testing after development is prohibitively

expensive

Cost of late testingSlide11

Who should Test Software? 11

CS

1110

students

Software designers

Software developers/programmersSoftware testersProject managersEnd users

CS 1110Slide12

How do We Test Software ?

12

Acceptance testing

(equivalence classes)

Boundary testing (corner cases)

Exception testingCS 1110Slide13

How do We Test Software ?Input Space Partitioning (ISP) – Choose Inputs

13

Input space partitioning

describes the

input domain

of the softwareInput domain contains all the possible values that the input parameters can have

Input parameters can be Parameters to a functionData read from a fileGlobal variablesUser level inputs

Testing is fundamentally about choosing finite sets of values from the input domain of the software being testedCS 1110Slide14

ISP: Choosing Test Inputs14

Identify

inputs

Find their

characteristics

Each characteristic allows the testers to define a partition.

Partition each characteristic and identify valuesThe partition must cover the entire domain; not overlap

Two approachesInterface-based (simpler)Develop characteristics from individual input parametersCan be partially automated in some situations

Functionality-based (harder)Develop characteristics from a behavior view May result in better tests, or fewer tests that are as effective

Choose tests by combining values of inputs CS 1110Slide15

How do We Test Software ?Input Space Partitioning (ISP) – Compare Results

15

Give the test inputs, compare the actual results with the expected results

If the actual results == expected result,

the program passes the test

Otherwise, the program fails the testThe test inputs reflect the characteristics of the input parameters

The characteristics of the inputs signifies the kinds of faults/bugs in programThe kinds of faults/bugs tell what to fixCS 1110Slide16

Identify

inputs

Two parameters:

string

, letter

Find their characteristicsC1

= string is empty

C2 = letter is empty

ISP: Example 1 (Interface-based)Choose Test Inputs16

#

Return index

of the first occurrence

of a

letter in string,

# Otherwise

, return -1def

get_index_of

(string, letter):

CS 1110Slide17

Partition

each

characteristic and

identify

values

Choose tests

by combining values of test inputs (assume choosing letter = "p")

ISP: Example 1 (Interface-based)Choose Test Inputs

17

Characteristic

P1

P2

C1 =

string

is empty

True

possible values:

""

False

possible values:

"python"

C2 =

letter

is empty

True

possible values:

""

False

possible values:

"p" or "z"

Test

string

letter

T1 (C1=True,

C2=True)

""

""

T2 (C1=True,

C2=False)

""

"p"

T3 (C1=False,

C2=True)

"python"

""

T4 (C1=False,

C2=False)

"python"

"p"

CS 1110Slide18

ISP:

Example

1

(Interface-based

)

Compare Results18

Test

string

letter

Expected result

T1 (C1=True,

C2=True)

""

""

-1

T2 (C1=True,

C2=False)

""

"p"

-1

T3 (C1=False,

C2=True)

"python"

""

-1

T4 (C1=False,

C2=False)

"python"

"p"

0

Put a test case T4 in Python

print

(

get_index_of

("python",

"p"

))

# expected:

0

If actual result ==

expected

result, the program passes the test

Otherwise, the program fails the test

Specify the

expected result

of the given test inputs

We don’t even have to look at the code to come up with the test cases

CS 1110Slide19

Identify

inputs

Two parameters:

string

, letter

Find their characteristics

C1 = number of occurrence of letter in string

C2 = letter occurs first in string

C3 = letter

occurs last in string ISP: Example

1 (Functionality-based)Choose Test Inputs19

#

Return index

of the first occurrence of letter in string,

#

Otherwise

, return -1

def

get_index_of

(string, letter)

:

No need to look at the code to come up with the test cases. The specification is sufficient.

CS 1110Slide20

Partition

each

characteristic

and

identify

values

Possible values

20

Characteristic

P1

P2

P3

C1 = number of occurrence of

letter

in

string

0

1

>

1

C2 =

letter

occurs first in

string

True

False

C3 =

letter

occurs last in

string

True

False

C

P1

P2

P3

C1

"intro to python", "z"

or

"intro to python", ""

or

"", "z"

"intro to python", "r"

"intro to python", "o"

C2

"intro to python", "

i

"

"intro to python", "o"

C3

"intro to python", "n"

"intro to python", "o"

ISP: Example

1

(Functionality-based)

Choose Test Inputs

CS 1110Slide21

Choose

tests

by combining values of test

inputs

Choose a characteristic, then choose test values for the characteristic

21

C

P1

P2

P3

C1

T1("intro to python", "z")

T2(

"intro to python", "")

T3(

"", "z")

T4("intro to python", "r")

T5("intro to python", "o")

C2

T6("intro to python", "

i

")

T5("intro to python", "o")

C3

T7("intro to python", "n")

T5("intro to python", "o")

ISP: Example

1

(Functionality-based)

Choose Test Inputs

CS 1110Slide22

Specify the expected result

of the given test inputs

ISP:

Example

1

(Interface-based

)

Compare Results

22

Test

string

letter

Expected result

T1

"intro to python"

"z"

-1

T2

"intro to python"

""

-1

T3

""

"z"

-1

T4

"intro to python"

"r"

3

T5

"intro to python"

"o"

4

T6

"intro to python"

"

i

"

0

T7

"intro to python"

"n"

14

print

(

get_index_of

(

"

intro to python

"

, "

z"

))

# expected:

-1

…

print(

get_index_of

("

intro to python

", "

i

"

))

# expected: 0

print

(

get_index_of

("

intro to python

", "

n"

))

# expected:

14

CS 1110Slide23

When do We Stop Testing?23

Time constraint

(Assignments)

due date

Coverage

Test cases cover all characteristics and partitions Characteristics and partitions determine the quality of test casesThe quality of test cases determine how thorough the program is tested

CS 1110Slide24

Note and TipCS 1110

24

I didn’t fail the test. I just found 100 ways to do it wrong

.

– Benjamin

Franklin

Did you know?

There is an actual tactic called “rubber duck debugging” where programmers verbally explain a broken code to a rubber duck in hopes of

finding the solution by describing the problem