Requirement Specifications based Test Automation PowerPoint Presentation, PPT - DocSlides

Slide1

Requirement Specifications based Test Automation

Tao XieUniversity of Illinois at Urbana-Champaignhttp://taoxie.cs.illinois.edu/courses/testing/

Work described in the slides was done in collaboration with the

Pex

team (Nikolai Tillmann, Peli de Halleux,

Pratap

Lakshman

, et al.) @Microsoft Research, students @Illinois ASE, and other collaborators

Slide2

Techniques for writing tests

Black-box (from specifications)Equivalence partitioningBoundary value analysisWhite-box (from code)Branch coverageFault-based testing (from common faults)

2

Slide3

3

What is a Unit Test?

A unit test is a small program with assertions.

[

TestMethod

]

public void Add()

{

HashSet

set = new

HashSet

();

set.Add

(3);

set.Add

(14);

Assert.AreEqual

(

set.Count

, 2);

}

Many developers write such unit tests by hand, involving

determining a meaningful sequence of method calls,

selecting exemplary argument values (the test

inputs

),

stating assertions.

Slide4

Unit Testing: Measuring Quality

Coverage: Are all parts of the program exercised?

statements

basic blocks

explicit/implicit branches

…

Assertions: Does the program do the right thing?

test oracle

Experience:

Just high coverage or large number of assertions is no good quality indicator.

Only both together are!

Slide5

Advantages of tests as specs

Concrete, easy to understandDon’t need new languageEasy to see if program meets the specMaking tests forces you to talk to customer and learn the problemMaking tests forces you to think about design of system (classes, methods, etc.)

5

Slide6

Disadvantages of tests as specs

Too specificHard to test that something can’t happenCan’t withdraw more money than you have in the systemCan’t break into the systemCan’t cause a very long transaction that hangs the systemTends to be verbose

6

Slide7

Tests as specifications

Tests show how to use the systemTests need to be readableNeed comments that describe their purpose or need good namesKeep short, delete duplicate or redundant

7

Slide8

Parameterized Unit Test

Parameterized Unit Test

A parameterized unit test is a small program that takes some inputs and states assumptions and assertions.

8

See later slides on JUnit: @Theory (multiple parameters) @Parameters (single parameter)

Slide9

Parameterized Unit Test

Parameterized Unit Testing

Parameterized Unit Testsserve as specificationscan be leveraged by (automatic) test input generatorsfit in development environment, evolve with the code

Slide10

Hand-written

Test Generation Process

10

// FooTest.cs[TestClass, PexClass]partial class FooTest{ [PexMethod] void Test(Foo foo) {…}

// FooTest.Test.cspartial class FooTest{ [TestMethod] void Test_1() { this.Test(new Foo(1)); } [TestMethod] void Test_1() { this.Test(new Foo(2)); } …}

Pex

User writes parameterized tests Lives inside a test class

Generated unit tests Pex not required for re-execution xUnit unit tests

xUnit

Attributes

Pex

Attributes

Parameterized Unit Test

Partial Class

Generated

http://msdn.microsoft.com/en-us/library/wa80x488(VS.80).aspx

Slide11

PUTs separate concerns

PUTs

separate two concerns:

(1) The specification of external behavior

(i.e., assertions)

(2) The selection of internal test inputs

(i.e., coverage)

In many cases, a test generation tool (e.g.,

Pex

) can construct a

small test suite

with

high coverage

!

Slide12

PUTs are algebraic specs

A PUT can be read as a universally quantified, conditional axiom.



int

name,

int

data.

name

≠

null ⋀ data

≠

null

⇒

equals(

ReadResource

(name,

WriteResource

(name, data)),

data)

Slide13

Pex4Fun – Turning Pex Online

1,792,222

clicked 'Ask

Pex!'

http://pex4fun.com/default.aspx?language=CSharp&sample=_Template

Slide14

http://research.microsoft.com/pex/

Dynamic Symbolic Execution (DSE)

aka. Concolic Testing [Godefroid et al. 05][Sen et al. 05][Tillmann et al. 08]Instrument code to explore feasible paths

Behind the Scene of Pex4Fun

Slide15

http://research.microsoft.com/pex/

Slide16

void

CoverMe

(

int

[] a)

{ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

a.Length>0

a[0]==123…

T

F

T

F

F

a==null

T

Constraints to solve

a!=null

a!=null &&

a.Length

>0

a!=null &&

a.Length

>0 &&a[0]==123456890

Inputnull{}{0}{123…}

Execute&Monitor

Solve

Choose next path

Observed constraints

a==null

a!=null &&

!(

a.Length

>0)

a==null &&

a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890

Done: There is no path left.

Dynamic Symbolic Execution in

Pex

http://pex4fun.com/HowDoesPexWork

Slide17

Pex is Part of Visual Studio 2015 Enterprise Edition!

As new feature of “IntelliTest”

https://www.visualstudio.com/news/vs2015-vs#Testing

Pex

:

30K

downloads after 20 months

Active user community: 1.4K forum posts during ~3 years

Slide18

From

http://bbcode.codeplex.com/

NUnit Extension

– 13024 downloads

xUnit.net Extension

- 8589 downloads

Impact of

IntelliTest

Slide19

Domain Matrix for Testing Complex Condition

19

Slide20

Guide Pex to Generate Test Data

[PexMethod(TestEmissionFilter = PexTestEmissionFilter.All)] public void TestBoundaryValuesAndInputPartition(int x, int y) { //boundary values/partitions for x > 0 && x <= 10 && y >= 1 PexAssume.IsTrue((x > 0)); if (x == 1) { } else if (x > 0) { } PexAssume.IsTrue((x <= 10)); if (x == 10) { } else if (x <= 10) { } PexAssume.IsTrue((y >= 1)); if (y == 1) { } else if (y > 1) { } }

20

Details see http://taoxie.cs.illinois.edu/publications/icsm10-coverage.pdf

http://pex4fun.com/default.aspx?language=CSharp&sample=_Template

using System;

using

Microsoft.Pex.Framework

;

using

Microsoft.Pex.Framework.Settings

;

[

PexClass

]

public class Program {

//insert your PUT here

}

Slide21

Parameterized Unit Tests Supported by Pex/Pex4Fun

using System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;[PexClass]public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool exist = s.member(i); s.insert(j); PexAssert.IsTrue(exist); } ….}

21

Slide22

Interface for IntSet

Class IntSet { public IntSet() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … }}sort IntSet imports Int, Bool signatures new : -> IntSet insert : IntSet × Int -> IntSet member : IntSet × Int -> Bool remove : IntSet × Int -> IntSet

22

http://www.cs.unc.edu/~stotts/723/adt.html

Slide23

(Buggy) Implementation for IntSet

Class IntSet { public IntSet() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … }}See the Set.cs that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/Set.cs Let’s copy it to http://pex4fun.com/default.aspx?language=CSharp&sample=_Template And Click “Ask Pex”

23

Slide24

Parameterized Unit Tests Supported by Pex/Pex4Fun

using System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;[PexClass]public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool existOld = s.member(i); s.insert(j); bool exist = s.member(i); PexAssert.IsTrue(existOld == exist); } ….}

24

Pex4Fun supports only one

PexMethod

at a time;

you can write multiple

PexMethods

but comment out other lines of “[

PexMethod

]” except one

Slide25

Axioms for IntSet

variables i, j : Int; s : IntSet Axioms:member(new(), i) = false member(insert(s, j), i) = if i = j then true else member(s, i)

25

http://www.cs.unc.edu/~stotts/723/adt.html

Is this complete?

How do we know?

Slide26

Guidelines for Completeness

Classify methods:constructors: return IntSetinspectors: take IntSet as argument, returning some other value.Identify key constructors, capable of constructing all possible object statese.g., insert, new. Identify others as auxiliary, e.g., remove is a destructive constructorCompleteness requires (at least): every inspector/auxiliary constructor is defined by one equation for each key constructor. 

26

Slide27

Add More Axioms

remove(new(), i) = new() remove(insert(s, j), i) = if i = j then remove(s, i) else insert(remove(s, i), j)

27

Are we done yet?

The completeness criterion (an equation defining member and remove for each of the new and insert constructors) is satisfied.

Slide28

Guidelines for Completeness

But does this really specify sets? Do the following properties hold?Order of insertion is irrelevant.insert(insert(s, i), j) = insert(insert(s, j), i)Multiple insertion is irrelevant.insert(insert(s, i), i) = insert(s, i)

28

Slide29

Interface (Implementation) for UIntStack

Class UIntStack { public UIntStack() {…}; public void Push(int k) { … } public void Pop() { … } public int Top() { … } public bool IsEmpty() { … } public int MaxSize() { … } public bool IsMember(int k) { … } public bool Equals(UIntStack s) { … } public int GetNumberOfElements() { … } public bool IsFull() { … }}

29

See the

UIntStack.cs

that can be downloaded from

http://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs

Slide30

Exercise: Write Parameterized Unit Tests (PUTs)

Class UIntStack { public UIntStack() {…}; public void Push(int k) { … } public void Pop() { … } public int Top() { … } public bool IsEmpty() { … } public int MaxSize() { … } public bool IsMember(int k) { … } public bool Equals(UIntStack s) { … } public int GetNumberOfElements() { … } public bool IsFull() { … }}

30

Let’s copy it to http://pex4fun.com/default.aspx?language=CSharp&sample=_Template And Click “Ask Pex”

Reminder: you have to comment earlier written “[PexMethod]” before you try Pex on your current PUT (Pex4Fun can handle only one PUT at a time)

See the

UIntStack.cs

that can be downloaded from

http://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs

Slide31

One Sample PUT

Below is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method. [PexMethod] public void TestPush([PexAssumeUnderTest]UIntStack s, int i) { //UIntStack s = new UIntStack(); PexAssume.IsTrue(!s.IsMember(i)); int oldCount = s.GetNumberOfElements(); s.Push(i); PexAssert.IsTrue(s.Top() == i); PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1); PexAssert.IsFalse(s.IsEmpty()); }

31

Slide32

Guideline of Writing PUT

Setup:

basic set up for invoking the method under test

Checkpoint: Run

Pex

to make sure that you don't miss any

Pex

assumptions (preconditions) for the PUT

Assert:

add assertions for asserting behavior of the method under test, involving

Adding

Pex

assertions

Adding

Pex

assumptions for helping assert

Adding method sequences for helping assert

Slide33

Setup

Select your method under test

m

Put its method call in your PUT

Create a parameter for your PUT as the class under test

c

(annotated it with

[PexAssumeUnderTest]

)

Create other parameters for your PUT for parameters of

m

if any

Add Pex assumptions for preconditions for all these parameters of PUT if any

Slide34

Setup - Example

[

PexMethod

]

public void

TestPush

([

PexAssumeUnderTest

]

UIntStack

s,

int

i) {

s.Push

(i);

}

You may write your factory method (see backup slides) to help

Pex

in test generation

If you get exceptions thrown

if indicating program faults, fix them

If indicating lack of PUT assumptions, add PUT assumptions

If indicating insufficient factory method assumptions or inappropriate scenarios, add PUT assumptions or improve factory method.

Slide35

Assert

Think about how you can assert the behavior

Do you need to invoke other (observer) helper methods in your assertions (besides asserting return values)?

Do you need to add assumptions so that your assertions can be valid?

Do you need to add some method sequence before the method under test to set up desirable state and cache values to be used in the assertions?

Slide36

Targets for Asserting

Return value of the method under test (MUT)

Argument object of MUT

Receiver object properties being modified by MUT (if public fields, directly assertable)

How to assert them?

Think about the intended behavior!

If you couldn't do so easily, follow the guidelines discussed next

Slide37

Cached Public Property Value

A

property

value before

invoking MUT

may need to be cached

and later used

.

Pattern 2.1/2.2: Assume, Arrange, Act, Assert

[

PexMethod

]

void

AssumeActAssert

(

ArrayList

list, object item) {

PexAssume.IsNotNull

(list); // assume

var

count

=

list.Count

; // arrange

list.Add

(item); // act

Assert.IsTrue

(

list.Count

==

count

+ 1); // assert

}

Slide38

Argument of MUT

Argument value of MUT may be used

Pattern 2.3:Constructor Test

[

PexMethod

]

void Constructor(

int

capacity) {

var

list = new

ArrayList

(

capacity

); // create

AssertInvariant

(list); // assert invariant

Assert.AreEqual

(

capacity

,

list.Capacity

); // assert

}

Slide39

Reciever or Argument of Earlier Method

Receiver or argument value of a method before invoking MUTPattern 2.4/5:Roundtrip[PexMethod]void ToStringParseRoundtrip(int value) { // two-way roundtrip string s = value.ToString(); int parsed = int.Parse(s); // assert Assert.AreEqual(value, parsed);}

value  s  parsed

Slide40

Observer Methods

Invoking observer methods on the modified object state

Pattern 2.6: State Relation

[

PexMethod

]

void

InsertContains

(string value) {

var

list = new List<string>();

list.Add

(value);

Assert.IsTrue

(

list.

Contains

(value));

}

Each modified object property should be read by at least one observer method.

Slide41

Observer Methods cont.

Forcing observer methods to return specific values (e.g., true or false) can force you to add specific assumptions or scenarios

[

PexMethod

]

void

PushIsFull

(

[

PexAssumeUnderTest

]

UIntStack

s,

int

value) {

PexAssume.IsTrue

(

s.GetSize

() == (

s.GetMaxSize

()-1));

s.Push

(value);

Assert.IsTrue

(

s.

IsFull

());

}

Slide42

Introduction to Software Testing (Ch 1)

© Ammann & Offutt

42

Parameterized Unit Tests in JUnit

Problem: Testing A Function With Similar Values

How To Avoid Test Code Bloat?

Simple Example: Adding Two Numbers

Adding a Given Pair of Numbers Is Just Like Adding Any Other Pair

You Really Only Want to Write One Test

Parameterized Unit Tests Call Constructor For Each Logical Set of Data Values

Same Tests Are Then Run On Each Set of Data Values

List of Data Values Identified with @Parameters Annotation

Slide43

Introduction to Software Testing (Ch 1)

© Ammann & Offutt

43

Parameterized Unit Tests in JUnit

import

org.junit

.*;

import

org.junit.runner.RunWith

;

import

org.junit.runners.Parameterized

;

import

org.junit.runners.Parameterized.Parameters

;

import static

org.junit.Assert

.*;

import

java.util

.*;

@

RunWith

(

Parameterized.class

)

public class

ParamTest

{

public

int

sum, a, b;

public

ParamTest

(

int

sum,

int

a,

int

b) {

this.sum

= sum;

this.a

= a;

this.b

= b; }

@Parameters public static Collection<Object[]> parameters() {

return

Arrays.asList

(new Object [][] {{0, 0, 0}, {2, 1, 1}});

}

@Test public void

additionTest

() {

assertEquals

(sum,

a+b

); }

}

Slide44

Introduction to Software Testing (Ch 1)

© Ammann & Offutt

44

JUnit Theories

These Are Unit Tests With Actual ParametersSo Far, We’ve Only Seen Parameterless Test MethodsContract Model: Assume, Act, AssertAssumptions (Preconditions) Limit Values AppropriatelyAction Performs Activity Under ScrutinyAssertions (Postconditions) Check Result

@Theory public void removeThenAddDoesNotChangeSet(

Set<String> set, String string) { // Parameters!

assumeTrue(set.contains(string)) ; // Assume

Set<String> copy = new HashSet<String>(set); // Act

copy.remove(string);

copy.add(string);

assertTrue (set.equals(copy)); // Assert //

// System.out.println(“Instantiated test: “ + set + “, “ + string);

}

Slide45

Introduction to Software Testing (Ch 1)

© Ammann & Offutt

45

Question: Where Does Data Come From?

Answer:All Combinations of Values from @DataPoint Annotations Where Assume Clause is TrueFour (of Nine) Combinations in This Particular CaseNote: @DataPoint Format is an Array.

@DataPoints

public static String[] string = {"ant", "bat", "cat"};

@DataPoints

public static Set[] sets = {

new HashSet(Arrays.asList("ant", "bat")),

new HashSet(Arrays.asList(“bat", “cat", “dog“, “elk”)),

new HashSet(Arrays.asList(“Snap”, “Crackle”, “Pop"))

};

Slide46

Introduction to Software Testing (Ch 1)

© Ammann & Offutt

46

JUnit Theories Need BoilerPlate

import org.junit.*;

import org.junit.runner.RunWith;

import static org.junit.Assert.*;

import static org.junit.Assume.*;

import org.junit.experimental.theories.DataPoint;

import org.junit.experimental.theories.DataPoints;

import org.junit.experimental.theories.Theories;

import org.junit.experimental.theories.Theory;

import java.util.*;

@RunWith(Theories.class)

public class SetTheoryTest {

… // See Earlier Slides

}

Slide47

Coding Duels

Pex

computes “semantic diff” in cloud

secret reference implementation vs.

code written in browser

You win when

Pex

finds no differences

secret

For more info, see our ICSE 2013 SEE paper:

http://taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf

Slide48

Behind the Scene of

Pex

for Fun

Secret Implementation

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

Player Implementation

class Player { public static int Puzzle(int x) { return x; }}

class Test {

public static void Driver(

int

x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}

behavior

Secret

Impl

==

Player

Impl

48

1,594,092

1,594,092

Slide49

Code Hunt Programming Game

https://www.codehunt.com/

For more info, see our ICSE 2015 JSEET paper: http://taoxie.cs.illinois.edu/publications/icse15jseet-codehunt.pdf

Code Hunt has had over

450,000

users in a year

with around 1,000 users a day

Slide50

Modeling in Science and Engineering

A model:Is an abstraction of the system from a particular perspectiveSupports investigation, construction and prediction Is not necessarily comprehensiveCan be expressed as a table, graphical diagram, formal notation,etc. Ubiquitous and Universal

50

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide51

Software Modeling Styles and Notations

51

Focus of

Spec Explorer

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide52

Model-Based Testing in a Nutshell

Model

Implementation

Expected Outputs (Test Oracle)

Inputs

(Test Sequences)

Control

Requirements

Generate

Observe

Feedback

Feedback

Author

Issue

Feedback

Test Cases

Verdict

Feedback

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide53

Model-Based Testing with Spec Explorer

Explore

Analyze

Generate

Execute

C# Model

(or other

.Net

Language)

Model Graph

Test Suite

VSTT Result

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide54

Walkthrough: Modeling and Testing a Web Service

Example: A Simple Web Shop

Web Shop Service

Web Shop Client

Billing

Shipping

AddToCart

Checkout

Bill

Ship

System Under Test

Web Shop Service

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide55

Web Shop Implementation in Windows Communication Foundation (WCF)

[ServiceContract(CallbackContract = typeof(IWebShopCallback))] public interface IWebShop { [OperationContract] // Add item to stock of web shop void AddItem(string itemName, int price); [OperationContract] // Add item to shopping cart of user bool AddToCart(string userName, string itemName); [OperationContract] // Check out void Checkout(string userName); } public interface IWebShopCallback { [OperationContract] // Initiate billing, returns true on success bool Bill(string userName, int amount); [OperationContract(IsOneWay = true)] // Initiate shipping void Ship(string userName, string itemName); }

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide56

Modeling Web Shop: Basic Approach

Import or declare action vocabulary in configuration scriptDefine state variables in C# model programDefine state update rules in C# model programDefine scenarios and generate state transition system

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide57

config Actions{ action static void TestAdapter.AddItem(string itemName, int price); action static void TestAdapter.AddToCart(string userName, string itemName); action static void TestAdapter.Checkout(string userName); action event static void TestAdapter.Billed(string userName, int amount, bool success); action event static void TestAdapter.Shipped(string userName, string itemName);}

Modeling Web Shop: Action Vocabulary

Adapter for the System-Under-Test

Reference to test adapter method

Indicates that this action is an event raised by adapter

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide58

Modeling WebShop: Define Model State in C#

enum ShopperStatus{ Shopping, CheckedOut, Billed }class ShopperInfo{ // Current status of this user internal ShopperStatus Status; // Items on cart so far internal SequenceContainer<string> Items;}// Foreach user shopping, contains info static MapContainer<string, ShopperInfo> shoppers;// Foreach item in the shop, contains pricestatic MapContainer<string, int> items;

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide59

Modeling WebShop: Defining State Update Rules

[Action(“AddToCart(userName,itemName)”)]static void AddToCart(string userName, string itemName){ Contracts.Requires(items.ContainsKey(itemName)); if (!shoppers.ContainsKey(userName)) shoppers[userName] = new ShopperInfo(); Contracts.Requires(InStatus(userName, ShopperStatus.Shopping)); shoppers[userName].Items.Add(itemName);} static bool InStatus(string userName, ShopperStatus status){ return shoppers.ContainsKey(userName) && shoppers[userName].Status == status;}

Binding rule to action in action vocabulary

Condition under which rule is enabled. Fires a state update

State update

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide60

Modeling WebShop: Define Scenarios

machine MultipleItemScenario() : Config{ AddItem("Bread", 2); AddItem("Cheese", 5); AddItem("Wine", 10) AddToCart("Paul","Wine"); AddToCart("Paul", "Cheese"); (Checkout | Billed | Shipped)*}machine MultipleItemSlice() : Config { MultipleItemScenario || construct model program from Config}

Combining scenario with C# model program

Or’ing actions in repetition (Note parameters omitted)

Sequence of actions

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide61

Modeling WebShop: Exploration of the Scenario

Parameters and control flow undetermined

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide62

Modeling WebShop: Exploration of the Model Slice

Choice point for system-under-test

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide63

Modeling WebShop: Execution of generated test code

Model expects bill of $2,

implementation bills $4

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide64

“BlueLine” Technical Document Testing Project

222 open protocol specifications tested22,847 pages66,962 person days (250+ years)Vendor test teamsHyderabad: 250Beijing: 100Tool dev center in Beijing (Spec Explorer and other)~10,000 document bugs filed

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide65

Comparison of Model-Based/Traditional Testing

Vendor teams had (guided) choice which approach to useOut of 222 protocols approx. ½ modeled, ½ traditional Modeling resulted in 42% productivity gain in average (some teams reached more than 300%)Overall, modeling saved 50 person-years (12,547 days)No special education background was requiredTeams of junior engineersMostly vendors, fresh out of college

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp

Slide66

Summary

https://msdn.microsoft.com/en-us/library/ee620411.aspx

https://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-956b-d9bfa4902745

A Model-Based Testing tool from Microsoft

Parameterized Unit Test

http://research.microsoft.com/pex/

IntelliTest

http://nmodel.codeplex.com/

NModel

Slide67

Q & A

67

Slide68

Code Hunt Programming Game

Slide69

69

Slide70

Test-Driven Development (TDD)

Basic Idea:

Write tests before code

Refine code with new tests

In more detail, TDD is a cycle of steps:

Add a test,

Run it and watch it fail,

Change the code as little as possible such that the test should pass,

Run the test again and see it succeed,

Refactor the code if needed.

Slide71

Note: TDD and specifications

TDD encourages writing specifications before code

Exemplary specification

Later, we will generalize TDD to

Parameterized TDD

Axiomatic specifications

Slide72

Parameterized Test-Driven Development

Write/refine Contract as PUT

Write/refine Code of Implementation

Fix-it (with Pex),Debug with generated tests

Use Generated Tests for Regression

Run

Pex

Bug in PUT

Bug in Code

failures

no failures

Slide73

Code Hunt Programming Game

Slide74

Code Hunt Programming Game

Slide75

Code Hunt Programming Game

Slide76

Code Hunt Programming Game

Slide77

Code Hunt Programming Game

Slide78

Code Hunt Programming Game

Slide79

Code Hunt Programming Game

Slide80

Code Hunt Programming Game

Slide81

Code Hunt Programming Game

Slide82

Code Hunt Programming Game

Slide83

Code Hunt Programming Game

Slide84

It’s a game!

iterative gameplayadaptivepersonalizedno cheatingclear winning criterion

secret

code

test cases

Slide85

(Buggy) Implementation for IntSet

Class IntSet { public IntSet() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … }}See the Set.cs that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/Set.cs Let’s copy it to http://pex4fun.com/default.aspx?language=CSharp&sample=_Template And Click “Ask Pex”

85

Slide86

Recall: Parameterized Unit Tests Supported by Pex/Pex4Fun

using System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;[PexClass]public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool existOld = s.member(i); s.insert(j); bool exist = s.member(i); PexAssert.IsTrue(existOld == exist); } ….}

86

Slide87

Force Pex/Pex4Fun to Display All Explored Test Inputs/Paths

using System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;[PexClass]public class Set { [PexMethod(TestEmissionFilter=PexTestEmissionFilter.All)] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool exist = s.member(i); s.insert(j); PexAssert.IsTrue(exist); } ….}

87

Slide88

Factory Method: Help Pex Generate Desirable Object States

In class, we show the factory method as below automatically synthesized by Pex after a user clicks “1 Object Creation” issue and then click “Accept/Edit Factory Method”. But it is not good enough to generate various types of object states. [PexFactoryMethod(typeof(UIntStack))] public static UIntStack Create(int k_i) { UIntStack uIntStack = new UIntStack(); uIntStack.Push(k_i); return uIntStack; // TODO: Edit factory method of UIntStack // This method should be able to configure the object in all possible ways. // Add as many parameters as needed, // and assign their values to each field by using the API. }

88

Slide89

Factory Method: Help Pex Generate Desirable Object States

Below is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method. [PexFactoryMethod(typeof(UIntStack))] public static UIntStack CreateVariedSizeAnyElemsStack(int[] elems) { PexAssume.IsNotNull(elems); UIntStack s = new UIntStack(); PexAssume.IsTrue(elems.Length <= (s.MaxSize() + 1)); for (int i = 0; i < elems.Length; i++) s.Push(elems[i]); return s; }

89

Slide90

Pex4Fun Not Supporting Factory Method - Workaround

If you try PUTs on Pex4Fun, which doesn’t support factory method, you can “embed” the factory method like the highlighted code portion below [PexMethod] public void TestPush(int[] elems, int i) { PexAssume.IsNotNull(elems); UIntStack s = new UIntStack(); PexAssume.IsTrue(elems.Length <= (s.MaxSize() + 1)); for (int i = 0; i < elems.Length; i++) s.Push(elems[i]); //UIntStack s = new UIntStack(); PexAssume.IsTrue(!s.IsMember(i)); int oldCount = s.GetNumberOfElements(); s.Push(i); PexAssert.IsTrue(s.Top() == i); PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1); PexAssert.IsFalse(s.IsEmpty()); }

90

Slide91

Understanding the Web Shop

Which of the following traces is valid?(assuming an initialization of the shop with AddItem(“wine”,10) and AddItem(“cheese”, 5))1AddToCart(“Peter”, “wine”) Checkout(“Peter”) Bill(“Peter”,10)/true Ship(“Peter”,”wine”)2AddToCart(“Peter”, “wine”) Checkout(“Peter”) Bill(“Peter”,10)/false Ship(“Peter”,”wine”)3AddToCart(“Peter”, “wine”) Checkout(“Peter”) Bill(“Peter”,12)/true Ship(“Peter”,”wine”)4AddToCart(“Peter”, “wine”) AddToCart(“Peter”,”cheese”) Checkout(“Peter”) Bill(“Peter”,15)/true Ship(“Peter”,”wine”) Ship(“Peter”,”cheese”)5AddToCart(“Peter”, “wine”) AddToCart(“Peter”,”cheese”) Checkout(“Peter”) Bill(“Peter”,15)/true Ship(“Peter”,”cheese”) Ship(“Peter”,”wine”)

Slides from

https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316

©

W.

Grieskamp