Tao Xie University of Illinois at UrbanaChampaign httptaoxiecsillinoiseducoursestesting Work described in the slides was done in collaboration with the Pex team Nikolai Tillmann Peli de Halleux ID: 569444
Download Presentation The PPT/PDF document "Requirement Specifications based Test Au..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Requirement Specifications based Test Automation
Tao XieUniversity of Illinois at Urbana-Champaignhttp://taoxie.cs.illinois.edu/courses/testing/
Work described in the slides was done in collaboration with the
Pex
team (Nikolai Tillmann, Peli de Halleux,
Pratap
Lakshman
, et al.) @Microsoft Research, students @Illinois ASE, and other collaboratorsSlide2
Techniques for writing tests
Black-box (from specifications)Equivalence partitioningBoundary value analysisWhite-box (from code)Branch coverageFault-based testing (from common faults)
2Slide3
3
What is a Unit Test?
A unit test is a small program with assertions.
[
TestMethod
]
public void Add()
{
HashSet
set = new
HashSet
();
set.Add
(3);
set.Add
(14);
Assert.AreEqual
(
set.Count
, 2);
}
Many developers write such unit tests by hand, involving
determining a meaningful sequence of method calls,
selecting exemplary argument values (the test
inputs
),
stating assertions.Slide4
Unit Testing: Measuring Quality
Coverage: Are all parts of the program exercised?statementsbasic blocksexplicit/implicit branches…
Assertions: Does the program do the right thing?
test oracle
Experience:
Just high coverage or large number of assertions is no good quality indicator.
Only both together are!Slide5
Advantages of tests as specs
Concrete, easy to understandDon’t need new languageEasy to see if program meets the specMaking tests forces you to talk to customer and learn the problem
Making tests forces you to think about design of system (classes, methods, etc.)
5Slide6
Disadvantages of tests as specs
Too specificHard to test that something can’t happenCan’t withdraw more money than you have in the systemCan’t break into the systemCan’t cause a very long transaction that hangs the systemTends to be verbose
6Slide7
Tests as specifications
Tests show how to use the systemTests need to be readableNeed comments that describe their purpose or need good namesKeep short, delete duplicate or redundant
7Slide8
Parameterized Unit Test
Parameterized Unit Test
A parameterized unit test is a small program that takes
some
inputs and states assumptions and assertions.
8
See later slides on JUnit: @Theory (multiple parameters) @Parameters (single parameter)Slide9
Parameterized Unit Test
Parameterized Unit Testing
Parameterized Unit Tests
serve as specifications
can be leveraged by (automatic) test input generators
fit in development environment, evolve with the codeSlide10
Hand-written
Test Generation Process
10
//
FooTest.cs
[
TestClass
,
PexClass
]
partial class
FooTest
{
[
PexMethod
]
void
Test
(Foo foo)
{…}
//
FooTest.Test.cs
partial class
FooTest
{
[
TestMethod
]
void Test_1()
{
this
.
Test
(new
Foo
(1)); }
[TestMethod] void Test_1() { this.Test(new Foo(2)); } …}
Pex
User writes parameterized tests Lives inside a test class
Generated unit tests Pex not required for re-execution xUnit unit tests
xUnit
Attributes
Pex
Attributes
Parameterized Unit Test
Partial Class
Generated
http://msdn.microsoft.com/en-us/library/wa80x488(VS.80).aspx
Slide11
PUTs separate concerns
PUTs separate two concerns:(1) The specification of external behavior
(i.e., assertions)
(2) The selection of internal test inputs
(i.e., coverage)
In many cases, a test generation tool (e.g.,
Pex
) can construct a
small test suite
with
high coverage
!Slide12
PUTs are
algebraic specsA PUT can be read as a universally quantified, conditional axiom.
int
name,
int
data.
name
≠
null ⋀ data
≠
null
⇒
equals(
ReadResource
(name,
WriteResource
(name, data)),
data)Slide13
Pex4Fun – Turning
Pex Online
1,792,222
clicked 'Ask
Pex
!'
http://pex4fun.com/default.aspx?language=CSharp&sample=_Template
Slide14
http://research.microsoft.com/pex/
Dynamic Symbolic Execution (DSE)
aka.
Concolic
Testing
[Godefroid et al. 05][Sen et al. 05][Tillmann et al. 08]Instrument code to explore feasible paths
Behind the Scene of Pex4FunSlide15
http://research.microsoft.com/pex/Slide16
void
CoverMe
(
int
[] a)
{
if (a == null) return;
if (
a.Length
> 0)
if (a[0] ==
1234567890
)
throw new Exception("bug");
}
a.Length
>0
a[0]==123…
T
F
T
F
F
a==null
T
Constraints to solve
a!=null
a!=null &&
a.Length
>0
a!=null &&
a.Length
>0 &&
a[0]==
123456890
Input
null
{}
{0}
{123…}
Execute&Monitor
Solve
Choose next path
Observed constraints
a==null
a!=null &&
!(
a.Length
>0)
a==null &&
a.Length
>0 &&
a[0]!=
1234567890
a==null &&
a.Length
>0 &&
a[0]==
1234567890
Done: There is no path left.
Dynamic Symbolic Execution in
Pex
http://pex4fun.com/HowDoesPexWorkSlide17
Pex
is Part of Visual Studio 2015 Enterprise Edition!As new feature of “IntelliTest”
https://www.visualstudio.com/news/vs2015-vs#Testing
Pex
:
30K
downloads after 20 monthsActive user community: 1.4K forum posts during ~3 yearsSlide18
From
http://bbcode.codeplex.com/
NUnit Extension
– 13024 downloads
xUnit.net Extension
- 8589 downloads
Impact of
IntelliTestSlide19
Domain Matrix for Testing Complex Condition
19Slide20
Guide
Pex to Generate Test Data [PexMethod(TestEmissionFilter =
PexTestEmissionFilter.All
)]
public void
TestBoundaryValuesAndInputPartition
(int x, int y) { //boundary values/partitions for x > 0 && x <= 10 && y >= 1 PexAssume.IsTrue((x > 0));
if (x == 1) { } else if (x > 0) { }
PexAssume.IsTrue
((x <= 10));
if (x == 10) { }
else if (x <= 10) { }
PexAssume.IsTrue
((y >= 1));
if (y == 1) { }
else if (y > 1) { }
}
20
Details see
http://taoxie.cs.illinois.edu/publications/icsm10-coverage.pdf
http://pex4fun.com/default.aspx?language=CSharp&sample=_Template using System;using
Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;[PexClass]public class Program { //insert your PUT here}Slide21
Parameterized Unit Tests Supported by
Pex/Pex4Funusing System;using Microsoft.Pex.Framework;
using
Microsoft.Pex.Framework.Settings
;
[
PexClass]public class Set { [PexMethod] public static void
testMemberAfterInsertNotEqual(Set s, int i, int
j) {
PexAssume.IsTrue
(s != null);
PexAssume.IsTrue
(i != j);
bool
exist =
s.member
(i);
s.insert
(j);
PexAssert.IsTrue(exist); } ….}
21Slide22
Interface for IntSet
Class IntSet { public IntSet() {…};
public void insert(
int
e) { … }
public
Bool member(int e) { … } public void remove(int e) { … }}sort IntSet
imports Int, Bool signatures
new : ->
IntSet
insert :
IntSet
×
Int
->
IntSet
member :
IntSet
×
Int
->
Bool remove : IntSet × Int -> IntSet22
http://www.cs.unc.edu/~stotts/723/adt.htmlSlide23
(Buggy) Implementation for
IntSetClass IntSet { public IntSet
() {…};
public void insert(
int
e) { … }
public Bool member(int e) { … } public void remove(int e) { … }}See the Set.cs
that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/Set.cs
Let’s copy it to
http://pex4fun.com/default.aspx?language=CSharp&sample=_Template
And Click “Ask
Pex
”
23Slide24
Parameterized Unit Tests Supported by
Pex/Pex4Funusing System;using Microsoft.Pex.Framework;
using
Microsoft.Pex.Framework.Settings
;
[
PexClass]public class Set { [PexMethod] public static void
testMemberAfterInsertNotEqual(Set s, int i, int
j) {
PexAssume.IsTrue
(s != null);
PexAssume.IsTrue
(i != j);
bool
existOld
=
s.member
(i);
s.insert
(j);
bool exist = s.member(i); PexAssert.IsTrue(existOld == exist); } ….
} 24Pex4Fun supports only one PexMethod
at a time; you can write multiple PexMethods but comment out other lines of “[PexMethod]” except oneSlide25
Axioms for IntSet
variables i, j : Int; s : IntSet Axioms:member(new(), i) = false
member(insert(s, j), i) =
if i = j then true
else member(s, i)
25
http://www.cs.unc.edu/~stotts/723/adt.html
Is this complete?
How do we know?Slide26
Guidelines for Completeness
Classify methods:constructors: return IntSetinspectors: take IntSet as argument, returning some other value.Identify key constructors, capable of constructing all possible object states
e.g., insert, new.
Identify others as auxiliary,
e.g., remove is a destructive constructor
Completeness requires (at least):
every inspector/auxiliary constructor is defined by one equation for each key constructor. 26Slide27
Add More Axioms
remove(new(), i) = new() remove(insert(s, j), i) = if i = j then remove(s, i) else insert(remove(s, i), j)
27
Are we done yet?
The completeness criterion (an equation defining member and remove for each of the new and insert constructors) is satisfied.Slide28
Guidelines for Completeness
But does this really specify sets? Do the following properties hold?Order of insertion is irrelevant.insert(insert(s, i), j) = insert(insert(s, j), i)Multiple insertion is irrelevant.insert(insert(s, i), i) = insert(s, i)
28Slide29
Interface (Implementation) for
UIntStackClass UIntStack { public UIntStack() {…};
public void Push(
int
k) { … }
public void Pop() { … }
public int Top() { … } public bool IsEmpty() { … } public int
MaxSize() { … } public bool
IsMember
(
int
k) { … }
public
bool
Equals(
UIntStack
s) { … }
public
int
GetNumberOfElements
() { … }
public
bool IsFull() { … }}29
See the UIntStack.cs that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs Slide30
Exercise: Write Parameterized Unit Tests (PUTs)
Class UIntStack { public UIntStack() {…}; public void Push(
int
k) { … }
public void Pop() { … }
public
int Top() { … } public bool IsEmpty() { … } public int MaxSize
() { … } public bool IsMember(
int
k) { … }
public
bool
Equals(
UIntStack
s) { … }
public
int
GetNumberOfElements
() { … }
public
bool
IsFull() { … }}30
Let’s copy it to http://pex4fun.com/default.aspx?language=CSharp&sample=_Template And Click “Ask Pex”Reminder: you have to comment earlier written “[PexMethod
]” before you try Pex on your current PUT (Pex4Fun can handle only one PUT at a time)See the UIntStack.cs that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs Slide31
One Sample PUT
Below is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method.
[
PexMethod
]
public void
TestPush([PexAssumeUnderTest]UIntStack s, int i) { //
UIntStack s = new UIntStack();
PexAssume.IsTrue
(!
s.IsMember
(
i
));
int
oldCount
=
s.GetNumberOfElements
();
s.Push
(i); PexAssert.IsTrue(s.Top() == i); PexAssert.IsTrue(s.GetNumberOfElements
() == oldCount+1); PexAssert.IsFalse(s.IsEmpty()); }31Slide32
Guideline of Writing PUT
Setup: basic set up for invoking the method under test
Checkpoint: Run
Pex
to make sure that you don't miss any
Pex
assumptions (preconditions) for the PUT
Assert: add assertions for asserting behavior of the method under test, involving
Adding
Pex
assertions
Adding
Pex
assumptions for helping assert
Adding method sequences for helping assertSlide33
Setup
Select your method under test m
Put its method call in your PUT
Create a parameter for your PUT as the class under test
c
(annotated it with
[PexAssumeUnderTest])Create other parameters for your PUT for parameters of
m if any
Add Pex assumptions for preconditions for all these parameters of PUT if anySlide34
Setup - Example
[
PexMethod
]
public void
TestPush
([
PexAssumeUnderTest]
UIntStack
s,
int
i) {
s.Push
(i);
}
You may write your factory method (see backup slides) to help
Pex
in test generation
If you get exceptions thrown
if indicating program faults, fix them
If indicating lack of PUT assumptions, add PUT assumptions
If indicating insufficient factory method assumptions or inappropriate scenarios, add PUT assumptions or improve factory method.Slide35
Assert
Think about how you can assert the behavior
Do you need to invoke other (observer) helper methods in your assertions (besides asserting return values)?
Do you need to add assumptions so that your assertions can be valid?
Do you need to add some method sequence before the method under test to set up desirable state and cache values to be used in the assertions?Slide36
Targets for Asserting
Return value of the method under test (MUT)
Argument object of MUT
Receiver object properties being modified by MUT (if public fields, directly assertable)
How to assert them?
Think about the intended behavior!
If you couldn't do so easily, follow the guidelines discussed nextSlide37
Cached Public Property Value
A property
value before
invoking MUT
may need to be cached
and later used
.Pattern 2.1/2.2: Assume, Arrange, Act, Assert
[PexMethod
]
void
AssumeActAssert
(
ArrayList
list, object item) {
PexAssume.IsNotNull
(list); // assume
var
count
=
list.Count
; // arrange
list.Add(item); // act Assert.IsTrue
(list.Count == count + 1); // assert
}Slide38
Argument of MUT
Argument value of MUT may be used
Pattern 2.3:Constructor Test
[
PexMethod
]
void Constructor(int
capacity) {
var
list = new
ArrayList
(
capacity
); // create
AssertInvariant
(list); // assert invariant
Assert.AreEqual
(
capacity
,
list.Capacity
); // assert
}Slide39
Reciever or Argument of Earlier Method
Receiver or argument value of a method before invoking MUT
Pattern 2.4/5:Roundtrip
[
PexMethod
]
void
ToStringParseRoundtrip(
int
value) {
// two-way roundtrip
string s =
value
.ToString
();
int
parsed =
int.Parse
(s);
// assert
Assert.AreEqual
(
value
, parsed);}
value
s parsedSlide40
Observer Methods
Invoking observer methods on the modified object state
Pattern 2.6: State Relation
[
PexMethod
]
void InsertContains
(string value) {
var
list = new List<string>();
list.Add
(value);
Assert.IsTrue
(
list.
Contains
(value));
}
Each modified object property should be read by at least one observer method.Slide41
Observer Methods
cont.Forcing observer methods to return specific values (e.g., true or false) can force you to add specific assumptions or scenarios
[
PexMethod
]
void
PushIsFull([
PexAssumeUnderTest]UIntStack
s,
int
value) {
PexAssume.IsTrue
(
s.GetSize
() == (
s.GetMaxSize
()-1));
s.Push
(value);
Assert.IsTrue
(
s.IsFull ());
}Slide42
Introduction to Software Testing (
Ch
1)
© Ammann & Offutt
42
Parameterized Unit Tests in JUnit
Problem: Testing A Function With Similar Values
How To Avoid Test Code Bloat?
Simple Example: Adding Two Numbers
Adding a Given Pair of Numbers Is Just Like Adding Any Other Pair
You Really Only Want to Write One Test
Parameterized Unit Tests Call Constructor For Each Logical Set of Data Values
Same Tests Are Then Run On Each Set of Data Values
List of Data Values Identified with @Parameters AnnotationSlide43
Introduction to Software Testing (
Ch
1)
© Ammann & Offutt
43
Parameterized Unit Tests in JUnit
import
org.junit
.*;
import
org.junit.runner.RunWith
;
import
org.junit.runners.Parameterized
;
import
org.junit.runners.Parameterized.Parameters
;
import static
org.junit.Assert
.*;
import
java.util
.*;
@
RunWith
(
Parameterized.class
)
public class
ParamTest
{
public
int
sum, a, b;
public
ParamTest
(
int
sum,
int a, int b) { this.sum = sum; this.a = a; this.b = b; } @Parameters public static Collection<Object[]> parameters() { return Arrays.asList (new Object [][] {{0, 0, 0}, {2, 1, 1}}); } @Test public void additionTest() { assertEquals(sum, a+b); }}Slide44
Introduction to Software Testing (Ch 1)
© Ammann & Offutt
44
JUnit Theories
These Are Unit Tests With Actual Parameters
So Far, We’ve Only Seen
Parameterless
Test Methods
Contract Model: Assume, Act, Assert
Assumptions (Preconditions) Limit Values Appropriately
Action Performs Activity Under Scrutiny
Assertions (
Postconditions
) Check Result
@Theory public void removeThenAddDoesNotChangeSet(
Set<String> set, String string) { // Parameters!
assumeTrue(set.contains(string)) ; // Assume
Set<String> copy = new HashSet<String>(set); // Act
copy.remove(string);
copy.add(string);
assertTrue (set.equals(copy)); // Assert //
// System.out.println(“Instantiated test: “ + set + “, “ + string);
}Slide45
Introduction to Software Testing (
Ch
1)
© Ammann & Offutt
45
Question: Where Does Data Come From?
Answer:
All Combinations of Values from @
DataPoint
Annotations Where Assume Clause is True
Four (of Nine) Combinations in This Particular Case
Note: @
DataPoint
Format is an Array.
@DataPoints
public static String[] string = {"ant", "bat", "cat"};
@DataPoints
public static Set[] sets = {
new HashSet(Arrays.asList("ant", "bat")),
new HashSet(Arrays.asList(“bat", “cat", “dog“, “elk”)),
new HashSet(Arrays.asList(“Snap”, “Crackle”, “Pop"))
};Slide46
Introduction to Software Testing (
Ch
1)
© Ammann & Offutt
46
JUnit Theories Need
BoilerPlate
import org.junit.*;
import org.junit.runner.RunWith;
import static org.junit.Assert.*;
import static org.junit.Assume.*;
import org.junit.experimental.theories.DataPoint;
import org.junit.experimental.theories.DataPoints;
import org.junit.experimental.theories.Theories;
import org.junit.experimental.theories.Theory;
import java.util.*;
@RunWith(Theories.class)
public class SetTheoryTest {
… // See Earlier Slides
}
Slide47
Coding Duels
Pex
computes “semantic diff” in cloud
secret reference implementation vs.
code written in browser
You win when
Pex
finds no differences
secret
For more info, see our ICSE 2013 SEE paper:
http://taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf
Slide48
Behind the Scene of
Pex
for Fun
Secret
Implementation
class Secret {
public static
int Puzzle(int x) {
if (x <= 0) return 1;
return x * Puzzle(x-1);
}
}
Player
Implementation
class Player {
public static
int
Puzzle(
int
x) {
return
x
; }}
class Test {
public static void Driver(int x) { if (
Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}
behavior
Secret
Impl
==
Player
Impl
48
1,594,092
1,594,092Slide49
Code Hunt Programming Game
https://www.codehunt.com/
For more info, see our ICSE 2015 JSEET paper:
http://taoxie.cs.illinois.edu/publications/icse15jseet-codehunt.pdf
Code Hunt has had over 450,000
users in a yearwith around 1,000 users a daySlide50
Modeling in Science and Engineering
A model:Is an abstraction of the system from a particular perspectiveSupports investigation
,
construction
and
prediction Is not necessarily comprehensiveCan be expressed as a table, graphical diagram, formal notation,etc.
Ubiquitous and Universal
50
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
GrieskampSlide51
Software Modeling Styles and Notations
51
Focus of
Spec Explorer
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
GrieskampSlide52
Model-Based Testing in a Nutshell
Model
Implementation
Expected Outputs (Test Oracle)
Inputs
(Test Sequences)
Control
Requirements
Generate
Observe
Feedback
Feedback
Author
Issue
Feedback
Test Cases
Verdict
Feedback
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
GrieskampSlide53
Model-Based Testing with Spec Explorer
Explore
Analyze
Generate
Execute
C# Model
(or other
.Net
Language)
Model Graph
Test Suite
VSTT Result
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
GrieskampSlide54
Walkthrough: Modeling and Testing a Web Service
Example: A Simple Web Shop
Web Shop Service
Web Shop Client
Billing
Shipping
AddToCart
Checkout
Bill
Ship
System Under Test
Web Shop Service
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
GrieskampSlide55
Web Shop Implementation in Windows Communication Foundation (WCF)
[ServiceContract(CallbackContract = typeof(IWebShopCallback
))]
public interface
IWebShop
{
[OperationContract] // Add item to stock of web shop void AddItem(string itemName, int price); [OperationContract
] // Add item to shopping cart of user bool AddToCart(string userName
, string
itemName
);
[
OperationContract
] // Check out
void Checkout(string
userName
);
}
public interface
IWebShopCallback
{
[
OperationContract] // Initiate billing, returns true on success bool Bill(string userName, int amount); [OperationContract
(IsOneWay = true)] // Initiate shipping void Ship(string userName, string itemName); }
Slides from https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316 ©W. GrieskampSlide56
Modeling Web Shop: Basic Approach
Import or declare action vocabulary in configuration scriptDefine state variables in C# model programDefine state update rules in C# model programDefine scenarios and generate state transition system
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
GrieskampSlide57
config
Actions{ action static void TestAdapter.AddItem(string itemName, int price); action static void
TestAdapter.AddToCart
(string
userName
, string
itemName); action static void TestAdapter.Checkout(string userName); action event static void TestAdapter.Billed(string userName
, int amount, bool success); action event static void
TestAdapter.Shipped
(string
userName
, string
itemName
);
}
Modeling Web Shop: Action Vocabulary
Adapter for the System-Under-Test
Reference to test adapter method
Indicates that this action is an event raised by adapter
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
GrieskampSlide58
Modeling
WebShop: Define Model State in C#enum ShopperStatus
{ Shopping,
CheckedOut
, Billed }
class
ShopperInfo{ // Current status of this user internal ShopperStatus Status; // Items on cart so far internal SequenceContainer<string> Items;}
// Foreach user shopping, contains info static MapContainer
<string,
ShopperInfo
> shoppers;
//
Foreach
item in the shop, contains price
static
MapContainer
<string,
int
> items;
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
GrieskampSlide59
Modeling
WebShop: Defining State Update Rules[Action(“AddToCart(
userName,itemName
)”)]
static void AddToCart(string
userName
, string itemName){ Contracts.Requires(items.ContainsKey(itemName)); if (!shoppers.ContainsKey
(userName)) shoppers[userName] = new
ShopperInfo
();
Contracts.Requires
(
InStatus
(
userName
,
ShopperStatus.Shopping
));
shoppers[
userName
].
Items.Add(itemName);} static bool InStatus(string userName, ShopperStatus status)
{ return shoppers.ContainsKey(userName) && shoppers[userName].Status == status;}
Binding rule to action in action vocabulary
Condition under which rule is enabled. Fires a state update
State update
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
GrieskampSlide60
Modeling
WebShop: Define Scenariosmachine MultipleItemScenario() :
Config
{
AddItem
("Bread", 2); AddItem("Cheese", 5); AddItem("Wine", 10) AddToCart("Paul","Wine"); AddToCart("Paul", "Cheese"); (Checkout | Billed | Shipped)*}
machine MultipleItemSlice() : Config
{
MultipleItemScenario
|| construct model program from
Config
}
Combining scenario with C# model program
Or’ing
actions in repetition
(Note parameters omitted)
Sequence of actions
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
GrieskampSlide61
Modeling
WebShop: Exploration of the Scenario
Parameters and control flow undetermined
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
GrieskampSlide62
Modeling
WebShop: Exploration of the Model Slice
Choice point for system-under-test
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
GrieskampSlide63
Modeling
WebShop: Execution of generated test code
Model expects bill of $2,
implementation bills $4
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
GrieskampSlide64
“
BlueLine” Technical Document Testing Project222 open protocol specifications tested22,847 pages66,962 person days (250+ years)Vendor test teams
Hyderabad: 250
Beijing: 100
Tool
dev
center in Beijing (Spec Explorer and other)~10,000 document bugs filedSlides from https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316 ©W. GrieskampSlide65
Comparison of
Model-Based/Traditional TestingVendor teams had (guided) choice which approach to useOut of 222 protocols approx. ½ modeled, ½ traditional Modeling resulted in 42%
productivity gain in average (some teams reached more than
300%
)
Overall, modeling saved 50 person-years (12,547 days)
No special education background was requiredTeams of junior engineersMostly vendors, fresh out of collegeSlides from https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©W. GrieskampSlide66
Summary
https://msdn.microsoft.com/en-us/library/ee620411.aspx
https://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-956b-d9bfa4902745
A Model-Based Testing tool from Microsoft
Parameterized Unit Test
http://research.microsoft.com/pex/
IntelliTest
http://nmodel.codeplex.com/
NModelSlide67
Q & A
67Slide68
Code Hunt Programming GameSlide69
69Slide70
Test-Driven Development (TDD)
Basic Idea:Write tests before codeRefine code with new testsIn more detail, TDD is a cycle of steps:Add a test,Run it and watch it fail,
Change the code as little as possible such that the test should pass,
Run the test again and see it succeed,
Refactor the code if needed.Slide71
Note: TDD and specifications
TDD encourages writing specifications before codeExemplary specificationLater, we will generalize TDD toParameterized TDDAxiomatic specificationsSlide72
Parameterized
Test-Driven DevelopmentWrite/refine Contract as PUT
Write/refine Code
of Implementation
Fix-it (with
Pex
),
Debug with generated tests
Use Generated Tests
for Regression
Run
Pex
Bug in PUT
Bug in Code
failures
no failuresSlide73
Code Hunt Programming GameSlide74
Code Hunt Programming GameSlide75
Code Hunt Programming GameSlide76
Code Hunt Programming GameSlide77
Code Hunt Programming GameSlide78
Code Hunt Programming GameSlide79
Code Hunt Programming GameSlide80
Code Hunt Programming GameSlide81
Code Hunt Programming GameSlide82
Code Hunt Programming GameSlide83
Code Hunt Programming GameSlide84
It’s a game!
iterative gameplayadaptivepersonalizedno cheating
clear winning criterion
secret
code
test casesSlide85
(Buggy) Implementation for
IntSetClass IntSet { public IntSet
() {…};
public void insert(
int
e) { … }
public Bool member(int e) { … } public void remove(int e) { … }}See the Set.cs
that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/Set.cs
Let’s copy it to
http://pex4fun.com/default.aspx?language=CSharp&sample=_Template
And Click “Ask
Pex
”
85Slide86
Recall: Parameterized Unit Tests Supported by
Pex/Pex4Funusing System;using Microsoft.Pex.Framework;
using
Microsoft.Pex.Framework.Settings
;
[
PexClass]public class Set { [PexMethod] public static void
testMemberAfterInsertNotEqual(Set s, int i, int
j) {
PexAssume.IsTrue
(s != null);
PexAssume.IsTrue
(i != j);
bool
existOld
=
s.member
(i);
s.insert
(j);
bool exist = s.member(i); PexAssert.IsTrue(existOld == exist); } ….
} 86Slide87
Force
Pex/Pex4Fun to Display All Explored Test Inputs/Pathsusing System;using Microsoft.Pex.Framework;
using
Microsoft.Pex.Framework.Settings
;
[
PexClass]public class Set { [PexMethod(TestEmissionFilter
=PexTestEmissionFilter.All)
]
public static void
testMemberAfterInsertNotEqual
(Set s,
int
i
,
int
j) {
PexAssume.IsTrue
(s != null);
PexAssume.IsTrue
(i != j); bool exist = s.member(i); s.insert(j); PexAssert.IsTrue(exist);
} ….} 87Slide88
Factory Method: Help
Pex Generate Desirable Object StatesIn class, we show the factory method as below automatically synthesized by Pex after a user clicks “1 Object Creation” issue and then click “Accept/Edit Factory Method”. But it is not good enough to generate various types of object states.
[
PexFactoryMethod
(
typeof
(UIntStack))] public static UIntStack Create(int k_i) {
UIntStack uIntStack = new
UIntStack
();
uIntStack.Push
(
k_i
);
return
uIntStack
;
// TODO: Edit factory method of
UIntStack
// This method should be able to configure the object in all possible ways.
// Add as many parameters as needed,
// and assign their values to each field by using the API.
}
88Slide89
Factory Method: Help
Pex Generate Desirable Object StatesBelow is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex
also generates argument values for the factory method.
[
PexFactoryMethod
(
typeof(UIntStack))] public static UIntStack CreateVariedSizeAnyElemsStack(int[]
elems) { PexAssume.IsNotNull
(
elems
);
UIntStack
s = new
UIntStack
();
PexAssume.IsTrue
(
elems.Length
<= (
s.MaxSize
() + 1));
for (int i = 0; i < elems.Length; i++)
s.Push(elems[i]); return s; }89Slide90
Pex4Fun Not Supporting Factory Method - Workaround
If you try PUTs on Pex4Fun, which doesn’t support factory method, you can “embed” the factory method like the highlighted code portion below [PexMethod
]
public void
TestPush
(
int[] elems, int i) {
PexAssume.IsNotNull(elems);
UIntStack
s = new
UIntStack
();
PexAssume.IsTrue
(
elems.Length
<= (
s.MaxSize
() + 1));
for (
int
i
= 0; i < elems.Length; i++)
s.Push(elems[i]); //UIntStack s = new
UIntStack(); PexAssume.IsTrue(!s.IsMember(i)); int oldCount
=
s.GetNumberOfElements
();
s.Push
(
i
);
PexAssert.IsTrue
(
s.Top() == i); PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1); PexAssert.IsFalse(s.IsEmpty()); }90Slide91
Understanding the Web Shop
Which of the following traces is valid?
(assuming an initialization of the shop
with
AddItem
(“wine”,10) and
AddItem(“cheese”, 5))
1
AddToCart(“Peter”, “wine”)
Checkout(“Peter”)
Bill(“Peter”,10)/true
Ship(“
Peter”,”wine
”)
2
AddToCart(“Peter”, “wine”)
Checkout(“Peter”)
Bill(“Peter”,10)/false
Ship(“
Peter”,”wine
”)
3
AddToCart(“Peter”, “wine”)
Checkout(“Peter”)
Bill(“Peter”,12)/true
Ship(“
Peter”,”wine
”)
4
AddToCart(“Peter”, “wine”)
AddToCart(“
Peter”,”cheese
”)
Checkout(“Peter”)
Bill(“Peter”,15)/true
Ship(“
Peter”,”wine
”)
Ship(“
Peter”,”cheese
”)
5
AddToCart(“Peter”, “wine”)
AddToCart(“
Peter”,”cheese
”)
Checkout(“Peter”)
Bill(“Peter”,15)/true
Ship(“
Peter”,”cheese
”)
Ship(“
Peter”,”wine
”)
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp