Presentations text content in Requirement Specifications based Test Automation
Slide1
Requirement Specifications based Test Automation
Tao XieUniversity of Illinois at Urbana-Champaignhttp://taoxie.cs.illinois.edu/courses/testing/
Work described in the slides was done in collaboration with the
Pex
team (Nikolai Tillmann, Peli de Halleux,
Pratap
Lakshman
, et al.) @Microsoft Research, students @Illinois ASE, and other collaborators
Slide2Techniques for writing tests
Black-box (from specifications)Equivalence partitioningBoundary value analysisWhite-box (from code)Branch coverageFault-based testing (from common faults)
2
Slide33
What is a Unit Test?
A unit test is a small program with assertions.
[
TestMethod
]
public void Add()
{
HashSet
set = new
HashSet
();
set.Add
(3);
set.Add
(14);
Assert.AreEqual
(
set.Count
, 2);
}
Many developers write such unit tests by hand, involving
determining a meaningful sequence of method calls,
selecting exemplary argument values (the test
inputs
),
stating assertions.
Slide4Unit Testing: Measuring Quality
Coverage: Are all parts of the program exercised?
statements
basic blocks
explicit/implicit branches
…
Assertions: Does the program do the right thing?
test oracle
Experience:
Just high coverage or large number of assertions is no good quality indicator.
Only both together are!
Slide5Advantages of tests as specs
Concrete, easy to understandDon’t need new languageEasy to see if program meets the specMaking tests forces you to talk to customer and learn the problemMaking tests forces you to think about design of system (classes, methods, etc.)
5
Slide6Disadvantages of tests as specs
Too specificHard to test that something can’t happenCan’t withdraw more money than you have in the systemCan’t break into the systemCan’t cause a very long transaction that hangs the systemTends to be verbose
6
Slide7Tests as specifications
Tests show how to use the systemTests need to be readableNeed comments that describe their purpose or need good namesKeep short, delete duplicate or redundant
7
Slide8Parameterized Unit Test
Parameterized Unit Test
A parameterized unit test is a small program that takes some inputs and states assumptions and assertions.
8
See later slides on JUnit: @Theory (multiple parameters) @Parameters (single parameter)
Slide9Parameterized Unit Test
Parameterized Unit Testing
Parameterized Unit Testsserve as specificationscan be leveraged by (automatic) test input generatorsfit in development environment, evolve with the code
Slide10Hand-written
Test Generation Process
10
// FooTest.cs[TestClass, PexClass]partial class FooTest{ [PexMethod] void Test(Foo foo) {…}
// FooTest.Test.cspartial class FooTest{ [TestMethod] void Test_1() { this.Test(new Foo(1)); } [TestMethod] void Test_1() { this.Test(new Foo(2)); } …}
Pex
User writes parameterized tests Lives inside a test class
Generated unit tests Pex not required for re-execution xUnit unit tests
xUnit
Attributes
Pex
Attributes
Parameterized Unit Test
Partial Class
Generated
http://msdn.microsoft.com/en-us/library/wa80x488(VS.80).aspx
Slide11
PUTs separate concerns
PUTs
separate two concerns:
(1) The specification of external behavior
(i.e., assertions)
(2) The selection of internal test inputs
(i.e., coverage)
In many cases, a test generation tool (e.g.,
Pex
) can construct a
small test suite
with
high coverage
!
Slide12PUTs are algebraic specs
A PUT can be read as a universally quantified, conditional axiom.
int
name,
int
data.
name
≠
null ⋀ data
≠
null
⇒
equals(
ReadResource
(name,
WriteResource
(name, data)),
data)
Slide13Pex4Fun – Turning Pex Online
1,792,222
clicked 'Ask
Pex!'
http://pex4fun.com/default.aspx?language=CSharp&sample=_Template
Slide14
http://research.microsoft.com/pex/
Dynamic Symbolic Execution (DSE)
aka. Concolic Testing [Godefroid et al. 05][Sen et al. 05][Tillmann et al. 08]Instrument code to explore feasible paths
Behind the Scene of Pex4Fun
Slide15http://research.microsoft.com/pex/
Slide16void
CoverMe
(
int
[] a)
{ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
a.Length>0
a[0]==123…
T
F
T
F
F
a==null
T
Constraints to solve
a!=null
a!=null &&
a.Length
>0
a!=null &&
a.Length
>0 &&a[0]==123456890
Inputnull{}{0}{123…}
Execute&Monitor
Solve
Choose next path
Observed constraints
a==null
a!=null &&
!(
a.Length
>0)
a==null &&
a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890
Done: There is no path left.
Dynamic Symbolic Execution in
Pex
http://pex4fun.com/HowDoesPexWork
Slide17Pex is Part of Visual Studio 2015 Enterprise Edition!
As new feature of “IntelliTest”
https://www.visualstudio.com/news/vs2015-vs#Testing
Pex
:
30K
downloads after 20 months
Active user community: 1.4K forum posts during ~3 years
Slide18From
http://bbcode.codeplex.com/
NUnit Extension
– 13024 downloads
xUnit.net Extension
- 8589 downloads
Impact of
IntelliTest
Slide19Domain Matrix for Testing Complex Condition
19
Slide20Guide Pex to Generate Test Data
[PexMethod(TestEmissionFilter = PexTestEmissionFilter.All)] public void TestBoundaryValuesAndInputPartition(int x, int y) { //boundary values/partitions for x > 0 && x <= 10 && y >= 1 PexAssume.IsTrue((x > 0)); if (x == 1) { } else if (x > 0) { } PexAssume.IsTrue((x <= 10)); if (x == 10) { } else if (x <= 10) { } PexAssume.IsTrue((y >= 1)); if (y == 1) { } else if (y > 1) { } }
20
Details see http://taoxie.cs.illinois.edu/publications/icsm10-coverage.pdf
http://pex4fun.com/default.aspx?language=CSharp&sample=_Template
using System;
using
Microsoft.Pex.Framework
;
using
Microsoft.Pex.Framework.Settings
;
[
PexClass
]
public class Program {
//insert your PUT here
}
Slide21Parameterized Unit Tests Supported by Pex/Pex4Fun
using System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;[PexClass]public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool exist = s.member(i); s.insert(j); PexAssert.IsTrue(exist); } ….}
21
Slide22Interface for IntSet
Class IntSet { public IntSet() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … }}sort IntSet imports Int, Bool signatures new : -> IntSet insert : IntSet × Int -> IntSet member : IntSet × Int -> Bool remove : IntSet × Int -> IntSet
22
http://www.cs.unc.edu/~stotts/723/adt.html
Slide23(Buggy) Implementation for IntSet
Class IntSet { public IntSet() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … }}See the Set.cs that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/Set.cs Let’s copy it to http://pex4fun.com/default.aspx?language=CSharp&sample=_Template And Click “Ask Pex”
23
Slide24Parameterized Unit Tests Supported by Pex/Pex4Fun
using System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;[PexClass]public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool existOld = s.member(i); s.insert(j); bool exist = s.member(i); PexAssert.IsTrue(existOld == exist); } ….}
24
Pex4Fun supports only one
PexMethod
at a time;
you can write multiple
PexMethods
but comment out other lines of “[
PexMethod
]” except one
Slide25Axioms for IntSet
variables i, j : Int; s : IntSet Axioms:member(new(), i) = false member(insert(s, j), i) = if i = j then true else member(s, i)
25
http://www.cs.unc.edu/~stotts/723/adt.html
Is this complete?
How do we know?
Slide26Guidelines for Completeness
Classify methods:constructors: return IntSetinspectors: take IntSet as argument, returning some other value.Identify key constructors, capable of constructing all possible object statese.g., insert, new. Identify others as auxiliary, e.g., remove is a destructive constructorCompleteness requires (at least): every inspector/auxiliary constructor is defined by one equation for each key constructor.
26
Slide27Add More Axioms
remove(new(), i) = new() remove(insert(s, j), i) = if i = j then remove(s, i) else insert(remove(s, i), j)
27
Are we done yet?
The completeness criterion (an equation defining member and remove for each of the new and insert constructors) is satisfied.
Slide28Guidelines for Completeness
But does this really specify sets? Do the following properties hold?Order of insertion is irrelevant.insert(insert(s, i), j) = insert(insert(s, j), i)Multiple insertion is irrelevant.insert(insert(s, i), i) = insert(s, i)
28
Slide29Interface (Implementation) for UIntStack
Class UIntStack { public UIntStack() {…}; public void Push(int k) { … } public void Pop() { … } public int Top() { … } public bool IsEmpty() { … } public int MaxSize() { … } public bool IsMember(int k) { … } public bool Equals(UIntStack s) { … } public int GetNumberOfElements() { … } public bool IsFull() { … }}
29
See the
UIntStack.cs
that can be downloaded from
http://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs
Slide30
Exercise: Write Parameterized Unit Tests (PUTs)
Class UIntStack { public UIntStack() {…}; public void Push(int k) { … } public void Pop() { … } public int Top() { … } public bool IsEmpty() { … } public int MaxSize() { … } public bool IsMember(int k) { … } public bool Equals(UIntStack s) { … } public int GetNumberOfElements() { … } public bool IsFull() { … }}
30
Let’s copy it to http://pex4fun.com/default.aspx?language=CSharp&sample=_Template And Click “Ask Pex”
Reminder: you have to comment earlier written “[PexMethod]” before you try Pex on your current PUT (Pex4Fun can handle only one PUT at a time)
See the
UIntStack.cs
that can be downloaded from
http://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs
Slide31
One Sample PUT
Below is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method. [PexMethod] public void TestPush([PexAssumeUnderTest]UIntStack s, int i) { //UIntStack s = new UIntStack(); PexAssume.IsTrue(!s.IsMember(i)); int oldCount = s.GetNumberOfElements(); s.Push(i); PexAssert.IsTrue(s.Top() == i); PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1); PexAssert.IsFalse(s.IsEmpty()); }
31
Slide32Guideline of Writing PUT
Setup:
basic set up for invoking the method under test
Checkpoint: Run
Pex
to make sure that you don't miss any
Pex
assumptions (preconditions) for the PUT
Assert:
add assertions for asserting behavior of the method under test, involving
Adding
Pex
assertions
Adding
Pex
assumptions for helping assert
Adding method sequences for helping assert
Slide33Setup
Select your method under test
m
Put its method call in your PUT
Create a parameter for your PUT as the class under test
c
(annotated it with
[PexAssumeUnderTest]
)
Create other parameters for your PUT for parameters of
m
if any
Add Pex assumptions for preconditions for all these parameters of PUT if any
Slide34Setup - Example
[
PexMethod
]
public void
TestPush
([
PexAssumeUnderTest
]
UIntStack
s,
int
i) {
s.Push
(i);
}
You may write your factory method (see backup slides) to help
Pex
in test generation
If you get exceptions thrown
if indicating program faults, fix them
If indicating lack of PUT assumptions, add PUT assumptions
If indicating insufficient factory method assumptions or inappropriate scenarios, add PUT assumptions or improve factory method.
Slide35Assert
Think about how you can assert the behavior
Do you need to invoke other (observer) helper methods in your assertions (besides asserting return values)?
Do you need to add assumptions so that your assertions can be valid?
Do you need to add some method sequence before the method under test to set up desirable state and cache values to be used in the assertions?
Slide36Targets for Asserting
Return value of the method under test (MUT)
Argument object of MUT
Receiver object properties being modified by MUT (if public fields, directly assertable)
How to assert them?
Think about the intended behavior!
If you couldn't do so easily, follow the guidelines discussed next
Slide37Cached Public Property Value
A
property
value before
invoking MUT
may need to be cached
and later used
.
Pattern 2.1/2.2: Assume, Arrange, Act, Assert
[
PexMethod
]
void
AssumeActAssert
(
ArrayList
list, object item) {
PexAssume.IsNotNull
(list); // assume
var
count
=
list.Count
; // arrange
list.Add
(item); // act
Assert.IsTrue
(
list.Count
==
count
+ 1); // assert
}
Slide38Argument of MUT
Argument value of MUT may be used
Pattern 2.3:Constructor Test
[
PexMethod
]
void Constructor(
int
capacity) {
var
list = new
ArrayList
(
capacity
); // create
AssertInvariant
(list); // assert invariant
Assert.AreEqual
(
capacity
,
list.Capacity
); // assert
}
Slide39Reciever or Argument of Earlier Method
Receiver or argument value of a method before invoking MUTPattern 2.4/5:Roundtrip[PexMethod]void ToStringParseRoundtrip(int value) { // two-way roundtrip string s = value.ToString(); int parsed = int.Parse(s); // assert Assert.AreEqual(value, parsed);}
value s parsed
Slide40Observer Methods
Invoking observer methods on the modified object state
Pattern 2.6: State Relation
[
PexMethod
]
void
InsertContains
(string value) {
var
list = new List<string>();
list.Add
(value);
Assert.IsTrue
(
list.
Contains
(value));
}
Each modified object property should be read by at least one observer method.
Slide41Observer Methods cont.
Forcing observer methods to return specific values (e.g., true or false) can force you to add specific assumptions or scenarios
[
PexMethod
]
void
PushIsFull
(
[
PexAssumeUnderTest
]
UIntStack
s,
int
value) {
PexAssume.IsTrue
(
s.GetSize
() == (
s.GetMaxSize
()-1));
s.Push
(value);
Assert.IsTrue
(
s.
IsFull
());
}
Slide42Introduction to Software Testing (Ch 1)
© Ammann & Offutt
42
Parameterized Unit Tests in JUnit
Problem: Testing A Function With Similar Values
How To Avoid Test Code Bloat?
Simple Example: Adding Two Numbers
Adding a Given Pair of Numbers Is Just Like Adding Any Other Pair
You Really Only Want to Write One Test
Parameterized Unit Tests Call Constructor For Each Logical Set of Data Values
Same Tests Are Then Run On Each Set of Data Values
List of Data Values Identified with @Parameters Annotation
Slide43Introduction to Software Testing (Ch 1)
© Ammann & Offutt
43
Parameterized Unit Tests in JUnit
import
org.junit
.*;
import
org.junit.runner.RunWith
;
import
org.junit.runners.Parameterized
;
import
org.junit.runners.Parameterized.Parameters
;
import static
org.junit.Assert
.*;
import
java.util
.*;
@
RunWith
(
Parameterized.class
)
public class
ParamTest
{
public
int
sum, a, b;
public
ParamTest
(
int
sum,
int
a,
int
b) {
this.sum
= sum;
this.a
= a;
this.b
= b; }
@Parameters public static Collection<Object[]> parameters() {
return
Arrays.asList
(new Object [][] {{0, 0, 0}, {2, 1, 1}});
}
@Test public void
additionTest
() {
assertEquals
(sum,
a+b
); }
}
Slide44Introduction to Software Testing (Ch 1)
© Ammann & Offutt
44
JUnit Theories
These Are Unit Tests With Actual ParametersSo Far, We’ve Only Seen Parameterless Test MethodsContract Model: Assume, Act, AssertAssumptions (Preconditions) Limit Values AppropriatelyAction Performs Activity Under ScrutinyAssertions (Postconditions) Check Result
@Theory public void removeThenAddDoesNotChangeSet(
Set<String> set, String string) { // Parameters!
assumeTrue(set.contains(string)) ; // Assume
Set<String> copy = new HashSet<String>(set); // Act
copy.remove(string);
copy.add(string);
assertTrue (set.equals(copy)); // Assert //
// System.out.println(“Instantiated test: “ + set + “, “ + string);
}
Slide45Introduction to Software Testing (Ch 1)
© Ammann & Offutt
45
Question: Where Does Data Come From?
Answer:All Combinations of Values from @DataPoint Annotations Where Assume Clause is TrueFour (of Nine) Combinations in This Particular CaseNote: @DataPoint Format is an Array.
@DataPoints
public static String[] string = {"ant", "bat", "cat"};
@DataPoints
public static Set[] sets = {
new HashSet(Arrays.asList("ant", "bat")),
new HashSet(Arrays.asList(“bat", “cat", “dog“, “elk”)),
new HashSet(Arrays.asList(“Snap”, “Crackle”, “Pop"))
};
Slide46Introduction to Software Testing (Ch 1)
© Ammann & Offutt
46
JUnit Theories Need BoilerPlate
import org.junit.*;
import org.junit.runner.RunWith;
import static org.junit.Assert.*;
import static org.junit.Assume.*;
import org.junit.experimental.theories.DataPoint;
import org.junit.experimental.theories.DataPoints;
import org.junit.experimental.theories.Theories;
import org.junit.experimental.theories.Theory;
import java.util.*;
@RunWith(Theories.class)
public class SetTheoryTest {
… // See Earlier Slides
}
Slide47
Coding Duels
Pex
computes “semantic diff” in cloud
secret reference implementation vs.
code written in browser
You win when
Pex
finds no differences
secret
For more info, see our ICSE 2013 SEE paper:
http://taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf
Slide48
Behind the Scene of
Pex
for Fun
Secret Implementation
class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}
Player Implementation
class Player { public static int Puzzle(int x) { return x; }}
class Test {
public static void Driver(
int
x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}
behavior
Secret
Impl
==
Player
Impl
48
1,594,092
1,594,092
Slide49Code Hunt Programming Game
https://www.codehunt.com/
For more info, see our ICSE 2015 JSEET paper: http://taoxie.cs.illinois.edu/publications/icse15jseet-codehunt.pdf
Code Hunt has had over
450,000
users in a year
with around 1,000 users a day
Slide50Modeling in Science and Engineering
A model:Is an abstraction of the system from a particular perspectiveSupports investigation, construction and prediction Is not necessarily comprehensiveCan be expressed as a table, graphical diagram, formal notation,etc. Ubiquitous and Universal
50
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide51Software Modeling Styles and Notations
51
Focus of
Spec Explorer
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide52Model-Based Testing in a Nutshell
Model
Implementation
Expected Outputs (Test Oracle)
Inputs
(Test Sequences)
Control
Requirements
Generate
Observe
Feedback
Feedback
Author
Issue
Feedback
Test Cases
Verdict
Feedback
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide53Model-Based Testing with Spec Explorer
Explore
Analyze
Generate
Execute
C# Model
(or other
.Net
Language)
Model Graph
Test Suite
VSTT Result
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide54Walkthrough: Modeling and Testing a Web Service
Example: A Simple Web Shop
Web Shop Service
Web Shop Client
Billing
Shipping
AddToCart
Checkout
Bill
Ship
System Under Test
Web Shop Service
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide55Web Shop Implementation in Windows Communication Foundation (WCF)
[ServiceContract(CallbackContract = typeof(IWebShopCallback))] public interface IWebShop { [OperationContract] // Add item to stock of web shop void AddItem(string itemName, int price); [OperationContract] // Add item to shopping cart of user bool AddToCart(string userName, string itemName); [OperationContract] // Check out void Checkout(string userName); } public interface IWebShopCallback { [OperationContract] // Initiate billing, returns true on success bool Bill(string userName, int amount); [OperationContract(IsOneWay = true)] // Initiate shipping void Ship(string userName, string itemName); }
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide56Modeling Web Shop: Basic Approach
Import or declare action vocabulary in configuration scriptDefine state variables in C# model programDefine state update rules in C# model programDefine scenarios and generate state transition system
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide57config Actions{ action static void TestAdapter.AddItem(string itemName, int price); action static void TestAdapter.AddToCart(string userName, string itemName); action static void TestAdapter.Checkout(string userName); action event static void TestAdapter.Billed(string userName, int amount, bool success); action event static void TestAdapter.Shipped(string userName, string itemName);}
Modeling Web Shop: Action Vocabulary
Adapter for the System-Under-Test
Reference to test adapter method
Indicates that this action is an event raised by adapter
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide58Modeling WebShop: Define Model State in C#
enum ShopperStatus{ Shopping, CheckedOut, Billed }class ShopperInfo{ // Current status of this user internal ShopperStatus Status; // Items on cart so far internal SequenceContainer<string> Items;}// Foreach user shopping, contains info static MapContainer<string, ShopperInfo> shoppers;// Foreach item in the shop, contains pricestatic MapContainer<string, int> items;
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide59Modeling WebShop: Defining State Update Rules
[Action(“AddToCart(userName,itemName)”)]static void AddToCart(string userName, string itemName){ Contracts.Requires(items.ContainsKey(itemName)); if (!shoppers.ContainsKey(userName)) shoppers[userName] = new ShopperInfo(); Contracts.Requires(InStatus(userName, ShopperStatus.Shopping)); shoppers[userName].Items.Add(itemName);} static bool InStatus(string userName, ShopperStatus status){ return shoppers.ContainsKey(userName) && shoppers[userName].Status == status;}
Binding rule to action in action vocabulary
Condition under which rule is enabled. Fires a state update
State update
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide60Modeling WebShop: Define Scenarios
machine MultipleItemScenario() : Config{ AddItem("Bread", 2); AddItem("Cheese", 5); AddItem("Wine", 10) AddToCart("Paul","Wine"); AddToCart("Paul", "Cheese"); (Checkout | Billed | Shipped)*}machine MultipleItemSlice() : Config { MultipleItemScenario || construct model program from Config}
Combining scenario with C# model program
Or’ing actions in repetition (Note parameters omitted)
Sequence of actions
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide61Modeling WebShop: Exploration of the Scenario
Parameters and control flow undetermined
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide62Modeling WebShop: Exploration of the Model Slice
Choice point for system-under-test
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide63Modeling WebShop: Execution of generated test code
Model expects bill of $2,
implementation bills $4
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide64“BlueLine” Technical Document Testing Project
222 open protocol specifications tested22,847 pages66,962 person days (250+ years)Vendor test teamsHyderabad: 250Beijing: 100Tool dev center in Beijing (Spec Explorer and other)~10,000 document bugs filed
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide65Comparison of Model-Based/Traditional Testing
Vendor teams had (guided) choice which approach to useOut of 222 protocols approx. ½ modeled, ½ traditional Modeling resulted in 42% productivity gain in average (some teams reached more than 300%)Overall, modeling saved 50 person-years (12,547 days)No special education background was requiredTeams of junior engineersMostly vendors, fresh out of college
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Slide66Summary
https://msdn.microsoft.com/en-us/library/ee620411.aspx
https://visualstudiogallery.msdn.microsoft.com/271d0904-f178-4ce9-956b-d9bfa4902745
A Model-Based Testing tool from Microsoft
Parameterized Unit Test
http://research.microsoft.com/pex/
IntelliTest
http://nmodel.codeplex.com/
NModel
Slide67Q & A
67
Slide68Code Hunt Programming Game
Slide6969
Slide70Test-Driven Development (TDD)
Basic Idea:
Write tests before code
Refine code with new tests
In more detail, TDD is a cycle of steps:
Add a test,
Run it and watch it fail,
Change the code as little as possible such that the test should pass,
Run the test again and see it succeed,
Refactor the code if needed.
Slide71Note: TDD and specifications
TDD encourages writing specifications before code
Exemplary specification
Later, we will generalize TDD to
Parameterized TDD
Axiomatic specifications
Slide72Parameterized Test-Driven Development
Write/refine Contract as PUT
Write/refine Code of Implementation
Fix-it (with Pex),Debug with generated tests
Use Generated Tests for Regression
Run
Pex
Bug in PUT
Bug in Code
failures
no failures
Slide73Code Hunt Programming Game
Slide74Code Hunt Programming Game
Slide75Code Hunt Programming Game
Slide76Code Hunt Programming Game
Slide77Code Hunt Programming Game
Slide78Code Hunt Programming Game
Slide79Code Hunt Programming Game
Slide80Code Hunt Programming Game
Slide81Code Hunt Programming Game
Slide82Code Hunt Programming Game
Slide83Code Hunt Programming Game
Slide84It’s a game!
iterative gameplayadaptivepersonalizedno cheatingclear winning criterion
secret
code
test cases
Slide85(Buggy) Implementation for IntSet
Class IntSet { public IntSet() {…}; public void insert(int e) { … } public Bool member(int e) { … } public void remove(int e) { … }}See the Set.cs that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/Set.cs Let’s copy it to http://pex4fun.com/default.aspx?language=CSharp&sample=_Template And Click “Ask Pex”
85
Slide86Recall: Parameterized Unit Tests Supported by Pex/Pex4Fun
using System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;[PexClass]public class Set { [PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool existOld = s.member(i); s.insert(j); bool exist = s.member(i); PexAssert.IsTrue(existOld == exist); } ….}
86
Slide87Force Pex/Pex4Fun to Display All Explored Test Inputs/Paths
using System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;[PexClass]public class Set { [PexMethod(TestEmissionFilter=PexTestEmissionFilter.All)] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j); bool exist = s.member(i); s.insert(j); PexAssert.IsTrue(exist); } ….}
87
Slide88Factory Method: Help Pex Generate Desirable Object States
In class, we show the factory method as below automatically synthesized by Pex after a user clicks “1 Object Creation” issue and then click “Accept/Edit Factory Method”. But it is not good enough to generate various types of object states. [PexFactoryMethod(typeof(UIntStack))] public static UIntStack Create(int k_i) { UIntStack uIntStack = new UIntStack(); uIntStack.Push(k_i); return uIntStack; // TODO: Edit factory method of UIntStack // This method should be able to configure the object in all possible ways. // Add as many parameters as needed, // and assign their values to each field by using the API. }
88
Slide89Factory Method: Help Pex Generate Desirable Object States
Below is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method. [PexFactoryMethod(typeof(UIntStack))] public static UIntStack CreateVariedSizeAnyElemsStack(int[] elems) { PexAssume.IsNotNull(elems); UIntStack s = new UIntStack(); PexAssume.IsTrue(elems.Length <= (s.MaxSize() + 1)); for (int i = 0; i < elems.Length; i++) s.Push(elems[i]); return s; }
89
Slide90Pex4Fun Not Supporting Factory Method - Workaround
If you try PUTs on Pex4Fun, which doesn’t support factory method, you can “embed” the factory method like the highlighted code portion below [PexMethod] public void TestPush(int[] elems, int i) { PexAssume.IsNotNull(elems); UIntStack s = new UIntStack(); PexAssume.IsTrue(elems.Length <= (s.MaxSize() + 1)); for (int i = 0; i < elems.Length; i++) s.Push(elems[i]); //UIntStack s = new UIntStack(); PexAssume.IsTrue(!s.IsMember(i)); int oldCount = s.GetNumberOfElements(); s.Push(i); PexAssert.IsTrue(s.Top() == i); PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1); PexAssert.IsFalse(s.IsEmpty()); }
90
Slide91Understanding the Web Shop
Which of the following traces is valid?(assuming an initialization of the shop with AddItem(“wine”,10) and AddItem(“cheese”, 5))1AddToCart(“Peter”, “wine”) Checkout(“Peter”) Bill(“Peter”,10)/true Ship(“Peter”,”wine”)2AddToCart(“Peter”, “wine”) Checkout(“Peter”) Bill(“Peter”,10)/false Ship(“Peter”,”wine”)3AddToCart(“Peter”, “wine”) Checkout(“Peter”) Bill(“Peter”,12)/true Ship(“Peter”,”wine”)4AddToCart(“Peter”, “wine”) AddToCart(“Peter”,”cheese”) Checkout(“Peter”) Bill(“Peter”,15)/true Ship(“Peter”,”wine”) Ship(“Peter”,”cheese”)5AddToCart(“Peter”, “wine”) AddToCart(“Peter”,”cheese”) Checkout(“Peter”) Bill(“Peter”,15)/true Ship(“Peter”,”cheese”) Ship(“Peter”,”wine”)
Slides from
https://channel9.msdn.com/Events/TechEd/Europe/2009/DEV316
©
W.
Grieskamp
Requirement Specifications based Test Automation
Download Presentation - The PPT/PDF document "Requirement Specifications based Test Au..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.