Achievements and Challenges Tao Xie North Carolina State University In collaboration with Nikolai Tillmann Peli de Halleux Wolfram Schulte Microsoft Research and students NCSU ASE ID: 360170
Download Presentation The PPT/PDF document "Automated Developer Testing:" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Automated Developer Testing:
Achievements and Challenges
Tao
XieNorth Carolina State UniversityIn collaboration with Nikolai Tillmann, Peli de Halleux, Wolfram Schulte @Microsoft Research and students @NCSU ASESlide2
Why Automate Testing?Software testing is importantSoftware errors cost the U.S. economy about $59.5 billion each year (0.6% of the GDP)
[NIST 02]Improving testing infrastructure could save 1/3 cost [NIST 02]Software testing is
costlyAccount for even half the total cost of software development
[Beizer 90]Automated testing reduces manual testing effortTest execution: JUnit, NUnit, xUnit, etc.Test generation: Pex, AgitarOne, Parasoft Jtest, etc.Test-behavior checking: Pex, AgitarOne
, Parasoft Jtest,
etc.Slide3
Automation in Developer TestingDeveloper testinghttp://www.developertesting.com/
Kent Beck’s 2004 talk on “Future of Developer Testing”http://www.itconversations.com/shows/detail301.html
This talk focuses on tool automation in
developer testing (e.g., unit testing)Not system testing etc. conducted by testersSlide4
Software Testing Setup
=?
Outputs
Expected Outputs
Program
+
Test
inputs
Test OraclesSlide5
Software Testing Problems
=
?
Outputs
Expected Outputs
Program
+
Test
inputs
Test Oracles
Test Generation
Generating high-quality test inputs (e.g., achieving high code coverage)Slide6
Software Testing Problems
=
?
Outputs
Expected Outputs
Program
+
Test
inputs
Test Oracles
Test Generation
Generating high-quality test inputs (e.g., achieving high code coverage)
Test Oracles
Specifying high-quality test oracles (e.g., guarding against various faults)Slide7
The Recipe of Unit Testing var
list = new List(); list.Add(item);
var count =
list.Count; Assert.AreEqual(1, count);}Three essential ingredients:DataMethod SequenceAssertionsvoid Add() {
int item = 3;Slide8
The (problem with) DataWhich value matters?Bad choices cause incomplete test suites.Hard-coded values get stale when product code changes.Why pick a value if it doesn’t matter?
list.Add(3); Slide9
Parameterized Unit Testingvoid Add(List list
, int item) {
var count =
list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}Parameterized Unit Test = Unit Test with ParametersSeparation of concernsData is generated by a toolDeveloper can focus on functional specification[Tillmann&Schulte
ESEC/FSE 05]Slide10
Parameterized Unit Tests areAlgebraic Specifications
A Parameterized Unit Test can be read as a universally quantified, conditional axiom.
void
ReadWrite(string name, string data) { Assume.IsTrue(name != null && data != null); Write(name, data); var readData = Read(name); Assert.AreEqual(data, readData);}
string name, string data:
name
≠
null ⋀ data
≠
null
⇒
equals(
ReadResource
(
name,WriteResource
(
name,data
)),
data)Slide11
Parameterized Unit Testingis going mainstreamParameterized Unit Tests (PUTs) commonly supported by various test frameworks.NET: Supported by .NET test frameworks
http://www.mbunit.com/http://www.nunit.org/…Java: Supported by JUnit
4.Xhttp://www.junit.org/
Generating test inputs for PUTs supported by tools.NET: Supported by Microsoft Research Pexhttp://research.microsoft.com/Pex/Java: Supported by Agitar AgitarOnehttp://www.agitar.com/Slide12
Test GenerationHumanExpensive, incomplete, …Brute ForcePairwise, predefined data, etc…Random:
Cheap, Fast“It passed a thousand tests” feelingDynamic Symbolic Execution: Pex, CUTE,EXEAutomated white-boxNot random – Constraint SolvingSlide13
Dynamic Symbolic Execution
Code to generate inputs for:
Constraints to solve
a!=null a!=null &&a.Length>0 a!=null &&a.Length>0 &&a[0]==1234567890void CoverMe
(int[] a)
{ if (a == null) return;
if (a.Length
> 0)
if (a[0] ==
1234567890
)
throw new Exception("bug");
}
Observed constraints
a==null
a!=null &&
!(
a.Length
>0)
a!=null &&
a.Length
>0 &&
a[0]!=
1234567890
a!=null &&
a.Length
>0 &&
a[0]==
1234567890
Data
null
{}
{0}
{123…}
a==null
a.Length
>0
a[0]==123…
T
T
F
T
F
F
Execute&Monitor
Solve
Choose next path
Done: There is no path left.
Negated conditionSlide14
Loops Fitnex [Xie et al. DSN 09]Generic API functions e.g., RegEx matching IsMatch
(s1,regex1) Reggae [Li et al. ASE 09-sp]Method sequences MSeqGen [Thummalapenta
et al. ESEC/FSE 09]Environments e.g., file systems, network, db, …
Parameterized Mock Objects [Marri AST 09]OpportunitiesRegression testing [Taneja et al. ICSE 09-nier]Developer guidance (cooperative developer testing)Challenges of DSESlide15
Loops Fitnex [Xie et al. DSN 09]Generic API functions e.g.,
RegEx matching IsMatch(s1,regex1) Reggae [Li et al. ASE 09-sp]
Method sequences
MSeqGen [Thummalapenta et al. ESEC/FSE 09]Environments e.g., file systems, network, db, …Parameterized Mock Objects [Marri AST 09]ApplicationsTest network app at Army division@Fort Hood, TexasTest DB app of hand-held medical assistant device at FDANCSU Industry Tech TransferSlide16
Pex on MSDN DevLabsIncubation Project for Visual Studio
Download counts
(20 months)
(Feb. 2008 - Oct. 2009 ) Academic: 17,366 Devlabs: 13,022 Total: 30,388Slide17
Loops Fitnex [Xie et al. DSN 09]Generic API functions e.g.,
RegEx matching IsMatch(s1,regex1) Reggae [Li et al. ASE 09-sp]
Method sequences
MSeqGen [Thummalapenta et al. ESEC/FSE 09]Environments e.g., file systems, network, db, …Parameterized Mock Objects [Marri AST 09]ApplicationsTest network app at Army division@Fort Hood, TexasTest DB app of hand-held medical assistant device at FDANCSU Industry Tech TransferSlide18
There are decision procedures for individual path conditions, but…Number of potential paths grows exponentially with number of branchesReachable code not known initiallyWithout guidance, same loop might be unfolded foreverFitnex search strategy
[Xie et al. DSN 09]
Explosion of Search SpaceSlide19
public bool TestLoop
(int x, int
[] y) {
if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false;}
Path condition:!(x == 90)
↓
New path condition:(x == 90)
↓
New test input:
TestLoop
(90, {0
})
Test input:
TestLoop
(0, {0})
DSE ExampleSlide20
Path condition:(x == 90) && !(y[0]
== 15)
↓
New path condition:(x == 90) && (y[0] == 15) ↓New test input:TestLoop(90, {15})
Test input:
TestLoop
(90
,
{0})
public
bool
TestLoop
(
int
x,
int
[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[
i
] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
DSE ExampleSlide21
public bool TestLoop
(int x, int
[] y) {
if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; }
return false;
}
Test input:
TestLoop
(90
, {15})
Path condition:
(x == 90) && (y[0]
== 15
)
&&
!(x+1 == 110)
↓
New path condition:
(x == 90) && (y[0]
== 15
)
&&
(x+1 == 110)
↓
New test input:
No solution
!?
Challenge in DSESlide22
public bool TestLoop
(int x, int
[] y) {
if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; } return false;}
Path condition:
(x == 90) && (y[0]
== 15)
&&
(0 <
y.Length
)
&&
!(1 <
y.Length
)
&& !(x+1 == 110)
↓
New path condition:
(x == 90) && (y[0]
== 15
)
&&
(0 <
y.Length
)
&&
(1 <
y.Length
)
Expand array size
Test input:
TestLoop
(90
, {15})
A Closer LookSlide23
We can have infinite paths
! (both length and number)Manual analysis
need at least 20 loop iterations to cover the target branchExploring all paths up to 20 loop iterations is practically infeasible: 220 pathsTest input:TestLoop(90, {15})
public
bool TestLoop
(int
x,
int
[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[
i
] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
A Closer LookSlide24
Fitnex: Fitness-Guided Explorationpublic
bool TestLoop(
int x,
int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; }
return false;
}
Test input:
TestLoop
(90, {15, 15})
Our
solution:
Prefer to flip
nodes on
the most
promising
path
Prefer to flip the most
promising
nodes
on
path
Use
fitness
function
as a proxy for
promising
Key observations: with respect to the coverage target,
not all paths are equally promising for flipping nodes
not all nodes are equally
promising to flipSlide25
Fitness FunctionFF computes fitness value (distance between the current state and the goal state)Search tries to minimize fitness value
[Tracey et al. 98, Liu at al. 05,
…]Slide26
Fitness Function for (x == 110)public
bool TestLoop(int
x, int
[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[i] == 15) x++; if (x == 110) return true; }
return false;}
Fitness function: |110 – x |Slide27
Compute Fitness Values for Paths
(90, {0}) 20
(90,
{15}) 19(90, {15, 0}) 19(90, {15, 15}) 18(90, {15, 15, 0}) 18(90, {15, 15, 15}) 17 (90, {15, 15, 15, 0}) 17(90, {15, 15, 15, 15}) 16(90, {15, 15, 15, 15, 0}) 16(90,
{15, 15, 15, 15, 15}) 15…
Fitness
Value
(x, y)
Give preference to flip
a node in paths
with better fitness
values.
We still need to address which
node
to flip on paths …
public
bool
TestLoop
(
int
x,
int
[] y) {
if (x == 90) {
for (int i = 0; i < y.Length; i++)
if (y[
i
] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
Fitness function: |110 – x |Slide28
FitnessValue
Compute Fitness Gains for Branches
public
bool TestLoop(int x, int[] y) { if (x == 90) { for (int i = 0; i < y.Length; i++) if (y[
i] == 15)
x++;
if (x == 110)
return true;
}
return false;
}
(90,
{0}
)
20
(90,
{15}
)
flip b4
19
(90,
{15, 0}
)
flip b2 19
(90,
{15, 15}
)
flip b4
18
(90,
{15, 15, 0}
)
flip b2
18
(90,
{15, 15, 15}
)
flip b4
17
(90,
{15, 15, 15, 0}
)
flip b2
17
(90,
{15, 15, 15, 15}
)
flip b4
16
(90,
{15, 15, 15, 15, 0}
)
flip b2
16
(90,
{15, 15, 15, 15, 15}
)
flip b4
15
…
(x, y)
Fitness function: |110 – x |
Branch b1:
i
<
y.Length
Branch b2:
i
>=
y.Length
Branch b3: y[
i
] == 15
Branch b4: y[
i
] != 15
Flipping branch node of b4
(b3) gives us average 1 (-1) fitness gain (loss)
Flipping branch node of b2
(b1) gives us average 0
(0) fitness
gain (loss)Slide29
Let p be an already explored path, and n a node on that path, with explored outgoing branch b.After (successfully) flipping n, we get path p’ that goes to node n, and then continues with a different branch b’.Define fitness gains as follows, where F(.) is the fitness value of a path.
Set FGain(b) := F(p) – F(p’)
Set FGain
(b’) := F(p’) – F(p)Compute the average fitness gain for each program branch over timeCompute Fitness Gains for BranchesSlide30
Implementation in PexPex: Automated White-Box Test Generation tool for .NET, based on Dynamic Symbolic Execution
Pex maintains global search frontierAll discovered branch nodes are added to frontier
Frontier may choose next branch node to flip
Fully explored branch nodes are removed from frontierPex has a default search frontierTries to create diversity across different coverage criteria, e.g. statement coverage, branch coverage, stack traces, etc.Customizable: Other frontiers can be combined in a fair round-robin schemeSlide31
Implementation in PexWe implemented a new search frontier “
Fitnex”:Nodes to flip are prioritized by their composite fitness
value:F(
pn) – FGain(bn),wherepn is path of node nbn is explored outgoing branch of n Fitnex always picks node with lowest composite fitness value to flip.To avoid local optimal or biases, the fitness-guided strategy is combined with Pex’s search strategiesSlide32
A collection of micro-benchmark programs routinely used by the Pex developers to evaluate Pex’s performance, extracted from real, complex C# programs
Ranging from string matching like
if (
value.StartsWith("Hello") && value.EndsWith("World!") && value.Contains(" "))
{ … }to
a small parser for a Pascal-like language where the target is to create a legal program.
Evaluation SubjectsSlide33
Pex with the Fitnex strategyPex without the
Fitnex strategyPex’s previous default strategy
Random
a strategy where branch nodes to flip are chosen randomly in the already explored execution treeIterative Deepeninga strategy where breadth-first search is performed over the execution treeSearch Strategies Under ComparisonSlide34
#runs/iterations required to cover the target
Pex
w/o
Fitnex: avg. improvement of factor 1.9 over RandomPex w/ Fitnex: avg. improvement of factor 5.2 over RandomEvaluation ResultsSlide35
Object CreationPex normally uses public methods to configure non-public object fieldsHeuristics built-in to deal with common typesUser can help if needed
void (
Foo
foo) { if (foo.Value == 123) throw …[PexFactoryMethod]Foo Create(Bar bar) { return new Foo(bar);}Slide36
QuickGraph ExampleA graph example from QuickGraph
library
36
36
interface
IGraph
{
/* Adds given vertex to the graph */
void
AddVertex
(
IVertex
v);
/* Creates a new vertex and adds it to the graph */
IVertex
AddVertex
();
/* Adds an edge to the graph. Both vertices should
already exist in the graph */
IEdge
AddEdge
(
IVertex
v1,
Ivertex
v2);
}Slide37
Method Under TestDesired object state for reaching targets 1 and 2: graph object should contain vertices and edges
Class
SortAlgorithm
{ IGraph graph; public SortAlgorithm
(IGraph
graph
) {
this.
graph
= graph;
}
public void
Compute
(
IVertex
s) {
foreach
(
IVertex
u in
graph.Vertices
)
{
//
Target 1
}
foreach
(
IEdge
e in
graph
.
Edges
)
{
//Target 2
}
}
}
m
ethod
sequenceSlide38
Method Under Test
VertexAndEdgeProvider v0 = new
VertexAndEdgeProvider
();Boolean v1 = false;BidirectionalGraph v2 = new BidirectionalGraph((IVertexAndEdgeProvider)v0, v1);IVertex v3 = v2.AddVertex();IVertex v4 = v0.ProvideVertex();IEdge v15 = v2.AddEdge(v3, v4);
Achieved 31.82%
(7 of 22)
branch coverage
Reason for low coverage: Not able to generate graph with vertices and edges
Applying
Randoop
, a random testing approach
that constructs test inputs by randomly selecting method calls
Example sequence generated by
Randoop
v4
not in the graph, so edge cannot be added
to graph.Slide39
New MSeqGen Approach
VertexAndEdgeProvider
v0;
bool bVal;IGraph ag
= new AdjacencyGraph
(v0,
bVal);
IVertex
source =
ag.AddVertex
();
IVertex
target =
ag.AddVertex
();
IVertex
vertex3 =
ag.AdVertex
();
IEdge
edg1 =
ag.AddEdge
(source, target);
IEdge
edg2 =
ag.AddEdge
(target,
vertex3);
IEdge
edg3 =
ag.AddEdge
(source, vertex3);
Use mined sequences to assist
Randoop
and
Pex
Both
Randoop
and
Pex
achieved
86.40% (19 of 22)
branch coverage with assistance from
MSeqGen
Mine sequences from existing code bases
Reuse mined sequences for achieving desired object states
A Mined sequence from an existing codebase
Graph object
includes both vertices and edgesSlide40
Challenges Addressed by MSeqGen
Existing codebases are often large and complete analysis is expensive Search and analyze only relevant portions
Concrete values in mined sequences may be different from desired values
Replace concrete values with symbolic values and use dynamic symbolic executionExtracted sequences individually may not be sufficient to achieve desired object states Combine extracted sequences to generate new sequencesSlide41
MSeqGen: Code Searching
Problem: Existing code bases are often large and complete analysis is expensive
Solution:
Use keyword search for identifying relevant method bodies using target classesAnalyze only those relevant method bodies
Target classes:
System.Collections.Hashtable
QuickGraph.Algorithms.TSAlgorithm
Keywords:
Hashtable
,
TSAlgorithm
Shortnames
of target classes are used as keywordsSlide42
MSeqGen: Sequence Generalization
Problem: Concrete values in mined sequences are different from desired values to achieve target states
Solution: Generalize sequences by replacing concrete values with symbolic values
Class A {
int
f1 { set; get; }
int
f2 { set; get; }
void
CoverMe
()
{
if (f1 != 10) return;
if (f2 > 25)
throw new Exception(“bug”);
}
}
Method Under Test
A
obj
= new A();
obj.setF1(14);
obj.setF2(-10);
obj.CoverMe
();
Mined Sequence for A
Sequence cannot help in exposing bug since desired values are
f1=10 and f2>25Slide43
MSeqGen: Sequence Generalization
Replace concrete values 14 and -10 with symbolic values X1 and X2
A obj = new A();obj.setF1(14);
obj.setF2(-10);obj.CoverMe();
Mined Sequence for A
int
x1 = *, x2 = *;
A
obj
= new A();
obj.setF1(x1);
obj.setF2(x2);
obj.CoverMe
();
Generalized Sequence for A
Use DSE for generating desired values for X1 and X2
DSE explores
CoverMe
method and generates desired values (X1 = 10 and X2 = 35)
Slide44
Improvement of State-of-the-Art
44
44
Randoop
Without assistance from
MSeqGen
: achieved
32%
branch coverage
achieved
86%
branch coverage
In evaluation, help
Randoop
achieve 8.7% (maximum 20%) higher branch coverage
Pex
Without assistance from
MSeqGen
: achieved
45%
branch coverage
achieved
86%
branch coverage
In evaluation, help
Pex
achieve 17.4% (maximum 22.5%) higher branch coverageSlide45
Test OraclesWrite assertions and Pex will try to break themWithout assertions, Pex can only find violations of runtime contracts causing NullReferenceException
, IndexOutOfRangeException, etc.Assertions leveraged in product and test codePex can leverage Code ContractsSlide46
Summary:Automated Developer Testing
=
?
Outputs
Expected Outputs
Program
+
Test
inputs
Test Oracles
Test Generation
Test inputs for PUT generated by
tools
(e.g.,
Pex
)
Fitnex
: guided exploration of paths [DSN 09]
MSeqGen
: exploiting real-usage sequences [ESEC/FSE 09]
Test Oracles
Assertions in PUT specified by
developers
Division of LaborsSlide47
Thank you
http://research.microsoft.com/pex
http://pexase.codeplex.com/
https://sites.google.com/site/asergrp/Slide48
Code Contractshttp://research.microsoft.com/en-us/projects/contracts/Library to state preconditions, postconditions, invariantsSupported by two tools:
Static CheckerRewriter: turns Code Contracts into runtime checksPex analyses the runtime checksContracts act as Test OraclePex may find counter examples for contracts
Missing Contracts may be suggestedSlide49
Example: ArrayList
Class invariant specification:public class ArrayList {
private Object[] _items;
private int _size; ... [ContractInvariantMethod] // attribute comes with Contracts protected void Invariant() { Contract.Invariant(this._items != null); Contract.Invariant(this._size >= 0); Contract.Invariant(this._items.Length >=
this._size); }Slide50
ParameterizedModelsSlide51
Unit test: while it is debatable what a ‘unit’ is, a ‘unit’ should be small.Integration test: exercises large portions of a system.Observation: Integration tests are often “sold” as unit testsWhite-box test generation does not scale well to integration test scenarios.Possible solution: Introduce abstraction layers, and mock components not under test
Unit Testing vs.
Integration TestingSlide52
AppendFormat
(null, “{0} {1}!”, “Hello”,
“World”); “Hello World!” .Net Implementation:public StringBuilder AppendFormat( IFormatProvider provider,
ch
ar[] chars, param
s object[] args)
{
if (chars == null || args == null)
throw new ArgumentNullException(
…);
int pos = 0;
int len = chars.Length;
char ch = '\x0';
ICustomFormatter cf = null;
if (provider != null)
cf = (
ICustomFormatter)provider.GetFormat(typeof(ICustomFormatter
));
…
Example
Testing with
I
nterfacesSlide53
Introduce a mock class which implements the interface.
Write assertions over expected inputs, provide concrete outputspublic class MFormatProvider : IFormatProvider
{
public object GetFormat(Type formatType) { Assert.IsTrue(formatType != null); return new MCustomFormatter(); }}
Problems:
Costly to write detailed behavior by exampleHow many and which mock objects do we need to write?
Stubs / Mock O
bjectsSlide54
Parameterized M
ock Objects - 1
Introduce a mock
class which implements the interface. Let an oracle provide the behavior of the mock methods.public class MFormatProvider : IFormatProvider { public object GetFormat(Type formatType) { … object o = call.ChooseResult<object>();
return o;
}
}
Result: Relevant result values can be generated by white-box test input generation tool, just as other test inputs can be generated!
54Slide55
Chosen values can be shaped by assumptions
public class MFormatProvider : IFormatProvider {
public
object GetFormat(Type formatType) { … object o = call.ChooseResult<object>(); PexAssume.IsTrue(o is ICustomFormatter); return o; }}
(Note: Assertions and assumptions are “reversed” when compared to parameterized unit tests.)
Parameterized
M
ock
Objects - 2
55Slide56
Models from ChoicesChoices to build parameterized models
class PFileSystem :
IFileSystem {
// cached choices PexChosenIndexedValue<string,string> files; string ReadFile(string name) { var content = this.files[name]; if (content == null) throw new FileNotFoundException(); return content; }}Slide57
57
57
Subjects:
QuickGraph
Facebook
Research Questions:
RQ1: Can our approach assist
Randoop
(random testing tool) in achieving higher code
coverages
?
RQ2: Can our approach assist
Pex
(DSE-based testing tool) in achieving higher code
coverages
?
MSeqGen
EvaluationSlide58
58
RQ1: Assisting
RandoopSlide59
59
RQ2: Assisting
Pex
Legend:
#c: number of classes
P: branch coverage achieved by
Pex
P + M: branch coverage achieved by
Pex
and
MSeqGenSlide60
Assumptions and Assertions
void PexAssume.IsTrue(bool c) { if (!c)
throw new AssumptionViolationException();}void PexAssert.IsTrue(bool c) { if (!c) throw new AssertionViolationException();}Assumptions and assertions induce branchesExecutions which cause assumption violations are ignored, not reported as errors or test casesSlide61
How to test this code?
(Actual code from .NET base class libraries)
Motivation: Unit Testing Hell
ResourceReader61Slide62
Motivation: Unit Testing Hell
ResourceReaderSlide63
ResourceReader
PUT
[
PexClass, TestClass][PexAllowedException(typeof(ArgumentNullException))][PexAllowedException(typeof(ArgumentException))][PexAllowedException(typeof(FormatException))][PexAllowedException(
typeof(BadImageFormatException
))][PexAllowedException
(typeof(IOException
))]
[
PexAllowedException
(
typeof
(
NotSupportedException
))]
public partial class
ResourceReaderTest
{
[
PexMethod
]
public unsafe void
ReadEntries
(byte[] data) {
PexAssume.IsTrue
(data != null);
fixed (byte* p = data) using (var stream = new UnmanagedMemoryStream
(p, data.Length)) {
var
reader = new
ResourceReader
(stream);
foreach
(
var
entry in reader) { /* reading entries */ }
}
}
}Slide64
Creating Complex ObjectsExploration of constructor/mutator method sequencesTesting with class invariantsSlide65
Write class invariant as boolean-valued parameterless methodRefers to private fieldsMust be placed in implementation codeExploration of valid states by setting public/private fields
May include states that are not reachable
Object Creation:Class InvariantsSlide66
Example: ArrayList
Class invariant specification:public class ArrayList {
private Object[] _items;
private int _size; ... [ContractInvariantMethod] // attribute comes with Contracts protected void Invariant() { Contract.Invariant(this._items != null); Contract.Invariant(this._size >= 0); Contract.Invariant(this._items.Length >=
this._size); }Slide67
Example: ArrayList
PUT:[PexMethod]
public void ArrayListTest
(ArrayList al, object o){ int len = al.Count; al.Add(o); PexAssert.IsTrue(al[len] == o);}Slide68
Example: ArrayList
Generated Test:[TestMethod]
public void Add01() {
object[] os = new object[0]; // create raw instance ArrayList arrayList = PexInvariant.CreateInstance<ArrayList>(); // set private field via reflection PexInvariant.SetField<object[]>(arrayList, "_items", os); PexInvariant.SetField<
int>(arrayList, "_size", 0);
// invoke invariant method via reflection
PexInvariant.CheckInvariant(arrayList
);
// call to PUT
ArrayListTest
(
arrayList
, null);
}