Testing Theory and Practice Tao Xie University of Illinois at UrbanaChampaign http taoxiecsillinoiseducoursestesting Work described in the slides was done in collaboration with the ID: 535969
Download Presentation The PPT/PDF document "Parameterized Unit" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Parameterized Unit Testing:Theory and Practice
Tao XieUniversity of Illinois at Urbana-Champaignhttp://taoxie.cs.illinois.edu/courses/testing/
Work
described in
the slides was
done in collaboration with the
Pex
team (Nikolai Tillmann, Peli de Halleux,
Pratap
Lakshman
, et
al.) @Microsoft Research, students @Illinois ASE, and other collaboratorsSlide2
©
Ammann
& Offutt
Faults, Errors & Failures
Fault
: A static defect in the software (i.e., defect, bug)
Infected State
:
An incorrect internal state that is the manifestation of some fault (often also referred to as
error
)
Software Failure
: External, incorrect behavior with respect to the requirements or other description of the expected behaviorSlide3
Mistake, Fault, Error, Failure
Programmer makes a
mistake
.
Fault
(
defect, bug
) appears in the program.
Fault remains undetected during
testing (running
test inputs
).
The program
fails
(based on test oracles) during execution i.e. it behaves unexpectedly.
Error
: difference between computed, observed, or measured value or condition
and true, specified, or theoretically correct value or condition
What does
Bug
mean in “
Bug Report
”?Slide4
What is fault, error, failure?
Doubling the balance and then plus 10 int calAmount
() {
int ret = balance * 3; ret = ret + 10;
return ret;
}
1-
4
void
testCalAmount() { Account a = new Account(); Account.setBalance(0
);
int amount =
Account.calAmount();
assertTrue(amount == 10);} Where is test input? Where is test oracle?Slide5
What is fault, error, failure?
Doubling the balance and then plus 10 int calAmount
() {
int ret = balance * 3; ret = ret + 10;
return ret;
}
1-
5
void
testCalAmount() { Account a = new Account(); Account.setBalance(1
);
int
amount = Account.calAmount
(); assertTrue(amount == 12);} Where is test input? Where is test oracle?Slide6
What is fault, error, failure?
Should not allow withdrawal when there is a balance of 100 or less boolean doWithdraw
(
int amount) { if (Balance<100)
return
false;
else
return
wthDraw
(amount); }1-6void
testWithDraw() {
Account a = new Account();
Account.setBalance(
100); boolean success = Account.doWithdraw(10); assertTrue(success == false);
}
Slide7
Who should test?Developer? Separate “quality assurance” group?
Programmer? User? Someone with a degree in “testing”?7Slide8
Types of Test Activities
Testing can be broken up into four general types of activitiesTest DesignTest Automation
Test Execution
Test EvaluationEach type of activity requires different skills, background knowledge, education and training
© Ammann & OffuttSlide9
Test Design
This is the most technical job in software testingRequires knowledge of :Discrete mathProgrammingTestingRequires much of a traditional CS degree
This is
intellectually stimulating, rewarding, and challengingTest design is analogous to software architecture on the development sideUsing people who are not qualified to design tests is a sure way to get ineffective tests
Design test values to satisfy coverage criteria or other engineering goal
©
Ammann
& OffuttSlide10
Test Automation
This is slightly less technicalRequires knowledge of programmingFairly straightforward programming – small pieces and simple algorithmsRequires very little theoryVery boring for test designersProgramming is out of reach for many
domain experts
Who is responsible for determining and embedding the expected outputs ?Test designers may not always know the expected outputsTest evaluators need to get involved early to help with this
Embed test values into executable scripts
©
Ammann
& OffuttSlide11
Test Execution
This is easy – and trivial if the tests are well automatedRequires basic computer skillsInternsEmployees with no technical backgroundAsking qualified test designers to execute tests is a sure way to convince them to look for a
development job
If, for example, GUI tests are not well automated, this requires a lot of manual laborTest executors have to be very careful and meticulous with bookkeeping
©
Ammann
& Offutt
Run tests on the software and record the resultsSlide12
Test Evaluation
This is much harder than it may seemRequires knowledge of :DomainTestingUsually requires almost no traditional CSA background in the domain
of the software is essential
An empirical background is very helpful (biology, psychology, …)A logic background is very helpful (law, philosophy, math, …)This is intellectually stimulating, rewarding, and challengingBut not to typical CS majors – they want to solve problems and build things
©
Ammann
& Offutt
Evaluate results of testing, report to developersSlide13
Types of Test Activities – Summary
These four general test activities are quite differentIt is a poor use of resources to use people inappropriately
©
Ammann
& Offutt
1.
Test design
Design test values to satisfy coverage criteria or other engineering goal
Requires technical knowledge of discrete math, programming and testing
2.
Test automation
Embed test values into executable scripts
Requires knowledge of scripting
3.
Test execution
Run tests on the software and record the results
Requires very little knowledge
4.
Test evaluation
Evaluate results of testing, report to developers
Requires domain knowledge
But most test organizations use the same people for ALL FOUR activities !!Slide14
Testing Model – Black Box Testing
You know the functionalityGiven that you know what it is supposed to do, you design tests that make it do what you think that it should doFrom the outside, you are testing its functionality against the specsFor software, this is testing the interfaceWhat is input to the system?What you can do from the outside to change the system? (controllability)What is output from the system? (observability)
Impossible to thoroughly exercise all inputs
Exhaustive testing grows without boundTests the functionality of the system by observing its external behaviorNo knowledge of how it goes about meeting the goals
©L. WilliamsSlide15
Testing Model – White Box Testing
You know the codeGiven knowledge of the internal workings, you thoroughly test what is happening on the insideClose examination of procedural level of detailLogical paths through code are testedConditionalsLoopsBranchesStatus is examined in terms of expected valuesImpossible to thoroughly exercise all paths
Exhaustive testing grows without bound
Can be practical if a limited number of “important” paths are evaluatedCan be practical to examine and test important data structures
©L. WilliamsSlide16
Group Exercise
A program needs to be developed so that given an integer valueit outputs 0 when the integer value is 0it outputs 1 when the integer value > 0It outputs -1 when the integer value < 0
What would be your black box tests?
How would you generate your white box tests?Would black box tests alone be good enough to find bugs/faults in the program? Why?Would white box tests alone be good enough be find bugs/faults in the program? Why?Slide17
Black-box vs. White-box
White-box - look at code to write testTests are based on codeBetter for finding crashes, out of bounds failures, file not closed failuresBetter at finding faults of extra logicBlack-box - don’t look at code to write testTests are based on specificationsBetter at telling if program meets specBetter at finding faults of omission
17Slide18
Types of Testing
Unit Testing (white)testing of individual hardware or software units or groups of related unitsDone by programmer(s)Generally all white boxAutomation desirable for repeatability
Integration Testing (black and white)
testing in which software components, hardware components, or both are combined and tested to evaluate the interaction between themDone by programmer as they integrate their code into code baseGenerally white box, maybe some black boxAutomation desirable for repeatability
©L. WilliamsSlide19
Types of Testing -II
Functional/System Testing (black)testing conducted on a complete, integrated system to evaluate the system compliance with its specified requirementsstress testing, performance testing, usability testingit is recommended that this be done by external test groupmostly black box so that testing is not ‘corrupted’ by too much knowledgetest automation desirable
Acceptance Testing (black)
formal testing conducted to determine whether or not a system satisfies its acceptance criteria (the criteria the system must satisfy to be accepted by a customer) and to enable the customer to determine whether or not to accept the systemGenerally done by customer/customer representative in their environment through the GUI . . . Definitely black box
©L. WilliamsSlide20
Types of Testing -III
Regression Testing (black and white)Regression testing is selective retesting of a system or component to verify that modifications have not caused unintended effects and that the system or component still complies with its specified requirementsSmoke test group of test cases that establish that the system is stable and all major functionality is present and works under “normal” conditionsBeta Testing (black)(1~many) potential users or beta testers install software and use it as they wish and report any revealed errors to the development organization. A/B Testing
https://
en.wikipedia.org/wiki/A/B_testing
©L. WilliamsSlide21
Techniques for writing testsBlack-box (from specifications)
Equivalence partitioningBoundary value analysisWhite-box (from code)Branch coverageFault-based testing (from common errors)http://www.exampler.com/testing-com/writings.html from Brian Marick
21Slide22
Planning a Black Box Test Case
©L. WilliamsSlide23
Important Consideration for Black Box Test Planning
Look at requirements/problem statement to generate.Test cases need to be traceable to a requirement.You must write the repeatable test case so anyone on the team can run the exact test case and get the exact same result/sequence of events.
The inputs must be very
specific.Example: “Students who receive a grade of 70 or higher pass the exam.”Correct test cases: Grade = 80; Grade =20Incorrect test cases: “input a passing grade” “input a failing grade”The expected results must be very specific. “Pass” “Fail”
©L. WilliamsSlide24
Equivalence Class Partitioning
Divide your input conditions into groups (classes).Input in the same class should behave similarly in the program.Be sure to test a mid-range value from each class.Example: for tests of “Go to Jail” the most important thing is whether the player has enough money to pay the $50 fineTest input values clearly in the two partitions: 25 and 75.
©L. WilliamsSlide25
Equivalence partitioning
Example: sortingsort(array,len) takes an array of integers of length len and sorts them in ascending order, i.e. permutes them so that each element of the array is less than or equal to the succeeding one.len = 0,1, 2, 17Array is already sorted, has duplicates, has negative numbers25Slide26
Equivalence class test ideasAny object: the null pointer
Strings: the empty stringCollections:The empty collectionContains exactly one elementContains the maximum number of elements (or at least more than one)26Slide27
Equivalence class test ideasLinked structures (trees, queues, etc.)
EmptyMinimal but non-emptyCircularDepth greater than one (or maximally deep)Equality comparison of objectsEqual but not identicalDifferent at lowest level, the same at upper27Slide28
Boundary Value Analysis
Focus on boundaries . . . because a greater number of faults tend to occur at the boundaries of the input domainRange input, a to b, test with a, b, a-1, a+1, b-1, b+1 if integer range; otherwise, slightly less than
a
and slightly more than b.If you can only have a certain quantity (q) of something, try to create q-1, q, q+1
©L. WilliamsSlide29
Decision Table Testing
©L. WilliamsSlide30
Monopoly Decision Table
If a Player (A) lands on property owned by another player (B), A must pay rent to B. If A does not have enough money to pay B, A is out of the game.
©L. WilliamsSlide31
Dirty/Failure Test Cases
Can something cause division by zero?What if the input type
is wrong (You’re expecting an integer, they input a float. You’re expecting a character, you get an integer.)?
What if the customer takes an illogical path through your functionality?What if mandatory fields are not entered?What if the program is
aborted abruptly or input or output
devices
are
unplugged
?
©L. Williams
Think diabolically!
Robustness TestingSlide32
Techniques for writing testsBlack-box (from specifications)
Equivalence partitioningBoundary value analysisWhite-box (from code)Branch coverageFault-based testing (from common errors)Reading online: catalog from Brian Marick32Slide33
CoverageMake sure tests
cover each part of programEvery statementEvery branchEvery conditionEvery pass through a loopEvery path(?)Measures the quality of testsHow much of the program do the tests cover?
33Slide34
34
Coverage toolsTool “instruments” the programYou run your tests, it builds databaseTool looks at database to see which parts of the program were executed, and reports test coverageSome Java open source tools: EclEmma, Quilt, NoUnit,
InsECT
, Jester, jcoverage, Coverlispe, Hansel…Slide35
35
White-box testsPurpose: exercise all the codeLarge number - take a long time to writeGood for finding run-time errorsNull object, array-bounds errorIn practice, coverage is better for evaluating tests than for creating themSlide36
White Box Testing - Review
You know the codeGiven knowledge of the internal workings, you thoroughly test what is happening on the insideClose examination of procedural level of detailLogical paths through code are testedConditionalsLoopsBranchesStatus is examined in terms of expected valuesImpossible to thoroughly exercise all pathsExhaustive testing grows without bound
Can be practical if a limited number of “important” paths are evaluated
Can be practical to examine and test important data structures
©L. WilliamsSlide37
Devising a prudent set of test cases
Equivalence Class/Boundary Value AnalysisStill applies!A metric for assessing how good your test suite isMethod CoverageStatement CoverageDecision/Branch CoverageCondition CoverageThink diabolically
©L. WilliamsSlide38
Recall: Mistake, Fault, Error, Failure
Programmer makes a
mistake
.
Fault
(
defect, bug
) appears in the program.
Fault remains undetected during
testing (running
test inputs
).
The program
fails
(based on test oracles) during execution i.e
. it behaves unexpectedly.
Error
: difference between computed, observed, or measured value or condition and true
, specified, or theoretically correct value or condition
What does
Bug
mean in “
Bug Report
”?Slide39
© Ammann & Offutt
39
Fault & Failure Model
Three conditions necessary for a failure to be observed
Execution/
Reachability
: The location or locations in the program that contain the fault must be reached
Infection
: The state of the program must be incorrect
Propagation
: The infected state must propagate to cause some output of the program to be incorrect
PIE modelSlide40
40Slide41
41
Conversation 1:
Tester
Test Manager
Test Manager: Looks like the code under test is not achieving high statement coverage. Please work hard to achieve high statement coverage.
Tester: Hmm… boss, our goal is to detect faults. I don’t think I need to spend more efforts to achieve high statement
coverge
.
Test Manager: Well, according to the PIE model, …..[You fill in here]Slide42
42
Conversation 2:
Tester
Test Manager
Tester: Boss, following your command, I work very hard and I have already achieved 100% statement coverage! I would like to take
a vacation in
Hawaii. Could you approve?
Test Manager:
Well, according to the PIE model, …..[You fill in here]Slide43
43
Testing & Debugging
Testing
: Finding inputs that cause the software to fail
Debugging
: The process of finding a fault given a failureSlide44
© Ammann & Offutt
44
Test Case
Test Case Values/Test Input/Test Data
: The values that directly satisfy one test requirement
Expected Results
: The result that will be produced when executing the test if the program satisfies it intended behavior
Related Term:
Test Oracles
Tests
can mean different things in different contextsSlide45
© Ammann & Offutt
45
Observability and Controllability
Software Observability
: How easy it is to observe the behavior of a program in terms of its outputs, effects on the environment and other hardware and software components
Software that affects hardware devices, databases, or remote files have low observability
Software Controllability
: How easy it is to provide a program with the needed inputs, in terms of values, operations, and behaviors
Easy to control software with inputs from keyboards
Inputs from hardware sensors or distributed software is harder
Data abstraction reduces controllability and observabilitySlide46
46
What is a Unit Test?
A unit test is a small program with assertions.
[TestMethod] public void Add()
{
HashSet set = new HashSet();
set.Add(3);
set.Add(14);
Assert.AreEqual(set.Count, 2);
}Many developers write such unit tests by hand. This involvesdetermining a meaningful sequence of method calls,selecting exemplary argument values (the test inputs),stating assertions.Slide47
Unit Testing: Benefits
Design and specificationby exampleCode coverage and regression testingconfidence in correctnesspreserving behaviorShort feedback loopunit tests exercise little codefailures are easy to debugDocumentationSlide48
Unit Testing: Measuring Quality
Coverage: Are all parts of the program exercised?statementsbasic blocksexplicit/implicit branches…Assertions: Does the program do the right thing?test oracleExperience:Just high coverage or large number of assertions is no good quality indicator.Only both together are!Slide49
Advantages of tests as specs
Concrete, easy to understandDon’t need new languageEasy to see if program meets the specMaking tests forces you to talk to customer and learn the problemMaking tests forces you to think about design of system (classes, methods, etc.)
49Slide50
Disadvantages of tests as specs
Too specificHard to test that something can’t happenCan’t withdraw more money than you have in the systemCan’t break into the systemCan’t cause a very long transaction that hangs the systemTends to be verbose50Slide51
Tests as specificationsTests show how to use the system
Tests need to be readableNeed comments that describe their purpose or need good namesKeep short, delete duplicate or redundant51Slide52
Parameterized Unit Test
Parameterized Unit TestA parameterized unit test is a small program that takes
some
inputs and states assumptions and assertions.
52
JUnit: @Theory (multiple parameters) @Parameters (single parameter)Slide53
Parameterized Unit Test
Parameterized Unit Testing
Parameterized Unit Tests
serve as specificationscan be leveraged by (automatic) test input generatorsfit in development environment, evolve with the codeSlide54
Hand-written
Test Generation Process
54
//
FooTest.cs
[
TestClass
,
PexClass
]partial class
FooTest{ [
PexMethod]
void Test(Foo foo) {…}
// FooTest.Test.cspartial class FooTest{ [TestMethod] void Test_1() { this.Test(new Foo
(1)); }
[TestMethod] void Test_1() { this.Test(new Foo(2)); } …}Pex
User writes parameterized tests Lives inside a test class
Generated unit tests Pex not required for re-execution xUnit unit testsxUnit Attributes
Pex
Attributes
Parameterized Unit Test
Partial Class
Generated
http://msdn.microsoft.com/en-us/library/wa80x488(VS.80).
aspx
Slide55
PUTs separate concerns
PUTs separate two concerns:(1) The specification of external behavior (i.e., assertions)
(2) The selection of internal test inputs
(i.e., coverage) In many cases, a test generation tool (e.g., Pex) can construct a small test suite with high coverage
!Slide56
PUTs are algebraic specs
A PUT can be read as a universally quantified, conditional axiom.
int name, int data.
name
≠
null ⋀ data
≠
null
⇒ equals( ReadResource(name, WriteResource(name, data)), data)Slide57
Pex4Fun – Turning Pex Online
1,750,069
clicked 'Ask
Pex!'http://pex4fun.com/default.aspx?language=CSharp&sample=_
Template Slide58
http://research.microsoft.com/pex/
Dynamic Symbolic Execution (DSE) aka. Concolic Testing [Godefroid
et al. 05][
Sen et al. 05][Tillmann et al. 08]Instrument code to explore feasible pathsBehind the Scene of Pex4FunSlide59
http://research.microsoft.com/pex/Slide60
Walkthrough: Unit Testing in VS
Create ProjectCreate Test ClassCreate Testspassing, failing, expected to failRun TestsView CoverageNote: Other unit test frameworks exist for .Net, e.g. Nunit
Use
[PexMethod(TestEmissionFilter=PexTestEmissionFilter.All)] to force generation/displaying of all explored test dataSlide61
void
CoverMe
(
int
[] a)
{
if (a == null) return;
if (
a.Length
> 0)
if (a[0]
==
1234567890)
throw new Exception("bug");}a.Length>0a[0]==123…
T
FTF
F
a==null
T
Constraints to solve
a!=null
a!=null &&
a.Length
>0
a!=null &&
a.Length
>0 &&
a[0]==
123456890
Input
null
{}
{0}
{123…}
Execute&Monitor
Solve
Choose next path
Observed constraints
a==null
a!=null &&
!(
a.Length
>0)
a==null &&
a.Length
>0 &&
a[0]!=
1234567890
a==null &&
a.Length
>0 &&
a[0]==
1234567890
Done: There is no path left.
Dynamic Symbolic Execution in
Pex
http://
pex4fun.com/HowDoesPexWorkSlide62
Pex is Part of Visual Studio 2015 Eneterprise Edition!
As new feature of “IntelliTest”
https://
www.visualstudio.com/news/vs2015-vs#Testing Slide63
Domain Matrix for Testing Complex Condition
63Slide64
Guide Pex to Generate Test Data
//[PexMethod(TestEmissionFilter = PexTestEmissionFilter.All)] [PexMethod] public void TestBoundaryValuesAndInputPartition(
int
x, int y) { //boundary values/partitions for x > 0 && x <= 10 && y >= 1 PexAssume.IsTrue((x > 0)); if (x == 1) { } else if (x > 0) { } PexAssume.IsTrue((x <= 10));
if (x == 10) { } else if (x <= 10) { }
PexAssume.IsTrue
((y >= 1));
if (y == 1) { }
else if (y > 1) { } }64Details see http://taoxie.cs.illinois.edu/publications/icsm10-coverage.pdf Slide65
Parameterized Unit Tests Supported by Pex/Pex4Fun
using System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;[PexClass]
public class Set {
[PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue
(i != j); bool exist = s.member
(i);
s.insert
(j);
PexAssert.IsTrue(exist); } ….} 65Slide66
Interface for IntSet
Class IntSet { public IntSet() {…}; public void insert(int e) { … } public Bool
member(
int e) { … } public void remove(int e) { … }}sort IntSet imports Int, Bool signatures
new : -> IntSet
insert
:
IntSet
× Int -> IntSet member : IntSet × Int -> Bool remove : IntSet × Int -> IntSet66http://www.cs.unc.edu/~stotts/723/adt.htmlSlide67
(Buggy) Implementation for IntSet
Class IntSet { public IntSet() {…}; public void insert(int e) { … } public
Bool
member(int e) { … } public void remove(int e) { … }}See the Set.cs that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/Set.cs
Let’s copy it to http://pex4fun.com/default.aspx?language=CSharp&sample=_Template
And Click “Ask
Pex
”
67Slide68
Parameterized Unit Tests Supported by Pex/Pex4Fun
using System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;[PexClass]
public class Set {
[PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue
(i != j); bool existOld =
s.member
(i);
s.insert
(j); bool exist = s.member(i); PexAssert.IsTrue(existOld == exist); } ….} 68Pex4Fun supports only one PexMethod at a time; you can write multiple PexMethods but comment out other lines of “[PexMethod]” except oneSlide69
Axioms for IntSet
variables i, j : Int; s : IntSet Axioms:member(new(), i) = false member(insert(s, j), i) = if i = j then true
else member(s, i)69http://www.cs.unc.edu/~stotts/723/adt.htmlIs this complete?
How
do we know?Slide70
Guidelines for Completeness
Classify methods:constructors: return IntSetinspectors: take IntSet as argument, returning some other value.Identify key constructors, capable of constructing all possible object statese.g., insert, new. Identify
others as auxiliary,
e.g., remove is a destructive constructorCompleteness requires (at least): every inspector/auxiliary constructor is defined by one equation for each key constructor. 70Slide71
Add More Axioms
remove(new(), i) = new() remove(insert(s, j), i) = if i = j then remove(s, i) else insert(remove(s, i), j)71
Are we done yet
?The completeness criterion (an equation defining member and remove for each of the new and insert constructors) is satisfied.Slide72
Guidelines for Completeness
But does this really specify sets? Do the following properties hold?Order of insertion is irrelevant.insert(insert(s, i), j) = insert(insert(s, j), i)Multiple insertion is irrelevant.insert(insert(s, i), i) = insert(s, i)72Slide73
Interface (Implementation) for UIntStack
Class UIntStack { public UIntStack() {…}; public void Push(int k) { … } public void Pop() { … } public int Top() { … }
public
bool IsEmpty() { … } public int MaxSize() { … } public bool IsMember(int k) { … } public bool Equals(UIntStack
s) { … } public int GetNumberOfElements() { … }
public
bool
IsFull
() { … }}73See the UIntStack.cs that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs Slide74
Group Exercise: Write Parameterized Unit Tests (PUTs)
Class UIntStack { public UIntStack() {…}; public void Push(int k) { … } public void Pop() { … }
public int Top() { … } public bool IsEmpty() { … } public int MaxSize() { … } public bool
IsMember(int k) { … }
public
bool
Equals(
UIntStack s) { … } public int GetNumberOfElements() { … } public bool IsFull() { … }}74Let’s copy it to http://pex4fun.com/default.aspx?language=CSharp&sample=_Template And Click “Ask Pex”Reminder: you have to comment earlier written “[
PexMethod]” before you try Pex on your current PUT (Pex4Fun can handle only one PUT at a time)
See the UIntStack.cs
that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/UIntStack.cs Slide75
(Buggy) Implementation for IntSet
Class IntSet { public IntSet() {…}; public void insert(int e) { … } public
Bool
member(int e) { … } public void remove(int e) { … }}See the Set.cs that can be downloaded fromhttp://taoxie.cs.illinois.edu/courses/testing/Set.cs Let’s copy it to
http://pex4fun.com/default.aspx?language=CSharp&sample=_Template
And Click “Ask
Pex
”
75Slide76
Recall: Parameterized Unit Tests Supported by Pex/Pex4Fun
using System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;[PexClass]
public class Set {
[PexMethod] public static void testMemberAfterInsertNotEqual(Set s, int i, int j) { PexAssume.IsTrue(s != null); PexAssume.IsTrue(i != j);
bool existOld = s.member(i);
s.insert
(j);
bool exist =
s.member(i); PexAssert.IsTrue(existOld == exist); } ….} 76Slide77
Force Pex/Pex4Fun to Display All Explored Test Inputs/Paths
using System;using Microsoft.Pex.Framework;using Microsoft.Pex.Framework.Settings;[PexClass
]
public class Set { [PexMethod(TestEmissionFilter=PexTestEmissionFilter.All)] public static void testMemberAfterInsertNotEqual(Set s, int
i, int j) {
PexAssume.IsTrue
(s != null);
PexAssume.IsTrue(i != j); bool exist = s.member(i); s.insert(j); PexAssert.IsTrue(exist); } ….} 77Slide78
Factory Method: Help Pex Generate Desirable Object States
In class, we show the factory method as below automatically synthesized by Pex after a user clicks “1 Object Creation” issue and then click “Accept/Edit Factory Method”. But it is not good enough to generate various types of object states. [PexFactoryMethod(typeof(
UIntStack
))] public static UIntStack Create(int k_i) { UIntStack uIntStack = new UIntStack(); uIntStack.Push(k_i
); return uIntStack;
// TODO: Edit factory method of
UIntStack
// This method should be able to configure the object in all possible ways.
// Add as many parameters as needed,
// and assign their values to each field by using the API. }78Slide79
Factory Method: Help Pex Generate Desirable Object States
Below is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method. [PexFactoryMethod
(
typeof(UIntStack))] public static UIntStack CreateVariedSizeAnyElemsStack(int[] elems) { PexAssume.IsNotNull(elems);
UIntStack s = new UIntStack();
PexAssume.IsTrue
(
elems.Length
<= (s.MaxSize() + 1)); for (int i = 0; i < elems.Length; i++) s.Push(elems[i]); return s; }79Slide80
One Sample PUT
Below is a manually edited/created good factory method to guide Pex to generate various types of object states. Note that Pex also generates argument values for the factory method. [PexMethod]
public void
TestPush([PexAssumeUnderTest]UIntStack s, int i) { //UIntStack s = new UIntStack(); PexAssume.IsTrue(!s.IsMember(
i)); int
oldCount
=
s.GetNumberOfElements
();
s.Push(i); PexAssert.IsTrue(s.Top() == i); PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1); PexAssert.IsFalse(s.IsEmpty()); }80Slide81
Pex4Fun Not Supporting Factory Method - Workaround
If you try PUTs on Pex4Fun, which doesn’t support factory method, you can “embed” the factory method like the highlighted code portion below [PexMethod] public void TestPush(int
[]
elems, int i) { PexAssume.IsNotNull(elems); UIntStack s = new
UIntStack();
PexAssume.IsTrue
(
elems.Length
<= (
s.MaxSize() + 1)); for (int i = 0; i < elems.Length; i++) s.Push(elems[i]); //UIntStack
s = new UIntStack(); PexAssume.IsTrue(!s.IsMember(
i)); int
oldCount = s.GetNumberOfElements(); s.Push(
i); PexAssert.IsTrue(s.Top() == i); PexAssert.IsTrue(s.GetNumberOfElements() == oldCount+1); PexAssert.IsFalse(s.IsEmpty()); }81Slide82
Guideline of Writing PUT
Setup: basic set up for invoking the method under testCheckpoint: Run Pex to make sure that you don't miss any
Pex
assumptions (preconditions) for the PUTAssert: add assertions for asserting behavior of the method under test, involving Adding Pex assertions
Adding Pex
assumptions for helping assert
Adding method sequences for helping assertSlide83
Setup
Select your method under test mPut its method call in your PUTCreate a parameter for your PUT as the class under test
c
(annotated it with [PexAssumeUnderTest])Create other parameters for your PUT for parameters of m if anyAdd Pex assumptions for preconditions for all these parameters of PUT if anySlide84
Setup - Example
[PexMethod] public void TestPush([PexAssumeUnderTest]UIntStack s, int i) { s.Push(i);
}
You may write your factory method to help Pex in test generationIf you get exceptions thrown
if indicating program faults, fix them
If indicating lack of PUT assumptions, add PUT assumptions
If indicating insufficient factory method assumptions or inappropriate scenarios, add PUT assumptions or improve factory method.Slide85
Assert
Think about how you can assert the behaviorDo you need to invoke other (observer) helper methods in your assertions (besides asserting return values)?Do you need to add assumptions so that your assertions can be valid?
Do you need to add some method sequence before the method under test to set up desirable state and cache values to be used in the assertions?Slide86
Targets for Asserting
Return value of the method under test (MUT)Argument object of MUTReceiver object properties being modified by MUT (if public fields, directly assertable)
How to assert them?
Think about the intended behavior!If you couldn't do so easily, follow the guidelines discussed nextSlide87
Cached Public Property Value
A property value before invoking MUT may need to be cached and later used.
Pattern 2.1/2.2: Assume, Arrange, Act, Assert
[PexMethod]void
AssumeActAssert(ArrayList
list, object item) {
PexAssume.IsNotNull
(list); // assume
var count = list.Count; // arrange list.Add(item); // act Assert.IsTrue
(list.Count ==
count
+ 1); // assert}Slide88
Argument of MUT
Argument value of MUT may be usedPattern 2.3:Constructor Test[
PexMethod
]void Constructor(int capacity) { var list = new ArrayList
(capacity
); // create
AssertInvariant
(list); // assert invariant
Assert.AreEqual(capacity, list.Capacity); // assert}Slide89
Reciever or Argument of Earlier Method
Receiver or argument value of a method before invoking MUTPattern 2.4/5:Roundtrip
[
PexMethod]void ToStringParseRoundtrip(int value) { // two-way roundtrip
string s =
value
.ToString
();
int parsed = int.Parse(s); // assert Assert.AreEqual(value, parsed);}
value
s parsedSlide90
Observer Methods
Invoking observer methods on the modified object statePattern 2.6: State Relation[
PexMethod
]void InsertContains(string value) { var list = new List<string>();
list.Add(value);
Assert.IsTrue
(
list.
Contains(value));}Each modified object property should be read by at least one observer method.Slide91
Observer Methods cont.
Forcing observer methods to return specific values (e.g., true or false) can force you to add specific assumptions or scenarios[PexMethod]
void
PushIsFull([PexAssumeUnderTest]UIntStack s, int value) {
PexAssume.IsTrue(
s.GetSize
() == (
s.GetMaxSize
()-1));
s.Push (value); Assert.IsTrue(s.IsFull ());}Slide92
Alternative Computation
Invoking another method/method sequence to produce a value to be usedPattern 2.7: Commutative Diagram[
PexMethod
]void CommutativeDiagram1(int x, int y) { // compute result in one way
string z1 = Multiply(x, y).
ToString
();
// compute result in another way
string z2 = Multiply(
x.ToString(), y.ToString()); // assert equality if we get here PexAssert.AreEqual(z1, z2);}Slide93
Divide and Conquer
Split possible outcomes into cases (each with pre and post condition)Pattern 2.8: Cases[
PexMethod
]void BusinessRules(int age, Job job) { var salary = SalaryManager.ComputeSalary(age, job); PexAssert
.Case(age < 30)
.Implies
(() => salary < 10000)
.
Case(job == Job.Manager && age > 35) .Implies(() => salary > 10000) .Case(job == Job.Manager && age < 20) .Implies(() => false); }Slide94
Class Invariant Checker
If class invariant checker (repOk) exists or you would be willing to write one, use it to assert
Pattern 2.3:Constructor Test
[PexMethod]void Constructor(int capacity) {
var
list = new
ArrayList
(capacity); // create
AssertInvariant(list); // assert invariant Assert.AreEqual(capacity, list.Capacity); // assert}Slide95
Other Patterns
Pattern 2.9: Allowed exceptions[PexAllowedException(
typeof
(ArgumentNullException))][ExpectedException(typeof
(
ArgumentNullException
))]
Pattern 2.10: Reachability
[
PexExpectedGoals] + throw new PexGoalException();Pattern 2.11: Parameterized StubNo scenarios or assertionsPattern 2.12: Input Output Testvoid Add(int
a, int b, out
int
result) { … }int
Substract(int a, int b) { … }Pattern 2.13/14: Regression Testsbool Parse(string input) { … }PexStore.ValueForValidation("result", result);http://research.microsoft.com/en-us/projects/pex/patterns.pdf Slide96
96Slide97
Test-Driven Development (TDD)
Basic Idea:Write tests before codeRefine code with new testsIn more detail, TDD is a cycle of steps:Add a test,Run it and watch it fail,Change the code as little as possible such that the test should pass,Run the test again and see it succeed,Refactor the code if needed.Slide98
Note: TDD and specifications
TDD encourages writing specifications before codeExemplary specificationLater, we will generalize TDD toParameterized TDDAxiomatic specificationsSlide99
Parameterized Test-Driven Development
Write/refine Contract as PUTWrite/refine Code of Implementation
Fix-it (with
Pex),Debug with generated tests
Use Generated Tests for Regression
Run
Pex
Bug in PUT
Bug in Code
failures
no failuresSlide100
Coding Duels
1,750,069
clicked 'Ask
Pex!'Slide101
Coding Duels
Pex
computes “semantic diff” in cloud
secret reference implementation vs.
code written in
browser
You win when
Pex
finds no differences
secret
For more info, see our ICSE 2013 SEE paper:
http://
taoxie.cs.illinois.edu/publications/icse13see-pex4fun.pdf
Slide102
Behind the Scene of
Pex for Fun
Secret
Implementation class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}
Player Implementation
class Player {
public static
int
Puzzle(
int x) { return x; }} class Test {public static void Driver(int x) {
if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}
behavior
Secret
Impl == Player Impl102
1,594,092
1,594,092Slide103
Code Hunt Programming Game
https://www.codehunt.com/ Slide104
Code Hunt Programming GameSlide105
Code Hunt Programming GameSlide106
Code Hunt Programming GameSlide107
Code Hunt Programming GameSlide108
Code Hunt Programming GameSlide109
Code Hunt Programming GameSlide110
Code Hunt Programming GameSlide111
Code Hunt Programming GameSlide112
Code Hunt Programming GameSlide113
Code Hunt Programming GameSlide114
Code Hunt Programming GameSlide115
Code Hunt Programming GameSlide116
It’s a game!
iterative gameplayadaptivepersonalizedno cheatingclear winning criterion
secret
code
test cases