Software Verification and Validation Lecture No. 9

slide1. Software Verification and Validation Lecture No. 9<br>
slide2. Module 57:Specification based Testing<br>
slide3. Specification based testing Specification or model?
Recall
Models typically provide some abstract representation of the behavior of the system.
Typical notations are:
Algebraic Specifications
Control/Dataflow Graphs
Logic-based Specifications<br>
slide4. Specification based testing Specification or model?
Typical notations are (Contd.):
Finite State Machine Specifications
Grammar-based Specifications
Considering Specifications written using informal languages, e.g., English<br>
slide5. Specification based testing Independently Testable Feature (ITF):
depends on the control and observation that is available in the interface to the system
Test case:
inputs, environment conditions, expected results.
Test case specification:
a property of test cases that identifies a class of test cases.<br>
slide6. Specification based testing Functional specification:
This can be some kind of formal specification that claims to be comprehensive.
Often it is much more informal comprising a short English description of the inputs and their relationship with the outputs.
For some classes of system the specification is hard to provide (e.g. a GUI, since many of the important properties relate to hard to formalize issues like usability).<br>
slide7. Specification based testing<br>
slide8. Specification based testing We slice specification into features (that may spread across many code modules).
Each feature should be independent of the other, i.e. we can concentrate on testing one at a time.
The design of the code will make this easier or more difficult depending on how much attention has been given to testability in the systems design.
called as Independently Testable Function (ITF)<br>
slide9. Specification based testing For each of the inputs we consider classes of values that will all generate similar behavior.
This will result in a very large number of potential classes of input for non-trivial programs.
We then identify constraints that disallow certain combinations of classes. The goal is to reduce the number of potential test cases by eliminating combinations that do not make sense.<br>
slide10. Specification based testing Command: find

Syntax: find <pattern> <filename>

Function: The find command is used to locate one or more instances of the given pattern in the named file. All matching lines in the named file are written to standard output. …<br>
slide11. Specification based testing We study methods where we have specifications written in plain English and we extract test cases out of them

We also study how we can extract test cases from model of a system

In upcoming lecture modules<br>
slide12. Module 58:Scenario-based Testing<br>
slide13. Scenario-based Testing What is a scenario:
A scenario is a hypothetical story, used to help a person think through a complex problem or system.
While doing scenario based testing: test case is based on a story about how the program is used, including information about the motivations of the people involved.<br>
slide14. Scenario-based Testing What is scenario-based testing:
The story is motivating. A stakeholder with influence would push to fix a program that failed this test. (Anyone affected by a program is a stakeholder.

A person who can influence development decisions is a stakeholder<br>
slide15. Scenario-based Testing What is scenario based testing:
The story is credible. It not only could happen in the real world; stakeholders would believe that something like it probably will happen.
The story involves a complex use of the program or a complex environment or a complex set of data.
The test results are easy to evaluate. This is valuable for all tests, but is especially important for scenarios because they are complex.<br>
slide16. Scenario-based Testing In a nutshell:
Learn the product
Connect testing to documented requirements
Expose failures to deliver desired benefits
Explore expert use of the program
Bring requirements-related issues to the surface, which might involve reopening old requirements discussions (with new data) or surfacing not-yet-identified requirements.<br>
slide17. Scenario-based Testing There are two types of scenario testing.
Type 1 – scenarios used as to define input/output sequences. They have quite a lot in common with requirements elicitation.
Type 2 – scenarios used as a script for a sequence of real actions in a real or simulated environment<br>
slide18. Scenario-based Testing Quite a lot of what is needed in order to write a good scenario is closely related to what is needed to write good requirements.

One of the reasons why scenario testing is so efficient may be that we, in some sense, we repeat the requirements process but with other people involved.<br>
slide19. Scenario-based Testing Type – 1 scenario:
Some rules for writing a good scenario:
List possible users – analyze their interest and objectives
List system events. How does the system handle them?
List special events. What accommodations does the system make for these?<br>
slide20. Scenario-based Testing List benefits and create end-to-end tasks to achieve them
Work alongside users and to see how they work and what they do
Read about what systems like this is supposed to do
Create a mock business. Treat it as real and process its data<br>
slide21. Scenario-based Testing Users
When we later say users, we mean the prospective users – those who will later use the system that we are currently developing and testing.

What we need to do is to understand how the system is used by its real users.<br>
slide22. Scenario-based Testing List possible users:
For each identified user, identify his interests. A user will value the system if it furthers his interests.

Focus on one interest at the time. Identify the user’s objectives.

How can we test that each objective is easy to achieve?<br>
slide23. Scenario-based Testing List system events:
An event is any occurrence that the system is supposed to respond to.
E.g. for a real time system – anything that generates an interrupt is an event.
For each event we need to understand
Its purpose
What the system is supposed to do in it
Rules related to the event<br>
slide24. Scenario-based Testing List special events:
Special events are events that are predictable but unusual. They require special handling.

They will also require special circumstances in order to be triggered. Make sure the scenario includes at least the most important of these circumstances.<br>
slide25. Scenario-based Testing List benefits:
What are the benefits that the system is supposed to provide to the users?
Ask the stakeholders about their opinions.
Watch out for
Misunderstandings
Conflicts – e.g. between groups of users or between users and customers.<br>
slide26. Scenario-based Testing Work alongside real users:
Observe then at their work. What do they
Usually do during a working day
Have problems with
How do they usually solve their problems
These observations help us to understand the users and give use ideas for scenarios.<br>
slide27. Scenario-based Testing Read about this type of systems:
Before writing a scenario it is important to have knowledge on
What is the new system supposed to do – system requirements.
What does other, similar systems do – system expectations. This knowledge can be obtained through books, manuals etc.<br>
slide28. Scenario-based Testing Create mock business:
To crate a mock business we need first of all need good knowledge of how the business works. The mock business must be realistic even if it isn’t real. It might be necessary to use external consultants.

Creating a mock business takes a lot of resources but can give valuable results.<br>
slide29. Scenario-based Testing Risks of scenario testing – 1:
Scenario testing is not good for testing new code. If we hit a bug early in the scenario test, the rest of the test will most likely have to be postponed until the bug is fixed.

The we can resume the testing and go on until we find a new bug and so on.

Thus, scenario testing should mainly be used as an acceptance test.<br>
slide30. Scenario-based Testing A scenario test aims to cover all or part of the functionality in a system. It does not consider code coverage of any kind.

Scenario tests discover more design errors than coding errors. Thus, it is not useful for e.g. regression testing or testing a new fix.<br>
slide31. Scenario-based Testing Scenario testing type 1:
When we do scenario testing type 1, we use the scenarios to write transactions as sequences of
Input
Expected output

The result can e.g. be an extremely detailed textual use case.<br>
slide32. Scenario-based Testing Scenario testing type-2:
When it comes to realism, scenario testing type 2 is a good testing method. The goal of scenario testing is to test how the system will behave
In real word situations – described by scenarios
With real users, supplied by the system owner
With real customers – if necessary<br>
slide33. Scenario-based Testing Scenario testing type-2:
A scenario tests is done as follows:
The environment is arranged according to the scenario description. Customers are instructed as needed.
A person – the game master – reads out each step of the scenario.<br>
slide34. Scenario-based Testing Scenario testing type-2:
A scenario tests is done as follows:
Users and customers react to the situations created by the game master.
The events resulting from each scenario is documented – e.g.
by a video camera or by one or more observers
for later assessment.<br>
slide35. Scenario-based Testing The number of possible scenarios is large. Which scenarios we select depends on the customer’s priorities. In addition, since scenario tests are expensive, we can usually just afford to run a few.
Scenarios are most efficient when we want to test requirements involving a strong interaction with the systems environment – users, customers, networks, file servers, a stressful work situation etc.<br>
slide36. Module 59:Scenario based testing - example<br>
slide37. Scenario based testing - example Example:
Requirement – MTTR < 1 hr.
Scenario for order handling system:
We have 50 Mb. of information on the system’s hard disk.
When we are registering a new order, we suddenly loose the electrical power.
At the same time several customers call us on the phone to enquire the status of their order.<br>
slide38. Scenario based testing - example Example (Contd.)
When we run the scenario one or more times, we want to measure the time from the power loss to the system is fully operational again.
The test may identify problems pertaining to:
Operational routines – e.g back-up
Operator training – stress handling
Customer handling
What about the half-filled-in order?<br>
slide39. Scenario based testing - example Example – 2:

Imagine we are working on a home heating system
We have a specification at hand
We would do two iterations
First we do use case based testing
Next we do scenario based testing<br>
slide40. Scenario based testing - example Use case: Power Up (Normal)
Actors: Home Owner (initiator)
Type: Primary and essential
Description: The Home Owner turns the power on. (Power fails)
Perform Adjust Temp.
Check temperature in all rooms, (Thermostat fails)
temperature OK, (Thermometer fails)
no actions taken. (Due to thermostat/meter failure)
Cross Ref.: Requirements XX, YY, and ZZ
Use-Cases: Perform Adjust Temp Home Heating Use-Cases<br>
slide41. Scenario based testing - example Use case: Power Up (Temp Low)
Actors: Home Owner (initiator)
Type: Primary and essential
Description: Owner turns the power on.
room temperature checked.
One of the room is below desired temperature
Valve for the room is opened,
water pump started.
water temp falls below threshold,
the fuel valve is opened, burner ignited.

Cross Ref.: Requirements XX, YY, and ZZ
Use-Cases: None<br>
slide42. Scenario based testing - example Use case: Adjust Temp
Actors: System (initiator)
Type: Secondary and essential
Description: Check the temperature in each room. For each room:
Below target: Perform Temp Low
Above target: Perform Temp High
Cross Ref.: Requirements XX, YY, and ZZ
Use-Cases: Temp Low, Temp High<br>
slide43. Scenario based testing - example Use case: Temp Low
Actors: System (initiator)
Type: Secondary and essential
Description: Open room valve, start pump if not started.
If water temp falls below threshold,
open fuel value and ignite burner.
Cross Ref.: Requirements XX, YY, and ZZ
Use-Cases: None<br>
slide44. Scenario based testing - example From Use Cases to Test Cases<br>
slide45. Scenario based testing - example Test Cases<br>
slide46. Scenario based testing - example Scenarios development Scenario - I User switches the heating on
All temperatures (current temperature and set temperatures are shown)
System starts working<br>
slide47. Scenario based testing - example Scenario – II
User switches the system on
All temperatures (current temperature and set temperatures are shown)
System starts working
The room temperature is lower than the set temperature
System opens valves for water movement and ignites burner
System open fuel valve.<br>
slide48. Scenario based testing - example Scenario – III
User switches the system on, temperatures (current temp./set temp. shown), System starts working
The room temperature is lower than the set temperature
System opens valves for water movement and ignites burner
System open fuel valve, burner fails and fuel supply is closed with error notification to user<br>
slide49. Scenario based testing - example Conclusion:
Scenarios can be viewed differently from:
System’s usage perspective
Usability perspective
Functionality validation perspective
It is important to find out what you are planning to conduct or perform<br>
slide50. Module 60:Equivalence Class Testing<br>
slide51. Equivalence Class Testing We start our discussion with an example.
Let us have a program specification: We require a routine that should take two integer values as input and returns a+b if a < b and a/b if a > b.
Let there be the following code against this specification:
Int myDiv (int a; int b) {
a > b ? return a / b : return a + b;
}<br>
slide52. Equivalence Class Testing Considering our previous knowledge, we only know handling situations w.r.t CFG or DFG Test it with respect to MCDC
a>b -> (2, 1), 2
A<b -> (1, 2), 3
With respect to CFG and we find out that there all paths are covered yet, (2, 0) would lead to a crash.
We need alternatives
We study equivalence class partitioning, applicable to all levels i.e., unit, integration and system levels<br>
slide53. Equivalence Class Testing Equivalence class testing uses mathematical concept of equivalence class to generate test cases for largely Functional (Black-box) testing
However, it can be used equally at unit and integration level since it works on the inputs and classifies it into disjoint sets.
The key goals for equivalence class testing are:
completeness of test coverage
Lessens effort of test coverage<br>
slide54. Equivalence Class Testing Let A = ( a1, a2, - - - -, an )
We divide A into disjoint subsets such that:
a1 U a2 U - - - - U an = A
for any i and j, ai ∩ aj = Ø
Equivalence relation R, Let R is defined out input variable’s value set X, is a relation defined on the set A, such that it is reflexive, transitive, and symmetric
reflexive : -2 € X then (-2, -2) € R
symmetric : (3, 5) € R, then (5, 3) € R
transitive: (-5, -7) € R and (-7, -200) € , then (-5, -200) € R<br>
slide55. Equivalence Class Testing Let us consider Code for our earlier example once again
Int myDiv (int a; int b)
{
a > b ? return a / b : return a + b;
}
We take values for inputs a and b from set of integers I:
I = { …, -3, -2, -1, 0, 1, 2, 3, …}
A = A1 U A2 U A3 such that
A1 = {…, -3, -2, -1}
A2 = {0};
A3 = {1, 2, 3, …}
A1 Ո A2 Ո A3 = Ф<br>
slide56. Equivalence Class Testing A1 = {…, -3, -2, -1}
A2 = {0};
A3 = {1, 2, 3, …}
Test cases {(A1, A1), (A1, A2), (A1, A3), …
Test cases {(A3, A3), (A1, A2), (A1, A3),…
(1,2), 3; (1,1), 2; (1,3), 4; .. All of them equivalent
So we need to devise a strategy<br>
slide57. Equivalence Class Testing We select test cases
Within range
At the boundaries of the range
Outside range (“illegal”)
A1 = {…, -3, -2, -1}
A2 = {0};
A3 = {1, 2, 3, …}
Range = 8 bit wide +ve integer value
A1 = {…, -3, -2, -1}
A2 = {0};
A3 = {1, 2, 3, …}<br>
slide58. Equivalence Class Testing A2 = {0};
A3 = {1, 2, 3, …, 256}
Test cases {(1, 2), (2, 1), (1, 0), (0, 1) … (2, 2)
Test cases (1, 0), (2, 2) would surface program defects
We have a mechanism to:
select data from equivalence classes
Less number of equivalent test cases
Consider boundary values (boundary value analysis)<br>
slide59. Module 61:Weak and Strong Normal and Robust Equivalence<br>
slide60. Weak and Strong Normal and Robust Equivalence Weak Normal equivalence testing:
Assumes the “independence of input variables.”
e.g. If there are 2 input variables, these input variables are independent of each other.
Partition the test cases of each input variable separately into different equivalence classes.
Choose the test case from each of the equivalence classes for each input variable independently of the other input variable<br>
slide61. Weak and Strong Normal and Robust Equivalence Strong Normal equivalence testing:
This is the same as the weak normal equivalence testing except for

“dependence” among the inputs

All the combinations of equivalence classes of the variables must be included.<br>
slide62. Weak and Strong Normal and Robust Equivalence Weak robust equivalence testing:
Up to now we have only considered partitioning the valid input space.
“Weak robust” is similar to “weak normal” equivalence test except that the invalid input variables are now considered.<br>
slide63. Weak and Strong Normal and Robust Equivalence A note about considering invalid input is that there may not be any definition specified for the different invalid inputs
This makes
the considering the output as the defining process for equivalence classes a bit more difficult.<br>
slide64. Weak and Strong Normal and Robust Equivalence Strong robust equivalence testing:
assumes dependency of input variables

“Strong robust” is similar to “strong normal” equivalence test except that the invalid input variables are now considered.<br>
slide65. Module 62:Equivalence Class (with boundary value) Example<br>
slide66. Equivalence Class (with boundary value) Example We take case:
Specification
We require a system that takes 3 inputs (a, b c), and returns if a triangle is possible evaluating if a<b+c, b<a+c and c<a+b.
In case a triangle is possible, it returns if resulting triangle is
scalene (no sides equal),
equilateral (two sides equal) or
isosceles (all sides equal)
Input range is 1-100<br>
slide67. Equivalence Class (with boundary value) Example “valid” inputs:
1<= a <= 200
1<= b <= 200
1<= c <= 200
and
for triangle:
a < b + c
b < a + c
c < b + a output Not triangle Equilateral Isosceles Scalene inputs a b c 15 10 4 15 15 15 15 15 7 15 18 14<br>
slide68. Equivalence Class (with boundary value) Example Since there is no further sub-intervals inside the valid inputs for the 3 sides a, b, and c, Strong Normal Equivalence is the same as the Weak Normal Equivalence<br>
slide69. Equivalence Class (with boundary value) Example Weak robust equivalence test cases:
Include 6 invalid test case in addition to Weak Normal
<101, 45, 50 >
< -5, 76, 89 >
<45, 104, 78 >
< 56, -5, 89 >
<50, 78, 108 >
< 56, 89, 0 > <100,100,100> <1, 1, 1><br>
slide70. Equivalence Class (with boundary value) Example Strong Robust equivalence test cases
Similar to Weak robust, but all combinations of “invalid” inputs but be included.
Look at the “cube” figure and consider the corners (two diagonal ones)
Consider one of the corners: there should be (23 – 1) = 7 cases of “invalids”
< 101, 101, 101 > < 50 , 101, 50 >
< 101, 101, 50 > < 50 , 101, 101 >
< 101, 50 , 101 > < 50, 50 , 101 >
< 101, 50 , 50 >
There will be 7 more “invalids” when we consider the other corner , <0,0,0 ><br>
slide71. Equivalence Class (with boundary value) Example The following testing schemes are overlapping in their concepts:
Equivalence classes based testing
Boundary value analysis
Category-partitioning
We have covered equivalence class testing with few references to other two.<br>
slide72. Module 63:Decision Table Based Testing<br>
slide73. Decision Table Based Testing Decision table is based on logical relationships just as the truth table.

It is a tool that helps us look at the “completeness” and “consistency” of combination of conditions<br>
slide74. Decision Table Based Testing C1
C2
C3 a1
a2
a3
a4
a5 T T T T F F F F T T F F T T F F T F T F T F T F x x x x x x x x x x x x x x conditions actions values of conditions actions taken R1 R2 R3 R4 R5 R6 R7 R8 rules<br>
slide75. Decision Table Based Testing The conditions in the decision table may take on any number of values. When it is binary, then the decision table conditions are just like a truth table set of conditions.

The decision table allows the iteration of all the combinations of values of the condition, thus it provides a “completeness check.”

The conditions in the decision table may be interpreted as the inputs, and the actions may be thought of as outputs.<br>
slide76. Module 64:Decision Table Based Testing Example<br>
slide77. Decision Table Based Testing Example We take case:
Specification
We require a system that takes 3 inputs (a, b c), and returns if a triangle is possible evaluating if a<b+c, b<a+c and c<a+b.
In case a triangle is possible, it returns if resulting triangle is
scalene (no sides equal),
equilateral (two sides equal) or
isosceles (all sides equal)
Input range is 1-100<br>
slide78. Decision Table Based Testing Example a < b + c
b < a + c
c < a + b

a = b
a = c
b = c Not triangle

Scalene
Isosceles
Equilateral
“impossible” F T T T T T T T T T T
- F T T T T T T T T T
- - F T T T T T T T T

- - - T T T T F F F F
- - - T T F F T T F F
- - - T F T F T F T F X X X

X
X X X
X
X X X Assume a, b and c are
all between 1 and 100<br>
slide79. Decision Table Based Testing Example There is the “invalid situation” --- not a triangle:
There are 3 test conditions in the Decision table
Note the “-” entries, which represents “don’t care,” when it is determined that the input sides <a, b, c> do not form a triangle
There is the “valid” triangle situation:
There are 3 types of valid; so there are 23 = 8 test conditions
But there are 3 “impossible” situations
So there are only 8 – 3 = 5 test conditions
So, for values of a, b, and c, we need to come up with 8 sets of <a, b, c> to test the (3 + 5) = 8 test conditions.<br>
slide80. Decision Table Based Testing Example Advantages: (check completeness & consistency)
Allow us to start with a “complete” view, with no consideration of dependence
Allow us to look at and consider “dependence,” “impossible,” and “not relevant” situations and eliminate some test cases.
Allow us to detect potential error in our Specifications<br>
slide81. Decision Table Based Testing Example Disadvantages:
Need to decide (or know) what conditions are relevant for testing - - - this may require Domain knowledge
e.g. need to know leap year for “next date” problem in the book
Scaling up can be massive: 2n rules for n conditions - - - that’s if the conditions are binary and gets worse if the values are more than binary<br>