Combinatorial - PowerPoint Presentation

Slide1

Combinatorial Interaction Testing for Automated Constraint Repair

Angelo Gargantini1, Justyna Petke2, Marco Radavelli11University of Bergamo, Italy2UCL, London, UK International Workshop on Combinatorial Testing 2017

Angelo

Gargantini - CIT for Automated Constraint Repairtests

13 March 2017 - IWCT2017

Ack

: thanks to BEN team (Laleh Shikh Gholamhosseingh, UTA) and Paolo Vavassori (UniBG)Slide2

OutlineThe contextCombinatorial models x highly configurable systemsModel M and System S (Problem and solution spaces)ConstraintsFaults in constraints in CIT modelsFault identification and repair processDiscovering failure inducing combinations by using BENRepairing the constraintsExperiments: Is it possible to

repair faulty models?Using mutation A real case study DjangoConclusions13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint RepairtestsSlide3

Configurable system S AssumptionsMost software systems can be configured to better address users’ needs. Modelling parameters and constraints among them bring great benefits

analysis, test generation, etc.Constraints model inconsistencies between certain hardware components, limitations of the possible system configurations, or design choicesWe assume parameters and their domains are correctly captured in the specification,while the constraints may contain some faults. 13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint RepairtestsCombinatorial model MParameters and constraints

Problem space

Solution spaceSlide4

Configurable system S Goal of the approach13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint Repairtests

Combinatorial model MConformance???

Model

WashingMachine

Parameters

:

Boolean

HalfLoad

;

Enumerative

Rinse

{Delicate

Drain

Wool

};

Enumerative

Spin

{ Low Mid High };

end

Constraints

:

#

HalfLoad

=> Spin!=

Spin.High

#

#

Rinse

==

Rinse.Delicate => HalfLoad #end

Rinse

Spin

HalfLoad

Assume: faults in the model and not in the system

Goal:

discovering and repairing faults in the constraints of the model

A

novel

automated

approach

for

finding

and

fixing

conformance

faults

between

the

given

software

system

and

its

combinatorial

model.Slide5

What is a fault?val(t) iff t is correct according to the model Mi.e. t satisfies the constraints

13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint RepairtestsConfigurable system S Combinatorial model MA configuration/test t

oracle

val{true

, false}

{true,

false}

correctness

iff

,

for every t,

We

say that the model contains a

conformance fault

if there exists a

such that

oracle(

t

)

iff

t

is correct according to the system Si.e., S correctly works with tAssume exists!May be expensiveSlide6

Two types of faultsif is true and is

falseA valid configuration is rejected by the actual systemif is true, but is false the model does not accept a test case that is allowed by the system 13 March 2017 - IWCT2017

Angelo Gargantini - CIT for Automated Constraint Repairtests

S

Configuration

spaceSlide7

Example of faultRinse

HalfLoadSpinDelicateTrueHigh13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint RepairtestsSystem Constraints

:# HalfLoad => Spin!=High ## Rinse

==Delicate =>

HalfLoad #

end

valtrue

oracle

false

The model M «

forgets

» the

constraint

that

if

the

HalfLoad

then

Spin

cannot be highModel Constraints:# Rinse==Delicate => HalfLoad #endSlide8

Partial tests/combinations Failure inducingA combination comb is an assignment of values to every parameter in a (non-empty) subset of

all the parameters . comb is failure-inducing if every test t containing comb (i.e., comb ⊆ t) fails, i.e. To find failure-inducing combinations we use BENLaleh Ghandehari, A Combinatorial testing based approach to fault localization, PhD thesis 2016

L. S. G. Ghandehari, Y. Lei, T. Xie, R. Kuhn and R. Kacker, "Identifying Failure-Inducing Combinations

in a Combinatorial Test Set,"ICST 2012Available as

jar (used as

external tool)

 13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint RepairtestsSlide9

Example of fault13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint Repairtests

System Constraints:# HalfLoad => Spin!=High ## Rinse==Delicate => HalfLoad #end

valtrue

oracle

false

The model M «

forgets

» the

constraint

that

if

HalfLoad

then

Spin

cannot

be high

Model

Constraints

:# Rinse==Delicate => HalfLoad #endRinseHalfLoadSpin-TrueHigh

Failure inducing combinationSlide10

Two types of failure inducing combinationsA failure inducing combination has type (w.r.t. the model)over-constraining

if for every assignment t containing , is false and is trueThe model must be relaxed in order to allow

under-constraining

if for every assignment t containing ,

= true and = false.The model must be strengthened in order to disallow

13 March 2017 - IWCT2017

Angelo Gargantini - CIT for Automated Constraint RepairtestsSlide11

ExampleRinseHalfLoad

Spin-TrueHigh13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint RepairtestsModel Constraints:#

Rinse==Delicate =>

HalfLoad #

end

val

trueoracle

false

This

faulty

combination

is

of

type

«under

constraining

»

true

in the model and false in the

systemIt must be excluded !Model Constraints:

# Rinse==Delicate => HalfLoad ## not (HalfLoad and Spin == High) #endSlide12

Constraints repair processWe can build a process to automatically repair models

13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint RepairtestsSlide13

Constraint repair process (external)13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint Repairtests

Apply CIT policyCIT ModelTest suiteSystem

forall tests

?

u

se BEN to generate (minimal) failure inducing combinations

Repair

the

constraints

exit

f

alse

: a

failure

found

true

 Slide14

Constraint repair process (internal)13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint RepairtestsApply CIT policy

Test suiteforall tests ? 

execute BEN with the current test suite

exit

f

alse

: a failure found

true

MFI

found?

true

false

new tests

produced by BEN

Increase

CIT

strength

false

true

BEN

cycle

Increase strength cycleforall new tests ? addSlide15

Experiments – 2 typesMutation analysiswe have taken some combinatorial models from the literatureadded some faultschecked if our process is able to repair the mutated modelIn this case the oracle is the availableM: original modelM’: mutated model – to be fixedM plays the role of the actual model, so M is the oracle

Django: configurable software system, in order to test our framework on a real case study.The oracle is the real system (it can be expensive to call)13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint RepairtestsSlide16

Mutation analysisbenchmarks13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint Repairtests

m

utations:

add a constraint that excludes a value of a parameter, remove an entire constraint,

negate a constraint, change a logical operator to another one (AND to OR etc.).Slide17

CIT policiesWe use several test generation policiesUC (Unconstrained CIT) generate the tests ignoring the constraintsCC (Constrained CIT) generates only tests satisfying the constraints. CV

(Constraints Violating CIT) produces only the tests violating constraints,CuCV (union CC and CV)ValC (CIT of Constraint Validity) requires the interaction of each parameter with the validity of the whole CIT model. 13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint RepairtestsParameters:…

Constraints:#

C1 ## C2

# ….

Parameters

:…Boolean validity;…Constraints:

# validity <=> C1 and C2

#

Gargantini

A.,

Petke

J.,

Radavelli

M.,

Vavassori

P. (2016)

Validation

of

Constraints

Among

Configuration Parameters Using Search-Based Combinatorial Interaction Testing. In Search Based Software Engineering. SSBSE 2016.Slide18

Comparison among policies13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint Repairtests

CC (Constrained

CIT) is rather cheap but it also

has the lowest repair capability

Some faults can be

found only by violating the

constraintsCuCV (CC + Constraints violation) is the best in terms of removed faultsbut is very expensive also in terms of

oracle calls

UC

(

ignoring

constraints

)

seems

not

so

badSlide19

Results in brief (mutation)How efficient is our constraint repair approach?the mutated models were repaired within seconds.How effective is our automated constraint repair approach at fixing faults in an existing CIT model?our approach can automatically fix on average 37% of conformance faults.Is it realistic to completely repair a model?For big models, very few mutations were completely repaired.Can our approach worsen the model?In theory yes, in practice it never happened.

13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint RepairtestsSlide20

2. Repairing DjangoDjango is a web application for websites. each Django project has a configuration file.we selected and modelled one Enumerative and 23 Boolean parameters. automated oracle: a configuration is correct if and only if the HTTP response code of the Django project homepage is 200 (HTTP OK).This oracle invocation is costly in terms of time (6 seconds)two starting modelsDjango0 without constraints – inferring constraints

Manual with constraints derived from the documentation – actual repair13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint RepairtestsSlide21

Results13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint Repairtests

Fi: how many tests (of the union of all the test suites) failInferring is costly. Best result 1h faults 8%

Reparing

is easier

than inferringBest results: 0

remaining faults UC is the worst

Every

policy

repairs

the

initial

modelSlide22

Django models13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint RepairtestsDjango0 # true

# Manual (constraints "inferred " manually)# ! PREPEND_WWW ## ! SECURE_SSL_REDIRECT #// if not DEBUG, "localhost" must be allowed # ! DEBUG => (ALLOWED_HOSTS==LOCALHOSTIP or ALLOWED_HOSTS==LOCALHOST) # CuCV (repairing Manual)# ! PREPEND_WWW ## ! SECURE_SSL_REDIRECT # # ! DEBUG => ALLOWED_HOSTS==LOCALHOSTIP or ALLOWED_HOSTS==LOCALHOST

## ALLOWED_HOSTS==IP => ! DEBUG or PREPEND_WWW or SECURE_SSL_REDIRECT #

ValC (repairing Manual)# ! PREPEND_WWW ## ! SECURE_SSL_REDIRECT # # ! DEBUG => ALLOWED_HOSTS==LOCALHOSTIP or ALLOWED_HOSTS==LOCALHOST #

# ALLOWED_HOSTS==IP => !DEBUG or SECURE_SSL_REDIRECT #Slide23

ConclusionsWe use CIT to repair constraintsOngoing effort to have a complete validation CIT-based processTo speed up fault localization and repair13 March 2017 - IWCT2017Angelo Gargantini - CIT for Automated Constraint Repairtests

Test generartion(several CIT policies)Fault localization (BEN)

Model repair(automated

process)

It

gives good results and it can be used in practice (see Django case study)