A Controlled Experiment on Android Applications Which Factor Impacts GUI TraversalBased Test Case Generation Technique Most Bo Jiang amp Yaoyue Zhang Beihang University WK ID: 792063
Download The PPT/PDF document "QRS 2017 Beihang University, China" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
QRS
2017
Beihang University, China
— A
Controlled Experiment on Android Applications
Which Factor Impacts GUI Traversal-Based Test Case Generation Technique Most?
Bo
Jiang &
Yaoyue
Zhang
,
Beihang
University
W.K
.
Chan,
City
Univeristy of Hong KongZhenyu Zhang, Chinese Academy of Sciences, Institute of Software
Slide2Outline
1
BACKGROUND & RELATED WORK
3
5
DESIGN
FACTORS
CONCLUSION
2
TEST CASE GENERATION FRAMEWORK
4
CONTROLLED EXPERIMENT
Slide3Outline
3
5
DESIGN
FACTORS
CONCLUSION
1
BACKGROUND & RELATED WORK
2
TEST CASE GENERATION FRAMEWORK
4
CONTROLLED EXPERIMENT
Slide4Background
1
3
2
Android Applications requires testing for ensure quality
Automated test case generation techniques are major research focus.
StateTraversal
: one of the most popular type of techniques
.
4
Slide5Related Work
1
3
2
Automated Test Input Generation for Android: Are We There Yet?
R. C.
Shauvik
et al. (ASE2015)
PUMA
: Programmable UI-automation for large-scale dynamic analysis of mobile apps.
S.
Hao
, B. Liu et al.
(
MobiSys2014) SwiftHand
: Guided GUI Testing of Android Apps with Minimal Restart and Approximate Learning. W.
Choi et al. (OOPSLA2013)
4
Acteve: Automated Concolic Testing of Smartphone Apps.S. Anand et al. (FSE2012)5
Slide6Outline
3
5
DESIGN
FACTORS
CONCLUSION
4
CONTROLLED EXPERIMENT
2
TEST CASE GENERATION FRAMEWORK
1
BACKGROUND & RELATED WORK
Slide7Test Case Generation Framework
PUMA Framework
7
Slide8Test Case Generation Framework
Generic Framework
Generic GUI
Traversal
-
base
d
Test Case generation framework of
PUMA
01
02
03
State
equivalence
Search strategy
Waiting time
8
Slide9Outline
1
BACKGROUND & RELATED WORK
5
CONCLUSION
2
TEST CASE GENERATION FRAMEWORK
4
CONTROLLED EXPERIMENT
3
DESIGN
FACTORS
Slide10Design Factors
State Equivalence
01
03
02
01
Cosine
Eigenvectors
of the UI widgets
.
Threshold:
0.95
DECAF&PUMA
ActivityID
UIAutomator's API
getCurrentActivityName
().
S
tring comparison. A3EUI HierarchyUse Widgets tree structure to represent . The widgets trees are the same.
SwiftHand
10
Slide11Search S
trategy
02
Rand
monkey
BFS
PUMA
DFS
A3E
Design Factors
11
Slide12Waiting Time (between Two Events)
0
3
Design Factors
11
Strategy
Used
By
watiForIdle
PUMA
wait200ms
Monkey
used
by
Shauvik
et
al in ASE 2015
wait3000msActeve
wait5000msSwifthand
Slide13Factor Level
Factor 1:
State Equivalence
Factor 2:
Search Strategy
Factor 3:
Waiting Time0
Cosine
BFSwaitForIdle
1
UI Hierarchy
DFS
wait200ms
2
ActivityID
Randomwait3000ms3
—
—wait5000msThree Factors and Their LevlesDesign Factors12
Slide14Outline
1
BACKGROUND & RELATED WORK
3
5
DESIGN
FACTORS
CONCLUSION
2
TEST CASE GENERATION FRAMEWORK
4
CONTROLLED EXPERIMENT
Slide15Controlled Experiment
Benchmarks and
Experimental
Setup
01
33 real-world open-source mobile apps
from Dynodroid, A3E, ACTEve,
SwiftHand.
We
implemented all
factor levels
in
the PUMA
framework.
Two virtual machines installed with ubuntu 14.04 operating systems.
14
Slide16Experimental Procedure
02
36 (i.e., 3*3*4) combinations of factor levels for the three
factors.
Took
1188 testing hours in total on
2
virtual machines
.
ANOVAs(one-way
ANalyses
Of
VAriances
)
Multiple
comparison
Controlled Experiment
15
Slide17Results
and
Analysis-state equivalence
03
Controlled Experiment
16
Slide18Results
and
Analysis-state equivalence
03
Controlled Experiment
17
Slide19Results
and
Analysis-state equivalence
03
Cosine Similarity > UI Hierarchy >
ActivityID
in
Failure
detection
ability
C
ode
coverage rate
Controlled Experiment
18
Slide20Results
and Analysis-search strategy
03
Controlled Experiment
19
Slide21Results
and Analysis-search strategy
03
Controlled Experiment
20
Slide22Results
and Analysis-search strategy
03
Randomized
search
strategy
was statistically comparable
to BFS and DFS in
Failure
detection
ability
C
ode
coverage rate
Controlled Experiment
21
Slide23Results
and
Analysis-waiting time
03
Controlled Experiment
22
Slide24Results
and
Analysis-waiting time
03
Controlled Experiment
23
Slide25Results
and
Analysis-waiting time
03
The strategy to
wait until GUI state is stable
before sending the next input event is not statistically more effective than the strategy of
waiting for a fixed time
interval
in
Failure
detection ability
Code coverage rate
Controlled Experiment
25
Slide26Results
and Analysis-Best
Treatment-Failure Detection Rate
03
Controlled Experiment
25
Slide27Results
and Analysis-Best
Treatment-Code Coverage
03
Controlled Experiment
26
Slide28Results
and Analysis-Best
Treatment
03
There
were many combinations of factor levels can attain the same high level of failure detection rate and high level of statement code
coverage
There
could be many good configurations in configuring
StateTraversal.
Controlled Experiment
27
Slide29Outline
1
BACKGROUND & RELATED WORK
3
DESIGN
FACTORS
2
TEST CASE GENERATION FRAMEWORK
4
CONTROLLED EXPERIMENT
5
CONCLUSION
Slide30Conclusion
State
equivalence
:Different state equivalence definitionswill
significantly affect the
failure detection rates and the code coverage.Search strategy: BFS and
DFS are comparable to Random.Waiting time: Waiting for idle and waiting for a fixed time period have no
significant difference.Failure detection rate: <Cosine Similarity, BFS, wait5000ms
>(best).Code coverage:<Cosine Similarity, DFS, wait5000ms>(best).
29
Slide31THANKS
Q&A