Zhen Huang David Lie Department of Electrical and Computer Engineering University of Toronto Outline Motivation Design Evaluation User Study Conclusion 2 Configuration errors Configuration errors are a leading cause of computer failures ID: 738224
Download Presentation The PPT/PDF document "Ocasta: Clustering Configuration Setting..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Ocasta: Clustering Configuration Settings for Error Recovery
Zhen Huang
, David Lie
Department of Electrical and Computer Engineering
University of TorontoSlide2
Outline
Motivation
Design
EvaluationUser StudyConclusion
2Slide3
Configuration errors
Configuration errors are a leading cause of computer failures.
Fixing a configuration error has two essential steps:
Identifying configuration settings that cause the error
Changing the faulty settings with values that fix the error
3Slide4
Configuration errors
Two categories of configuration errors:
Single-setting configuration errors
require changing one configuration setting to fix.
Multi-setting configuration errors
require changing more than one configuration settings,
related configuration settings
, to fix.
Example multi-setting configuration for Acrobat Reader4InlineAutoCompleteAutoCompleteOnTabSlide5
Configuration errors
Two categories of configuration errors:
Single-setting configuration errors
require changing one configuration setting to fix.
Multi-setting configuration errors
require changing more than one configuration settings,
related configuration settings
, to fix.
Example multi-setting configuration for Acrobat Reader5InlineAutoComplete: falseAutoCompleteOnTab : falseSlide6
Configuration errors
Two categories of configuration errors:
Single-setting configuration errors
require changing one configuration setting to fix.
Multi-setting configuration errors
require changing more than one configuration settings,
related configuration settings
, to fix.
Example multi-setting configuration for Acrobat Reader6InlineAutoComplete: trueAutoCompleteOnTab : falseSlide7
Configuration errors
Two categories of configuration errors:
Single-setting configuration errors
require changing one configuration setting to fix.
Multi-setting configuration errors
require changing more than one configuration settings,
related configuration settings
, to fix.
Example multi-setting configuration for Acrobat Reader7InlineAutoComplete: trueAutoCompleteOnTab : trueSlide8
Multi-setting configuration e
rrors
A significant number of configuration errors (14.9% - 34.7%) are multi-setting configuration errors [1].
However, previous techniques [2,3]
cannot
fix multi-setting configuration errors.
Ocasta
is designed to fix both single-setting and multi-setting configuration errors.1. “An empirical study on configuration errors in commercial and open source systems”, SOSP 20112. “
Automated Diagnosis of Software Configuration Errors”, ICSE 20133. “Automating configuration troubleshooting with dynamic information flow analysis,” OSDI 20108Slide9
Ocasta
Ocasta works in two modes:
Recording mode: record access to configuration settings
Recovery mode: fixing configuration errors with
rollbacks
access history of configuration settings
RECORDING MODE
Search for a fix
Identify related configuration settings
t
rial to execute
RECOVERY MODE
candidate fixes
9Slide10
Ocasta
Ocasta works in two modes:
Recording mode: record access to configuration settings
Recovery mode: fixing configuration errors with
rollbacks
access history of configuration settings
RECORDING MODE
Search for a fix
Identify related configuration settings
t
rial to execute
RECOVERY MODE
candidate fixes
10Slide11
Ocasta
Ocasta works in two modes:
Recording mode: record access to configuration settings
Recovery mode: fixing configuration errors with
rollbacks
access history of configuration settings
RECORDING MODE
Search for a fix
Identify related configuration settings
t
rial to execute
RECOVERY MODE
candidate fixes
11Slide12
Ocasta
Ocasta works in two modes:
Recording mode: record access to configuration settings
Recovery mode: fixing configuration errors with
rollbacks
access history of configuration settings
RECORDING MODE
Search
for a fixIdentify related configuration settings
t
rial to execute
RECOVERY MODE
candidate fixes
12Slide13
Ocasta
Ocasta works in two modes:
Recording mode: record access to configuration settings
Recovery mode: fixing configuration errors with
rollbacks
access history of configuration settings
RECORDING MODE
Search for a fix
Identify related configuration settings
t
rial to execute
RECOVERY MODE
candidate fixes
13Slide14
Recording access to configuration settings
Ocasta
records application’s access to configuration settings into a database.
Provides input for identifying related configuration settings
Supplies historical values of configuration settings for rollback recovery
Ocasta
treats
application as black-box and intercepts access to configuration settings.Windows registry API callsGConf API callsFile read and
write14Slide15
Identify related configuration settings
We observe that related configuration settings are
more
likely to be modified together than unrelated settings
.
InlineAutoComplete
AutoCompleteOnTab
MaxMRUCnt
UseCustomPaper
0
500
1000
1500
2000
Time (second)
Example access history for Acrobat Reader
15Slide16
Identify related configuration settings
We define a
correlation metric
between every pair of two settings to identify settings that are often modified together, but rarely modified individually.
Ocasta uses
hierarchical agglomerative
clustering algorithm to cluster related configuration settings using correlation metric.
16Modified TogetherTotal ModificationsSlide17
Automated Search
Execute trial
Take original screenshot
Rollback a cluster of related settings
Execute trial
Take screenshot
Select a cluster to rollback
17
Undo rollbackSlide18
Automated Search
18
Execute trial
Take original screenshot
Rollback a cluster of related settings
Execute trial
Take screenshot
Select a cluster to rollback
Undo rollbackSlide19
Automated Search
19
Execute trial
Take original screenshot
Rollback a cluster of related settings
Execute trial
Take screenshot
Select a cluster to rollback
Undo rollbackSlide20
Automated Search
20
Execute trial
Take original screenshot
Rollback a cluster of related settings
Execute trial
Take screenshot
Select a cluster to rollback
Undo rollback
Configuration ErrorSlide21
Automated Search
21
Execute trial
Take original screenshot
Rollback a cluster of related settings
Execute trial
Take screenshot
Select a cluster to rollback
Undo rollbackSlide22
Automated Search
1.
InlineAutoComplete
AutoCompleteOnTab
2.
MaxMRUCnt
3. UseCustomPaper
22Execute trialTake original screenshotRollback a cluster of related settingsExecute trialTake screenshot
Select a cluster to rollback
Undo rollbackSlide23
Automated Search
InlineAutoComplete
AutoCompleteOnTab
2013-09-03:
false, false
2013-10-12:
true, false
2013-12-02: false, false2014-02-25: true, true
23Execute trialTake original screenshotRollback a cluster of related settingsExecute trialTake screenshot
Select a cluster to rollback
Undo rollbackSlide24
Automated Search
24
Execute trial
Take original screenshot
Rollback a cluster of related settings
Execute trial
Take screenshot
Select a cluster to rollback
Undo rollbackSlide25
Automated Search
25
Execute trial
Take original screenshot
Rollback a cluster of related settings
Execute trial
Take screenshot
Select a cluster to rollback
Undo rollbackSlide26
Automated Search
26
Execute trial
Take original screenshot
Rollback a cluster of related settings
Execute trial
Take screenshot
Select a cluster to rollback
Undo rollbackSlide27
Automated Search
27
Execute trial
Take original screenshot
Rollback a cluster of related settings
Execute trial
Take screenshot
Select a cluster to rollback
Undo rollback
1.
InlineAutoComplete
AutoCompleteOnTab
2.
MaxMRUCnt
3.
UseCustomPaperSlide28
Recovery of configuration errors
Users view screenshots to select which fix to
apply.
28
Error
FixSlide29
Evaluation
We installed Ocasta on 24 Linux and 5 Windows desktop computers for several months.
We evaluated the accuracy of
Ocasta’s
clustering algorithm, the effectiveness and performance of
Ocasta’s
error recovery for 11 Linux and Windows desktop applications.
mail client: MS OUTLOOK, Evolutionweb browser: Internet Explorer, Chrome browserdocument editor/viewer: MS Word, GNOME Edit, Acrobat readerimage editor/viewer: MS Paint, GNOME Image Viewermedia player: Windows Media PlayerWindows shell: Explorer
29Slide30
Accuracy of clustering algorithm
Ocasta
identified 1,005 clusters for the 11 applications, involving 1,871 settings.
Ocasta has a 88.6% accuracy of identifying multi-setting clusters.
We manually examined all 255 multi-setting
clusters.
30Slide31
Configuration errors used in evaluation
We use 16 real-world configuration errors from literature, online forums, FAQ and configuration document.
Complex real-world applications: MS Outlook, MS Word, Internet Explorer, Chrome, Acrobat reader
Variety of
complex and simple errors:
List
of recently opened documents missing
User is unable to save any documentMenu bar missing for certain PDF documents
31Slide32
Performance of fixing configuration errors
The configuration errors were simulated by being injected into the recorded access history, 14 days before the end of the history.
Ocasta
successfully fixed all 16 errors.
Without clustering, 5 of them
cannot
be fixed.
For 14 errors, user only needs to examine less than 5 screenshots.All errors were fixed within 7 minutes. 32Slide33
User study
We performed a user study on 19 participants with various backgrounds to evaluate Ocasta.
Participants first used Ocasta to fix some configuration errors and then tried to manually fix them.
To limit the length of user study, we used 4 simple configuration errors among the 16 configuration errors.
33Slide34
User study
We use a scale of 1 to 5 indicating from very easy to very difficult.
74% of the participants rated
creating trials as very easy.
80% of the participants rated selecting the correct screenshots as very easy.
34Slide35
Conclusion
Ocasta accurately identifies 88.6% of clusters on average.
Ocasta fixes all 16 real-world configuration errors, 5 of them are multi-setting configuration errors.
Our user study shows that it is easy and efficient to use Ocasta to fix configuration errors.
35Slide36
Thank You!
contact:
z.huang@mail.utoronto.ca
demo:
http://
youtu.be/aRvJlTj-0F0
36