Online Software Testing Using Transactional Memory Jayaram Bobba Weiwei Xiong Luke Yen Mark D Hill and David A Wood Multifacet Project wwwcswiscedumultifacet ID: 361412
Download Presentation The PPT/PDF document "StealthTest: Low Overhead" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
StealthTest: Low Overhead Online Software Testing Using Transactional Memory
Jayaram Bobba, Weiwei Xiong*, Luke Yen†,Mark D. Hill, and David A. Wood
Multifacet Project (www.cs.wisc.edu/multifacet)Dept. of Computer SciencesUniversity of Wisconsin-Madison
*Dept. of Computer SciencesUniversity of Illinois at Urbana-Champaign† Currently at Advanced Micro DevicesSlide2
Executive Summary (1/2)The Problem
Software testing hardMultithreading makes harderOnline software testing can helpRun tests on deployed softwareE.g., Delta Execution for patch testing[Tucek et al., ASPLOS 2009]Non-intrusive mechanismsfork (existing)
Functionally
Hidden
LowOverhead
GoodScaling
fork
© 2009 Multifacet Project University of Wisconsin-Madison
2Slide3
Executive Summary (2/2)StealthTest
Leverage Transactional Memory for online testingNon-Intrusive?transaction { test(); abort}Fast TM mechanisms© 2009 Multifacet Project University of Wisconsin-MadisonDemonstrate two uses
Delta ExecutionIn vivo Testing
Functionally
Hidden
Low
Overhead
Good
Scaling
StealthTest
3Slide4
OutlineOnline Software TestingE.g., Patch ValidationStealthTest: TM for online testing
Delta Execution using StealthTestIn vivo Testing using StealthTest© 2009 Multifacet Project University of Wisconsin-Madison4Slide5
Online Patch Validation
Bug fixes can introduce more bugsPatches must be validatedOnline Validation [Nagaraja et al., OSDI 2004]Increased resource usageLockstep execution© 2009 Multifacet Project University of Wisconsin-Madison
Testing
Production
Input
Output
Diff
5Slide6
Delta Execution[Tucek et al., ASPLOS 2009]
Online Patch Validation Most patches are small Patched and Un-patched executions similarDelta ExecutionRun together except when they differ© 2009 Multifacet Project University of Wisconsin-Madison
Prior WorkDelta ExecutionIncreased Resource UsageO
PLockstep ExecutionOP
6Slide7
Delta Execution using fork
© 2009 Multifacet Project University of Wisconsin-Madison
Production
Testing
fork
Install
D
data
Unpatched
execution
Patched
execution
Compute
D
data
Isolate
D
data
Merged
execution
Time
7Slide8
Multi-threading and
fork© 2009 Multifacet Project University of Wisconsin-Madison
Production
Testing
fork
Install
D
data
Unpatched
execution
Patched
execution
Compute
D
data
Isolate
D
data
Merged
execution
Time
8
Stop all threads to get a consistent memory snapshot
‘Park’ all other threadsSlide9
fork
Poor Performance ~9.8ms for split/~106ms for merge [Tucek et al, ASPLOS 2009]Poor ScalabilityWeb-server response rate reduced by 43%Want an alternate mechanism© 2009 Multifacet Project University of Wisconsin-Madison
9Slide10
OutlineOnline Software TestingE.g., Patch ValidationStealthTest: TM for online testing
Delta Execution using StealthTestIn vivo Testing using StealthTest© 2009 Multifacet Project University of Wisconsin-Madison10Slide11
Delta Execution using StealthTest© 2009 Multifacet Project University of Wisconsin-Madison
Isolatepatched executionIntrospectpatched executionMonitordelta data access
Delta ExecutionTransactionalMemory
transaction{…}Version ManagementTracks new/old values
Conflict DetectionMonitor accesses
StealthTest
11
fork
Execute on child process
Page
diffing
mprotectSlide12
StealthTest Interface© 2009 Multifacet Project University of Wisconsin-Madison
Isolatepatched executionIntrospectpatched executionMonitordelta data access
Delta Executiontransaction{…}
Version ManagementTracks new/old valuesConflict DetectionMonitor accesses
StealthTest
ST_begin_transaction
ST_abort_transaction
ST_get_old
ST_get_new
ST_protect_set
ST_protect_clear
12
Transactional
MemorySlide13
Requirements from TMStrong Atomicity [Martin et al., CAL 2006]Transactions isolated from non-transactions
=> Test transactions isolated from application codeFlexible Conflict ResolutionCan abort transactions if necessary=> Abort tests if they block application Communication from within transactions=> Expose result of a test© 2009 Multifacet Project University of Wisconsin-Madison13Slide14
OutlineOnline Software TestingE.g., Patch ValidationStealthTest: TM for online testing
Delta Execution using StealthTestIn vivo Testing using StealthTest© 2009 Multifacet Project University of Wisconsin-Madison14Slide15
Delta Execution
using StealthTest© 2009 Multifacet Project University of Wisconsin-Madison
Production
Testing
fork
Install
D
data
Unpatched
execution
Patched
execution
Compute and Isolate
D
data
Merged
execution
fork
StealthTest
Production
Install
D
data
Unpatched
execution
Patched
execution
Compute and Isolate
D
data
Merged
execution
ST_abort_tran
…
ST_begin_tran
…
ST_protect_set
transaction
15
ST_get_new
ST_get_old
Introspect and
rollbackSlide16
Multi-threaded
Delta Execution© 2009 Multifacet Project University of Wisconsin-Madison
Production
Testing
fork
Install
D
data
Unpatched
execution
Patched
execution
Compute and Isolate
D
data
Merged
execution
fork
StealthTest
Original
Install
D
data
Unpatched
execution
Patched
execution
Compute and Isolate
D
data
Merged
execution
ST_abort_tran
…
ST_begin_tran
…
ST_protect_set
transaction
16
ST_get_new
ST_get_old
Introspect and
rollbackSlide17
Evaluation(1) Effective? (2) Non-intrusive?Workloads
Collection of multi-threaded server appsSame as Tucek et al., ASPLOS 2009Pin-based TM Emulation2-way SMP with 2.4GHz Pentium 4 CPUs and 2.5GB RAM© 2009 Multifacet Project University of Wisconsin-Madison17Slide18
(1) Effective?
ProgramDescriptionPatchDescriptionPatch Verified?forkStealthTestCraftyChess App
Code refactoringPPRaytraceRaytracer
Result reporting fixPPTarArchive UtilIncremental archiving fix
PP
Apache1
Web Server
Buffer overflow fix
P
P
Apache2
Web Server
Buffer overflow fix
P
P
DNSCache
DNS Cache
Behavior Change
P
P
MySQL5.0
DB Server
Extra permission checks
P
O
OpenSSL
Security Lib
Added bug in TLS handling
P
O
Squid
Web CacheBuffer overflow fixP
O
ATPhttpd
Web Server
Buffer
overflow fix
P
O
© 2009 Multifacet Project University of Wisconsin-Madison
Memory allocation
Works
18
socketsSlide19
(2) Non-intrusive?© 2009 Multifacet Project University of Wisconsin-Madison
ProgramDescriptionforkForkOverhead(%)PatchDuration(%)CraftyChess App
0.1<0.1RaytraceRaytracer0.20.5
TarArchive Util417.3Apache1Web Server
2.80.1
Apache2Web Server
12
0.1
DNSCache
DNS Cache
65
0.1
MySQL5.0
DB Server
4.7
5.0
OpenSSL
Security Lib
12
<0.1
Squid
Web Cache
2.9
0.2
ATPhttpd
Web Server
65
0.8
19Slide20
OutlineOnline Software TestingE.g., Patch ValidationStealthTest: TM for online testing
Delta Execution using StealthTestIn vivo Testing using StealthTest© 2009 Multifacet Project University of Wisconsin-Madison20Slide21
In vivo Testing[Murphy et al. TR 2007, Chu et al. ICST 2008]
Run unit tests on deployed software+ More testing+ More realistic Catch bugs early© 2009 Multifacet Project University of Wisconsin-Madison21Slide22
In vivo Testingusing StealthTest
© 2009 Multifacet Project University of Wisconsin-MadisonST_begin_transaction();try { test(); ST_begin_escape();
fprintf(log, “…”, success); ST_end_escape();} catch/except() {
ST_begin_escape(); fprintf(log, “…”, fail); ST_end_escape();
}ST_abort_transaction
(NO_RETRY);
22Slide23
EvaluationWorkloadsBugbench
Server WorkloadsSTAMPTransactional Memory benchmarksImplementationIntel STMLanguage-Based TMTL2 STMLibrary-Based TMQuad-core workstation with RHEL5© 2009 Multifacet Project University of Wisconsin-Madison23Slide24
(1) Effective?© 2009 Multifacet Project University of Wisconsin-Madison
ProgramDescriptionSize (LOC)Bug TypeErrorDetected?NCOMfile compress1.9KStack SmashYes
POLYfile “unixier”0.7KStack SmashYesGZIP
file compress8.2KBuffer OverflowYesMANdocumentation4.7KBuffer OverflowYes
BCcalculator17.0K
Buffer Overflow
Yes
HTPD1
web server
224K
Atomicity
Yes
SQUD
proxy
cache
93.5K
Buffer Overflow
Possible
CVS
version control
114.5K
Double
Free
Possible
MSQL2
DBMS
514K
AtomicityPossible
MSQL3DBMS1028K
AtomicityPossible
Unsupported Library Calls
Works
Built on Intel STM.
Run tests on
Bugbench
applications
24Slide25
(2) Non-intrusive? © 2009 Multifacet Project University of Wisconsin-Madison
Built on TL2 STM. Run tests on STAMP applications (1000 tests per min)25Slide26
Conclusions and Future WorkStealthTest can provide non-intrusive online testing
Synergistic benefit for TM and testingFutureMore testing frameworksBand-aid patch testing [Sidiroglou et al., 2007]Multi-threaded tests© 2009 Multifacet Project University of Wisconsin-Madison26Slide27
Executive SummarySoftware testing hardOnline software testing can help
Existing mechanisms inadequateStealthTest leverages TM for non-intrusive online testingDemonstrate two usesDelta Execution In vivo Testing© 2009 Multifacet Project University of Wisconsin-Madison
Functionally
HiddenLowOverheadGood
Scaling
StealthTest
27Slide28
© 2009 Multifacet Project University of Wisconsin-Madison
28Slide29
© 2009 Multifacet Project University of Wisconsin-Madison
29Slide30
Atomicity Violation Bugs?
© 2009 Multifacet Project University of Wisconsin-Madison30Slide31
Degree-2 TransactionsIsolate only writes.Implementation
Reads in escape actionEarly ReleaseAdd new type of transaction to TM© 2009 Multifacet Project University of Wisconsin-Madison31Slide32
StealthTest Wish ListHardware SupportSystem Calls within TransactionsInteraction between Locks and Transactions
© 2009 Multifacet Project University of Wisconsin-Madison32Slide33
© 2009 Multifacet Project University of Wisconsin-Madison
33Slide34
Related WorkBinary Translation (SPROCKETS)Code Emulation (STEM)TLS (
Oplinger&LAM, PathExpander)© 2009 Multifacet Project University of Wisconsin-Madison34Slide35
In vivo Testing MotivationOrdering Bug in MySQL
In vivo Test: Data Consistency Checks© 2009 Multifacet Project University of Wisconsin-Madison35Buggy Code (In mysys/thr_lock.c):void thr_lock_delete(THR_LOCK *lock) { … pthread_mutex_destroy
(&lock->mutex); … list_delete(thr_lock_thread_list, &lock->list); …}