Lenin Ravindranath Jitu Padhye Sharad Agarwal Ratul Mahajan Ian Obermiller Shahin Shayandeh Microsoft Research gt One Million Apps gt 300000 Developers Theres an App For That Too slow killing the usefulness when you really need to go ID: 750636
Download Presentation The PPT/PDF document "AppInsight: Mobile App Performance Moni..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
AppInsight: Mobile App Performance Monitoring in the Wild
Lenin Ravindranath, Jitu Padhye, Sharad Agarwal, Ratul Mahajan, Ian Obermiller, Shahin Shayandeh
Microsoft ResearchSlide2Slide3Slide4
> One Million Apps
> 300,000 DevelopersSlide5
There’s an App
For That
“… Too slow - killing the usefulness when you really need to go.”
But its slowSlide6
“Slower than a snail.”
“Slow
and
unresponsive like
mud”
“Consistently
3 seconds behind where I touch.”
“So slow
. Did an intern write this app??”
“Very
very slow compared to even browsing web.”“Sluggish and freezes my HTC
phone.”
“Loading
GPS data is
***
slow”Slide7
“Slower than a snail.”
“Slow
and
unresponsive like
mud”
“Consistently
3 seconds behind where I touch.”
“So slow
. Did an intern write this app??”
“Very
very slow compared to even browsing web.”“Sluggish and freezes my HTC
phone.”
“Loading
GPS data is
***
slow”
Performance problems are
inevitable in the wild
Hard to emulate in the lab
Diverse environmental conditions
Network connectivity, GPS signal quality, etc
Variety of hardware and OS versions
Wide range of user interactionsSlide8
What is the user-perceived delay?
Where is the bottleneck?
Monitor Performance in the Hands of Users
Significant barrier for most app developers
Little platform support
Only option is to instrument your app
Manage your own logging infrastructureSlide9
AppInsight
Automatic App Instrumentation
Zero developer effort
Binary Instrumentation
Readily deployable
No changes to the OS or runtimeLow overhead
What is the user-perceived delay?
Where is the bottleneck?Slide10
App
Instrumenter
App
Instrumented
App Store
Downloads
Analysis
Developer
Feedback
Server
Traces
DeveloperSlide11
App
Instrumenter
App
Instrumented
App Store
Downloads
Analysis
Developer
Feedback
Server
Traces
DeveloperSlide12
App
Instrumenter
App
Instrumented
Downloads
Analysis
Developer
Feedback
Server
Traces
Developer
App StoreSlide13
Instrumentation impacts app performanceThey are already slow enough!!!
Limited Resources
Network, Battery, Memory
App Instrumentation is ChallengingSlide14
Highly interactive, UI centricSingle UI thread that should not be blocked
Most tasks are performed asynchronouslyAsynchronous APIs for Network, Sensor, IO etc.Computation performed on background threads
App Instrumentation is Challenging
Highly Asynchronous Programming Pattern
Tracing
a
sync
c
ode is challengingSlide15
Internet
5
3%
“Mitt Romney might vote for Obama as well”
“We
recommend Mitt Romney for
president”
“I
would definitely trust
Mitt
Romney with my money.”
“If
I had a nickel for
every time
Mitt Romney said something
stupid
I'd be in his tax
bracket”
Mitt Romney
Mitt
Rating
Mitt
Rating
TweetsSlide16
ClickHandler
() {
tweets =
HttpGet(url); rating = ProcessTweets
(tweets); display.Text = rating;}
ProcessTweets(tweets
){ ...}
Click Handler
Start
Click Handler
End
LogStart();
LogEnd();
Thread
User Perceived Delay
53%
Mitt
Rating
Hypothetical Synchronous CodeSlide17
ClickHandler
()
{
AsyncHttpGet
(url, DownloadCallback);}
DownloadCallback(tweets
) { rating =
ProcessTweets(tweets); UIDispatch
(DisplayRating, rating);}
DisplayRating(rating
)
{
display.Text =
rating;
}
ProcessTweets
(
tweets
)
{
...
}
Display
Rating
UI Thread
Background Thread
UI Dispatch
ClickHandler
End
ClickHandler
Start
Mitt
Rating
System
Async Get Call
ProcessTweets
Download
Callback
53%
User Click
Asynchronous CodeSlide18
Mitt
Rating
53%
User Perceived Delay
System
UI Thread
Background Thread
Display
Rating
UI Dispatch
Async Get Call
Download
Callback
ClickHandler
Start
User Transaction
Transaction timeSlide19
UI Thread
Background Thread
Transaction time
User Transaction
Get Rating
53%Slide20
UI Thread
Background Thread
User Transaction
7 edges
Mitt
Rating
53%Slide21
Apps are highly asynchronous
Where is the bottleneck?
Focus development efforts
30 popular apps
167,000 transactions
On average,
19 asynchronous calls
per user transactionOn average,
8 parallel threads per user transactionUp to 7000 edgesSlide22
UI Thread
Background Thread
Background Thread
Background Thread
Twitter
Thread Wakeup
Thread Blocked
Fire
Fire
Display
User Transaction
Callback
Callback
Tweets
Posts
Twitter
Facebook
Process
Posts
Process
Tweets
Facebook
Mitt
Rating
4
7
%
User ClickSlide23
UI Thread
Background Thread
Background Thread
Background Thread
User Click
Twitter
Thread Wakeup
Thread Blocked
Fire
Fire
Callback
Callback
Twitter
Facebook
Process
Posts
Process
Tweets
Facebook
Critical Path
Optimizing the critical path reduces the user perceived delay
User Transaction
DisplaySlide24
AppInsight
What is the user-perceived delay?
Where is the bottleneck?
Automatically instruments the app to
track
user transactions and critical
pathSlide25
App
Instrumenter
App
Instrumented
Downloads
Analysis
Developer
Feedback
Server
Traces
Developer
App StoreSlide26
Low Overhead
User Transaction
UI Thread
Background Thread
Should not impact app performance
Limited Resources
Capturing User TransactionSlide27
User Transaction
UI Thread
Background ThreadSlide28
UI Thread
Background Thread
User Click
Event Handler
UI Manipulations
Capture
User TransactionSlide29
UI Manipulations
Capture
User Transaction
UI Thread
Background ThreadSlide30
UI Manipulations
Thread Execution
Capture
Click Handler
Start
Click Handler
End
Update UI
S
tart
End
Callback
Start
Callback End
User Transaction
User Transaction
UI Thread
Background ThreadSlide31
UI Manipulations
Thread Execution
Capture
User Transaction
UI Thread
Background ThreadSlide32
UI Manipulations
Thread Execution
Async Calls and Callbacks
Async Get Call
UI Dispatch Call
UI Dispatch
Callback
Capture
Download
Callback
User Transaction
UI Thread
Background ThreadSlide33
UI Manipulations
Thread Execution
Async Calls and Callbacks
Capture
User Transaction
UI Thread
Background ThreadSlide34
UI Manipulations
Thread Execution
Async Calls and Callbacks
UI Updates
Capture
User Transaction
UI Update
User Transaction
UI Thread
Background ThreadSlide35
UI Manipulations
Thread Execution
Async Calls and Callbacks
UI Updates
Capture
User Transaction
UI Thread
Background ThreadSlide36
UI Thread
Background Thread
Background Thread
Background Thread
User Transaction
UI Manipulations
Thread Execution
Async Calls and Callbacks
UI Updates
Thread Synchronization
Fire
Fire
Thread
Blocked
Thread
Wakeup
CaptureSlide37
UI Manipulations
Thread Execution
Async Calls and Callbacks
UI Updates
Thread Synchronization
CaptureSlide38
UI Manipulations
Thread Execution
Async Calls and Callbacks
UI Updates
Thread Synchronization
Capture
User Transaction
UI Thread
Background ThreadSlide39
UI Manipulations
Thread Execution
Async Calls and Callbacks
UI Updates
Thread Synchronization
Capture
Click Handler
Start
Click Handler
End
Update UI
S
tart
End
Callback
Start
Callback End
User Transaction
User Transaction
UI Thread
Background ThreadSlide40
Trace every methodProhibitive overheadEnough to log thread boundaries
Log entry and exit of UpcallsCapturing Thread Execution
Upcalls
System
App
Click Handler
Start
Click Handler
End
Update UI
S
tart
End
Callback
Start
Callback End
UI Thread
Background ThreadSlide41
ClickHandler
()
{
AsyncHttpGet
(url,
DownloadCallback);}
DownloadCallback(tweets)
{ rating =
ProcessTweets(tweets); UIDispatch
(DisplayRating, rating);}
DisplayRating
(
rating
)
{
display.Text =
rating;
}
ProcessTweets
(
tweets
){ ...
}Event Handlers are UpcallsF
unction pointers point to potential UpcallsCallbacks are passed as function pointers<Button Click
=“ClickHandler” />
Identify UpcallsSlide42
Instrument Upcalls
Rewrite app
Trace Upcalls
ClickHandler
()
{
Logger.UpcallStart(1);
AsyncHttpGet(url,
DownloadCallback
);
Logger.UpcallEnd(1);
}
DownloadCallback
(
response
)
{
Logger.UpcallStart(2);
rating
= ProcessTweets(tweets); UIDispatch(
DisplayRating, rating); Logger.UpcallEnd(2);}
DisplayRating(rating)
{ Logger.UpcallStart(3);
display.Text = rating;
Logger.UpcallEnd(3);}
ProcessTweets(tweets)
{ ...}
Low Overhead
7000 times less overheadClick Handler
Start
Click Handler End
Update UI
Start
End
Callback
Start
Callback End
UI Thread
Background ThreadSlide43
UI Manipulations
Thread Execution
Async Calls and Callbacks
UI Updates
Thread
Synchronization
Capture
Async Get Call
UI Dispatch Call
UI Dispatch
Callback
Download
Callback
User Transaction
UI Thread
Background ThreadSlide44
Async Get Call
Callback
Async Calls and Callbacks
Log Callback Start
We capture start of the thread
Log Async Call
Any call that accepts a function pointer
Match Async Call to its Callback
ClickHandler
()
{
AsyncHttpGet
(url
,
DownloadCallback
);
Logger.AsyncCall(5);
}
DownloadCallback
(
tweets
)
{
Logger.UpcallStart(2); ....}Slide45
Matching Async Call to its Callback
A method could be a callback to many async callsReplicate the callback for each Async call [Hirzel’01]
Async call called in a loop (e.g. download list of URLs)
Cannot be matched correctly
Solution: Detour CallbacksSlide46
System
App
AsyncHttpGet
Async Call
DownloadCallback
DownloadCallback
(
response
){}
CallbacksSlide47
Detour Callbacks
DetourCallback
(
response
)
{
DownloadCallback(response);}
class
DetourObject {
}
MatchId = 3
obj.DetourCallback
MatchId = 3
MatchId = 3
AsyncHttpGet
(url,
GetCallback
);
AsyncHttpGet
(url,
obj.DetourCallback
);
o
bj = new DetourObject(DownloadCallback, MatchId++);
System
App
AsyncHttpGet
Async Call
DownloadCallback
DownloadCallback
(
response
)
{}
while(...)
{}Slide48
Detour Callbacks
DetourCallback
(
response
)
{
DownloadCallback(response);}
class
DetourObject {
}
MatchId = 4
obj.DetourCallback
MatchId = 4
MatchId = 4
o
bj = new
DetourObject(DownloadCallback,
MatchId++);
System
App
AsyncHttpGet
Async Call
DownloadCallback
(
response
)
{
}
AsyncHttpGet
(url,
obj.DetourCallback
);
w
hile(...) {
}Slide49
Detour Callbacks
DetourCallback
(
response
)
{
DownloadCallback(response);}
class
DetourObject {
}
MatchId = 5
obj.DetourCallback
MatchId =
5
MatchId =
5
o
bj = new
DetourObject(DownloadCallback,
MatchId++);
System
App
AsyncHttpGet
Async Call
DownloadCallback
(
response
)
{
}
AsyncHttpGet
(url,
obj.DetourCallback);
while(...) {
}Slide50
Detour CallbacksEvent Subscriptions
Delayed CallbacksTrack Object IdsMatching Async Call to its Callback
Low Overhead
Async Get Call
UI Dispatch Call
UI Dispatch
Callback
Download
Callback
UI Thread
Background ThreadSlide51
UI Manipulations
Thread Execution
Async Calls and Callbacks
UI Updates
Thread Synchronization
Capture
User Transaction
UI Thread
Background ThreadSlide52
App
Instrumenter
App
Instrumented
App Marketplace
Downloads
Analysis
Developer
Feedback
Server
Traces
DeveloperSlide53
App
Instrumenter
App
Instrumented
App Marketplace
Downloads
Analysis
Developer
Feedback
Server
Traces
DeveloperSlide54
Critical PathAggregate AnalysisException Path
AnalysisSlide55
Critical Path
Bottleneck path in the user transactionOptimizing the critical path reduces user perceived delay
UI Update
User Manipulation
Critical Path
Multiple UI updates
Timers, SensorsSlide56
Critical PathAggregate AnalysisException Path
AnalysisSlide57
Aggregate Analysis
Group similar transactionsSame transaction graph
Outliers
Points to corner cases
Highlight common critical paths
Focus development effort
Root causes of Performance Variability
Highlight what really matters in the wildSlide58
Critical PathAggregate AnalysisException
PathAnalysisSlide59
UI Thread
Background Thread
Background Thread
Exception
a
t
parseXML()
DownloadCallback()
Stack trace
parseXML()
…
…
DownloadCallback()
AsyncHttpGet(url)
ThreadCallback()
ClickHandler()
ThreadStart()
User
Manipulation
Exception Path
Failures in the wildSlide60
Critical PathAggregate AnalysisException Path
AnalysisSlide61
App
Instrumenter
App
Instrumented
App Store
Downloads
Analysis
Developer
Feedback
Server
Traces
DeveloperSlide62
Developer Feedback
Web based Interface
Long User
Transactions
Critical
Path
Aggregate
Analysis
Outliers
Common
Case
Factors affecting
Crashes - Exception PathSlide63
App
Instrumenter
App
Instrumented
App Store
Downloads
Analysis
Developer
Feedback
Server
Traces
Developer
DeploymentSlide64
30 Windows Phone apps 30 UsersOver 4 months of data
6750 app sessions33,000 minutes in apps167,000 user transactionsDeploymentSlide65
User Transactions and Critical Path
Apps are highly asynchronous
# edges in the critical path << # edges in the transaction
Top 2 edges responsible for 82% of transaction time
App developer can focus on optimizing these edges
15% of the user transactions take more than
5 seconds
Key reason why a system like AppInsight is needed
Up to 7000 edgesSlide66
High variability in the wild
Aggregate Analysis
29% transaction groups has multiple critical paths
Network, GPS state, Device model, User
a
ll affects performance
Key reason why
performance monitoring
in the wild
is importantSlide67
AppInsight Overhead
Compute
Network
Memory
Binary Size
Battery
<1%
1.2%
2%
4%
0.02%
Negligible Overhead
Impact on app performance is minimal
Our users reported no cases of performance degradation
Low resource consumptionSlide68
App
Instrumenter
App
Instrumented
App Store
Downloads
Analysis
Developer
Feedback
Server
Traces
DeveloperSlide69
App
Instrumenter
App
Instrumented
App Store
Downloads
Analysis
Developer
Feedback
Server
Traces
Developer
ImprovedSlide70
App: My App
Problem: UI HogAppInsight
:
Aggregate
analysis showed high performance variabilityAttributed to UI thread
Abnormal latencies only at the start of the sessionSystem loading DLLs in the critical path
Developer Case StudySlide71
App: Popular App
Problem: Slow TransactionsAppInsight
:
Aggregate
analysis showed 3G latencies significantly affecting critical pathsKnown issue but
no quantitative data Current caching policy insufficientQuantitative data
Developer Case StudySlide72
Developer Case Study
App: Professional AppProblem:
Slow Transactions
AppInsight
:
Custom instrumentation in the critical pathAffecting user perceived delaySlide73
AppInsight
Extremely low overhead
Zero
developer effort
Readily deployable
Automatically tracks User TransactionsIdentifies Critical Paths and
Exception PathsAggregate Analysis highlights factors affecting performance
Helps app developers understand performance bottlenecks in the wildSlide74
Who cares about Windows Phone. Can you do it for Android and iPhone?
This was done in 1982. Don’t you read related work?Are you one of those who don’t care about user privacy?
Sounds
too good. Are there any limitations
?
Are you going to release it? Or is it a one off paper?I totally don’t get why you are doing this!?
FAQSlide75
AppInsight can miss certain causal relationshipsDoes not track data dependencies
Two threads communicating through a shared variableOne thread pollingUses heuristics to warn the developerWill miss implicit casual relationshipsResource contentionDoes not track that state left behind
Dependencies resulting from such state
Limitations