Jian Xu Qingqing Cao Aditya Prakash Aruna Balasubramanian Donald E Porter Lab Wearable devices are becoming a major growth ID: 1003795
Download Presentation The PPT/PDF document "UIWear : Easily Adapting User Interfac..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. UIWear: Easily Adapting User Interfaces for Wearable DevicesJian Xu, Qingqing Cao, Aditya Prakash, Aruna Balasubramanian, Donald E. PorterLab
2. Wearable devices are becoming a major growth sector for computing…Introductionhttps://www.techrepublic.com/article/smartwatch-sales-to-hit-17-8-billion-in-2020-but-business-use-lagging/
3. Wearable use cases
4. A Dominant model for wearable apps: Companion appSpotify appCompanionImageView => CardImageButton => Card ImageViewTextView => Card TextView1. Companion app mirrors and retailors smartphone UI
5. A Dominant model for wearable apps: Companion appSpotify appCompanion1. Companion app mirrors and retailors smartphone UI2. User action on watch relayed to phoneUser actionUI updates
6. Survey: Top 100 apps in AndroidWear App CategoryPercentageExamplesMirrors UI and performs no local computation78%Spotify, WeatherStandalone10%Google Translate, TalkbackCollects local data6%Fit, WeChatPerforms some local computation6%Outlook, Google MusicCompanion Apps
7. Lots of popular apps do not have companions… Problem: Writing companion apps is tediousLines of Code
8. Deconstructing Spotify companion appTotal > 10000 LoC (excluding libraries)Data Structure< 1000 LoCNetwork> 4000 LoCCreate UI widgets> 4000 LoCTakeaway:Code entangles UI design with UI managementMost development effort on app-agnostic tasks
9. UIWear Goal:Easily build companion apps with minimal developer effort
10. Decouple UI design from UI managementUIWear Key IdeaDeveloper writes simple Metaprogram (considerably lower effort)UIWear creates companionapp, performs UI management
11. UIWear ArchitectureMetaprogram: UI Design specified by a humanCompiler: Compiles metaprogram to UICompanion* appUIWear runtime: Relays UI events from wearable to phone and performs UI updatesCommon thread : Leverage UI abstraction(1) Extract UI model, (2) Retailor UI model , and (3) Sync at the UI layerSimilar to developer-written app, but generated by UIWear
12. MetaProgramUIWearCompilerUIAbstractionUIWear Compiler and Metaprogram UICompanion AppUIRetailoringPhone app
13. Challenge:How to abstract the UI from off-the-shelf apps?
14. The app’s UI is made up of several UI elementsPossible Solution: Extracting UI from hardware buffer
15. Extracting UI from hardware buffer is too late, since semantics are LOST! UI elements cannot be retailoredPossible Solution: Extracting UI from hardware bufferRDP protocol
16. UIWear leverages UI trees UI Tree is the Intermediate Representation of UIUI trees can be extracted using Accessibility API
17. The accessibility API does not extract image data from UI17Limitations of accessibility APIUI Tree extractedby accessibility APIUIWear augments existing accessibility API to extract images on-demand.
18. UIWear Meta Program specificationWhich UI elements are mapped to wearableHow UI elements are retailored and placed
19. Meta program: UI placement and retailoring<wearable.view.CardScrollView> id="@+id/backgroundID" layout_width="match_parent" layout_height="wrap_content" layout_box="bottom"> <wearable.view.CardFrame> <TextView id="@+id/titleID" layout_width="match_parent" layout_height="wrap_content"/> <ImageView id="@+id/iconID" layout_width="30dp" layout_height="30dp”/> <TextView id="@+id/albumID" layout_width="match_parent" layout_height="wrap_content" /> </wearable.view.CardFrame></wearable.view.CardScrollView>Meta Program for Spotify
20. Compiling UICompanion apps from metaprogramGenerated by parsingMeta ProgramUIWear Stub
21. UIWear Runtime: Sync at UI layerUIWearWearableProxyUIWear Phone ProxyPollUIChangeUI UpdateUser ActionUser InputThrough AccessibilityPhoneWearable Device
22. Challenge:Phone OSes destroy the UI (and the UI Tree) when application is in background
23. UIWear idea: Destroy graphics stack but keep UI tree (no negative effect on power or performance)
24. UIWear behavior when app moves to backgroundApp (moved to background)UI Tree nodeGraphics stackStopAccessibility ServiceStop rendering
25. UIWearUIWear behavior when app moves to backgroundApp (moved to background)UI Tree nodeGraphics stackNon-stop Accessibility ServiceStop renderingCached UI tree
26. ImplementationAndroid OS, AndroidWear, and Sony Smartglasses20 UICompanion apps Metaprogram mimics developer-written companion appPicked from top 150 apps with different categories and popularities
27. Generated UICompanion apps
28. Evaluations
29. Comparing Lines-of-CodeApp (downloads) Companion LoCMeta prog. LoC WorkoutTrainer (10M)226536Anghami (10M)2,263 79 BandLab (500M)1,802 8Spotify(100M)10,21547Endomondo (10M) 6,522 18 Ghostracer (10K) 5,324 95 HydroCoach (1M) 4,022 12 iHeartRadio (50M) 7,926 96 LoC of meta program covers all activitiesUICompanion apps require an order-of-magnitude fewer developer-written LoC
30. Energy Consumption on the watchUICompanion apps have comparable performance in terms of energy consumption on phone, CPU, latency
31. User studiesMetricUICompanion Better or EqualAndroidWearBetterFunctionality78%22%Look and feel74%26%On average, each subject created a working meta program <= 16 minutes
32. Other EvaluationsKeeping the UI Tree active does not affect power or performanceUIWear can be used to createApps on smart glassesThird-party companion apps
33. ConclusionsContributes a new design point that decouples the application logic and UI designProvides a much easier way to generate a UICompanion app with few LoCThe UICompanion app has comparable performance and powerhttp://sbuwear.cs.stonybrook.edu/https://github.com/SBUNetSys/UIWear