Andreea Haras 11092013 Agenda Introduction Mobile Testing vs Traditional WebBased Testing Challenges in Mobile Testing Guidelines amp Best Practices Conclusion QampA About me ID: 750651
Download Presentation The PPT/PDF document "Tap into Mobile App Testing" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1Slide2
Tap into Mobile App Testing
Andreea
Haras
11/09/2013Slide3
Agenda
Introduction
Mobile
Testing
vs. Traditional
Web-Based
Testing
Challenges in Mobile Testing
Guidelines & Best
Practices
Conclusion
Q&ASlide4
About me
Andreea Haras – andreea.haras@gmail.com
4+ years experience as software tester
Mobile testing enthusiastic
Interested in new technologySlide5
Why mobile
testing?Slide6
Why mobile testing?
“The mobile market is no longer just communication. It’s internet, music, video, web, social networks, navigation and everything else... The mobile ecosystem now embraces anything a consumer wishes to do either on the move or on a handheld device at home”
- ForresterSlide7
Why mobile testing?
“The mobile market is no longer just communication. It’s internet, music, video, web, social networks, navigation and everything else... The mobile ecosystem now embraces anything a consumer wishes to do either on the move or on a handheld device at home”
- Forrester
Applications running on mobile devices (e.g., mobile applications running on smart
phones
or new generation tablets) are becoming so popular that they are representing a
revolution
in the IT sector
.Slide8
Why mobile testing?
“The mobile market is no longer just communication. It’s internet, music, video, web, social networks, navigation and everything else... The mobile ecosystem now embraces anything a consumer wishes to do either on the move or on a handheld device at home”
- Forrester
Applications running on mobile devices (e.g., mobile applications running on smart
phones
or new generation tablets) are becoming so popular that they are representing a
revolution
in the IT sector
.
Over
1.6 million
mobile applications have been developed, available in Apple App Store and Google Play (
www.mobithinking.com
)Slide9
Why mobile testing?
“The mobile market is no longer just communication. It’s internet, music, video, web, social networks, navigation and everything else... The mobile ecosystem now embraces anything a consumer wishes to do either on the move or on a handheld device at home”
- Forrester
Applications running on mobile devices (e.g., mobile applications running on smart
phones
or new generation tablets) are becoming so popular that they are representing a
revolution
in the IT sector
.
Over
1.6 million
mobile applications have been developed, available in Apple App Store and Google Play (
www.mobithinking.com
)
Estimates for downloads of apps in 2013 range from
56 to 82 billion
(
www.idc.com
) and a prediction of
200
billion
global downloads in 2017 that will be worth
US$63.5
billion (
www.portioresearch.com
)Slide10
Native
, Traditional, Hybrid, Oh My!
Native
Apps
:
coded with a specific programming language. They are fast, reliable, and powerful, but are tied to a mobile platform and downloaded from a central portal by OS (i.e. Apple App Store, Google Play). That means you must duplicate them using the appropriate programming language in order to target another mobile platform.
Netflix, Pandora,
Shazam
.Slide11
Native
, Traditional, Hybrid, Oh My!
Native
Apps
:
coded with a specific programming language. They are fast, reliable, and powerful, but are tied to a mobile platform and downloaded from a central portal by OS (i.e. Apple App Store, Google Play). That means you must duplicate them using the appropriate programming language in order to target another mobile platform.
Netflix, Pandora,
Shazam
.
Traditional Mobile Web Apps
: mobile web sites designed to match every web-enabled phone. They’re accessed through the mobile device’s web browser and do not need to be downloaded/installed. They re-direct to the site’s M Site. Amazon, eBay, MSN are good examples
.Slide12
Native
, Traditional, Hybrid, Oh My!
Native
Apps
:
coded with a specific programming language. They are fast, reliable, and powerful, but are tied to a mobile platform and downloaded from a central portal by OS (i.e. Apple App Store, Google Play). That means you must duplicate them using the appropriate programming language in order to target another mobile platform.
Netflix, Pandora,
Shazam
.
Traditional Mobile Web Apps
: mobile web sites designed to match every web-enabled phone. They’re accessed through the mobile device’s web browser and do not need to be downloaded/installed. They re-direct to the site’s M Site. Amazon, eBay, MSN are good examples
.
Hybrid Apps
:
rely on development frameworks and are written with the same technology used for websites and mobile web
implementations. They
are hosted or run inside a native container on a mobile device. They ensure cross-platform compatibility and can access the phone’s hardware (camera, GPS, user’s contacts, etc.).
Foursquare, Twitter, Wikipedia and Facebook are good examples.Slide13
Pop Quiz
There are seven (yes, seven!) mobile operating systems that are recognized in the industry, other than the top 2 (
iOS
and Android), can you name one or more of the other five? Slide14Slide15
Can you name the differences between
Mobile Testing
and
Traditional
Web-Based
Testing?Slide16
Mobile
Testing vs. Traditional Web-Based Testing
Integration testing
Within
areas of the Device Hardware
Battery
, light, ringer, sleep, wake, usage, connectivity loss or change in speeds (“elevator test”), camera
Integration
with other Apps on the device
Facebook
, Twitter, etc. Slide17
Mobile
Testing vs. Traditional Web-Based Testing
Integration testing
Within
areas of the Device Hardware
Battery
, light, ringer, sleep, wake, usage, connectivity loss or change in speeds (“elevator test”), camera
Integration
with other Apps on the device
Facebook
, Twitter, etc.
Interruption Testing
How
the App behaves with user-controlled and
automatic interruptionsSlide18
Mobile
Testing vs. Traditional Web-Based Testing
Integration testing
Within
areas of the Device Hardware
Battery
, light, ringer, sleep, wake, usage, connectivity loss or change in speeds (“elevator test”), camera
Integration
with other Apps on the device
Facebook
, Twitter, etc.
Interruption Testing
How
the App behaves with user-controlled and
automatic interruptions
Device-specific
Gesture Testing
Pinching
, Zooming, Shaking, Orientation, Eye Scrolling, Eye Pausing, Tilting, Swiping, Press and Hold,
TappingSlide19Slide20
Pop quiz
What mobile testing challenges do you think there are?Slide21
Challenges in Mobile Testing
The time
frames
Variety of Devices & OS Versions
Display density
Different browsers
Network connections
Installation issuesSlide22
Challenges in Mobile Testing
The time
frames
Variety of Devices & OS Versions
Display density
Different browsers
Network connections
Installation issues
Simulating real life scenarios
Call interrupts
Low battery
Alarms
Network issuesSlide23
Usability
Accessibility
Intuitive
Mobile experience
More challenges
in Mobile TestingSlide24
Usability
Accessibility
Intuitive
Mobile experience
Automation
What to automate?
Tools
Effort
vs
Value
More challenges
in Mobile TestingSlide25
More challenges
in Mobile Testing
Usability
Accessibility
Intuitive
Mobile experience
Automation
What to automate?
Tools
Effort
vs
Value
Non functional testing
Performance
Security
Load testingSlide26
Guidelines & Best Practices
Determining
your
Mobile Test Matrix
Setting up your Lab: Phones
Setting
up your Lab: Tablets
Checklist
Key
types of Mobile QA Testing: Manual or Automated Testing
?
Things to remember
Test
Automation Tool Selection – Few factors to considerSlide27
The most difficult facet of the testing matrix is device fragmentation. There are nearly 4000 separate Android device models! How do you make sure that all users get a consistent experience?
Device manufacturer market share (Q1 2013)
Determining your
Mobile Test MatrixSlide28
Determining your Mobile Test Matrix
Android device fragmentation
Source: BGRSlide29
Determining your Mobile Test Matrix
Multiple approaches to build a “Mobile Test Matrix”
Group devices based on the hardware and/or software specification and prioritize them
User base – Build a test matrix based on the device and software version the user uses
Use statistics from vendor market place (especially useful for native apps like games, utilities, etc.)
Risk based and functionality based matrixSlide30
Determining your Mobile Test Matrix
Multiple approaches to build a “Mobile Test Matrix”
Group devices based on the hardware and/or software specification and prioritize them
User base – Build a test matrix based on the device and software version the user uses
Use statistics from vendor market place (especially useful for native apps like games, utilities, etc.)
Risk based and functionality based matrix
Example:
Group 1, priority C: Small devices with a small CPU, RAM and low resolution. Older software versions and older browsers.
Group 2, priority B: Mid-range devices with an avg. CPU, RAM (<512 MB), good screen size and resolution. The software is not the latest.
Group 3, priority A: High-end devices with a dual/quad-core CPU, RAM (>512 MB) and a high screen resolution. Latest software versions.Slide31
Setting up your Lab:
Phones
Newest, highest resolution,
high
density, normal sized phone running latest version of OS. (Samsung Galaxy S4, iPhone 5S)
Oldest
, smallest, low density small sized phone running oldest supported version of
OS. (iPhone 3, HTC
Tattoo)
Middle
of the range phone matches highest usage, normal size, high density, normal
resolution,
highest usage OS (Samsung Galaxy S2)
Phone
with Highest Market Share
(Samsung Galaxy S3, iPhone 4S)
Budget
smartphone - normal size, large screen,
mdpi
, old OS (Dell Venue
)Slide32
Setting up your Lab:
Tablets
XLarge
size high end tablet (10”), xhdpi, latest OS. (Nexus 10)
Large
size Tablet (5-7”), med density, legacy OS (Kindle Fire)
Phablet
(Phone/Tablet Hybrid) (5”)
xhdpi
, large size, latest OS, strange aspect ratio (LG Intuition, Samsung Note)
iOS
running
iOS
5 or lower (
iPad
2)
iOS
running
iOS
6 (
iPad
mini,
iPad
3)Slide33
ChecklistSlide34
Key types of Mobile QA Testing: Manual or Automated Testing?
Functional testing
Performance
testing
Memory Leakage Testing
Interrupt testing
Usability testing
Installation testing
Upgrade
testingSlide35
Things to remember
Start testing early. Use emulators / simulators
Use the app on different carriers and network connections like 3G, 4G,
WiFi
, or LTE
Combine testing tools and integrate them into a continuous integration system
When possible, have a group of BETA testers to help with real-world testing
Consider the problem space: go mobile to test mobile
Don’t assume that because it works on one device, it will work on allSlide36
Test Automation Tool Selection – Few factors to consider
Supported devices
Emulator
support
Availability
of integrated simulators
Types
of connection supported
Multi-device
support (this is important to run same test on multiple devices)
Compatibility
with existing automation tools
Security
- VPN connections, data sharing, etc.
Functionality
supported - swipe, drag and drop
,
zoom
,
scrolling
,
pinching
, rotating, etc
Image
/ text recognition
Debugging
and test reportingSlide37
Conclusions
Despite the challenges in mobile application testing, careful selection of target devices, using statistical techniques for test optimization and tools that maximize automation can ensure a cost effective mobile testing
process.Slide38
Conclusions
Despite the challenges in mobile application testing, careful selection of target devices, using statistical techniques for test optimization and tools that maximize automation can ensure a cost effective mobile testing process.
Given the stiff competition in the mobile apps market and variety of apps available for users to download, it is vital that you thoroughly test each and every aspect of the app before releasing it into the market, to ensure that the product meets the quality standard demanded by users. Slide39Slide40