App flighting and beta testing - PowerPoint Presentation

Slide1

App flighting and beta testingin the Windows Store

Jonathan Garrigues & Beth Anne Katz

Program ManagersSlide2

App Flighting Goals

The Windows Store allows developers to

do controlled

testing in production…

…to confirm the quality of updates before deploying them to all customers.

…to minimize the impact of bugs that only reveal themselves at scale.

…to learn about the effectiveness of new experiences from a limited set of customers, before these experiences are ready to ship to everyone.Slide3

New tools for testing in production

Available now

Package flighting

Deliver

different packages to different people who already own the app.

Manage groups of known customers

Specify people by Microsoft account email addresses.

Coming soon

Targeted

app availability

Restrict who can see and acquire your app in the Store.

Gradual rollout

Control how quickly package updates are delivered.

Let

users

opt-in to

app beta groups

Invite

customers

to

opt-in

to receive your beta quality experiences.Slide4

Everyone else gets the non-flighted packages

Package flights

Different

packages

delivered only

to group members

New concept of customer groups

K

nown

customers specified by Microsoft account email address

Customers who have

opted-in to an

app beta group

Customer groups

Who can see /

acquire the app?

Everyone can see / Everyone can acquire

Only those with direct link can see / Anyone with direct link can acquire

Targeted app availabilityOnly group members with a direct link can see / acquire the app

Only those with direct link can see / Only those w/ promo code can acquire

Which packages do they get?Slide5

Package flighting

Sample scenario

You

already

have an

app in

the Windows Store, with several million customers.You want to validate that your latest bug fix works end to end with a limited set of known people who already have the app installed.Once you have confidence in the fix, you’ll make the update available to everyone.Slide6

Flight packages to multiple groups

simultaneously

Package flights contain only binary packages.

Only group members get these updates.

Your app or game has to be published before you can create a package flight.

Non-flighted submission

Packages v1.0

Store listings

App distribution settings

(

available to everyone)

Group of 100 co-workers

Package flight C

Packages v1.1

Group of 15 testers

Group of 1500 enthusiasts

Package flight B

Packages v1.2

Package flight A

Packages v1.3Slide7

Demo

Creating and publishing a package flightSlide8

Supported client versions

Package

flighting works

on

Windows 10 Desktop

Version 10586+ (November)

Windows 10 Mobile Version 10586.63+ (February)For earlier clientsUsers not eligible to receive package flights continue to get updates from the non-flighted submission.Slide9

Package flighting customer experience

Updates to flights happen automatically

Unless your app tells them, customers don’t know that an update is from a package flight.

Make sure you maintain the quality level in your flights your customers expect.

Clients check for updates automatically once per day.

Customers expecting flights (such as testers) can manually check for updates in the Store to get a new flight immediately.

Get consent before flighting “beta” quality packages

If you are offering customers a way to sign up to receive a “

V.Next

beta” experience by providing their Microsoft account email addresses, make sure they understand the quality level of the experience you will be sending them.Slide10

Customer feedback channels

Windows Feedback App

Your app can invite your customers to provide feedback using the Windows Feedback App.

Customers can to see and up-vote feedback for the same version they are running.

Learn

more in the video

App

Feedback: Connect with your customers Ratings and reviewsRatings and reviews for packages in a flight are hidden from other customers.

All ratings and reviews are visible in Dev Center.If a package becomes available to everyone, ratings and reviews for that package will be visible to everyone.Slide11

Adjust audience based

on

analytics

Per-version telemetry

You

can make decisions about

increasing the audience for specific packages based on real world performance data.Each package in your flights has a specific version.Dev Center health report allows you to filter crashes, hangs, etc. by packageversion.Slide12

Demo

Moving

packages

between flightsSlide13

Coming soonSlide14

In the Store, gradual rollout in progress

View gradual package rollout telemetry

Coming soon

Gradual rollout

Control

the

speed of

any

package update by percentage

of the target audience.

No need to identify specific users.Halt the rollout if you find an issue.Independently and simultaneously rollout flights and non-flights .Slide15

Coming soon

Targeted

app availability

Restrict who can see and acquire your app or game in the Store.

Only people who are group members can see and acquire the app or game, using a direct link.

Controlled as part of your non-flighted submission.

Colleagues

Enthusiasts

Group of 100 Co-workers

Test TeamSlide16

Coming soon

Let

users opt-in

to app

beta groups

Describe how you will use the app beta

group and invite your customers to join.Customers can directly opt-in / opt-out.

Enables

V.First

beta scenarios when used with targeted app availability

(app not public yet)Enables V.Next beta scenarios when used with package flighting

(app is public, beta is for next version).Slide17

Summary

Try

out in Dev Center Today

:

Package

flighting

Manage groups of known customersFeatures coming soon:Targeted app availabilityGradual rolloutLet users opt-in to app beta groupsThank you.