AB Testing Overview AB testing is a user experience research methodology. AB tests consist of a randomized experiment with two variants, A and B. It includes application of statistical hypothesis testing or two-sample hypothesis
"A/B Testing Overview A/B testing is a user" is the property of its rightful owner. Permission is granted to
download and print the materials on this website for personal, non-commercial use only, and to display it
on your personal computer provided you do not modify the materials and that you retain all copyright
notices contained in the materials. By downloading content from our website, you accept the terms of this
agreement.
Presentation Transcript
01
A/B Testing Overview A/B testing is a user experience research methodology. A/B tests consist of a randomized experiment with two variants, A and B. It includes application of statistical hypothesis testing or “two-sample hypothesis testing” as used in the field of statistics.<br>
02
Product Roadmap via A/B Testing Market Needs + Customer Expectations + Business Goals 01 02 03 04 05<br>
03
A/B Testing Process CONTINUOUS
IMPROVEMENT 01 02 03 04 05<br>
04
A/B Testing Process BEFORE OPTIMIZATION WITH OPTIMIZATION AFTER OPTIMIZATION Original -1% Variation 1-4.5% $1,000 in Sales $4,500 in Sales<br>
05
A/B Testing Process 50% visitors
see variation A 30% Conversion rate 50% visitors
see variation B 16% Conversion rate<br>
06
A/B Testing Campaign CAMPAIGN Ad 01 Ad 01 Ad 02 Ad 03 Audience 01 Audience 02 Audience 03 Audience 04 Campaign with a single objective. For example, website conversations. Four separate Ad sets targeting two different audience. One Ad per Ad set. Notice there are only 2 different ads. Each Ad is displayed in its own Ad set to each audience.<br>
07
A/B Testing Decision Tree<br>
08
A/B Test Example +1.5 Landing Page +0.8 -4.0% +6.0% A/B Testing Improvements To Statistically Significant Confidence 6 8 4.5 2<br>
09
Landing Page A/B Test SUM OF NEWSLETTER SIGN UPS SUM OF AMOUNT<br>
A/B Test Dashboard EDIT WIDGET - SAVE WIDGET EDIT WIDGET - 6 MONTH LTV EDIT WIDGET IN TRIAL - PURCHASE EDIT WIDGET - SAVE WIDGET EDIT WIDGET - 6 MONTH LTV EDIT WIDGET IN TRIAL - PURCHASE AB Test Feature Y Funnels AB Test Feature X Funnels<br>
12
A/B Test Analytics Average session
length 1% Views Content occurrences/session 28% Sessions per daily user 7% Lifetime value per daily user 60% Subscribing occurrence per day 24% Time in subscribing per daily user 24% Sign In occurrences per daily user 32% Subscribed occurrences per daily user 9% Daily Active Users<br>
13
A/B Test Results A/B Testing Line Graph Total visitors Average conversion rate Original Variation 01<br>
14
A/B Test Results Sum of Newsletter Sign Ups Chance of B Outperforming A<br>
