damn time and what to do about it by Peep Laja ConversionXL Call to Action Conference 2014 You should test that Dude wtf peeplaja Typical scenario A company runs 100 tests year ID: 808771
Download The PPT/PDF document "15 A/B testing mistakes I see all the" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Slide215 A/B testing mistakes I see all the
damn
time
(and what to do about it)
by Peep Laja, ConversionXL
Call to Action Conference 2014
Slide3“You should test that!”
Slide4Slide5“Dude wtf?”
Slide6@peeplaja
Typical scenario:
A company runs 100 tests / year.
After 12 months their conversion rate is still the same.
Slide7@peeplaja
Testing mistake #1:
Precious time wasted on stupid tests
Slide8@peeplaja
Testing mistake #2:
You don’t have a data-driven, learning-oriented hypothesis
Slide9@peeplaja
Testing mistake #3:
You think you know what will work
Slide10@peeplaja
Testing mistake #4:
You run tests when you have no traffic
Slide11@peeplaja
Testing mistake #5:
Your absolute conversions are way too low
Slide12Slide13@peeplaja
You need *at least*
250-400 conversions PER variation.
Way more if you want to segment the data
Slide14@peeplaja
Our "thumb rule" is: 3000-4000 conversions per variation and 3-4 week test duration. That is enough traffic so we can even talk about valid data if we drill down into segments.
Testing "sin" nr.1: search in segments for uplifts although you have no statistical validity, e.g. 85 versus 97 conversions. That’s bullshit.
Slide15@peeplaja
Testing mistake #6:
Your tests don’t run long enough
Slide16Slide17@peeplaja
Testing mistake #7:
You don’t test full weeks at a time
Slide18Slide19@peeplaja
Testing mistake #8:
Test data is not sent to third-party analytics
Slide20Slide21@peeplaja
Pro tip:
If you have over 500,000 pageviews / mo, you don’t (necessarily) need to spend $100,000 on GA Premium. Get unsampled data with
Analytics Canvas
Slide22@peeplaja
Testing mistake #9:
You give up after your first test for a hypothesis fails.
Slide23Slide24Full case study:
http://conversionxl.com/case-study-how-we-improved-landing-page-conversion/
79.3% uplift in conversions
Slide25@peeplaja
Testing mistake #11:
You’re not aware of validity threats
Slide26@peeplaja
History effect
Instrumentation effect
Selection effect
Slide27@peeplaja
Testing mistake #12:
You’re ignoring small gains
Slide28@peeplaja
5% monthly increase in your conversion rate will result in 80% uplift for the year.
That’s how math works.
Slide29@peeplaja
Testing mistake #13:
You don’t know html/css/jquery - and thus only test the simple stuff
Slide30@peeplaja
Testing mistake #14:
You’re not running tests at all times
Slide31@peeplaja
Testing mistake #15:
You’re late for your lunch break!
Slide32Follow me @peeplaja
@peeplaja
Keep reading ConversionXL!
Thank you
Slide33