Erik Andersen PhD student Center for Game Science University of Washington Computer Science Dept Music Gameplay Sound effects Art engagement retention revenue AB Testing AB Testing ID: 273032
Download Presentation The PPT/PDF document "Game Optimization through Large-Scale Ex..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Game Optimization through Large-Scale Experimentation
Erik Andersen
Ph.D. student
Center for Game Science
University of Washington Computer Science Dept.Slide2
Music
Gameplay
Sound effects
ArtSlide3
engagement
retentionrevenue
?Slide4
A/B TestingSlide5
A/B Testing
A
BSlide6
Metrics
Levels completed Time played
Return rateSlide7
Aesthetics
Secondary Objectives
TutorialsSlide8
Foldit
Refraction
Hello WorldsSlide9
RefractionSlide10
Rating: 3.8 / 5
400,000 playsSlide11
Hello Worlds!Slide12
Rating: 4.2 / 5
1,385,000 playsSlide13
FolditSlide14
240,000 playersSlide15
Player Tracking
Flash cache / login nameNew players onlySlide16
Statistical significance
95% confidence level (p<0.05)Slide17
Experiment #1: Audio
Sound Effects
MusicSlide18
Result:
Music and sound effects did not
matterSlide19
Experiment #2: AnimationsSlide20
Result:
Animations improved engagementSlide21
No Animations
RefractionSlide22
No Animations
RefractionSlide23
No Animations
RefractionSlide24
No Animations
Hello WorldsSlide25
No Animations
Hello WorldsSlide26
Experiment #3: Secondary Objectives
(Super Mario Bros., Nintendo 1985)Slide27Slide28Slide29
Levels
% of
players
100
0Slide30
Levels
% of
players
100
0
With Coins
HypothesisSlide31
Levels
% of
players
100
0
With Coins
RealitySlide32Slide33Slide34
Time
% of
players
100
0
What about time played?Slide35
Time
% of
players
100
0
With Coins
HypothesisSlide36
Time
% of
players
100
0
Hypothesis
With CoinsSlide37
Time
% of
players
100
0
With Coins
Reality: Many Players Quit SoonerSlide38
Time
% of
players
100
0Slide39Slide40Slide41
Result:
secondary objectives harmed
engagementSlide42
(Assassin’s Creed,
Ubisoft
2007)Slide43
Maybe easier is better?
(Super Mario Bros., Nintendo 1985)Slide44Slide45Slide46Slide47Slide48
Time
% of
players
100
0
Off-path CoinsSlide49
Time
% of
players
100
0
On-path CoinsSlide50Slide51Slide52
Result:
secondary objectives were good if they supported the primary objectivesSlide53
Experiment #4: TutorialsSlide54
No Animations
FolditSlide55
No Animations
FolditSlide56
Refraction and Hello Worlds:
no effectSlide57
Result:
text tutorials helped
only in the most complex gameSlide58
Number of concepts
11
13
24Slide59
Context-sensitivity
Context-sensitive
Context-insensitiveSlide60
Result:
context-sensitive help was better
(if tutorials helped at all)Slide61
No Animations
FolditSlide62
No Animations
FolditSlide63
Let’s gather more data
Let us A/B test your games!Slide64
Coauthors
Yun-En Liu
Richard SniderRoy SzetoSeth CooperEleanor O’Rourke
Jeff LowdermilkDavid TruongZoran
PopovićSlide65
Acknowledgements
Kongregate: August
Brown and Anthony PecorellaLogging:
Kefan XuRefraction Team: Erik Andersen, Yun-En Liu, Marianne Lee, Eric Butler, Brian
Britigan, Stephen Sievers, Roy Szeto, Mai Dang, Christian
Lee, Ethan Apter, Emma Lynch, Happy Dong, Zorah Lea Fung, Justin Irwen, Seth Cooper, François Boucher-
Genesse, Zoran PopovićHello Worlds Team
: Rich Snider, Michael Eng, Marianne Lee, Blake Thompson, Jeff Flatten
Foldit Team: Seth Cooper, Adrien Treuille, Firas
Khatib
, Janos
Barbaros
, Joshua Snyder, Alex Cho Snyder, Jeff Flatten, Jeff
Lowdermilk
, Dun-Yu Hsiao,
Jeehyung
Lee, David
Salesin
, David Baker,
Zoran
PopovićSlide66
e
land@cs.washington.edu