collaborative dynamic personalized experimentation Joseph Jay Williams Harvard Office of the Vice Provost for Advances in Learning National University of Singapore Im originally from the Caribbean Trinidad and Tobago ID: 803491
Download The PPT/PDF document "Perpetually enhancing human learning thr..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Perpetually enhancing human learning through collaborative, dynamic, personalized experimentation
Joseph Jay WilliamsHarvard (Office of the Vice Provost for Advances in Learning) National University of Singapore[I’m originally from the Caribbean, Trinidad and Tobago]
Slide2Digital Resources are Increasingly Used in EducationMOOCs Massive Open Online Courses
On-Campus CoursesK12 Online homeworkLearning from Mathematics Problems(Stigler et al 2010, Heffernan et al, 2016)
Slide3Resources for Learning Need ImprovementLack of measurable learning (Steenbergen-Hu & Cooper, 2013; 2014)Problem: Can’t predict
what works in real-worldVision: Systems that perpetually improve – like real teachersx = matrix(rnorm(m*n),m,n)What is the standard error of this random variable?
Explanation
A z-score is defined as the number of standard deviations a specific point is away from the mean.
0.078
Explanation 2
You should imagine that people’s ranking depends on how well they do relative to how everyone else is doing.
Explanation 3
Think of it by analogy to comparing teams in the NBA’s Western and Eastern conferences.
Slide4Novel Opportunities for Experimentation
ResearchPracticeResearchers’ Lab
Students’ Class
Slide5Approach: Making Experiments Collaborative, Dynamic, Personalized AB
Dynamic AnalysisAX%
100-X%
B
0%
100%
Enhancement
A
B
0%
100%
Personalization
100%
0%
A
B
Outcome Metric
Education
Cognitive Science
Online Education
Bayesian Statistics
& Machine Learning
NIPS 2008, UAI 2013
ACIC 2016
Cognitive Science 2010
J. of Exp. Psych., 2013
EDM
2015
IJAIED
2016
CHI 2016,
ACM LAS 2016
Crowdsourcing
&
Human Computation
MOOClet
github.com
/
kunanit
/
mooclet
-engine
+
N
...
Continually add conditions
50%
50%
Slide6Overview+ N
...Dynamic AnalysisAX%
100-X%
B
0%
100%
Personalization
100%
0%
A
B
A
B
A
B
Enhancement
Vision: Perpetually Improving Systems
Approach:
Collaborative, Dynamic
,
Personalized Experimentation
Motivation & Reflection
Adaptive
eXplanation
Improvement
System (AXIS)
Discovering how to personalize
Future
Bridging Teachers,
Social-Behavioral Scientists, & Machine
Learning
Apps for Student Goal
Setting & Study Strategies
Slide7Overview+ N
...Dynamic AnalysisAX%
100-X%
B
0%
100%
Personalization
100%
0%
A
B
A
B
A
B
Enhancement
Motivation & Reflection
Adaptive
eXplanation
Improvement
System (AXIS)
Discovering how to personalize
Future
Slide8Experiments on Math Problems: Motivation & Reflection
x = matrix(rnorm(m*n),m,n)What is the standard error?
Remember, the more you practice the smarter you become!
Answer:
x = matrix(
rnorm
(m*n),
m,n
)
What is the standard error?
Answer:
Explain why this answer is correct.
Motivational Messages
Prompts to Reflect
Online educational experiments:
Bridging psychological research &
real-world learning
Slide9Growth Mindset MessageEncouraging Growth Mindset about IntelligenceSome of these problems are hard. Do your best!
Remember, the more you practice the smarter you become!N = 200 000 learnersDweck, 2007Practice-as-usual Encouraging Message
Slide10Effects of Messages?Growth Mindset > Practice-as-Usual and Encouraging Message1% increase in Number problems attempted (p < 0.05)Encouraging Messages not better than Practice-as-Usual (p > 0.3)“You can learn anything” campaign: 11 million learners
Competition inviting researchers to do similar experiments
Slide11Experiments on Problems: Motivation & Reflection
x = matrix(rnorm(m*n),m,n)What is the standard error?
Remember, the more you practice the smarter you become!
Answer:
x = matrix(
rnorm
(m*n),
m,n
)
What is the standard error?
Answer:
Explain why this answer is correct.
Motivational Messages
Prompts to Reflect
Slide12Reflection: Help Students Help Themselves
x = matrix(rnorm(m*n),m,n)What is the standard error?
Answer:
Explain why this answer is correct.
Prompts to Reflect
Prompts to explain “why?” help people find patterns
Cognitive Science, 2010
JEP: General, 2013
Slide13Benefits of Question Prompts to explain “Why?”Psychology, Education, Philosophy, ML & AILombrozo, 2012; Aleven
& Koedinger, 2002; Chi et al, 1994; McNamara, 2004; Wellman, 2012; Woodward, 2013; DeJong & Mooney, 1986General benefits: Engagement (e.g. Siegler, 2002)Selective effects: A Subsumptive
Constraints
account
(
Williams
&
Lombrozo
, 2010,
Cognitive Science
)
Explanation for “why?” interprets fact as instance of pattern
Explaining engages search for underlying generalizations
Slide14Explanation & LearningDiscovery of patterns(Williams & Lombrozo, 2010, Cognitive Science
)Overgeneralize at expense of specific facts (Williams et al, 2013, Journal of Experimental Psychology: General)Use of prior knowledge(Williams & Lombrozo, 2013, Cognitive Psychology)Children’s causal learning (Walker, Williams, Lombrozo, Gopnik, 2012)Comparison of examples (Edwards, Williams, Lombrozo, Gentner, 2013)Contradictions & belief revision (Williams et al 2016, Computer-Human Interaction)
Slide15Explaining Anomalies – Contradictions to Prior BeliefsAnomalies often ignored (Chinn & Brewer, 1993)Prompts to explain:No effectReduce belief revision b/c favors contradictory prior knowledge (Williams & Lombrozo, 2013; Williams, Lombrozo,
Rehder, 2013)Promote belief revisionUnifying generalizations (Williams & Lombrozo, 2010)15
Slide16✖Relative Rank using standard deviationLearn to rank using z-scores/standard deviation (Schwartz & Martin, 2004; Belenky & Nokes, 2011)
16
Sarah
got 85% in a Sociology class, where the average score was 79%, the average deviation was 8
%
, the minimum score was 67%, and the maximum score was 90%.
Tom got 69% in a Art History class, where the average score was 65%, the average deviation was 3
%
, the minimum score was 42%, and the maximum score was 87%
.
Tom was ranked higher.
Ranking
Rule
Use
of rule
Higher
ranked
Higher score
85 > 69
Sarah
Greater distance from average
(85 – 79) > (69 – 65)
Sarah
Closer to maximum
(90 – 85) < (87 – 69)
Sarah
More
deviations above the average
(85-79)/8 < (69-65)/3
Tom
Personal Score
Class Average
Class Maximum
Class Deviation
Sarah
85%
79%
90%
8%
Tom
69%
65%
87%
3%
✖
✖
✔
Slide17Design17Pre-test
Study ItemsPost-test~6 ranked student pairs
Few Anomalies
Many Anomalies
Explain
Write thoughts
Overlapping
Distributed
Sarah
got 85% in a Sociology class, where the average score was 79%, the average deviation was
3%
, the minimum score was 67%, and the maximum score was 90%.
Tom got 69% in a Art History class, where the average score was 65%, the average deviation was
8%
, the minimum score was 42%, and the maximum score was 87%
.
Slide18Overlapping vs. Distributed
2 out of 6 anomalies conditionOverlapping condition
Distributed
condition
1
2
3
4
5
6
1
2
3
4
5
6
Ranking Rule
Higher score
✖
✖
✔
✔
✔
✔
✖
✖
✔
✔
✔
✔
Greater distance from average
✖
✖
✔
✔
✔
✔
✔
✔
✖
✖
✔
✔
Closer to maximum
✖
✖
✔
✔
✔
✔
✔
✔
✔
✔
✖
✖
More deviations above
average
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
✔
18
Slide19Explaining x Anomalous Information19
Few Anomalies
Slide20Contributions: Increasing Motivation & Reflection
x = matrix(rnorm(m*n),m,n)What is the standard error?
Remember, the more you practice the smarter you become!
Answer:
x = matrix(
rnorm
(m*n),
m,n
)
What is the standard error?
Answer:
x = matrix(
rnorm
(m*n),
m,n
)
What is the standard error?
Answer:
Explanation
A z-score is defined as the number of standard deviations a specific point is away from the mean.
Explain why this answer is correct.
Motivational Messages
Prompts to Reflect
A
B
+
N
...
Continually add conditions
Analyze & Dynamically Adapt
A
X%
100-X%
B
Slide21Overview+ N
...Dynamic AnalysisAX%
100-X%
B
0%
100%
Personalization
100%
0%
A
B
A
B
A
B
Enhancement
Motivation & Reflection
Adaptive
eXplanation
Improvement
System (AXIS)
Discovering how to personalize
Future
Slide22AXIS: Adaptive eXplanation Improvement SystemA
B+ N...Continually add conditions
Analyze & Dynamically Adapt
A
X%
100-X%
B
x = matrix(
rnorm
(m*n),
m,n
)
What is the standard error?
Answer:
Explanation
A z-score is defined as the number of standard deviations a specific point is away from the mean.
ACM Learning @ Scale 2016
Explain why this answer is correct.
CHI 2016
(Nominee for best paper)
Renkl
, 1997
Slide23CExplanationLearners Rate & Generate Explanations
Linda is training for a marathon, which is a race that is 26 miles long.Her average training time for the 26 miles is 208 minutes,but the day of the marathon she was x minutes faster than her average time.What was Linda's running speed for the marathon in miles per minute?Explanation
Linda's speed is the distance she ran divided by the time it took. The distance Linda ran was 26 miles. The time it took her was 208 – x. Linda's speed was 26/(208 - x)
26/(208 - x)
To help you learn, explain in your own words why the answer is correct.
How helpful was the above information for your learning?
Completely Perfectly
Unhelpful Helpful
0 1 2 3 4 5 6 7 8 9 10
Explanation
A
B
Explanation
C
Explanation
Slide24ParametersAction aDynamic Experimentation: Exploration vs Exploitation
Multi-Armed Bandit (Reinforcement Learning)Randomized Probability Matching (Thompson Sampling)AReward R
Policy
Explanation
The probability is 3/7 * 5/8, because the number of cookies is changing.
Rating
How helpful was the above information for your learning?
0 1 2 3 4 5 6 7 8 9 10
Exp1
Exp
2
Exp
3
15%
65%
20%
(Probability of Explanation being Rated Helpful)
(0 to 10 Rating by Student)
Slide25AXIS deployed with n=150
12345678910
11
1
2
3
4
5
6
7
8
9
10
11
0
0
100
0
0
0
0
0
0
0
0
AXIS Deployment
1
2
3
4
5
6
7
8
9
10
11
18
13
4
5
8
18
22
6
3
1
2
AXIS Policy: Probability distribution over explanations
1
2
50
50
1
2
20
80
1
2
3
10
60
30
1
100
Slide26Do AXIS explanations help learning?Evaluation of AXIS explanationsProblem
x = matrix(rnorm(m*n),m,n)What is the standard error?
Answer:
Problem
x = matrix(
rnorm
(m*n),
m,n
)
What is the standard error?
Answer:
AXIS Explanation
Problem
x = matrix(
rnorm
(m*n),
m,n
)
What is the standard error?
Answer:
Filtered Explanation
Problem
x = matrix(
rnorm
(m*n),
m,n
)
What is the standard error?
Answer:
Instructor Explanation
Slide27Impact of AXIS Explanations on LearningInstructor reported the AXIS explanations comparable to their own
Slide28ContributionsStudents help peers in course of learningDynamic experimentation put data into practiceLimitations & FutureInvolve teachersBroader applications to goal-setting and motivation
Slide29Overview+ N
...Dynamic AnalysisAX%
100-X%
B
0%
100%
Personalization
100%
0%
A
B
A
B
A
B
Enhancement
Motivation & Reflection
Adaptive
eXplanation
Improvement
System (AXIS)
Discovering how to personalize
Future
Slide30Discover how to personalize emailsEmails
Slide31Compare introductory messageQuestion about course participationWould you please take this short survey, so we can improve the course for future students?Click here to take the survey.
Dear Sam,It has been a while since you logged into the course, so we are eager to learn about your experience. Would you please take this short survey, so we can improve the course for future students? BriefMention Absence
Slide32Optimization through Personalization0%100%
Personalization100%0%
A
B
14.5% more responses
Slide33Overview+ N
...Dynamic AnalysisAX%
100-X%
B
0%
100%
Personalization
100%
0%
A
B
A
B
A
B
Enhancement
Motivation & Reflection
Adaptive
eXplanation
Improvement
System (AXIS)
Discovering how to personalize
Future
Slide34Collaborative, Dynamic Experimentation
Social-Behavioral ScientistsTeachersAB
+
N
...
C
On-Campus
Courses
Dynamic Analysis
A
X%
100-X%
B
A
B
50%
50%
Enhancement
A
B
0
%
10
0
%
Atlantic Causal Inference Conference,
2016; CSCW,
under review
Slide35App for Experimentation on Problems in Canvastiny.cc/cdesite is a website for using the app
Slide36Co-Design of Explanations, Hints, Learning Tips
Slide37Version
Probability of ConditionMean Student Rating
Number of Students
Standard Deviation of Rating
Standard Error of the Mean
Instructor Rating
Instructor Confidence
1. Quantitative Explanation
0.23
7.26
46.00
1.87
0.28
7/10
2/5
2. Analogical Explanation
0.77
7.48
56.00
1.59
0.21
5/10
2/5
Analogical
explanations in Public Policy course
Slide38Student Interaction with Calculus ProblemFrom the graph of y=f'(x) on its entire domain of [a,h], at which x-value(s) is f' least?
x = ex = bx = ax = cCorrect! This is the lowest point on the graph of f'. FeedbackCorrect! This is the lowest point on the graph of f'. If you would like to review this sort of question further, you can look back at your notes about how we found where the function g was greatest and least. Or you can look at the relevant video. Click this link to open the video in a new window. Feedback and Prompt to Review
Slide39Version
Probability of ConditionMean Student Rating
Number of Students
Standard Deviation of Rating
SEM - Rating
Next Problem Accuracy
SEM - Next Problem Accuracy
Instructor Rating
Instructor Confidence
1. Feedback
0.48
9.00
18.00
2.18
0.51
0.64
0.07
7/10
4/5
2. Feedback and Prompt to Review
0.52
9.10
18.00
1.91
0.45
0.72
0.07
8/10
4/5
Learning
T
ips
in Calculus course
Slide40Qualitative Instructor FeedbackLowered Barriers: “I’m not aware of any tools that do this sort of thing… even if I found one, I don't think that I have the technical expertise to incorporate it”Reflection on pedagogy: “I never really seriously considered typing up multiple versions as we are now doing. So even if we don't get any significant data, that will have been a benefit in my mind”
Making research practical: “a valuable tool. Putting in the hands of the teacher to understand how their students learn. Not just in broad terms, but specifically in their course” “you must know plenty of general things about how students learn, whereas I know specific things about how they get calculus”Directly helping students: “improved the experience of many of the students by giving them answers that are more helpful… the earlier ones can help improve the experience of the later students. That’s pretty neat”
Slide41Apps for Student Goal Settingtiny.cc/keepingengagedOn-Demand prompts to specify goals, think through obstacles, receive reminders
Slide42Student Study Strategies: Reflective Questions
Slide43App providing Situational, On-Demand Promptstiny.cc/rqs Reflective Questions Strategy
Slide44Conclusion+ N
...Dynamic AnalysisAX%
100-X%
B
0%
100%
Personalization
100%
0%
A
B
A
B
A
B
Enhancement
Vision: Perpetually Improving Systems
Approach:
Collaborative, Dynamic
,
Personalized Experimentation
Motivation & Reflection
Adaptive
eXplanation
Improvement
System (AXIS)
Discovering how to personalize
Future
Bridging
Teachers & Scientists
Apps for
Goal
Setting & Study Strategies
Postdoc available at NUS
josephjaywilliams@gmail.com
nus.ed.sg/alset
Institute for Application of Learning Sciences to Educational Technology
Slide45Thank You!
Juho Kim, Krzysztof Gajos, Anna RaffertyHarvard VPAL (Vice Provost for Advances in Learning) ResearchTania Lombrozo & Tom Griffiths
Candace
Thille
& John Mitchell
Jascha
Sohl
-Dickstein, PERTS, Khan Academy
Sam Maldonado
Lytics
Lab
Slide46Conclusion+ N
...Dynamic AnalysisAX%
100-X%
B
0%
100%
Personalization
100%
0%
A
B
A
B
A
B
Enhancement
Vision: Perpetually Improving Systems
Approach:
Collaborative, Dynamic
,
Personalized Experimentation
Motivation & Reflection
Adaptive
eXplanation
Improvement
System (AXIS)
Discovering how to personalize
Future
Bridging Teachers & Scientists
Apps for Goal Setting & Study Strategies
Postdoc available at NUS
josephjaywilliams@gmail.com
nus.ed.sg/alset
Institute for Application of Learning Sciences to Educational Technology