Maria Costa Statistics Programming and Data Strategy GSK Prior Elicitation Teaching Old Dogs New Tricks PSI Annual Conference 15 th 17 th May 2017 Outline Introduction Background ID: 639481
Download Presentation The PPT/PDF document "Kim berley Hacquoil , Statistics, Progra..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Kimberley Hacquoil, Statistics, Programming and Data Strategy GSKMaria Costa, Statistics, Programming and Data Strategy GSK
Prior Elicitation:
Teaching Old Dogs New Tricks
PSI Annual Conference
15
th
– 17
th
May 2017Slide2
OutlineIntroductionBackground
Prior Elicitation eLearning Overview
eLearning Development Process
eLearning DemonstrationChallenges & LearningsConclusion
PSI 2017
Prior Elicitation: Kimberley Hacquoil & Maria Costa
2Slide3
IntroductionPrior elicitation is the process through which expert knowledge about a quantity of interest (e.g., treatment effect) is elicited from a subject matter expert and represented through a probability distribution
In order to participate, experts need to understand basic probability concepts
What is a subjective probability?
How can uncertainty be represented through a probability distribution?Which subjective judgements should be made and how to assess these?For a prior elicitation to be effective it is important to train experts (typically non-statisticians) in these concepts
This training component is a challenging and time-consuming task
In this talk we will share
GSK’s
experience in developing and implementing an eLearning to train experts in the fundamental statistical concepts underlying a prior elicitationObjective: “To create a fit-for-purpose eLearning which would train experts for prior elicitation using Roulette and Quartile methods”
In collaboration with Tony O’Hagan and Grifo Multimedia
PSI 2017
Prior Elicitation: Kimberley Hacquoil & Maria Costa
3Slide4
BackgroundPrior Elicitation: Kimberley Hacquoil & Maria Costa
4
Since 2014 GSK have conducted ~ 35 Prior Elicitation sessions
Currently, training happens in a separate ~45 min meeting that occurs prior to the actual elicitation session
This can be challenging as experts are unlikely to be available at the same time (particularly if in multiple time zones) leading to multiple training sessions being required
Development of eLearning streamlines training process and allows “just in time” approach to training
Post-elicitation phase (facilitator)
Elicitation phase (experts + facilitator)
Pre-elicitation phase (project statistician & physician + facilitator)
Select experts
Documentation
Decision problem or statistical model
Limited /conflicting evidence;
high uncertainty
Problem definition (project team)
Select method
Frame problem
Decision to conduct elicitation
Prepare evidence dossier
Training
Carry out elicitation
PSI 2017Slide5
Prior Elicitation eLearning OverviewThe eLearning curriculum is composed of 4 modules...Prior Elicitation: Kimberley Hacquoil & Maria Costa
5
Module 0: Introduction
Introduction to eLearning and the concept of expert judgement elicitation
Overview of eLearning modules
Module 1: Probabilities
How subjective probabilities can be used to represent knowledge and uncertainty about events of interest
How uncertainty can be described by a probability distribution and how to interpret it
Module 2: Elicitation
Methods for expert elicitation and overview of structure of elicitation session
How the Quartile and Roulette methods work – which subjective judgements are required?
Group elicitation: the consensus prior and the role of the rational impartial observer
Module 3: Assessment
Assessment: is the expert able to make the necessary judgements using either the Quartile or Roulette methods and does he/she understand the implications of those judgements?
PSI 2017Slide6
eLearning Development ProcessPrior Elicitation: Kimberley Hacquoil & Maria Costa
6
Roll out and Embedding
Integration
Development
Maintenance
Scope of work
Storyboards
Planning and Design
Create Modules
Test Modules
GSK Systems
UAT
Implement
PSI 2017Slide7
Roll out and Embedding
Integration
Development
Maintenance
Scope of work
Storyboards
Planning and Design
Create Modules
Test Modules
GSK Systems
UAT
Implement
eLearning Development Process
Prior Elicitation: Kimberley Hacquoil & Maria Costa
7
PSI 2017Slide8
Roll out and Embedding
Integration
Development
Maintenance
Scope of work
Storyboards
Planning and Design
Create Modules
Test Modules
GSK Systems
UAT
Implement
eLearning Development Process
Prior Elicitation: Kimberley Hacquoil & Maria Costa
8
PSI 2017Slide9
Roll out and Embedding
Integration
Development
Maintenance
Scope of work
Storyboards
Planning and Design
Create Modules
Test Modules
GSK Systems
UAT
Implement
eLearning Development Process
Prior Elicitation: Kimberley Hacquoil & Maria Costa
9
PSI 2017Slide10
Roll out and Embedding
Integration
Development
Maintenance
Scope of work
Storyboards
Planning and Design
Create Modules
Test Modules
GSK Systems
UAT
Implement
eLearning Development Process
Prior Elicitation: Kimberley Hacquoil & Maria Costa
10
PSI 2017Slide11
Roll out and Embedding
Integration
Development
Maintenance
Scope of work
Storyboards
Planning and Design
Create Modules
Test Modules
GSK Systems
UAT
Implement
eLearning Development Process
Prior Elicitation: Kimberley Hacquoil & Maria Costa
11
Communication of new eLearning to target audience
Documentation of eLearning completion to ensure compliance
Gather feedback on content and usability
PSI 2017Slide12
Roll out and Embedding
Integration
Development
Maintenance
Scope of work
Storyboards
Planning and Design
Create Modules
Test Modules
GSK Systems
UAT
Implement
eLearning Development Process
Prior Elicitation: Kimberley Hacquoil & Maria Costa
12
PSI 2017Slide13
eLearning Demonstration
PSI 2017
Prior Elicitation: Kimberley Hacquoil & Maria Costa
13Slide14
Challenges & LearningsChallenges Teaching in eLearning environment
Balancing teaching, examples, assessment
Balancing length of eLearning and information overload
Ideal vs too hard/long to implement/program Coordination issuesIntegrating different systems
Understanding what people need and when
Learnings
Project management
Regular meetings/interactions
Talking to the right people
Capture comments in a more structured way
PSI 2017
Prior Elicitation: Kimberley Hacquoil & Maria Costa
14Slide15
ConclusionCurrently going through User Acceptance Testing stages of implementationSelected 2 clinical experts: one with and one without experience in prior elicitation
Also included experienced statisticians (experienced facilitator) as part of UAT
Plan is to roll out eLearning by end of May/ early June (depending on
UAT outcome)
Where we are...
PSI 2017
Prior Elicitation: Kimberley Hacquoil & Maria Costa
15Slide16
AcknowledgementsTony O’Hagan
Antonio De Girolamo
Livio Melfi
Roberta MemeoEd Morris Jeremy Oakley
Bill O’Shea, Denise Bird, Kate Foster
Nigel Dallow, Nicky Best, Tim Montague
Duncan Richards,
Rajnish Saini
PSI 2017
Prior Elicitation: Kimberley Hacquoil & Maria Costa
16Slide17
Thank YouSlide18
Back Up18Slide19
Screen ShotsModule 0 - IntroductionPrior Elicitation: Kimberley Hacquoil & Maria Costa
19
PSI 2017Slide20
Screen ShotsModule 1 – Subjective ProbabilitiesPrior Elicitation: Kimberley Hacquoil & Maria Costa
20
PSI 2017Slide21
Screen ShotsModule 1 – Subjective ProbabilitiesPrior Elicitation: Kimberley Hacquoil & Maria Costa
21
PSI 2017Slide22
Screen ShotsModule 1 – Representing Knowledge through Probability DistributionsPrior Elicitation: Kimberley Hacquoil & Maria Costa
22
PSI 2017Slide23
Screen ShotsModule 1 – Interpreting Probability DistributionsPrior Elicitation: Kimberley Hacquoil & Maria Costa
23
PSI 2017Slide24
Screen ShotsModule 1 – Assessing Understanding of Probability DistributionsPrior Elicitation: Kimberley Hacquoil & Maria Costa
24
PSI 2017Slide25
Screen ShotsModule 2 – Elicitation MethodsPrior Elicitation: Kimberley Hacquoil & Maria Costa
25
PSI 2017Slide26
Screen ShotsModule 2 – Example to Illustrate Elicitation Methods Prior Elicitation: Kimberley Hacquoil & Maria Costa
26
PSI 2017Slide27
Screen ShotsModule 2 – Quartile MethodPrior Elicitation: Kimberley Hacquoil & Maria Costa
27
PSI 2017Slide28
Screen ShotsModule 2 – Quartile MethodPrior Elicitation: Kimberley Hacquoil & Maria Costa
28
PSI 2017Slide29
Screen ShotsModule 2 – Roulette MethodPrior Elicitation: Kimberley Hacquoil & Maria Costa
29
PSI 2017Slide30
Screen ShotsModule 2 – Roulette MethodPrior Elicitation: Kimberley Hacquoil & Maria Costa
30
PSI 2017Slide31
Screen ShotsModule 2 – Structure of an Elicitation SessionPrior Elicitation: Kimberley Hacquoil & Maria Costa
31
PSI 2017
Session starts with individual elicitation: Each expert is asked to represent their prior knowledge through a distribution
Facilitator then leads group discussion: Objective is to reach a consensus prior, representing the views of a Rational Impartial Observer Slide32
Screen ShotsModule 3 – The Quantity of InterestPrior Elicitation: Kimberley Hacquoil & Maria Costa
32
PSI 2017Slide33
Screen ShotsModule 3 – The Evidence DossierPrior Elicitation: Kimberley Hacquoil & Maria Costa
33
PSI 2017Slide34
Screen ShotsModule 3 – Assessing the Quartile Method using MATCHPrior Elicitation: Kimberley Hacquoil & Maria Costa
34
PSI 2017Slide35
Screen ShotsModule 3 – Assessing the Quartile Method using MATCHPrior Elicitation: Kimberley Hacquoil & Maria Costa
35
PSI 2017Slide36
Screen ShotsModule 3 – Assessing the Roulette MethodPrior Elicitation: Kimberley Hacquoil & Maria Costa
36
PSI 2017Slide37
Screen ShotsModule 3 – FeedbackPrior Elicitation: Kimberley Hacquoil & Maria Costa
37
PSI 2017