Linda M Collins PhD The Methodology Center and Department of Human Development amp Family Studies Penn State Presented at Grand Rounds Yale Center for Implementation Science cosponsor Center for Methods of Implementation and Prevention Science ID: 935556
Download Presentation The PPT/PDF document "Optimization of interventions for effect..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Optimization of interventions for effectiveness, efficiency, economy, and sustainability
Linda
M. Collins, Ph.D.
The Methodology Center and
Department of Human Development & Family
Studies
Penn State
Presented at Grand Rounds
Yale Center for Implementation Science
co-sponsor: Center for Methods of Implementation and Prevention Science
December 11, 2018
Slide2Late 20
th
century
(mid 1980’s)
Early 21
st century(today)
18 mpg
No airbags
27 mpg
Driver, passenger, head and side airbags
Massive improvements in technology over the past 30 years
2
Have interventions
improved this much?
Slide3Outline
Definitions
Critique of “business as usual”
What is MOST? What is optimization?
OK, how do you do this? Closing remarks
3
Slide4What is an intervention?
Examples:
Smoking cessation
School-based drug abuse prevention
Online intervention to prevent excessive drinking and risky sex in college studentsAdult weight lossIntervention to get HIV+ individuals into the health care system and treated with antriretrovirals
Typically made up of multiple components.
Slide5What is an intervention component?
Definition:
Any aspect of an intervention that can be separated out for study
Parts of intervention content
e.g., each major topic to be coveredFeatures that affect quality of implementation by……promoting engagement/compliance/adherence
e.g., MEMScaps…improving fidelity/quality of deliverye.g., 800 number for program delivery staff to call with questions
Slide6Outline
Definitions
Critique of “business as usual”
What is MOST? What is optimization?
OK, how do you do this? Closing remarks
6
Slide7Classical treatment package approach
Intervention
component
component
component
Evaluation via RCT
component
component
Slide8What is wrong with evaluating a treatment package via an RCT?
Absolutely nothing!
Slide9Classical treatment package approach
Intervention
component
component
component
Evaluation via RCT
component
component
Slide10An RCT that finds a significant effect DOES NOT
provide information about:
Which components are making positive contributions to overall effect
Whether the inclusion of one component has an impact on the effect of another
Whether a component’s contribution offsets its costHow to make the intervention more effective, efficient, and scalable/sustainable
10
Slide11An RCT that finds a non-significant effect DOES NOT
provide information about:
Whether any components are worth retaining
Whether one component had a negative effect that offset the positive effect of others
Specifically what went wrong and how to do it better the next time
11
Slide12Outline
Definitions
Critique of “business as usual”
What is MOST? What is optimization?
OK, how do you do this? Closing remarks
12
Slide13The multiphase optimization strategy (MOST)
An engineering-inspired framework for development, optimization, and evaluation of interventions
Using MOST it is possible to engineer an intervention to meet a specific criterion
Slide14Desiderata for a sustainable intervention
Effectiveness
Extent to which the intervention does more good than harm (under real-world conditions; Flay, 1986)
Efficiency
Extent to which the intervention avoids wasting time, money, or other valuable resourcesEconomyExtent to which the intervention is effective without exceeding budgetary constraints, and offers a good value
ScalabilityExtent to which the intervention can be implemented in the intended setting exactly as evaluated
Slide15Optimization of an intervention is:
T
he
process of identifying
the intervention that provides the best expected outcome obtainable……within key constraints imposed by the need for efficiency, economy, and/or scalability
.
Slide16Figure taken from Collins, L.M. (2018).
Optimization of Behavioral,
Biobehavioral
, and Biomedical Interventions: The Multiphase Optimization Strategy (MOST).
New York: Springer.
Flow
chart of the three phases of the multiphase optimization strategy (MOST). Rectangle = action.
Diamond = decision.
Slide17Phases of MOST: Preparation
, optimization, evaluation
Preparation
Purpose: to lay groundwork for optimization
Review prior research, take stock of clinical experience, conduct secondary analyses, etc.Derive conceptual modelSelect intervention components to examine
Conduct pilot/feasibility work Identify clearly operationalized optimization criterion
Slide18Selecting an optimization criterion
Optimization always involves a clearly stated
optimization criterion
This is the goal you want to achieve
Once achieved, it is the bar that sets a standard for later efforts
Slide19One possible optimization criterion (out of many)
Key constraint: Money
Most effective intervention that can be delivered for ≤ some $$
CONSIDER a primary-care-based smoking cessation intervention.
Suppose insurers will pay for a program that costs no more than $500/person to implement, including materials and staff time. Achieve this by selecting set of components that represents the most effective intervention that can be delivered for ≤
$500/person.
Slide20Figure taken from Collins, L.M. (2018).
Optimization of Behavioral,
Biobehavioral
, and Biomedical Interventions: The Multiphase Optimization Strategy (MOST).
New York: Springer.
Flow
chart of the three phases of the multiphase optimization strategy (MOST). Rectangle = action.
Diamond = decision.
Slide21Phases of MOST: Preparation,
optimization
, evaluation
Optimization
Objective: To form a treatment package that meets the optimization criterionCollect and analyze empirical data on performance of individual intervention components relying on efficient randomized experiments
Based on information gathered, select components and levels that meet optimization criterion.
Slide22Figure taken from Collins, L.M. (2018).
Optimization of Behavioral,
Biobehavioral
, and Biomedical Interventions: The Multiphase Optimization Strategy (MOST).
New York: Springer.
Flow
chart of the three phases of the multiphase optimization strategy (MOST). Rectangle = action.
Diamond = decision.
Slide23Phases of MOST: Preparation, optimization,
evaluation
Evaluation
Objective: To establish whether the optimized intervention has a statistically significant effect compared to a control or alternative intervention
Conduct an RCT
Slide24Outline
Definitions
Critique of “business as usual”
What is MOST? What is optimization?
OK, how do you do this? Closing remarks
24
Slide25Example: Primary-care-based smoking cessation study
PIs: Mike Fiore and Tim Baker, University of Wisconsin
Investigators include Robin
Mermelstein
(University of Illinois, Chicago) and Megan Piper (UW)Funded by the National Cancer InstitutePiper et al. (2018),
Annals of Behavioral Medicine; Baker et al. (2017), Behavior Therapy; Piper et al. (2017a,b), Drug and Alcohol Dependence; Baker et al. (2016), Addiction
; Cook et al. (2016), Addiction;
Schlam et al. (2016), Addiction; Piper et al. (2016),
Addiction; Collins et al. (2014), Translational Behavioral Medicine…
Slide26Components being considered for the
smoking cessation intervention
Component
Higher
(intensive) level
Lower level
Precessation
nicotine patch
YesNo
26
Slide27Components being considered for the
smoking cessation intervention
Component
Higher
(intensive) level
Lower level
Precessation
nicotine patch
YesNo
Precessation ad lib oral NRT (gum)
YesNo
27
Slide28Components being considered for the
smoking cessation intervention
Component
Higher
(intensive) level
Lower level
Precessation
nicotine patch
YesNo
Precessation ad lib oral NRT (gum)
YesNo
Precessation
counseling3 20-min sessions (2 in-person, 1 phone)
No
28
Slide29Components being considered for the
smoking cessation intervention
Component
Higher
(intensive) level
Lower level
Precessation
nicotine patch
YesNo
Precessation ad lib oral NRT (gum)
YesNo
Precessation
counseling3 20-min sessions (2 in-person, 1 phone)
No
Cessation in-person counselin
g
3 20-min sessions
1 3-min
session
29
Slide30Components being considered for the
smoking cessation intervention
Component
Higher
(intensive) level
Lower level
Precessation
nicotine patch
YesNo
Precessation ad lib oral NRT (gum)
YesNo
Precessation
counseling3 20-min sessions (2 in-person, 1 phone)
No
Cessation in-person counselin
g
3 20-min sessions
1 3-min
session
Cessation telephone
counseling
3 15-min sessions
1 10-min session
30
Slide31Components being considered for the
smoking cessation intervention
Component
Higher
(intensive) level
Lower level
Precessation
nicotine patch
YesNo
Precessation ad lib oral NRT (gum)Yes
No
Precessation counseling
3 20-min sessions (2 in-person, 1 phone)No
Cessation in-person counselin
g
3 20-min sessions
1 3-min
session
Cessation telephone
counseling
3 15-min sessions
1 10-min session
Maintenance medication duration starting at quit date
(combo NRT)
16 weeks
8 weeks
31
Slide32MOST as implemented in smoking cessation study
Precess
. counseling
Cess
. in-
pers
couns
.
Precess
. NRT: patch
Evaluation via RCT
Cess. phone couns.
Maint
. med. duration
EMPIRICALLY-BASED
OPTIMIZATION
Precess. NRT: gum
component
component
component
Optimized
smoking cessation intervention
Slide33Figure taken from Collins, L.M. (2018).
Optimization of Behavioral,
Biobehavioral
, and Biomedical Interventions: The Multiphase Optimization Strategy (MOST).
New York: Springer.
Flow
chart of the three phases of the multiphase optimization strategy (MOST). Rectangle = action.
Diamond = decision.
Slide34Slide35Choosing an efficient design for the optimization trial*
35
Design
Approximate N to achieve power≥.85
(Cohen’s
d
=.27)
Number of experimental conditions
Can interactions be examined?
Option
A: Six
individual experiments
Option
B:
Comparative treatment
Option
C:
Factorial experiment
*We are developing a fixed intervention, so we are considering factorial experimental designs and related designs
Slide36Design option A: Six individual treatment/control experiments
Precessation
patch vs. no patch
Precessation
oral NRT (gum) vs. no oral NRTPrecessation counseling vs. no precessation
counselingIntensive cessation counseling vs. minimalIntensive cessation phone counseling vs. minimal16 weeks of NRT during cessation/maintenance vs. 8 weeks
36
Slide37Choosing an efficient design for the optimization trial
37
Design
Approximate N to achieve power≥.85
(Cohen’s
d
=.27)
Number of experimental conditions
Can interactions be examined?
Option
A: Six
individual experiments
3,072
12
None
Option
B:
Comparative treatment
Option
C:
Factorial experiment
Slide38Design option B: Comparative treatment experiment
Treatment conditions
Control
Precessation
p
atch
=
yes
Precessation
gum =
yes
Precessation
counseling =
yes
Cessation counseling
=
intensive
Cessation
phone counseling =
intensive
Cessation NRT =
16 weeks
All =
low
All others
= low
All others
= low
All others
= low
All others = lowAll others = low
All others = low38
Experimental conditions:
Slide39Choosing an efficient design for the optimization trial
39
Design
Approximate N to achieve power≥.85
(Cohen’s
d
=.27)
Number of experimental conditions
Can interactions be examined?
Option
A: Six
individual experiments
3,072
12
None
Option
B:
Comparative treatment
1,792
7
None
Option
C:
Factorial experiment
Slide40Design option C
2
6
factorial experiment
This will have 64 experimental conditions40
Slide41Choosing an efficient design for the optimization trial
41
Design
Approximate N to achieve power≥.85
(Cohen’s
d
=.27)
Number of experimental conditions
Can interactions be examined?
Option
A: Six
individual experiments
3,072
12
None
Option
B:
Comparative treatment
1,792
7
None
Option
C:
Factorial experiment
512
64
Yes, all
Slide42Choosing an efficient design for the optimization trial
42
Design
Approximate N to achieve power≥.85
(Cohen’s
d
=.27)
Number of experimental conditions
Can interactions be examined?
Option
A: Six
individual experiments
3,072
12
None
Option
B:
Comparative treatment
1,792
7
None
Option
C:
Factorial
experiment
*
512
64
Yes, all
We actually used a Resolution VI fractional factorial design with 32 experimental conditions.
Slide43Optimize based on results of optimization trial
Analyze data, obtain estimates of effects of each of the components
Use this information to select components
Discard components that do not perform adequately
If desired, based on predicted outcomes and estimated costs, select components that will make up optimized interventionDeveloping better decision-making approaches is one area I want to head next
Slide44Components/levels selected based on optimization trial
Based on the results of experimentation on 15 components, 5 “winners”:
From the optimization trial I described:
Precessation
oral NRTCessation phase in-person counseling at intensive level
From another optimization trial conducted as part of the P01:Extended medication (26-week postquit combination NRT)
Maintenance phase counseling telephone callsMaintenance phase automated adherence calls
Slide45Figure taken from Collins, L.M. (2018).
Optimization of Behavioral,
Biobehavioral
, and Biomedical Interventions: The Multiphase Optimization Strategy (MOST).
New York: Springer.
Flow
chart of the three phases of the multiphase optimization strategy (MOST). Rectangle = action.
Diamond = decision.
Slide46Slide47Outline
Definitions
Critique of “business as usual”
What is MOST? What is optimization?
OK, how do you do this? Closing remarks
47
Slide48Some possibilities offered by MOST
Engineer interventions to be cost-effective
Engineer interventions to be immediately scalable and sustainable
Based on one optimization trial, optimize using different criteria for different situations
Slide49Is MOST catching on?
More than 25 funded projects funded by 9 different NIH ICs, plus USDA and IES
Example areas:
Substance use and addiction
HIVObesity/weight managementHeart disease/Cardiac rehabilitationDiabetesSleep
Mental healthEducation 49
Slide50What’s next
More applications of MOST
Decision-making
Multi-criteria decision analysis
Decision-making based on results of optimization trial designs such as SMARTs and MRTsCost-effectivenessData analysisNon-normal modelsMediation analysis of data from an optimization trial
Slide51Imagine a 21st century in which interventions…
…include only active components
…are built in a principled manner to clearly-defined specifications
…operate transparently
…become incrementally better over time…are effective, efficient, economical, and immediately scalable, AND THEREFORE SUSTAINABLE
51
Slide5252
Slide53For more information:
http://methodology.psu.edu
Section on MOST with
Suggested reading
FAQAdvice for people writing grant proposals involving MOSTTraining May 13-17, 2019 in Bethesda, MD
Slide5454
Slide55Extra slides
Slide56For you skeptics:
When
used to address suitable research questions, balanced factorial experimental designs often require many FEWER subjects than alternative designs.
Don’t believe me? Try reading:
Collins, L.M., Dziak, J.J.,
Kugler, K.C., & Trail, J.B. (2014). Factorial experiments: Efficient tools for evaluation of intervention components. American Journal of Preventive Medicine, 47, 498-504. Chapter 3 in Collins, L.M. (2018),
Optimization of Behavioral, Biobehavioral, and Biomedical Interventions: The Multiphase Optimization Strategy (MOST)
. New York: Springer.
56
Slide57Quick intro to factorial experiments
Slide58Factorial experiments 101
Example: 2 X 2, or 2
2
, factorial design
Factorial experiments can have≥ 2 factors≥ 2 levels per factor
On the next slide is a 24 factorial design
Component
A
Component B
Off
On
Off
A,B
off
A on, B off
On
A off, B on
A,B on
Slide59Experimental conditions in a factorial experiment with four factors
Experimental condition
Factor A
Factor B
Factor C
Factor D
1
Off
Off
Off
Off
2
Off
Off
Off
On
3
Off
Off
On
Off
4
Off
Off
On
On
5
Off
On
Off
Off
6
Off
On
Off
On
7
Off
On
On
Off
8
Off
On
On
On
9
On
Off
Off
Off
10
On
Off
Off
On
11
On
Off
On
Off
12
On
Off
On
On
13
On
On
Off
Off
14
On
On
Off
On
15
On
On
On
Off
16
On
On
On
On
Slide60What are we trying to estimate with a factorial experiment?
Most important for decision making: Main effect of each factor
DEFINITION OF MAIN EFFECT OF FACTOR A:
Effect of Factor A averaged across all levels of all other factors
Also selected interactionsDEFINITION OF INTERACTION BETWEEN FACTOR A AND FACTOR B (assuming each factor has two levels):½ ((effect of Factor A at level 1 of Factor B) – (effect of Factor A at level 2 of Factor B))
Slide61Experimental
condition
Factor A
Factor B
Factor C
Factor D
1
Off
Off
Off
Off
2
Off
Off
Off
On
3
Off
Off
On
Off
4
Off
Off
On
On
5
Off
On
Off
Off
6
Off
On
Off
On
7
Off
On
On
Off
8
Off
On
On
On
9
On
Off
Off
Off
10
On
Off
Off
On
11
On
Off
On
Off
12
On
Off
On
On
13
On
On
Off
Off
14
On
On
Off
On
15
On
On
On
Off
16
On
On
On
On
MAIN EFFECT OF FACTOR A is mean of conditions 1-8 vs. mean of conditions 9-16
Slide62Experimental
condition
Factor A
Factor B
Factor C
Factor D
1
Off
Off
Off
Off
2
Off
Off
Off
On
3
Off
Off
On
Off
4
Off
Off
On
On
5
Off
On
Off
Off
6
Off
On
Off
On
7
Off
On
On
Off
8
Off
On
On
On
9
On
Off
Off
Off
10
On
Off
Off
On
11
On
Off
On
Off
12
On
Off
On
On
13
On
On
Off
Off
14
On
On
Off
On
15
On
On
On
Off
16
On
On
On
On
MAIN EFFECT OF FACTOR B is mean of conditions 5—8 and 13—16 vs. mean of conditions 1—4 and 9—12
Slide63Experimental
condition
Factor A
Factor B
Factor C
Factor D
1
Off
Off
Off
Off
2
Off
Off
Off
On
3
Off
Off
On
Off
4
Off
Off
On
On
5
Off
On
Off
Off
6
Off
On
Off
On
7
Off
On
On
Off
8
Off
On
On
On
9
On
Off
Off
Off
10
On
Off
Off
On
11
On
Off
On
Off
12
On
Off
On
On
13
On
On
Off
Off
14
On
On
Off
On
15
On
On
On
Off
16
On
On
On
On
MAIN EFFECT OF FACTOR C is mean of conditions 3,4,7,8,11,12,15, and 16 vs. mean of conditions 1,2,5,6,9,10, 13, and 14
Slide64Experimental
condition
Factor A
Factor B
Factor C
Factor D
1
Off
Off
Off
Off
2
Off
Off
Off
On
3
Off
Off
On
Off
4
Off
Off
On
On
5
Off
On
Off
Off
6
Off
On
Off
On
7
Off
On
On
Off
8
Off
On
On
On
9
On
Off
Off
Off
10
On
Off
Off
On
11
On
Off
On
Off
12
On
Off
On
On
13
On
On
Off
Off
14
On
On
Off
On
15
On
On
On
Off
16
On
On
On
On
MAIN EFFECT OF FACTOR D is mean of conditions 1,3,5,7,9,11,13,15 vs. mean of conditions 2,4,6,8,10,12,14,16
Slide65Design of Wisconsin experiment
Slide66Design of Wisconsin optimization trial
This is a factorial experiment with six factors.
It is a
2
6-1 fractional factorial.Resolution VIThe design has 32 experimental conditions.Hey! No
“control group”!!!