Implications for Measurement Design and Analysis Implementation Research Methods Meeting September 2021 2010 Chris S Hulleman PhD Implementation vs Implementation Fidelity Descriptive ID: 651825
Download Presentation The PPT/PDF document "Conceptualizing Intervention Fidelity:" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Conceptualizing Intervention Fidelity:
Implications for Measurement, Design, and Analysis
Implementation Research Methods MeetingSeptember 20-21, 2010
Chris S. Hulleman,
Ph.D.Slide2
Implementation vs. Implementation Fidelity
Descriptive
What happened as the intervention was implemented?A priori modelHow much, and with what quality, were the core intervention components implemented?
Implementation Assessment Continuum
Fidelity: How faithful was the implemented intervention (tTx) to the intended intervention (TTx)?Infidelity: TTx – tTx
Most assessments include bothSlide3
Linking Fidelity to Causal Models
Rubin’s Causal Model:True causal effect of X is (YiTx – YiC)RCT is best approximation
Tx – C = average causal effectFidelity Assessment Examines the difference between implemented causal components in the Tx and CThis difference is the
achieved relative strength (ARS) of the interventionTheoretical relative strength = TTx – TC
Achieved relative strength = tTx – tCIndex of fidelitySlide4
Implementation assessment typically captures…
(1) Essential or core components (activities,
processes, structures) (2) Necessary, but not unique, activities, processes and structures (supporting the essential components of Tx)
(3) Best practices (4) Ordinary features
of the setting (shared with the control group)Intervention FidelityassessmentSlide5
Why is this Important?
Construct ValidityWhich is the cause? (TTx - TC) or (tTx – t
C)Degradation due to poor implementation, contamination, or similarity between Tx and CExternal ValidityGeneralization is about tTx –
tCImplications for future specification of TxProgram failure vs. Implementation failure
Statistical Conclusion ValidityVariability in implementation increases error, and reduces effect size and powerSlide6
Why is this important? Reading First implementation results
Components
Sub-components
Performance Levels
ARS
RF
Non-RF
Reading Instruction
Daily (min.)
105.0
87.0
0.63
Daily in 5 components (min.)
59.0
50.8
0.35
Daily with High Quality practice
18.1
16.2
0.11
Overall Average
0.35
Adapted from Gamse et al. (2008) and Moss et al. (2008)
Effect Size Impact of Reading First on Reading Outcomes = .05Slide7
5-Step Process(Cordray, 2007)
Specify the intervention modelDevelop fidelity indicesDetermine reliability and validity
Combine indicesLink fidelity to outcomesConceptual
Measurement
AnalyticalSlide8
Some Challenges
Intervention Models
Unclear interventionsScripted vs. Unscripted
Intervention Components vs. Best Practices
MeasurementNovel constructs: Standardize methods and reporting (i.e., ARS) but not measures (Tx-specific)Measure in both Tx & C
Aggregation (or not) within and across levels
Analyses
Weighting of components
Psychometric properties?
Functional form?
Analytic frameworks
Descriptive vs. Causal (e.g., ITT) vs.
Explanatory (e.g., LATE)
See Howard’s Talk Next!
Future Implementation
Zone of Tolerable Adaptation
Systematically test impact of fidelity to core components
Tx
Strength (e.g., ARS): How big is big enough?Slide9
Treatment Strength (ARS): How Big is Big Enough?
Effect Size
StudyFidelity ARS
OutcomeMotivation – Lab
1.880.83Motivation – Field0.800.33
Reading First*
0.35
0.05
*Averaged over 1
st
, 2
nd
, and 3
rd
grades (Gamse et al., 2008).Slide10
Thank You!
And Special Thanks to
My Collaborators:
Catherine Darrow, Ph. D.
Amy Cassata-Widera, Ph.D.
David S.
Cordray
Michael Nelson
Evan Sommer
Anne Garrison
Charles MunterSlide11Slide12
Extras and NotesSlide13
Achieved Relative Strength = 0.15
Infidelity
“Infidelity”
(85)-(70) = 15
t
C
t
tx
T
Tx
T
C
.45
.40
.35
.30
.25
.20
.15
.10
.05
.00
Treatment Strength
Expected
Relative Strength = TTx - TC = (0.40-0.15) = 0.25100 90 85 80 75 70 65 60 55 50 Outcome Slide14
Linking Fidelity to OutcomesSlide15
Concerns and QuestionsBest practices vs. model-specific Fidelity
i.e., be wary of measures that find 100% fidelity!Fidelity as Achieved Relative Strength (ARS)How much ARS is enough?How to include ARS/fidelity into analytic frameworkWeighting of core componentsCombining (or not) of indicesAnalytic frameworkDescriptive
Causal (ITT)Explanatory (LATE, TOT, instrumental variables)ARS within multi-level modelsCombining indices within and across levelsZone of tolerable adaptationResearcher vs. teacher perspective (fidelity is bad b/c I’m like everyone else and need to be creative)Fidelity to process vs. structureFidelity as a moderator/mediatorWhen does fidelity = mediation, when does it not?
Tolerable adaptation vs. moderationDevelopmental studiesDevelop valid and reliable indicesDetermine which components matterImplementation drivers