/
Conceptualizing Intervention Fidelity: Conceptualizing Intervention Fidelity:

Conceptualizing Intervention Fidelity: - PowerPoint Presentation

celsa-spraggs
celsa-spraggs . @celsa-spraggs
Follow
374 views
Uploaded On 2018-03-15

Conceptualizing Intervention Fidelity: - PPT Presentation

Implications for Measurement Design and Analysis Implementation Research Methods Meeting September 2021 2010 Chris S Hulleman PhD Implementation vs Implementation Fidelity Descriptive ID: 651825

components fidelity implementation ttx fidelity components ttx implementation ars intervention strength causal relative reading big effect achieved core implemented

Share:

Link:

Embed:

Download Presentation from below link

Download Presentation The PPT/PDF document "Conceptualizing Intervention Fidelity:" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.


Presentation Transcript

Slide1

Conceptualizing Intervention Fidelity:

Implications for Measurement, Design, and Analysis

Implementation Research Methods MeetingSeptember 20-21, 2010

Chris S. Hulleman,

Ph.D.Slide2

Implementation vs. Implementation Fidelity

Descriptive

What happened as the intervention was implemented?A priori modelHow much, and with what quality, were the core intervention components implemented?

Implementation Assessment Continuum

Fidelity: How faithful was the implemented intervention (tTx) to the intended intervention (TTx)?Infidelity: TTx – tTx

Most assessments include bothSlide3

Linking Fidelity to Causal Models

Rubin’s Causal Model:True causal effect of X is (YiTx – YiC)RCT is best approximation

Tx – C = average causal effectFidelity Assessment Examines the difference between implemented causal components in the Tx and CThis difference is the

achieved relative strength (ARS) of the interventionTheoretical relative strength = TTx – TC

Achieved relative strength = tTx – tCIndex of fidelitySlide4

Implementation assessment typically captures…

(1) Essential or core components (activities,

processes, structures) (2) Necessary, but not unique, activities, processes and structures (supporting the essential components of Tx)

(3) Best practices (4) Ordinary features

of the setting (shared with the control group)Intervention FidelityassessmentSlide5

Why is this Important?

Construct ValidityWhich is the cause? (TTx - TC) or (tTx – t

C)Degradation due to poor implementation, contamination, or similarity between Tx and CExternal ValidityGeneralization is about tTx –

tCImplications for future specification of TxProgram failure vs. Implementation failure

Statistical Conclusion ValidityVariability in implementation increases error, and reduces effect size and powerSlide6

Why is this important? Reading First implementation results

Components

Sub-components

Performance Levels

ARS

RF

Non-RF

Reading Instruction

Daily (min.)

105.0

87.0

0.63

Daily in 5 components (min.)

59.0

50.8

0.35

Daily with High Quality practice

18.1

16.2

0.11

Overall Average

0.35

Adapted from Gamse et al. (2008) and Moss et al. (2008)

Effect Size Impact of Reading First on Reading Outcomes = .05Slide7

5-Step Process(Cordray, 2007)

Specify the intervention modelDevelop fidelity indicesDetermine reliability and validity

Combine indicesLink fidelity to outcomesConceptual

Measurement

AnalyticalSlide8

Some Challenges

Intervention Models

Unclear interventionsScripted vs. Unscripted

Intervention Components vs. Best Practices

MeasurementNovel constructs: Standardize methods and reporting (i.e., ARS) but not measures (Tx-specific)Measure in both Tx & C

Aggregation (or not) within and across levels

Analyses

Weighting of components

Psychometric properties?

Functional form?

Analytic frameworks

Descriptive vs. Causal (e.g., ITT) vs.

Explanatory (e.g., LATE)

See Howard’s Talk Next!

Future Implementation

Zone of Tolerable Adaptation

Systematically test impact of fidelity to core components

Tx

Strength (e.g., ARS): How big is big enough?Slide9

Treatment Strength (ARS): How Big is Big Enough?

Effect Size

StudyFidelity ARS

OutcomeMotivation – Lab

1.880.83Motivation – Field0.800.33

Reading First*

0.35

0.05

*Averaged over 1

st

, 2

nd

, and 3

rd

grades (Gamse et al., 2008).Slide10

Thank You!

And Special Thanks to

My Collaborators:

Catherine Darrow, Ph. D.

Amy Cassata-Widera, Ph.D.

David S.

Cordray

Michael Nelson

Evan Sommer

Anne Garrison

Charles MunterSlide11
Slide12

Extras and NotesSlide13

Achieved Relative Strength = 0.15

Infidelity

“Infidelity”

(85)-(70) = 15

t

C

t

tx

T

Tx

T

C

.45

.40

.35

.30

.25

.20

.15

.10

.05

.00

Treatment Strength

Expected

Relative Strength = TTx - TC = (0.40-0.15) = 0.25100 90 85 80 75 70 65 60 55 50 Outcome Slide14

Linking Fidelity to OutcomesSlide15

Concerns and QuestionsBest practices vs. model-specific Fidelity

i.e., be wary of measures that find 100% fidelity!Fidelity as Achieved Relative Strength (ARS)How much ARS is enough?How to include ARS/fidelity into analytic frameworkWeighting of core componentsCombining (or not) of indicesAnalytic frameworkDescriptive

Causal (ITT)Explanatory (LATE, TOT, instrumental variables)ARS within multi-level modelsCombining indices within and across levelsZone of tolerable adaptationResearcher vs. teacher perspective (fidelity is bad b/c I’m like everyone else and need to be creative)Fidelity to process vs. structureFidelity as a moderator/mediatorWhen does fidelity = mediation, when does it not?

Tolerable adaptation vs. moderationDevelopmental studiesDevelop valid and reliable indicesDetermine which components matterImplementation drivers