Strand A: How Can We Make Intensive Intervention Happen? Considerations for Knowledge - PowerPoint Presentation

Slide1

Strand A: How Can We Make Intensive Intervention Happen? Considerations for Knowledge Development, Implementation, and Policy

Session 3: From Know-How to Action: Assessing and Improving School-Level Implementation of Data-Based IndividualizationLaura Berry Kuchle, Christopher Lemons, T. Chris Riley-Tillman, Louis Danielson

This document was produced under U.S. Department of Education, Office of Special Education Programs, Award No. H326Q110005. Celia 

Rosenquist

serves as the project officer. The views expressed herein do not necessarily represent the positions or policies of the U.S. Department of Education. No official endorsement by the U.S. Department of Education of any product, commodity, service or enterprise mentioned in this document is intended or should be inferred.Slide2

Introduction to DBIWhy Do We Need to Assess DBI Implementation?Developing a

RubricDBI Implementation RubricUse in NCII Intensive Technical Assistance SitesLessons Learned: Strategies for Improving Technical Assistance and DBI ImplementationNCII Resources to Support Implementation

Session OverviewSlide3

3

What is Intensive Intervention?Intensive intervention addresses severe and persistent learning or behavior difficulties. Intensive intervention should be Driven by data Characterized by increased intensity (e.g., smaller group, expanded time) and individualization of academic instruction and/or behavioral supportsSlide4

4

A Bird’s Eye View of DBISlide5

Why Do We Need to Assess DBI Implementation?

5Slide6

Extent to which an intervention’s core components have been implemented as planned (Nelson et al., 2012)For DBI, this includes the instructional platform, adapted iterations of intervention, ongoing progress monitoring, and decision-making procedures.

See IRIS module for additional information: http://iris.peabody.vanderbilt.edu/module/fid/ What Is Fidelity?

6Slide7

Student levelFor a given student, are assessment and intervention being carried out as planned?Student-level fidelity may be assessed through teacher logs, observations, or record review

Systems level (school or team)Are essential components of DBI being implemented consistently?Are there systems-level problems that hinder DBI implementation?A systems-level rubric or interview might examine team meeting checklists, logs, or other recordsLevels of Fidelity

7Slide8

Fidelity assessment provides evidence that DBI is being implemented as

intendedAssessment, intervention, and decision-making processesStudent and systems levelsProvides guidance on how to improve DBI implementationDoes the interventionist need additional training or support?Is an intervention adaptation needed?Is there a systems-level problem? (e.g., scheduling prevents sufficient intervention time, staff do not have access to evidence-based instructional platforms)

Why Does Fidelity Matter?

8Slide9

Developing a Rubric

9Slide10

Based on the structure of the Center on Response to Intervention’s RTI Essential Components Integrity Rubric http://www.rti4success.org/resource/essential-components-rti-integrity-rubric-and-worksheet

Identify structures, resources, and practices needed for quality school-level implementation of DBIReflect lessons learned from NCII knowledge development sitesDeveloping a Systems-Level Rubric

10Slide11

Purpose: to learn about strategies for implementing intensive intervention from sites that have demonstrated positive outcomes for students with disabilities (SWDs)Implementing

Intensive Intervention: Lessons Learned From the Field (NCII, 2013) http://www.intensiveintervention.org/resource/implementing-intensive-intervention-lessons-learned-field

Key Findings From Knowledge

Development Sites

11Slide12

Intensive intervention is most likely to be facilitated when implemented as a component of a multi-tiered system of support (MTSS)

Family engagement supports implementationIntensive behavioral intervention is often more challenging because of limited progress monitoringInconsistent decision rules for intensifying supports can hinder intervention planning and resource allocationHidden inefficiencies may overtax staff unnecessarily

Lessons Learned

12Slide13

DBI Implementation Rubric

13Slide14

NCII’s DBI Implementation Rubric and Interview: http://

www.intensiveintervention.org/resource/dbi-implementation-rubric-and-interviewAligned with the essential components of DBI and the infrastructure that is necessary for successful implementation in Grades K–6Measuring School-Level Implementation

14Slide15

System FeaturesData

and Decision MakingInterventionDBI ProcessDBI EvaluationComponents/Infrastructure Required for DBI Implementation

15Slide16

LeadershipSchool Schedules

ResourcesCultural and Linguistic ResponsivenessCommunication With and Involvement of ParentsCommunication With and Involvement of All Staff

1.

System

Features

16Slide17

Process to Identify Students in Need of Intensive Intervention

Academic Progress-Monitoring ToolsBehavior Progress-Monitoring ToolsProgress-Monitoring ProceduresDiagnostic Assessment

Fidelity of Assessment Implementation

2. Data and Decision Making

17Slide18

Intensive InterventionAdaptation of Intervention

Fidelity of Implementation of InterventionInterventionist CharacteristicsRelationship to Core Instruction

3. Intervention

18Slide19

Team or Appropriate Support MechanismRegular Meetings

Data ReviewDocumentationDecision RulesOverall Implementation of DBI Process

4. DBI Process

19Slide20

Evaluation

5. DBI Evaluation20Slide21

21

Rating Scale1–5 points with anchors for1 point = little or no implementation3 points = partial or inconsistent implementation5 points = complete and consistent implementationSlide22

22

DBI Implementation InterviewScript and note-taking template for gathering information to evaluate rubricSample questions for each rubric itemSlide23

Use in NCII Intensive Technical Assistance Sites

23Slide24

Spring and fall 2014: 17 schools in 4 statesOngoing NCII training and coaching for approximately 1–2 school yearsTwo facilitators led interview

NCII staff and consultants, including NCII school coachCo-rated for reliability checksDBI Implementation “Pulse Checks”24Slide25

When two facilitators scored the same item for the same school:Mean/average difference was 0.51 points on the 5-point scaleMean difference across items ranged from 0.12 to 0.94 points

How Much Did Raters’ Scores Vary?25Slide26

26

How Often Did Both Raters Give the Same Score? Within One Point?Inter-rater Agreement on Item Ratings

Across All Items

Range

Perfect Agreement

58%

29%–88%

Agreement Within 1 Point

94%

76%

–

100%Slide27

Guideline: Refine rubric anchors and/or interview questions for items with <80% agreement within one pointItem 5a, Evaluation76% agreement within one point

Mean point difference = 0.82Item 4a, Team or Appropriate Support Mechanism82% agreement within one pointMean point difference = 0.94Items With Low Agreement

27Slide28

Mean score across all items and schools was 2.99Suggests partial or inconsistent implementationExpected at this point in time

Schools’ mean scores ranged from 2.17 to 3.58Overall Scores28Slide29

29

Items With Highest Scores

Item

Mean

3d. Interventionist Characteristics

3.88

4a. Team or Appropriate Support Mechanism

3.82

4b. Regular Meetings

3.82

3e. Relationship to Core Instruction

3.65

1a. Leadership

3.53

2b. Academic Progress-Monitoring Tools

3.47Slide30

Trained, experienced interventionistsTeams to support decision making for DBI, with a regular meeting scheduleAligning intervention to student need and core expectations, addressing prerequisite skills as appropriate

District and school leadership support for DBITechnically rigorous academic progress-monitoring tools that are sensitive to student improvementThese Scores Suggest Relative Strengths in the Areas of…

30Slide31

31

Items With Lowest Scores

Item

Mean

5a. Evaluation

2.12

2c. Behavior Progress-Monitoring Tools

2.18

2f.

Fidelity of Assessment Implementation

2.24

3c. Fidelity of Implementation of Intervention

2.29

4e. Decision Rules

2.35

4f. Overall Implementation of DBI Process

2.35Slide32

EvaluationSchools may informally review implementation without setting specific goalsEvaluation may be difficult or a low priority if other pieces are not clearly articulated and in place

Behavior progress monitoringFewer validated tools available compared to academicsNew to many schoolsWhy Might These Items Have Relatively Low Scores?

32Slide33

FidelityFew schools systematically monitor

Decision rulesMay not be clearly articulated in one or more areasApplication may not be consistentOverall implementationMay be inconsistent or poorly documentedConsidering Low Scores, Continued…

33Slide34

Lessons Learned 2

Strategies for Improving Technical Assistance and DBI Implementation34

Lessons Learned Slide35

Cross-state coach meeting in June 2014Discussed DBI Interviews and coaching activities throughout the year to identify essential elements of DBI implementation—implications for expanding to new schoolsIncorporated interview findings into 2014–15 TA plans; now more individualized at school level

Created Coaches’ Corner website and Professional Learning Community to share resources throughout the yearLearning From Initial DBI Interviews35Slide36

Staff commitmentStudent plansStudent meetings

Valid, reliable dataInclusion of students with disabilitiesEssential Elements of DBI Implementation

36Slide37

Staff Commitment

Key Element

Flexibility Within Implementation

Commitment of:

Principal

Intervention staff

Special educators

Specific

intervention

staff involved including staff who work with students with intensive needs in the area(s) of concern. (

e.g., reading specialists,

social

workers) Slide38

38

Student Plans

Key Element

Flexibility Within Implementation

Student plans are developed and reflect:

Accurate and timely

student data

Goal(s) for the

intervention based on valid, reliable assessment tools

Timeline for executing and revisiting the intervention plan

Content

area

(s)

Number of student plans

Grade level(s) Slide39

39

Student Meetings

Key Element

Flexibility Within Implementation

Student meetings are data driven

There is a regularly scheduled time

to

meet

Meetings are structured to maximize efficiency and focused problem solving

Frequency

Schedule

Team members  Slide40

40

Progress Monitoring

Key Element

Flexibility Within Implementation

Valid, reliable

progress monitoring tools are used.

Data are

graphed.

Data are collected

at regular

intervals.

Choice of tool

Use of

progress-monitoring

data at

other

tiers Slide41

41

Students With Disabilities

Key Element

Flexibility Within Implementation

Students with disabilities must

have access to intensive

intervention.

Who delivers intervention for

students with disabilities

Inclusion of students with and without IEPsSlide42

NCII Resources to Support Implementation

42Slide43

Systems level considerations—infrastructure and staff commitmentModule to assess and support readiness to implement DBIIdentifying intervention and assessment tools for mathematics and behavior

Tools chartsSample lessons and activitiesConsistent procedures and documentationStudent meeting toolsAddressing Common Barriers to Implementation

43Slide44

Module introducing schools to DBI and infrastructure needed for implementation: http://www.intensiveintervention.org/resource/getting-ready-implement-intensive-intervention-infrastructure-data-based-individualization

Handouts and activities to assess readiness and begin action planningGetting Ready to Implement Intensive Intervention: Infrastructure for DBI

44Slide45

Tools

Charts

Academic Progress Monitoring

http://www.intensiveintervention.org/chart/progress-monitoring

Academic Intervention

http://www.intensiveintervention.org/chart/instructional-intervention-tools

Behavioral Progress Monitoring

http://www.intensiveintervention.org/chart/behavioral-progress-monitoring-tools

Behavioral Intervention

http://www.intensiveintervention.org/chart/behavioral-intervention-chart

Slide46

46

Mathematics: Sample Lessons & Activities

http://

www.intensiveintervention.org/resources/sample-lessons-activities/mathematics

Slide47

47

Implementing Behavioral Strategies: Considerations and Sample Resources

http://

www.intensiveintervention.org/implementing-behavioral-strategies-considerations-and-sample-resourcesSlide48

48

Individual Student Meeting Tools

http://

www.intensiveintervention.org/tools-support-intensive-intervention-data-meetings

Slide49

NCII websiteExamples of Standards-Aligned Instruction Across TiersDBI Training Series

WebinarsConnect to NCIILearn More About DBI49Slide50

50

Find Out What NCII Has to Offer

www.intensiveintervention.orgSlide51

51

Examples of Standards-Aligned Instruction Across Tiers

http://

www.intensiveintervention.org/illustration-standards-relevant-instruction-across-levels-tiered-system

Slide52

52

DBI Training Series Eight training modules focusing on components of DBI for academics and behaviorOne module focused on

readiness and action planning

Include

Slides and speaker notes

Activities

Coaching guides

http://www.intensiveintervention.org/content/dbi-training-seriesSlide53

53

Webinars

View archived webinars and look for announcements about the next live webinar:

www.intensiveintervention.org

Slide54

54

Connect to NCIISign up on our website to receive our newsletter and announcementsFollow us on YouTube and TwitterYouTube Channel: National Center on Intensive Intervention

Twitter handle:

@

T

heNCIISlide55

National Center on Intensive Intervention. (2013). Implementing intensive intervention: Lessons learned from the field. Washington, DC: U.S. Department of Education, Office

of Special Education Programs. http://www.intensiveintervention.org/sites/default/files/Lessons_Learned_From_Field_0.pdfNelson, M. C., Cordray, D. S., Hulleman, C. S., & Sommer, E. C. (2012). A procedure for assessing intervention fidelity in experiments testing educational and behavioral interventions.

The Journal of Behavioral Health Services & Research, 39(4)

,

374–396

.

References

55Slide56

Reference

National Center on Intensive Intervention1000 Thomas Jefferson Street NW

Washington, DC 20007-3835

866-577-5787

www.intensiveintervention.org

ncii@air.org

@

TheNCII