/
84.305L: Low-Cost, Short-Duration Evaluation of Education Interventions 84.305L: Low-Cost, Short-Duration Evaluation of Education Interventions

84.305L: Low-Cost, Short-Duration Evaluation of Education Interventions - PowerPoint Presentation

test
test . @test
Follow
355 views
Uploaded On 2018-09-18

84.305L: Low-Cost, Short-Duration Evaluation of Education Interventions - PPT Presentation

84324L LowCost ShortDuration Evaluation of Special Education Interventions Allen Ruby National Center for Education Research Kimberley Sprague National Center for Special Education Research ID: 669185

research education agency intervention education research intervention agency outcomes data student 324l project plan evaluation state secondary gov agencies

Share:

Link:

Embed:

Download Presentation from below link

Download Presentation The PPT/PDF document "84.305L: Low-Cost, Short-Duration Evalua..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.


Presentation Transcript

Slide1

84.305L: Low-Cost, Short-Duration Evaluation of Education Interventions84.324L: Low-Cost, Short-Duration Evaluation of Special Education Interventions

Allen Ruby

National Center for Education Research

Kimberley Sprague

National Center for Special Education ResearchSlide2

Overview Overview of IES and its mission

Requirements

Specifics

PurposeThe project narrativeSignificancePartnershipResearch PlanPersonnelResourcesOther important sections of the applicationPreparing and submitting an application

2Slide3

Legislative Mission of IES

Describe the condition and progress of education in the United States

Identify education practices that improve academic achievement and access to education opportunities

Evaluate the effectiveness of Federal and other education programs

3Slide4

Organizational

Structure of IES

4

National Board for Education Sciences

Standards & Review Office

Office of the Director

National Center for Education

Evaluation

National Center for Education

Statistics

National Center for Education

Research

National Center for Special

Education

ResearchSlide5

IES Grant Programs: Research ObjectivesDevelop or identify education interventions (i.e., practices, programs, policies, and approaches)

that enhance academic

achievement

that can be widely deployedIdentify what does not work and thereby encourage innovation and further researchUnderstand the processes that underlie the effectiveness of education interventions and the variation in their effectiveness

5Slide6

Partnerships & IES Priorities

IES seeks to...

E

ncourage education researchers to develop partnerships with stakeholder groups to advance relevance of research and usability of its findings for day-to-day work of education practitioners and policymakers

Increase capacity

of education policymakers and practitioners to use

knowledge

generated from high quality data analysis, research, and evaluation through

wide variety of communication and outreach

strategies

(See

http://ies.ed.gov/director/board/priorities.asp)

6Slide7

Short DescriptionCarried out by PartnershipsNew or established

Minimum: research institution and a state or local education agency

Purpose

Carry out rigorous evaluations of education interventions implemented by state or local education agencies High importance to the education agencyUse secondary data (e.g., administrative data)Low-cost: maximum grant of $250,000Short-duration: 2 yearsSlide8

Impetus for Low-Cost Grant ProgramTake advantage of opportunities to use administrative data to do evaluations

Provide useful information to education agencies in a more timely manner than traditional evaluations

Create additional opportunities for research institutions and education agencies to work together

Identify the strengths, weaknesses, and applicability of this type of evaluationSlide9

General RequirementsFocus on student education outcomes

84:305L: For students from prekindergarten through postsecondary and adult education

84.324L: For

infants/toddlers through students in grade 12 with or at-risk for disabilityResearch occurs in an authentic education settingEvaluate education interventions using secondary dataPartnership between research institutions and state and local education agenciesDisseminate findings in ways useful to agency decision-making Slide10

Student Population84.305L: Students from prekindergarten through postsecondary and adult

education

84.324L: Students from infants/toddlers

through grade 12 with or at-risk for disabilityA student with a disability is defined in Public Law 108-446, the Individuals with Disabilities Education Improvement Act of 2004 (IDEA)Additional requirements for identifying students at risk for developing a disabilitysee

http://

ies.ed.gov/ncser/definition.asp

10Slide11

Focus on Student Education Outcomes

Research must address education outcomes of students. For both 305L and 324L these include

Academic outcomes

Social and behavioral competencies that support student success in schoolFor 324L, these also includeDevelopmental, functional, and transitional outcomes for students with or at-risk for disability

11Slide12

Student Outcomes

Age/Grade

Outcome

Infants/Toddlers 324L

Developmental outcomes pertaining to cognitive, communicative, linguistic, social, emotional, adaptive, functional or physical development.

Prekindergarten

305L

& 324L

324L

School readiness

(e.g.,

pre-reading, language, vocabulary, early skills in STEM,

social and behavioral competencies)

Developmental

outcomes

12Slide13

Student Outcomes

Grade

Outcome

K - 12 305L & 324L

324L

Learning and achievement in reading, writing, and STEM;

Progress through the education system (e.g.,

course and grade completion

or

retention, high school graduation, and dropout);

Social and behavioral competencies important to academic and

post-academic success.

Functional outcomes that improve educational results;

Transitions to employment, independent living, and postsecondary education.

13Slide14

Additional 305L Student Outcomes

Grade

Outcome

Postsecondary(Grades 13 – 16) (baccalaureate and sub-baccalaureate)

Access to, persistence in, progress through, and completion of postsecondary education; f

or students in developmental programs, additional outcomes include achievement in reading, writing, English language proficiency, and mathematics;

success in

gateway STEM courses

and

introductory

English composition

Adult Education

(Adult Basic Education, Adult

Secondary Education

, Adult ESL, and HS equivalency preparation)Student achievement in reading, writing, English language

proficiency, and mathematics; access to, persistence in, progress through, and completion of adult education programs

14Slide15

Education InterventionsThe wide range of education curricula, instructional approaches, professional development, technology, and practices, programs, and policies that are implemented at the

child/student

, classroom, school, district, state, or federal level to improve student education

outcomesThe intervention is of high importance to the SEA or LEAThe implementation of the intervention is managed or overseen by the SEA or LEA (not just allowing a researcher or organization to implement the intervention)Implementation of the intervention will occur in Year 1 of the projectThe intervention is expected to produce meaningful improvement in student education outcomes within a short period (e.g., within a quarter, semester, or year)

At a minimum, the administrative data (or other secondary source) contains student education outcomesSlide16

Applications must be from a Partnership

Applications

must include at least one Principal Investigator

(PI) from a research institution and at least one PI from a U.S. state or local education agency

PI from research institution

:

Must have the ability and capacity to conduct scientifically valid

research and expertise in the education issue to be addressed

PI from state

or local education

agency

: Should have decision-making authority for the issue within his or her agency

16Slide17

PartnershipPartnership may be new or existingResearch institution has a broad

definition

Ability and capacity to conduct scientifically valid research

17Slide18

Partnership: SEAsState education agencies Examples: education agencies, departments, boards, commissions

Oversee

early

learning, elementary, secondary, postsecondary and/or adult educationFor 324L, oversee infant and child care, and/or early intervention services Also includes education agencies in

District of Columbia, the Commonwealth of Puerto Rico, and each of the outlying

areasSlide19

Partnership: LEAsLocal education agencies which are primarily public school districts

County

or city agencies that have primary responsibility for infant and child care (324L), early intervention services (324L), or prekindergartenCommunity college districtsTribal education agencies

19Slide20

Partnerships: LEAs (continued)State and city postsecondary systemsA public postsecondary education system can apply as the education agency partner

If there is

a

state or city education agency that oversees the postsecondary system, the application will be stronger if they are also part of the partnershipA postsecondary system that applies as an education agency partner cannot also serve as the research institution partner in the same projectAdult education providers (defined under WIOA) can serve as the partner when there is no state or local education agency for adult educationSlide21

Additional PartnersPartnerships may include more than one state

or local education

agency if they share similarities and interests

Intermediary/service districts that provide services to multiple districts but do not have decision-making authority over implementing programs and policies, Non-education state and local agencies may be partners as long as an education agency is a partnerPartnerships may include more than one research institution if they have shared interests and make unique contributionsPartnerships may include other non-research organizations (e.g., issue-oriented or stakeholder groups) that will contribute to the partnership and its work

21Slide22

DisseminationApplicants must

describe their plans to disseminate the findings from their project in Appendix

A

of their application.Required dissemination throughout the partner agencyAgency-wide oral briefingA free non-technical written brief available to the publicDissemination to other education agenciesPresentations at practitioner and policymaker meetings and publications in practitioner and policymaker journalsToolkit or guide for other education agencies on how to conduct a similar

study

Dissemination to the research community

Academic presentations and publications in peer reviewed journalsSlide23

Check the Fit of Your Research and Low-Cost, Short-Duration Evaluation Grant ProgramIf

you are not looking at

student outcomes, then IES is not the appropriate funding agency

If you need time and effort to build a partnership and prepare for an evaluation, consider:Researcher-Practitioner Partnerships under 84.305HIf the intervention you want to evaluate Is not implemented by a state or local education agency,Cannot be evaluated using secondary data, Will not be implemented in Year 1 of the project, orIs not expected to improve student outcomes within a short period (e.g., a quarter – a year).

Then consider the:

Education Research Grants Program (84.305A) or

Special Education Research Grants Program (84.324A)Slide24

Low-Cost Evaluation: Purpose

Promote joint evaluation research by research institutions and state and local education agencies

On an education intervention identified as having great importance by the education agency

That includes practitioner input into the researchThat will provide timely rigorous evidence for the agency’s decision-making regarding the intervention

And the results of which will be broadly disseminated in ways easily accessible throughout the agency partner and its stakeholders, and to other practitioners, researchers and the public

24Slide25

What should the partnerships do prior to and during the grant?

Identify a specific

education

intervention (prior)Implemented by an SEA or LEA in Year 1 of projectOf high priority to that agency

Intended to improve

student

education outcomes within a year

Carry

out

an evaluation of that intervention

(during)

Using a RCT or RDD design (or a SCD for NCSER applications)

Using secondary dataEstimate overall impactsIf data are available, you should consider:Estimating subgroup impacts for important subgroups

Examining other moderators and mediators of interest, fidelity of implementation, and comparison group practice

25Slide26

Expected Products of the GrantCausal evidence of the impact of a clearly specified

intervention

implemented by

an SEA or LEAOverall impactsImpacts for available subgroups of interestAdvice for the SEA or LEA Continuing and/or expanding the use of the interventionFurther research needs, e.g., Evaluation, e.g., variation in impacts, moderation and mediation, generalizability, replication

Development, e.g., modifications to the intervention or its implementation

26Slide27

The Project NarrativeSignificancePartnership

Research Plan

Personnel

Resources

27Slide28

SignificanceThe education intervention to be evaluated

The education problem/issue the intervention is to address within the SEA/LEA

Relevance to other SEAs or LEAs (secondary importance)

Components of the interventionRationale for why the intervention can improve student outcomes within a short period (e.g., 1 quarter - 1 year)May include theory of changeDifference from status quoRelated findings from previous studies and how this study will improve upon past workSlide29

SignificanceThe implementation of the intervention

Who will implement it and how will it be implemented

The

education agency will implement or will oversee implementationAdequate funding available for implementationImplementation during Year 1 of the project at a level expected to impact student outcomesSources of secondary data to be used in the evaluationHow these data are collected

How these data will be obtained

by researchers by the 1

st

quarter of Year 2 of

the projectSlide30

The Project NarrativeSignificancePartnership

Research Plan

Personnel

Resources30Slide31

PartnershipDescribe the partnersThe research institution and the education agency

Offices or divisions within the agency whose cooperation is necessary

Any other members of the

partnershipCommon interest in and benefit from this evaluationThe process through which the partners determined the specific intervention to evaluateData sharing agreement – the strategy to obtain the secondary data and provide it for analysis by the 1st quarter of the second yearSlide32

The Project NarrativeSignificancePartnership

Research Plan

Personnel

Resources32Slide33

Research PlanState research questions and hypotheses

Describe

sample and setting

Define population and how your sample and sampling procedures will allow inferences to the populationExclusion and inclusion rules and their justificationStrategies used to increase participation and reduce attritionDescribe the setting and its implications for

the generalizability of your study

33Slide34

Research Plan: DesignDiscuss how design will support causal inferences and identify potential

t

hreats

to internal validityDiscuss how degree of equivalence at baseline will be determinedDiscuss possibility of bias from overall and differential attrition305L: Required use of RCT or RDDPotential to meet WWC evidence standards without reservations

324L:

Required use of

RCT, RDD, or Single-Case Experimental Design

34Slide35

Research Plan: DesignRandomized Controlled

Trial

(RCT)

Note unit of randomization and justify choiceDescribe process for random assignment and maintaining its integrityDifferent Approaches to RCTs - Potential IssuesEntire population-mandatory: Treatment fidelity

Volunteers: Comparison group status

Lotteries: Attrition of non-accepted parties

Staggered roll out: Little time for true comparison

Variations of program/policy: Issue of overall significance

35Slide36

Research Plan: DesignRegression Discontinuity Design (RDD)

Appropriateness of assignment variable

Show true discontinuity

Discuss possibility of manipulation of design variable and analyses to determine such manipulationSensitivity analyses to assess influence of key procedural or analytic decisions on results

36Slide37

Research Plan: Design

Single-Case Experimental Design (324L Only)

Justify the use of a single-case experimental design as opposed to an RCT or RDD (e.g., a focus on students with a low-incidence disability)

Describe the repeated, systematic measurement of a dependent variable before, during, and after the active manipulation of an independent variable (i.e., intervention)Include outcome measures that are not strictly aligned with the interventionDescribe any quantitative analytic techniques, in addition to visual analysis, for analyzing the resulting data (e.g., between-case effect size calculations)

37Slide38

Research Plan: Statistical PowerDetailed description of

power analysis

Justify method

used to calculate powerJustify parameters used and assumptions madeProvide power for main analyses and important subgroup analysesAlong with identifying minimum detectable effect for your analysis, justify itsReasonablenessPractical meaning

Reviewers

should be able to check power

calculations

38Slide39

Research Plan: Outcome MeasuresStudent education outcome measures relevant

to states, districts, and

schools

Found in administrative data or other secondary dataDiscuss reliability, validity, and appropriatenessMust be collected during Year 1 of projectAdditional data from previous years of intervention’s implementation may also be used if appropriate to the evaluation designClearly link measures to rationale for the intervention

39Slide40

Research Plan: Optional MeasuresIf available, describe measures ofIntermediate outcomes

Moderators (subgroups expected)

Mediators (intermediate outcomes)

Fidelity of implementationComparison group practiceSlide41

Research Plan: Analysis

Detail impact analyses

Make clear how analyses directly answer your research questions

Show that analyses are based on the designAddress clustering of students in classrooms in schools Address missing data

If multiple datasets are to be linked, detail how this will be done

Describe any other analyses to be done (e.g.,

s

ubgroups, other moderators, mediators

,

and

fidelity of

implementation)

41Slide42

The Project NarrativeSignificancePartnership

Research Plan

Personnel

Resources42Slide43

PersonnelIdentify all key personnel on the project team

The PI from the research institution

The PI from the education

agency Other key personnelRoles and responsibilities on the projectEach individual’s roles and responsibilities on the projectTheir qualifications (i.e., expertise and experience) for their roleTheir % FTE on the project

Past success at working in similar partnerships

PI qualifications for managing a grant of this type

Ensure objectivity of evaluation

43Slide44

ResourcesDescribe

the institutional resources of all the institutions involved in the partnership and how these resources will contribute to

building

the partnership and to the researchInstitutional capacity to manage the grantResources available at the partner institutions to support the projectPlans to acquire any major resources not yet in hand (e.g., secondary data)Joint Letter of Agreement by partners (Appendix E)

Letter of Agreement to provide administrative data (App. E)

Resources to carryout the Dissemination Plan

44Slide45

Other Important Sections of the ApplicationAppendix AAppendix B

Appendix C

Appendix D

Appendix EBudget & Budget Narrative45Slide46

Appendix A: RequiredDissemination Plan Required: dissemination

throughout the partner agency

Agency-wide oral briefing

A free non-technical written brief available to the publicDissemination to other education agenciesPresentations at practitioner and policymaker meetings and publications in practitioner and policymaker journalsDissemination to the research communityAcademic presentations and publications in peer reviewed journalsSlide47

Appendix B (required for resubmissions)

If you are resubmitting an application,

discuss how you responded to reviewer comments47Slide48

Appendix C (Optional)

Figures

, charts, or tables that supplement the project

narrativeTimelines for the project (very useful)Examples of instruments used in the collection of the administrative or other secondary sources of dataDo NOT include narrative textSlide49

Appendix D (Optional)

Examples

of materials used in the

intervention:curriculum materialscomputer screen shotstraining documentsassessment itemsother materialsDo

NOT

include narrative text

49Slide50

Appendix E (Required)Required Letters

of

Agreement

Joint Letter from the research institution and the SEA/LEADocument participation and cooperation in the partnershipSet out each’s roles and responsibilities under the projectLetter from the office in charge of the agency’s dataProject will have access to data required in time to do analysisOptional Letters of AgreementSeparate Letters from other organizations taking part

Letters from any consultants and schools taking part

50Slide51

Budget & Budget Narrative

Maximum project length is 2 years

Maximum award is $250,000

Funds must be used for evaluation only (e.g., cannot be used for implementation of the intervention)Award size depends on project scopeInclude a detailed budget form (SF 424) AND a budget narrative that links the activities, personnel, etc. from the Project Narrative to the funds requested

51Slide52

Preparing Your ApplicationImportant datesInformation sourcesRead the RFA

Talk with a program officer

Review processSlide53

Important Dates & Deadlines

53

Important Dates/Deadlines are listed in the Request for Applications

Letter of IntentApplication package postedApplication deadline

Possible start datesSlide54

Information SourcesRequest for Applications

http://ies.ed.gov/funding/

Abstracts of Projectshttp://ies.ed.gov/funding/grantsearch/index.asp Application Package

www.grants.gov

Program Officers

Kimbereley.Sprague@ed.gov

84.324L

Phill.Gagne@ed.gov

84.305L

Allen.Ruby@ed.gov

84.305L

54Slide55

Peer Review(Standards & Review Office)

Compliance screening for

required parts

Responsiveness screening for program requirementsAssignment to review panel2 to 3 reviewers (substantive and methodological)The most competitive proposals are

reviewed by full panel

Many panelists will be generalists to your topic

Panels contain experts in relevant methodologies

Panel provides an overall

score plus

specific scores

on Significance,

Partnership, Research Plan, Personnel, and

Resources55Slide56

Notification

All applicants will receive e-mail notification that the following information is available via the Applicant Notification System (ANS):

Status of award

Reviewer summary statementIf you are not granted an award the first time, consider resubmitting and talking with your Program Officer

56Slide57

For More Information

http://ies.ed.gov/funding

Kimberley Sprague

National Center for Special Education ResearchKimberely.Sprague@ed.gov

Phill Gagne

Allen

Ruby

National Center for Education Research

Phill.Gagne@ed.gov

Allen.Ruby@ed.gov

57