84324L LowCost ShortDuration Evaluation of Special Education Interventions Allen Ruby National Center for Education Research Kimberley Sprague National Center for Special Education Research ID: 669185
Download Presentation The PPT/PDF document "84.305L: Low-Cost, Short-Duration Evalua..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
84.305L: Low-Cost, Short-Duration Evaluation of Education Interventions84.324L: Low-Cost, Short-Duration Evaluation of Special Education Interventions
Allen Ruby
National Center for Education Research
Kimberley Sprague
National Center for Special Education ResearchSlide2
Overview Overview of IES and its mission
Requirements
Specifics
PurposeThe project narrativeSignificancePartnershipResearch PlanPersonnelResourcesOther important sections of the applicationPreparing and submitting an application
2Slide3
Legislative Mission of IES
Describe the condition and progress of education in the United States
Identify education practices that improve academic achievement and access to education opportunities
Evaluate the effectiveness of Federal and other education programs
3Slide4
Organizational
Structure of IES
4
National Board for Education Sciences
Standards & Review Office
Office of the Director
National Center for Education
Evaluation
National Center for Education
Statistics
National Center for Education
Research
National Center for Special
Education
ResearchSlide5
IES Grant Programs: Research ObjectivesDevelop or identify education interventions (i.e., practices, programs, policies, and approaches)
that enhance academic
achievement
that can be widely deployedIdentify what does not work and thereby encourage innovation and further researchUnderstand the processes that underlie the effectiveness of education interventions and the variation in their effectiveness
5Slide6
Partnerships & IES Priorities
IES seeks to...
E
ncourage education researchers to develop partnerships with stakeholder groups to advance relevance of research and usability of its findings for day-to-day work of education practitioners and policymakers
Increase capacity
of education policymakers and practitioners to use
knowledge
generated from high quality data analysis, research, and evaluation through
wide variety of communication and outreach
strategies
(See
http://ies.ed.gov/director/board/priorities.asp)
6Slide7
Short DescriptionCarried out by PartnershipsNew or established
Minimum: research institution and a state or local education agency
Purpose
Carry out rigorous evaluations of education interventions implemented by state or local education agencies High importance to the education agencyUse secondary data (e.g., administrative data)Low-cost: maximum grant of $250,000Short-duration: 2 yearsSlide8
Impetus for Low-Cost Grant ProgramTake advantage of opportunities to use administrative data to do evaluations
Provide useful information to education agencies in a more timely manner than traditional evaluations
Create additional opportunities for research institutions and education agencies to work together
Identify the strengths, weaknesses, and applicability of this type of evaluationSlide9
General RequirementsFocus on student education outcomes
84:305L: For students from prekindergarten through postsecondary and adult education
84.324L: For
infants/toddlers through students in grade 12 with or at-risk for disabilityResearch occurs in an authentic education settingEvaluate education interventions using secondary dataPartnership between research institutions and state and local education agenciesDisseminate findings in ways useful to agency decision-making Slide10
Student Population84.305L: Students from prekindergarten through postsecondary and adult
education
84.324L: Students from infants/toddlers
through grade 12 with or at-risk for disabilityA student with a disability is defined in Public Law 108-446, the Individuals with Disabilities Education Improvement Act of 2004 (IDEA)Additional requirements for identifying students at risk for developing a disabilitysee
http://
ies.ed.gov/ncser/definition.asp
10Slide11
Focus on Student Education Outcomes
Research must address education outcomes of students. For both 305L and 324L these include
Academic outcomes
Social and behavioral competencies that support student success in schoolFor 324L, these also includeDevelopmental, functional, and transitional outcomes for students with or at-risk for disability
11Slide12
Student Outcomes
Age/Grade
Outcome
Infants/Toddlers 324L
Developmental outcomes pertaining to cognitive, communicative, linguistic, social, emotional, adaptive, functional or physical development.
Prekindergarten
305L
& 324L
324L
School readiness
(e.g.,
pre-reading, language, vocabulary, early skills in STEM,
social and behavioral competencies)
Developmental
outcomes
12Slide13
Student Outcomes
Grade
Outcome
K - 12 305L & 324L
324L
Learning and achievement in reading, writing, and STEM;
Progress through the education system (e.g.,
course and grade completion
or
retention, high school graduation, and dropout);
Social and behavioral competencies important to academic and
post-academic success.
Functional outcomes that improve educational results;
Transitions to employment, independent living, and postsecondary education.
13Slide14
Additional 305L Student Outcomes
Grade
Outcome
Postsecondary(Grades 13 – 16) (baccalaureate and sub-baccalaureate)
Access to, persistence in, progress through, and completion of postsecondary education; f
or students in developmental programs, additional outcomes include achievement in reading, writing, English language proficiency, and mathematics;
success in
gateway STEM courses
and
introductory
English composition
Adult Education
(Adult Basic Education, Adult
Secondary Education
, Adult ESL, and HS equivalency preparation)Student achievement in reading, writing, English language
proficiency, and mathematics; access to, persistence in, progress through, and completion of adult education programs
14Slide15
Education InterventionsThe wide range of education curricula, instructional approaches, professional development, technology, and practices, programs, and policies that are implemented at the
child/student
, classroom, school, district, state, or federal level to improve student education
outcomesThe intervention is of high importance to the SEA or LEAThe implementation of the intervention is managed or overseen by the SEA or LEA (not just allowing a researcher or organization to implement the intervention)Implementation of the intervention will occur in Year 1 of the projectThe intervention is expected to produce meaningful improvement in student education outcomes within a short period (e.g., within a quarter, semester, or year)
At a minimum, the administrative data (or other secondary source) contains student education outcomesSlide16
Applications must be from a Partnership
Applications
must include at least one Principal Investigator
(PI) from a research institution and at least one PI from a U.S. state or local education agency
PI from research institution
:
Must have the ability and capacity to conduct scientifically valid
research and expertise in the education issue to be addressed
PI from state
or local education
agency
: Should have decision-making authority for the issue within his or her agency
16Slide17
PartnershipPartnership may be new or existingResearch institution has a broad
definition
Ability and capacity to conduct scientifically valid research
17Slide18
Partnership: SEAsState education agencies Examples: education agencies, departments, boards, commissions
Oversee
early
learning, elementary, secondary, postsecondary and/or adult educationFor 324L, oversee infant and child care, and/or early intervention services Also includes education agencies in
District of Columbia, the Commonwealth of Puerto Rico, and each of the outlying
areasSlide19
Partnership: LEAsLocal education agencies which are primarily public school districts
County
or city agencies that have primary responsibility for infant and child care (324L), early intervention services (324L), or prekindergartenCommunity college districtsTribal education agencies
19Slide20
Partnerships: LEAs (continued)State and city postsecondary systemsA public postsecondary education system can apply as the education agency partner
If there is
a
state or city education agency that oversees the postsecondary system, the application will be stronger if they are also part of the partnershipA postsecondary system that applies as an education agency partner cannot also serve as the research institution partner in the same projectAdult education providers (defined under WIOA) can serve as the partner when there is no state or local education agency for adult educationSlide21
Additional PartnersPartnerships may include more than one state
or local education
agency if they share similarities and interests
Intermediary/service districts that provide services to multiple districts but do not have decision-making authority over implementing programs and policies, Non-education state and local agencies may be partners as long as an education agency is a partnerPartnerships may include more than one research institution if they have shared interests and make unique contributionsPartnerships may include other non-research organizations (e.g., issue-oriented or stakeholder groups) that will contribute to the partnership and its work
21Slide22
DisseminationApplicants must
describe their plans to disseminate the findings from their project in Appendix
A
of their application.Required dissemination throughout the partner agencyAgency-wide oral briefingA free non-technical written brief available to the publicDissemination to other education agenciesPresentations at practitioner and policymaker meetings and publications in practitioner and policymaker journalsToolkit or guide for other education agencies on how to conduct a similar
study
Dissemination to the research community
Academic presentations and publications in peer reviewed journalsSlide23
Check the Fit of Your Research and Low-Cost, Short-Duration Evaluation Grant ProgramIf
you are not looking at
student outcomes, then IES is not the appropriate funding agency
If you need time and effort to build a partnership and prepare for an evaluation, consider:Researcher-Practitioner Partnerships under 84.305HIf the intervention you want to evaluate Is not implemented by a state or local education agency,Cannot be evaluated using secondary data, Will not be implemented in Year 1 of the project, orIs not expected to improve student outcomes within a short period (e.g., a quarter – a year).
Then consider the:
Education Research Grants Program (84.305A) or
Special Education Research Grants Program (84.324A)Slide24
Low-Cost Evaluation: Purpose
Promote joint evaluation research by research institutions and state and local education agencies
On an education intervention identified as having great importance by the education agency
That includes practitioner input into the researchThat will provide timely rigorous evidence for the agency’s decision-making regarding the intervention
And the results of which will be broadly disseminated in ways easily accessible throughout the agency partner and its stakeholders, and to other practitioners, researchers and the public
24Slide25
What should the partnerships do prior to and during the grant?
Identify a specific
education
intervention (prior)Implemented by an SEA or LEA in Year 1 of projectOf high priority to that agency
Intended to improve
student
education outcomes within a year
Carry
out
an evaluation of that intervention
(during)
Using a RCT or RDD design (or a SCD for NCSER applications)
Using secondary dataEstimate overall impactsIf data are available, you should consider:Estimating subgroup impacts for important subgroups
Examining other moderators and mediators of interest, fidelity of implementation, and comparison group practice
25Slide26
Expected Products of the GrantCausal evidence of the impact of a clearly specified
intervention
implemented by
an SEA or LEAOverall impactsImpacts for available subgroups of interestAdvice for the SEA or LEA Continuing and/or expanding the use of the interventionFurther research needs, e.g., Evaluation, e.g., variation in impacts, moderation and mediation, generalizability, replication
Development, e.g., modifications to the intervention or its implementation
26Slide27
The Project NarrativeSignificancePartnership
Research Plan
Personnel
Resources
27Slide28
SignificanceThe education intervention to be evaluated
The education problem/issue the intervention is to address within the SEA/LEA
Relevance to other SEAs or LEAs (secondary importance)
Components of the interventionRationale for why the intervention can improve student outcomes within a short period (e.g., 1 quarter - 1 year)May include theory of changeDifference from status quoRelated findings from previous studies and how this study will improve upon past workSlide29
SignificanceThe implementation of the intervention
Who will implement it and how will it be implemented
The
education agency will implement or will oversee implementationAdequate funding available for implementationImplementation during Year 1 of the project at a level expected to impact student outcomesSources of secondary data to be used in the evaluationHow these data are collected
How these data will be obtained
by researchers by the 1
st
quarter of Year 2 of
the projectSlide30
The Project NarrativeSignificancePartnership
Research Plan
Personnel
Resources30Slide31
PartnershipDescribe the partnersThe research institution and the education agency
Offices or divisions within the agency whose cooperation is necessary
Any other members of the
partnershipCommon interest in and benefit from this evaluationThe process through which the partners determined the specific intervention to evaluateData sharing agreement – the strategy to obtain the secondary data and provide it for analysis by the 1st quarter of the second yearSlide32
The Project NarrativeSignificancePartnership
Research Plan
Personnel
Resources32Slide33
Research PlanState research questions and hypotheses
Describe
sample and setting
Define population and how your sample and sampling procedures will allow inferences to the populationExclusion and inclusion rules and their justificationStrategies used to increase participation and reduce attritionDescribe the setting and its implications for
the generalizability of your study
33Slide34
Research Plan: DesignDiscuss how design will support causal inferences and identify potential
t
hreats
to internal validityDiscuss how degree of equivalence at baseline will be determinedDiscuss possibility of bias from overall and differential attrition305L: Required use of RCT or RDDPotential to meet WWC evidence standards without reservations
324L:
Required use of
RCT, RDD, or Single-Case Experimental Design
34Slide35
Research Plan: DesignRandomized Controlled
Trial
(RCT)
Note unit of randomization and justify choiceDescribe process for random assignment and maintaining its integrityDifferent Approaches to RCTs - Potential IssuesEntire population-mandatory: Treatment fidelity
Volunteers: Comparison group status
Lotteries: Attrition of non-accepted parties
Staggered roll out: Little time for true comparison
Variations of program/policy: Issue of overall significance
35Slide36
Research Plan: DesignRegression Discontinuity Design (RDD)
Appropriateness of assignment variable
Show true discontinuity
Discuss possibility of manipulation of design variable and analyses to determine such manipulationSensitivity analyses to assess influence of key procedural or analytic decisions on results
36Slide37
Research Plan: Design
Single-Case Experimental Design (324L Only)
Justify the use of a single-case experimental design as opposed to an RCT or RDD (e.g., a focus on students with a low-incidence disability)
Describe the repeated, systematic measurement of a dependent variable before, during, and after the active manipulation of an independent variable (i.e., intervention)Include outcome measures that are not strictly aligned with the interventionDescribe any quantitative analytic techniques, in addition to visual analysis, for analyzing the resulting data (e.g., between-case effect size calculations)
37Slide38
Research Plan: Statistical PowerDetailed description of
power analysis
Justify method
used to calculate powerJustify parameters used and assumptions madeProvide power for main analyses and important subgroup analysesAlong with identifying minimum detectable effect for your analysis, justify itsReasonablenessPractical meaning
Reviewers
should be able to check power
calculations
38Slide39
Research Plan: Outcome MeasuresStudent education outcome measures relevant
to states, districts, and
schools
Found in administrative data or other secondary dataDiscuss reliability, validity, and appropriatenessMust be collected during Year 1 of projectAdditional data from previous years of intervention’s implementation may also be used if appropriate to the evaluation designClearly link measures to rationale for the intervention
39Slide40
Research Plan: Optional MeasuresIf available, describe measures ofIntermediate outcomes
Moderators (subgroups expected)
Mediators (intermediate outcomes)
Fidelity of implementationComparison group practiceSlide41
Research Plan: Analysis
Detail impact analyses
Make clear how analyses directly answer your research questions
Show that analyses are based on the designAddress clustering of students in classrooms in schools Address missing data
If multiple datasets are to be linked, detail how this will be done
Describe any other analyses to be done (e.g.,
s
ubgroups, other moderators, mediators
,
and
fidelity of
implementation)
41Slide42
The Project NarrativeSignificancePartnership
Research Plan
Personnel
Resources42Slide43
PersonnelIdentify all key personnel on the project team
The PI from the research institution
The PI from the education
agency Other key personnelRoles and responsibilities on the projectEach individual’s roles and responsibilities on the projectTheir qualifications (i.e., expertise and experience) for their roleTheir % FTE on the project
Past success at working in similar partnerships
PI qualifications for managing a grant of this type
Ensure objectivity of evaluation
43Slide44
ResourcesDescribe
the institutional resources of all the institutions involved in the partnership and how these resources will contribute to
building
the partnership and to the researchInstitutional capacity to manage the grantResources available at the partner institutions to support the projectPlans to acquire any major resources not yet in hand (e.g., secondary data)Joint Letter of Agreement by partners (Appendix E)
Letter of Agreement to provide administrative data (App. E)
Resources to carryout the Dissemination Plan
44Slide45
Other Important Sections of the ApplicationAppendix AAppendix B
Appendix C
Appendix D
Appendix EBudget & Budget Narrative45Slide46
Appendix A: RequiredDissemination Plan Required: dissemination
throughout the partner agency
Agency-wide oral briefing
A free non-technical written brief available to the publicDissemination to other education agenciesPresentations at practitioner and policymaker meetings and publications in practitioner and policymaker journalsDissemination to the research communityAcademic presentations and publications in peer reviewed journalsSlide47
Appendix B (required for resubmissions)
If you are resubmitting an application,
discuss how you responded to reviewer comments47Slide48
Appendix C (Optional)
Figures
, charts, or tables that supplement the project
narrativeTimelines for the project (very useful)Examples of instruments used in the collection of the administrative or other secondary sources of dataDo NOT include narrative textSlide49
Appendix D (Optional)
Examples
of materials used in the
intervention:curriculum materialscomputer screen shotstraining documentsassessment itemsother materialsDo
NOT
include narrative text
49Slide50
Appendix E (Required)Required Letters
of
Agreement
Joint Letter from the research institution and the SEA/LEADocument participation and cooperation in the partnershipSet out each’s roles and responsibilities under the projectLetter from the office in charge of the agency’s dataProject will have access to data required in time to do analysisOptional Letters of AgreementSeparate Letters from other organizations taking part
Letters from any consultants and schools taking part
50Slide51
Budget & Budget Narrative
Maximum project length is 2 years
Maximum award is $250,000
Funds must be used for evaluation only (e.g., cannot be used for implementation of the intervention)Award size depends on project scopeInclude a detailed budget form (SF 424) AND a budget narrative that links the activities, personnel, etc. from the Project Narrative to the funds requested
51Slide52
Preparing Your ApplicationImportant datesInformation sourcesRead the RFA
Talk with a program officer
Review processSlide53
Important Dates & Deadlines
53
Important Dates/Deadlines are listed in the Request for Applications
Letter of IntentApplication package postedApplication deadline
Possible start datesSlide54
Information SourcesRequest for Applications
http://ies.ed.gov/funding/
Abstracts of Projectshttp://ies.ed.gov/funding/grantsearch/index.asp Application Package
www.grants.gov
Program Officers
Kimbereley.Sprague@ed.gov
84.324L
Phill.Gagne@ed.gov
84.305L
Allen.Ruby@ed.gov
84.305L
54Slide55
Peer Review(Standards & Review Office)
Compliance screening for
required parts
Responsiveness screening for program requirementsAssignment to review panel2 to 3 reviewers (substantive and methodological)The most competitive proposals are
reviewed by full panel
Many panelists will be generalists to your topic
Panels contain experts in relevant methodologies
Panel provides an overall
score plus
specific scores
on Significance,
Partnership, Research Plan, Personnel, and
Resources55Slide56
Notification
All applicants will receive e-mail notification that the following information is available via the Applicant Notification System (ANS):
Status of award
Reviewer summary statementIf you are not granted an award the first time, consider resubmitting and talking with your Program Officer
56Slide57
For More Information
http://ies.ed.gov/funding
Kimberley Sprague
National Center for Special Education ResearchKimberely.Sprague@ed.gov
Phill Gagne
Allen
Ruby
National Center for Education Research
Phill.Gagne@ed.gov
Allen.Ruby@ed.gov
57