Garron Gianopulos PhD North Carolina Department of Public Instruction Marianne Motley Ohio Department of Education John White PhD SAS Instititute June 25 2014 Moving Beyond Math amp Reading 38 How ValueAdded Models Can Incorporate Other Types of Assessments ID: 692254
Download Presentation The PPT/PDF document "Moving Beyond Math & Reading 3-8: Ho..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Moving Beyond Math & Reading 3-8: How Value-Added Models Can Incorporate Other Types of Assessments
Garron Gianopulos, PhDNorth Carolina Department of Public InstructionMarianne Motley, Ohio Department of EducationJohn White, PhDSAS, InstitituteJune 25, 2014Slide2
Moving Beyond Math & Reading 3-8: How Value-Added Models Can Incorporate Other Types of Assessments
Marianne Mottley, Assistant Director, Office of AccountabilityOhio Department of EducationSlide3
“A principal or a superintendent must be
able to say to the school board and the public:Everyone who teaches here is good – and here’s how I know.”- Charlotte DanielsonSlide4Slide5
Improve teaching and learning
Ultimate goal of teacher evaluationSlide6
How do we get there?Slide7
Evaluation Framework
AccomplishedSkilledDevelopingIneffectiveFinal Summative Rating
Teacher Performance on Standards 50%
Student Growth Measures 50%Slide8
RttT Teacher-Student Linkage
Spring 201130% of Ohio’s LEAs ParticipatedSpring 201260% of Ohio’s LEAs Participated
Spring 2013
100% of Ohio’s LEAs ParticipatedSlide9
Linkage
Required Linkage Grades 4-8 - Reading - MathOptional Linkage for Extended Testing - Grades K-3 reading and math - Grades K-4, 6, 7 science - Grades K-8 social studies - Grades 9-12 all content areasSlide10
Extended Testing Programs
RttT Mini-Grant - Grades 1-3 R/M - Grades 2-3 Science - Grade 3 Social StudiesProject SOAR - Grade 3 R/M - Grades 3-8 Social Studies - Grades 3, 4, 6, 7 Science - High School EOC ExamsSlide11
RttT Mini-Grant
Not a mandatory part of RttTDistricts could opt in by submitting a mini-grant applicationDistrict agreement with ODETwo-Year Grant = Two-Year CommitmentSlide12
RttT Mini-Grant Overview
Grant pays for assessments in grades K-3 and for teacher value added reportsPools created based on which assessments were most widely usedThree pools in Ohio - Terra Nova (CTB McGraw Hill) - MAP (NWEA) - STAR (Renaissance Learning) Slide13
Mini-Grant Overview
Univariate Response Model (URM) Predicted Mean Approach Tests Properties - High correlation to curricular objectives - Have sufficient stretch - Must be sufficiently reliableSlide14
Mini-Grant Overview
Univariate Response Model (URM) Minimum 3 prior year’s or years’ tests (predictor tests) Pool must have same predictors Pool must have same current year tests (response tests)Slide15
RttT Mini-Grant Overview
Predictor Testing RequirementsTerra Nova - Spring window in at least 3 subjects - Reading and math mandatory - Science or social studies optionalMAP/STAR - Fall & spring windows in at least 2 subjects - Reading and math mandatorySlide16
Mini-Grant Overview
Some districts had prior test data and received reports in 2013For others, 2014 will be their first year when reports will be generatedPools must remain stable across yearsDistricts cannot deviate from a poolSlide17
Mini-Grant Overview
Reports must be used for teacher evaluationConsidered to be a vendor assessment for the evaluation systemMust be used for at least 10% of the 50% that comprises student growth measures Slide18
Project SOAR
Battelle for Kids Project SOAR Created in 2002 with 42 districts 2014 over 100 SOAR districts Uses value-added data to drive decision making and enhance student learning Must use for teacher evaluationsSlide19
Project SOAR Grades 3-8
Predictor DataResponse DataSlide20
Project SOAR High School
Uses ACT Quality CoreEnd-of-Course ExamsFour Content AreasMathematicsAlgebra IGeometryAlgebra IIPre-CalculusEnglish/Language ArtsEnglish 9English 10English 11English 12ScienceBiologyChemistry I
Social StudiesU.S. HistorySlide21
Other BFK Clients
NWEA and possibly other vendorsUsing “single year” Multivariate Response Model (MRM) for Year 1 - Uses fall tests as baseline data - Uses spring tests to measure growthUsing “across years” MRM approach for Year 2 and beyondSlide22
Other BFK Clients
Districts will partner directly with BFKExact grades/subjects yet to be determinedGrade 3 reading and math will use Ohio’s achievement assessments as response (using URM calculation)Slide23
education.ohio.govSlide24
Social Media
@OHEducationohio-department-of-educationOhio Families and EducationOhio Teachers’ HomeroomOhioEdDeptstorify.com/ohioEdDeptSlide25
Moving Beyond Math & Reading 3-8: How Value-Added Models Can Incorporate Other Types of Assessments
Garron Gianopulos, PhDPsychometrician, Technical Development of Statewide Assessments, Design and Implementation of Tests, North Carolina Department of Public InstructionJune 25, 2014Slide26
Overview
A Year of ChangeHow NC Uses EVAASOutcomes and Plans for the FutureSlide27Slide28
2012-13: A Year of Change
2005: Implemented new growth formula28READY AssessmentsIntroduced all new READY end-of-grade (EOG) Assessments in grades 3 through 8New READY end-of-course (EOC) assessments
All EOGs and EOCs were aligned to the common core state standards, adopted in 2010All new standards classifying students as ready for the next grade level and on-track to college
New proficiency levelsSlide29
2012-13: A Year of Change
2005: Implemented new growth formula29New LawsRead to Achieve Law required third graders to demonstrate proficiency in reading
35 new third grade portfolio reading assessments
Beginning-of-grade 3 Reading assessment
School L
etter
G
rades
Grade letter determined by proficiency and EVAAS growthSlide30
2012-13: A Year of Change
2005: Implemented new growth formula30Race to the Top Assessments for Teacher EvaluationNew Teacher Effectiveness model required teachers to demonstrate growth6 New Occupational
Course of Study (OCS) assessments
30 New NC Final Exams
Science exams in grade 4,6,7, and high school
New H.S. Math exams
Social Studies exams in grade 4 – 9 and high school
Historically
non-tested
subjectsSlide31
2012-13: A Year of Change
2005: Implemented new growth formula31State Board of Education (SBE)Not all board members were comfortable with the new proficiency levelsA policy decision was made to change original 4 achievement levels to 5 achievement levels
The new level differentiates between readiness for next grade level and being on-track for college
NC is a governing member of the Smarter Balanced Assessment Consortium (SBAC); however, the SBE is re-evaluating consortia and future testing optionsKeeping current READY assessments through 2015-16Slide32
Overview
A Year of ChangeHow NC Uses EVAASOutcomes and Plans for the FutureSlide33
How NC Uses the Educator Value Added Assessment System (EVAAS)
33
Educator Instructional Planning
School Accountability
Educator EffectivenessSlide34
Instructional Planning
2012-13: EVAAS provides projections of EOG/EOC percentile scores at the beginning of the school year to help with instructional plansStudent-level EVAAS projections are also provided for ACT, PLAN, and EXPLORE 2013-14: EOG/EOC Scale score projections will be providedSlide35
35
The School Accountability SystemSlide36
We have a
total of 6 standards in our teacher evaluation system. All standards, 1-6, are of equal value. Our goal:Identify our strongest teachers and explore their methodologies, andSupport teachers who need to increase their effectivenessThree years of EVAAS growth measures needed before ratings applyNorth Carolina Educator Evaluation ProcessTeachers
1
2
3
4
5
6
Establish Environment
Know
Content
Facilitate
Learning
Demonstrate Leadership
Reflect on
Practice
Contribute to Academic SuccessSlide37
Training Provided
37
Webinars with EVAAS team focused on how staff will continue to get value-added data with a change in standardsUnderstanding the EVAAS system of value-added growth
Conversion of assessment data to Normal Curve Equivalents (NCE’s)
Understanding the use of assessments in various growth modelsSlide38
EVAAS Models Used in NC
38
Multivariate Response Model (MRM)17 End-of-grade
Math and ELA assessmentsUsed in the school accountability system
Univariate
Response Model (URM)
3 End-of-course assessments
6 Occupational Course of Study assessments
70 Career
and Technical
Exams
30 NC
Final
Exams (historically non-tested subjects)
All available assessments used in educator effectivenessSlide39
What Types of
Scores Can be Used in EVAAS?39
Sufficient spread in the scale
Limited ceiling
or floor effects Correlation between covariates and
score
> .60
High reliability
Test
content needs to align closely to the instructed
curriculum
H
igh Validity
Standardized
assessments need curricula that are also standardizedSlide40
Overview
A Year of ChangeHow NC Uses EVAASOutcomes and Plans for the FutureSlide41
Outcomes from 2012-13
41
READY EOG/EOC AssessmentsPublic
backlash against quantity of
testing (especially in grade 3), but not against our chosen growth modelAll
READY EOG/EOC assessments
and most Final
Exams had the required properties to
be used by
EVAAS
No changes were needed to the READY EOG/EOC assessments as these were developed over a three year time
frame with sufficient funding
The legislatively-driven calculations for school letter grades in the Accountability Model generated a disproportionately large percent of Fs and DsSlide42
Outcomes from 2012-13
42
Final ExamsCertain Final Exams and OCS assessments did not meet requirements to be utilized in the EVAAS models
Insufficient sample size
Insufficient correlations between covariates and test score
Insufficient reliabilities
Floor
effects surfaced on some High School Math
assessmentsSlide43
Outcomes from 2012-13
43
Constructed Response Items (CR)NCDP provided general guidelines for managing CR item scoring process, but each district implemented their own process
Constructed response (CR) items too labor intensive to score by teachers
Many concerns about the
security, inter-rater reliability, and accuracy
of teacher-scored CR
itemsSlide44
Outcomes from 2012-13
44
Response from EducatorsThe
large majority (79%) of teachers met or exceeded growth expectationsTeachers with insufficient individual estimates were excluded from the rating system
Few groups are comparing their own growth measures to EVAASMost educators seem to be responding favorably to the EVAAS reportsSlide45
Changes in
2013-1445
Managerial ChangesNew chief in charge of the test development section
The test development section is now managing the test development process for all Final Exams
Increased test development staff to manage work loadSlide46
Changes in
2013-1446
Final ExamsName was changed from Common Exams to Final Exams to emphasize that they are intended to replace teacher-made final exams
Testing
time increased from 90 minutes to 120 minutesReduced the number of Final Exams with CR items
Reduced the number of CR items per exam
Tests lengthened with MC items to increase
reliability
CR items scored by contractor rather than by NC
teachersSlide47
47
Final Exams
Moving many of the Final Exams to online administrationImplementing common item non-equivalent equating design
Experimental items embedded to provide greater statistical control in form assembly
Reduced difficulty of some Math High School tests to improve reliability and remove floor effects
Changes in 2013-14Slide48
48
Providing more detail in test specifications to help teachers know what to instruct in those subjects where the curriculum standards are not specific enough
The EVAAS team will determine if they can use the beginning-of-grade 3 assessment to measure growth
Changes in 2013-14Slide49
2013-14: More Change Ahead?
2005: Implemented new growth formula49The weight given to EVAAS growth within the school Accountability Model will likely increase to produce a more acceptable distribution of letter grades Two bills approved by senate and the house to rescind the Common Core State Standards
Will funding for Final Exams remain at the same level?Slide50
Questions
Every Student READY