Understanding SBA Scores From the Assessment development Team - PowerPoint Presentation

UnderstandingSBA Scores From the Assessment development Team

Introduction Purpose : to answer some basic questions about Smarter Balanced Summative and Interim test scoresAudience: DACs, principals, data/instructional coaches, teachers, parentsShare: staff meetings, professional development, PLC meetings 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 2

Summative Test Scores For each content area (ELA/Literacy and Mathematics), Washington reports: Overall scale score Achievement levelClaim achievement category4/26/2017OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 3

Summative Test Scores Family Report/ Individual Student Report 4/26/2017OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION4

Summative Test Scores—scale scores 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 5

Summative Test Scores—scale scores Scale scores: Are not determined only by a calculation of “raw-points-earned divided by total-points-possible.”Are related to the difficulty of the items a student was presented with.4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 6

.. . . . . . . . Calli . . . . . . . .. Beatrice . . . . . Anya Easy items Difficult items 2000 2500 3000 Summative Test Scores—scale scores

Summative Test Scores—achievement levels 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 8

Summative Test Scores—achievement levels Scale scores are divided into four categories of performance called “achievement levels” (Levels 1, 2, 3, and 4)Achievement levels are defined by Achievement Level Descriptors, the specifications for what knowledge and skills students display at each level.The detailed Achievement Level Descriptors are available here:http://www.k12.wa.us/assessment/StateTesting/PLD/default.aspx The score that divides between two levels is called a “cut score.”The cut score that divides achievement Level 2 from Level 3 is referred to as the “standard.” Students with scores above this cut have “met the achievement standard.” 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 9

Summative Test Scores—achievement levels Example of ELA grade 7: 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION10 https://www.smarterbalanced.org/assessments/scores/

Summative Test Scores—achievement levels Educators and other stakeholders from Smarter Balanced member states participated in achievement level setting meetings in October 2014 to recommend cut scores. In December 2014, the state education chiefs voted to approve the cut scores for Smarter Balanced. Washington’s State Board of Education accepted these cut scores for Washington students in January 2015.More information about the process, and WA’s involvement can be found:https://www.smarterbalanced.org/assessments/scores/ http://www.k12.wa.us/SMARTER/Achievement.aspx 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 11

Summative Test Scores—SEM Each student scale score is reported along with a Standard Error of Measurement value (SEM). This is the ± value displayed next to the 4-digit scale score. This value can be different for each student.This value is dependent on the items a student saw, which ones were answered correctly, and those the student did not answer correctly.The SEM:Indicates a range where the student’s score is likely to be if they took the test several timesCould also be called the “margin of error” or “standard error”4/26/2017OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 12

Summative Test Scores—SEM 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 13

Summative Test Scores—SEM An SEM is also included with state, district, school, and classroom/roster average scores in the Online Reporting System.The smaller the group of students, the larger the SEM.4/26/2017OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION14

Summative Test Scores—SEM Average comparisons students 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 15 S tate District School Classroom

Summative Test Scores--Claims 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 16

Summative Test Scores—Claims Claim reporting category results come from comparing the student’s claim scale score to the “standard” cut score, and incorporating the student’s SEM on the items in that claim. Step 1: Break the items up into their claim categories:Reading—15Writing—13Listening—8Research—9 4/26/2017OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 17

Summative Test Scores—Claims Step 2: Calculate the scale score for only the items in the claim category. This is called the claim scale score.4/26/2017OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 18 Research Listening Writing Reading 2780

Summative Test Scores—Claims Step 3: Find the SEM for each of these claim scale scores. These claim scale scores and claim SEM can be found in score files downloaded from WAMS and downloaded from ORS.4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 19 ± 50

Summative Test Scores—Claims Step 4: Get the “standard cut score” for the grade level and content area and do some math:standard cut score + (Claim SEM x 1.5) = high standard cut score - (Claim SEM x 1.5) = low 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 20 2552 + (50 x 1.5) 2552 - (50 x 1.5) 2552 + (75) 2552 - (75) 2627 2477 David’s Reading c laim scale score = 2780

Summative Test Scores—Claims Step 5: Compare the answers from step 4 to the claim scale score.4/26/2017OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION21 low high low from #4 = 2477 At/Near Standard Above Standard Below Standard David’s Reading c laim scale score = 2780 high from #4 = 2627

Summative Test Scores—Claims Step 5: Compare the answers from step 4 to the claim scale score.4/26/2017OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION22 low high low from #4 = 2477 At/Near Standard Above Standard Below Standard David’s Reading c laim scale score = 2780 high from #4 = 2627

Summative Test Scores—Claims 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION Another way to illustrate:23 Claim SEM x 1.5 Claim SEM x 1.5 At/Near Standard Below Standard Above Standard low high

Summative Test Scores—Claims 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION The larger the SEM, the larger the “at/near” range:24 Claim SEM x 1.5 At/Near Standard Below Standard Above Standard Claim SEM x 1.5

Summative Test Scores—Claims Above Standard: student’s claim scale score > 1.5 SEMs above the “standard” cut scoreBelow Standard: student’s claim scale score > 1.5 SEMs below the “standard” cut scoreAt/Near Standard: student’s claim scale score within 1.5 SEMs of the “standard” cut score 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 25

Summative Test Scores Scores are not determined by a calculation of “raw-points-earned divided by total-points-possible”Scores are related to:the particular set of items the student saw,the difficulty of those items,which of those items the student answered correctly,the standard error value (SEM) of the scale score, and the distance between the scale score and the “standard cut score.” The achievement levels (1, 2, 3 and 4) are defined by Achievement Level Descriptors . 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 26

Interim Comprehensive Assessment The ICAs are reported using the same three types of scores as the Summative: Overall scale score Achievement levelClaim achievement categoryThe scores are determined the same way as the Summative scores described on the previous slides.4/26/2017OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 27

Interim Assessment Block (IAB) Scores The IABs are reported using a “performance category”: Above At/NearBelowThis is very similar to the “claim achievement category” on the Summative.The IAB performance categories are determined the same way as described on the previous slides for the summative claim score categories:4/26/2017OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 28 Raw score/item difficulty Scale score and SEM Compare to standard cut Performance category

Interim Assessment Block (IAB) Scores 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 29Raw score/item difficulty Scale score and SEM Compare to standard cut Performance category

Interim Assessment Block (IAB) Scores Percentage of points earned Scores are not determined by a calculation of “raw-points-earned divided by total-points-possible”4/26/2017OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION30

Interim Assessment Block (IAB) Scores Item difficulty across the content The IABs are groups of items about similar topics. IABs with harder content likely have more items with higher difficulty values.Raw points on one IAB topic cannot be compared to raw points on another IAB topic.4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 31

Interim Assessment Block (IAB) Scores Standard Error of Measurement value (SEM) The smaller the number of items on a test, the greater the SEM. IABs with a small number of items (5 or 6) have the largest SEM.4/26/2017OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION32

Now what? Look past the colors and labels Think about…. Were there items that all your students struggled with?Were there items that all of your students did well on?Where are the outliers—items that all but a few students did well on? What instruction would benefit those few?Were there trends in answers based on particular types of items?What did you notice while hand scoring in THSS?4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 33

Now what? Remember: T he IABs are one tool that can be used to gather evidence about a student’s understanding. The teacher should also gather additional evidence through formative practices. The IABs are not intended to be all the evidence that is collected, but they can be used to inform an educator’s thoughts about a student’s understanding.  Several data points (including the IAB) should be used to formalize a conclusion about a student’s understanding .4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 34

Interim Test Scores Interim Comprehensive Assessment scores are calculated the same way as the Summative Assessment scores.Interim Assessment Block performance categories are calculated the same way as the claim achievement categories.Focus on the items and patterns of student responses to those items.4/26/2017OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 35

Resources and reminders FAQ document posted along with this video on the WCAP Portal includes:Links included in this presentationAnswers to frequently asked questionsLinks to documents for further, more technical informationAudience: DACs, principals, data/instructional coaches, teachers, parentsShare: staff meetings, professional development, PLC meetings4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 36

Thank you! Please direct questions and comments about this presentation to: Kara Todd, Content Coordinator for Test Development kara.todd@k12.wa.us Please direct content specific questions to:Anton Jackson, Mathematics Assessment Specialistanton.jackson@k12.wa.us Shelley O’Dell, English Language Arts Assessment Specialistshelley.odell@k12.wa.us 4/26/2017 OFFICE OF SUPERINTENDENT OF PUBLIC INSTRUCTION 37