summative assessment Nathan L Wall Anne H Davidson CCSSO National Conference on Student Assessment Annual Meeting Austin TX 2017 Background amp Purpose Test platforms for computerbased assessments must incorporate flexibility in how students can interact with test content ID: 681011
Download Presentation The PPT/PDF document "Evaluation of how students used online ..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Evaluation
of how students used online accessibility tools for a
summative
assessment
Nathan L. Wall
Anne H. Davidson
CCSSO - National Conference on Student Assessment Annual Meeting
Austin, TX : 2017Slide2
Background & PurposeTest platforms for computer-based assessments must incorporate flexibility in how students can interact with test content.Several online features and supports are designed to be available to all students, many of which have previously been available only as accommodations.
Test developers need guidance when specifying how content should be paired with each tool.
Empirical data can shed some light on how students accessed each tool and whether the tool was repeatedly accessed.Patterns of student behavior can help developers reflect back to decision makers what tools are effectively used.
Tools reviewed: Answer Masking, Calculator, Color Chooser, Custom Masking, Guideline Reading Tool, Notepad, Read Aloud, Reverse Contrast, Sketch, and Zoom.Slide3
Tools and Tool Use Classification3
Tool
Description
ClassificationAnswer
MaskingThe student “hides” possible answer choices (for multiple choice items only).Universal Tool
CalculatorBasic
calculator (Grades 3-6)
Scientific calculator (Grades 7-8)
Non-Universal Tool (Available for specific items.)
Color
Chooser
Students can choose a text and background color.
Accommodation
Custom Masking
Provides the ability to mask certain parts of the test interface or question.
Universal Tool
Guideline
The student uses an onscreen tool to assist in reading each line of text onscreen.
Universal Tool
Notepad
The notepad is provided for students to write a note for items.
Universal Tool
Read Aloud
(Play)
Students can play audio
to have content read aloud.
Accommodation
Reverse Contrast
Inverts all color values in the user interface.
Universal Tool
Sketch Pad
Sketch
or draw using pens or highlighters.
Universal
Tool
Zoom (Decrease)
Students can magnify the entire screen.
Universal Tool
Zoom (Increase)
Students can magnify the entire screen.
Universal ToolSlide4
Current Study MethodDescriptive data analysis of three tests (Mathematics, Reading, and Writing/Language) in grades 3-8 from a statewide testing program.Data set included a transaction record for each instance of a student’s interaction within the platform, including when the student opened or closed a tool.Analytics
Counts of the times a student accessed a given tool were computed.
Usage patterns looked at across quartiles of usage (quartile was defined as 25% of the student’s item completion).Proportion of unique students to the times a tool was accessed.
4Slide5
Results – Student Count5
GradeMathematics
ReadingWriting/Language3
11,56410,1759,074
412,33711,1759,9175
12,34311,0979,881
6
12,682
11,276
9,931
7
12,827
11,013
9,647
8
12,236
10,122
8,146
Current StudySlide6
Exploratory Study Results - 16
Tool
Percentage of Total Tools Accessed –Writing Field Test(Times Accessed Rank Across Grades 3-8)
3
4567
8Sketch Pad44 (1)57 (1)
46 (1)
49 (1)
49 (1)
54 (1)
Zoom
28 (2)
22 (2)
29 (2)
21 (2)
27 (2)
19 (2)
Reverse Contrast
20 (3)
13 (3)
14 (3)
18 (3)
12 (3)
14 (3)
Custom Masking
5
(4)
7 (4)
8 (4)
10 (4)
9 (4)
9 (4)
Guideline
2 (5)
2 (5)
2 (5)
2 (5)
3 (5)
3 (5)
Color Chooser
0 (6)
--
--
--
--
--Slide7
Exploratory Study Results - 27Slide8
Results – Number of Tools (Overall)8Slide9
Results – Number of Tools (Averaged Across Items)9Slide10
Results – Percentage of Times Tools Accessed (Mathematics)10
Tool
Percentage of Total Tools AccessedMathematics 3-8
3
4
56
7
8
Answer Masking
31
34
29
31
27
28
Calculator
9
6
10
10
19
13
Color Chooser
0
0
0
0
0
0
Custom Masking
3
4
4
4
4
5
Guideline
1
1
1
1
1
1
Notepad
3
3
3
2
2
2
Read Aloud (Play)
7
5
5
3
2
2
Reverse Contrast
6
8
9
10
10
9
Sketch Pad
30
31
30
30
26
33
Zoom (Decrease)
2
2
3
3
2
2
Zoom (Increase)
7
6
6
7
6
5Slide11
Results - Percentage of Times Tools Accessed (Reading)11
Tool
Percentage of Total Tools AccessedReading 3-8
3
4
5
6
7
8
Answer Masking
40
38
39
37
36
37
Color Chooser
0
0
0
0
0
0
Custom Masking
5
6
5
7
6
7
Guideline
3
3
2
2
2
2
Notepad
3
3
2
2
2
2
Reverse Contrast
13
12
13
14
14
12
Sketch Pad
23
25
26
27
31
32
Zoom (Decrease)
3
3
3
3
2
2
Zoom (Increase)
10
10
9
8
7
7Slide12
Results - Percentage of Times Tools Accessed (Writing & Language)12
Tool
Percentage of Total Tools AccessedWriting and Language 3-8
3
4
56
7
8
Answer Masking
53
50
52
45
46
50
Color Chooser
0
0
0
0
0
0
Custom Masking
4
5
4
5
6
5
Guideline
2
2
1
1
1
1
Notepad
2
2
2
2
2
1
Reverse Contrast
9
11
10
13
13
12
Sketch Pad
19
20
20
25
25
26
Zoom (Decrease)
2
2
2
2
2
2
Zoom (Increase)
8
8
7
7
6
6Slide13
Results – Proportion of Unique Students to Times Accessed13Slide14
Results – Tool Usage Across Test quarters – Grade 3 Example14Slide15
Results – Tool Usage Across Test quarters – Grade 7 Example15Slide16
SummaryAs a reminder, 2 studies are presented here. The findings from the current study were consistent with the exploratory one.Over the course of the test most students 4 or fewer tools but on average 80% used only one tool per item.Answer Masking and Sketch Pad were the most popular tools.A higher proportion of students to tool use were found for the guideline tool and notepad, indicating a small group of students accounted for the tool’s use.
A low proportion was found for the answer masking and sketch pad, indicating many students used these tools.
Though not consistent across grades and subjects, students did use the tools a bit more in the beginning portion of the test rather than evenly throughout.eMetric, LLC Confidential
16Slide17
ImplicationsReports of tool usage should be provided to inform test designers and decision makers.Training and professional development could take this information into account so that students were aware of all of the tools they may have access to.Special educators may find value in learning how their students with IEPs used the tools so that they might have the option to intervene if tool usage is not aligned to how students were instructed to use such tools.
Such reports and studies could aid validity arguments for test score interpretations as well.
eMetric, LLC Confidential 17Slide18
Tool Usage Reporting FrameworkAnalyzing tool use can be thought of in 3 categories.1. General tool use patterns at the test level.Most of the analyses presented here, fall into this category.2. Tool use added to traditional item analyses.
eMetric, LLC Confidential
18
Item ID12
Item Pos.
Status
Key
N-Count
Pval
Pt.
Bis
.
Calc.
Rv.
Cont.
Ans.
Mask
100199
5
OP
A
12,151
0.75
.28
1,200
1,100
4,860
100200
29
FT
D
5,602
0.52
.32
500
495
489Slide19
Tool Usage Reporting Framework3. Examining student performance in relation to tool usage.Adding to individual student reports.Summarizing alongside test performance data.Analyzing test data using tool use/non-tool use as a disaggregating variable.
eMetric, LLC Confidential
19Slide20
Relating to Validity ArgumentationThe latest version of the AERA/NCME/APA Standards for Educational and Psychological Testing (2014) New chapter, “Fairness in Testing,” emphasizes accessibility and fairness as fundamental issues in testing (p. 4).
Students’ accessibility to test constructs is a basic assumption to all validity argumentation.
Construct-irrelevant factors must be addressed in validity arguments.eMetric, LLC Confidential
20Slide21
Importance of the Practice TestAccessibility tools, including but not limited to accommodations, are now a part of the test development and test administration processes.Computer-based testing allows for technology-driven tools. Rapid innovation drives need for rapid specification and evaluation.
Students must have facility and opportunity to practice with the tools.Practice tests should address accessibility features as well as item types and navigation functions.
eMetric, LLC Confidential
21Slide22
Questions?Nathan L. Wall – nwall@emetric.netAnne H. Davidson – anne.Davidson@edmetricllc.com
22