Wednesday June 28 2017 830 am1000 am JW Grand Ballroom Salon 2 JW Marriott Austin Presenters 3 Catherine Welch Univ of IowaITP Jon S Twing Pearson Wayne Camara ID: 707612
Download Presentation The PPT/PDF document "Technology Enhanced Items — Signal or..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Technology Enhanced Items — Signal or Noise?
Wednesday, June 28, 2017 | 8:30 am–10:00
am
JW
Grand Ballroom | Salon 2 (JW Marriott Austin)Slide2Slide3
Presenters
3
Catherine Welch,
Univ
of Iowa/ITP
Jon S.
Twing
, Pearson
Wayne
Camara
, ACT
Stephen T. Murphy, Measured Progress
Discussant: Joyce
Zurkowski
, CDOE
Moderator: Douglas F. Becker, HMHSlide4
TEIs in RFPs
4
The vendor’s response must reflect familiarity with computer-based testing and the use of a variety of item types, including
technology enhanced items (TEI)
to assess students’ higher order cognitive skills as well as their knowledge of core ideas and concepts.
(RFP 2017-073 DOE New Hampshire Statewide Assessments)
STATE desires use of multiple item types and
technology enhanced items
to capitalize on efficiency while ensuring that NDSA assessments are aligned to the full breadth, depth, and cognitive complexity of STATE’s content standards.
(North Dakota State Assessment RFP 201-2017-175)
Technology: Assessments will be delivered primarily online, include effective
technology-enhanced items
, and a facile, intuitive test management system.
(
Nebraska Statewide Assessments
RFI11162016)Slide5
Previous Studies
5
Wan &
Henly
(AME 2012
)
TEIs and MCs in statewide science achievement
test
Figural
Response, Constructed Response
Crabtree & Welch
(NCME 2016)
TEIs
and MCs in an end-of-course Algebra
exam
Bukhari,
Boughton
, & Kim
(NCME 2016
)
TEIs and MCs in grade 8 math and grade 7 ELA in PARCC and
SBAC
Matching
, Equation and Expression Entry,
Select and Order, Multiple Correct Response;
Multiple Correct Response, Evidence-Based Selected
ResponseSlide6
Previous Studies
6
IRT based
ICC
, average item information, information
efficiency
Conclusions
TEIs are more difficult than MCs.
TEIs are more discriminating than MCs.
TEIs provide more information at high
abilitySlide7
Tech-Enhanced Items: How
Can They Provide Better
Measurement?
This
presentation will critically evaluate a number of TEI formats and how they might be scored, from both a classical and item response theory (IRT) perspective. This discussion will be couched in the consideration of actual data delivered in K-12 educational assessments. One notable aspect is that substantial information can be reaped from incorrect responses, which are generally not considered except for advanced IRT models like partial credit or nominal response models. Even these do not always consider all available data. For example, a straightforward multiple- response item with six possible options and two correct answers might be scored as 0-1-2 points—better than simply 0-1—but not considering which of the four incorrect options were selected ignores any possible information that might be present there. Of course, this typically happens with multiple-choice items too, as the nominal response model is virtually never used in practice. In other cases— such as PARCC two-part items—IRT modeling frankly falls apart.
TEI’s at other conferences
7Slide8
Investigation of Psychometric Properties of Technology-Enhanced Items Considering Content
CharacteristicsRong
Jin
,
Stephen
Murphy, Sid
Sharairi
-- Houghton
Mifflin Harcourt Publishing
Recent
research explored technology-enhanced (TE) items only by subject . This study dives into DOK, master domain, and grade level to explore over six hundreds TEs in twelve formats from a K-8 large scale mathematics assessment and compares the measurement properties between multiple-choice and TE, and between TE formats .
TEI’s at other conferences
8Slide9
Domain and Item Type
9
Mean
TEI > MC in each domain.
Range
for
TEI and MC is
grade related
low: CC
, OA, NBT,
MD (grade
K-5
)
high: EE
,
NS, SP, RP (grade 6+)
Standard Deviation
MC < TEI in each domain.
Large (40+) for TEI in domain G, NS, SP, especially in EE. Slide10
Conclusions
10
MC Vs. TEI
On average, TEI are more difficult, discriminating, and time consuming than MC in each DOK or domain.
On average, when DOK increases, both MC and TEI become more challenging, less discriminating, and more time consuming but
to different
extents.
On average, both MC and TEI in domains available in grade 6+ are more challenging and time consuming than those in domains in grade K-5.
TEI Types
Easy: Number Line, Ordering, and Matching.
Hard: Multiple Select, Matrix, Graph, Enter Math.
Time consuming: Enter Math and Graph.Slide11
Presenters
11
Catherine Welch,
Univ
of Iowa/ITP
Stephen T. Murphy, Measured Progress
Wayne
Camara
, ACT
Jon
S.
Twing
, Pearson
Discussant: Joyce
Zurkowski
, CDOE
Moderator: Douglas F. Becker, HMH