/
Technology  Enhanced Items — Signal or Noise Technology  Enhanced Items — Signal or Noise

Technology Enhanced Items — Signal or Noise - PowerPoint Presentation

alida-meadow
alida-meadow . @alida-meadow
Follow
349 views
Uploaded On 2018-11-01

Technology Enhanced Items — Signal or Noise - PPT Presentation

  Wednesday June 28 2017 830 am1000 am JW Grand Ballroom Salon 2 JW Marriott Austin Presenters 3 Catherine Welch Univ of IowaITP Jon S Twing Pearson Wayne Camara ID: 707612

response tei grade items tei response items grade teis enhanced technology multiple domain item mcs information assessments irt time

Share:

Link:

Embed:

Download Presentation from below link

Download Presentation The PPT/PDF document "Technology Enhanced Items — Signal or..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.


Presentation Transcript

Slide1

Technology Enhanced Items — Signal or Noise?

 

Wednesday, June 28, 2017 | 8:30 am–10:00

am

JW

Grand Ballroom | Salon 2 (JW Marriott Austin)Slide2
Slide3

Presenters

3

Catherine Welch,

Univ

of Iowa/ITP

Jon S.

Twing

, Pearson

Wayne

Camara

, ACT

Stephen T. Murphy, Measured Progress 

Discussant: Joyce

Zurkowski

, CDOE 

Moderator: Douglas F. Becker, HMHSlide4

TEIs in RFPs

4

The vendor’s response must reflect familiarity with computer-based testing and the use of a variety of item types, including

technology enhanced items (TEI)

to assess students’ higher order cognitive skills as well as their knowledge of core ideas and concepts. 

(RFP 2017-073 DOE New Hampshire Statewide Assessments) 

STATE desires use of multiple item types and

technology enhanced items

to capitalize on efficiency while ensuring that NDSA assessments are aligned to the full breadth, depth, and cognitive complexity of STATE’s content standards. 

(North Dakota State Assessment RFP 201-2017-175)

Technology: Assessments will be delivered primarily online, include effective

technology-enhanced items

, and a facile, intuitive test management system. 

(

Nebraska Statewide Assessments 

RFI11162016)Slide5

Previous Studies

5

Wan &

Henly

(AME 2012

)

TEIs and MCs in statewide science achievement

test

Figural

Response, Constructed Response

Crabtree & Welch

(NCME 2016)

TEIs

and MCs in an end-of-course Algebra

exam

Bukhari,

Boughton

, & Kim

(NCME 2016

)

TEIs and MCs in grade 8 math and grade 7 ELA in PARCC and

SBAC

Matching

, Equation and Expression Entry,

Select and Order, Multiple Correct Response;

Multiple Correct Response, Evidence-Based Selected

ResponseSlide6

Previous Studies

6

IRT based

ICC

, average item information, information

efficiency

Conclusions

TEIs are more difficult than MCs.

TEIs are more discriminating than MCs.

TEIs provide more information at high

abilitySlide7

Tech-Enhanced Items: How

Can They Provide Better

Measurement?

This

presentation will critically evaluate a number of TEI formats and how they might be scored, from both a classical and item response theory (IRT) perspective. This discussion will be couched in the consideration of actual data delivered in K-12 educational assessments. One notable aspect is that substantial information can be reaped from incorrect responses, which are generally not considered except for advanced IRT models like partial credit or nominal response models. Even these do not always consider all available data. For example, a straightforward multiple- response item with six possible options and two correct answers might be scored as 0-1-2 points—better than simply 0-1—but not considering which of the four incorrect options were selected ignores any possible information that might be present there. Of course, this typically happens with multiple-choice items too, as the nominal response model is virtually never used in practice. In other cases— such as PARCC two-part items—IRT modeling frankly falls apart.

TEI’s at other conferences

7Slide8

Investigation of Psychometric Properties of Technology-Enhanced Items Considering Content

CharacteristicsRong

Jin

,

Stephen

Murphy, Sid

Sharairi

-- Houghton

Mifflin Harcourt Publishing 

Recent

research explored technology-enhanced (TE) items only by subject . This study dives into DOK, master domain, and grade level to explore over six hundreds TEs in twelve formats from a K-8 large scale mathematics assessment and compares the measurement properties between multiple-choice and TE, and between TE formats . 

TEI’s at other conferences

8Slide9

Domain and Item Type

9

Mean

TEI > MC in each domain.

Range

for

TEI and MC is

grade related

low: CC

, OA, NBT,

MD (grade

K-5

)

high: EE

,

NS, SP, RP (grade 6+)

Standard Deviation

MC < TEI in each domain.

Large (40+) for TEI in domain G, NS, SP, especially in EE. Slide10

Conclusions

10

MC Vs. TEI

On average, TEI are more difficult, discriminating, and time consuming than MC in each DOK or domain.

On average, when DOK increases, both MC and TEI become more challenging, less discriminating, and more time consuming but

to different

extents.

On average, both MC and TEI in domains available in grade 6+ are more challenging and time consuming than those in domains in grade K-5.

TEI Types

Easy: Number Line, Ordering, and Matching.

Hard: Multiple Select, Matrix, Graph, Enter Math.

Time consuming: Enter Math and Graph.Slide11

Presenters

11

Catherine Welch,

Univ

of Iowa/ITP

Stephen T. Murphy, Measured Progress 

Wayne

Camara

, ACT

Jon

S.

Twing

, Pearson

Discussant: Joyce

Zurkowski

, CDOE 

Moderator: Douglas F. Becker, HMH