Item Difficulty Modeling on - PowerPoint Presentation

Slide1

Item Difficulty Modeling on Logical and Verbal Reasoning Tests

Kuan Xing1 and Kirk Becker21 University of Illinois – Chicago; 2 Pearson VUE, Chicago IL

Acknowledgement: This pilot study was done during first author’s internship at Pearson VUE. The first author wants to thank Pearson VUE, and especially Dr. Kirk Becker for his great support and mentoring.

1Slide2

Background

Psychometric researchers and testing practitioners must investigate the quality of large-scale tests. Checking item parameters such as item difficulty is important in the test quality control process.However, researchers and practitioners want to know more, such as what makes items difficult (or ea

sy)?2Slide3

Background (continued)

It’s essential to investigate the item features, and use those information to make predictions on the item qualities for future item generation (Irvine, 2003). In practice, often there are some discrepancies between test writers’ predicted item characteristics (e.g., item difficulty) and operational item characteristics. Item difficulty modeling may function as an effective way to generate new items with good quality.

3Slide4

Theoretical framework

Earlier work: Mental rotation (Shepard & Metzler, 1971)Carroll and Johnson (1991) showed item difficulties could be predicted by radicals (those theoretically consonant structural components of items)Psychology of reasoning (Newstead et al., 2003): Using think-aloud protocols and experimental studies, they identified factors which could affect the item difficulties in Analytical Reasoning (AR), such as negatives, types of reasoning task (assignment or placing), overall informativeness, among others.

4Slide5

Logical Reasoning

Items are grouped into sets;Each set provides a set of rules;The answer to each item can be deduced from the information provided.5Slide6

Example of logical reasoning content

The game is for 3 playersIf A is playing, B is notOnly 2 of B, C, or D can play at the same timeC will always playIf E plays, A will also playQuestion – if A and E are playing, who is also playing?

6Slide7

Methods

Participants: A new-designed college administration test was piloted in a private university. This test consists of logical, quantitative, and verbal reasoning subtests, but we only focus on the logical and verbal parts.Procedures: We identified and coded test features from the previous research work (Newstead et al., 2003), answer sheets developed by test writers, and our discussions. Item level and set level features were coded.

Multiple regressions were applied to predict item difficulties using different combinations of coded test features. 7Slide8

Results

Part 1: Logical reasoning testA total of 14 valid item difficulty parameters were finally entered in the regression equation for logical reasoning. Values used were the average difficulty of the 4 items within a set.Predictors including people, repeating groups, and cultural background explained a marginally significant proportion of variance in item difficulty parameters, R

2 = .50, F

(3,13) = 3.37,

p

= .06.

(A list of test features is shown in Table 1 on next page.)

8Slide9

Results (continued)

Test featuresCoding criteriaPeople

Categorical variable: coded as “1” if there are people (and objects) in test scenarios, and as “0” if there are only objects in test scenarios.

Repeating groups

Categorical variable: coded as “1” if the people/objects could be reassigned into different reasoning questions (usually one question set

has

4

reasoning questions), and as “0” if they couldn’t.

Cultural background

Categorical variable: coded as “1” if the items

possibly require

examinees’ cultural background (e.g., sports knowledge), and as “0” if the items don’t require.

Information ratio

Continuous variable: it equals to the number of reasoning rules divided by the number of information pieces. [Note. One rule may consists of more than one information piece.

Table 1. Test features in logical reasoning test

9Slide10

Part 2. Verbal Reasoning

Candidates are asked to read a paragraph with 4 missing words. Each “item” consists of the missing word and 3 distractors. I had called upon my friend Sherlock Holmes upon the second morning after Christmas, with the _______ of wishing him the compliments of the season. He was ______ upon the sofa in a purple dressing-gown, a pipe-rack within his reach upon the right, and a pile of crumpled morning papers, evidently newly studied, near at hand. Beside the couch was a wooden chair, and on the angle of the back hung a very seedy and ________ hard-felt hat, much the worse for wear, and cracked in several places. A lens and a forceps lying upon the seat of the chair suggested that the hat had been suspended in this manner for the purpose of _______.

10Slide11

Verbal Reasoning Features

Word Frequency (key - ngram from ngramr R package)Reading EaseGrade level Word count

CharactersSentences Passive sentencesWords per sentence

Characters per word

11Slide12

Verbal Reasoning Correlations

12Slide13

Discussions

Regression analysis for logical and verbal reasoning tests showed promising results (given the sample size is small). Test features were essential components to predict item difficulty parameters.Additional studies with more items and with verification of the predictions will be forthcoming.13Slide14

Thanks for listening!Questions?

E-mails: kxing2@uic.edu kirk.becker@pearson.comKey referencesIrvine, S. H. (2003). The foundations of item generation for mass testing. In S. H. Irvine & P. C. Kyllonen (Eds.), Item generation for test development (pp 3-34). Hillsdale, NJ: Erlbaum.Newstead, S., Bradon, P., Handley, S., Evans, J., & Dennis, I. (2003). Using the psychology of reasoning to predict the difficulty of analytical reasoning problems. In S. H. Irvine & P. C. Kyllonen

(Eds.), Item generation for test development (pp 35-52). Hillsdale, NJ: Erlbaum.14