Presentation on theme: "Item Difficulty Modeling on"— Presentation transcript
Slide1
Item Difficulty Modeling on Logical and Verbal Reasoning Tests
Kuan Xing1 and Kirk Becker21 University of Illinois – Chicago; 2 Pearson VUE, Chicago IL
Acknowledgement: This pilot study was done during first author’s internship at Pearson VUE. The first author wants to thank Pearson VUE, and especially Dr. Kirk Becker for his great support and mentoring.
1Slide2
Background
Psychometric researchers and testing practitioners must investigate the quality of large-scale tests. Checking item parameters such as item difficulty is important in the test quality control process.However, researchers and practitioners want to know more, such as what makes items difficult (or ea
sy)?2Slide3
Background (continued)
It’s essential to investigate the item features, and use those information to make predictions on the item qualities for future item generation (Irvine, 2003). In practice, often there are some discrepancies between test writers’ predicted item characteristics (e.g., item difficulty) and operational item characteristics. Item difficulty modeling may function as an effective way to generate new items with good quality.
3Slide4
Theoretical framework
Earlier work: Mental rotation (Shepard & Metzler, 1971)Carroll and Johnson (1991) showed item difficulties could be predicted by radicals (those theoretically consonant structural components of items)Psychology of reasoning (Newstead et al., 2003): Using think-aloud protocols and experimental studies, they identified factors which could affect the item difficulties in Analytical Reasoning (AR), such as negatives, types of reasoning task (assignment or placing), overall informativeness, among others.
4Slide5
Logical Reasoning
Items are grouped into sets;Each set provides a set of rules;The answer to each item can be deduced from the information provided.5Slide6
Example of logical reasoning content
The game is for 3 playersIf A is playing, B is notOnly 2 of B, C, or D can play at the same timeC will always playIf E plays, A will also playQuestion – if A and E are playing, who is also playing?
6Slide7
Methods
Participants: A new-designed college administration test was piloted in a private university. This test consists of logical, quantitative, and verbal reasoning subtests, but we only focus on the logical and verbal parts.Procedures: We identified and coded test features from the previous research work (Newstead et al., 2003), answer sheets developed by test writers, and our discussions. Item level and set level features were coded.
Multiple regressions were applied to predict item difficulties using different combinations of coded test features. 7Slide8
Results
Part 1: Logical reasoning testA total of 14 valid item difficulty parameters were finally entered in the regression equation for logical reasoning. Values used were the average difficulty of the 4 items within a set.Predictors including people, repeating groups, and cultural background explained a marginally significant proportion of variance in item difficulty parameters, R
2 = .50, F
(3,13) = 3.37,
p
= .06.
(A list of test features is shown in Table 1 on next page.)
8Slide9
Results (continued)
Test featuresCoding criteriaPeople
Categorical variable: coded as “1” if there are people (and objects) in test scenarios, and as “0” if there are only objects in test scenarios.
Repeating groups
Categorical variable: coded as “1” if the people/objects could be reassigned into different reasoning questions (usually one question set
has
4
reasoning questions), and as “0” if they couldn’t.
Cultural background
Categorical variable: coded as “1” if the items
possibly require
examinees’ cultural background (e.g., sports knowledge), and as “0” if the items don’t require.
Information ratio
Continuous variable: it equals to the number of reasoning rules divided by the number of information pieces. [Note. One rule may consists of more than one information piece.
Table 1. Test features in logical reasoning test
9Slide10
Part 2. Verbal Reasoning
Candidates are asked to read a paragraph with 4 missing words. Each “item” consists of the missing word and 3 distractors. I had called upon my friend Sherlock Holmes upon the second morning after Christmas, with the _______ of wishing him the compliments of the season. He was ______ upon the sofa in a purple dressing-gown, a pipe-rack within his reach upon the right, and a pile of crumpled morning papers, evidently newly studied, near at hand. Beside the couch was a wooden chair, and on the angle of the back hung a very seedy and ________ hard-felt hat, much the worse for wear, and cracked in several places. A lens and a forceps lying upon the seat of the chair suggested that the hat had been suspended in this manner for the purpose of _______.
10Slide11
Verbal Reasoning Features
Word Frequency (key - ngram from ngramr R package)Reading EaseGrade level Word count
CharactersSentences Passive sentencesWords per sentence
Characters per word
11Slide12
Verbal Reasoning Correlations
12Slide13
Discussions
Regression analysis for logical and verbal reasoning tests showed promising results (given the sample size is small). Test features were essential components to predict item difficulty parameters.Additional studies with more items and with verification of the predictions will be forthcoming.13Slide14
Thanks for listening!Questions?
E-mails: kxing2@uic.edu kirk.becker@pearson.comKey referencesIrvine, S. H. (2003). The foundations of item generation for mass testing. In S. H. Irvine & P. C. Kyllonen (Eds.), Item generation for test development (pp 3-34). Hillsdale, NJ: Erlbaum.Newstead, S., Bradon, P., Handley, S., Evans, J., & Dennis, I. (2003). Using the psychology of reasoning to predict the difficulty of analytical reasoning problems. In S. H. Irvine & P. C. Kyllonen
(Eds.), Item generation for test development (pp 35-52). Hillsdale, NJ: Erlbaum.14
