Criterion-Referenced Proficiency Testing BILC 26

Transcript:Criterion-Referenced Proficiency Testing BILC 26:
Criterion-Referenced Proficiency Testing BILC 26 May 2016 Ray Clifford Norm-Referenced (NR) or Criterion-Referenced (NR) Does it make a difference? The classic distinction made between CR and NR has focused on why the test is given. Norm-Referenced tests compare people with each other. Criterion-Referenced tests compare each person against a criterion or standard. But there are other differences as well! Test Characteristics of Typical NR and CR Tests Are all language tests proficiency tests? No. Achievement tests assess rote, direct application learning. Performance tests assess rehearsed near transfer learning. Proficiency tests assess far-transfer learning. And not all tests that are called “proficiency” tests are criterion-referenced tests. Creating a C-R Test The Construct must have Task, Context, and Accuracy (TCA) expectations. The following elements must be aligned: The construct to be tested. The test design and development . The scoring model. If all of these elements are aligned, the test is legally defensible. Shrock, S. A. & Coscarelli, W. C. (2007). Criterion-referenced test development: Technical and legal guidelines for corporate training and certification. (3rd.ed.). San Francisco, CA: John Wiley and Sons. Creating a CR Language Test Define incremental TCA stages of the trait (where each stage is more complex in dimensionality than the preceding stage). Maintain strict alignment of the: Theoretical construct model. Test development model. Psychometric scoring model. Luecht, R. (2003). Multistage complexity in language proficiency assessment: A framework for aligning theoretical perspectives, test development, and psychometrics. Foreign Language Annals, 36, 527-535. Testing Experts Stress the Need for Alignment Further Research With the approval of ATC, “Permissive” BAT research continued using English language learners interested in applying for admittance to a U.S. university. A diversity of first languages was represented among the test takers. The number who have taken the BAT Reading test now exceeds 600. With 600+ test takers, we have done the IRT analyses needed for adaptive testing. Validating the BAT WinSteps IRT Analyses confirmed that The BAT test items did “cluster” by difficulty level and the clusters didn’t overlap. The average logit values for each level were separated by more than 1 logit. Clustered items were then assembled into testlets, and the 5-item testlets were assigned to the appropriate stage. For every level, the testlets were of comparable difficulty – within 0.02 logits. The BAT Reading Test Is a Criterion-Referenced Proficiency Test. Defines the construct with a hierarchy of level-by-level TCA requirements. Follows those level-by-level criteria in the