Putting subject comparability right is not sufficient to make grades from exams in different subjects interchangeable. Mike Cresswell Ó Mike Cresswell 2016 Grade Interchangeability HEIs might select the wrong students for their courses
"Putting subject comparability “right” is not" is the property of its rightful owner. Permission is granted to
download and print the materials on this website for personal, non-commercial use only, and to display it
on your personal computer provided you do not modify the materials and that you retain all copyright
notices contained in the materials. By downloading content from our website, you accept the terms of this
agreement.
Presentation Transcript
01
Putting subject comparability “right” is not sufficient to make grades from exams in different subjects interchangeable. Mike Cresswell Ó Mike Cresswell 2016<br>
02
Grade Interchangeability “HEIs might select the wrong students for their courses because they assume that A level grades for different subjects can all be counted as equal.” (Ofqual ISC Working Papers 1 and 3)
If a student had taken an A-level which was easier/harder than the one they did take, they would have had a greater/lesser chance of being selected.
This situation is not fair for students or efficient for HE. Ó Mike Cresswell 2016<br>
03
The linking construct “If all the subjects being compared do measure ‘general aptitude’ – at least to some extent – then their outcomes can be compared, and linked, along this dimension. We might then interpret grades in terms of this linking construct. So, having adopted and applied this conception of comparability, we could conclude that a particular grade in (say) physics indicated the same level of general ability for learning as the same grade in French, or geography, or English.” (Ofqual ISC Working Paper 2) Ó Mike Cresswell 2016<br>
04
But... Putting subject comparability “right” means that, on average, candidates who have the same level of “general ability” get the same grade whatever subject(s) they take.
But does that actually increase the interchangeability of grades for individual candidates?
Does it make the grades of more individuals interchangeable?
How many more? Ó Mike Cresswell 2016<br>
05
Modelling Randomly generate linking “general ability” scores for 1000 candidates: N(50,100)
Add two sets of random variation N(0,e2) to model the candidates’ scores in two “subject” exams.
These random variations model the effect on each candidate’s score of both unreliability and the factors which might differentially affect their performance in different subjects (see Ofqual ISC Working Paper 2).
Choose the value of e so as to produce a specific level of correlation between the results in the two “subjects”. Ó Mike Cresswell 2016<br>
06
Three subject pairs Ó Mike Cresswell 2016<br>
07
Modelling Grade the scores in the two “subjects” so as to reflect current (non-comparable) grade standards in them both.
Compare the candidates’ grades in the two “subjects” to see how interchangeable they are.
Re-grade the scores in the two “subjects” so that the two grade distributions are the same.
Compare the candidates’ new grades in the two “subjects” to see if they are more interchangeable now that the “subject” standards have been made statistically comparable. Ó Mike Cresswell 2016<br>
08
Results Pair 1 Ó Mike Cresswell 2016<br>
09
Results Pair 2 Ó Mike Cresswell 2016<br>
10
Results Pair 3 Ó Mike Cresswell 2016<br>
11
Summary Ó Mike Cresswell 2016<br>
12
Correlations between subjects Ó Mike Cresswell 2016<br>
13
Conclusions Only for highly correlated subject exams does putting subject comparability “right” significantly increase the interchangeability of grades.
Conceptualising subject exams as measures of “general ability” is theoretically neat but they are necessarily low validity/reliability measures of such a construct so mainly have middling inter-correlations.
Putting subject comparability “right” would do very little to improve the quality of selection processes which assume that grades from exams in different subjects are interchangeable. Ó Mike Cresswell 2016<br>
