19922016 George Leckie Centre for Multilevel Modelling Graduate School of Education University of Bristol Outline Outline Every year the Government publishes school league tables ID: 553482
Download Presentation The PPT/PDF document "The evolution of school league table mea..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
The evolution of school league table measures in England 1992-2016
George Leckie
Centre for Multilevel Modelling
Graduate School of Education
University of BristolSlide2
OutlineSlide3
OutlineEvery year the Government publishes ‘
school league tables
’
summarising pupils’ average GCSE results and progress
In the last 10 years the headline measure of progress has changed from
contextual value-added
(
CVA, 2006-10) to
expected
progress
(
EP, 2011-15) to
progress
8
(
P8, 2016-)
In this talk we
review
and
statistically critique
this evolution
We question
Government’s reasons for withdrawing
CVA
We argue
that
EP
suffers from fundamental design flaws
We explain
how
P8
should address a subset of these flaws
We show
that
CVA
,
EP
and
P8
give very different
rankingsSlide4
BackgroundSlide5
Two types of performance measureAttainment vs. Progress
‘
Attainment
’
measures (e.g.,
5 A*-C
and
A8
) aim to report average
‘
status
’ of pupils
at the end of
secondary
schooling
May give information regarding school inequalities, but crucial to realise that they reflect differences in school intake composition more than school processes
‘
Progress
’ measures
(e.g.,
CVA
,
EP
, and
P8
) aim
to report
average
‘
growth
’ made by pupils
during
secondary schooling
Generally
considered
the fairer and more meaningful way
to measure school
performanceSlide6
KS2 tests and GCSE examinations
Primary phase of education (ages 4–11, years R–6)
KS2 tests
in English and maths
Measured as a
fine point score
, but discretised for
reporting into
national curriculum
levels
: W, 1
, 2, 3, 4,
5
Secondary phase of education (ages 11–16, years 7–11)
GCSE examinations
in English, maths
and
other
subjects
Measured using
GCSE grades
: U, G, F, E, D, C, B, A, A*Slide7Slide8
Gov. justifications for publishing school performance tables
To support parental
school choice
based on schools’ ability to teach the national
c
urriculum, and to therefore create competition and a free market in education
To enable
school
accountability
; publically funded schools should be held publically accountable
To promote
school improvement
via school self-reflection and the identification of effective practices being employed in successful schoolsSlide9Slide10Slide11Slide12Slide13
Contextual value-added2006-2010Slide14
Contextual value-added (CVA)2006-2010
Conceptually, CVA scores were school-level averages of the differences between pupils’ actual & predicted GCSE scores
Each pupil’s predicted GCSE score was derived as the average GCSE score among pupils nationally with the same KS2 score and socioeconomic and demographic background characteristics
Free school meal status, ethnicity, English as an additional language, special education needs
,
…
Statistically, CVA scores were derived from a simple
multilevel modelSlide15Slide16
CVA score
95% CISlide17Slide18
Gov. reasons for ending CVA 1Schools white
p
aper 2010
‘[CVA] is
difficult for the public to understand
’
Agreed, but no need to understand the technical details of the model in order to interpret the CVA scores, only the general principle of adjusting for intake differences
Having said that, more guidance should have been given as to how to do this, for example one had to delve deep into the technical documentation to find out what the CVA unit of measurement was
Graphical approaches could have been considered as alternatives to 95
% confidence intervalsSlide19
Gov. reasons for ending CVA 2
Schools white
p
aper
2010
‘We should expect every child to succeed and measure schools on how much value they add for all pupils,
not rank them on the ethnic make-up of their intake
...’
This statement suggests
a fundamental
misunderstanding
CVA
explicitly adjusted for as many of the observed differences between schools’ intakes as possible in order to remove their influence from schools’
rankings
In
contrast, it is when one ignores these differences that one implicitly ranks schools on the make-up of their
intakesSlide20
Gov. reasons for ending CVA 3
Schools
white
p
aper
2010
‘[CVA] also has
the effect of expecting different levels of progress from different groups of pupils
on the basis of their ethnic background, or family circumstances, which we think is wrong in principle’
CVA
did not
apriori
expect different levels of progress from different pupil groups, rather it
adjusted for such
differences if they
arose
The reality is that nationally some pupil groups
do
make less progress than others and that this must be adjusted for if we are to make fair comparisons between schools
Failure to do so leads to ‘
comparing apples and oranges
’Slide21
Gov. reasons for ending CVA 4
Schools white
p
aper
2010
‘It is morally wrong to have an attainment measure which
entrenches low aspirations for children because of their background
’
The Government are arguing that by adjusting for pupil
background, CVA
led to a system-level acceptance that socially and other disadvantaged pupil groups
will
make less progress than their more advantage
peers
One concern is that some schools started to use the published CVA model to set differential GCSE targets for current pupils based
on
their background. This was never the purpose of CVA and reflects the
perverse incentives
that so often arise with high-stakes school league tablesSlide22Slide23
Expected progress2011-2015Slide24
Expected
progress (EP)
2011-2015
EP simply reports the percentage of pupils making
three levels of progress
between KS2 and GCSE separately in English and maths.
I
t is seemingly intuitive and ignores pupils’ socioeconomic and demographic characteristicsSlide25
Statistical concern 1Borderline effects
Transition values are binary (EP is a
threshold
measure)
No partial rewards for just missing target grades; no additional rewards for greatly surpassing target grades
Schools
are perversely incentivised to concentrate
on
borderline
pupils (within
each
KS2 level) (c.f., 5+ A*-C)
KS2 level
GCSE
grade
U
G
F
E
D
C
B
A
*
Working towards
level
1
0
1
1
1
1
1
1
1
1
Level 1
0
0
1
1
1
1
1
1
1
Level 2
0
0
0
1
1
1
1
1
1
Level 3
0
0
0
0
1
1
1
1
1
Level 4
0
0
0
0
0
1
1
1
1
Level 5
0
0
0
0
0
0
1
1
1Slide26
S
tatistical concern
2
Dependency on prior attainment
Schools with higher attaining intakes will do better on EP; EP under-adjusts for school differences in prior attainmentSlide27
S
tatistical concern 2 (cont’d)
Dependency on prior attainment
EP also has an illogical
sawtooth
(zig
zag
) dependency
on prior attainment with sharp discontinuities in probability of making EPSlide28
Statistical concern 3Statistical uncertainty
EP makes no attempt to quantify and communicate the statistical uncertainty in measuring school effects
Consider
a school with 180
pupils where
70% make EP.
The
associated 95% Wald binomial confidence interval ranges from 63% to 77% and so the school has a
±7 percentage point margin of
error
which would
be completely unacceptable in any survey or poll of public
opinion
There is no obvious way for users to establish whether measured differences between schools, or differences from national averages and floor standards, are meaningful, or whether they more likely reflect the variations of
chanceSlide29
Statistical concern
3
Statistical
uncertainty
Over a third of schools cannot be distinguished from the national average in English and in maths, but this is not communicatedSlide30
Progress 8
2016-20??Slide31
Progress 8 (P8)2016-20??
As with CVA, P8 scores are
school-level averages of the differences between pupils’ actual & predicted GCSE
scores
However,
e
ach
pupil’s predicted GCSE score
is now derived more simply as just the
average GCSE score among pupils nationally with the same KS2 score
P8 (like CVA) will
avoid the borderline effects and biases of EP
P8 (like CVA)
will also be presented with 95%
CIs
However
,
P8 (unlike CVA) will
ignore pupil
backgroundsSlide32Slide33
CVA vs. EP2010Slide34
Correlation between CVA and EP
Many schools ranked high on CVA are ranked low on EP and vice versa
Slide35
Change in national ranking due to switch from CVA to EP in 2010
As expected, EP is strongly biased in favour of schools with high prior attaining intakes (especially
Grammar
schools)Slide36
EP vs. P82015 and 2016Slide37
Correlation between
EP and P8
Stronger correlation between EP and P8, but change of progress measure still leads to substantial rank changes
Slide38
Change in national ranking due to switch from EP to P8 in 2016
P8 will pull back schools with high prior attaining intakes (especially
Grammar
schools)Slide39
ConclusionSlide40
ConclusionCVA
recognised that poor pupils make less progress than their richer peers and adjusted for
this in an attempt
to make fair and meaningful comparisons between
schools
Reasons for withdrawing CVA are questionable
EP
was an ideological shift away from CVA whereby the Government declared all pupils must make the same progress, irrespective of their prior attainment and socioeconomic
circumstances
EP suffers from fundamental design flawsSlide41
Conclusion (cont’d)P8 represents a partial return to CVA in that it
recognises
that pupils with higher prior attainment make more
progress
However
, P8 continues
to ignore pupils’
backgrounds
CVA, EP and P8 lead to very different rankings and each reform generates new winners and losers
Move from CVA to EP unfairly benefited Grammars
Move from EP to P8 largely corrects for this bias
Policy
makers, schools and parents should view progress measures and school league tables
with
far more caution and scepticism than has often been the case to
dateSlide42
End of talk – Thank youg.leckie@bristol.ac.uk
bristol.ac.uk/cmm/team/leckie.htmlSlide43
ReferencesLeckie, G., & Goldstein, H. The evolution of school league tables in England 1992-2016: ‘contextual value-added’, ‘expected progress’ and ‘progress 8’.
British Educational Research Journal
.
Forthcoming
.
Leckie
, G., Goldstein, H. (2016).
How should we measure and hold schools accountable for the progress of their pupils?
Graduate School of Education Research
Briefing,
No. 32.
Leckie, G., & Goldstein, H.
The evolution of school league tables in England 1992-2016: ‘contextual value-added’, ‘expected progress’ and ‘progress 8’
.
Bristol Working Paper
in Education
Series
. Working Paper, 2/16
.Slide44Slide45