The evolution of school league table measures in England - PowerPoint Presentation

Slide1

The evolution of school league table measures in England 1992-2016

George Leckie

Centre for Multilevel Modelling

Graduate School of Education

University of BristolSlide2

OutlineSlide3

OutlineEvery year the Government publishes ‘

school league tables

’

summarising pupils’ average GCSE results and progress

In the last 10 years the headline measure of progress has changed from

contextual value-added

(

CVA, 2006-10) to

expected

progress

(

EP, 2011-15) to

progress

8

(

P8, 2016-)

In this talk we

review

and

statistically critique

this evolution

We question

Government’s reasons for withdrawing

CVA

We argue

that

EP

suffers from fundamental design flaws

We explain

how

P8

should address a subset of these flaws

We show

that

CVA

,

EP

and

P8

give very different

rankingsSlide4

BackgroundSlide5

Two types of performance measureAttainment vs. Progress

‘

Attainment

’

measures (e.g.,

5 A*-C

and

A8

) aim to report average

‘

status

’ of pupils

at the end of

secondary

schooling

May give information regarding school inequalities, but crucial to realise that they reflect differences in school intake composition more than school processes

‘

Progress

’ measures

(e.g.,

CVA

,

EP

, and

P8

) aim

to report

average

‘

growth

’ made by pupils

during

secondary schooling

Generally

considered

the fairer and more meaningful way

to measure school

performanceSlide6

KS2 tests and GCSE examinations

Primary phase of education (ages 4–11, years R–6)

KS2 tests

in English and maths

Measured as a

fine point score

, but discretised for

reporting into

national curriculum

levels

: W, 1

, 2, 3, 4,

5

Secondary phase of education (ages 11–16, years 7–11)

GCSE examinations

in English, maths

and

other

subjects

Measured using

GCSE grades

: U, G, F, E, D, C, B, A, A*Slide7
Slide8

Gov. justifications for publishing school performance tables

To support parental

school choice

based on schools’ ability to teach the national

c

urriculum, and to therefore create competition and a free market in education

To enable

school

accountability

; publically funded schools should be held publically accountable

To promote

school improvement

via school self-reflection and the identification of effective practices being employed in successful schoolsSlide9
Slide10
Slide11
Slide12
Slide13

Contextual value-added2006-2010Slide14

Contextual value-added (CVA)2006-2010

Conceptually, CVA scores were school-level averages of the differences between pupils’ actual & predicted GCSE scores

Each pupil’s predicted GCSE score was derived as the average GCSE score among pupils nationally with the same KS2 score and socioeconomic and demographic background characteristics

Free school meal status, ethnicity, English as an additional language, special education needs

,

…

Statistically, CVA scores were derived from a simple

multilevel modelSlide15
Slide16

CVA score

95% CISlide17
Slide18

Gov. reasons for ending CVA 1Schools white

p

aper 2010

‘[CVA] is

difficult for the public to understand

’

Agreed, but no need to understand the technical details of the model in order to interpret the CVA scores, only the general principle of adjusting for intake differences

Having said that, more guidance should have been given as to how to do this, for example one had to delve deep into the technical documentation to find out what the CVA unit of measurement was

Graphical approaches could have been considered as alternatives to 95

% confidence intervalsSlide19

Gov. reasons for ending CVA 2

Schools white

p

aper

2010

‘We should expect every child to succeed and measure schools on how much value they add for all pupils,

not rank them on the ethnic make-up of their intake

...’

This statement suggests

a fundamental

misunderstanding

CVA

explicitly adjusted for as many of the observed differences between schools’ intakes as possible in order to remove their influence from schools’

rankings

In

contrast, it is when one ignores these differences that one implicitly ranks schools on the make-up of their

intakesSlide20

Gov. reasons for ending CVA 3

Schools

white

p

aper

2010

‘[CVA] also has

the effect of expecting different levels of progress from different groups of pupils

on the basis of their ethnic background, or family circumstances, which we think is wrong in principle’

CVA

did not

apriori

expect different levels of progress from different pupil groups, rather it

adjusted for such

differences if they

arose

The reality is that nationally some pupil groups

do

make less progress than others and that this must be adjusted for if we are to make fair comparisons between schools

Failure to do so leads to ‘

comparing apples and oranges

’Slide21

Gov. reasons for ending CVA 4

Schools white

p

aper

2010

‘It is morally wrong to have an attainment measure which

entrenches low aspirations for children because of their background

’

The Government are arguing that by adjusting for pupil

background, CVA

led to a system-level acceptance that socially and other disadvantaged pupil groups

will

make less progress than their more advantage

peers

One concern is that some schools started to use the published CVA model to set differential GCSE targets for current pupils based

on

their background. This was never the purpose of CVA and reflects the

perverse incentives

that so often arise with high-stakes school league tablesSlide22
Slide23

Expected progress2011-2015Slide24

Expected

progress (EP)

2011-2015

EP simply reports the percentage of pupils making

three levels of progress

between KS2 and GCSE separately in English and maths.

I

t is seemingly intuitive and ignores pupils’ socioeconomic and demographic characteristicsSlide25

Statistical concern 1Borderline effects

Transition values are binary (EP is a

threshold

measure)

No partial rewards for just missing target grades; no additional rewards for greatly surpassing target grades

Schools

are perversely incentivised to concentrate

on

borderline

pupils (within

each

KS2 level) (c.f., 5+ A*-C)

KS2 level

GCSE

grade

U

G

F

E

D

C

B

A

*

Working towards

level

1

0

1

1

1

1

1

1

1

1

Level 1

0

0

1

1

1

1

1

1

1

Level 2

0

0

0

1

1

1

1

1

1

Level 3

0

0

0

0

1

1

1

1

1

Level 4

0

0

0

0

0

1

1

1

1

Level 5

0

0

0

0

0

0

1

1

1Slide26

S

tatistical concern

2

Dependency on prior attainment

Schools with higher attaining intakes will do better on EP; EP under-adjusts for school differences in prior attainmentSlide27

S

tatistical concern 2 (cont’d)

Dependency on prior attainment

EP also has an illogical

sawtooth

(zig

zag

) dependency

on prior attainment with sharp discontinuities in probability of making EPSlide28

Statistical concern 3Statistical uncertainty

EP makes no attempt to quantify and communicate the statistical uncertainty in measuring school effects

Consider

a school with 180

pupils where

70% make EP.

The

associated 95% Wald binomial confidence interval ranges from 63% to 77% and so the school has a

±7 percentage point margin of

error

which would

be completely unacceptable in any survey or poll of public

opinion

There is no obvious way for users to establish whether measured differences between schools, or differences from national averages and floor standards, are meaningful, or whether they more likely reflect the variations of

chanceSlide29

Statistical concern

3

Statistical

uncertainty

Over a third of schools cannot be distinguished from the national average in English and in maths, but this is not communicatedSlide30

Progress 8

2016-20??Slide31

Progress 8 (P8)2016-20??

As with CVA, P8 scores are

school-level averages of the differences between pupils’ actual & predicted GCSE

scores

However,

e

ach

pupil’s predicted GCSE score

is now derived more simply as just the

average GCSE score among pupils nationally with the same KS2 score

P8 (like CVA) will

avoid the borderline effects and biases of EP

P8 (like CVA)

will also be presented with 95%

CIs

However

,

P8 (unlike CVA) will

ignore pupil

backgroundsSlide32
Slide33

CVA vs. EP2010Slide34

Correlation between CVA and EP

Many schools ranked high on CVA are ranked low on EP and vice versa

 Slide35

Change in national ranking due to switch from CVA to EP in 2010

As expected, EP is strongly biased in favour of schools with high prior attaining intakes (especially

Grammar

schools)Slide36

EP vs. P82015 and 2016Slide37

Correlation between

EP and P8

Stronger correlation between EP and P8, but change of progress measure still leads to substantial rank changes

 Slide38

Change in national ranking due to switch from EP to P8 in 2016

P8 will pull back schools with high prior attaining intakes (especially

Grammar

schools)Slide39

ConclusionSlide40

ConclusionCVA

recognised that poor pupils make less progress than their richer peers and adjusted for

this in an attempt

to make fair and meaningful comparisons between

schools

Reasons for withdrawing CVA are questionable

EP

was an ideological shift away from CVA whereby the Government declared all pupils must make the same progress, irrespective of their prior attainment and socioeconomic

circumstances

EP suffers from fundamental design flawsSlide41

Conclusion (cont’d)P8 represents a partial return to CVA in that it

recognises

that pupils with higher prior attainment make more

progress

However

, P8 continues

to ignore pupils’

backgrounds

CVA, EP and P8 lead to very different rankings and each reform generates new winners and losers

Move from CVA to EP unfairly benefited Grammars

Move from EP to P8 largely corrects for this bias

Policy

makers, schools and parents should view progress measures and school league tables

with

far more caution and scepticism than has often been the case to

dateSlide42

End of talk – Thank youg.leckie@bristol.ac.uk

bristol.ac.uk/cmm/team/leckie.htmlSlide43

ReferencesLeckie, G., & Goldstein, H. The evolution of school league tables in England 1992-2016: ‘contextual value-added’, ‘expected progress’ and ‘progress 8’.

British Educational Research Journal

.

Forthcoming

.

Leckie

, G., Goldstein, H. (2016).

How should we measure and hold schools accountable for the progress of their pupils?

Graduate School of Education Research

Briefing,

No. 32.

Leckie, G., & Goldstein, H.

The evolution of school league tables in England 1992-2016: ‘contextual value-added’, ‘expected progress’ and ‘progress 8’

.

Bristol Working Paper

in Education

Series

. Working Paper, 2/16

.Slide44
Slide45