Measuring School Effectiveness: - PowerPoint Presentation

Slide1

Measuring School Effectiveness:Introduction and Background

Lorraine Dearden, Institute of Education, University of LondonSlide2

IntroductionSet the scene for the rest of the dayIntroduction to school testing regime in England

Potted history of how school effectiveness has been measured in England over timePoint out the strengths and difficulties with approaches that have been taken (briefly)Give some illustrations using work I am currently doing with Alfonso Miranda and Sophia Rabe-HeskethPoint out the similarities and differences with experience/approaches in some other countries  Slide3

Testing/Data in EnglandHave forms of testing at age 5 (FSP), age 7 (Key Stage 1), age 11* (Key Stage 2), age 14 (Key Stage 3 – now abolished); age 16* (GCSEs – Key Stage 4) and at age 18* (A levels – Key stage 5)

Have individual background information on students every year from 2001/02 (basic background characteristics - PLASC)Data has been linked to vocational courses done in e.g. FE colleges (ILR/NISVQ) as well as HE participation/outcomes(HESA)

* Externally markedSlide4
Slide5

History of School Accountability in England1988 National Curriculum introduced which all maintained schools are obliged to follow (but not private schools)For KS1, KS2, KS3 national system of test/assessment introduced (ages 5 to 14)

First school league tables published in 1992 for GCSEs and A levels (secondary schools)Aim: to inform parents about choice of school and to provide an incentive for schools to raise standardsSlide6

Development of School AccountabilityIn 1992 first secondary league tables showed ‘raw’ school GCSE results and one A level indicator of results Similar tables introduced for primary schools in 1996 based on KS2 test results in English, Maths and Science

Problems of using these ‘raw’ measures immediately pointed out (e.g. Goldstein & Spiegelhalter (1996))Simply reflected differences in school intake, background of students, etcSimple change in composition of intake could improve/worsen results from one year to next......Slide7

Value Added ModelsIf want to measure school effectiveness need to come up with measure that measures the value actually being added by school (e.g. Ladd and Walsh (2000))

Raw test results do not come close to doing this for most schoolsCouldn’t actually estimate VA models until could link individual school data some pilot studies were done in the 1990s but nothing nationallyOnly could start doing this for secondary schools in 2001/02 when had baseline KS2 performance data for children who sat KS4 exams in that year 2001/02Slide8

Other detailsKS4 measure is capped score of 8 best GCSE scores (maximum mark is 58 for A* in subject) so maximum score is 464 KS2 score is based on levels achieved in English, Maths and Science (not actual test score)Level 4 (27 points) is expected level, Level 5 (33 points) is 2 years ahead of where expected to be, level 3 (21 points) is 2 years behind where expected to be, level 2 (15 points) is 4 years behind where meant to be at age 11.

For CVA now use raw test scores to give individuals APS between 15 and 36 in each subject VA and CVA average 3 KS2 APSs (if one missing just average 2). Slide9

Value AddedVA model introduced in 2002/03 and used average of KS2 crude APS in English, Maths and Science Split this KS2 score into 10 groups and looked at the median outcome for these 10 groups join the median points to get the 'national median line'

Value added was simply the school average of individual deviation from this line (added to 100)Slide10

CVACritisized that still problematic (see Ray (2006) for full background) – need to take other factors into account to truly measure the value added by schools

Ceiling effects (e.g. KS1 to KS2 VA)Stability issuesMoved to a CVA model where controlled for prior attainment (fine APS), FSM status, ethnicity, gender, SEN, deprivation(local area), relative age, EAL, mobilityRegression model, with school clustering (multi-level/heirarchical model)CVA is school average of residuals from this model (+1000)Slide11

Ready ReckonerModel estimated every year on latest data and so way CVA calculated changes every yearSchools given ready

reckoner so they can see CVA for every child in schoolPretty obvious that if they have choice over-classifying a student’s background, they know which is more favourable for CVADo not have to add as much value (other things being equal) if non-EAL, or White British or unclassifiable ethnicity Slide12

Do school understand CVA?Slide13

1) Has the school CVA measure solved the problem of informing parents about school choice and ensuring school accountability? No – and this will be the subject of some talks today2) Could we do better? Yes – and this will be the subject of the other talks today

3) Should we even attempt to measure school effectiveness?.............Slide14

Problems with CVA in EnglandNot very stable (Leckie and Goldstein)Differences between schools rarely statistically significant (

Leckie and Goldstein)Not very transparent and difficult to understand (Gorard; Dearden, Micklewright and Vignoles)

Evidence of differential effectiveness within school (Brown and Tzavidis; Dearden, Micklewright and Vignoles

)Evidence that parents don’t use it when making decisions – prefer ‘raw scores’ (Machin and Hansen)Newspapers don’t highlight it generally – still go back to ‘easy to understand’ raw scoresSlide15
Slide16

Other problemsDrawing on work I am doing with Alfonso Miranda and Sophia Rabe-Hesketh (using linked survey “Next Steps” data and NPD data for kids born in 1991/92)Big left censoring in KS2 scores used which disadvantages school with kids at bottom of KS2 ability distribution (and some right censoring)

Also –way the fine point KS2 APS is constructed (transforming raw test scores into the APS system slightly strange)Slide17

Raw KS2 Maths ScoresSlide18

Maths score put in CVA Model

Left censoring makes it much harder to add value – why do it?Slide19

How serious is it that key covariates not measured?Key variables, like parental education not measured in the administrative dataBut we know, that more highly educated parents are likely, on average, to provide more educational input in the homeAlso know that education of parents varies markedly by school – not random

Has implications for CVASlide20

Is this a serious problem in England?Our work suggests it is (Dearden, Miranda and Rabe-Hesketh (2010))Use “Next Steps” survey data linked to the NPD for cohort of children who took KS2 in 2001 and KS4 in 2006

Observe parental education for just over 12,000 children in our sample (total cohort in NPD just over 550,000)Slide21

Regress individual CVA on mother’s educationMother’s

EducationGirlsBoys

Estimate (SE) Estimate

(SE)Constant991.7(2.6)

998.5(1.9)

Level 2

7.5

(2.7)

9.9

(2.2)

Level 3

14.2

(2.9)

18.5

(2.4)

Level 4/5

25.7

(3.3)

30.4

(3.2)

No.

obs

6442

6551

Note: Cluster at school level (698 out of total of just over 3,000 schools and use “Next Steps” survey weights)Slide22

What can we do about this?Use model of Miranda and Rabe-Hesketh (2010) to re-calculate CVA model for whole NPD sample accounting for missing mother’s educationPaper will be presented on Wednesday at Festival

Exploits that fact that for some individuals have survey and NPD dataExplicitly models missingness in survey and administrative dataSlide23

ResultsSlide24

School league tables in other countries?Increasingly in a number of States in US being used but not nationallyIn Australia introduced national testing (NAPLAN) tests with report cards to parents in Years 5, 7 and 9

Have website where parents can check performance of schools next to statistically ‘similar’ school School accountability becoming an issue world widePISA world league tables...Slide25

ConclusionsSchool League tables here to stay in EnglandBut very difficult to measure the value added by schoolsGiven not likely to go away, we need to do it betterLook forward to rest of today’s talks to take us forward on this issueSlide26

ReferencesGoldstein, H. and Spiegelhalter, D. J. (1996) League tables and their limitations: statistical issues in comparisons of institutional performance. Journal of the Royal Statistical Society: Series A, 159, 385-443.

Goldstein H, Rasbash J, Yang M, Woodhouse, G, Pan H, Nuttall, D, and Thomas, S (1993) ‘A multilevel analysis of school examination results’ Oxford Review of Education, 19: 425-33. Gorard

, S. (2010) All evidence is equal: the flaw in statistical reasoning, Oxford Review of Education, (forthcoming).Ladd and Walsh (2000) ‘Implementing value-added measures of school effectiveness: getting the incentives right’, Economics of Education Review, vol. 2 part 1 pp. 1–17.Leckie

, G. and Goldstein, H. (2009) The limitations of using school league tables to inform school choice. Journal of the Royal Statistical Society: Series A. vol. 127 part 4, pp835-52.Ray, A. (2006) School Value Added Measures in England. Paper for the OECD Project on the Development of Value-Added Models in Education Systems. London, Department for Education and Skills

http://www.dcsf.gov.uk/research/data/uploadfiles/RW85.pdf.