METRICS— - PowerPoint Presentation

Slide1

METRICS—and other alternatives to the REF

RENU CONFERENCE

: Research Excellence and Funding

Birmingham City University

28

th

April

2015

DEREK SAYERSlide2

Introduction: the REF and other university rankings Slide3

At the level of the institution there is a good correlation between RAE/REF performance and position in the major—and largely metrics-based—global university rankings

despite differences in methodology

The institution is the appropriate level of analysis for purposes of allocating QR funding because (1) the REF does not evaluate individuals and (2) universities may spend QR money how they wish, irrespective of UoA REF scores

Of the 7 UK universities that make it into the THE World Rankings top 50, six are in the REF2014 top 10 (and the other is #11

)

Of 11 UK universities in THE top 100, only one is outside the REF top

20

Of 22 UK universities in THE top 150, only two are outside REF top

30

Of the 28 UK universities in THE top 200, only two are outside the REF top

40 Slide4

REF performance and traditional “research elite” status

One major driver for RAE regime was abolition of binary system in 1992.

RAE was a means of ensuring that scarce resources were not diverted from research-intensive universities to teaching-focused former polytechnics

.Throughout RAE/REF era the research elite universities have been major beneficiaries of QR funding. Despite lip-service to “encouraging excellence wherever it is found,” government and HEFCE have manipulated QR funding formula to ensure Russell Group and a handful of others retain dominance.

Russell Group

QR

funding fell from 65% in 2008-9 to 62% as a result of RAE2008

.

HEFCE’s

changes to the funding formula during 2010-11 ensured that by 2013-14 that share had risen by almost 10 percentage points to 71.9%. Russell Group and

1994

Group universities

received

almost 85% of QR

on

the eve of

REF2014

.

The Russell Group share of QR fell by less than one percentage point (to 70.8%) following HEFCE’s post-REF2014 “tweaks” to the funding

formula

. Slide5

If these very costly, time-consuming and contentious research assessment exercises, which take place every six years or so,

deliver results that correlate closely with

other global university rankings

; and confirm the excellence of the UK’s established elite schools

;

Isn’t the whole ritual a waste of time and money—an elaborate way of confirming what we already know?

T

here

is at least a prima facie case for looking for less costly

alternatives for allocating the research funding that is currently disbursed in the form of QR

. Slide6

Costs of the REF

“Although

few people are prepared to admit it, the REF has become a monster, a Minotaur that must be appeased by bloody sacrifices

.” (Peter Scott, 2013).Slide7

from RAE to REF: milestones

1986

4-pp. “general description of strengths” and 5 outputs required for each department, submissions judged by 37 UGC subject committees.

1989 70 specially-convened panels

evaluate 152

UoAs; each individual staff member now has to submit 2 research outputs.

1996

4 outputs required per staff member

; panel assessment criteria and working methods published in advance.

2010

2* outputs de-funded

. This leads many universities to institute their own labor-intensive “

internal REFs

” to select staff for the REF.

REF 2014

REF disciplinary

panels reduced from 67 to 36.

I

mpact added to assessment

, counting 20% of overall score.Slide8

REF 2014 by numbers

36 subpanels

with 10-30 members—898 academic members, 259 research users (total 1157)—evaluated 1911

UoA

submissions containing

191,148 individual research outputs

and

6975 impact

case

studies

f

rom 52,060 staff (FTE) located in 154 institutionsall in less than a year

Source: “Submissions data,” HEFCE REF2014 website,

http://www.ref.ac.uk/results/analysis/submissionsdata/

Slide9

REF2014—direct costs

Cost originally officially said to be comparable to RAE 2008, i.e.

nearly £60m—£47m within universities, £12m HEFCE admin costs

HEFCE

now puts its own direct costs of REF2014 at £14.4m.

Rand Europe has estimated

costs to universities of preparing impact submissions alone at £55m.

One anonymous University Research Head quoted in THE says “

an honest assessment” of the overall cost would be nearer £200m

.

Robert Bowman, director of the Centre for Nanostructured Media at Queen’s University Belfast,

has estimated that the real cost of the REF was more than £1 billion, with impact alone costing nearly £100 million.

His

figures included full economic costing for salaries,

which

Rand’s do not

.Slide10

REF2014—opportunity costs

Vast amounts of time are spent

both by panelists and those involved in REF preparation at university level from department research directors

upward.Panelists—basically a full year’s work. Assuming 898 academic members of REF panels each earns the average professorial salary of £76,395, opportunity costs of time academics spend on REF panels amount to £68,602,710

.

Universities—assume 1 departmental research director on average professorial salary of £76,395 devotes 1/3 of her time to REF over the two years 2012-14 (=£50,930). Multiply by total number of

UoA

submissions in REF2014 (1911). Net opportunity cost of departmental research directors alone: £97,327,230.

“

The amount of valuable research that this could have funded is immense [and the REF is] unlikely to tell us anything significant that we don’t know already” (anonymous university head of research, in THE, February 2014

)

Slide11

REF2014—costs for other academic valuesREF increasingly drives (and distorts) university priorities

, affecting e.g. research/teaching balance and hiring decisions.

Interdisciplinary and other “risky” research get marginalized

as universities try to second-guess discipline-based REF panels.

REF funding formula incentivizes research likely to have short-term impact

—even when its quality is judged only 2*.

Pressures to be selective in staff submitted to REF undermine morale and jeopardize collegiality.

‘It's difficult to imagine how I would ever have enough peace and quiet in the present sort of climate to do what I did in 1964 …

Today I wouldn't get an academic job … I don't think I would be regarded as productive enough.” Peter Higgs, Winner of 2013 Nobel Prize in Physics,

Guardian

, 6 December 2013Slide12

Is it worth it?

‘The one question a modern civil servant fails to

ask is

it worth the extra effort?’ (Peter Swinnerton-Dyer, 2013) Slide13

towards a balance sheet

Such costs are in my view justifiable only if they deliver benefits

that could not be achieved by other means that are less costly

in financial and other terms. This is the appropriate standpoint from which to evaluate metrics and other proposed alternatives to the REF. I want to consider two key questions, before moving to a wider consideration of possible alternatives to the REF.

In its own terms, is REF reliable as a means of evaluating research quality? Does it do what it says on the tin?

Could similar findings be generated using alternative methods of research evaluation—like metrics—at significantly lower cost?

The REF is not inevitable. These costs are

not

incurred by many other university systems, including the USA, where there is no equivalent of REF—but whose universities make up 7 of the top 10 and 15 of the top 20 in the THE World University

Rankings.Slide14

“What I challenge is the claim from which the REF derives its entire authority as a mechanism for funding allocation and on which it stakes its entire legitimacy as a process of research evaluation—the claim that it is a process of expert peer review. It was this claim that convinced the government to back down on its plans to replace the RAE with metrics after RAE 2008. If the claim is false, the case for metrics needs to be reconsidered

—

along with other ways of funding universities’ research infrastructure, which might include scrapping any such centralized national research audits altogether.”

THE ARGUMENTSlide15

Why the REF is not expert peer review

Comparisons with expert peer reviewing procedures used in other academic contexts in the UK and internationally—journal and book publication, research grant competitions, tenure and promotion proceedings—embarrass the REF Slide16

inadequacies of the REF All review is done in-house

by members of REF subpanels. REF2014 abandoned RAE2008’s option of use of external specialist reviewers.

O

n some panels, the volume of work is such that just one assessor may read each output—an absolute no-no in other peer reviewing contexts.

HEFCE’s

prohibitions on using bibliometric data (and standing of journals and publishers)

reinforce reliance on panelists’ subjective judgments.

Despite presence of “international members” on Main Panels, sub-panelists (who do the grading) are

almost all drawn from British universities alone

.

Reducing the number of disciplinary panels

from 67 in RAE 2008 to 36 in REF2014 additionally

reduced the range of expertise available.

HEFCE

discouraged cross

-reference of outputs to other panels. That it was used much more than expected suggests there were many areas in which panels did not feel competent to

assess submitted materials,

despite HEFCE’s

claims.Slide17

REF panels lack the expertise to do their job

REF panels rarely contain sufficient “breadth and depth of expertise” to evaluate the full range of outputs that fall under their remit

.

While panel members may be professionally eminent, they often lack the specialist knowledge to distinguish “internationally recognized” (2*), “internationally excellent” (3*) and “world-leading” (4*) outputs in the relevant fields.

Without specialist expertise, how can they judge the

originality

,

significance and rigor of a contribution—since all of these are field-specific qualities?

The REF History panel has

no

members who specialize in the history of China, Latin America, the Middle East, or many European countries—including such historically significant players as Spain, Turkey, and Austria-Hungary.

They don’t know the languages, the archives, the literatures: on what possible basis are they meant to judge?

“[The RAE] was not “peer review” in any case; peer review is done by journal referees, selected among the world’s top experts, in the case of the best journals, not from a single country’s rag-tag generic panel” (Stevan Harnad, January 2008). Slide18

REF panelists lack the time to do their jobLess than 1000 REF assessors grade nearly 200,000 outputs in less than a year

—the same number as the US National Endowment for the Humanities uses to evaluate 5700 applications for 40 grant programs!

Peter Coles calculates that each member of the Physics panel must read about 640 papers, concluding: “

It is blindingly obvious that whatever the panel does do will not be a thorough peer review of each paper, equivalent to refereeing it for publication in a journal.”

One anonymous RAE2008 panelist told Times Higher Education it would take him

“two years’ full-time work, while doing nothing else

,”

to properly read the 1200 journal articles he had been allocated.

HEFCE has acknowledged

panelists’ workload

as a problem, suggesting that any future REF instead consider “sampling” outputs despite REF panels’ unanimous opposition to this suggestion. Slide19

Is metrics a viable alternative?

“It is transparent and objective, it would not require departments to decide who they do and don’t enter for the assessment, and most importantly, it wins hands down on cost-effectiveness.” (Dorothy Bishop) Slide20

metrics and the REF—milestones2006 Gordon Brown announces

RAE to be replaced by metrics

after 2008.

HEFCE 2007-8 consultations show the UK’s academic establishment (UUK, Russell Group, 1994 Group, Research Councils, Royal Society, British Academy, HEIs, professional associations) overwhelmingly opposed to this.

HEFCE caves in: “

We just don’t think bibliometrics

are sufficiently mature at this stage to be used in a formulaic way or, indeed, to replace expert review

” (Graeme Rosenberg, HEFCE REF manager, June 2009).The quid pro quo was acceptance of

impact as an independent dimension of the assessment

(originally announced as

worth 25%

).

On the instruction of then Universities Minister David Willetts, HEFCE set up a panel in 2014 chaired by James Wilsdon

, to revisit the option of using metrics in the REF. Its preliminary findings suggest the views of the sector have not changed and any future introduction of metrics into the REF will be gradual and partial. Report due in May 2015. Slide21

The most common arguments against metricsJournal impact factors (

jifs

) cannot be used as proxy for quality because

high-impact journals may publish poor quality articlesUsing citations as evidence of quality ignores self-citation and negative citation

Major citation indexes ignore monographs

, which are more important than journal articles in humanities and some social science disciplines

Volumes of both outputs and citations differ across disciplines

, making comparisons problematic (to the assumed disadvantage of the humanities)

In my view, all

these objections could be accommodated by designing better metrics—don’t use

jifs

, exclude self-citation, use Google Scholar or Publish or Perish, which include monographs,

and standardize for disciplines or fields.Slide22

metrics reconsidered

Whatever the difficulties of metrics,

ignoring whether an output has gone through any prior process of expert peer review, where it has been published, how it has been received and how often it has been cited in

favor

of the subjective opinion of evaluators who may have no specialized expertise in the field is hardly a defensible

alternative.

Whatever their limitations as a valid

measure of the quality of individual outputs, metrics as used in global rankings have proved a very

reliable predictor of REF performance

, at least at the level of the institution.

Adoption of metrics

would deliver comparable outcomes to the REF for a fraction of financial and other costs—and without the pernicious and side-effects associated with staff selection at university level.

The

UK academic establishment’s insistence on maintaining “expert peer review” under these circumstances suggests they have a stake in the REF panel process itself, independent of its merits as a means of research evaluation.Slide23

Benefits of the REF?

“The

awful truth is that too many of us have learnt to love, as well as – even more than – fear

it” (Peter Scott, 2013) Slide24

“intolerable” process or “key instrument”?“

The

rot really set in when vice-chancellors ceased to see the RAE as a funding mechanism

and regarded it, instead, as a ‘free-standing assessment of research quality,’ with the added advantage of being ‘useful as a means to get rid of people not doing any research or to make them do more teaching. If that is what vice-chancellors want, they can conduct their own internal processes, but, nationwide, I don’t think such an exercise is justified.” Peter

Swinnerton

-Dyer (2013)

“

The RAE has also been the key instrument for performance management

in institutions

, and much of the obloquy that has been heaped on it has arisen from university managements doing what they should do but sheltering behind the pretext of the RAE.

To

this extent, the RAE has done more than drive research quality; it has been crucial to modernisation.” David Eastwood (2007)Could they both be right?Slide25

5 reasons why established academic elites like the REF (follow the money)?The REF is an

excellent tool of internal discipline for university managers

,

making all research activity—not just that which is externally funded—financially accountable.The REF machinery is also tailor-made for the reproduction of disciplinary hierarchies and networks

(selection of panel members, composition of panels, secrecy of panel evaluative procedures).

The REF is

a crucial vehicle of legitimation

: it is the fact that this is a national exercise carried out under the watchful eye of a public

body that gives REF scores their

unique authority. Birmingham CCCS example.

The

major beneficiaries of this have been Russell Group and other traditional elite pre-1994 universities—who also dominate REF panels.The REF thus both legitimates this dominance and is a key vehicle for financially and otherwise maintaining it. Slide26

Alternatives to the REF?Slide27

General principles for restoring sanity

Return to what the RAE was originally intended to be, viz. a mechanism for allocating research funding between universities

—rather than an all-purpose tool of university management, or a vehicle for maintaining the power and prestige of academic elites.

Confine information gathered for the assessment to what is necessary and sufficient to enable reasonably informed, objective, and fair judgments of institutional quality to be made. Do we

really

need to have four outputs from every academic in the land? Fewer outputs might allow for more rigorous peer review, involving genuine subject-matter experts.

Wherever possible

, use discipline-appropriate metrics to substitute for or at least inform subjective “expert” judgment.

If we must have an REF

, make it as cost-efficient as possible, and try to minimize opportunities for gaming and any unfairness to individuals resulting from “strategic” processes of staff selection

. Slide28

Some specific suggestionsReconsider the dual support system

. Abolishing QR and routing all support through the research councils could offer

a more level playing field for universities and individual scholars

if the councils were mandated to use the enhanced budget to support individual scholarship and emergent excellence. Neither REF-style peer-review nor metrics seem capable of delivering valid judgments of the quality of research outputs, yet

the REF bases 65

% of its ranking on

outputs

. It seems silly to give such weight to something that we cannot measure and that may be inherently unmeasurable.

Perhaps we should instead

ask

which

features of the research environment (which currently counts for a mere 15% of the REF assessment) are most conducive to a vibrant research culture and focus funding accordingly. Library and laboratory resources, research grant income, faculty members’ involvement in conferences, journal or series editing, and professional associations, PhD student numbers and the intellectual life of a department as reflected in research seminars and public lectures are all good indicators of research vitality.  They are also eminently measurable.