Achieving and Demonstrating Research Impact - PowerPoint Presentation

Slide1

Achieving and Demonstrating Research Impact

John ScottSlide2

Workshop Outline

What

is ‘impact’ and how can it be achieved?

This part of the workshop will consider the REF definition of impact, the various types of impact, and the criteria for scoring impact. It will consider the broad range of types of impact and their varying advantages and disadvantages. It will also consider how the achievement of impact might be built into research projects and the support mechanisms that Universities can establish to promote the achievement of impact.

Coffee break

How can impact be demonstrated?

In this part of the workshop I will consider the issue of producing evidence on impact and of constructing a plausible account of the impact achieved. We will consider the issues involved in constructing and presenting appropriate case studies.

The Future of the REF.

This final session will be an open discussion of wider issues in the REF: panel structure, assessment mechanisms,

bibliometrics

, etc. It is an opportunity to explore issues and potentialities in the approach to REF 2020.

Final Questions Slide3

What is ‘impact’ and how can it be achieved?Slide4

The Development of Research Assessment

1986 UGC Review

Subject areas rated as

’outstanding’, ‘above average’,

‘average’

or

‘below average’.

1989 UFC Research Selectivity Exercise

1992 HEFCE (S&W) Research Assessment Exercise

Subject areas rated on scale from 1 to 5.

Grade 4 means ‘some evidence of international excellence’.

1996 RAE

Grade 3 split into

3a

and

3b

.

3a

‘

 possibly showing

evidence

of international excellence

.’

2001 RAE

2008 RAE

Each subject area given a ‘profile’ with % of activity at

levels from 1 to 4. Grade 2 means ‘

Quality that is

recognised

internationally in terms of originality,

significance

and

rigour

2014 Research Excellence Framework (REF)Slide5

All exercises involved, to a greater or lesser degree, an assessment

of publications and the research environment. Later exercises

a

dded assessment of ‘esteem’.

20014 REF introduced the assessment of ‘impact’, replacing ‘esteem’.Slide6

Subject Panels

Assessments made by ‘expert’ panels, generally

discipline-based, but have become broader in composition.

Subject panels coordinated by ‘Main Panels’ covering large

areas.

Universities can choose how to group their staff in

subject areas and which panels to submit them to.Slide7

A

B

C

D

1 Clinical Medicine

2 Public Health, Health Services and Primary Care

3 Allied Health Professions, Dentistry, Nursing and Pharmacy

4 Psychology, Psychiatry and Neuroscience

5 Biological Sciences

6 Agriculture, Veterinary and Food Science

7 Earth Systems and Environmental Sciences

8 Chemistry

9 Physics

10 Mathematical Sciences

11 Computer Science and Informatics

12 Aeronautical, Mechanical, Chemical and Manufacturing Engineering

13 Electrical and Electronic Engineering, Metallurgy and Materials

14 Civil and Construction Engineering

15 General Engineering

16 Architecture, Built Environment and Planning

17 Geography, Environmental Studies and Archaeology

18 Economics and Econometrics

19 Business and Management Studies

20 Law

21 Politics and International Studies

22 Social Work and Social Policy

23 Sociology

24 Anthropology and Development Studies

25 Education

26 Sport and Exercise Sciences, Leisure and Tourism

27 Area Studies

28 Modern Languages and Linguistics

29 English Language and Literature

30 History

31 Classics

32 Philosophy

33 Theology and Religious Studies

34 Art and Design: History, Practice and Theory

35 Music, Drama, Dance and Performing Arts

36 Communication, Cultural and Media Studies, Library and Information ManagementSlide8

The ‘Profile’ system

Each element of the assessment (outputs, environment, and

esteem/impact) assessed separately. Panels judge the % of

Each element that is rated at the quality levels.

Profiles for each element combined into an overall profile using

a ‘weighting’.

Weightings used in REF:

Outputs 65%

Environment 15%

Impact 20%Slide9

Structure of the Impact Assessment

Universities must submit one case study for about every eight staff submitted,

with a minimum of two case studies.

Must also submit an Impact Statement – strategy and approach to impact.

Case Studies and Statement are equal constituents of the assessment.Slide10

Where did ‘impact’ come from?

Government demand for ‘value for money’.

Public reaction to ‘ivory towers’

HEFCE pre-empted Government action by introducing ‘impact

assessment’.

Consultation and debate led to broadening of meaning of ‘impact’.Slide11

Definition of impact for the REF

(From

Assessment Framework and Guidance on Submissions

)

140. For the purposes of the REF, impact is defined as an effect on, change or benefit

to the economy, society, culture, public policy or services, health, the environment

or quality of life, beyond academia (as set out in paragraph 143).

141. Impact includes, but is not limited to, an effect on, change or benefit to:

• the activity, attitude, awareness, behaviour, capacity, opportunity, performance,

policy, practice, process or understanding

• of an audience, beneficiary, community, constituency, organisation or individuals

• in any geographic location whether locally, regionally, nationally or internationally.

142. Impact includes the reduction or prevention of harm, risk, cost or other

negative effects.

Paragraph 143 excludes ‘academic impact’ and impact on students.Slide12

This definition is specified by each main panel.

There are slight differences, so it is important to know which

panel is to be submitted to.

I will illustrate with the criteria for Main Panel C:

‘The main panel acknowledges that impact within its remit may take many forms

and occur in a wide range of spheres. These may include (but are not restricted to):

creativity, culture and society; the economy, commerce or organisations;

the environment; health and welfare; practitioners and professional services;

public policy, law and services.

‘The categories used to define spheres of impact, for the purpose of this document,

inevitably overlap and should not be taken as restrictive. Case studies may describe

impacts which have affected more than one sphere.’Slide13
Slide14
Slide15

The criteria for assessing impacts are ‘reach and significance’.:

‘Reach

will be understood in terms of the extent and diversity of the communities, environments, individuals, organisations or any other beneficiaries that have benefited or been affected

.

‘Significance

will be understood in terms of the degree to which the impact has enriched, influenced, informed or

changed

policies, opportunities, perspectives or practices of communities, individuals or organisations

.’ (Panel C Criteria)

These are assessed on the scale of starred levels:

Four star Outstanding impacts in terms of their reach and significance.

Three star Very considerable impacts in terms of their reach and significance.

Two star Considerable impacts in terms of their reach and significance.

One star Recognised but modest impacts in terms of their reach and significance.

Unclassified The impact is of little or no reach and significance; or the impact was not eligible; or the impact was not underpinned by excellent research produced by the submitted unit.’Slide16

Key considerations:

Eligibility of impact

Research must have been carried out at the institution between 1

st

Jan 1993 and 31

st

December 2013. i.e., 20 year period up to submission date. For REF 2020, probably research from 1

st

Jan 2000.

Impact must be apparent between 1

st

January 2008 and 31

st

July 2013. For REF 2020, probably impact after 1

st

January 2014.

Presentation of evidence

Must put in place mechanisms for collecting evidence on impact. This needs to be an integral part of the research and of Department/Faculty practice.

Support at institution/unit level

Must have mechanisms and strategy in place that support

project-level activities.Slide17

How can impact be demonstrated?Slide18

What is the panel looking for?

Clear Evidence

. Assessment must be based on what is on the

paper.

This involves three key aspects:

Quality of underpinning research: the 2* threshold

Significance and reach of impact: the evidence

Compelling narrative linking research to impactSlide19

Key questions considered by panel in assessing case studies:

What was the research activity: a person, a project, or a programme?

Was research actually carried out at the university?

Was the research carried out within the time period allowed?

Did the impact occur within the time period allowed?

Did the research actually contribute to the impact?

Is there supporting evidence for the impact?Slide20

Issues in considering Impact Statements

Is an impact strategy demonstrated?

Are institutional support mechanisms actually taken up?

Relation of case studies to Impact Statement. Are they

examples of a successful application of strategy,

examples that have informed the development of a strategy, or

serendipitous successes from which something can be learned.Slide21

Problems apparent in the assessment

Impact assessment was based on a small number of elements, so results

were ‘lumpy’ compared with outputs.

This affected the overall assessment profile

What may change for 2020

Impact weighting unlikely to go down

More metrics to help avoid the ‘lumpy’ distribution

Panel configuration – key issues about which panel to submit to.