Evaluating evaluation :

Presentation on theme: "Evaluating evaluation :"— Presentation transcript:

Slide1

Evaluating evaluation :Looking at assessing social and wellbeing impacts of projects for heritage and cultural institutions_Lessons from the HLF experience

VSG conference

London, March 2017

Eirini Gallou,

Kalliopi

Fouseki

./ UCL, ISH Institute

TRACEY-ANN SMITH / TAS creationsSlide2

PhD research: the experience of HLF Landscape , multiple goals and activity typesA 3 year scheme2 years after post project evaluation of social and professional practice impacts

A new integrated framework?

Heritage benefits

(Biodiversity, built and archaeological features, museums and archives, intangible heritage)

>>>holistic

People benefits

(Community, access and learning, training and skills) Slide3

Existing evaluation framework by HLF

Outputs

Desired outcomes

Actual outcomes

Legacy & recommendations

-Usually quantifiable concrete products/results Eg. creation of infrastructure (action paths)Expected changes due to outputsEg. Improved public access to an areaRealised changes due to outputs-Quantifiable; eg. Increased numbers of visitors in sites -Qualitative changes: Eg.Public acceptanceLong term persisting changesLocal suggestions -crowdsource opinions on improvementDefine certain relevant themes: access, visibility, future participation Slide4

PhD research: the experience of HLF Landscape schemes in post project evaluationWhat is there to learn?

Types of impacts: heritage benefits and people benefits

Integration in execution best in planning phase-formative

Legacy and recommendations: towards holistic framework and evaluating sustainability

What is there to add and explore further?

From outcomes to impacts; pathways to changeLong term evaluation of social impacts : adapt evaluation design? Social impacts and non-audiences: who is left aside?Organizational impacts and changes in our practice!Slide5

Close-ended questionnaires with a sample of 250 community people, residing in the area of projects

Semi structured interviews with selected participants & managing team in

two specific projects

Post project evaluation & Social impact assessment

PhD Research design

Data collection & analysis

1

3

>>18 Semi structured

Interviews

with both managers and participants in

various projects

within same schemeSlide6

Step 1_Assessing social impacts for whom?Community needs and assets mappingComplementary approaches to community mapping;

asset mapping

social network mapping

social needs mapping

Some elements to map include;

‘physical spaces - used by the communitycommunity institutions organizations connected to those institutions or doing related workservices, informal groups, and the last key individuals (REVEALING INTERCONNECTIONS AND LINKS WILL HELP YOU UNDERSTAND YOUR FUTURE PARTNERS!Slide7

Step 2_ Process, relevant social/wellbeing impact areas + frameworksSlide8

Step 2_ Focus and relevant framework(S)Health and wellbeing: personal –social, direct /indirect impacts?

Interlinkages & overlaps

Sub-aspects and indicators

Some frameworks to consider for assessing social and personal wellbeing impacts :

Life satisfaction and Affectivity indicators

SWE Mental Wellbeing Scale

5 ways to wellbeing-NEF

The Generic Social outcomes

Personal and social impacts-NEF indicators

HLF 3 year scheme community impacts indicators

Social capital- Edinburgh,

Scot.

Government Council Indicators and toolkitSlide9

Step 3_ Formulate questions to assess social impacts: identifying levels and relevance

LEVELS

CORE

QUESTIONSSlide10

Advice for improvements and methodological tips Understand Current Needs In A Community Level >>community needs mappingUnderstand

unaffected groups

+ underrepresented Groups>> community mapping and Identification

Participation types

: understand levels of engagement and relation with impacts to be sought!

Think of representativeness of sample and depth of knowledge of population>> Knock On effects: Family And Close Ones>> Need for new tools , like social network AnalysisLong term Changes: repeatability of Surveys and consistency in documentation to support longitudinal evaluation design!Slide11

Indicators and metrics: what is out there?Slide12

Notions of wellbeing within social impacts

GLO learning outcomes >> GSO social outcomes >>Generic wellbeing outcomes

Personal well-being:

3 aspects

: Emotional,

Vitality

& positive functioning

Social well-being: Supportive relationships

Recovery from

physical illness

& improved hospital care

Source: (Ander et all, 2011) Slide13

Health and wellbeing , health within wellbeing or health vs wellbeing? NEF definition of wellbeing: A holistic definition of wellbeing: generic /

eudaimonic

/ hedonic

Scales of assessment ; personal & socialSlide14

Personal level & community-social level

Interlinkages

Interlinkages & overlapsSlide15

Subjective WellbeingCovers Hedonic aspects: :‘a person’s cognitive and affective evaluations of his or her life’ (Diener

, Lucas, &

Oshi

, 2002, p. 63)

The

cognitive element : what one thinks about his or her life satisfaction in global terms (life as a whole) and in domain terms (in specific areas of life such as work, relationships, etc.) The affective element : emotions, moods and feelings > 3 components of SWB, life satisfaction, positive affect and negative affect, (Andrews & Withey, 1976, Lucas et al., 1996). Life satisfaction questionnairesAffectivity (positive and negative) the PANAS [balance between positive affect negative affect schedule] (Watson, Clark & Tellegan, 1988) or Affect Balance Scale (ABS) (Bradburn 1969) Slide16

Psychological WellbeingPlaces less of an emphasis on how people feel (affect and satisfaction), and more of an emphasis on how well they function.

A conception of well-being as an

active

process that has to be constituted and sustained through intentional

activities

– rather than a more passive process rooted in people’s feelings. Covers Eudaimonic aspectsSlide17

A. Psychological General Well-being Scale (Dupuy

1984)

Consists of eighteen items within six dimensions (

anxiety, depression, positive well-being, self-control, vitality, general health

).

+ captures many dimensions including clinically recognized aspects, of mental health- outdated/ complexity of questions and length increases/ difficulty in use in certain groups (see SWB)B. The Affect Balance Scale (ABS, Bradburn 1969) :‘an indicator of happiness or general psychological well-being’ (Bowling 2005, 132). In this self-administered scale, well-being is conceptualized as a balance between: positive affect and negative affect.See also Panas scales eg. scale looking at mood and emotion (Watson, Clark, and Tellegen, 1988), + self administered can be both positive, less bias in asking questions

_positive and negative aspect are not always fully representative of truth, only affective aspects covered

Metrics & indicators:

Mental health, psychological wellbeing around concept of

SWB and hedonic aspects Slide18

Metrics & indicators: Happiness and psychological wellbeing_PSW and eudaimonic aspects

C. Warwick-Edinburgh Mental Well-being scale:

feelings and functioning

, all positive-validated !

D. Foresight & NEF looked at happiness as determined by:Our circumstances (where we are)Our personal resources (who we are)The activities where we focus our energy (what we do).+holistic understanding of internal and external factors, simplified aspects-difficulty in determining all aspects of identity and prosocial behavior E. Ryff (1989) - scale of psychological wellbeing six dimensions of well-being, 20 items for each! Springer and Hauser (2006) reduced it to 3 dimensions!+A six-point answering scale, enabling even statistical analysis/bigger samples!

-empirical tests have been applied to validate only in specific contexts

Self acceptance

Environmental mastery

Autonomy,

Purpose in life/ personal growth,

Positive relations with othersSlide19

1. Existing toolkit of indicators on social capital: NEF’s Proving and improving

NEF

INDIVIDUAL

Social Capital

Communities Needs

COMMUNITIES

Well-being

Skills Development

Health

High Risk

BehaviourSlide20

2. Existing toolkit for social capital and health_ Scottish GovernmentScottish Edinburgh Health Inequalities Standing Group

,

indicators for three aspects of social capital; bridging, bonding and linking

and relation with specific activities!Slide21
Slide22

3. Existing HLF indicators toolbox –3 year volunteer programme evaluation_PWB and individual level

Predominantly

PWB measures

that investigate

cognitive

functioning and social relationships, in addition to the subjective measure of ‘happiness’. The five items are:Ablity to concentrate Capability to make decisions Social engagement and self worth (‘playing a useful part in thingsAbility to enjoy normal day-to-day activities ƒLevels of happiness Based on General Health Questionnaire (GHQ). Additional well-being issues: curiosity and absorption in activities + wider ‘quality of life’ questions, (how enjoyable -what is the single best thing that they gain )Slide23

3. HLF indicators_ 3 year volunteer programme evaluation _Social capital and community level

Civic engagement :

participation in parent-teachers associations, citizens’ juries, or community forums (individual level overlaps mentioned)

Collective :

individuals

acting together can affect outcomes in their community. This can range from the informal regulation of codes of conduct, stopping an adult hitting a child for example, or a young person from vandalising, to more formal outcomes. (individual level overlaps mentioned)Other aspects relevant to social life:SocialisingIntergenerational outcomesCommunity focusSense of belonging to placeCommunity cohesionSlide24

4. Impacts to health and wellbeing –The GSO’sSlide25

APPENDIX_ METRICS AND INDICATORS RESOURCES 3 resources for social capital related questions:NEF Prove and Improve: Available at: http://www.proveandimprove.org/meaim/SocialIndicators-Communities.php

Edinburgh Health Inequalities Standing Group , Available at: http://www.scdc.org.uk/media/resources/what-we-do/mtsc/Social%20Capital%20Health%20and%20Wellbeing%20toolkit.pdf

HLF, Assessment of the social impact of volunteering in HLF funded projects: Year 2 and year 3 available

online

6 resources for SWB and PWB related questions:

5 item satisfaction with life questionnaire (Diener, Emmons, Larsen & Griffin, 1985; Pavot & Diener, 1993)A relevant paper is available at: http://www.midss.org/sites/default/files/swls_english.pdf Affectivity : the PANAS [balance between positive affect negative affect schedule] (Watson, Clark & Tellegan, 1988) or Affect Balance Scale (ABS) (Bradburn 1969)a self-administered scaleWarwick-Edinburgh Mental Well-being scale . Relevant publication available at: http://www.brandeis.edu/roybal/docs/PANAS-GEN_website_PDF.pdfNEF 5 ways to wellbeing: As applied by a local council in Rochdale available at: http://fiveways.rochdale.gov.uk/survey/Ryff (1989) - scale of psychological wellbeing and follow-up researches. Relevant paper available at: https://www.uam.es/gruposinv/esalud/Articulos/Psicologia%20Positiva/Ryff-six-factor.pdfSlide26

APPENDIX_1 SENARIO(S)A real life and up to date scenario:Talkin ’Bout My Generation: Young and older people in the Kings Cross area

Context: Covering one aspect of cultural diversity ,in an urban area with increased racial tensions

Participants Group: 40 people, multiethnic, intergenerational

Activities: older people with young adults

bring stories together,

within the community centre/ museum visits/ performance & exhibition as end result Frequency: twice a month for 1 yearSlide27

Follow up exercise & reflectionDevelop at least 5 principal questions that you could use for interviews or surveys with the participants.Reflect on 3 themes for following discussion:

Long and short term impacts covered by the framework you used

Personal and community level impacts, areas of impacts covered in relation to project outputs and evaluation aims

Adaptation and alignment to current frameworks you have been using…Slide28

Suggested matrix for exercise

Long term or shorter term

Social impacts covered

Method and comments: adapted to current frameworks you use/to funders’ standards?

Questionnaire 1: WEMWB Scale

Questionnaire 2: NEF 5 ways to wellbeingSlide29

Suggested matrix for exercise-social capital

Long term or shorter term

Social impacts covered

Method and comments: adapted to current frameworks you use/to funders’ standards?

Questionnaire 1:NEF

Questionnaire 2: Scot. Government Council social capital toolboxQuestionnaire 3:HLF