Looking at assessing social and wellbeing impacts of projects for heritage and cultural institutions Lessons from the HLF experience VSG conference London March 2017 Eirini Gallou Kalliopi ID: 565929
Download Presentation The PPT/PDF document "Evaluating evaluation :" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Evaluating evaluation :Looking at assessing social and wellbeing impacts of projects for heritage and cultural institutions_Lessons from the HLF experience
VSG conference
London, March 2017
Eirini Gallou,
Kalliopi
Fouseki
./ UCL, ISH Institute
TRACEY-ANN SMITH / TAS creationsSlide2
PhD research: the experience of HLF Landscape , multiple goals and activity typesA 3 year scheme2 years after post project evaluation of social and professional practice impacts
A new integrated framework?
Heritage benefits
(Biodiversity, built and archaeological features, museums and archives, intangible heritage)
>>>holistic
People benefits
(Community, access and learning, training and skills) Slide3
Existing evaluation framework by HLF
Outputs
Desired outcomes
Actual outcomes
Legacy & recommendations
-Usually quantifiable concrete products/results Eg. creation of infrastructure (action paths)Expected changes due to outputsEg. Improved public access to an areaRealised changes due to outputs-Quantifiable; eg. Increased numbers of visitors in sites -Qualitative changes: Eg.Public acceptanceLong term persisting changesLocal suggestions -crowdsource opinions on improvementDefine certain relevant themes: access, visibility, future participation Slide4
PhD research: the experience of HLF Landscape schemes in post project evaluationWhat is there to learn?
Types of impacts: heritage benefits and people benefits
Integration in execution best in planning phase-formative
Legacy and recommendations: towards holistic framework and evaluating sustainability
What is there to add and explore further?
From outcomes to impacts; pathways to changeLong term evaluation of social impacts : adapt evaluation design? Social impacts and non-audiences: who is left aside?Organizational impacts and changes in our practice!Slide5
Close-ended questionnaires with a sample of 250 community people, residing in the area of projects
Semi structured interviews with selected participants & managing team in
two specific projects
Post project evaluation & Social impact assessment
PhD Research design
Data collection & analysis
1
3
>>18 Semi structured
Interviews
with both managers and participants in
various projects
within same schemeSlide6
Step 1_Assessing social impacts for whom?Community needs and assets mappingComplementary approaches to community mapping;
asset mapping
social network mapping
social needs mapping
Some elements to map include;
‘physical spaces - used by the communitycommunity institutions organizations connected to those institutions or doing related workservices, informal groups, and the last key individuals (REVEALING INTERCONNECTIONS AND LINKS WILL HELP YOU UNDERSTAND YOUR FUTURE PARTNERS!Slide7
Step 2_ Process, relevant social/wellbeing impact areas + frameworksSlide8
Step 2_ Focus and relevant framework(S)Health and wellbeing: personal –social, direct /indirect impacts?
Interlinkages & overlaps
Sub-aspects and indicators
Some frameworks to consider for assessing social and personal wellbeing impacts :
Life satisfaction and Affectivity indicators
SWE Mental Wellbeing Scale
5 ways to wellbeing-NEF
The Generic Social outcomes
Personal and social impacts-NEF indicators
HLF 3 year scheme community impacts indicators
Social capital- Edinburgh,
Scot.
Government Council Indicators and toolkitSlide9
Step 3_ Formulate questions to assess social impacts: identifying levels and relevance
LEVELS
CORE
QUESTIONSSlide10
Advice for improvements and methodological tips Understand Current Needs In A Community Level >>community needs mappingUnderstand
unaffected groups
+ underrepresented Groups>> community mapping and Identification
Participation types
: understand levels of engagement and relation with impacts to be sought!
Think of representativeness of sample and depth of knowledge of population>> Knock On effects: Family And Close Ones>> Need for new tools , like social network AnalysisLong term Changes: repeatability of Surveys and consistency in documentation to support longitudinal evaluation design!Slide11
Indicators and metrics: what is out there?Slide12
Notions of wellbeing within social impacts
GLO learning outcomes >> GSO social outcomes >>Generic wellbeing outcomes
Personal well-being:
3 aspects
: Emotional,
Vitality
& positive functioning
Social well-being: Supportive relationships
Recovery from
physical illness
& improved hospital care
Source: (Ander et all, 2011) Slide13
Health and wellbeing , health within wellbeing or health vs wellbeing? NEF definition of wellbeing: A holistic definition of wellbeing: generic /
eudaimonic
/ hedonic
Scales of assessment ; personal & socialSlide14
Personal level & community-social level
Interlinkages
Interlinkages & overlapsSlide15
Subjective WellbeingCovers Hedonic aspects: :‘a person’s cognitive and affective evaluations of his or her life’ (Diener
, Lucas, &
Oshi
, 2002, p. 63)
The
cognitive element : what one thinks about his or her life satisfaction in global terms (life as a whole) and in domain terms (in specific areas of life such as work, relationships, etc.) The affective element : emotions, moods and feelings > 3 components of SWB, life satisfaction, positive affect and negative affect, (Andrews & Withey, 1976, Lucas et al., 1996). Life satisfaction questionnairesAffectivity (positive and negative) the PANAS [balance between positive affect negative affect schedule] (Watson, Clark & Tellegan, 1988) or Affect Balance Scale (ABS) (Bradburn 1969) Slide16
Psychological WellbeingPlaces less of an emphasis on how people feel (affect and satisfaction), and more of an emphasis on how well they function.
A conception of well-being as an
active
process that has to be constituted and sustained through intentional
activities
– rather than a more passive process rooted in people’s feelings. Covers Eudaimonic aspectsSlide17
A. Psychological General Well-being Scale (Dupuy
1984)
Consists of eighteen items within six dimensions (
anxiety, depression, positive well-being, self-control, vitality, general health
).
+ captures many dimensions including clinically recognized aspects, of mental health- outdated/ complexity of questions and length increases/ difficulty in use in certain groups (see SWB)B. The Affect Balance Scale (ABS, Bradburn 1969) :‘an indicator of happiness or general psychological well-being’ (Bowling 2005, 132). In this self-administered scale, well-being is conceptualized as a balance between: positive affect and negative affect.See also Panas scales eg. scale looking at mood and emotion (Watson, Clark, and Tellegen, 1988), + self administered can be both positive, less bias in asking questions
_positive and negative aspect are not always fully representative of truth, only affective aspects covered
Metrics & indicators:
Mental health, psychological wellbeing around concept of
SWB and hedonic aspects Slide18
Metrics & indicators: Happiness and psychological wellbeing_PSW and eudaimonic aspects
C. Warwick-Edinburgh Mental Well-being scale:
feelings and functioning
, all positive-validated !
D. Foresight & NEF looked at happiness as determined by:Our circumstances (where we are)Our personal resources (who we are)The activities where we focus our energy (what we do).+holistic understanding of internal and external factors, simplified aspects-difficulty in determining all aspects of identity and prosocial behavior E. Ryff (1989) - scale of psychological wellbeing six dimensions of well-being, 20 items for each! Springer and Hauser (2006) reduced it to 3 dimensions!+A six-point answering scale, enabling even statistical analysis/bigger samples!
-empirical tests have been applied to validate only in specific contexts
Self acceptance
Environmental mastery
Autonomy,
Purpose in life/ personal growth,
Positive relations with othersSlide19
1. Existing toolkit of indicators on social capital: NEF’s Proving and improving
NEF
INDIVIDUAL
Social Capital
Communities Needs
COMMUNITIES
Well-being
Skills Development
Health
High Risk
BehaviourSlide20
2. Existing toolkit for social capital and health_ Scottish GovernmentScottish Edinburgh Health Inequalities Standing Group
,
indicators for three aspects of social capital; bridging, bonding and linking
and relation with specific activities!Slide21Slide22
3. Existing HLF indicators toolbox –3 year volunteer programme evaluation_PWB and individual level
Predominantly
PWB measures
that investigate
cognitive
functioning and social relationships, in addition to the subjective measure of ‘happiness’. The five items are:Ablity to concentrate Capability to make decisions Social engagement and self worth (‘playing a useful part in thingsAbility to enjoy normal day-to-day activities Levels of happiness Based on General Health Questionnaire (GHQ). Additional well-being issues: curiosity and absorption in activities + wider ‘quality of life’ questions, (how enjoyable -what is the single best thing that they gain )Slide23
3. HLF indicators_ 3 year volunteer programme evaluation _Social capital and community level
Civic engagement :
participation in parent-teachers associations, citizens’ juries, or community forums (individual level overlaps mentioned)
Collective :
individuals
acting together can affect outcomes in their community. This can range from the informal regulation of codes of conduct, stopping an adult hitting a child for example, or a young person from vandalising, to more formal outcomes. (individual level overlaps mentioned)Other aspects relevant to social life:SocialisingIntergenerational outcomesCommunity focusSense of belonging to placeCommunity cohesionSlide24
4. Impacts to health and wellbeing –The GSO’sSlide25
APPENDIX_ METRICS AND INDICATORS RESOURCES 3 resources for social capital related questions:NEF Prove and Improve: Available at: http://www.proveandimprove.org/meaim/SocialIndicators-Communities.php
Edinburgh Health Inequalities Standing Group , Available at: http://www.scdc.org.uk/media/resources/what-we-do/mtsc/Social%20Capital%20Health%20and%20Wellbeing%20toolkit.pdf
HLF, Assessment of the social impact of volunteering in HLF funded projects: Year 2 and year 3 available
online
6 resources for SWB and PWB related questions:
5 item satisfaction with life questionnaire (Diener, Emmons, Larsen & Griffin, 1985; Pavot & Diener, 1993)A relevant paper is available at: http://www.midss.org/sites/default/files/swls_english.pdf Affectivity : the PANAS [balance between positive affect negative affect schedule] (Watson, Clark & Tellegan, 1988) or Affect Balance Scale (ABS) (Bradburn 1969)a self-administered scaleWarwick-Edinburgh Mental Well-being scale . Relevant publication available at: http://www.brandeis.edu/roybal/docs/PANAS-GEN_website_PDF.pdfNEF 5 ways to wellbeing: As applied by a local council in Rochdale available at: http://fiveways.rochdale.gov.uk/survey/Ryff (1989) - scale of psychological wellbeing and follow-up researches. Relevant paper available at: https://www.uam.es/gruposinv/esalud/Articulos/Psicologia%20Positiva/Ryff-six-factor.pdfSlide26
APPENDIX_1 SENARIO(S)A real life and up to date scenario:Talkin ’Bout My Generation: Young and older people in the Kings Cross area
Context: Covering one aspect of cultural diversity ,in an urban area with increased racial tensions
Participants Group: 40 people, multiethnic, intergenerational
Activities: older people with young adults
bring stories together,
within the community centre/ museum visits/ performance & exhibition as end result Frequency: twice a month for 1 yearSlide27
Follow up exercise & reflectionDevelop at least 5 principal questions that you could use for interviews or surveys with the participants.Reflect on 3 themes for following discussion:
Long and short term impacts covered by the framework you used
Personal and community level impacts, areas of impacts covered in relation to project outputs and evaluation aims
Adaptation and alignment to current frameworks you have been using…Slide28
Suggested matrix for exercise
Long term or shorter term
Social impacts covered
Method and comments: adapted to current frameworks you use/to funders’ standards?
Questionnaire 1: WEMWB Scale
Questionnaire 2: NEF 5 ways to wellbeingSlide29
Suggested matrix for exercise-social capital
Long term or shorter term
Social impacts covered
Method and comments: adapted to current frameworks you use/to funders’ standards?
Questionnaire 1:NEF
Questionnaire 2: Scot. Government Council social capital toolboxQuestionnaire 3:HLF