The credit union difference: how to measure and report it - PowerPoint Presentation

1. The credit union difference: how to measure and report it – .PPT materials for the Toolkit Funded by:

2. Social Impact Measurement for Credit UnionsWorkshop 1 on Design Unify 23 July 2018 Hoot 24 July 2018

3. Agenda11.00 - Introduction11.10 - Session A – Unpacking the ideas 11.50 - Session B – Mapping stakeholders' issues12.30 - Session C – Formulating the theory 1.10 - Lunch1.30 - Session D – Refining the theory 2.10 - Session E – Reviewing the theory 2.50 – Conclusion3.00 – Ends

4. 11.00 - IntroductionBackground to the project AgendaExpectations

5. 11.10 - Session A - Unpacking the ideas in ‘SIM’Impact...Measurement...Reporting...ToC thinking...Management...Learning Questions A

6. Impact...Is a long term, positive, durable & sustainable feature of the financial circumstances of members & their communities to which the credit union contributes through working with them; Flows from the members’ engagement with the credit union; Varies with the financial circumstances of the members, which reflects differences between ‘affluent’, ‘coping’ and ‘vulnerable’ members;

7. Measurement...Is the means through which we collect evidence to demonstrate impact; Makes use of either or both qualitative & quantitative methods; Should extract value from existing before collecting new data; May rely on direct or indirect (proxy) sources; Should be proportionate (costs to benefits); E.g. SROI / HACT’s Social Value; BSC's Outcomes Matrix;

8. Reporting...Typically written reports, making extensive use of graphics to enhance visual appeal; ‘Case studies, ‘stories’, ‘dashboards’, ‘scorecards’, ‘personas’, etc; The issue is whether the reporting meets the legitimate interests of the stakeholder; Does it convince them that the credit union is having the impact it claims?

9. ToC thinking… 1) Analyse context about needs, barriers, enablers, etc, 2) examine assumptions about how we may intervene, 3) assess the evidence that what we do brings about the desired change; Articulating / surfacing & linking / connecting the imp elements of contexts, assumptions, evidence; Making explicit how what we deliver (memberships, loans, savings) has effects that lead to positive changes in financial circumstances;

10. Management...Sustains relationships with stakeholders; Evidences the case for resource support from public sector; Enhances governance through facilitating accountability; Creates basis for persuasive / marketing communications; Contributes to more informed decision-making at operational and strategic levels;

11. Discussion QuestionsWhat do you want social impact measurement to deliver for the credit union? What challenges for the credit union arise from social impact measurement? Who wants to know about the impact of the credit union - to whom should the credit union report?

12. 11.50 - Session B - Mapping stakeholders' issuesWhy map? Clarifying (any) gaps in current reporting to these stakeholders? Consider what is the impact that the interested and influential stakeholders want the credit union to deliver.

13. Why map?Who is the audience for the reportage? Map / matrix of stakeholders (s), what aspect of your impact (i) do you want to communicate to them?

14. The key questions are: What is the issue that you and they want addressed? What are the key factors that influence this issue? Whose behaviour needs to change in ways that address the issue? How ill the credit union enable these people to change their behaviour?

15. Map / matrix of stakeholders (s), showing what aspect of your impact (i) do you want to communicate to them? Influence is high s) i) s) i) s) i) Influence is moderate a) i) a) i) a) i) Influence is low a) i) a) i) a) i) Stakeholders’ Influence on, by Interest in, CU Interest is low Interest is moderate Interest is high Esp housing, bodies and local government's interest in impacts flowing from saving & loans

16. Clarifying (any) gaps in current reporting to these stakeholders? Discuss (any) gaps in the reporting through which the credit union communicates impact to these stakeholders (esp. those in blue cells). What evidence would convince the stakeholder(s) about this aspect of the impact? is the credit union providing this evidence? What needs to be done to fill these gaps? Is it feasible that the credit union would do this? Would this be proportionate to the benefits?

17. Consider what is the impact that the interested and influential stakeholders want the credit union to deliver. Is it sensible to frame this issue as 'too few people borrow from (or save with) the credit union'? EXERCISE 'But why do too few...'

18. 12.30 - Session C - Formulating the ToCWhat is ToC? Why is ToC important for impact measurement? Assumptions & PropositionsPathways & HypothesesWhat should they look like?Understanding the theory / logic underlying the credit union's impact

19. What is ToC? an approacha methoda diagrama definitionclaims for ToC

20. What is the ToC approach?Is about understanding the dynamic in which the credit union intervenes (to secure the impact) Financial wellbeing, and how it is fostered, is complex rather than linear; We should have a set of ideas to understand the process, i.e. a theory; With the theory we will be better able to learn about what does or not work and communicate this to interested parties;

21. ToC, a definitionA set of hypotheses and critical assumptions that make up a causal pathway of change which is the basis of the ... design [of the intervention].Hypotheses are ‘if-then’ statements between different levels of the change pathway (Drinkwater; 2009)

22. Claims for ToCIntegrate planning, implementation, and evaluation.Prevent mismatches between activities and effects.Enhance accountability by keeping stakeholders focused on outcomes.Help set priorities for allocating resources.Reveal data needs and provide a framework for interpreting results.Enhance learning by integrating research findings and practice wisdom.Define a shared language and shared vision for community change. (Ordonez; 2014)

23. Why is ToC important for impact measurement? Facilitates analysis of assumptions about context and effect of the intervention Helps show 'reality’ of the causal pathwaysExplores propositions about 'if A, then B’About the logic / why of pathwaysDoes the intervention make sense?

24. Hypothesesabout change as it occurs in different domainsDomain A: changing people's behaviours?COM-B - Capability, Opportunity, Motivation & BehaviourDomain B: changing agency practices?Advocacy in media about change as it occurs in different domains

25. What they look like?'left to right' v 'top to bottom’ (too static?)'box & arrow’ (too messy?) mechanistic / linear v holistic / complexA ‘boxy’ example…EXERCISE: What additional elements would align this to ‘good practice’ (as above, what aspect of context is missing)?

26.

27. Basics of how to ‘do’ ToC? Forward mapping: inputs  activities outputs  outcomes  impact; Backward mapping from desired impact to inputs is often better as it challenges us; If we have the [specified] impact, then we need...To have delivered these outcomes, we need...To have produced these outputs, we need...To have carried out activities, we need...To have obtained these inputs for resources.

28. ToC EXERCISEUsing the suggested ‘backward mapping’ & starting from the impact the credit union intends to have, outline / sketch the elements and how they are linked through the causal process… Impact, what is it, who benefits from it, at what level does it occur (community v individual) what is the ‘problem’ to which it is a ‘solution’?;

29. ToC Exercise ContinuedWhat are the outcomes from the work that contribute to the impact – esp. moving to ‘low value / high income’ loans? How do the outputs, (savings, loans, others) feed into these outcomes? In what ways do the credit union’s activities produce the outputs? What inputs does the credit union require as resources fuel the process?

30. ToC QUESTIONSAre the links from inputs to impacts clear? Would the way you present the links help convince a sceptical person? Does the ToC 'add up' as a way of addressing the issues you want the credit union to address? Are there assumptions in the theory that may not be true? What are these? Should the ToC change so that it wont matter? How would you summarise the basic hypothesis(es) about credit / wellbeing that underpins the ToC? (Try to formulate this as a testable idea).

31. 1.30 - Session D - Refining the ToC, Recap...Inputs resource activities that produce outputs that generate results that deliver outcomes that create impact. If you were sceptical what would allay any concerns you might have about the claims made about the process? EXERCISE Taking each of the questions in turn, write brief notes on how you would convince a sceptical observer.

32. How would we answer the sceptic? Is the process in the ToC consistent with different explanations about how and why the actions influence the effects? Is the impact identified plausibly linked to other explanations (apart from that of the credit union's intervention)? Are the claims made in the ToC open to communicate in plain English what is the theory about the links in the chain of causality, e.g., If we do x action, then y change will follow because...? Are there particular / key / critical points in time (between, e.g., loan made' and 'enhanced financial resilience'), which we would like to know about in order to trace the change in resilience back to the borrowing profile?

33. 2.10 - Session E - Review of ToCDoes the theory make sense as a response to the financial circumstances of people's lives in the area (the context)? Would it make sense to a) a senior officer in local government and b) a member of the credit union?

34. Summary Review QuestionsIs the theory plausible as a statement of how local people live / stakeholders operate? Is if feasible for the credit union to implement it? Is it sensible to think that we could test the theory?

35. Review Questions cont’dDoes the impact that the theory identifies offer a reasonable prospect of addressing the 'too few / too many' issue you identified in the stakeholder mapping exercise? Does the diagram make clear the causal process? Are the assumptions made explicit? What is the quality of the evidence for the assumptions?EXERCISE In the light of your answers to these questions, how would you wish to have the theory further refined?

36. Does the diagram make clear the causal process?Is there enough detail that links the 'if... then' logic? Are there any missing links? Is it clear how the different elements relate to each other (activities >>> results >>> outcomes)? Does it communicate enough of the specific character of the credit union?

37. Are the assumptions made explicit...about the causal processabout the actions of the credit union (to activate this process)about the enablers and barriers that arise from the context?

38. What is the quality of the evidence?Is there evidence noted for the major hypotheses (about behaviour change?) that underpin the theory? Is there some assessment of the strength of this evidence? Is the assessment plausible?

39. EXERCISE In the light of your answers to these questions, how would you wish to have the theory further refined?

40. 2.50 - ConclusionParticipants to say if, and extent to which, workshop fulfilled their expectations and comment on the nature of any gaps in these expectations left unfilled.Expectations revisited – did it deliver? Evaluation – what worked, what didn’t’?

41. Next StepsSmall Change to write-up report on workshop for the credit union; Small Change to prepare for Workshop 2 on implementation in light of this; Ends [3.00]

42. Social Impact Measurement for Credit UnionsWorkshop 2 on Data Unify 31st July 2018 Hoot 1st August 201842

43. Agenda11.00 - Session A - Overview of workshop 1 & progress 11.30 - Session B - Practical measurement options12.00 - Session C – Quantification & wellbeing12.30 - Session D – Monetisation & partners’ savings 1.00 - Lunch1.20 - Session E - Qualitative commentary on impact1.50 - Session F - Questions for discussion on 3 options 2.20 - Session G - Data system given preferred option 2.50 - Conclusion, evaluation & next steps43

44. 11.00 - Session A - Overview of workshop 1 & progress to dateUnify logic model matrix form of ToCHoot systems mapping Levels / domains form of ToC44

45. Questions for discussion on progress If the CU reports on these impacts to local government / housing & local employers will its credibility rise? What factors will enhance the credibility of the reportage? What factors could detract from its credibility? What else would need to be done? For these stakeholders, what are the hallmarks of a quality report? 45

46. 11.30 - Session B - Outlining the practical measurement options that demonstrates impactOption 1) in-house requiring staff time and quality assurance of independence / validity / credibility; Option 2) out-source to external consultant or University through securing grant-aid; Discuss advantages & disadvantages of the two options; 46

47. 12.00 - Session C - Quantitative measurement of contribution to members' wellbeing (HACT)What is HACT’s wellbeing valuation approach? The value of a change in satisfaction with life that correlates with (predicts) an outcome (e.g. ‘reduced stress over unmanageable debt,’ ‘feeling more in control of own finances’) expressed as the {average} amount of additional money a person would require for s/he to report an equivalent change in satisfaction; 47

48. How does HACT value wellbeing? Use quality survey data e.g. Understanding Society Identify outcome variables, e.g. health status, fin inclusionConstruct statistical model that predicts outcomes‘Controls’ for other factors e.g. working, education Model amount of additional money that predicts this Make allowance for ‘what would have happened anyway’Do surveys of CU’s members before loan & after loan Use comparable questions, note diffs ‘before & after’ Multiply number changed by HACT’s £ estimates 48

49. HACT’s Financial Inclusion Outcome Values OutcomeValue▲ Debt-free£2,548.06▲ Afford to keep house well-decorated£6,272.09▲ Able to save regularly£1,293.02▲ Relief from heavily burdened with debt£13,376.7▲ Able to pay for housing £8,974.36▲ Financial comfort£8,878.57▲ Access to internet£1,663.12▲ Able to insure home contents£5,843.7649

50. HACT’s Health Outcome ValuesOutcomeValue☼ High confidence (adult)£12,549.31☼ Relief from dep’n/anxiety (adult)£39,301.75☼ Good overall health£20,186.34☼ Relief from drug/alcohol problems£15,674.32☼ Smoking cessation£3,948.13☼ Feel in control of life£15,733.72☼ Can rely on family£7,036.0750

51. Questions for discussion on HACT approach Does this seem like a reasonable approach to social impact measurement? One criticism is that it represents a narrow, limited & mercenary view of impacts. What is your view? One claim is that it would allow for the comparison of the impact of two different organisations because it uses the same monetary metric for both. Is this realistic? Is it desirable? What needs to happen for the CU to report its impact using HACT’s approach? Is this feasible, is it desirable? 51

52. 12.30 - Session D - Quantitative measurement of contribution to partner agencies’ cost savings Way to address measurement challenge; Approach ‘makes sense’ to audiences; Lower costs & more efficient for CU; Which savings should it evidence? (audience); Be upfront about how representative case is; Back up user comments with other credible data; 52

53. Session D – Costed case studies, overview of components Calculate cost of CU’s work on loan; Research alternative credit sources’ costs; Locate comparable data on statutory costs (end slide]; Consider attribution v contribution balance issue; Consider ‘would have happened anyway’ (deadweight) Write-up BUT costs of collating data given range of loan uses? 53

54. 1.20 - Session E - Qualitative commentary on impact; case studies of impactConvincing & credible narratives of impactA realist perspective on the narrative; May work better for non-typical benefits; Low cost option; Could incorporate multiple sources, user & agency;Links well to ToC thinking;54

55. Session E – Qualitative… contdContext + Mechanism = Outcome Configuration; ‘what work for whom in which circumstances’;Context – circumstances, needs & resources; Mechanism – what the loan pays for, why imp; Configuration – how the C+M=O ‘fits’ together; Write-up – one para on each; Use infographic style (end-slide); 55

56. 56

57. CorePartner specificFinancial resilience57

58. 1.50 - Session F - Questions for discussion on the 3 options Option 1 = HACT; Option 2 = Costed savings case studies; Option 3 = impact narratives; Which of the 3 offers best prospect to communicate the impact? Which the least? What would a balanced approach look like?In making your choices, consider the option’s practicality / time requirement and its credibility; 58

59. 2.20 – Session G – Qs on data system given preferred optionsWhat use is to be made of Curtains / other existing data? What additional data will be required? What are the sources of the additional data? What instruments should we use to collect this data? Who leads on data, collection, analysis & write-up?Quality assurance / auditing arrangement? How frequently should CU do the impact reports? When would you like to complete the first report; Exercise: List sources & instruments of data you will use; 59

60. 2.50 - ConclusionParticipants to say if, and extent to which, workshop fulfilled their expectations and comment on the nature of any gaps in these expectations left unfilled.Expectations revisited – did it deliver? Evaluation – what worked, what didn’t’? [Ends: 3.00] 60

61. Some Useful Resources HACT’s Social Value Bank http://socialvaluebank.org/ Data on statutory service costs https://www.pssru.ac.uk/ Case studies https://en.wikipedia.org/wiki/Case_study Local area statistics https://www.nomisweb.co.uk/ Infographic templates, etc https://www.canva.com/templates/ NB Importance of compliance with HACT’s licensing 61

62. Social Impact Measurement for Credit UnionsWorkshop 3 on Data Systems Hoot 21st August 2018Unify 22nd August 201862

63. Agenda for Data Systems Workshop11.00 - Session A – Overview of workshop 2 on data 11.30 - Session B – What should the (data) system do? 12.00 - Session C – Internal / administrative data 12.30 - Session D – Additional membership data 1.00 – Lunch – discussion on Friday Skype call 1.20 - Session E – Public domain data 1.50 - Session F – Integrating the three data sources 2.20 - Session G – Outputs required from the system 2.50 - Conclusion, evaluation & next steps63

64. 11.00 - Session A – Overview of workshop 2 on data & progress to dateUnify See Sheet 1Hoot See Sheet 2What learning questions would you want to answer with reports based on these indicators? 64

65. 11.30 - Session B – What should the system do? What are your requirements of the data system? Who will have access to it? Are there ethical issues? 65

66. 12.00 - Session C – Internal / administrative data What are the processes through which the administrative data will generate the required impact indicators? What are the database issues?How will the data be processed? Where will it be stored securely? 66

67. 12.30 - Session D – Additional membership data What are the processes through which the members’ data will generate the required impact indicators? What are the questionnaire items? What is the survey process?What is the sampling process?67

68. 1.00 – Lunch – discussion on Friday Skype call 68

69. 1.20 - Session E – Public domain data What are the processes through which the public domain data will generate the required impact indicators? How will this material be sourced? Who will review it for relevance, quality, comparability, etc?69

70. 1.50 - Session F – Integrating the three data sources What software is likely to facilitate the analysis of the data? Will Excel allow for the data to be integrated? What would be in the rows and columns? What is the ‘unit of analysis’? 70

71. 2.20 - Session G – Outputs required from the system What outputs do you want the software data system to produce? Charts? Text material? 71

72. 2.50 - Conclusion, evaluation & next stepsParticipants to say if, and extent to which, workshop fulfilled their expectations and comment on the nature of any gaps in these expectations left unfilled.Evaluation – what worked, what didn’t’? [Ends: 3.00] 72