Risk Management Indicators – Verifying Change - PowerPoint Presentation

1. Risk Management Indicators – Verifying Change NORHED seminar in Addis Ababa 13 – 14 March 2014 Lill-Ann Bjaarstad Medina

2. Today’s mission Where are we? Wen have formulated some generic Outcomes of strengthening capacityWe have identified some essential Outputs to reach these Outcomes We need indicators to monitor progress in delivering these essential Outputs and to whether they have the intended effect.2

3. Risk ManagementRisk management as a method to address the risks of not reaching planned or expected results. A project risk management plan should include management risksDoes not include risks for own staff3

4. What are risks? What is risks management? Risk“Factors or events that may occur and influence the results achievements in a negative way». Risks surge from uncertainties that have negative impacts Risk Management A step-wise method to be able to handle risksIdentificationAnalysis MitigationTolerance 4

5. EActivities1.1.1.Output1.1.2Output1.1 Outcome1.2 Outcome1.0 Outcome ( Strengthened Institutions) 1.1.3OutputResources Impact1.1.4OutputRisks

6. Types of risks «Internal” vs. “external” risk factors Interne risiko Eksterne risiko6

7. Main Principles in Risks ManagementRisks must include both aspects; uncertainties and negative impact on achievement of results Risk management divides into “phases” (identification, assessment, mitigation and tolerance) of which all are necessary steps to take The assessment of probability and consequences helps identify the most significant risks to be prioritized when planning mitigating measures and how much risk you may tolerate7

8. Risk AssessmentProbability and Consequence 8

9. Mitigation and Tolerance Implementation of preventive measures to reduce likelihood (ref. probability) Preventive measures to reduce the degree of negative impact (ref. consequence) Early intervention Contingencies plans Mitigation measures may reduce risks but not remove it. 9Mitigation measures reduces, but not necessarily removes the risk

10. Some common flaws in risk managementRisk factors are identified, but the consequences for the intervention are not identified nor assessedExternal risks are mistaken as not being possible to mitigate “Jumping to conclusions”; Risk factors are identified, and mitigation measures are discussed, without a sound assessment of probability and consequenceRisk assessments needs to be updated .Problems can be removed and should be addressed. 10

11. 11ImpactOutcomesOutputsIndicators We need indicators at each results level

12. What is an indicator?A variable to verify and/or measure change or processes towards predefined ends. Measurement: continuous vs. discrete measures (not present/present) Indication: direct vs indirectVerification: evidence-based vs. perception-based12

13. What is a good indicator? To be able to measure or verify the achievement of results, indicators should;Reflect the entire results statement in an adequate manner Be relevant – and give the needed (strategic) informationBe at the right (corresponding) levelBe realistic (“cost-efficient” and feasible) 13

14. Ez1.1.1.Output1.1.2Output1.1 Outcome1.2 Outcome1.3 OutcomeImpact1.0 Outcome ( Strengthened Institutions) 1.3.1 Output1.2.1Output1.2.2Output1.1.3Output

15. Types of indicators Standard Indicators (core indicators) Key Indicators Indicators to demonstrate progress (sequenced outputs) Composite indicators (cluster) 15There is no standard terminology

16. Baseline – the current situation Baseline data is narrower then a Problem AnalysesBaseline is established when the Outcomes has been agreed upon. Baseline does not address causesBaseline data should be relevant to the Results Statements at each level Baselines may help reconfirm or adjust indicators and targets16

17. Baseline values Baseline values need to be ascribed to each indicator 17If you can’t find your baseline value you probabely have the wrong indicator

18. TargetsTargets on Output level; Set relative to resource inputTargets on Outcome level; When is it feasible? Targets as symbols of a political consensus or visions18Illustration: Unicef Global monitoring vs reporting on results of interventions

19. The most common errors Starting by selecting indicators is starting in the wrong end Results Statements define indicators and not the other way around. Indicators should not be phrased as objectives Indicators are measurements. Sometimes what they measure have to be supplemented with a narrative description. Indicators are seen as “neutral” measures Indicators should reflect strategic choices, may have management implications and aim at learning. They may be and often are political 19

20. Measuring capacity by its effectsIn NORHED the effects of capacity is identified as, more and better research produced more and better qualified graduates20“At the Programme level it is evident that there is a need for more comprehensive indicators for project outputs, achievements and outcomes beyond an annual summery of graduates produced and research papers published” Norad, Evaluation report 7/2009

21. Illustration: Indicators to measure quantity and quality of researchHEEACT – Higher Education Evaluation and Accreditation Council of Taiwan Number of article in the past 11 yearsNumber of articles last yearNumber of citations for past 11 yearsNumber of highly cited papersNumber of articles in highly cited journals ARWU – Academic Ranking of World Universities (Shanghai Jiao) Papers published in Nature and SciencePapers indexed in Science in Sconce Citation Index-expanded and Social Science Citation IndexStaff/alumni winning Nobel Prices and Field Medals 21Does it say something about the quality of the institution?

22. Illustration: Indicators to measure institutions ARWU – Academic Ranking of World Universities (Shanghai Jioa) Research Outputs Papers indexed in Science in Sconce Citation Index-expanded and Social Science Citation IndexPer capita academic performance THE - Higher Times Education Teaching and learning environment Results on reputational survey on teachingStaff to student ratioRatio PhD to BA degrees awarded by institutions measured against number of academic staff Research volume, income and reputation. Results of reputational surveyResearch income scaled against staffNumber of articles in academic journals (Thompson Reuter) per staff member22

23. EExpanded and better qualified, gender balanced workforce Increased, better and relevant knowledge baseIncreased evidence base for policies and public management1.1.1.Output1.1.2Output1.1 Outcome1.2 Outcome1.3 Outcome1.0 Outcome ( Strengthened Institutions) 1.2.1Output1.2.2Output1.3.1 Output1.1.3OutputNORHED IMPACT

24. Contribution to impactAre we missing some indicatorsExpanded and better qualified, gender balanced workforce Increased, better and relevant knowledge baseIncreased evidence base for policies and public managementTwo remaining questions; How to measure contribution to better qualified work force?How do we demonstrate the relevance of knowledge?How do we verify that there is and increased evidence base for polices?Can we demonstrate the influence of our research findings? 24

25. How to measure high quality teaching?Suggestions for composite indicators Curriculum includes/updated with knowledge that is relevant for the local setting Curriculum in adapted to the global knowledge baseShare of lecturers with PhDShare of lecturers that with research experience that is relevant for the MA programme 25

26. How to demonstrate the relevance knowledge from the six sub-programmesand how it influence society? 26Suggestion: Reporting in narrativeCan we organize the reporting on the relevance of research findings according to thematic categories or research questions? How to report on influence? Should we put different requirements for reports on different types of research?Basic AppliedTechnology and InnovationR&D for management of public and natural resourcesAction research

27. Do we need a communication strategy?Good stories - Anecdotal evidence of relevance and influenceA more systematic approach? 27