impact evaluation for realtime decisionmaking Why are WE here today We are here to get better at what we do A simple idea research to inform a process of adaptive policy making How best do I get there ID: 661617
Download Presentation The PPT/PDF document "Arianna Legovini Development Impact Eval..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Arianna Legovini
Development Impact Evaluation (DIME)
impact evaluation for real-time decision-makingSlide2
Why are WE here today?Slide3
We are here to get better at what we doSlide4
A simple ideaSlide5
research to inform a process of adaptive policy making
How best do I get there?Slide6
Actionable answers
Iterative learningSlide7
Rwanda Rural Roads: understanding the impact of road rehabilitation on national agricultural markets
Set up high frequency national information and mapping on agricultural marketsMeasure effect of rural roads investments on:
agricultural prices across national markets,availability of consumer goods and migration patternsSlide8
Objective to–over time—improve policy choices and policy effectivenessSlide9
2 types of inputs into decisions
Data
EvidenceSlide10Slide11Slide12
Nairobi public transport IE: a system approach to road safety
Set up real-time data system to measure movement of vehicles and people across the city
Understand set of contractual arrangements between all actors in public transportsExperiment with use of information and incentives to improve outcomes (mobility, road safety, emissions, etc) Slide13
What is a sufficient basis for a decision?
Call for action“
Lack of” sufficient basis for deciding to do somethingno data, no system of accountability, perverse incentivesDiagnosisHigh quality data to understand problem and underlying population.
Use it to debunk priors and formulate policy hypothesis.
Test hypothesis and select policy modality
Causal evidence
to sort out complex supply & demand response to policy
Use it to select most effective policy alternative.Slide14
The importance of being earnest
Causal inference
Scientifically valid tests to obtain actionable answersSlide15
what is causal inference?Slide16
Invest in
the cause to get more of an effect
Policy objectiveSlide17
Evaluation objective
Identify
cause - effect
enable good policy decisionsSlide18
mmm…WHEN I USE MY UMBRELLA MY SHOES GET WET
I THINK I WILL STOP USING UMBRELLAS
It is easy to confuse correlation with causation
WHY IS UNDERSTANDING
CAUSALITY SO IMPORTANT?
TO AVOID MISTAKES THAT CAN CAUSE HARMSlide19
Monitoring:trends and correlations not causality
Monitoring tracks indicators over time
(usually among participants)It is descriptive before-after analysis It tells us whether things are moving in the right direction
It does not tell us
why
things happen (causality)Slide20
How do we sort our way through competing hypotheses?Slide21
Does the BRT or metro line increase incomes in the low-income areas we connect?
YES
NO
NOT SURESlide22
Invest in the metro
Outward migration of poorer population
Increase in land and real estate values
Metro line
INCREASE INCOME
Metro line announced
Developers move in
Increase access to jobsSlide23
Dar Es Salaam BRT IE: evaluate & experiment to learn
Evaluate BRT investments
Impact of mass transit on improving labor market linkages
23
Experiment with subsidies to understand cost of displacement and pricing strategies for access
Dynamics of gentrification
Strategies for inclusive urban growthSlide24
Ethiopia Highway and manufacturing growth
Evaluate the corridor
Unleasing new opportunities for growth in manufacturing Awasa biggest industrial park in Ethiopia
Understand how to increase welfare
Big labor market shock for young women in
poor rural areas near industrial parks
How do we protect them from exploitation?
24Slide25
Rio Supervia IE: social and economic aspects
Measuring the incidence of harassment against women
Estimating willingness to pay and economic costs of harassmentSlide26
Evaluation
Compares
WHAT HAPPENED
WHAT WOULD HAVE HAPPENED
TOSlide27
What is counterfactual analysis?
Compare same individual
with & without interventionat the same point in time Missing dataCompare statistically identical groups of individuals
with & without intervention
at the same point in time
Comparable dataSlide28
Counterfactual criteria
Treated & control groups
Have identical initial average characteristics (observed and unobserved)So that, the only difference is the treatment
Therefore the only reason for the difference in outcomes is due to the treatmentSlide29
Some important ideas
Data must be embedded in a rigorous analytical framework to clarify:
the process through which data is generated and how data can be interpreted. Strong team with research skills needed to:Impose analytical structure on design, implementation, measurement system and rigorous testing
Critical for adaptive process to take place as to ensure greater effectiveness
S
killed research team need the promise of analytical rewardsSlide30
2 things will be fundamental
The quality of the thinking and
The quality of the engagementSlide31
Quality of thinking
Going from the impact of a road to thinking about the combinations of policies and solutions to maximize improvements in economic outcomesSlide32
Quality of engagement
Each discovery leads to more refined and interesting questions
It is by working together and thinking together that we can aim at solving problems and improving outcomesSlide33
the idea
Capitalize on the World Bank/IFIs billion $ investments to pilot ideas and exert influence
Empower policy people to exert control on their local environment
Invest in impact evaluation research
generate data & evidence and motivate changeSlide34
DIME objectives
Develop institutional
capacity
for data & evidence-based policy
Generate useful
knowledge
that solve development problemsSlide35
We work through the policy cycle
35
Systematic use of evidence
Capacity building
IE ProductsSlide36
36
Global outreach
5 regions
60 countries
173 IEs
300 partnersSlide37
Across all sectorsSlide38
USD 12 billion in underlying WB lendingSlide39
What we learned doing this for the last 12 years
It takes time to change the way projects operateIt takes time to build the capacity, relationships, shared understanding
It requires changing the skill composition of the teamsEven when the skills are there, individuals may not have the bandwidthNeed team members dedicated to the taskIt’s a journey not a destinationSlide40
But we have also learned that
Together,
we can affect transformative change and have huge payoffsSlide41
Thank You!
facebook.com/
ieKnow
#
impacteval
blogs.worldbank.org/
impactevaluations
microdata.worldbank.org/
index.php
/catalog/
impact_evaluation
http://dime.worldbank.org
WEB