Monitoring and Evaluation - PowerPoint Presentation

Slide1

Monitoring and Evaluation of Development Programs and Projects

Class

7

–

Randomization*

*(how to design an RCT that accommodates real world constraints)Slide2

Randomly

sample

from area of interest

Clarification:

Random

sampling

vs.random

assignmentSlide3

Randomly

sample

from area of interest

Randomly

assign

to

treatment

and

control

Random sampling and random assignment

Randomly

sample

From both

treatment and controlSlide4

BEFORE YOU EVEN TALK ABOUT RANDOMIZATION DESIGN…Explain why randomization is necessaryTalk about attribution

A

rticulate

why

(non-random) comparison group may be different and specify way this may bias resultsExplain that the intervention may have unintended (good or bad) consequences worth measuring

Can measure spilloverCost-benefit analysesLecture OutlineSlide5

Lecture OutlineUnit of RandomizationWays to RandomizeSimple Randomization (eg

: lottery)

Randomization “in the bubble”

Phase-inRotationEncouragement Design

Multiple treatments, 2 stage(if time) StratificationSlide6

Unit of Randomization?

Options

Individual

ClusterWhich level to randomize?ConsiderationsWhat unit does the program target for treatment?

What is the unit of analysis?Slide7

Unit of Randomization: Individual?Slide8

Unit of Randomization: Individual?Slide9

Unit of Randomization: Clusters?

“Groups of individuals”: Cluster Randomized TrialSlide10

Unit of Randomization: Class?Slide11

Unit of Randomization: Class?Slide12

Unit of Randomization: School?Slide13

Unit of Randomization: School?Slide14

School Tutoring programIn the example discussed last week, students were tutored after school  how did we randomize last week?

What if students are taken out of class to be tutored, say by a teaching assistant?

 how would you randomize?Slide15

Randomizing at the child-level within classesRandomizing at the class-level within schoolsRandomizing at the community-level

Think about:

logistics, fairness

, politicsSlide16

Nature of the TreatmentHow is the intervention administered?What is the catchment area of each “unit of intervention”How wide is the potential impact?

Aggregation level of available data

Do you want to measure spillover?

Power requirementsGenerally, best to randomize at the level at which the treatment is

administered.How to Choose the Level / UnitSlide17

Sometimes a program is only large enough to serve a handful of communitiesPrimarily an issue of statistical powerWill be addressed

next week

Think about: sample sizeSlide18

Lecture OutlineUnit of RandomizationWays to RandomizeSimple Randomization (

eg

: lottery)

Randomization “in the bubble”Phase-inRotation

Encouragement DesignMultiple treatments, 2 stage(if time) StratificationSlide19

Simple Randomization of bank branches:

Interest Rate Elasticity

“high rate”

“

low rate”

Final

sample included 132 branch offices in 80 geographic

clustersSlide20

Simple Randomization: LotterySchool Voucher Program in Colombia (PACES)Student

must be entering 6th grade and under 15 years old

Students must provide evidence that they live in poor

neighborhoodRenewable

through graduation unless student is retained in a gradeVouchers awarded by lottery if demand exceeds supplyCovered about 60% of feesKey Findings (after 3 years) on Voucher Recipients:

Increased Usage of Private SchoolsHigher Educational AttainmentNo Difference in Drop-out RatesLess Grade RepetitionHigher Test Scores

Less Incidence of Teen-age EmploymentFurther research on peer effectsSlide21

Lotteries are simple, common and transparentRandomly chosen from applicant poolParticipants know the “winners” and “losers”Simple lottery is useful when there is no a priori reason to discriminate

Perceived as fair

Transparent

LotteriesSlide22

What if you have 500 applicants for 500 slots?Could increase outreach activities(but think of external validity)Sometimes screening mattersSuppose there are 2000 applicants for 500 slots

Screening of applications produces 500 “worthy”

candidates

A simple lottery will not

work - What are our options?Lottery (continued)Slide23

What are they screening for?Which elements are essential?

Selection procedures may exist only to reduce eligible candidates in order to meet a capacity constraint

If certain filtering mechanisms appear “arbitrary” (although not random), randomization can serve the purpose of filtering

and

help us evaluateConsider the screening rulesSlide24

Randomization “at the margin”Organizations may not be willing to randomize among eligible people.But

might be willing to randomize

those at the margin –

ie, those who are borderline in terms of eligibility

Just above the threshold  not eligible, but almostWhat treatment effect do we measure? What does it mean for external validity?(hint: review RDD from last week)Slide25

Randomization at the margin: Impact of consumer creditDean Karlan and Jonathan

Zinman

worked with a bank in South Africa

Loan officers ranked applicants as “egregiously uncreditworthy” or “marginally

uncreditworthy”Randomly selected marginal applicants to be reconsidered53% of those were offered a loanLook at impact of credit on those randomized into the “reconsider group” Not just those offered loan… more on this next week. Found that access to credit increased likelihood that clients would retain job, increase income, feel less food insecurity

Marginal loans are profitable, but less than regular loansSlide26

Take advantage of operational constraintsTypical during expansion phaseEveryone gets program eventuallyFigure out what determines order of expansion

Examples:

Progresa

(Mexico), Deworming

(Kenya)Phase-inSlide27

Phase-in design

Round 1

Treatment

: 1/3

Control

:

2/3

Round 2

Treatment

: 2/3

Control

:

1/3

Round 3

Treatment

: 3/3

Control

:

0

1

1

1

1

1

1

1

1

1

1

1

1

1

1

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

Round 1

Treatment

: 1/3

Control

:

2/3

Round 2

Treatment

: 2/3

Control

:

1/3

Randomized evaluation ends

Randomized evaluation endsSlide28

AdvantagesEveryone gets something eventually

Provides incentives to maintain contact

Concerns

Can complicate estimating long-run effects

Care required with phase-in windows

Do expectations of change actions today?Phase-in designsSlide29

Round 1

Treatment

: 1/2

Control

: 1/2

Rotation design

Round 2

Treatment

from Round 1



Control

——————————————————————————

Control

from Round 1



Treatment

Round 1

Treatment

: 1/2

Control

:

1/2Slide30

Groups get treatment in turnsGroup A gets treatment in first periodGroup B gets treatment in second period

Advantages

Perceived as fairer; easier to get accepted

Concerns

If people in Group B anticipate they’ll receive the treatment the next period, they can have a different behavior in the first periodImpossible to measure long-term impact since no control group after first period

RotationSlide31

Extra Teacher program: Rotation with schools

Group

Year

1

Year 2

Year 3

AGrade 3

Grade 4Grade 3

B

Grade 4Grade 3

Grade 4

Schools in Varodara, India were divided into two groups: ( A and B)and teaching assistant offered in schools according to the following schedule:Slide32

Sometimes it’s practically or ethically impossible to randomize program accessBut most programs have less than 100% take-upRandomize encouragement to receive treatment

Encouragement design: What to do

when you can’t randomize accessSlide33

Something that makes some folks more likely to use program than othersNot itself a “treatment”

For whom are we estimating the treatment effect?

Think about who responds to

encouragement

Do not choose an encouragement that affects those who are different than the entire populationWhat is “encouragement”?Slide34

Methods of randomization - recapSlide35

Lecture OutlineUnit of RandomizationWays to Randomize

Simple Randomization (

eg

: lottery)

Randomization “in the bubble”Phase-inRotationEncouragement DesignMultiple treatments, 2 stage

(if time) StratificationSlide36

Treatment 1

Treatment 2

Treatment 3

Multiple treatmentsSlide37

Manage or Measure SpilloverSpillover: when the control group, although “untreated”, is affected (positively or negatively) by the treatmentChoose unit that contains

spillover (

ie

randomize at school or village rather than individual level)Measure Spillover: TUP (

Bangladesh, Honduras, Peru, Pakistan, Ghana, Ethiopia, Yemen)Slide38

2 Stage Example: Targeting the Ultra PoorProgram which consists of targeting poorest families in a village and providing consumption support ($$ or food), asset transfer, livelihood training

Eventual graduation to

microfinanceSlide39

TUP ASSET TRANSFERSSlide40

TUP Evaluation Design I

Communities

(Total

= 80)

40 treatment

40 control

C 20 cont

Households(Total =

1600)

A 20 treat

Ho do we determine impact?

Program direct impact: A-C Slide41

TUP: INDIRECT EFFECTS

Villagers may benefit from neighbors being treated

Or they may be negatively affected

Why would it be desirable (from a policy perspective) to measure spillover?Slide42

TUP Evaluation Design II

Communities (Total

= 80)

40 treatment

40 control

C

20 cont

Households(Total = 2400)

A 20 treat

B 20

cont

Ho do we determine impact?

Program direct impact: A-C

Program indirect impact: B-C Slide43

StratificationObjective: when you have a small

sample, make sure key variables are balanced between Treatment and Control

What is it:

dividing the sample into different subgroups

selecting treatment and control from each subgroupStratify on variables that could have important impact on outcome variable (bit of a guess

)Stratify on subgroups that you are particularly interested in (where may think impact of program may be different)Can

get complex to stratify on too many variablesMakes the draw less transparent the more you stratifySlide44

Next WeekSampling!