Randomised controlled trials - PowerPoint Presentation

Slide1

Randomised controlled trials

Peter JohnSlide2

Causation in policy evaluationSlide3

Do

not

do this (before and after study)Slide4

A Randomised Controlled Trial

A RCT is an experiment where two or more groups of subjects who are compared with a control group that does not get the treatment

Randomisation between treated and control subjects ensures that there are no other outcome differences between the groups than from the intervention

Focus on comparing

average outcomes

in each group Slide5

Figure 7.1 Simple Random Allocation Design

Simple Random Allocation Design Slide6

Policy experiments

A distinct form of RCT which involves the policy-maker delivering an intervention. Goes back to the 1920s - e.g. school milk experiment (sabotaged by teachers)

Policy-maker

randomises

, researcher evaluates

Different to participation experiments, such as GOTV + various forms of experiment we in NNTT

Can be part of an official evaluation, but new model whereby done informally Slide7

The new nirvana?

Promoted by UK Behavioural Insights Team – had little money, but look at what it did with it: RCTs on tax letters, texting for court fines, charitable giving, DVLA reminders, organ donations with DVLA, job centres, work with firms

Massive expansion in development field

US interest - education federal funding, social policy, welfare to work

Use of experiments by economists (e.g.Duflo)

Now experiments in local governmentSlide8

Example: Essex County Council

Blue Badge Renewals: Channel Shift

Peter John and Toby

BlumeSlide9
Slide10

Essex County CouncilSlide11

Research design

We set out to test whether local authorities can

‘

nudge

’ residents to renew their Blue Badge permits online rather than by post – channel shift

Three types of message were tested for Blue Badge permit holders over a period of two months (December 2014-Jan 2015)

Two months to goSlide12

Treatments and sample

2798 renewals were sent out: 699 allocated to control, 698 to the simplified condition, 698 to the incentive group and 699 to the messenger group

Treatments:

Simplification

Messenger - the messenger treatment included a picture and testimonial from another Blue Badge holder of how they have renewed online and encouraged others to do the same.

Incentive - appealing to common goodSlide13

Incentive letterSlide14

Results

Simplification significantly increased online renewal rates compared against the control group by 7.3 percentage points.

Offering an intrinsic incentive – that does not offer a personal benefit but rather a collective incentive, so the individual is prompted to act for the benefit of others – also significantly increased renewal rates compared with the control group by 7.3 percentage points.

The use of a peer messenger – another Blue Badge holder – to encourage online renewal did not appear to have an eff

Slide15

ResultsSlide16

The seven things you need to plan to do a RCT

Establish a question that can be answered by a trial

Work out the units

Determine sample size

Specify the treatment arm or arms

5

. Set

out

randomisation

procedure

6

. Specify outcomes and their measurement

7

. Address

ethical

issues and data

protectionSlide17

Step 1: Establish a question that can be answered by a trial

Not all questions can

Need to be a good question – you do not know the answer

Is the theory plausible (a mechanism you expect)?

Theory of change

(ToC)

Can you intervene in a way that changes behaviour or outcomes (or attitudes) in a set time period?Slide18

Step 2: Work out the units

Can be individuals, households, streets, communities, larger areas, firms, other organisations, classes

Can be nested in units, so individuals in streets, or students in classesSlide19

More on units

Governed by a set of practical question about availability

Data records will drive this

Costs of measurement (sequeway to sample size)

Recruitment – there will be drop out > selection biasSlide20

Step 3: work out sample size

A topic of its own involving use of statistics

Basic message is that RCTs need large groups

Rough minimum of 400 per group

Avoid too small trials

Sample size calculators can help (be careful with the online ones)Slide21

More on size

To calculate the right sample size, you need to have an idea of what the expected effect is going to be, usually expressed as point differences between two groups, (e.g. .5 and .6 for a ten percentage point difference).

You need to specify in advance what probability level you are prepared to accept. Conventionally we have a two-sided test at .05  

You need to set the statistical power that is desirable, which is usually 80 per cent. Slide22

Step 4: Specify the treatment arm or arms

Need to ensure the treatment reflects what you want to measure,

Need for precision in delivery

Ensure the treatment is not doing too many things at once or that you break up different elements into treatment arms Slide23

Step 5: Set out

randomisation

procedure

A

llocate subjects using random numbers: >

http://www.random.org/

Or create a random number seed e.g. in Excel

Good idea to ask an outside person to

randomise

to ensure it is implemented effectively

You can test for balance across T and CSlide24

Step

6

: Specify outcomes and their measurement

Need something measurable that is standard across the units and T&C, e.g. achievement, volunteering

Can be interval (that is in equal units, such as money donated) or categorical (e.g. volunteer not not volunteer) or even ordered (e.g. who arrives first), can be measured over time, or before and after (desirable but not essential)Slide25

Step 7:

Address

ethical and data protection

issues

All research needs to be guided by ethical issues – consent, avoidance of harm, respect for participants (e.g. right to withdrawal, privacy), maintain safety

Trials often thought to be vulnerable because of randomisation may

‘

deny

’ benefits or impose costs (in practice these happen anyway)Also may involve deception that needs to be justified