Randomized Control Trials: What, Why, How, When, and Where - PowerPoint Presentation

Slide1

Randomized Control Trials: What, Why, How, When, and WhereMark L. DavisonMESI ConferenceMarch 11, 2016

Department of Educational PsychologySlide2

What Is A Randomized Control Trial (RCT)?A randomized control trial is an experiment in which the people being studied are randomly assigned to one of several treatment conditions.People can be students, teachers, administrators, etc.

Can also randomly assign schools, districts

Treatment conditions, e.g.

No treatment vs. treatment New treatment vs. business-as- usual

Department of Educational PsychologySlide3

Why Do an RCT?RCT designs support causal inferences; that is if there is a difference in outcomes for treatment and control, RCT supports conclusion that treatment caused the differenceAs an educator, you need to know what actions you can take that will CAUSE an improvement in outcomesWe need to understand the causal processes affecting outcomes if we are to do education well

Not the only design that can support causal inferences

Slide4

How to Do An RCT?Logic Model (or Theory of Change)Sampling PlanMeasurementsIndependent Variables

Dependent Variables

Covariates

MediatorsModeratersTreatments, fidelity measures, and fidelity observationsSlide5

Logic ModelTheory of how intervention has its effectsInclude the various variables that you will include in study as measured variables.Text to accompany figure that explains how the variables are related to the change processSlide6

How to Do an RCT: Planning Process?Logic Model which includesIntermediates steps between the intervention and the outcome (mediators)Factors that may weaken or strengthen the effect of the intervention (moderators)Alternative variables that may explain why the treatment group may have better outcomes than the control group (covariates) Slide7

RCT VariablesIndependent variable = treatment conditionDependent variable = outcome variableCovariates = variables to be statistically controlled in the analysisMust be measured in data collection

E.g. prior achievement in studies of achievement

Mediator variables = processes through which independent variables has its effects

Moderator variables = variables that influence the size of the treatment effect

Department of Educational PsychologySlide8

Logic ModelDepartment of Educational PsychologySlide9

Random Sampling and the Independent VariableRandom Assignment options (influence analysis strategy)Randomly assign studentWhen people receive the treatment independentlyStudents in same classroom can receive different treatments

Randomly assign by teacher or classroom

When treatment is administered by classroom (all students in same classroom receive same treatment)

When treatment is by teacher (all students with same teacher receive same

treatment)

Studies of classroom or teacher, policy, or interventions

Randomly assign by school if treatment administered at school level (all students in same school receive same

treatment)

Studies of school policy

Avoids cross-over effects

Generally not feasible for district research (not enough schools)Slide10

Stratified Random SamplingSimple random sampling controls confounding variables only within limits of random samplingStratified random sampling identifies variables to be controlled, called strata, and samples in way that controls Sex as stratifying variable in readingHalf males are randomly assigned to treatment and control; half females are randomly assigned to treatment and controlSlide11

Stratified Random SamplingSex and ESL as stratifying variablesFour groups: boys/ESL, boys/non-ESL, girls/ESL, girls/non-ESLHalf of each group is randomly assigned to treatment and controlsStratified random sampling actually controls stratifying variables, not just within limits of samplingSlide12

Sample SizeSelect sample size so power of detecting effect is large (e.g. > .8)Power is the probability of detecting an improvement resulting from the treatment if there is oneConsult applied statistician and websiteshttp://sitemaker.umich.edu/group-based/optimal_design_softwareSlide13

Dependent VariableIf students are not the target of the intervention, do you want a student outcome variable? For example, Teacher intervention designed to improve reading achievementTeacher measures e.g. pedagogical knowledge, fidelity of implementation in classroomStudent outcome

Reading achievement measure, number of books read independently Slide14

CovariatesAlternative explanations for an effect (if it occurs) and that were not controlled by stratificationUsed to determine if randomization well-controlled the variableUsed to statistically control for alternative explanationTest hypothesis of whether there is still a treatment effect controlling for differences in gender, prior achievement, etc.

In studies of achievement, use prior achievement as covariate; increases power and controls for alternative explanationSlide15

MediatorsVariables that represent intermediate steps in the processes leading from the intervention to the outcomeSchool processes, teacher processes, student

processes

Mediator variables are used to assess whether the intervention led to the intermediate processes and whether intermediate process are related to outcomeSlide16

ModeratorsUsed to test hypotheses about whether the intervention is equally effective for all E.g. wonder whether intervention will be equally effective for boys and girls? Include sex as a

moderator?

Used to test hypotheses about whether intervention will help close gaps?

E. g. Wonder if intervention will close gaps between blacks and whites? Include race as a moderatorSlide17

TreatmentsDescribe each treatmentTreatment fidelityDevelop or select a measure of how well treatment is administered in treatment and control conditionsQuantitative treatment fidelity index can be used as a

mediator and to see if outcome increases as function of how well it is implemented

Observe classrooms (treatment and control) to see what goes on in each treatment and how different they really are

Qualitative observations of treatment and controlRCTs tend to be mixed quantitative and qualitative studiesSlide18

When RCT?RCT canNOT be used ifIndependent variable is not malleable (it is not under the control of educators)E.g. sex of students, race of students

Manipulation of independent variable by researchers would be ethically unacceptable or practically not feasible

E.g. randomly assigning students to a suspension and no suspension condition

E.g. randomly assigning people to high income and low income conditionsSlide19

When RCT? Alternatives to RCTRegression discontinuity designs Assignment to treatment is based on a strict cut-off on a continuous variableSingle-subject designs, such as

ABA

designs in which intervention is

turned off and on at designated points to see if outcome changes as the intervention is turned off and on

Recognized by special education more than regular education (NCSER, but not NCER)

Must distinguish between single subject designs that show effects in one-on-one settings vs those that show effectiveness in class settings

Used when treatments have strong, immediate effects

that are reversible

Slide20

When RCT? As Research CapstoneMy opinion. RCT’s are best usedAfter “bugs” have been worked out of the interventionhttp://www.whale.to/vaccine/bayly.html#THE%20SALK%20VACCINE%20DISASTERAfter research methods have been piloted in less rigorous quasi-experimental studies

After pilot studies (quasi-experimental studies or small RCTs) have provided some evidence for the intervention

In short, after a lengthy program of intervention development and pilot studiesSlide21

SummaryWhat?RCT is an experiment in which people are randomly assigned to treatment groupsWhy?RCT permits causal inference, and educators need to know what actions they can take to cause an improvement in student outcomes?Slide22

SummaryHowBased on plan including logic model, sampling plan, variables (independent, dependent, mediators, moderators, covariates), treatment description, and fidelity measures and observationsWhen and whereIf feasible and ethically acceptable

If independent variable is malleable educational factor

If regression discontinuity or single subject design is less appropriate

If research design and treatment implementation have been well worked outSlide23

ReferencesBooth, J. L., Oyer, M. H., Pare-Blagoev, E. J., Elliott, A. J., Barbieri

, C., Augustine, A., &

Koedinger

, K. R. (2015). Learning algebra by example in real-world classrooms. Journal of Research on Educational Effectiveness, 8(4), 530 – 551.

Borman

, G. D. & Dowling, N. M. (2006). Longitudinal achievement: Effects of multi-year summer school: Evidence from the Baltimore randomized field trial.

Educational

Evaluation

and Policy Analysis

,

28

, 25 – 48

.Slide24

ReferencesFehr, C. N., Davison, M. L., Graves, M. G., Sales, G. C. & Seipel, B. E. (2012). The effect of individualized, online vocabulary instruction on picture vocabulary scores: An efficacy study. Computer Assisted Language Learning, 25(1), 87-102.

Food

and Nutrition Service, U. S. Department of Agriculture (February, 2004).

Evaluation of the school breakfast program pilot project. Washington, D. C.: Author. Slide25

What Works Clearinghouseies.ed.gov/ncee/wwc Contains evaluations of research evidence on various interventionsEvidence for a causal effect of the interventionSlide26

Group Work: Research PlanStatement of research questionLogic ModelResearch PlanSampling PlanIndependent Variable

Dependent Variable(s)

Covariates

Mediator variablesModerator variablesTreatment conditionsFidelity measure