EMR 6550: Experimental and Quasi-Experimental Designs - PowerPoint Presentation

1. EMR 6550:Experimental and Quasi-Experimental DesignsDr. Chris L. S. CorynKristin A. HobsonFall 2013

2. AgendaBasic design elements and notationQuasi-experimental designs that either lack a control group or lack pretest observations on the outcomeMidterm examinationCase study

3. Questions to ConsiderWhat are the limitations of designs lacking either control groups and/or pretest observations?What simple strategies can be used to improve these types of designs?Why are such designs sometimes the only ones that can be used?

4. Basic Design Elements and Notation

5. AssignmentRandom assignmentCutoff-based assignmentOther nonrandom assignmentMatching and stratifyingMasking

6. MeasurementPosttest observationsSingle posttestsNonequivalent dependent variablesMultiple substantive posttestsPretest observationsSingle pretestRetrospective pretestProxy pretestRepeated pretests over timePretests on independent samplesModerator variable with predicted interactionMeasuring threats to validity

7. Comparison GroupsSingle nonequivalent groupsMultiple nonequivalent groupsCohortsInternal versus external controlsConstructed contrastsRegression extrapolation contrastsNormed contrastsSecondary data contrasts

8. TreatmentsSwitching replicationsReversed treatmentsRemoved treatmentsRepeated treatments

9. NotationX = treatmentO = observationR = random assignmentNR = nonrandom assignmentX = removed treatmentX+ = treatment expected to produce an effect in one directionX- = conceptually opposite treatment expected to reverse an effectC = cutting score- - - = non-randomly formed groups… = cohort

10. Logic of Quasi-Experimentation

11. RationaleQuasi-experiments are often a necessity given practical and logistical constraintsGreater emphasis on construct or external validity rather than cause-effect associations (least common)Funding, ethics, administration (somewhat common)The intervention has already occurred (most common)Sometimes they are the best alternative, even if causal inferences are weaker than is possible with other designsEven so, great care must be taken when planning such studies as numerous threats that cannot be controlled are often operating

12. Central PrinciplesIdentification and study of plausible threats to internal validityCareful scrutiny of plausible alternative explanations for treatment-outcome covariationPrimacy of control by designUse carefully planned and implemented design elements rather than statistical controls for anticipated confoundsCoherent pattern matchingComplex (a priori) causal hypotheses that reduce the plausibility of alternative explanations

13. Designs without Control Groups

14. One-Group Posttest Only DesignAbsence of pretest makes it difficult to know if change has occurredAbsence of a control group makes it difficult to know what would have happened without treatmentXO1

15. One-Group Pretest-Posttest DesignAdding a pretest provides weak information concerning what might have happened to participants had the treatment not occurredO1XO2

16. One-Group Pretest-Posttest Design with Double PretestAdding multiple pretests reduces the plausibility of maturation and regression effectsAdditional pretests can confirm maturational trendsO1O2XO3

17. One-Group Pretest-Posttest Design Using a Nonequivalent VariableMeasure A is expected to change because of treatment, B is notBoth A and B are expected to respond to the same validity threats in the same way{O1A , O1B} X{O2A , O2B}

18. Lottery ticket sales in convenience stores after introduction of signs in store windows reading “did you buy your ticket?”A = sale of lottery ticketsB = sale of alcoholC = sale of tobaccoAABBCC

19. Removed-Treatment DesignDemonstrates that outcomes rise and fall with the presence or absence of treatmentO1XO2O3XO4

20. Generally interpretable outcome pattern O1OutcomeWorseBetterO2O3O4XXUninterpretableoutcomeInterpretableoutcome

21. Repeated-Treatment DesignFew threats could explain a close relationship between treatment introductions and removals and parallel outcome changesO1XO2XO3XO4

22. Mean narcotics use over multiple Methadone maintenance on/off conditions

23. A-B DesignsMultiple-baseline design (a class of single-subject designs), or collection of A-B designs, to assess the effects of an intervention across separate baselinesA = baselineB = treatmentThe intervention is introduced in a staggered manner and the baseline provides a predicted level of the dependent variable in absence of the treatmentA-B-A designs are sometimes called removal designs (i.e., the treatment is removed)

24. Number of AccidentsWeeksBaselineBaselineBaselineTreatmentTreatmentTreatmentSite 1Site 2Site 3EffectEffectEffect

25. Designs that use a Control Group but no Pretest

26. Posttest-Only Design with Nonequivalent Control GroupUnknown pretest group differences make it extremely difficult to separate treatment effects from selection effectsNRXO1NRO1

27. Posttest-Only Design using an Independent Sample PretestAssumes overlapping group membershipUseful whenPretest measurements may be reactive, Cannot follow same groups over time, orWhen interested in studying intact communities whose members change over timeNRO1XO2NRO1O2

28. Posttest-Only Design using Proxy PretestProxy measures should be conceptually related to and correlated with the outcomeCan be used for a variety of purposes including indexing selection bias and/or attritionNROA1XOB2NROA1OB2

29. Case Control StudiesPredominant method for many forms of epidemiological researchUsed to identify factors that may contribute to a condition by comparing subjects who have that condition (i.e., 'cases') with those who do not have the condition but are otherwise similar (i.e., 'controls')Famously, the association between smoking and lung cancerSimilar in many respects to Scriven’s GEM and MOM

30. Midterm Examination

31. Midterm ExaminationThe examination will consist of 50-75 multiple-choice items, scored as 0 or 1You will have 2½ hours to complete the examinationYou may use one page of notes (front and back) on 8½” X 11’’ paperYou will be asked questions about statistical power, but will not be required to calculate power

32. Case Study

33. Case Study ActivityAn aid agency implemented a project in Bangladesh with the objective of improving the nutritional and physical health status of men and womenThe intervention consisted of a package of services including: nutrition education, primary health care, and other activitiesTo determine whether the intervention might be effective, the project was field-tested in a small rural community prior to large-scale implementation throughout the countryA small monetary incentive was provided and slightly more than half of the community’s men and women participated in the studyAll men and women in the community were weighed and height measurements taken prior to the intervention - body mass index (BMI) was calculated and then again six months after the interventionThose who did not participate were used as a control group and the evaluators found significant improvements in nutritional and physical health indicators for the treatment group contrasted with the control

34. QuestionsWhat is the design of the study?What internal validity threats are most plausible?How might the design feasibly be improved?