/
Individual-level Moderation Individual-level Moderation

Individual-level Moderation - PowerPoint Presentation

alida-meadow
alida-meadow . @alida-meadow
Follow
413 views
Uploaded On 2017-05-19

Individual-level Moderation - PPT Presentation

in ClusterRandomized Control Trials Sharon Wolf NYU Abu Dhabi Additional Insights Summer Training Institute June 15 2015 1 Outline Conceptual overview Analytic considerations PowerMinimum detectable differential effects ID: 550083

school level reading age level school age reading outcome cluster power predictor interaction subgroup effect treatment scores cross program subgroups intervention variables

Share:

Link:

Embed:

Download Presentation from below link

Download Presentation The PPT/PDF document "Individual-level Moderation" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.


Presentation Transcript

Slide1

Individual-level Moderation in Cluster-Randomized Control Trials

Sharon WolfNYU Abu Dhabi Additional Insights Summer Training InstituteJune 15, 2015

1Slide2

OutlineConceptual overviewAnalytic considerations

Power/Minimum detectable differential effectsCross-level interactions in MLMCentering variablesRecommendations and tips

2Slide3

Conceptual OverviewWhen is the story in the subgroups?

3Slide4

Research Questions that Motivate Subgroup Analyses

Guide questions about how to target resources most efficiently:

How

widespread

are the effects of an intervention?

Is the intervention effective for a specific subgroup

?

Is the intervention effective for any subgroup?

Exploratory* versus confirmatory subgroup findings

4Slide5

Individual-level Moderation versus Cluster-level Moderation

Two examples from welfare reform in the United States and the different policy implications.

Michalopoulos

& Schwartz

(2000

) assessed two types of subgroups:

A range

of

person-level subgroups (e.g., education level

, prior employment experience, and risk of

depression).The nature of the program and program office practices.

5Slide6

Characteristics that Define Individual-level Subgroups

Characteristics believed to be related to the need for a particular intervention or the likelihood of benefiting from it.Demographic characteristics – e.g., gender, age, education levelRisk factors - past smoking, drug abuse, severity of disease, poverty statusCombinations

of characteristics – e.g., gender and age; cumulative levels of risk/risk index

6Slide7

Subgroups that are Exogenous versus Endogenous to the Intervention

Exogenous to the intervention: not affected by the intervention or correlated with its receipt (all pre-random assignment characteristics).

Endogenous

to the intervention: affected by the intervention or correlated with its receipt (e.g., dosage of the intervention). Valid

causal inferences much more

difficult.

Gambia: higher “dosage”

(i.e.,

higher attendance)

 more learning?

Increased

attendance

could bring less advantaged students into the intervention

group, biasing the average treatment effect (ATE) downward

.

7Slide8

Exploratory versus Confirmatory Subgroup Analyses

Exploratory subgroup analysesProvide a basis for hypothesis-generationEssential step in the scientific method

Should be considered suggestive

Confirmatory subgroup analyses

Appropriate basis for testing hypotheses

Provide strong evidence if findings are: (a) consistent with existing findings, (b) large enough magnitude to be meaningful, (c) robust.

Bloom & Michalopoulos, 2010

8Slide9

Contextual considerationsInternal contextual considerationsFeatures of findings internal to a given study

E.g., pattern across all outcomes for a particular subgroup in a studyExternal contextual considerationsFeatures

of findings

external

to

a

given

study

E.g

., consistency with prior study findings

9Slide10

Subgroup analysis assess two questionsWhat is the impact of the program for each subgroup?

What are the relative impacts of the program across subgroups?

10Slide11

Power Calculations for Subgroup Analyses

Minimum detectable differential effects

11Slide12

Some considerations for power calculations of subgroups

1. Did the program work for a particular subgroup ? Assess impacts separately for this subgroup

Assess power to detect impacts for this subgroup

2. Were

the effects

different

for particular subgroups?

Assess impacts using a cross-level interaction

Assess power to detect a cross-level interaction

12Slide13

#1: Did the program work for a particular subgroup?Minimum Detectable Effect Size (MDES): the smallest true effect, in standard deviations of the outcome, that is detectable for a given level of power and statistical significance.

Accepted parameters:Power: 80% Statistical significance level: 0.0513Slide14

Establishing common notationρ = intraclass correlationδ

= MDESλ = non-centrality parameterJ = number of clustersn = number of units per cluster

14Slide15

Power in a 2-level CRT (J. Spybrook)Main effect:Main effect

with covariate:Cluster level Moderator:

Individual level

Moderator:Slide16

Considerations for power analysisThe

number of clusters (highest level units) is more important than the size of the cluster (lower level units) in reducing the MDES.

A higher

intra-cluster correlation

(ICC) increases the

MDES

(i.e., if

τ

00

is relatively large).

The proportion of variance in the outcome you can predict

with L1 and L2 variables (i.e.,

R

|X2 and R

|W

2

) reduces the

MDES.

16Slide17

Subgroup specific analysisMaintains a significant portion of power because the number of clusters (or L2 units) remains the same.

The only statistical difference between the subexperiment and the full experiment is the number of L1 units per cluster.17Slide18

Example using OD software18Slide19

#2: Did the program work differently for different subgroups?Minimum Detectable Effect Size Differences (MDESD): the smallest true effect of the

difference in program impacts for two subgroups, in standard deviations of the outcome, that is detectable for a given level of power and statistical significance.Accepted parameters:Power: 80%

Statistical significance level: 0.05

19Slide20

Power in a 2-level CRT (J. Spybrook)Main effect:

Main effect with covariate:Cluster level Moderator:

Individual level

Moderator:Slide21

Considerations for power analysisWithin-level variance becomes increasingly important. Implications include:

The number of cases per cluster (lower

level units)

become more important for increasing power.

The

intra-cluster

correlation

(ICC)

becomes less significant in affecting power (though still important).

The

proportion of variance in the outcome you can predict with L1 variables (not L2; i.e

.,

R

|X2) increases power.

21Slide22

Cross-level interactions

Assessing individual-level moderation in cluster-randomized trials using multi-level models22Slide23

Testing relationships using multi-level modeling

Lower level direct effects. Does a L1 predictor X (e.g., student gender) have a relationship with the L1 outcome variable Y (e.g., student reading

)?

Cross-level direct effects.

Does a L2 predictor (e.g.,

school treatment status

) have a relationship with an L1 outcome variable Y (e.g.,

student reading

)?

Cross-level interaction effects. Does the nature or strength of the relationship between a L1 variable (e.g., gender) and the outcome (e.g.,

reading

) change as a function of a higher-level variable (e.g.,

school treatment status)?

23Slide24

Model of Program Impacts

Level 1

Level 2

Y

ij

= outcome for individual

i

in cluster j

T

j

=

1

for program-group members,

0

for control-group

γ

00

= mean outcome for the control group

γ

01

= true program impact

r

ij

= error component for individual

i

from cluster j

u

0j

= error component for cluster j

 

24Slide25

Combined equation

 

25Slide26

Adding in a cross-level interactionLevel 2 (e.g., school treatment status

) and Level 1 (e.g., student gender) variables interacting to produce an effect on the outcome (e.g., student reading scores).

In terms of your impact estimation equation:

(a) Add Level 1 predictor (moderator).

(b) Expand Level 2 model to include a fixed slope (

1

).

(c) Add a level 2 predictor (treatment status) to the slope.

26Slide27

Model of Moderated Program Impacts

Level 1

Level 2

 

Added L1 predictor (moderator)

Expand L2 slope

Add L2 predictor to the slope

27Slide28

Model of Moderated Program Impacts

Level 1

Level 2

γ

00

= mean outcome for the control group

γ

01

= estimated program impact for

M

ij

=0

γ

10

= main effect for the moderating variable,

T

j

=0

γ

11

= moderated effect (i.e., interaction)

 

Coefficient for cross-level interaction

28Slide29

Combined equation

 

29Slide30

How to graph a cross-level interaction effect“Simple Regression Equation”: Calculate the expected values of Y

ij under different conditions of Tj and MijFor continuous moderators, plot at values of one standard deviation below the mean, the mean, and one standard deviation above the mean for

M

.

I

t may also be useful to choose additional values that may be informative in specific contexts.

30Slide31

Simple Regression EquationE(Yij

| Mij ,Tj) =

γ

00

+

γ

01

(

T

j

)+ γ10(Mij )+ γ11(

M

ij

)(Tj)

Under control conditions:

E

(

Y

ij

|

M

ij

,Tj = 0) = γ00 + γ10(Mij)Under treatment conditions:E

(

Y

ij

|

M

ij

,

T

j

= 1) =

γ

00

+

γ

01

+

γ

10

(

M

ij

)

+

γ

11

(

M

ij

)

31Slide32

BREAKIf we need it32Slide33

Centering Variables in Multi-level Models

Implications for interpreting effect estimates and detecting impact variation

33Slide34

Decisions about centering depend on your data and your research questionHow do you want to interpret the intercept in your model? The coefficients?Example: School diversity/cultural awareness program

H1: Improved sense of belonging for minority students (L1 moderator).

H2:

Improved

sense

of belonging for minority

students in less diverse schools (L1 & L2 moderators).

The distribution of the moderator variable across clusters needs to be considered.

34Slide35

Two options to center in multi-level modelsCGM = Centering at the grand meanDeviations calculated from the sample mean for all individualsCGM L1 with all individuals; L2 with all clusters

CWC = Centering within clustersaka, group-mean centeringDeviations calculated around the mean of the cluster j to which case i belongs

35Slide36

Y

(outcome)

X

(predictor)

The distribution of M is highly variable across clusters.

36

Cluster 1

Cluster 2

Cluster 3Slide37

Y

(outcome)

CGM

M

37

X

(predictor)Slide38

Y

(outcome)

X

(predictor)

CGM

M

38Slide39

Centering at the Grand Mean (CGM)Does not affect the rank order of scores on the variable. The complex, multilevel association between the L1 and L2 variables is unaffected.Yields scores that are correlated with variables at both levels of the hierarchy. (This is a critical differences with CWC.)

Produces an interaction coefficient (γ11) that is a weighted combination of the within- and between-cluster regression coefficients.

39Slide40

Y

(outcome)

X

(predictor)

CWC

M

1

M

2

M

3

40

The distribution of M is highly concentrated within clusters.Slide41

Y

(outcome)

X

(predictor)

CWC

M

1

M

2

M

3

41Slide42

Centering within Cluster (CWC)Affects the rank order of scores of variables within the sample.Produces scores that are uncorrelated with Level 2 variables (because the mean for all L2 variables is zero).

Produces an interaction coefficient (γ11) that is an unbiased estimate of the Level 1 associationγ11 is a pure estimate of the cross-level interaction, no longer confounded with the Level 2 interaction.

42Slide43

Y

(outcome)

X

(predictor)

43

The distribution of M is even across clusters.Slide44

Y

(outcome)

X

(predictor)

CGM

M

44Slide45

Y

(outcome)

X

(predictor)

CWC

M1

M2

M3

45Slide46

Main TakeawaysCentering will affect estimates more if the predictor variable is not evenly distributed across clusters.Cross-level interaction term using CGM will provide a coefficient estimate that is a mix of the L1 and L2 effects.

Cross-level interaction term using CWC will provide a pure estimate of the L1 relationship.Decisions on how to center depend on your data and your research question (!!).

46Slide47

Example

Predictor: Treatment status (L2)Individual level moderator: Student age (L1) (continuous)Outcome: Reading score

Some options on how to center the data and what it means for interpreting your moderated effect…

47Slide48

Option 1: No centering

Level 1

Level 2

γ

00

is the average school mean reading score for the control group when age=0

γ

10

is the composite

of

the relationship of within

school age-reading

scores and

between-school age reading

scores

γ

11

is the composite

of

the interaction between treatment and within

school age-reading

scores and treatment and between-school

age reading

scores.

 

48Slide49

Option 1: CGM age

Level 1

Level 2

γ

00

is the average school mean reading

score for

schools

for the control group.

γ

10

is the composite of the relationship of within school age-reading scores and between-school age reading

scores.

γ

11

is

still the

composite of the interaction between treatment and within school age-reading scores and treatment and between-school age reading scores.

 

49Slide50

Option 3: CWC age at L1, CGM age at L2

50

Level 1

Level 2

γ

00

is the average school mean reading

score across

the

schools for the control group.

γ

10

is the average change in school mean reading

score for

a 1 unit increase in school mean age across schools (between school age relationship)

γ

11

is the composite of the interaction between treatment and within school age-reading scores and treatment and between-school age reading scores.

 Slide51

Option 4: CWC age at L1, CGM age at L2 interacted with treatment status51

Level 1

Level 2

γ

00

is the average school mean reading fluency across the

schools for the control group.

γ

10

is the average change in school mean reading fluency for a 1 unit increase in school mean age across schools (between school age relationship

).

γ

03

is the moderated relationship between treatment and between school age reading scores.

γ

11

is the

moderated relationship between

treatment and within school age-reading

scores.

 Slide52

General Recommendations and Tips

52Slide53

Multiple Hypothesis T

estingDistortions to statistical inferences

can

occur when multiple related hypothesis tests

are conducted.

Suggested approaches:

Explicitly

distinguish between exploratory and confirmatory

findings

Minimize the number of confirmatory hypothesis tests conducted by a given study.

Create an omnibus hypothesis test about the intervention’s effects that considers all outcome measures and subgroups together. (e.g., composite measure of individual outcomes).

Consider family-wise error correction (reduces statistical

power

considerably).

53Slide54

Practical tips and recommendations

Calculate ρ for all levels.Determine your research question and relevant approach to assessing subgroup affects.

Calculate the power needed to detect a subgroup effect (either for a particular subgroup, or for a cross-level interaction, depending on your research question).

Rescale (i.e., center) predictor

variables as needed.

Assess the practical significance of your findings (i.e., calculate effect sizes

).

Report results regarding each step of the model building process including all coefficients, standard errors and variance components

.

54Slide55

ReferencesAguinis, H., Gottfredson, R. K., & Culpepper, S. A. (2013). Best-practice recommendations for estimating cross-level interaction effects using multilevel modeling. Journal of Management

, 0149206313478188.Bloom, H. S. (Ed.). (2005). Learning more from social experiments: Evolving analytic approaches. Russell Sage Foundation.Bloom, H. & Michalopoulos, M. (2013). When Is the Story in the Subgroups? MDRC Working Paper.

Enders, C. K., & Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: a new look at an old issue.

Psychological methods

,

12

(2), 121.

Mathieu, J. E., Aguinis, H., Culpepper, S. A., & Chen, G. (2012). Understanding and estimating the power to detect cross-level interaction effects in multilevel modeling.

Journal of Applied Psychology

,

97(5), 951.55