Propensity Score Analysis with SEM - PowerPoint Presentation

Slide1

Propensity Score Analysis with SEM

A tutorial with MPLUS

Walter L. Leite, University of Florida

Laura M. Stapleton, University of MarylandSlide2

Learning ObjectivesDescribe quasi-experimental research designs

Identify propensity score analysis methodsExplain how PSA can benefit from SEM

Detail the use of MPLUS to implement PSA with SEMSlide3

Characteristics of a Quasi-experimental/Observational study

The objective is to estimate the effect of a condition (e.g. treatment, program, intervention) on outcomes. The condition was not randomly assigned to participants.Manipulation of the condition is possible. Slide4

ExamplesStudy of the effect

of high school student participation in career academies on future income;Study of the effect of having a full time security personnel in schools on the use of harsh discipline;Study of the effect of mothers having a job that provides or subsidizes child care on the length that they breastfeed their children

Study the effect

of

participation in a virtual

learning environment

on algebra achievement.

Study

the effect of self-employment on job

satisfaction;

Study e

effect of center-based care in

kindergarten

on

mathematics achievement;Slide5

Rubin’s Causal Model

All individuals in the population have potential outcomes and in the presence of the treatment and control conditions.; The outcomes of the treatment group are only observed in the presence of the treatment condition ; The outcomes of the control group are only observed in the presence of the control condition. Slide6

Average Treatment Effect (ATE)It is the difference between the outcomes of the individuals in the treated and untreated conditions.Slide7

Conventional estimation of the ATE

In randomized designs, this estimator is valid because:

Sample mean of treatment group

Sample mean of control groupSlide8

Strong Ignorability of Treatment Assignment

The treatment assignment is independent of the potential outcome distributions, given observed covariates.

It also requires that for every value of

Z,

the probability of treatment assignment is neither zero nor one

Slide9

Average Treatment Effect for the TreatedIt is the difference between the expected value of the outcomes of the treated individuals and the expected value of the potential outcomes of the treated individuals.Slide10

Average Treatment Effect for the ControlIt is the difference between the expected value of the outcomes of the control individuals and the expected value of the potential outcomes of the control individuals.Slide11

Regression Analysis to estimate a treatment effectLimitations:It assumes a linear relationship between the outcome and covariates.

It assumes homogeneity of regression slopes. is not the ATE or ATT if the functional form of the relationship between any X and Y is misspecified, and there is any interaction between X and Z.Slide12

Advantages of Propensity Score Methods over RegressionThe ATE or ATT can be estimated with smaller models where fewer parameters are estimated;

Linearity assumptions are not made;12Slide13

Propensity scores

The propensity score is defined as a conditional probability of treatment assignment, given observed covariates (Rosembaum & Rubin, 1983);

13Slide14

Strong Ignorability of Treatment Assignment and Propensity Scores

If treatment selection is strongly ignorable given an observed set of individual covariates X, then it is also strongly ignorable when these individual covariates are combined into a propensity score e(X

), (proved Rosenbaum & Rubin 1983):Slide15

Steps of Propensity Score Analysis

Data PreparationPropensity Score Estimation

Propensity Score Method Implementation

Covariate Balance Evaluation

Treatment Effect Estimation

Sensitivity Analysis

Design Stage

Analysis StageSlide16

Data PreparationExamined proportion of participants in treated/untreated groupsSelect covariates

Diagnose and deal with missing dataSlide17

Propensity Score EstimationParametric models: logistic regression, probit regression

Data mining methods: classification trees, random forests, boosted logistic regression, neural networksExamine common support: There should be an area of the propensity score distributions where values exist for both treated and untreated groupsSlide18

Propensity Score Method ImplementationPropensity scores methods:

MatchingStratificationWeightingSlide19

Theoretical justification of pair matchingThe difference in response of treatment and untreated units in a matched pair with the same value of PS equals the ATE at that value of PS.

The mean of the matched pair differences is the ATE. Slide20

Matching MethodsMatching methods can be classified by ratio and algorithm:Ratios: One-to-one, one-to-K, variable ratio

Algorithm: Greedy, genetic, optimalSlide21

StratificationDivide propensity score distribution of treated and untreated into strata, typically five.

Estimate the treatment effect within strataCombine the treatment effect estimates across strata using weights.Stratification can be considered as a coerce form of matching, or as a non-parametric form of weighting.Slide22

WeightingSimilarly to sampling weights, propensity score weights adjust the distributions of covariates so that they are similar across treated and untreated groups.

Inverse probability of treatment weighting: For estimating the ATE, weight observations by the inverse of the probability of receiving the treatment they received. Weighting by the odds: For estimating the ATT, treated receive a weight of one, untreated receive a weight equal to the odds of treatment.Slide23

Evaluation of covariate balanceBalance refers to the equivalence of the treatment and untreated groups’ joint distribution on all observed covariates:

In practice, balance on the joint distribution is hard to check, therefore the focus is on each covariates distribution separately;Covariate balance can be checked by visual, descriptive or inferential methods.Slide24

Checking for BalanceVisual inspection

: Comparison of histograms, kernel density plots, QQ-plotsDescriptive criteria that compare the covariate distribution of the treatment and control group:

Standardized mean difference

Variance ratioSlide25

Treatment Effect EstimationNon-parametric estimators: Mean differencesParametric estimators: statistical models, typically regression, but can be complex such as multilevel models and structural equation models.Slide26

Sensitivity AnalysisIt asks the question: Would the conclusion change if an important covariate was omitted?

Goals: Determine how strong the effect of an omitted covariate would have to be for the significance test of the treatment effect to changeDetermine the degree of robustness of treatment effects to hidden bias. Slide27

Limitations of Propensity Score AnalysisIt assumes that all true confounders are observed and included in the propensity score model.

It assumes that the propensity score model is correctly specified.It assumes that covariates are reliably measured.It assumes measurement equivalence between treated and untreated groups.Slide28

Consequence of ignoring random measurement error in propensity score analysisPropensity scores will not balance the latent confounding variables across treated and control

groups if they are measured with substantial error. Estimated treatment effects on latent variables measured with error will be attenuated.Slide29

Latent Variables in propensity score analysisLatent variables are not observed directly, but they influence performance on observed indicators.In propensity score analysis, latent variables may appear as confounding variables in the design stage or outcomes in the analysis stage.Slide30

Uses of structural equation modeling in propensity score analysis1) SEM can estimate a propensity score model with latent covariates;

2) SEM can estimate treatment effects on latent variables;3) SEM can provide validity evidence for the measurement of latent variables used in PSA. Slide31

ExampleThe objective is to estimate the effect of new teacher participation in a network of teachers on their perception of workload

manageability. The hypothesis is:Teachers that participate in a network of teachers have higher levels of perceived workload manageability.Slide32

Steps of design stage of PSA with latent confounding variables

Data preparation and covariate selection: Identify observed and latent confounding variables.Fit measurement model for latent confounding variables

Evaluate measurement invariance.

Estimate propensity

s

core scores

Implement propensity score method

Evaluate covariate balance.Slide33

Data PreparationData sources:

1999-2000 School and Staffing Survey (SASS), Teacher Follow Up Survey (TFS). Sampling design: stratified multi-stage sampling design with a stratified sample of schools and teachers

sampled

within schools.

Treatment indicator:

“In

the past 12 months, have you participated in the following activities related to teaching?”

Option: “Participating

in a network of teachers (e.g., one organized by an outside agency or over the Internet)”.

Sample:

1030 new teachers with zero to three years of teaching

experience, with 223 (21.7%) treated.Slide34

Covariate SelectionFive latent

covariates: Perception of school managementPerception of family background of students Perception of student delinquency

P

erception

of student

participation

Perception

of teacher

support

20

observed

variablesSlide35

Multiple-group CFA Model and Evaluation of Measurement Invariance

Invariance testing: estimate parameters of the two groups simultaneously with and without constraints for parameter equality across groups. Perform likelihood ratio test between constrained and unconstrained models.Slide36

Multiple-group CFA for Latent Confounding VariablesSlide37

Taxonomy for Invariance

Strict factorial invariance: Requires that loadings, error variances, and intercepts/thresholds are the same across multiple groups. Strong factorial invariance, scalar invariance:

Requires that the loadings and intercepts/thresholds are constant across groups.

Weak factorial invariance, metric invariance:

Only requires that loadings to be equivalent across groups.

Configural

invariance:

the pattern of zero and non-zero loadings is constant across groups, and the sign of the non-zero loadings is the same for all the groups examined. Slide38

Invariance requirements for multiple group SEM with latent meansIn order to compare latent means, there should be at least scalar (strong factorial invariance).This means that factor loadings and intercepts should be equal across groups.

Measurement invariance testing should be performed before mean differences can be interpreted.38Slide39

Title: Multiple-Group CFA and Invariance Testing with MPLUSdata: file is …;

variable: names are …;categorical are all;grouping is treat (1=Network 0=

noNetwork

);

weight

= TFSFINWT

;

analysis

:

estimator =

wlsmv

;MODEL = CONFIGURAL SCALAR

;Slide40

MODEL:PSCHMAN by T0299* T0300 T0306 T0307 T0310 T0312;PSCHMAN@1;

PFAMBACK by T0335* T0336 T0337 T0338;PFAMBACK@1;PSTUDEL by T0325* T0326 T0327 T0331 T0332;PSTUDEL@1;

PSTUPAR

by T0321* T0322 T0324;

PSTUPAR@1;

PTEACSUP

by T0308* T0309 T0311 ;

PTEACSUP@1;Slide41

Measurement Invariance Testing Results

Number of Degrees of Model Parameters Chi-Square Freedom P-Value

Configural

188 503.366 358 0.0000

Scalar 135 549.460 411 0.0000

Degrees of

Models Compared Chi-Square Freedom P-Value

Scalar against

Configural

59.734 53 0.2443Slide42

Fit of Multiple-group CFA with Scalar Invariance

Chi-Square Test of Model Fit Value 549.460*

Degrees of Freedom 411

P-Value 0.0000

RMSEA

(Root Mean Square Error Of Approximation)

Estimate 0.026

90 Percent C.I. 0.020 0.031

Probability RMSEA <= .05 1.000

CFI/TLI

CFI 0.978

TLI 0.977Slide43

Perception of School Management

The principal lets staff members know what is expected of them.

0.805

The school administration’s behavior toward the staff is supportive and encouraging.

0.801

My principal enforces school rules for student conduct and backs me up when I need it.

0.784

The principal talks with me frequently about my instructional practices.

0.693

The principal knows what kind of school he/she wants and has communicated it to the staff.

0.863

In this school staff members are recognized for a job well done.

0.768Slide44

Perception of Family Background

Problem-parental involvement

0.821

Problem-poverty

0.743

Problem-unprepared students

0.938

Problem-student health

0.760Slide45

Perception of Student Delinquency

Problem-physical conflicts

0.713

Problem-theft

0.723

Problem-vandalism

0.785

Problem-weapons

0.758

Problem-disrespect for teachers

0.838Slide46

Perception of Student Participation

Problem – Student tardiness

0.791

Problem-student absenteeism

0.877

Problem-class cutting

0.825Slide47

Perception of Teacher Support

Rules for student behavior are consistently enforced by teachers in this school even for students who are not in their classes.

0.820

Most of my colleagues share my beliefs and values about what the central mission of the school should be.

0.671

There is a great deal of cooperative effort among the staff members.

0.736Slide48

Propensity Score Estimation with Latent Variables as CovariatesSingle step:

Estimate measurement model and the propensity score model simultaneously with SEM;Two step: Save factor scores from the confirmatory factor analysis, then fit a logistic regression with factor scores as predictors.Slide49

Estimation of Propensity Scores with SEMZ is the treatment indicator;

X are indicators of latent covariates. are latent covariates;W are observed covariates.Slide50

Estimation of Propensity Scores with SEM using MPLUS model:

PSCHMAN by T0299@1 T0300 T0306 T0307 T0310 T0312;PFAMBACK by T0335@1 T0336 T0337 T0338;PSTUDEL

by

T0325@1

T0326 T0327 T0331 T0332;

PSTUPAR

by

T0321@1

T0322 T0324;

PTEACSUP

by

T0308@1 T0309 T0311 ;treat

on PSCHMAN PFAMBACK PSTUDEL PSTUPAR PTEACSUP LEP_T PLAN T0059 T0106 T0120 T0122 T0124 T0125 T0126 T0127 T0147 T0150 T0153 T0154 T0158 T0248 T0250

T0208 PUPILS

teachImp

;Slide51

Saving of propensity scores with MPLUS savedata: save = propensity;

file = propensityscores.txt; format = free;Slide52

Estimation of Propensity Scores with Logistic RegressionZ is the treatment indicator;

X are indicators of latent covariates.F are factor scores of latent covariates;W are observed covariates.Slide53

Estimation of Logistic Regression with MPLUS analysis:

estimator = mlr; model:treat on PSCHMAN PFAMBACK PSTUDEL PSTUPAR PTEACSUP

LEP_T

PLAN T0059 T0106 T0120

T0122 T0124

T0125 T0126 T0127

T0147 T0150 T0153

T0154 T0158 T0248 T0250

T0208 PUPILS

teachImp

;

savedata:

save = propensity; file = propensityscores2.txt; format = free;Slide54

Estimation of propensity scores with mixture modeling with known classesVariable:

CLASSES = group (2);KNOWNCLASS = group (treat = 0 1);analysis:type=mixture;

algorithm=integration;

integration=

montecarlo

;

savedata

:

save = CPROBABILITIES;

file = propensityscores3.txt;

format = free;Slide55

Weight for estimating the ATE

Condition: Treated or untreated.

Propensity ScoreSlide56

Weight for estimating the ATTThe weight is 1 for treated individuals and the odds of treatment for untreated individuals.

Condition: Treated or untreated.

Propensity ScoreSlide57

Calculation of Weights with MPLUSATEDefine:IF (treat EQ 1) THEN

wghtATE = 1/PropScr;IF (treat EQ 0) THEN wghtATE = 1/(1-PropScr);

ATT

:

Define:

IF (treat EQ 1) THEN

wghtATT

= 1;

IF (treat EQ 0) THEN

wghtATT

=

PropScr/(1-PropScr);Slide58

Calculation of Weights with MPLUS combined with sampling weightsATE

Define:IF (treat EQ 1) THEN wghtATE = TFSFINWT/PropScr;IF (treat EQ 0) THEN wghtATE = TFSFINWT/(1-PropScr);

ATT

:

Define:

IF (treat EQ 1) THEN

wghtATT

= TFSFINWT;

IF (treat EQ 0) THEN

wghtATT

= TFSFINWT*

PropScr

/(1-PropScr);Slide59

Covariate balance evaluation of latent variablesExamination of differences between treated and control groups on latent means can performed by using

multiple-group CFA.For identification, the latent variable means of one group are fixed at zero and the means of the other group are the between-group mean differences.The latent variable means are balanced if they are close to zero for the second group.Slide60

Title: Multiple-Group CFA for covariate balance evaluation of latent variables with MPLUS

data: file is …;variable: names are …;categorical are all;

grouping

is treat (1=Network 0=

noNetwork

);

weight

=

wghtATE

;

model:

PSCHMAN by T0299@1 T0300 T0306 T0307 T0310 T0312;PFAMBACK by T0335@1 T0336 T0337 T0338;

PSTUDEL by T0325@1 T0326 T0327 T0331 T0332;PSTUPAR by T0321@1 T0322 T0324;PTEACSUP by T0308@1 T0309 T0311 ;Slide61

Results for Covariate Balance Evaluation of Latent Variables Estimate S.E

. Est./S.E. P-ValueMeans PSCHMAN 0.043 0.106 0.410 0.682 PFAMBACK -0.040 0.113 -0.352 0.724 PSTUDEL

-

0.140 0.115 -1.214 0.225

PSTUPAR

-

0.093 0.099 -0.936 0.349

PTEACSUP

0.006

0.097 0.063 0.950Slide62

Multiple-group

analysis of covariate balance evaluation of observed variables

model Network:

[

LEP_T](t1);

[PLAN](t2);

[T0106](t3);

[T0208](t4);

[PUPILS] (t5);

[T0059$1] (t6)

[T0120$1] (t7) [T0120$2] (t8) [T0124$1] (t9)

model

noNetwork

:

[LEP_T](c1);

[PLAN](c2);

[T0106](c3);

[T0208](c4);

[PUPILS] (c5);

[T0059$1] (c6)

[T0120$1](c7)

[T0120$2] (c8)

[T0124$1] (c9)Slide63

Constraints in multiple-group analysis of covariate balance with observed covariatesMODEL CONSTRAINT:

NEW(D1 D2 D3 D4 D5 D6 D7 D8 D9); D1 = T1-C1; D2 = T2-C2; D3 = T3-C3;

D4 = T4-C4;

D5 = T5-C5;

D6 = T6-C6;

D7 = T7-C7;

D8 = T8-C8;

D9 = T9-C9;Slide64

Results of balance evaluation of observed covariates with propensity score weights Estimate S.E.

Est./S.E. P-Value D1 -0.014 0.288 -0.049 0.961 D2 -0.008 0.136 -0.058 0.954 D3 -0.005 0.188 -0.026 0.979

D4 0.035 0.127 0.277 0.782

D5 0.048 0.271 0.177 0.860

D6 0.045 0.179 0.253 0.801

D7 -0.009 0.166 -0.055 0.956

D8 0.040 0.163 0.245 0.807

D9 0.035 0.194 0.180 0.857Slide65

Steps of analysis stage of propensity score analysis with a latent outcome Identify

indicators for the latent outcome; Fit measurement model; Assess model fit and re-specify as needed; Evaluate measurement invariance of latent outcome across treated and control groups;

Fit

structural equation model for estimation of treatment effects;

Evaluate

fit of structural equation model;

If

fit is acceptable, interpret treatment effect estimate;Slide66

Estimation of treatment effect with Multiple-Group SEM

Workload Manageability

(Teachers in networks)

X

2

X

3

X

1

X

4



2



3



1



4



2



3



1



4

Workload Manageability

Teachers not in networks)

X

2

X

3

X

1

X

4



2



3



1



4



2



3



1



4

1



1



0

=

0



Slide67

Multiple-group Model Latent Outcome Variable data: file is MPLUSdata_with_weights2.txt;

variable: names are ….;usevariables = F0105 F0108 F0113 F0116; categorical are all;

grouping is treat (1=Network 0=

noNetwork

);

weight = WGHTATE;

analysis:

estimator =

wlsmv; MODEL = CONFIGURAL SCALAR;

model:WORKMAN BY F0105* F0108@1 F0113 F0116; WORKMAN@1;Slide68

Results of invariance testing for latent outcomeInvariance Testing

Number of Degrees of Model Parameters Chi-Square Freedom P-Value

Configural

40 8.823

4 0.0657

Scalar

26 24.364

18

0.1435

Degrees of

Models Compared Chi-Square Freedom P-Value

Scalar against Configural 16.730 14 0.2708Slide69

Model Fit Information with Scalar InvarianceChi-Square Test of Model Fit

Value 24.364* Degrees of Freedom 18 P-Value 0.1435

RMSEA (Root Mean Square Error Of Approximation)

Estimate 0.026

90 Percent C.I. 0.000 0.050

Probability RMSEA <= .05 0.948

CFI 0.962

TLI 0.975Slide70

Results of Estimation of the ATE Estimate S.E. Est./S.E. P-Value

Group NETWORK WORKMAN BY F0105 0.393 0.060 6.523 0.000 F0108 0.798 0.087 9.187 0.000 F0113 0.495 0.074 6.697 0.000

F0116 0.385 0.060 6.448 0.000

Means

WORKMAN 0.363 0.175 2.076 0.038Slide71

Estimation of treatment effect with mimic modelThe multiple indicator and multiple causes model (MIMIC)

is a single-group model where the treatment indicator is added as a covariate.Slide72

MIMIC Model to Estimate the ATE with MPLUS data: file is MPLUSdata_with_weights2.txt; variable:

names are …; usevariables = F0105 F0108 F0113 F0116 treat; categorical are F0105 F0108 F0113 F0116;

weight = WGHTATE;

analysis

:

estimator =

wlsmv

;

model:

WORKMAN BY F0105* F0108@1 F0113 F0116; WORKMAN on treat;Slide73

Fit of MIMIC ModelChi-Square Test of Model Fit

Value 8.495* Degrees of Freedom 5 P-Value 0.1310RMSEA (Root Mean Square Error Of Approximation)

Estimate 0.026

90 Percent C.I. 0.000 0.055

Probability RMSEA <= .05 0.903

CFI/TLI

CFI 0.978

TLI 0.957Slide74

Results of MIMIC Model Estimate S.E. Est./S.E. P-Value

WORKMAN BY F0105 0.396 0.058 6.826 0.000 F0108 0.827 0.090 9.213 0.000 F0113 0.494 0.076 6.535 0.000 F0116 0.365 0.059 6.224 0.000

WORKMAN ON

TREAT 0.117 0.060 1.944 0.052Slide75

Comparison of multiple-group SEM with MIMIC modelMIMIC models always assume equal variances across groups, but multiple-group SEM can allow unequal variances.

Multiple-group SEM for latent mean comparisons require that observed means are available, but MIMIC models can be fit to covariance matrices only.MIMIC models assume strict factorial invariance, while multiple-group SEM only needs strong factorial invariance for the latent mean comparison to be valid.

75Slide76

Related WorkLeite, W. L. (in press). Practical Propensity Score Analysis Using R. Sage.

Publication date: December, 2016. Leite, W. L., Jimenez, F., Kaya, Y., Stapleton, L. M., MacInnes, J. W., & Sandbach, R. (2015). An Evaluation of Weighting Methods Based on Propensity Scores to Reduce Selection Bias in Multilevel Observational Studies. Multivariate Behavioral Research, 50

, 265-284.

doi

:

10.1080/00273171.2014.991018

Leite, W. L.,

Sandbach

, R.,

Jin

, R.,

MacInnes, J. W., & Jackman

, M. G. (2012). An Evaluation of Latent Growth Models for Propensity Score Matched Groups. Structural Equation Modeling: A Multidisciplinary Journal, 19, 437-456. doi: 10.1080/10705511.2012.687666Stuart, E. A. (2010). Matching Methods for Causal Inference: A Review and a Look Forward.

Statistical Science, 25

, 1-21.

doi

:

10.1214/09-sts313

Schafer, J. L., & Kang, J. (2008). Average causal effects from nonrandomized studies: A practical guide and simulated example.

Psychological methods, 13

, 279-313.

doi

:

10.1037/a0014268Slide77

Thank you!

Contact:Walter.leite@coe.ufl.edu