The Use of Test Scores in Secondary Analysis - PowerPoint Presentation

The Use of Test Scores in Secondary Analysis PIAAC Methodological Seminar, June 2019, Paris Dr. Sabine Meinck The Use of Survey Weights in Regression Analysis (Wooldridge) Discussion

Concept note:“… primary analysis is mainly about … efficient estimation of the main parameters … (the cognitive skills) in the population of interest. Secondary analysis … is mainly focused on the … parameters of a statistical model that aims at uncovering the causal relationship between different variables.” Is there really a significant difference between “primary” and “secondary” analysis?“Primary” vs “Secondary” Analysis? 2

I don’t fully agree on the arguments made in the concept note.Reason to mostly utilize simple statistical indicators for “primary” reports is the wealth of the data“Primary” analysis is still more broad than just measuring some domains. Most of the papers with “secondary analysis” I read/reviewed, actually aim for making population inferences, even when using more advanced analysis types“Primary” vs “Secondary” Analysis? 3

In which circumstance would we actually NOT be interested in inferring on the population? I actually have a hard time in thinking of any.Hence, this difference alone clearly is no argument for altering recommendations on using weights.“Primary” vs “Secondary” Analysis? 4

Perhaps. Perhaps not.Using sampling weights for design-unbiased estimation of simple statistical indicators is out of question.Weighting in simple regression analysis? Weighting for propensity score modelling?Weighting in MLM?Weighting in SEM?More evidence is needed to develop efficient weighting schemes for advanced analysis methods.Technical doc’s/User Guides should cover related topics in more comprehensive ways. Different Weighting Schemes Needed? 5

Randomized treatments can hardly ever be observed in LSA. LSA are never RCT’s.If external variables are imposed as “treatments” (e.g., some specific reform in an education sub-system), bias may arise from not having considered this variable in the conditioning model used for scaling. Related discussion really relevant?Treatment effect estimation? 6

There is a large body of literature with a very critical view on the possibilities of uncovering causal effects with LSA data.E.g., special issue of Large-scale Assessments in Education on “Quasi-causal methods” (May 2016). Need to encourage further discussion among economists and educational researchers?Estimating Causal Effects of LSA? 7

Using sampling weights can have an effect on sampling errors. It can go in both directions (increasing or decreasing sampling errors), BUT it reflects the true sampling variance resulting from unequal sample allocation. Weighting Reduces Precision of Estimates?8

Weighting Reduces Precision of Estimates? 9 Think of a country in which all public schools follow the same curriculum Each private school follows a different curriculum If private schools are over-sampled, we overestimate the variety in the school system and also sampling variance. U nless we use weights.

Neglecting weights will lead to biased sampling error estimates. In other words: WLS may produce higher S.E.’s than OLS, but WLS produces unbiased S.E. estimates. Weighting Reduces Precision of Estimates?10

“To ensure consistency use the sampling weights – if they can be trusted.”Generally, all procedures to derive sampling weights are methodologically sound and well established. No doubt that design weights are correctly reflected in the final weights!Sampling frames highly reliable; selection probabilities tracked for each sampling stageTrusting Sampling Weights? 11

Critical issue: weight adjustmentsAssumption: non-informative response model It can be very well argued that this assumption is often violatedThere is even evidence that nonresponse is not occurring at randomWhy don’t we do something about it?Information on the mechanisms of nonresponse is often not available (not at all or not in time)Even if a thorough NRBA is possible, this is still no proof of unbiasedness This is why LSA require such high participation rates Trusting Sampling Weights?12

Enforce statements about using weights (e.g., Carstens and Hastedt, 2010; Braun and von Davier, 2017; Von Davier, Gonzalez and Mislevy, 2009, and now Wooldridge): All recommend using sampling weights for most if not all statistical analysis.More research is needed.  Summary 13

Thank you! Sabine Meincksabine.meinck@iea-hamburg.de