A little history first Why most published findings are false Ioannidis 2005 biostatistician Clinical trials epidemiological studies molecular research Less likely to be true if Studies are smaller ID: 675765
Download Presentation The PPT/PDF document "Reproducibility Or, “Who knew that rep..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Meta-analysisSlide2
Significance and effect sizes
What is the problem with just using p-levels to determine whether one variable has an effect on another?
Be careful with comparisons--sample results:
For boys,
r
(87) = .31,
p
= .03
For girls,
r
(98) = .24,
p
= .14
How does sample size affect effect size? Significance?
Why are effect sizes important?
What is the difference between statistical, practical, and clinical significance?Slide3
What should you report?
2 group comparison—treatment vs. control on anxiety symptoms
3 group comparison—positive prime vs. negative prime vs. no prime on number of problems solved
2 continuous variables—relationship between neuroticism and goal directedness
3 continuous variables—anxiety as a function of self-esteem and authoritarian parenting
2 categorical variables—relationship between answers to 2 multiple choice questionsSlide4
Types of reviews
Narrative
review vs
.
meta-analysis vs. integrative data analysis
When was the first meta-analysis?
When was the term first used?
What are the advantages of quantitative reviews?
What are problems with them? Slide5
Steps to meta-analysisSlide6
1. Define
your variables/question
1
df
contrasts (or multivariate meta-analysis or SEM-based meta-analysis)
What is a contrast?Slide7
2. Decide on inclusion/exclusion criteria
What factors do you want to consider here?Slide8
3. Collect studies systematically
Where do you find studies?
File drawer problem
Gray litSlide9
4. Code your studies
What should you code?
Inter-rater
reliability
If there is more than 1 effect per study, what do you do?
What does the sign mean on an effect size?
What are small, medium, and large effects?
How can you convert from one to another?
r or d?
http
://
www.soph.uab.edu/Statgenetics/People/MBeasley/Courses/EffectSizeConversion.pdfSlide10
Families of effect sizes—d family
2 group comparisons (difference between the means)
Cohen’s d (with various subscripts)
Hedge’s g
Glass’s d or delta
Within vs. between-participants designs
https://
www.frontiersin.org/articles/10.3389/fpsyg.2013.00863/full
Lakens
, 2013 (Table 1) Slide11
Families of effect sizes—R family
Continuous or multi-group (proportion of variability)
η
2
η
p
2
η
G
2
ω
2
and its parts
r
, fisher’s z, R
2
, adjusted R
2
difference
between
η
2
and R
2
family
https://www.frontiersin.org/articles/10.3389/fpsyg.2013.00863/full
Lakens
, 2013 (Table
2) Slide12Slide13Slide14
Other effect sizes
Nonparametric effect sizes
Nonnormal
data: convert z to r or d
Categorical data:
Rho
Cramer’s V
Goodman-
Kruskal’s
Lambda
How can you increase your effect sizes?Slide15
CIs
How can you calculate confidence intervals around your effect sizes?
http://
daniellakens.blogspot.com/2014/06/calculating-confidence-intervals-for.html
https://thenewstatistics.com/itns/esci
/
http
://www.cem.org/effect-size-calculator
https://www.aggieerin.com/shiny-server/Slide16
Interpretation of effect sizes
Recommended
for at least most important
findings
Benchmarks?
SD units
Practical or clinical significance and compare to lit
PS or common language effect size
U
Binomial effect size
display
Relative
risk
Odds ratio
Risk differenceSlide17
5.
Combine effect sizes
When should you do fixed vs. random effects?
Should you weight effect sizes, and if so, on what?
How can you deal with dependent effect sizes?
Hunter and Schmidt method vs. Hedges et al. methodSlide18
6
.
Calculate confidence intervals
Credibility intervals vs. confidence intervalsSlide19
7. Check for/correct for biases
m-a effect = True effect + effect of pub and
exp
bias
Outliers
Correct for unreliabilitySlide20
Publication bias
Rosenthal’s
fail-safe N
# studies needed at p < .05= (K/2.706) (K(mean Z squared) = 2.706)
Z = Z for that level of p
K = number of studies in meta-analysis
Funnel plot
(Egger’s test)
Rank
correlation test for pub
bias
Correlation between n and ESSlide21
Fig. 3. Funnel plots of 11 (subsets of) meta-analyses from 2011 and Greenwald, Poehlman, Uhlman, and Banaij (2009).
Marjan Bakker et al. Perspectives on Psychological Science 2012;7:543-554
Copyright © by Association for Psychological ScienceSlide22
Responses to publication bias
Trim
and fill
Sensitivity analysis
WAPP-WLS
PET-PEESE (Figure 1; van Elk et al., 2015)
Critiques of
PET-PEESE
,
http://
datacolada.org/59
p-uniform
3PSM
Cumulative meta-analysis
Bayesian approaches (e.g., BALM)
What did Carter et al. assess the effects of (Table 1)?
What effects did QRPs have on meta-analytic estimates? Slide23
p-curves
P-curve analysis (Figure 1; Simmons &
Simonsohn
, 2017
)
Critiques of p-curves
www.p-curve.com
What do Carter et al. recommend? (p. 135)Slide24
8. Look at heterogeneity of effect sizes
Chi-square test
I
2
(measure based on Chi-square)
Cochran’s Q
Standard deviations of effect sizes
Stem and leaf plot
Box plot
Forest plotSlide25
Forest plotSlide26
9. Look for moderators
What are common moderators you might test?
How do you compare moderators? Slide27
“little ‘m’ meta-analysis”
Comparing and combining effect sizes on a smaller level—when might you want to do this?
How would you do it?
Average within-cell
r’s
with fisher z transforms
To compare
independent
r’s
: Z = z
1
-z
2
/
sqrt
((1/n-3) + (1/n-3))
To combine
independent
r’s
: z = z
1
+z
2
/2Slide28
Write-up
Inclusion criteria, search, what effect size
Which m-a tech and why
Stem and leaf plots of effect sizes (and maybe
mods
)
Forest plots
Stats on variability of effect sizes, estimate of pop effect size and
confidence/credibility
intervals
Publication bias analysesSlide29
Other suggestions/questions on meta-analysis? Slide30
Coming up
Data cleaning and basic analyses due today
Next class:
Replication readings
Meta-analysis assignment
The next
week:
Presentations
for proposal
Formal presentations—dress nice, stand up
No more than 12 minutes
I’ll take notes and send to you
Go through your FINAL plan for your study—background, method, expected results, and
discussion