Ashley Amaya Matthew Williams Devon Cribb Rachel Harter amp Katherine Morton August 2 2018 JSM 2018 1 Acknowledgements The presentation is sponsored by RTI Internationals Survey Research Division with the research included in the presentation stemming from ongoing methodological wor ID: 748834
Download Presentation The PPT/PDF document "Switching from Field Enumeration to an A..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Switching from Field Enumeration to an ABS Frame: The Effect on Coverage Bias
Ashley Amaya, Matthew Williams, Devon Cribb, Rachel Harter, & Katherine MortonAugust 2, 2018
JSM 2018
1Slide2
Acknowledgements
The presentation is sponsored by RTI International’s Survey Research Division, with the research included in the presentation stemming from ongoing methodological work conducted under the contract for the National Survey on Drug Use and Health (NSDUH). The NSDUH is funded by the Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Behavioral Health Statistics and Quality under contract no. 283-2017-00002C and project no. 0215638.
The views expressed in this presentation do not necessarily reflect the official position or policies of SAMHSA or the U.S. Department of Health and Human Services; nor does mention of trade names, commercial practices, or organizations imply endorsement by the U.S. Government.
2
JSM 2018Slide3
Introduction to ABS
What is address-based sampling?
Address list based on the US Postal Service’s databaseGeocode addresses into sample segmentsDraw a sample Why do we use it?
Great coverage (over 90%) (AAPOR 2016)Less expensiveEliminates human listing errorTimely
3
JSM 2018Slide4
Undercoverage on the ABS Frame
The ABS frame is not perfect
Undercoverage is much higher in rural areas23-35% in rural areas vs. 1-10% in urban areas (Dohrmann et al 2006; Dohrmann et al 2007; O’Muircheartaigh et al 2007)
Purposely excludes:Unique ZIP codes (e.g., AIAN tribal areas and universities)Vacant units in rural areasIncludes “unusable” addresses:
PO Boxes
Simplified addresses (e.g., rural routes)
4
JSM 2018Slide5
Research Question
Does the use of an ABS frame introduce coverage bias?
If so, how much?
5
JSM 2018Slide6
Methods - Datasets
The NSDUH provides national, state and substate data on substance use and mental health in the civilian, noninstitutionalized population age 12 and older. Data are collected on a quarterly basis each year.
Approximately 700 field interviewers (FIs) staffed.Approximately 140,000 household screenings and 67,500 interviews completed annually.
Conducted by RTI under contract with SAMHSA.Currently uses a field enumerated frame.
6
JSM 2018Slide7
Methods – Datasets (cont’d)
3 datasets constructed from 2015-2016 NSDUH
Field enumerated (FE) datasetAll respondentsN = 136,000
ABS Subsample 1FE dataset minus residents of description-based addressesN = 129,000
ABS Subsample 2
Subsample 1 minus residents of tribal areas and group quarters
N = 125,000
7
JSM 2018Slide8
Methods – Variables of Interest
8
JSM 2018
Alcohol
Past month binge alcohol use
Past month alcohol use
Past year alcohol use disorder
Other drugs
Past month stimulant use
Past year substance use disorder
Past month pain reliever use
Past year illicit drug use disorder
Past year specialty substance use treatment
Past month
cigarette
use
Past month
marijuana
use
Mental health
Past year serious mental illness
Past year any mental illness
Past year mental health service use
Past year major depressive episode
Past year major depressive episodeSlide9
Methods – Domains of Interest
9
JSM 2018
Age
Sex
Race
Hispanicity
Census division
County type
College education status
Pregnancy status
13 two-way cross domains
Up to 325 comparisons for each outcome variableSlide10
Methods – Overview of Analyses
Overall
differences by sampleSummary of differences by
measureSummary of differences by sample size
Summary of differences in
conclusions
drawn
10
JSM 2018Slide11
11
JSM 2018
1. Overall differences by sampleSlide12
1. Overall Differences by Sample
12
JSM 2018
*
p
<0.05Slide13
13
JSM 2018
2. Summary of significant subdomain comparisons by measureSlide14
2. Summary Relative Difference in Estimates: Marijuana Use
14
JSM 2018Slide15
2. Summary Relative Difference in Estimates: Marijuana Use
15
JSM 2018
6% (n=20)
3% (n=10)
Few comparisons were significantly different between FE and ABS samples.Slide16
2. Summary Relative Difference in Estimates: Marijuana Use
16
JSM 2018
But,
among
the significant differences, the magnitude was often larger than 1%.
90% (n=18)
100% (n=10)Slide17
2. Summary of Findings by Measure
ABS Subsample 2
Unaffected
Bias on few domains, but large
Bias on many domains but small
Bias on many domains and large
ABS Subsample 1
Unaffected
Binge alcohol
Stimulant use
Serious mental illness (18+)
Bias on few domains but large
Substance use disorder
Specialty substance use treatment
Marijuana use
Past month pain reliever use
Major depressive episode (18+)
Bias on many domains but small
Mental health service use (18+)
Alcohol use
Cigarette use
Any mental illness (18+)
Bias on many domains and large
Alcohol use disorder
Illicit drug use disorder
17
JSM 2018Slide18
2. Summary of Findings by Measure
ABS Subsample 2
Unaffected
Bias on few domains, but large
Bias on many domains but small
Bias on many domains and large
ABS Subsample 1
Unaffected
Binge alcohol
Stimulant use
Serious mental illness (18+)
Bias on few domains but large
Substance use disorder
Specialty substance use treatment
Marijuana use
Past month pain reliever use
Major depressive episode (18+)
Bias on many domains but small
Mental health service use (18+)
Alcohol use
Cigarette use
Any mental illness (18+)
Bias on many domains and large
Alcohol use disorder
Illicit drug use disorder
18
JSM 2018Slide19
19
JSM 2018
3. Summary of significant subdomain comparisons by sample sizeSlide20
3. Summary of Differences by Sample Size
20
JSM 2018Slide21
3. Summary of Differences by Sample Size
21
JSM 2018Slide22
22
JSM 2018
4. Comparisons between subdomain and overall measuresSlide23
4. Proportion of Comparisons that would Change Significance between Subsample 1 and FE Sample
23
JSM 2018Slide24
24
JSM 2018
SummarySlide25
Does an ABS Frame Introduce Coverage Bias?
Overall
differences by sample7 of 30 comparisons were significant, but the differences were small.Summary of differences by
measureVariables differed on the frequency and size of change across frames.No pattern or consistency across framesSummary of differences by sample size
Large samples drove many of the significant findings.
Summary of differences in
conclusions
drawn
Only 3% of conclusions changed, possibly due to chance
25
JSM 2018Slide26
Final Take-Away
An ABS frame has the potential to introduce coverage bias, but…
It will depend on the variable of interestIt will depend on desired precision and sample size
The magnitude of the bias will varySubstantive conclusions in bivariate analyses are unlikely to be affected
26
JSM 2018Slide27
Limitations
Identified differences are worst case scenarios.
Large sample sizes increase number of significant differences.All differences were attributed to error in the ABS frame.Areas with known coverage problems would be enumerated, in practice.The simulations are imperfect.
These findings are limited to health indicators.
27
JSM 2018Slide28
More Information
Ashley Amaya
Research Survey Methodologist202.728.2486aamaya@rti.org
JSM 2018
28Slide29
Works Cited
American Association for Public Opinion Research. (2016).
Address-based sampling (prepared for AAPOR Council by the Task Force on Address-Based Sampling; R. Harter, Chair). Oakbrook Terrace, IL: Author. Retrieved from http://www.aapor.org/Education-Resources/Reports/Address-based-Sampling.aspx
Dohrmann, S., Han, D., & Mohadjer, L. (2006). Residential address lists vs. traditional listing: Enumerating households and group quarters. In Proceedings of the 2006 Joint Statistical Meetings, American Statistical Association, Survey Research Methods Section, Seattle, WA
(pp. 2959-2964). Alexandria, VA: American Statistical Association.
Dohrmann, S., Han, D., & Mohadjer, L. (2007). Improving coverage of residential address lists in multistage area samples. In
Proceedings of the 2007 Joint Statistical Meetings, American Statistical Association, Section on Survey Research Methods, Salt Lake City, UT
(pp. 3219-3126). Alexandria, VA: American Statistical Association.
O'Muircheartaigh, C., English, N., & Eckman, S. (2007). Predicting the relative quality of alternative sampling frames. In
Proceedings of the 2007 Joint Statistical Meetings, American Statistical Association, Section on Survey Research Methods, Salt Lake City, UT
(pp. 3239-3248). Alexandria, VA: American Statistical Association
.
29
JSM 2018