Costs are Changingand Why Presented at AAPOR 2016 Austin TX May 13 2016 Updated 5232016 2 A unit of the Weldon Cooper Center for Public Service Thomas M Guterbock Center for Survey Research ID: 713339
Download Presentation The PPT/PDF document "How Cell Phone Interviewing" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
How Cell Phone InterviewingCosts are Changing—and Why
Presented at AAPOR 2016Austin, TXMay 13, 2016
[Updated 5/23/2016
]Slide2
2. . . A unit of the Weldon Cooper Center for Public Service
Thomas M. Guterbock
Center for Survey ResearchUniversity of Virginia
TomG@virginia.edu
Grant Benson
Survey Research Center
University of Michigangdbenson@umich.eduPaul LavrakasIndependent Consultantpjlavrakas@centurylink.net
David
Dutwin
SSRS
ddutwin@SSRS.com
Jenny Kelly
NORC
Kelly-Jenny@NORC.orgSlide3
Thanks to:Marketing Systems Group and ASDE Survey Sampler, for assisting us with recruitment of respondentsSpecial thanks to Trent Buskirk (M-S-G) and Michel Durocher (ASDE)University of Michigan SRC, for hosting the surveyGrant Benson, for managing survey fielding, data analysis assistance
3Slide4
4Overview
Study purposes: Have cost ratios declined?
What factors affect production ratios in dual-frame RDD surveys?
Trends in
cell phone survey costs
2015 Cell Phone Task Force survey
Comparison to similar surveys from 2010, 2013Comparison of cost ratios across yearsFactors that affect production ratesComponents of the production rate Call duration, effort & yield
Summary of
results, estimated cost examples
Whither cell phone survey costs?Slide5
Study purposesBuild upon, extend, and compare to 2010 Cell Phone Task Force survey and 2013 Cost SurveySample SROs more broadlyObtain greater detail on productionExamine distinct components of production Gain better understanding of factors that drive production ratios & their componentsOur main question: Are cost ratios of cell vs. landline calling trending downward further?
5Slide6
Cost per interview [CPI]: three componentsCPI is sum of three types of cost:Interviewer hoursIncluding interview time and non-interview time
Cost of incentives (plus any mailing costs)Cost of purchased sample phone numbers and pre-screening Our focus: ratio of interviewer hours per completeRatio of cell phone hours to landline hours
Interviewer hours captures most of the data collection labor costIncentive amount differences are a design decision rather than determined by the type of sample
Cost of samples is relatively low and similarSlide7
A closer look at interviewer hoursInterviewer Hours per Completion = HPC = hours/completes HPC = 1/CPHScreening and Recruiting Hours per Completion = SRHPCSRHPC = HPC ─ interview lengthSlide8
Four factors affect production ratesWorking number rateContact rateEligibility rate
Cooperation rateOf interest: differentials in these rates (CP/LL) on the same studyIn 2010, first three were all lower for cell phones!Result: lower yields, more hours of calling for cell phones
As noted: the production rate ratio drives the cost ratioThese rates are relevant in theory, but difficult to measure in practice
8Slide9
Why might the cost ratio be decreasing?Working number rateShift in working-number densities as households migrate to cell and CPODutwin shows strong trends in opposite directions for LL(↓) and CP (
↑)Working number rate now higher for CPContact rateMore likely to answer if CPO, and CPO share is increasingBut Dutwin shows increase in answering machine/no answer rates
Eligibility rate(Not clear how this might be changing)Cooperation rateDutwin’s
data show little change in rates of refusal on either LL or CP
In addition, SROs may have learned and trained on ways to achieve greater cell phone success
9Slide10
A look back at 2010Cell phone cost survey was part of 2010 AAPOR Cell Phone Task Force ReportCost section by Guterbock, Lavrakas, Tompson and zuWallackData collection: Winter 2009-2010Convenience sample of 8 SRO’s
4 academic, 4 private sector26 usable surveys reported10Slide11
Productivity Statistics and Cost Ratios for Dual Frame RDD Surveys
Screening and Recruiting Hours per Completion
Hours per Completion
Overall Cost per Interview
Ratio (cell/landline)
SRHPC Ratio
HPC Ratio
Cost Ratio
Mean
2.53
2.00
2.05
Minimum
1.21
1.17
1.35
Maximum
5.37
3.47
3.97
N
26
26
20
Std. deviation
1.02
.63
.77
A look back at 2010
Source: 2010 survey for AAPOR Task ForceSlide12
The 2013 Cost SurveyA dual-frame survey, too!Member organizations of AASROAll AAPOR cell phone session presenters, 2010-2012112 individuals in frame98 organizations
Web survey in Qualtrics, March-April 2013allowed reporting on up to 3 recent surveysResponses from 27 organizations (RR3 = 42%)Usable data on 37 dual-frame surveys
12Slide13
The 2015 TF Cost SurveyBroader sampling planMember organizations of AASROAll AAPOR cell phone session presenters, 2013-2015
Outreach to cell phone clients by sample vendors M-S-G and ADPE Survey SamplerWeb survey in Qualtrics, Sept – Oct 2015allowed reporting on up to 3 recent surveys
72 individuals responded, some had no data22 firms gave us data on 53 studies3 surveys: data on 114 studies in all, 2010-15
13Slide14
Thanks to 2015 participating SROs:Abt SRBI, Inc.Castleton College Polling InstituteElon
University PollIPSOS Public AffairsMarist College PollNORC at the University of ChicagoPenn State Harrisburg Center for Survey Research
Princeton Survey Research AssociatesPublic Opinion StrategiesRTIRutgers University Eagleton Center for Public Interest Polling
SSI
SSRS
South Dakota Department of Health
University of Nebraska-Lincoln Bureau of Sociological Research University of Northern Iowa Center for Social & Behavioral ResearchUniversity of Oklahoma Public Opinion Learning LaboratoryUniversity of Virginia Center for Survey ResearchUniversity of Waterloo Survey Research Centre University of Wisconsin Survey CenterU.S.Food and Drug Administrationplus . . . Four anonymous contributors14
Commercial firms in
blueSlide15
We asked about . . .Main survey topicSample geographyInterview length (for CP and LL)Screened or “overlap” design?Eligibility rate among the screened landline sample casesUse of advance mailings and gift card incentivesHow LL numbers were dialed
Use of “enhanced” cell phone sampleProduction details . . .15Slide16
Two formats for reporting production data:16
Average interview length (in minutes)
Method of telephone dialing (manual, one-at-a-time autodial, or predictive dialer?)
Number of phone numbers attempted
Total dialings for the numbers attempted
Number of completes
Total interviewing hours
Completes per hour (CPH)
[divide
n
of completes by interviewing hours]
Landline RDD
Cell Phone
RDD*
XXX
If you prefer, instead of reporting the numbers in the shaded boxes, you can just report the CPH ratio, as calculated below:CPH Ratio = (Cellphone CPH) / (Landline CPH) = ______________Detailed format (38 surveys):Alternative format (17 surveys):Slide17
Cell Phone Sample Share is Increasing17Slide18
Cash incentives for cell phone: now rare18Slide19
Hours per Completion, 2015
Mean HPCratio (CP/LL)
= 1.39
Ratio reported for 53 surveys; hours reported for 35 surveys providing detail
Landline
m
edian (reported)
Cell
(projected)Slide20
Screening and Recruitment HPC, 2015
Mean SRHPCratio (CP/LL)
= 1.53
Ratio reported for 53 surveys; hours reported for 35 surveys providing detail
Landline
m
edian (reported)
Cell
(projected)Slide21
Productivity Statistics for Dual Frame RDD Surveys
Screening and Recruiting Hours per Completion
Hours per Completion
Ratio (cell/landline)
SRHPC Ratio
HPC Ratio
2010
2013
2015
2010
2013
2015
Mean
2.5
1.7**
1.5
2.0
1.5**
1.4
Minimum
1.2
.77
.43
1.2
.85
.45
Maximum
5.4
3.7
3.8
3.5
2.9
3.3
N
27
37
50
27
38
53
Std. deviation
1.0
.71
.80
.63
.51
.64
2015
compared to
prior years
**2013 & 2015
significantly different from
2010.
2015 not significantly different from 2013.
Outliers (
hpcratio
> 4.0) excludedSlide22
HPC Ratios Compared22Slide23
SRHPC Ratios Compared23Slide24
Factors that might affect the ratioPredictive dialing of landlines (25 surveys*)Increase SRHPC ratio by making landline calling fasterGeography (26 national surveys)Expect lower ratio with national studiesBRFSS studies (10 surveys)Used modified cell-only design; required 15 attempts for landline (but not for cell phone); used out-of-state cases
Enhanced cell samples showing activity (31 surveys)Fewer bad cell phone numbers, lower SHRPC ratioMonetary or gift card incentives for cell phone (17)Lower SHRPC ratio by increasing cell phone cooperation
24
[*Based on 91 surveys reported in 2014 & 2015]Slide25
Log of SRHPC is well distributedSRHPC ratioLog of SRHPC ratio25
[*Based on 114 usable surveys reported in 2010, 2014 & 2015]Slide26
Factors affecting SRHPCLog: SRHPC
Factor:Cell Phone
Land Line
Ratio
(CP/LL)
LL Predictive Dialer
-.259-.546**.407**
National sample
.275
.
309*
-.089
BRFSS study
.172
.
209*
-.076
Enhanced cell sample
-.172
.
217
+
-.
463**
CP Gift
card
-.069
-.118
.074
R
2.070.434.59526SRHPC = Screening and recruitment hours per completionOLS Standardized CoefficientsN = 60 Slide27
Predictive dialing speeds LL calling: higher ratios27Slide28
Enhanced cell samples more efficient: lower ratios28Slide29
3 components of productionAverage Call Duration = Screening and Recruitment Hours [SRH] / number of dialingsAverage time spent on a non-interview call attemptEffort =
Dialings / number of phone numbers attemptedYield = Completes / number of phone numbers attemptedThese components are directly measurable!We can also calculate a CP
/LL ratio for each component.Slide30
Duration of non-interview calling, 2015
Mean Duration ratio (CP/LL) =
1.57
n = 29 surveys
Landline
m
edian (reported)
Cell
(projected)Slide31
Effort (dialings per number), 2015Mean
Effortratio (CP/LL) = 1.03
n = 29 surveys
Landline
m
edian
(reported
)
Cell
(projected)Slide32
Yield (completes per number), 2015Median Yield
ratio (CP/LL) = 1.09
About 1 in 20
About 1 in 15
Completing
about 1 in 14
n = 32 surveys
Landline
m
edian
(reported)
Cell
(projected)Slide33
Factors affecting effortLog: Effort
Factor:Cell Phone
Land Line
Ratio
(CP/LL)
LL Predictive Dialer
-.590**-.038-.462**
National sample
.141
-.
451**
.
632**
BRFSS study
.030
.205
-.
206
+
Enhanced cell sample
.
318*
.
261*
-.024
CP Gift
card
.314
.060
.201
R2.374.466.35433Effort = Dialings/attempted numberOLS Standardized CoefficientsN = 54 Slide34
Factors affecting yieldLog: Yield
Factor:Cell Phone
Land Line
Ratio
(CP/LL)
LL Predictive Dialer
-.423**-.551**
.
318*
National sample
-.156
-.
292*
.247
BRFSS study
.174
.
208*
-.103
Enhanced cell sample
.
430**
-.004
.
497**
CP Gift
card
.
367**
.083
.299R2.596.672.44134Yield = Completes/number attemptedOLS Standardized CoefficientsN = 54 Slide35
3 components SRHPCProduct of Duration times Effort divided by Yield equals SRHPC:
35
(Duration)
(Effort)
(1/Yield)
SRH
––––––––––Dialings
Dialings
––––––––––
Numbers
Numbers
––––––––––
CompletesSlide36
3 components SRHPCProduct of Dialtime times Effort divided by Yield equals SRHPC:
36
(Duration)
(Effort)
(1/Yield)
SRH
––––––––––Dialings
Dialings
––––––––––
Numbers
Numbers
––––––––––
Completes
(SRHPC)
SRH
Completes
––––––––––Slide37
3 components SRHPCProduct of Dialtime times Effort divided by Yield equals SRHPC:
37
(Duration)
(Effort)
(1/Yield)
SRH
––––––––––Dialings
Dialings
––––––––––
Numbers
Numbers
––––––––––
Completes
(SRHPC)
SRH
Completes
––––––––––
Therefore:
Log
Duration
+
Log Effort
─
Log Yield =
Log SRHPCSlide38
3 logged components account for all the variance in logged SRHPC ratio
Log
Duration
Ratio
Log
SRHPC
ratio
Log
Effort
Ratio
Log
Yield
Ratio
Log
HPC
ratio
0.994
R
2
= 0.998
R
2
=
0.989
-
1.314
**
1.081
**
1.400
**Updated for 2015Slide39
Main effects of Predictive Dialing
Log
Duration
Ratio
Log
Effort
Ratio
Log
Yield
Ratio
Log
HPC
ratio
R
2
=
0.989
Log
SRHPC
ratio
R
2
=
0.998
0.99
Predictive Dialing of Landlines
1.027
**
0.318*
-
1.3**1.1**1.4**-0.462**Updated for 2015Slide40
Main effect of National Geography
Log
Duration
Ratio
Log
Effort
Ratio
Log
Yield
Ratio
Log
HPC
ratio
R
2
=
0.989
Log
SRHPC
ratio
R
2
=
0.998
0.99
National
Geography
.632**
-
1.3
**1.1**1.4**Updated for 2015Slide41
Main Effect of BRFSS Study Design
Log
Duration
Ratio
Log
Effort
Ratio
Log
Yield
Ratio
Log
HPC
ratio
R
2
=
0.989
Log
SRHPC
ratio
R
2
=
0.998
0.99
BRFSS Study
-
0.206
+
-
1.3**1.1**1.4**Updated for 2015Slide42
Main Effect of Enhanced Cell Sample
Log
Duration
Ratio
Log
Effort
Ratio
Log
Yield
Ratio
Log
HPC
ratio
R
2
=
0.989
Log
SRHPC
ratio
R
2
=
0.998
0.99
Enhanced Cell Phone Sample
0.497**
-
1.3
**
1.1**1.4**Slide43
Main Effect of Gift Cards
Log
Duration
Ratio
Log
Effort
Ratio
Log
Yield
Ratio
Log
HPC
ratio
R
2
=
0.989
Log
SRHPC
ratio
R
2
=
0.998
0.99
Gift Card
for Cell
0.299
*
-
1.3**1.1**1.4**Updated for 2015Slide44
Putting it all together . . .
Log
Duration
Ratio
Log
Effort
Ratio
Log
Yield
Ratio
Log
HPC
ratio
-
1.314
**
R
2
=
0.988
Log
SRHPC
ratio
R
2
=
0.998
0.994
R
2
= 0.62R2 = 0.35R2 = 0.441.081**1.400**Predictive Dialing of LandlinesNational GeographyBRFSSStudyEnhanced Cell Phone SampleGift Cardfor Cell1.027**0.318+-0.206+0.497**0.299*-0.462*0.632**Updated for 2015Slide45
5 factors predict 60% of variance:
Predictive Dialing of Landlines
National
Geography
BRFSS
study
Enhanced Phone Sample
Gift Card
for Cell Phone
Log SRHPC ratio
0.407
**
-
0.089
-
0.076
-
0.463
**
0.074
R
2
=
0.60
(Really just two factors!)Slide46
46
n of surveys
Numbers attempted
Cost per number
Sample cost per complete
HPC
HPC Cost ($30/hr)
Total Cost per Interview
All CP
56
18
0.09
$
1.62
1.51
$45.30
$
46.92
All LL
56
14
0.07
$
0.98
1.16
$34.80
$
35.78
Conventional CP sample
38240.07$1.681.54$46.20$47.88Enhanced CP sample18130.11$1.541.44$43.20$44.74Predictive dial LL23310.07
$
2.17
0.94
$28.20
$
30.37
Regular dial LL
45
13
0.07
$0.91
1.46
$43.80
$44.71
So, what’s the cost?
Assumed:
no cash incentivesSlide47
Summary of findingsMean HPC and SRHPC ratios are down significantly from the 2010 resultsRatio of Hours Per Completion has decreased from 2.0 to 1.5 to 1.4Therefore, cost ratios are also downVariance in production ratios has lessened significantlySurvey shops have learned how to routinize dual-frame phone workSlide48
Summary of findings (cont.)Variation in cost ratios is driven primarily by:Predictive dialing of landlinesincreases efficiency on landline callingraises the cell/landline ratio for duration, SRHPCunder threat from TCPA regulatory climate
Enhanced cell phone sampleincreases efficiency of cell phone callingraises the cell/landline ratio for yield, SRHPCNot uncommon in 2015 for studies using enhanced sample to show SRHPC < 1.0 That is: Cell frame more productive than landline frameSlide49
Whither Cell Phone interviewing costs?Enhanced cell samples are changing the cost picture significantlyProduction rates for enhanced cell are often equal to or better than landline RDD production ratesThese cell-frame refining methods are being widely adoptedAs cost ratio decreases, allocation of sample to the cell phone frame can be further increased decreasing design effects in dual-frame studies
Trend to larger cell allocations will continueA cell-only (single) frame design is increasingly possible, as cell phone production rates improveSee Peytchev et al (2013)
49Slide50
Innovation and research continuing . . .Future experiments on cell phone calling productivity should give attention to the three separate components of productivityCall duration, effort, and yieldNew variations on screening designs are sure to emergeNote that BRFSS screens for CPO or
cell-mostly (90% or more cell calls)Dual-frame surveys are becoming routine; we are getting more efficient at doing them; CP and LL costs are equalizingBut: the pace of change in telephony and innovation in survey methods is not slowing down!
50Slide51
ReferencesAAPOR 2010. New Considerations for Survey Researchers When Planning and Conducting RDD Telephone Surveys in the U.S. with Respondents Reached via Cell Phone Numbers. AAPOR Cell Phone Task Force Report. Paul J. Lavrakas (task force chair), and 20 others. American Association for Public Opinion Research, June. Available at: www.aapor.org
.Guterbock, Thomas M., Paul J. Lavrakas, Trevor N. Tompson, and Randal ZuWallack, 2010. “The Variable Costs of Cell Phone Interviewing: Understanding Cost and Productivity Ratios in Dual-Frame Telephone Surveys.” AAPOR paper, Chicago.Guterbock, Thomas M., Paul J. Lavrakas, Trevor N.
Tompson, and Randal ZuWallac, 2011. “Cost and Productivity Ratios in Dual-Frame RDD Telephone Surveys.”
Survey Practice
, April:
www.surveypractice.org
.Guterbock, Thomas M., Robin A. Bebel, John Lee P. Holmes, and Peter A. Furia, 2012. “Why We No Longer Need Cell Phone Incentives.” AAPOR paper, Orlando, May.Guterbock, Thomas M., Andy Peytchev, and Deborah L. Rexrode. 2013. “Cell Phone Costs Revisited: Understanding Cost and Productivity Ratios in Dual-Frame Telephone Surveys.” AAPOR paper, Boston, May.Peytchev, Andy and Neely, Benjamin, 2013. “RDD Telephone Surveys: Toward a Single Frame Cell Phone Design.” Public Opinion Quarterly 77 (Spring): 283-304.51Slide52
How Cell Phone InterviewingCosts are Changing—and Why
Presented at AAPOR 2016Austin, TX
May 13, 2016
Thomas M. Guterbock
TomG@virginia.edu
Grant Benson
GDBenson@UMich.eduPaul Lavrakaspjlavrakas@centurylink.net
David
Dutwin
ddutwin@srss.com
Jenny Kelly
k
elly-jenny@norc.com
[Updated 5/23/2016
]