How Cell Phone Interviewing - PowerPoint Presentation

Presentation on theme: "How Cell Phone Interviewing"— Presentation transcript:

Slide1

How Cell Phone InterviewingCosts are Changing—and Why

Presented at AAPOR 2016Austin, TXMay 13, 2016

[Updated 5/23/2016

]Slide2

2. . . A unit of the Weldon Cooper Center for Public Service

Thomas M. Guterbock

Center for Survey ResearchUniversity of Virginia

TomG@virginia.edu

Grant Benson

Survey Research Center

University of Michigangdbenson@umich.eduPaul LavrakasIndependent Consultantpjlavrakas@centurylink.net

David

Dutwin

SSRS

ddutwin@SSRS.com

Jenny Kelly

NORC

Kelly-Jenny@NORC.orgSlide3

Thanks to:Marketing Systems Group and ASDE Survey Sampler, for assisting us with recruitment of respondentsSpecial thanks to Trent Buskirk (M-S-G) and Michel Durocher (ASDE)University of Michigan SRC, for hosting the surveyGrant Benson, for managing survey fielding, data analysis assistance

3Slide4

4Overview

Study purposes: Have cost ratios declined?

What factors affect production ratios in dual-frame RDD surveys?

Trends in

cell phone survey costs

2015 Cell Phone Task Force survey

Comparison to similar surveys from 2010, 2013Comparison of cost ratios across yearsFactors that affect production ratesComponents of the production rate Call duration, effort & yield

Summary of

results, estimated cost examples

Whither cell phone survey costs?Slide5

Study purposesBuild upon, extend, and compare to 2010 Cell Phone Task Force survey and 2013 Cost SurveySample SROs more broadlyObtain greater detail on productionExamine distinct components of production Gain better understanding of factors that drive production ratios & their componentsOur main question: Are cost ratios of cell vs. landline calling trending downward further?

5Slide6

Cost per interview [CPI]: three componentsCPI is sum of three types of cost:Interviewer hoursIncluding interview time and non-interview time

Cost of incentives (plus any mailing costs)Cost of purchased sample phone numbers and pre-screening Our focus: ratio of interviewer hours per completeRatio of cell phone hours to landline hours

Interviewer hours captures most of the data collection labor costIncentive amount differences are a design decision rather than determined by the type of sample

Cost of samples is relatively low and similarSlide7

A closer look at interviewer hoursInterviewer Hours per Completion = HPC = hours/completes HPC = 1/CPHScreening and Recruiting Hours per Completion = SRHPCSRHPC = HPC ─ interview lengthSlide8

Four factors affect production ratesWorking number rateContact rateEligibility rate

Cooperation rateOf interest: differentials in these rates (CP/LL) on the same studyIn 2010, first three were all lower for cell phones!Result: lower yields, more hours of calling for cell phones

As noted: the production rate ratio drives the cost ratioThese rates are relevant in theory, but difficult to measure in practice

8Slide9

Why might the cost ratio be decreasing?Working number rateShift in working-number densities as households migrate to cell and CPODutwin shows strong trends in opposite directions for LL(↓) and CP (

↑)Working number rate now higher for CPContact rateMore likely to answer if CPO, and CPO share is increasingBut Dutwin shows increase in answering machine/no answer rates

Eligibility rate(Not clear how this might be changing)Cooperation rateDutwin’s

data show little change in rates of refusal on either LL or CP

In addition, SROs may have learned and trained on ways to achieve greater cell phone success

9Slide10

A look back at 2010Cell phone cost survey was part of 2010 AAPOR Cell Phone Task Force ReportCost section by Guterbock, Lavrakas, Tompson and zuWallackData collection: Winter 2009-2010Convenience sample of 8 SRO’s

4 academic, 4 private sector26 usable surveys reported10Slide11

Productivity Statistics and Cost Ratios for Dual Frame RDD Surveys

Screening and Recruiting Hours per Completion

Hours per Completion

Overall Cost per Interview

Ratio (cell/landline)

SRHPC Ratio

HPC Ratio

Cost Ratio

Mean

2.53

2.00

2.05

Minimum

1.21

1.17

1.35

Maximum

5.37

3.47

3.97

N

26

26

20

Std. deviation

1.02

.63

.77

A look back at 2010

Source: 2010 survey for AAPOR Task ForceSlide12

The 2013 Cost SurveyA dual-frame survey, too!Member organizations of AASROAll AAPOR cell phone session presenters, 2010-2012112 individuals in frame98 organizations

Web survey in Qualtrics, March-April 2013allowed reporting on up to 3 recent surveysResponses from 27 organizations (RR3 = 42%)Usable data on 37 dual-frame surveys

12Slide13

The 2015 TF Cost SurveyBroader sampling planMember organizations of AASROAll AAPOR cell phone session presenters, 2013-2015

Outreach to cell phone clients by sample vendors M-S-G and ADPE Survey SamplerWeb survey in Qualtrics, Sept – Oct 2015allowed reporting on up to 3 recent surveys

72 individuals responded, some had no data22 firms gave us data on 53 studies3 surveys: data on 114 studies in all, 2010-15

13Slide14

Thanks to 2015 participating SROs:Abt SRBI, Inc.Castleton College Polling InstituteElon

University PollIPSOS Public AffairsMarist College PollNORC at the University of ChicagoPenn State Harrisburg Center for Survey Research

Princeton Survey Research AssociatesPublic Opinion StrategiesRTIRutgers University Eagleton Center for Public Interest Polling

SSI

SSRS

South Dakota Department of Health

University of Nebraska-Lincoln Bureau of Sociological Research University of Northern Iowa Center for Social & Behavioral ResearchUniversity of Oklahoma Public Opinion Learning LaboratoryUniversity of Virginia Center for Survey ResearchUniversity of Waterloo Survey Research Centre University of Wisconsin Survey CenterU.S.Food and Drug Administrationplus . . . Four anonymous contributors14

Commercial firms in

blueSlide15

We asked about . . .Main survey topicSample geographyInterview length (for CP and LL)Screened or “overlap” design?Eligibility rate among the screened landline sample casesUse of advance mailings and gift card incentivesHow LL numbers were dialed

Use of “enhanced” cell phone sampleProduction details . . .15Slide16

Two formats for reporting production data:16

Average interview length (in minutes)

Method of telephone dialing (manual, one-at-a-time autodial, or predictive dialer?)

Number of phone numbers attempted

Total dialings for the numbers attempted

Number of completes

Total interviewing hours

Completes per hour (CPH)

[divide

n

of completes by interviewing hours]

Landline RDD

Cell Phone

RDD*

XXX

If you prefer, instead of reporting the numbers in the shaded boxes, you can just report the CPH ratio, as calculated below:CPH Ratio = (Cellphone CPH) / (Landline CPH) = ______________Detailed format (38 surveys):Alternative format (17 surveys):Slide17

Cell Phone Sample Share is Increasing17Slide18

Cash incentives for cell phone: now rare18Slide19

Hours per Completion, 2015

Mean HPCratio (CP/LL)

= 1.39

Ratio reported for 53 surveys; hours reported for 35 surveys providing detail

Landline

m

edian (reported)

Cell

(projected)Slide20

Screening and Recruitment HPC, 2015

Mean SRHPCratio (CP/LL)

= 1.53

Ratio reported for 53 surveys; hours reported for 35 surveys providing detail

Landline

m

edian (reported)

Cell

(projected)Slide21

Productivity Statistics for Dual Frame RDD Surveys

Screening and Recruiting Hours per Completion

Hours per Completion

Ratio (cell/landline)

SRHPC Ratio

HPC Ratio

2010

2013

2015

2010

2013

2015

Mean

2.5

1.7**

1.5

2.0

1.5**

1.4

Minimum

1.2

.77

.43

1.2

.85

.45

Maximum

5.4

3.7

3.8

3.5

2.9

3.3

N

27

37

50

27

38

53

Std. deviation

1.0

.71

.80

.63

.51

.64

2015

compared to

prior years

**2013 & 2015

significantly different from

2010.

2015 not significantly different from 2013.

Outliers (

hpcratio

> 4.0) excludedSlide22

HPC Ratios Compared22Slide23

SRHPC Ratios Compared23Slide24

Factors that might affect the ratioPredictive dialing of landlines (25 surveys*)Increase SRHPC ratio by making landline calling fasterGeography (26 national surveys)Expect lower ratio with national studiesBRFSS studies (10 surveys)Used modified cell-only design; required 15 attempts for landline (but not for cell phone); used out-of-state cases

Enhanced cell samples showing activity (31 surveys)Fewer bad cell phone numbers, lower SHRPC ratioMonetary or gift card incentives for cell phone (17)Lower SHRPC ratio by increasing cell phone cooperation

24

[*Based on 91 surveys reported in 2014 & 2015]Slide25

Log of SRHPC is well distributedSRHPC ratioLog of SRHPC ratio25

[*Based on 114 usable surveys reported in 2010, 2014 & 2015]Slide26

Factors affecting SRHPCLog: SRHPC

Factor:Cell Phone

Land Line

Ratio

(CP/LL)

LL Predictive Dialer

-.259-.546**.407**

National sample

.275

.

309*

-.089

BRFSS study

.172

.

209*

-.076

Enhanced cell sample

-.172

.

217

+

-.

463**

CP Gift

card

-.069

-.118

.074

R

2.070.434.59526SRHPC = Screening and recruitment hours per completionOLS Standardized CoefficientsN = 60 Slide27

Predictive dialing speeds LL calling: higher ratios27Slide28

Enhanced cell samples more efficient: lower ratios28Slide29

3 components of productionAverage Call Duration = Screening and Recruitment Hours [SRH] / number of dialingsAverage time spent on a non-interview call attemptEffort =

Dialings / number of phone numbers attemptedYield = Completes / number of phone numbers attemptedThese components are directly measurable!We can also calculate a CP

/LL ratio for each component.Slide30

Duration of non-interview calling, 2015

Mean Duration ratio (CP/LL) =

1.57

n = 29 surveys

Landline

m

edian (reported)

Cell

(projected)Slide31

Effort (dialings per number), 2015Mean

Effortratio (CP/LL) = 1.03

n = 29 surveys

Landline

m

edian

(reported

)

Cell

(projected)Slide32

Yield (completes per number), 2015Median Yield

ratio (CP/LL) = 1.09

About 1 in 20

About 1 in 15

Completing

about 1 in 14

n = 32 surveys

Landline

m

edian

(reported)

Cell

(projected)Slide33

Factors affecting effortLog: Effort

Factor:Cell Phone

Land Line

Ratio

(CP/LL)

LL Predictive Dialer

-.590**-.038-.462**

National sample

.141

-.

451**

.

632**

BRFSS study

.030

.205

-.

206

+

Enhanced cell sample

.

318*

.

261*

-.024

CP Gift

card

.314

.060

.201

R2.374.466.35433Effort = Dialings/attempted numberOLS Standardized CoefficientsN = 54 Slide34

Factors affecting yieldLog: Yield

Factor:Cell Phone

Land Line

Ratio

(CP/LL)

LL Predictive Dialer

-.423**-.551**

.

318*

National sample

-.156

-.

292*

.247

BRFSS study

.174

.

208*

-.103

Enhanced cell sample

.

430**

-.004

.

497**

CP Gift

card

.

367**

.083

.299R2.596.672.44134Yield = Completes/number attemptedOLS Standardized CoefficientsN = 54 Slide35

3 components  SRHPCProduct of Duration times Effort divided by Yield equals SRHPC:

35

(Duration)

(Effort)

(1/Yield)

SRH

––––––––––Dialings

Dialings

––––––––––

Numbers

Numbers

––––––––––

CompletesSlide36

3 components  SRHPCProduct of Dialtime times Effort divided by Yield equals SRHPC:

36

(Duration)

(Effort)

(1/Yield)

SRH

––––––––––Dialings

Dialings

––––––––––

Numbers

Numbers

––––––––––

Completes

(SRHPC)

SRH

Completes

––––––––––Slide37

3 components  SRHPCProduct of Dialtime times Effort divided by Yield equals SRHPC:

37

(Duration)

(Effort)

(1/Yield)

SRH

––––––––––Dialings

Dialings

––––––––––

Numbers

Numbers

––––––––––

Completes

(SRHPC)

SRH

Completes

––––––––––

Therefore:

Log

Duration

+

Log Effort

─

Log Yield =

Log SRHPCSlide38

3 logged components account for all the variance in logged SRHPC ratio

Log

Duration

Ratio

Log

SRHPC

ratio

Log

Effort

Ratio

Log

Yield

Ratio

Log

HPC

ratio

0.994

R

2

= 0.998

R

2

=

0.989

-

1.314

**

1.081

**

1.400

**Updated for 2015Slide39

Main effects of Predictive Dialing

Log

Duration

Ratio

Log

Effort

Ratio

Log

Yield

Ratio

Log

HPC

ratio

R

2

=

0.989

Log

SRHPC

ratio

R

2

=

0.998

0.99

Predictive Dialing of Landlines

1.027

**

0.318*

-

1.3**1.1**1.4**-0.462**Updated for 2015Slide40

Main effect of National Geography

Log

Duration

Ratio

Log

Effort

Ratio

Log

Yield

Ratio

Log

HPC

ratio

R

2

=

0.989

Log

SRHPC

ratio

R

2

=

0.998

0.99

National

Geography

.632**

-

1.3

**1.1**1.4**Updated for 2015Slide41

Main Effect of BRFSS Study Design

Log

Duration

Ratio

Log

Effort

Ratio

Log

Yield

Ratio

Log

HPC

ratio

R

2

=

0.989

Log

SRHPC

ratio

R

2

=

0.998

0.99

BRFSS Study

-

0.206

+

-

1.3**1.1**1.4**Updated for 2015Slide42

Main Effect of Enhanced Cell Sample

Log

Duration

Ratio

Log

Effort

Ratio

Log

Yield

Ratio

Log

HPC

ratio

R

2

=

0.989

Log

SRHPC

ratio

R

2

=

0.998

0.99

Enhanced Cell Phone Sample

0.497**

-

1.3

**

1.1**1.4**Slide43

Main Effect of Gift Cards

Log

Duration

Ratio

Log

Effort

Ratio

Log

Yield

Ratio

Log

HPC

ratio

R

2

=

0.989

Log

SRHPC

ratio

R

2

=

0.998

0.99

Gift Card

for Cell

0.299

*

-

1.3**1.1**1.4**Updated for 2015Slide44

Putting it all together . . .

Log

Duration

Ratio

Log

Effort

Ratio

Log

Yield

Ratio

Log

HPC

ratio

-

1.314

**

R

2

=

0.988

Log

SRHPC

ratio

R

2

=

0.998

0.994

R

2

= 0.62R2 = 0.35R2 = 0.441.081**1.400**Predictive Dialing of LandlinesNational GeographyBRFSSStudyEnhanced Cell Phone SampleGift Cardfor Cell1.027**0.318+-0.206+0.497**0.299*-0.462*0.632**Updated for 2015Slide45

5 factors predict 60% of variance:

Predictive Dialing of Landlines

National

Geography

BRFSS

study

Enhanced Phone Sample

Gift Card

for Cell Phone

Log SRHPC ratio

0.407

**

-

0.089

-

0.076

-

0.463

**

0.074

R

2

=

0.60

(Really just two factors!)Slide46

46

n of surveys

Numbers attempted

Cost per number

Sample cost per complete

HPC

HPC Cost ($30/hr)

Total Cost per Interview

All CP

56

18

0.09

$

1.62

1.51

$45.30

$

46.92

All LL

56

14

0.07

$

0.98

1.16

$34.80

$

35.78

Conventional CP sample

38240.07$1.681.54$46.20$47.88Enhanced CP sample18130.11$1.541.44$43.20$44.74Predictive dial LL23310.07

$

2.17

0.94

$28.20

$

30.37

Regular dial LL

45

13

0.07

$0.91

1.46

$43.80

$44.71

So, what’s the cost?

Assumed:

no cash incentivesSlide47

Summary of findingsMean HPC and SRHPC ratios are down significantly from the 2010 resultsRatio of Hours Per Completion has decreased from 2.0 to 1.5 to 1.4Therefore, cost ratios are also downVariance in production ratios has lessened significantlySurvey shops have learned how to routinize dual-frame phone workSlide48

Summary of findings (cont.)Variation in cost ratios is driven primarily by:Predictive dialing of landlinesincreases efficiency on landline callingraises the cell/landline ratio for duration, SRHPCunder threat from TCPA regulatory climate

Enhanced cell phone sampleincreases efficiency of cell phone callingraises the cell/landline ratio for yield, SRHPCNot uncommon in 2015 for studies using enhanced sample to show SRHPC < 1.0 That is: Cell frame more productive than landline frameSlide49

Whither Cell Phone interviewing costs?Enhanced cell samples are changing the cost picture significantlyProduction rates for enhanced cell are often equal to or better than landline RDD production ratesThese cell-frame refining methods are being widely adoptedAs cost ratio decreases, allocation of sample to the cell phone frame can be further increased decreasing design effects in dual-frame studies

Trend to larger cell allocations will continueA cell-only (single) frame design is increasingly possible, as cell phone production rates improveSee Peytchev et al (2013)

49Slide50

Innovation and research continuing . . .Future experiments on cell phone calling productivity should give attention to the three separate components of productivityCall duration, effort, and yieldNew variations on screening designs are sure to emergeNote that BRFSS screens for CPO or

cell-mostly (90% or more cell calls)Dual-frame surveys are becoming routine; we are getting more efficient at doing them; CP and LL costs are equalizingBut: the pace of change in telephony and innovation in survey methods is not slowing down!

50Slide51

ReferencesAAPOR 2010. New Considerations for Survey Researchers When Planning and Conducting RDD Telephone Surveys in the U.S. with Respondents Reached via Cell Phone Numbers. AAPOR Cell Phone Task Force Report. Paul J. Lavrakas (task force chair), and 20 others. American Association for Public Opinion Research, June. Available at: www.aapor.org

.Guterbock, Thomas M., Paul J. Lavrakas, Trevor N. Tompson, and Randal ZuWallack, 2010. “The Variable Costs of Cell Phone Interviewing: Understanding Cost and Productivity Ratios in Dual-Frame Telephone Surveys.” AAPOR paper, Chicago.Guterbock, Thomas M., Paul J. Lavrakas, Trevor N.

Tompson, and Randal ZuWallac, 2011. “Cost and Productivity Ratios in Dual-Frame RDD Telephone Surveys.”

Survey Practice

, April:

www.surveypractice.org

.Guterbock, Thomas M., Robin A. Bebel, John Lee P. Holmes, and Peter A. Furia, 2012. “Why We No Longer Need Cell Phone Incentives.” AAPOR paper, Orlando, May.Guterbock, Thomas M., Andy Peytchev, and Deborah L. Rexrode. 2013. “Cell Phone Costs Revisited: Understanding Cost and Productivity Ratios in Dual-Frame Telephone Surveys.” AAPOR paper, Boston, May.Peytchev, Andy and Neely, Benjamin, 2013. “RDD Telephone Surveys: Toward a Single Frame Cell Phone Design.” Public Opinion Quarterly 77 (Spring): 283-304.51Slide52

How Cell Phone InterviewingCosts are Changing—and Why

Presented at AAPOR 2016Austin, TX

May 13, 2016

Thomas M. Guterbock

TomG@virginia.edu

Grant Benson

GDBenson@UMich.eduPaul Lavrakaspjlavrakas@centurylink.net

David

Dutwin

ddutwin@srss.com

Jenny Kelly

k

elly-jenny@norc.com

[Updated 5/23/2016

]