Nick Inchausti SurveyMonkey Mingnan Liu Facebook Response Rates 2 Response rates are a critical indicator of data quality for surveys and we as survey researchers are always trying to get to 100 response ID: 800143
Download The PPT/PDF document "Improving Survey Response Rates: The Eff..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Improving Survey Response Rates: The Effect of Embedded Questions in Web Survey Email Invitations
Nick
Inchausti
, SurveyMonkey
Mingnan
Liu, Facebook
Slide2Response Rates
2
Response rates are a critical indicator of data quality for surveys, and we as survey researchers are always trying to get to 100% response
For mixed-mode or web surveys, email invitations are widely used as the first point of contact to reach respondents
We wanted to find out what happens if you begin data collection in the initial survey invitation itself?
Slide3Literature Review
3
Email personalization sometimes boosts response rates
(
Heerwegh
, 2005
), but sometimes has no effect
(Porter & Whitcomb,
2003)
A white
background and simple header had higher response rates than other conditions (Whitcomb & Porter, 2004
)
Mentioning
the purpose of an email (requesting for survey participation) and the sponsor of the survey in the email subject line also had an impact on survey participation (Porter & Whitcomb, 2005
)
Several
other factors, including the length of the email, placement of the URL, and the estimated time of the survey, have also been explored in survey experiments in order to improve survey participation and response rate (
Kaplowitz
,
Lupi
, Couper, & Thorp, 2012;
Keusch
, 2012;
Trouteaud
, 2004).
Slide4Research Question
4
New product feature: ability to embed survey question within an email
Research Question: Do survey invitations that include an embedded question have higher response rates than standard invitations?
Click rate
Completion rate
Data quality check: comparison of responses to the first question
Slide5Embedded Question
5
Screenshot:
standard
survey invitation email
Screenshot:
embedded
survey invitation email
Slide6Experiment Design
6
Sample frame: group of SurveyMonkey customers who had agreed to participate in research projects and provided their email addresses
Data collected July 27 – August 8, 2016
Initial email invitation to complete a survey, follow-up reminder 4 days later
Random assignment into 1/2 conditions: standard and embedded first question
4333 valid emails for embedded condition
4347 valid emails for standard condition
13-question
survey, identical in each condition
Experience with SurveyMonkey
Satisfaction with the survey platform
Interest in additional features
Slide7Results – Improved email click rate
7
Higher click rate for embedded survey than the standard survey (32.0% vs. 26.2%)
Statistically significant p<.001
This means that respondents in the embedded condition were more likely to
click on the embedded question and start the survey than the respondents in the standard condition to click on the “Begin survey” button.
Slide8Results – Improved survey completion rate
8
Higher completion rate for the embedded survey than the standard survey (29.1% vs. 24.4%)
Statistically significant p<.001
Completion rate =
# complete
# sent
Slide9Results – Small negative effect on survey drop-out rate
9
The proportion of respondents who completed the survey divided by the number who clicked int
o the survey was slightly higher for those in the standard condition…
Which means that the embedded version had a slightly higher drop-out rate…
But this difference is not statistically significant (p=.07).
Valid email
Clicked
Completed
% Completed/Clicked
Embedded
4333
1388
1261
90.8%
Standard
4347
1141
1059
92.8%
Slide10Results – No effect on data quality
10
Do we get different responses when we ask a question embedded in an email vs. in the survey itself?
No.
The ratio
of the two NPS scores between the embedded and standard email invites was 0.98, suggesting the two responses to the first question were almost identical for the two conditions.
Summary of Results
11
Using an emailed survey invitation with the first question embedded:
Improves the email click rate
Improves the survey completion rate
Has only a small negative effect on the survey drop-out rate
Has no effect on data quality, in terms of responses to the first (embedded) question
Slide12Discussion
12
Overall, successful test for adding a new feature.
Additional advantage:
even if respondents drop out
of the
survey, their answers to the first question will be recorded in the embedded condition. In the standard condition, if respondents drop out before completing the first page, all data will be lost.
Future research opportunities:
Embedding more than one question
(or the whole survey
?) in the email itself
Experimenting with different survey lengths and question types
Will this work with a different population of respondents?
Slide13Thank you
13
Contact us at:
Nicki@surveymonkey.com
Mingnanliu@fb.com