Pretesting items ensures validity 2 Set pretest goals 3 Outline Methods to test new survey items Expert reviews Cognitive interviews Focus groups Field testing 4 Outline ID: 693460
Download Presentation The PPT/PDF document "Module 4 presentation Pretesting survey ..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Module 4 presentation
Pretesting survey itemsSlide2
Pretesting items
ensures
validity.
2Slide3
Set
pretest goals.
3Slide4
OutlineMethods to test new survey items:Expert reviews.
Cognitive
interviews.
Focus groups.
Field testing.4Slide5
OutlineMethods to test new survey items:Expert reviews.
Cognitive
interviews.
Focus
groups.Field testing.
5Slide6
Expert reviews6Slide7
Who
are the experts in the area you want to study?
7Slide8
OutlineMethods to test new survey items:Expert
reviews.
Cognitive
interviews.
Focus groups.Field testing.
8Slide9
Cognitive interviews9
1. Planning for a Survey
9Slide10
Cognitive interviews: Purpose
A cognitive interview is an
appraisal
of the survey questions
.
10Slide11
Cognitive interviews: PurposeA cognitive interview is
not
a test.
11Slide12
Cognitive interviews: Schedule
12Slide13
13
Structure
your cognitive interview.Slide14
Identify cognitive
interview
respondents
.
14Slide15
Cognitive interviewerWho
should
do the interviews?
15Slide16
Interview approachesConcurrent “think-aloud”Retrospective probingSlide17
Concurrent “think-aloud
”
Ask participants to
verbalize
what they
are
thinking
.
17Slide18
Retrospective probing
Ask specific questions
after
task completion
.18Slide19
“Think-aloud” advantages and disadvantages
19Slide20
“Retrospective probing” advantages and disadvantages
20Slide21
Types of probesScripted
and
spontaneous
21Slide22
Cognitive interview
example probes
What does the word
equivalent
mean to you
?
In
your own
words,
what
is this question
asking you to do
?
Can
you tell me how sure you are that
you. . .?
22Slide23
Types of
probes
How
do you
remember. . .?
Why do you think this question was
asked?
Was it
easy or hard to answer this question
?
23Slide24
Cognitive interview probes
Was this question easy or hard for you to answer?
It was easy!
24Slide25
Conduct
the
interview
.
25Slide26
Review
notes
and
document
findings.
26Slide27
Cognitive interviews: AnalysisReview
notes and
listen
to audio recordings of interviews.
27Slide28
Cognitive interviews: Example 1
28Slide29
Cognitive interviews: Example 1
29Slide30
Cognitive interviews: Example 2
30Slide31
Cognitive interviews: Example 2
31Slide32
OutlineMethods to test new survey items:Expert
reviews.
Cognitive
interviews.
Focus groups.Field testing.
32Slide33
Focus groups
1. Planning for a Survey
33Slide34
Ask participants to comment on survey items as a group
.
34Slide35
OutlineMethods to test new survey items:Expert
reviews.
Cognitive
interviews.
Focus groups.
Field testing.
35Slide36
Field testing
36Slide37
Activity 3:
Conducting cognitive interviews
37Slide38
Additional resourcesFor more information, please visit the following websites:The American Association for Public Opinion Research:
http://www.aapor.org
/
.
Public Opinion Quarterly: https://poq.oxfordjournals.org/.
Journal of Survey Statistics and Methodology: http://jssam.oxfordjournals.org/content/current
.Slide39
ReferencesJabine, T. B., Straf, M. L., Tanur, J. M., & Tourangeau, R. (1984).
Cognitive aspects of survey
methodology
: Building a bridge between disciplines. Washington, DC: The National Academies Press. Retrieved
April 1, 2016, from http://www.nap.edu/catalog/930/cognitive-aspects-of-survey-methodology-building-a-bridge-between-disciplines?_sm_au_=iVVjJQZpVk2tF7q.Lavrakas, P. J. (2008).
Encyclopedia of survey research methods. Thousand Oaks, CA: SAGE.Willis, G. B. (1999). Cognitive interviewing: A “how-to” guide
.
Rockville, MD
: Research Triangle Institute. Retrieved
April 1, 2016, from
http
://
www.appliedresearch.cancer.gov/areas/cognitive/interview.pdf
.
Willis, G. B. (2005).
Cognitive interviewing: A tool for improving questionnaire
d
esign
. Thousand Oaks, CA: SAGE.
39