Problems that careful questionnaire design can alleviate Consistency effect respondents attempt to make their later answers fit the earlier ones Fatigue effect questions toward the end of the survey are not carefully answered ID: 274599
Download Presentation The PPT/PDF document "Developing survey questions" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Developing survey questionsSlide2
Problems that careful questionnaire design can alleviate
Consistency effect – respondents attempt to make their later answers fit the earlier ones
Fatigue effect – questions toward the end of the survey are not carefully answered
Redundancy effect – all questions in the same style are answered in the same waySlide3
Question types
There are two major types of questions
Closed-ended questions
Open-ended questionsSlide4
Closed-ended questions
The respondent is provided with a limited number of response options
Each response option has a numerical value or code it representsSlide5
Closed-ended questions
Examples:
Are you:
18-34 years of age (1)
35-54 years of age (2)
55+ years of age (3)Slide6
Closed-ended questions
If you were about to purchase a digital TV, which source of information, if any, would you consult first?
1. Magazines
2. Advertising
3. Salesperson
4. Product brochure
5. Friends
6. Other
7. Would not consult information sourceSlide7
Closed-ended questions
Please say whether you
Strongly agree (5),
agree somewhat (4),
neither agree nor disagree (3),
disagree somewhat (2)
or strongly disagree (1)
with the following statement:
“People who spend over 400 dollars on a cell phone are vain.”Slide8
Open-ended questions
The question is standardized, but the respondent answers in her own words.
The interviewer usually ‘probes’ at least once for expansion and/or specificity after the initial response.
Example:
“What features would the ideal science fiction movie have?”Slide9
Closed-ended v. open-ended
Closed-ended questions take less time to complete
Closed-ended questions take less time to enter into the database
Closed-ended questions allow the respondent to interpret her position in relation to the questionSlide10
Closed-ended v. open-ended
Open-ended questions allow the respondent to elaborate and to make connections you may not have anticipated
Answers to open-ended questions must be ‘coded’ in order to be interpreted in a quantitative manner
Respondents often prefer open-ended questions and are bored and frustrated by too many closed-ended questionsSlide11
Closed-ended v. open-ended
Because they take longer and are more difficult to deal with, open-ended questions should be used sparingly
Mix open-ended questions through the survey instrument, breaking it up and adding interestSlide12
Question wording
Concrete
questions are best
a. Are you happy with the service at Radio Shack?
b. Better: Were you served within 5 minutes the last time you went to Radio Shack?Slide13
Question wording
Avoid abbreviations
In your view, does
USC
provide a liberal arts education worth its yearly tuition?
In your view, does the
University of South Charleston
provide a liberal arts education worth its yearly tuition?Slide14
Question wording
Avoid slang
and colloquial expressions
Should president Obama take his economic advisers behind the woodshed?
Avoid jargon
and technical terms
Should a
summative evaluation
of Head Start be commissioned by the U.S. government?
Better: Should the U.S. government commission a history of Head Start to review its activities and accomplishments?Slide15
Question wording
Avoid biasing
words and phrases
The
left-wing
student group protesting U.S. presence in Sumatra
claims
that American presence there is unwarranted. Do you
agree?
?
An exception:
If necessary
,
use
loaded questions
to prompt response, but be careful, respondents may be offended
Parents get really angry at their children sometimes.
In the past week, have you been really angry at your son?Slide16
Question wording
Avoid
double-barreled
questions
Do you think the University of Kentucky should subsidize student tuition
and
staff benefits?
Would you say that the decision to increase President Lee Todd’s salary by 67% will be a boon to the University of Kentucky
or
don’t you care?
You could think it would be a boon but not careSlide17
Question wording
Avoid negative wording
Double negatives are especially bad
“Please indicate whether you strongly agree, agree, neither agree nor disagree, disagree or strongly disagree with the following statement:
The United Nations should
not
have more authority to intervene in a nation’s military affairs.
Either
emphasize NOT
when asking the question or else
reword the question
in a positive mannerSlide18
Proper scale construction
Response options on scales should be
1) Mutually exclusive
2) Equivalent
3) ExhaustiveSlide19
Mutual exclusivity
A given respondent or item should not fit into more than one category
Example:
Are you:
a) Single; b) Married; c) Divorced; d) Widowed; e) Separated, or f) Living in a couple relationship but not married
In one interview I heard, a 70+ year old woman said “All of the above”Slide20
Category equivalency
Categories in a scale must differ on a
single dimension
Example: Would you say that the service in the shoe department is:
a) fast
b) slow
c) polite
d) knowledgeableSlide21
Exhaustiveness
All respondents or items must be classifiable
in one of the categories
Example: Does your Herald-Leader arrive on your doorstep by 6 AM:
a) Always
b) Seldom
c) Never
What about “usually”?Slide22
Scales used to measure knowledge, attitudes, beliefs
Developing measures of knowledge, attitudes, and/or beliefs is a challenging task.
While anyone can come up with a measure, it is hard to develop a truly
valid
measure of KAB’sSlide23
Main types of attitude measures
Semantic differentials
Likert
scales
Thurstone
scalesSlide24
Semantic differentialGood __ __ __ __ __ __ __ Bad
Weak __ __ __ __ __ __ __ Strong
Beautiful __ __ __ __ __ __ __ UglySlide25
Likert scales
Example: The Employment Self Esteem Scale
INSTRUCTIONS:
Please rate how strongly you agree or disagree with each of the following statements by placing a check mark in the appropriate box.
(Strongly Disagree/Somewhat Disagree/ Somewhat Agree/Strongly Agree)Slide26
I feel good about my work on the job.
On the whole, I get along well with others at work.
I am proud of my ability to cope with difficulties at work.
When I feel uncomfortable at work, I know how to handle it.
I can tell that other people at work are glad to have me there.
I know I'll be able to cope with work for as long as I want.
I am proud of my relationship with my supervisor at work.
I am confident that I can handle my job without constant assistance.
I feel like I make a useful contribution at work.
I can tell that my coworkers respect me. Slide27
Guttman scale
Statements are ordered so that they represent increasing agreement with or acceptance of one position on a dimension
If respondent accepts one statement higher on the scale, he should also accept those below it on the scaleSlide28
A type of Guttman scale:Bogardus
social distance scale
I am willing to permit immigrants to live in my country
I am willing to permit immigrants to live in my community
I am willing to permit immigrants to live in my neighborhood
I am willing to permit immigrants to live next door to me
I am willing to have immigrants for friends
I am willing to have an immigrant marry my son/daughter
Agreement with item 3 implies agreement with items 1 and 2, and so onSlide29
Thurstone scales
Thurstone
scales include a number of statements that have been developed to represent their position or magnitude on some dimension
The average of the scores of the statements agreed with by the respondent provides her placement on the scaleSlide30
People with AIDS are like my parents.
Because AIDS is preventable, we should focus our resources on prevention instead of curing.
People with AIDS deserve what they got.
Aids affects us all.
People with AIDS should be treated just like everybody else.
AIDS will never happen to me.
It's easy to get AIDS.
AIDS doesn't have a preference, anyone can get it.
AIDS is a disease that anyone can get if they are not careful.
If you have AIDS, you can still lead a normal life.
AIDS is good because it helps control the population.
I can't get AIDS if I'm in a monogamous relationship
.Slide31
Response option considerations
“Don’t know” option
Researchers debate whether the interviewer should explicitly ask whether the respondent doesn’t know an answer
Fear that the DK option gives the respondent an easy out without thinking about the question
If an explicit “Don’t Know” option is provided, you’ll get more DKs but your conclusions may actually be more validSlide32
“Neutral” option
Should scales have a middle option (odd-numbered) or no neutral point (even-numbered)?
Even-numbered scales force respondents to take a standSlide33
Indexes
Indexes combine scores from multiple items to generate a single score for each respondent on some variable of interest
Thurstone
scales are indexes
Some variables are complex, needing several items to provide a valid measure
The items could be organized in a single dimension (
unidimensional
) or in several dimensions (multidimensional)Slide34
Unidimensional indexes
Summated index
measured by a set of indicators that can be added together to derive a single, overall score
Items are related among themselves (correlated), (if you score high on one item you should score high on the other ones)
Make sure the items are not
too
highly correlated Slide35
Multidimensional indexes
If the construct you are measuring has more than one significant
subconcept
, you probably would gain by using a multidimensional index
Example -- communicative credibility is composed of three
subconcepts
:
Authoritativeness
Trustworthiness
Dynamism
To measure communicative credibility you would need to ask questions related to each of the
subconceptsSlide36
So:
Questionnaire design is the most important part of survey analysis
Developing valid measures of ‘black box’ phenomena is especially difficult
Writing good questions is probably the most important part of questionnaire design
A number of common problems in question wording existSlide37
Closed-ended questions are simpler and faster but open-ended questions allow respondents to provide more personalized information and are more interesting for them
A number of forms of attitude/belief measures have been developed over the years
Commonly used types are
Likert
scales, semantic differentials,
Thurstone
scales,
Guttman
scalesSlide38
Using multiple items to measure a single construct and then combining them increases measurement validity