Presentation on theme: "Questionnaire Design"— Presentation transcript
Slide1
Questionnaire Design
Van
Bennekom
, Chapter 4Slide2
Identifying the questions to ask
What are concerns and interests from your
U
pstream shareholders
E.g., monitor service quality to avoid future problems
Different managers might have different issues
Immediate recipients
The direct target, truly reflect their concerns (otherwise, low response rate or wrong answers)
Personal interview: interview representatives
Focus groups: small group interviews
Related literatures, documentations, white papers, reports, etc.
Downstream customers
Might be the indirect cause, their influence to the immediate recipients
Follow your statement of purpose with clear definition of scopes and goals.Slide3
Identifying the questions to ask
Personal Interviews
Defining a set of interview questions that helps elicit the survey purpose
Semi-structured questionnaire (creating your questions during the interview)
Questionnaire: add more questions, alter the sequences of questions, change wording, Slide4
Identifying the questions to ask
Personal Interviews
How to take the notes
Using tape recorder (please check the legal issues)
Start with the statement of the interview
The best notes are quotes from the interviewee.
How many interviewers should we interview?
1 to many
Who should conduct the interviews
Someone with the interview training
Person who will design the questionnaire, conduct the surveySlide5
Identifying the questions to ask
Personal Interviews
How should the interview be sampled
Purposive sample (most representative, hard and critical
Where should these interviews be conducted
Convenient for interviewees
Face-to-face is the best
How long should these interviews last?
30min-one hour
When should we stop
When you stop hearing new thingsSlide6
Identifying the questions to ask
Personal interview
Pros
Deep in context
Explore unexpected paths
Paints the initial picture
Cons
How structured a questionnaire?
How many interviewers
By phone or in person
lengthSlide7
Identifying the questions to ask
Focus group
To have a directed discussion with a small group of people
The dynamic interaction with groups can bring some issues or ideas that would have been missed or ignored during personal interview.
Time efficient
How many people should be in a focus group
7-15Slide8
Identifying the questions to ask
Focus group
Who should be invited
Purposive sample, attendee covers different fields
Who should not be invited
Limit the number of observers
What is the role of the moderator?
Help to bring out all sides
Encourage members to discuss/present their ideas
Control an overbearing opinion leader
What type of questionnaire should be used?
Same as personal interview: semi-structured Slide9
Identifying the questions to ask
Focus group
Where should the focus group be held and how should the discussion be captured.
A comfortable meeting room
Audio taping
How many focus groups should be conducted?
Until you did not hear new information.
How long should it last
Two hours, morning is better than afternoon, evening is also good
Serving a light snack is always good.Slide10
Identifying the questions to ask
Focus group
Pros
Deep in context
Explore unexpected paths
Interaction among participants
Cons
Planning & execution critical
Good moderator critical
Have one central theme
How many to invite
Whom to invite
Time consuming textual analysisSlide11
Documentary Data
Existing documents
White papers
Related literatureSlide12
Analyzing data
Doing content analysis of the textual notes
Boiling down the words into categories, supplemented with a few well-chosen statements
Identify patterns: group similar comments to reveal patterns
Some free-form comments can address the “why” for a question
Literal transcript of every word is not necessary, only capture key points and key focuses.
Connect words to interviewee’s background
A distillation processSlide13
Drafting the questionnaire
It is an iterative process:
Step 1: Organizing your essential findings from the interviews, group them into logic groups
Step 2:
Consider three basic types of questions (e.g., demographic, specific, overall)
Step 3:
Select question formats: scale type (limit the number of different scale types), open-end
Step 4:
Draft some questions
Step 5:
Take a break
Step 6: Repeat steps 3-5.
Step 7: seek other input (look for external reviewers to go through your questionnaire, ask domain experts
Step 8: Pilot test your draft questionnaire with a representative sample (critical stage)Slide14
Elements of a questionnaire
A questionnaire contains several components
An introduction
Instructions
Initial questions
Sections and their headings
Summary
A thank youSlide15
Elements of a questionnaire
Introduction
Personally addressed to the respondent
To explain the purpose of the survey
Who should complete the survey
The expected time to complete the survey
Emphasize anonymity, if appropriate
What to do with the completed questionnaire
When to do the survey (now!)
Definition of certain terminology
Example: Van
Bennekom
, p75Slide16
Elements of a questionnaire
Instructions
How to enter answers
Explain the scales
Example: Van
Bennekom
, p78Slide17
Elements of a questionnaire
Initial questions
Pay attention to the initial questions (engage them, subject matter, motivation to move ahead, easy to answer)
Never open with demographic questions in the beginningSlide18
Elements of a questionnaire
Section headings
Should be short
Group 50 questions into 3 section vs. put all 50 questions in one section.Slide19
Elements of a questionnaire
Summary and Thank youSlide20
Sequencing questions
From general to specific
Initial questions should be general
Demographic questions
Do not ask questions which data you do not need
Question interaction
Rating a person, rather than a system
Connection of different questions
Rotating questions
Branching: direct respondents to the right sectionsSlide21
Question formats
Unstructured
Open-ended, free-form
Limited the number of open-ended questions (respondent burden)
Post them after certain structured questions about key theme, or ask them to clarify or expand on previous answersSlide22
Question formats
S
tructured
(see page 85)
multiple choice or categorical,
ordinal scales,
interval-rating scales,
ratio scalesSlide23
Question formats
Structured: multiple choice or categorical,
For frequency distribution (e.g., job title, company size)
Kinds:
Multiple choice, multiple response
Multiple choice, single response
Binary choice
Adjective checklist (page 90)
Design issues:
Range of choices in response sets (include other category)
Distinctions among choices
Number of choices (max 6-8)
Sequencing of choicesSlide24
Question formats
Structured: Ordinal scales (page 93)
Basic ordinal scale: age range, or salary range
Forced-ranking scale: ask respondent to rank some items
Paired comparison
Design issues
Number of items
Clarify of instructions on what the rank meansSlide25
Question formats
Structured: Interval-rating scales (page 95)
Such as: level of agreement
Verbal scales: scale in text
Numeric scales: scale in number
Likert
-type scale: strongly disagree-disagree-neither agree nor disagree-agree-strongly agree
Many others (page 98-100)
Design issues
Advantages of grouping by type
Which step first, developing scale or writing questions?
Importance of anchor choice
Scale direction (1 is best or 5 is best?)
Interval equality
Anchor balancing (extremely dissatisfied, vs. very satisfied)
Even (1-7) vs. odd (1-6)-numbered scales
Number of points on the scale: 10 points are too much.
Presenting the scale low to high or high to low
Use of multiple scale types creates confusion
Avoiding response ruts
Dispersion of resultsSlide26
Question formats
Structured: Ratio scales (page 108)
Fractionation scale: open-ended scale, no upper limit, has better precision.Slide27
General guidelines
Goal
Maximizing the information we receive
Minimizing response burden
Do not ask meaningless questions
Keep the instrument focused
A survey is not a test (instruction should be clear)
Be concerned about the visual and layout of your questionnaire
Keep wording and grammar simple
Be creative in question designSlide28
Pilot testing
Don’s skip it
It is not complicated
It should be conducted as one-on-one or personal interview on purposive sampled targets
The purposes are
To know whether questions are well designed, difficult to answer, or have proper response choices
Engage the respondent?, too long? Clarity of instructions, find awkward wording
