Lorrie Faith Cranor September 2011 Surveys interviews and focus groups Surveys Ask people set list of questions possibly with conditional questions or branching with multiple choice or free response answers ID: 436454
Download Presentation The PPT/PDF document "Surveys, interviews, and focus groups" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Surveys, interviews, and focus groups
Lorrie Faith Cranor
September
2011Slide2
Surveys, interviews, and focus groups
Surveys
Ask people set list of questions (possibly with conditional questions or branching) with multiple choice or free response answers
Can be conducted in person, by postal mail, by phone, online
Interviews
Ask people open-ended questions, with follow-up questions depending on their responses
Usually conducted in person or by telephone
Focus groups
Group interview, usually with 5-10 participants
Usually conducted in personSlide3
What method should I use?
Survey
Allows you to collect data from a lot of people relatively quickly
Easy to collect data in standardized format, ready for analysis
May be difficult to answer
“
why”
questions
Useful for validating hypotheses from smaller studies with a larger population
Interview
Allows you to probe mental models, what people think and why
Semi-structured interviews allow you to explore issues as they are raised
Allows you to clarify if people don’t understand a question
Focus group
Similar to interviews
, but more efficient as you can interview 5-10 people
at
once
Less detail from any interviewee than you would get in an individual interview
Not
great
for testing usability because participants probably won’t all
use the
software
Sometimes an opinionated individual can dominate a focus group
Hard to publish
paper based only on
one or two focus groupsSlide4
SurveysSlide5
Designing good survey questions
Word questions clearly, without jargon or undefined abbreviations
Avoid leading questions, ambiguous terms, or emotionally-loaded terms
Design questions to evoke truthful responses
Non-threatening, don’t bias participants to provide what they think you want, protect confidentiality
Probe one dimension at a time
Design questions such that respondents are likely to provide a range of answersSlide6
Multiple-choice answers
Make sure answer choices are clear, mutually exclusive, cover entire space of possible answers
Make sure answer choices are at appropriate level of granularity
Where appropriate, allow respondents to indicate they don’t know, don’t have an opinion, or the question is not applicable to them
Use consistent rating scales throughout your survey
Be aware that the rating scale can skew responses (people like to think they are normal)Slide7
More survey design tips
Cluster similar questions together
Do not ask respondents to perform cognitively difficult tasks (unless you are testing their ability to perform these tasks) such as ranking more than 5 items
Use a clear and attractive layout
Pilot, pilot, pilot!Slide8
InterviewsSlide9
Developing an interview script
Keep your questions fairly open-ended
You can follow-up with specific probes (“What files do you have on your computer that you consider valuable?” Follow up with “Do you have valuable photos? Videos? School work? Letters?”)
Start with general questions and get more specific so you get their unbiased impressions before you direct their thinking to particular details
A semi-structured approach allows you to adjust as needed
Write-out all your questions and follow-up probesSlide10
Role play and hypothetical scenarios
Appropriate for some interview studies
Give participant a role to play or put them in a hypothetical scenario
Imagine you just saw this message on your computer screen….
Imagine your friend called you and told you he saw this message and asked you what to do….Slide11
Preparing for an interview
The day before the interview:
Print out:
Protocol – including detailed interview script
Official consent form
Payment sheet
Prepare:
Compensation payment
Audio/video recording (devices, batteries, extension cords, etc.)
Additional material
Send an email to the interviewee to remind him/her of the date/time/placeSlide12
During the interview
Explain the purpose of the study (unless you need to hide that)
Ask to read and sign the consent form
If recording/video taping, turn it on
!
Perform
the study!
Debrief the person (if applicable)
Ask to sign the signing sheet
Give the payment
Thank the person!Slide13
Coding interview data
Transcribe interview data (yourself or with hired transcription service
)
Depending on purpose of study, transcribing only selected quotes may be sufficient
Iteratively review transcripts and create code for concepts mentioned by participants
As new codes are added, check to see whether those concepts were mentioned in previously analyzed transcripts
Keep track of how many participants mentioned each concept to find concepts that resonate with a lot of participants
Group similar codes together into categories
Note interesting
quotesSlide14
Analyzing interview data
Interview can be formally analyzed through a coding process
Qualitative
approach
Use concepts and categories to develop theory (Grounded theory approach, does not start with hypotheses)
Quantitative approach
With large number of interviewees (~30), and questions that ask participants to provide numerical ratings, quantitative analysis may be feasible
Important to validate for coder reliability
Can be used to develop mental modelsSlide15
Focus groupsSlide16
Planning a focus group
Develop
very detailed
script to guide you
Pre-screen participants to get a good mix of people who meet your
criteria
Setup audio and video recorders, but don’t make people feel under surveillance
Helpful
to have at least 2 people, a moderator and a note taker
Give people name tags with their first name only
Plan
to do multiple focus groups to mitigate effects of dominant participant steering conversationSlide17
Conducting a focus group
Make
the session fun, informal, relaxed feel
Provide drinks and snacks
Promote a free flowing conversation that engages all participants
Ask open ended questions
Show people multiple things and ask them to compare
Give demos or show videos to start-off discussion
Give people handouts and ask them to circle things they like/don’t like, or jot down first impressions before discussing with the
groupSlide18
Using videos in focus groups
Videos can provide concrete scenarios for people to
discussSlide19