What are the major advantages and disadvantages of surveys Determine content and purpose of question Choose the response format Figure out how to word it Figure out where to put it Pilot test ID: 729227
Download Presentation The PPT/PDF document "Surveys! Diary methods What are the diff..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Surveys!Slide2
Diary methodsSlide3
Curran, 2016
What are the reasons participants may give us bad data?
How can you address each?
How frequent are C/IE responders?
How do they affect results? What are some ways to deal with C/IE?
Careless or inattentive responsesSlide4
Decide ahead of time how you will clean your data and
pre-register (including order of procedures)
Decide what order you’ll do these in as well as what you’ll do
Always collect response time
Consider collecting IP addresses and mapping them, collecting worker number (
mTurk
)
Before the studySlide5
Attention check items
Instructional manipulation checks
Ask participants if you should use their data
Ask participants if they were honest
Included in the designSlide6
Long string analysis
Mahalanobis
distance
Individual consistency
Semantic and psychometric antonyms/synonymsInter-tem SDPolytomous
Guttman
errorsPerson-total rRead about your study online (
mTurk
; turknation.com,
reddit
)
Data cleaningSlide7
Pick more than one method
Pre-register what you’ll use (and use that)
Report what you did and why
Run and report the analyses both ways
Overall best practicesSlide8
When is missing data likely to be a problem?
What are the different types of missing data you can have?
MCAR vs. MAR vs. MNAR
In reality, a continuum between MAR and MNAR
How can you tell if your data are MCAR? Little’s MCAR test (want ns chi-square)
What problems do they cause?
Missing data (Graham, 2009)Slide9
Listwise
deletion
Pairwise deletion
Mean substitutionMissingness dummy variable
Regression-based single
imputation
Old methods of detailing with missing valuesSlide10Slide11
If at least ½ variables there and good alpha and all item-total correlations are about the same, then just calculate with those items you have.
But don’t use mean(….) in SPSS. Write out the math, then look at those with missing data
Compute new=mean(Q1, Q2, Q3).
Compute new=(Q1+Q2+Q3)/3.
You will get different answers, depending on missing values
Use syntax!
Missing data on a scaleSlide12
In SPSS, need missing values analysis module
EM algorithm (expectation maximization)
Goes through values one at a time. If there is a value, it’s added to the model. If not, then the best guess based on predicting it in regression with all the other variables is put in. This continues until it becomes stable.
Good for: mean, variance, covariance estimates, correlation matrices, coefficient alpha, exploratory FA
Standard errors too small, so not good for hypo testing
Use SAS, NORM, EMCOV, SPSS*, R
Modern methodsSlide13
MI (multiple imputation):
Better because it doesn’t assume that responses lie on the regression line—it adds in random error
Use NORM, SAS,
Splus
, SPSS*, RFIML:Does it all in one step
SEM software: Amos, LISREL,
Mx, R Slide14
3 form design
2 method measurement
Include good predictors of the missing values in your data set
Measure
p’s plans to drop outFollow up and try to get measures for some drop outsAre there other options they didn’t mention?
Planning for missing dataSlide15
What are the differences between surveys, interviews, scales, and questionnaires?
What are the major advantages and disadvantages of surveys? Slide16
Determine content and purpose of question
Determine the survey mode
Choose the response format
Figure out how to word it
Figure out where to put itPilot test!
Ask for feedback from participants (at least have a comments box)
When writing items…Slide17
Is the question needed? At that level of detail?
Is there a double-barreled question? Do you need to ask more than 1 question?
Do p’s have the info and ability to answer the question?
Do you need to be more specific or more general? Is the question clear and unambiguous?
Are there biases or charged words in the question? Will people answer the question honestly?
Hypothetical projective respondent
Question wordingSlide18
Question wording
Can someone answer either way and still look “good” (social desirability concerns)?
Does the question include assumptions or need a time frame specified?
Does the question fit the population you’re sampling?
How personal is the wording?
Are there words people wouldn’t know?
Are the alternatives clear?Do you use “not” statements?
Does the questionnaire look nice? Slide19
Any problems?
When did you move to Cedar Falls?
In the past 30 days, were you able to climb the stairs with no difficulty?
On days when you drink alcohol, how many drinks do you usually have?
How many miles are you from the nearest hospital?
I feel completely secure in facing unknown new situations because I know that my partner will never let me down
.
How long have you been married?
I feel I do not have much to be proud of. Slide20
Options
Don't you agree that campus parking is a problem?
There are many people who believe that campus parking is a problem. Are you one of them?
Do you agree that campus parking is a problem and that the administration should be working diligently on a solution?
What do you think about parking?Slide21
More on question wording
(1 = strongly disagree to 7 = strongly agree; * = reverse scored)
I oppose raising taxes.* (willing to pay)
2.88
I would be willing to pay a few extra dollars in taxes to provide high-quality education to all children.
4.82
The primary task of the government should be to keep citizens safe from terrorism and crime.
3.81
The primary task of the government should be to preserve citizen’s rights and civil liberties.
4.50
nSlide22
USA Today/Gallup Poll Feb. 2009
1013 adults, +/- 3% error
Do you approve of the government temporarily
taking over
major banks in danger of failing? 54% approved
Do you approve of the government temporarily
nationalizing major banks in danger of failing? 57% dis
approvedSlide23
Other bad survey question examples?Slide24
Question placement and layout
What should be early vs. late in the survey?
What else affects placement?
When and what should you randomize?
What about layout issues?
Order effects? Slide25
In a Likert scale, how many options should you have?
Should there be a neutral point?
How can the responses you offer affect results?
How can you reduce social desirability with response alternatives?
What criteria do response categories have to have? What are the best options to use for gender or ethnicity?
Types of responses Slide26
When are open-ended items good or bad?
When should you give multiple options?
Do you have all the alternatives without going into too much detail? At what point do you quit adding categories?
More on responsesSlide27
Survey examples
How often do you exercise?
Infrequently 17%
Occasionally 48%
Often 35%
In the last six months, how often have you engaged in at least 20 minutes of aerobic activity?
Almost never 17% 3x/week 15%
Less than 1x/week 13% 4x/week 15%
1x/week 12% >4x/week 13%
2x/week 15%Slide28
Thank them to start
Keep it short
Be alert to discomfort
Look professional
Thank them at the endSend a copy of the results if they want them
Be nice to participantsSlide29
Mail survey
Group-administered questionnaire
Household drop-off survey
Electronic survey
Focus groupTelephone interviewFace-to-face interviewComputerized options
Mixed mode
Advantages/disadvantages and when to use:Slide30
Population issues
Sampling issues
Question type issues
Content issues
Bias issuesAdministrative issues
How to decide what method to use? Slide31
Things to consider in evaluating survey data
Who paid for the survey?
Who are the participants?
What was the sampling frame and method?
How were the questions worded?
What is the margin of error?
How were the data weighted? Slide32
Last week
This week
Outline coming up (Feb. 26)
AssignmentsSlide33
Experiments
2 chapters
Problems with experiments (2 articles)
Mediation and moderation
Placebo/Nocebo presentation
Next weekSlide34
What are the main strengths and weaknesses of this study?
Table 3
What were the main findings?
Mode study (Zhang,
Kuchinke
,
Woud, Velten, & Margraf
, 20170