Super Attention Check Items to Conduct WebBased Survey Research The Evolution of MTurk Survey Respondents Kateryna Sylaska PhD Carthage College John D Mayer PhD University of New Hampshire ID: 776399
Download Presentation The PPT/PDF document " It’s 2019: Do We Need " is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
It’s 2019: Do We Need
“Super” Attention Check Items to Conduct Web-Based Survey Research?The Evolution of MTurk Survey Respondents
Kateryna Sylaska, Ph.D., Carthage College John D. Mayer, Ph.D., University of New HampshireAssociation for Research in PersonalityJune 28, 2019
Slide2Why Do We Need Attention Checks?
Low control over testing conditions in online-surveys
(e.g., Johnson, 2005)Participant distraction and “multi-tasking” (e.g., Chandler et al., 2013)Participant satisficing to reduce cognitive demand (e.g., Oppenheimer et al., 2009) To support the integrity of our research (e.g., Curran, 2015; Mead & Craig, 2012)
Slide3Standard Screening
Missing > 50% of survey
Speedy Completion Longstring Responding Attention Check Items
Slide4Attention Check Items
For
a number of years, very simple attention-check items were
sufficient
Slide5Attention Check Items
But now things appear to be changing
Slide6How do we know things are changing?
Slide7Evidence from Research with the Test of Personal Intelligence
Personal Intelligence
(Mayer, 2008; 2014)Ability to reason about ourselves and others based on personality informationRecognize personality informationForm accurate models of personalityUse models to guide choices and make future plansTest of Personal Intelligence (TOPI)Objective, research-based questions
Slide8TOPI
A
person is straightforward and modest. Most likely, she also could be described as:
Valuing ideas and beliefs
Active and full of energy
Sympathetic to others and “tender minded”
Self-conscious and more anxious than average
Slide9the good old days (2013-2016)
Slide10Mturk
Sample on the TOPI-MINI-12 Data Collected February, 2013(reported in Mayer et al, 2018, Study 1)
M
= 0.25
Expectation if Randomly Responding
Slide11College and
Mturk Samples on TOPI-MINI-12 Data Collected January-April 2016 (Sylaska & Mayer, 2019)
College (N = 299 for MINI)no attention checks
Mturk
(N = 468 for MINI)attention checks
Slide12College Sample
of TOPI-MINI-12
Data Collected 2017-2018 (Sylaska, 2019a)
Slide13Now
Slide14Mturk
Sample for TOPI-MINICollected December 2018(Sylaska & Mayer, 2019b)
What’s wrong with this picture?
The negative skew has disappearedNearly half appear to be answering at or near a random level
M
= 0.25
Expectation if Randomly Responding
Slide15Slide16Slide17Attempt to Solve the Problem
Slide18New Sample
Paid for 150 participants
Removed 25 for speedy completion
N
= 125
Slide19Original Attention Check Item
Covert Attention Check Item
Slide20Comparing Original and Embedded Attention Checks
Slide21Evaluating TOPI
Traditional Attentional Checks
Covert Attention Checks
AFTER eliminating participants based on passing 50%+ attention checks
Slide22Cost Consideration
Paid for 150 participants
Removed 25 for speedy completion
Removed 53 for failing traditional and covert attention checks
Final
N
= 72
48% return on investment
Likely still
keeping some inattentive
responders
Expected mean for TOPI is closer to .80 (mean for using these criterion is .70)
Slide23Slide24Other Solutions
IP Address
CollectionGPS Coordinate TrackingOpen-Ended Response Comparisons Embedded Activity Tracking (e.g., TaskMaster)
Dennis et al., 2019; Kennedy et al., 2019; Permut et al., 2019
Slide25References
Chandler, J., Mueller, P., &
Paolacci
, G. (2013).
Nonnaïveté
among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers.
Behavior Research Methods, 46
, 112–130. doi:10.3758/s13428-013-0365-7
Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data.
Journal of Experimental Social Psychology, 66
, 4-19. doi:10.1016/j.jesp.2015.07.006
Dennis, S. A., Goodson, B. M., & Pearson, C. (March 14, 2019). Virtual Private Servers and the limitations of IP-based screening procedures: Lessons from the
MTurk
quality crisis of 2018.
doi
: 10.2139/ssrn.3233954
Johnson, J. A. (2005). Ascertaining the validity of Web-based personality inventories.
Journal of Research in Personality, 39
, 103–129.
doi
: 10.1016/j.jrp.2004.09.009
Kennedy, R., Clifford, S., Burleigh, T., Jewell, R., & Waggoner, P. (October 24, 2018). The shape of and solutions to the
MTurk
quality crisis.
doi
: 10.2139/ssrn.3272468
Mayer, J. D. (2008). Personal intelligence. Imagination,
Cognition and Personality, 27
, 209-232.
Mayer, J. D. (2014).
Personal intelligence: The power of personality and how it shapes our lives.
New York: Farrar, Straus and Giroux.
Mayer, J. D.,
Lortie
, B.,
Panter
, A. T., & Caruso, D. R. (2018). Employees high in personal intelligence differ from their colleagues in workplace perceptions and behavior.
Journal of Personality Assessment, 100,
539-550.
Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data.
Psychological Methods, 17
, 437-455.
doi
: 10.1037/a0028085
Oppenheimer, D. M.,
Meyvis
, T., &
Davidenko
, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power.
Journal of Experimental Social Psychology, 45,
867–872.
doi
: 10.1016/j.jesp.2009.03.009
Permut
, S., Fisher, M., & Oppenheimer, D. M. (2019).
TaskMaster
: A tool for determining when subjects are on task.
Advances in Methods and Practices in Psychological Science, 2,
188–196.
doi
: 10.1177/2515245919838479
Sylaska, K. (2019). [Monmouth College students and choosing a major.] Unpublished raw data.
Sylaska, K., & Mayer, J. D. (2019a).
Major Decisions: Personal intelligence and reasoning about college major contribute to success.
Manuscript submitted for publication.
Sylaska, K., & Mayer, J. D. (2019b). [Personal intelligence and choosing a college major.] Unpublished raw data.
Slide26Thank you