Its 2019 Do We Need Super Attention Check Items to Conduct WebBased Survey Research The Evolution of MTurk Survey Respondents Kateryna Sylaska PhD Carthage College John D Mayer PhD University of New Hampshire ID: 765899
Download Presentation The PPT/PDF document "It’s 2019: Do We Need" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
It’s 2019: Do We Need “Super” Attention Check Items to Conduct Web-Based Survey Research?The Evolution of MTurk Survey Respondents Kateryna Sylaska, Ph.D., Carthage College John D. Mayer, Ph.D., University of New HampshireAssociation for Research in PersonalityJune 28, 2019
Why Do We Need Attention Checks? Low control over testing conditions in online-surveys (e.g., Johnson, 2005)Participant distraction and “multi-tasking” (e.g., Chandler et al., 2013)Participant satisficing to reduce cognitive demand (e.g., Oppenheimer et al., 2009) To support the integrity of our research (e.g., Curran, 2015; Mead & Craig, 2012)
Standard Screening Missing > 50% of survey Speedy Completion Longstring Responding Attention Check Items
Attention Check Items For a number of years, very simple attention-check items were sufficient
Attention Check Items But now things appear to be changing
How do we know things are changing?
Evidence from Research with the Test of Personal Intelligence Personal Intelligence (Mayer, 2008; 2014)Ability to reason about ourselves and others based on personality informationRecognize personality informationForm accurate models of personalityUse models to guide choices and make future plans Test of Personal Intelligence (TOPI)Objective, research-based questions
TOPI A person is straightforward and modest. Most likely, she also could be described as:Valuing ideas and beliefsActive and full of energySympathetic to others and “tender minded”Self-conscious and more anxious than average
the good old days (2013-2016)
Mturk Sample on the TOPI-MINI-12 Data Collected February, 2013(reported in Mayer et al, 2018, Study 1) M = 0.25 Expectation if Randomly Responding
College and Mturk Samples on TOPI-MINI-12 Data Collected January-April 2016 (Sylaska & Mayer, 2019) College (N = 299 for MINI)no attention checksMturk (N = 468 for MINI)attention checks
College Sample of TOPI-MINI-12 Data Collected 2017-2018 (Sylaska, 2019a)
Now
Mturk Sample for TOPI-MINICollected December 2018(Sylaska & Mayer, 2019b) What’s wrong with this picture?The negative skew has disappearedNearly half appear to be answering at or near a random levelM = 0.25 Expectation if Randomly Responding
Attempt to Solve the Problem
New Sample Paid for 150 participantsRemoved 25 for speedy completion N = 125
Original Attention Check Item Covert Attention Check Item
Comparing Original and Embedded Attention Checks
Evaluating TOPI Traditional Attentional ChecksCovert Attention Checks AFTER eliminating participants based on passing 50%+ attention checks
Cost Consideration Paid for 150 participantsRemoved 25 for speedy completionRemoved 53 for failing traditional and covert attention checksFinal N = 7248% return on investmentLikely still keeping some inattentive respondersExpected mean for TOPI is closer to .80 (mean for using these criterion is .70)
Other Solutions IP Address CollectionGPS Coordinate TrackingOpen-Ended Response Comparisons Embedded Activity Tracking (e.g., TaskMaster)Dennis et al., 2019; Kennedy et al., 2019; Permut et al., 2019
References Chandler, J., Mueller, P., & Paolacci, G. (2013). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–130. doi:10.3758/s13428-013-0365-7Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4-19. doi:10.1016/j.jesp.2015.07.006Dennis, S. A., Goodson, B. M., & Pearson, C. (March 14, 2019). Virtual Private Servers and the limitations of IP-based screening procedures: Lessons from the MTurk quality crisis of 2018. doi : 10.2139/ssrn.3233954 Johnson, J. A. (2005). Ascertaining the validity of Web-based personality inventories. Journal of Research in Personality, 39 , 103–129. doi : 10.1016/j.jrp.2004.09.009 Kennedy, R., Clifford, S., Burleigh, T., Jewell, R., & Waggoner, P. (October 24, 2018). The shape of and solutions to the MTurk quality crisis. doi : 10.2139/ssrn.3272468 Mayer, J. D. (2008). Personal intelligence. Imagination, Cognition and Personality, 27 , 209-232. Mayer, J. D. (2014). Personal intelligence: The power of personality and how it shapes our lives. New York: Farrar, Straus and Giroux. Mayer, J. D., Lortie , B., Panter , A. T., & Caruso, D. R. (2018). Employees high in personal intelligence differ from their colleagues in workplace perceptions and behavior. Journal of Personality Assessment, 100, 539-550. Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17 , 437-455. doi : 10.1037/a0028085 Oppenheimer, D. M., Meyvis , T., & Davidenko , N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872. doi : 10.1016/j.jesp.2009.03.009 Permut , S., Fisher, M., & Oppenheimer, D. M. (2019). TaskMaster : A tool for determining when subjects are on task. Advances in Methods and Practices in Psychological Science, 2, 188–196. doi : 10.1177/2515245919838479 Sylaska, K. (2019). [Monmouth College students and choosing a major.] Unpublished raw data. Sylaska, K., & Mayer, J. D. (2019a). Major Decisions: Personal intelligence and reasoning about college major contribute to success. Manuscript submitted for publication. Sylaska, K., & Mayer, J. D. (2019b). [Personal intelligence and choosing a college major.] Unpublished raw data.
Thank you