It’s 2019: Do We Need - PowerPoint Presentation

Slide1

It’s 2019: Do We Need

“Super” Attention Check Items to Conduct Web-Based Survey Research?The Evolution of MTurk Survey Respondents

Kateryna Sylaska, Ph.D., Carthage College John D. Mayer, Ph.D., University of New HampshireAssociation for Research in PersonalityJune 28, 2019

Slide2

Why Do We Need Attention Checks?

Low control over testing conditions in online-surveys

(e.g., Johnson, 2005)Participant distraction and “multi-tasking” (e.g., Chandler et al., 2013)Participant satisficing to reduce cognitive demand (e.g., Oppenheimer et al., 2009) To support the integrity of our research (e.g., Curran, 2015; Mead & Craig, 2012)

Slide3

Standard Screening

Missing > 50% of survey

Speedy Completion Longstring Responding Attention Check Items

Slide4

Attention Check Items

For

a number of years, very simple attention-check items were

sufficient

Slide5

Attention Check Items

But now things appear to be changing

Slide6

How do we know things are changing?

Slide7

Evidence from Research with the Test of Personal Intelligence

Personal Intelligence

(Mayer, 2008; 2014)Ability to reason about ourselves and others based on personality informationRecognize personality informationForm accurate models of personalityUse models to guide choices and make future plansTest of Personal Intelligence (TOPI)Objective, research-based questions

Slide8

TOPI

A

person is straightforward and modest. Most likely, she also could be described as:

Valuing ideas and beliefs

Active and full of energy

Sympathetic to others and “tender minded”

Self-conscious and more anxious than average

Slide9

the good old days (2013-2016)

Slide10

Mturk

Sample on the TOPI-MINI-12 Data Collected February, 2013(reported in Mayer et al, 2018, Study 1)

M

= 0.25



Expectation if Randomly Responding

Slide11

College and

Mturk Samples on TOPI-MINI-12 Data Collected January-April 2016 (Sylaska & Mayer, 2019)

College (N = 299 for MINI)no attention checks

Mturk

(N = 468 for MINI)attention checks

Slide12

College Sample

of TOPI-MINI-12

Data Collected 2017-2018 (Sylaska, 2019a)

Slide13

Now

Slide14

Mturk

Sample for TOPI-MINICollected December 2018(Sylaska & Mayer, 2019b)

What’s wrong with this picture?

The negative skew has disappearedNearly half appear to be answering at or near a random level

M

= 0.25



Expectation if Randomly Responding

Slide15

Slide16

Slide17

Attempt to Solve the Problem

Slide18

New Sample

Paid for 150 participants

Removed 25 for speedy completion

N

= 125

Slide19

Original Attention Check Item

Covert Attention Check Item

Slide20

Comparing Original and Embedded Attention Checks

Slide21

Evaluating TOPI

Traditional Attentional Checks

Covert Attention Checks

AFTER eliminating participants based on passing 50%+ attention checks

Slide22

Cost Consideration

Paid for 150 participants

Removed 25 for speedy completion

Removed 53 for failing traditional and covert attention checks

Final

N

= 72

48% return on investment

Likely still

keeping some inattentive

responders

Expected mean for TOPI is closer to .80 (mean for using these criterion is .70)

Slide23

Slide24

Other Solutions

IP Address

CollectionGPS Coordinate TrackingOpen-Ended Response Comparisons Embedded Activity Tracking (e.g., TaskMaster)

Dennis et al., 2019; Kennedy et al., 2019; Permut et al., 2019

Slide25

References

Chandler, J., Mueller, P., &

Paolacci

, G. (2013).

Nonnaïveté

among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers.

Behavior Research Methods, 46

, 112–130. doi:10.3758/s13428-013-0365-7

Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data.

Journal of Experimental Social Psychology, 66

, 4-19. doi:10.1016/j.jesp.2015.07.006

Dennis, S. A., Goodson, B. M., & Pearson, C. (March 14, 2019). Virtual Private Servers and the limitations of IP-based screening procedures: Lessons from the

MTurk

quality crisis of 2018.

doi

: 10.2139/ssrn.3233954

Johnson, J. A. (2005). Ascertaining the validity of Web-based personality inventories.

Journal of Research in Personality, 39

, 103–129.

doi

: 10.1016/j.jrp.2004.09.009

Kennedy, R., Clifford, S., Burleigh, T., Jewell, R., & Waggoner, P. (October 24, 2018). The shape of and solutions to the

MTurk

quality crisis.

doi

: 10.2139/ssrn.3272468

Mayer, J. D. (2008). Personal intelligence. Imagination,

Cognition and Personality, 27

, 209-232.

Mayer, J. D. (2014).

Personal intelligence: The power of personality and how it shapes our lives.

New York: Farrar, Straus and Giroux.

Mayer, J. D.,

Lortie

, B.,

Panter

, A. T., & Caruso, D. R. (2018). Employees high in personal intelligence differ from their colleagues in workplace perceptions and behavior.

Journal of Personality Assessment, 100,

539-550.

Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data.

Psychological Methods, 17

, 437-455.

doi

: 10.1037/a0028085

Oppenheimer, D. M.,

Meyvis

, T., &

Davidenko

, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power.

Journal of Experimental Social Psychology, 45,

867–872.

doi

: 10.1016/j.jesp.2009.03.009

Permut

, S., Fisher, M., & Oppenheimer, D. M. (2019).

TaskMaster

: A tool for determining when subjects are on task.

Advances in Methods and Practices in Psychological Science, 2,

188–196.

doi

: 10.1177/2515245919838479

Sylaska, K. (2019). [Monmouth College students and choosing a major.] Unpublished raw data.

Sylaska, K., & Mayer, J. D. (2019a).

Major Decisions: Personal intelligence and reasoning about college major contribute to success.

Manuscript submitted for publication.

Sylaska, K., & Mayer, J. D. (2019b). [Personal intelligence and choosing a college major.] Unpublished raw data.

Slide26

Thank you