in ACM Conferences Jonathan Aldrich Carnegie Mellon University SGB Liason to the ACM Publications Board DoubleBlind Reviewing DBR shields author identities from reviewers SingleBlind Reviewing SBR ID: 916589
Download Presentation The PPT/PDF document "Considering a Mandate for Double-Blind R..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Considering a Mandate for Double-Blind Reviewingin ACM Conferences
Jonathan Aldrich, Carnegie Mellon University
SGB
Liason
to the ACM Publications Board
Slide2Double-Blind Reviewing (DBR) shields author identities from reviewers
Single-Blind Reviewing (SBR)
Reviewer identities hidden from authors
Author identities
visible
to reviewers
Double-Blind Reviewing (DBR):
Reviewer identities hidden from authors
Author identities
hidden
from reviewers
Slide3There is bias in single-blind reviewing[Snodgrass, SIGMOD 2006][Tomkins et al., PNAS 2017]
Clear literature review findings
Evidence of gender bias
Evidence of pro-US institution bias
Perception that DBR is more fair
Additional mixed/surprising results (not always in expected direction)
Bias related to prolific authors
Bias related to institution quality
Regardless of the source / kind of bias, DBR can reduce it
Slide4DBR Improves Quality
Articles published in journals using blinded peer review were cited significantly more than articles published in journals using non-blinded peer review, controlling for a variety of author, article, and journal attributes
.
[
Laband
and
Piette
, JAMA 1994]
Slide5DBR reviewers rarely guess author identity
Across 3 conferences, reviewers were asked to guess author identity
74-90% of reviews were submitted with no correct guesses
[Le Goues et al., CACM 2018]
Slide6DBR is practical
PC chairs did not report the extra administrative burden was large
[Le Goues et al., CACM 2018]
Most current ACM conferences use DBR
80% of those that reported (117 of 146)
Thank you for contributing to this data:
https
://
docs.google.com/spreadsheets/d/1XqN4PIUYDJAeWoqIzhXQodwJx8bQfVs9cS31vK0MQe0/edit
Slide7DBR has variations with tradeoffs[terminology from double-blind.org]
When are author identities revealed?
Partially double blind: author identities revealed after initial review
Allows authors to be considered in discussion, e.g. with respect to prior work
Fully double blind (blind-to-accept): author identities hidden until (cond.) accept
Stronger protection against bias
Most double-blind ACM conferences are blind-to-accept (93 of 117)
Some started partially double-blind, and moved to fully double-blind
arXiv
-restricted
S
ubmissions to
arXiv
are restricted for a period before/during review
Can help with blinding, as reviewers may be notified of preprint submissions
Relatively uncommon in ACM conferences (~3 of 127)
Pubs Board encourages
arXiv
preprints in general, so a total ban is counterproductive
Slide8Should ACM mandate DBR?
Some other publishers are doing it, e.g. IOP in Physics
https://ioppublishing.org/news/iop-publishing-commits-to-adopting-double-blind-peer-review-for-all-journals
/
Some publishers are going the other way, towards Open Peer Review
Let’s discuss
Slide9Some Resources on Double Blind Review
https://double-blind.org
/
Tracks double-blind reviewing in top CS conferences
Includes resources on DBR (at the bottom of the page)
https://
www.cs.utexas.edu/users/mckinley/notes/blind.html
Kathryn McKinley’s site on improving reviewing quality through DBR
Also includes many useful DBR resources
Two resources advocating for double-blind in SE conferences
https://www.cs.cmu.edu/~
clegoues/double-blind.html
https://people.cs.umass.edu/~
brun/doubleblind.html