evalhtml8SIGMOD Record Vol 35 No 3 Sep 2006 associateeditorcftheTODSpolicyySnodgrass2003fcoursetheEditorinChiefandtheProgramChairareknowntoeveryoneTheimportantpointhereisthatthesinglebli ID: 950982
Download Pdf The PPT/PDF document "ACMPolicyonPrePublicationEvaluationat" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
ACMPolicyonPre-PublicationEvaluation,at eval.html8SIGMOD Record, Vol. 35, No. 3, Sep. 2006 associateeditor(cf.theTODSpolicyySnodgrass2003fcourse,theEditor-in-ChiefandtheProgramChairareknowntoeveryone.Theimportantpointhereisthatthesingle-blindreviewingappliesonlytohidingtheidentityofthereviewerfromtheauthor.Inanefforttoachievemoreobjectivereviewing,avenuecanalsoreuestthattheidentityoftheauthorberemovedfromthesubmittedmanuscript,aprocesstermedblindingthemanuscript.Whentheidentityoftheau-thorsandtheirinstitutionsiskeptfromthereviewers,thisistermeddouble-blindreviewing.NotethattheEditor-in-ChiefandProgramChair,andgenerallytheAssociateEditor,aremadeawareofthisinformationviaaseparatecoversheetnotsharedwiththereviewers.Thepsychologicalsciencesutilizeadifferenttermi-nologythatconveysasubtlephilosophicalshift.Whentheidentitiesoftheauthorsandreviewersarenotre-vealedtoeachother,itistermedinthesesciencesamaskedreviewingprocess.Notethesymmetryofthisterminology.TheAmericanPsychologicalAssociationGuidetoPreparingManuscriptsforJournalPublica-tionnCalfeeValencia200states,Peerreviewisthebackboneofthereviewprocess.MostAPAjournals,likethemajorityofotherprofessionalpublications,prac-ticeanonymous,ormasked,reviews.Authorsandre-viewersareunawareofeachothersidentitiesinmostinstances,anarrangementdesignedtomaketheprocessmoreimpartial.Theimplicationisthatrevealingeitherthereviewersidentityortheauthorsidentitybreaksthemask.Presumablysingle-blindreviewingwouldthenbetermednon-masked,buttheAPAdoesntusetheterm.(Thetermunmaskingdenotesrevealingtheidentityofareviewertoaco-reviewerrvanRooyen1999wedonconsiderthatpracticehere.)Thepresentpaperwillusethetermssingle-blindanddouble-blindreviewing,aswellastheirrespectivethree-letteracronyms,Venuesdifferinwhodoestheblindingmaskingofasubmission.Wewillusethetermauthormaskingtheauthorremovesidenticationfromthepaperbeforesubmittingitandeditorialmaskingwhensuchidentication(generally,authornameandafliation)isremovedintheeditorialprocessbeforesendingthemanuscripttothereviewers.Proceduresdifferinhowaggressiveistheuiredauthormaskingandtheactualeditorialmask-ing.Self-citationsandotherrst-personreferencesinthebodyofpapersaregenerallyretainedinedito
rialmask-ing.Whileauthormaskingcanbemorethorough,be-causeauthorswouldknowwhatkindofinformationisrevealing,authorsthroughvariousdeviousmeanscancir-cumventbothkindsofmasking.3LiteratureReviewsBecauseofthecentralityofpeerreviewtothepropaga-tionofscienticknowledge,onewouldepectthatpeerreviewhasbeenthoroughlystudied,withitsbenetsandpotentialpitfallsehaustivelydocumented.Suchisnotthecase.Priorto197researchonpeerreviewwasrel-ativelyscarce,withdiscussionsbasedmoreonpersonalobservationsratherthansystematicdatagathering.Cam-panariohaswrittenthemostcomprehensive(1pagesummaryoftheresearchthathasbeendoneonpeerre-view,generallyoverthetwodecadesof197.Part1ofthissummarycoveredtheparticipantsinthesystemthecredentialsofreferees,editorialboardmembers,andeditorshoweditorsandeditorialboardmembersareap-pointedhowrefereesarechosenreviewerincentivesandtasksandsystemicproblemsofreliability,accuracy,andbiasreliabilityofreviewaccuracyofreviewisthesys-tembiasedtowardspositiveresultsandisthesystembi-asedagainstreplicationeplicationCampanario1998a.Part2cov-eredcurrentresearchndingsaboutfraud,favoritism,andself-interestinpeerreviewwCampanario1998bthispartincludedathree-pagesectiononDBR.TheInstituteofMathematicalStatistics(IMS)formedtheAdHocCommitteeonDouble-BlindRefereeinginFebruary1991,atleastpartlyinresponsetoareportoftheNewResearchersCommitteeitteeAltmanetal.1991.Thiscommitteeissuedareportthatcontainedathree-pagelit-eraturereviewwCoxetal.1993therreviewsincludeoneofthevoluminouslit-erature(over00items)onthemoregeneraltopicofjournalreviewing,includingseveralparagraphsre-latedtoDBRBRDalton199,anotheron8papersonempiricalevidenceconcerningjournalpeerre-viewwArmstrong1997,includingonepageonDBR,andasummaryoftheevidencefortheeffectivenessofpeerreviewingeneral,includingaboutapageandahalfonDBRRFletcher&Fletcher1997.Finally,twointer-nationalcongressesoneditorialpeerreviewhavebeenheld,withpapersrevisedandre-reviewedandappear-ingintheJournaloftheAmericanMedicalAssociation(JAMA)AMA)Rennie1990,RennieFlanagin199.Rele-vantpapersfromthesecongressesarediscussedinthefollowingsections.Theseliteraturereviewsemphasizethreeprimaryas-pectsrelevanttoblindreviewing,fairnesstoauthors(tounknownauthorso
rtoauthorsafliatedwithunknowninstitutions,toless-publishedortoprocientauthors,tobothgenders),reviewuality,andblindingefcacy.Thefollowingsectionswilladdresseachaspectinturn. theunacceptabilityoftheresubmittedmanuscripts,cou-pledwiththepresumablynearperfectagreementamongtheoriginalreviewersinfavorofpublishing,providead-ditionalconvergentsupportfortheresponsebiashypoth-esisisPetersCeci1982,page192.Theirproposedso-lutionIfinstitutionalafliationorprofessionalstatuscaninfactbiaspeerreview-andthisbiasprovestohavenovalidity,ornegativevalidity-thenonepossiblesolu-tiontothisproblem(asseveralcriticshaverecommended)wouldbetoestablishblindreviewsasstandardjournalpolicyyibid,page19AseminaleperimentmentBlank1991demonstratedsta-tusbiasinreviewingmoredirectly.Inthiseperiment,everyotherpaperthatarrivedattheAmericanEconomicReviewwasdesignatedasdouble-blind.Forthesepapers,aneditorialassistantremovedthenameandafliationoftheauthorfromthetitlepageandtypicallyscannedtherstpageforadditionaltitlesornotesthatwouldidentifytheauthor(i.e.,editorialmasking).Thiseperimentlastedfortwoyears.Therelevantissuewaswhethertheratioofaccep-tanceratesbetweeninstitutionalranksintheblindsam-plediffersfromthecorrespondingratiointhenonblindsample..Blank1991,page10.Itwasfoundthatthisratiodidnotdifferforthoseattop-rankeddepart-mentsandthoseatcollegesandlow-rankeduniversities.Allothergroups,inthatimportantgrayareawhereed-itorialjudgmentismostneeded,hadsubstantiallyloweracceptanceratesintheblindsamplethaninthenonblindsampleinsomecases,theacceptanceratedroppedbymorethan7percentagepoints.Shefoundsimilardiffer-enceswithrefereeratingsbetweenSBRandDBR.Aretrospectivestudyofsingle-blindreviewsfortheJournalofPediatricsandpublishedinJAMAfoundonlypartialevidenceforstatusbias,thatforthe17briefre-ports,lowerinstitutionalrankwasassociatedwithlowerratesofrecommendationforacceptancebyreviewersP.)....Forthe28majorpapers,however,therewasnosignicantrelationshipbetweeninstitutionalrankandeitherthereviewersrecommendations(ortheacceptancerate((Garfunkeletal.199page138Anotherretrospectiveanalysisofsingle-blindreviewsalsopublishedinJAMAfoundevidenceofstatusbiasatacoarsegeographicallevellLink1998.Inthisanalysisoforigina
lresearcharticlessubmittedtoGastroenterologyduring199and199,itwasfoundthatreviewersfromtheUnitedStatesandoutsidetheUnitedStatesevaluatenon-USpaperssimilarlyandevaluatepaperssubmittedbyUSauthorsmorefavorably,withUSreviewershav-ingasignicantpreferenceforUSpaperspapersLink1998,page2Theeperimentalevidenceismiedconcerningstatusbiaspresentfortop-rankedauthorsandinstitutions.Theevidenceisuitecompellingthatstatusbiasispossible,perhapsprevalent,inSBRformostotherauthorsandin-stitutions,presumablyforthosepapersmostneedingthecriticalevaluationofreviewers.4.2FairnesstoProliÞcAuthorsTherehavebeenseveralstudiesthathavelookedattheimpactofblindingonprolicauthors,withconictingre-sults.TheMahoneyeperimentdiscussedinthepre-vioussectionalsosuggestedthatself-citationmaybeadeterminantofareviewersevaluationofamanuscripttMahoneyetal.1978,page70.Inhalfofthepaperssenttovolunteerreviewers,theauthorde-fendedhiscontentionsbyreferencingthreeofhisownpresspublications.Fortheotherhalf,thesesamepre-publicationreferenceswerealsocited,butwereattributedtosomeoneelse....Reviewersratedthearticleasmoreinnovativeandpublishableifthectitiousauthorincludedself-referencesinthemanuscriptthanifnoself-referenceswereincludedudedibidAneperimentpublishedinJAMA7consecutivemanuscriptssubmittedtotheJournalofDevelopmentandBehavioralPediatricsthatwererandomlyassignedtoei-therblindedorunblindedreview(thatis,usingeditorialmasking)foundthatcontrarytotheoriginalhypothesisofthisstudy,seniorauthorswithmorepreviousarticlesreceivedsignicantlybetterscoresfromtheblindedre-viewers(),butnotfromthenonblindedreview-ers((Fisheretal.199,page1.Theauthorsinterpretthisndingtoindicatethattheblindedreview-ers,especiallythosewhowerereallyblindedandcouldnotguessauthoridentity,mayhaverecognizedimprovedualityintheworkofthoseauthorswithmorepreviouspublications.Incontrast,reviewerswhowereawareofauthoridentitydidnotgivebetterscorestothemoreeriencedauthors,likelyindicatingthatvarioustypesofbiasmayhaveenteredintotheirthinkingngibid,page1Aretrospectivestudyoftwodatabaseconfer--Madden&DeWitt200foundnoimpactonpro-cauthors.Thisstudyconsideredpapersauthoredbythosedesignatedvariouslyasafamouspersoncresearchermoreseniorresearcher,nedasth
oseindividualswhohavepublished20ormorepapersinSIGMDandVLDBconferencesconferencesibid,page29.TheanalysisfoundthatDBRreviewing(withauthormask-ing)intheACMSIGMDconferencehashadessen-tiallynoimpactonthepublicationratesofmoreseniorresearchersinthedatabaseeibid,page30.Thisre-sultmirrorsthoseofBlanksstudy,whichfoundthatac-ceptanceratesfortop-rankedinstitutions(wherepresum-ablymostoftheseprolicresearchersresided)werenotaffectedbyDBR. Anindependentanalysisofthisdata,usingmediansratherthanmeans,reachedtheoppositeconclusionthatdoubleblindreviewinSIGMDdohaveitsimpactontheperformanceoffamousperson’”[Tung200.Therea-sonforthedifferingconclusionsoverthesamedatamaybethatthisdata,consistingofyearlycountsofpapersbyprolicauthorsandbyothers,istoocoarsetomakeanaldetermination,asitdoesnttakenintoconsiderationthevaryingsubmissionratesofindividualsandthevary-ingparticipationbyprolicauthors.Thesecontradictoryresultsrenderitimpossibletosayanythingdenitiveabouttheimpactofblindingonpro-cauthors.However,theredoesseemtobeevidenceofsomekindsofbiaswithSBR.4.3GenderEquityWhenreviewersknowtheidentityoftheauthor(s)ofthesubmittedmanuscript,genderbiasisalsoapossibility.Severaldisciplineshavelaunchedin-depthstudiesbasedonconcernsofgendereuity.Aclassicandmuch-referencedstudyshowedthatevenwhentheworkofawomanwasidenticaltothatofaman,theformerwasjudgedtobeinferiorrGoldberg19.Inthisstudy,scholarlyessaysinanumberofacademicwerepresentedtofemalecollegestudents.Allofthestu-dentsratedthesameessays,buthalfofthemratedessaysbearingthenamesofmaleauthors(e.g.,JohnT.McKay),whereastheotherhalfratedthesameessayswiththenamesoffemaleauthors(e.g.,JoanT.McKay).Theresultsindicatedthatthoseessayswheretheauthorwasmalewasratedhigher.uantitativemeta-analysisofthisandsimilarstud-ies(overonehundred)overtheinterveningtwodecadesfoundthattheaveragedifferencebetweenratingsofmenandwomenisnegligibleleSwimetal.1989,pageConsistentwiththisanalysis,73ofstudiesfoundnosig-canteffectfortheJoan-Johnmanipulation,20thatJohnsworkwasratedhigher,andtheremaining7foundthatJoansworkwasratedhigher.Interestingly,therewassomeindication,however,thatwomenwillberatedlessfavorablythanmenwhenlessinformationispresentedentedibid,paget
herewasalsosomein-dicationofgreaterbiaswhenthestimulusmaterialwasaresumeorapplicationnibid,page.Morerele-vanttotheissueofpeerreviewistheobservationfromtheAPAtaskforcereportthatwhenJoanandJohnworkwashighinuality,theeffectsizewasclosetozerozerothenegativesignindicatingalowerevaluationoffemale-authoredworktheeffectwaslargerwhenJoanandJohnsworkwasmediuminuality(-.2)....theseresultsseemtoindicatethatevaluationofabsolutelyout-standingarticleswillnotbebiased,butarticlesofam-biguousmeritmaybejudgedbasedontheauthorsgen--Fouadetal.2000,page.Thisisagainconsistentwithpreviously-discussedstudiesthatconsideredstatusbias.Blankperiment,describedearlier,wasinfactini-tiatedduetoconcernsofgenderbias.TheAmericanEco-nomicReviewjournalhademployedSBRformostofitsrecenthistory,eceptduringaperiodof1973whenthethen-currenteditoradoptedDBRRBorts197Inthemid-1980s,theAmericanEconomicAssociationCommitteeontheStatusofWomenintheEconomicsProfessionformallyepresseditsconcernaboutthepo-tentialnegativeeffectonwomensacceptanceratesofasingle-blindsystemmBlank1991,page10.Asare-sult,BlankwasaskedbythecurrenteditoroftheandtheBoardofEditorstodesignandrunarandomizedperimentlookingintothispotentialeffect.Duetothecarefulrandomizationdesignofthiseperiment,onecancompareacceptanceratesbetweentheblindandnonblindsamples,andindeed,therewerestrik-ingdifferences.Forwomen,thereisnosignicantdif-ferenceinacceptanceratesbetweenthetwosamples.Formen,acceptanceratesaresignicantlyhigherinthenon-blindsample.ample.ibid,page10.Whenreviewersknewthatthatpaperwasauthoredbyamale,theyacceptedahigherpercentage(1,versus11)thanifthepaperwasblinded.necancompareacceptanceratesbetweentheblindandnonblindsampleswithoutothercontrolvari-ablesbecausetherandomizationprocessguaranteesthatpapersbywomen(andmen)ineachsamplehaveidenti-caldistributionsofcharacteristicsticsibidBlankemphasizedthecoreissuewhethertheratioofmaletofemaleacceptanceratesinthenonblindsam-pleisdifferentfromthatintheblindsample.Inbothsamples,womensacceptanceratesarelowerthanmenbutthedifferentialintheblindsampleissmaller.Whilewomenintheblindsamplehaveanacceptancerateonly1percentagepointbelowthatofmen,theirrateis3.8per-centagepointslowerinthenon-blindsampleeibid.
Heretheresultswerestatisticallyinsignicant,perhapsbecausethereweretoofewobservationsofpapersauthoredbywomen.WouldDBRresultinalargeincreaseinacceptancesofpapersbywomenWhilethereissomeindicationinthesedatathatwomendoslightlybetterunderadouble-blindsystem,bothintermsofacceptanceratesandrefereeratings,theseeffectsarerelativelysmallandstatisticallyinsignicant.Thus,thispaperprovideslittleevidencethatmovingtoadouble-blindreviewingsystemwillsubstan-tiallyincreasetheacceptancerateforpapersbyfemaleeconomistssibid,page10.Interestingly,theAmeri-canEconomicReviewnowemploysDBR.TheModernLanguageAssociations(MLA)eperi-encewasstrikinggoingtoDBRresultedinalargein-creaseinacceptancesbyfemaleauthors.Contributed scientistswhohavedonelargeamountsofgoodualityworkintherecentpastarelikelytocontinuedoingsointhenearfuturereibid,page11Thereisevidencethatevenwiththispotentiallyuse-fulinformation,journalsusingSBRpublishalargerfractionofpapersthatshouldnothavebeenpublishedthandojournalsusingDBRRLaband199,page1Thisstudyusednonlinearregressionandorderedpro-bittechniuestoestimatetheimpactofDBRoncita-tionsofasampleof101articlespublishedin28eco-nomicsjournalsduring198.Theanalysisfoundthatarticlesreviewedsingle-blindarelesslikelythanthoserevieweddouble-blindtobeidentiedcorrectlyasthehighest-impactarticles(thosewithnineormorecitationsintheensuingyears).Bythesametoken,articlesre-viewedsingle-blindaremorelikelythanthoserevieweddouble-blindtobemisidentiedasthelowest-impactpa-pers(thosewithnocitationsintheensuingyears)....Weconcludethatthesingle-blindreviewprocessappar-entlysuffersfromatypeIerrorbiastoagreateretentthanthedouble-blindreviewprocess..ibid,page1SeverallimitationsoftheLabandstudyhavebeenpointedoutTherearedifcultieswiththisanalysis,themainonebeingthatthepapersconsideredwereonlyre-viewedinoneway,eitherblindedornotblinded.Also,controllingforthestatusofthejournalinwhicheachpapersappearedisinevitablyadifcultprocess.Pa-persselectedforthemarketleaderjournalsbywhat-everprocessmustbemorelikelytobecitedthanthoseselectedformorespecialisedorlesswellrespectedcom-petitorsrsPoutney199,page10Theevidenceisthusverymiedaboutwhetherinfor-mationaboutprioraccomplishments,coupledwiththeob-servedcor
relationwithfutureaccomplishments,resultsinbetterjudgmentaboutaspecicsubmissionbeforeare-viewer.Somearguethatinformationabouttheauthorsinstitutionalafliationhelpsrefereesevaluatemanuscriptsbecausetheyconstitutepresumptivethatthere-searchdescribedwasactuallydonedoneCampanario1998b,page29.IthasalsobeenobservedthatSomerefer-eesbelievethattheycanjudgebetteriftheyknowtheauthorbecausethemanuscriptcanbeevaluatedinthecontetoftheauthorsentirecorpusofwork,butthisclaimisrare.MorefreuentisadvocacyofanonymityforauthorshorsDalton199,page23.Anotherasserted,itshouldbetheworkitself,andnotthereputationoftheauthor,whichinuences...Asstatisticians,oneofourimsisthatthedatashouldspeakforthemselves,solikewiseshouldwelettheworkspeakforitselfwithoutundueinuencefromoutsidepressuresresBillard1993WenoweaminethescienticevidencethatDBRcanin-creasereviewuality.AstudyofsityarticlesdrawnfromtheJournalorAbnormalPsychologyfoundthatthearticlesbyscholarsliatedwithhigh-statusinstitutionswerecitedconsider-ablymoreoftenthanthearticlesbyscholarsatlow-statusinstitutionssPerlman1982Therefore,itappearsthataninstitutionsprestigeisavalidpredictor,andeditorsmaybejustiedinusingthisasafactorintheirdecisionmaking.Advocatesofblindreview,however,maystillobjecttousingeitherinstitutionalafliationoranindi-vidualsreputationascriteriainselectingarticles.Theycouldclaimthattheecellenceofthemanuscriptshouldnotonlybeapparentovertime,itshouldalsobeimmedi-atelyapparentwithouttheaidofstatuscues.Thus,evenwithablindreviewprocess,assessorsshouldidentifyahigherproportionofitemssubmittedbyscholarsatpres-tigiousinstitutionsasworthyofpublicationpublicationibidAnotherdouble-blindstudyofDBR,carriedoutattheJournalofGeneralInternalMedicine,foundthatblind-ingreviewersimprovestheualityofreviewfromtheed-itorsperspectiveeMcNuttetal.1990,page137seealsooEvansetal.1990.Specically,editorsgradedtheualityofblindedreviewsbetteronthreeofthefourual-itydimensionsfromtheeditorsperspectiveimportanceoftheuestion,targetingkeyissues,andmethods(all.02).Thegreatestdifferencewasnotedinthegradesformethods.Editorsgradedblindedreviewsfromtheauthorspointofviewstatisticallysignicantlybetterononlyoneoftheualitymeasurestheblindedreviewerwasgradedasm
oreknowledgeable().Thegradesontheotherdimensionsofualityfavoredtheblindedre-viewer,eceptforcourteousness.Theeditorssummarygrades,takingintoaccountbotheditorsandauthorpointsofview,favoredtheblindedreviewers.Themeansummarygradewas3.forblindedreviewersand3.1forunblindedreviewers.Themeandifferencebetweentheblindedandunblindedreviewerswas0.Thedifferencebetweenthemediangradeforblindedandunblindedreviewerswas.0vs3.0,respectively,anentiregraderadeMcNuttetal.1990,pp.1373.TwolimitationshavebeennotededFletcher&Fletcher1997nedifcultywiththestudyisthattherefereesre-ceivingtheblindedcopyofthemanuscriptwouldhavebeenawarethattheywerepartofaneperimentandmayinconseuencehavebeenmorecarefulwiththeirre-portssCoxetal.1993,page31Thestudyrecognizedthattheresultsmayhavebeeninuencedbythenatureofthejournalnotamarketleader,butwithaverywideeditorialremitandthatuitedifferentresultsmightbefoundinsimilaranalysesoflargejournals,sub-specialtyjournalsandbasicsciencejournalssPoutney199,pageHowever,fourotherstudiesdidnotndthatmaskingpeerreviewerstoauthoridentityimprovestheualityofpeerreview.InaneperimentalsopublishedinJAMA,tworandomizationswereperformedforsubmissions overasi-monthperiodtovebiomedicaljour-nalssJusticeetal.1998.Therstassignedauarterofthesubmissionstothejournalsusualpractice,whichforfourofthejournalswasSBR.Fortherestofthesubmissions,oneofthetworeviewerswasrandomlyse-lectedtoreceiveamanuscriptthathadbeenmasked,byremovingauthorandinstitutionalidentityfromthetitlepage,runningheadersorfooters,andacknowledgmentsofthemanuscripts.Self-referencesinthetetwerenotremoved(thatis,editorialmasking)ng)ibid,page2uestionnaireswereprovidedtoeditors,authors,andre-viewers.Analysisoftheseuestionnairesrevealedthatauthorsandeditorsperceivednosignicantdifferenceualitybetweenmaskedandunmaskedreviews.Wealsofoundnodifferenceinthedegreetowhichthereviewuencedtheeditorialdecision....Whenanalysiswasrestrictedtomanuscriptsthatweresuccessfullymasked,reviewualityasassessedbyeditorsandauthorsstilldidnotdiffereribid,page2ThreeotherstudiesudiesTobiasibrin1978,vanRooyen1999,Smithetal.2002,intheeldsofspe-cializedmedicineandeducation,foundsimilar,negativeresultsItisimportanttonotethatthesestudiesutilizededi-
torialmasking.Wewillshowshortlythatmaskingsuc-cessdependshighlyonhowthatmaskingisdone.TheauthorsoftheJAMAstudyspeculatethatpooroverallmaskingsuccess,incombinationwiththeobservationthatanauthorsrenownisstronglyassociatedwithmaskingfailure,maycontributetothislackofadifferencebe-tweenunblindedandblindedreviewswsJusticeetal.1998,page2nemustconcludethatthejuryisstillout.IthasnotbeenshownconvincinglythateitherSBRorDBRcan,byrevealingorbyhidingtheidentityoftheauthorandinsti-tution,increasetheualityofthereviewsofasubmittedmanuscript.6EfÞcacyofBlindingAnybenetsascribedtodouble-blindreviewingassumethattheblindingofthesubmittedmanuscripthasbeensuccessful,thatreviewerscannotinfactidentifytheau-thor(s)northeirinstitutions.Howtrulyanonymousanypartycanbeinaworldinwhichrefereesareselectedfortheirin-depthknowledgeofasmallsliceoftheuniverseofknowledgeisopentouestionnDalton199,page23Thenotionthataneperimentedrefereecanidentifytheauthorofagivenpaperinaspecialtyjournalhasbeenusedbymanytoderogatetheclaimofanadvantagetodouble-blindreviewwCampanario1998b,page29Weearlierdifferentiatededitorialandauthorblinding.Eachcanbedoneinvariousways.Tosimplyblockoutthenameandafliationfromthetitlepagereuiresmini-maleffort,toblockoutself-referencesaddsalittlemore,andscrutinizingthemanuscriptforanyinternalcuesne-cessitateslaboriousline-by-linestudy.Therefore,theef-cacyoftheblindingprocesswillvarydirectlywiththeeffortependedonittPitkin199,page781Severalstudieshaveeaminedhowwellblindingworks.Thesestudies,acrossawiderangeofdisciplines,observedthatblindingachievedsuccessratesof.Wenowreviewthestudieschronologically.AstudyofreviewersofpaperssubmittedtotheJour-nalofSocialServiceResearchduring1978showedthatofthereviews,therefereesdidnotventureaguessastotheidentityoftheauthor.Inanother,therefereesguessedwrong.Inanadditional,therefer-eesmadecorrectguessesaboutsomebitofidentifyinginformation,buttheydidnotguessthenameoftheau-thorhorRosenblattKirk1980,page389.Thisworksouttosuccessfulblindingofthetime,foreditorialblindinginwhichthenamesofauthorsandtheirinstitu-tionalafliationsareremovedfromthetitlepage.neretrospectivestudyoversijournalsemployingDBRwithauthorblindingandrepresentingabroadrangeofareasin
psychologyshowedthatofthe1viewerswerecorrectintheiridenticationsoftheauthororofatleastoneoftheauthorsinthecaseofmultiau-thoredpapers....Therewerenosignicantdifferencesintheproportionofcorrectdetectionsamongthesijour-nals,rangingfrom2...norwasthereanyre-lationshipbetweendetectionaccuracyandthenumberofyearsofreviewingeeCeciPeters198,page.Wheneditorialstaffoversightsinnotremovingtitlepagesofmanuscriptsbeforesendingthemtoreview-ersorauthorsoversightsinpreparingtheirmanuscripts,suchaseplicitaggingofformerwork(Inourearlierwork...)orinappropriateinclusionofpersonalacknowl-edgmentsinthebodyofthetet...areecludedfromtheanalysis,overallonly2ofreviewersareabletodetectauthorsidentities,withverylittlevariationamongthesijournalssibidIntheMcNuttstudydescribedintheprevioussec-tion,editorialblindingwassuccessfultoinstitutionnamefor73ofthereviewersandtoauthor(s)namesforofthrreviewerswersMcNuttetal.1990.Theblind-ingprocesswasofmoderatethoroughnessAneditorialassistantcopiedeachmanuscript,retypedthetitlepage,andremovedauthorsandinstitutionsidentiersusinganopauetape.Todothis,theentiremanuscriptwasscannedheadersandfooters,bodyoftet,tables,andgures.Shemadenoattempttoremovereferencestotheauthorsownwork....Minimalchangesweremade,onaverage,tothebodyofthemanuscripttibid,page1372SubmissionstotheAmericanJournalofPublicHealtharepartiallyauthorblindedandpartiallyeditori- thorsandfailurestoscreenmanuscriptsbyeditorialstaff,weareimpressedbytheoverallefciencyofblindre-viewwCeciPeters198,page17RecommendationsofScholarlySeveralscholarlysocietieshaveweighedinonthisues-tion.AneaminationpublishedintheJournalofBusinessofpastabusesoftheeditorialprocesswarrantedtheproposaltousethedoubleblindprocessinwhichrefereesdonotknowauthorsnorauthors,refereesastoprotectrefereesfrominuenceandparticipantsfromdamageinanegativedecisiononCarlandetal.1992,page103TheInstituteofMathematicalStatistics(IMS)NewResearchersCommittee(NRC)reportstated,afteretensivediscussion,theconsensusoftheNRCisthattheadvantagesofthedouble-blindsystemoutweighthecosts,andwerecommendthatIMSjournalsevaluatethetsofadoptingsuchasystememAltmanetal.1991,page1.Inaresponsetodiscussantsofthisre-port,theNRCreassertedayearlater,thattheNRCstro
nglysupportsdouble-blindrefereeingforitspoten-tialtoremoveseparateconsideration,perceivedoroth-erwiseeAltmanetal.1992,page2TheviewoftheIMSAdHocCommitteeonDouble-BlindRefereeingin1993maybesummarizedascau-tiouslyreceptivetodouble-blindrefereeing.Wearenotconvincedthatthebenetsoutweighthedisadvan-tagesbutweareopentothepossibility.Werecom-mendthatifachangeinjournalpolicyiscontemplatedthataneperimentbeconductedtoassessthemeritsofdouble-blindrefereeingbeforeanypermanentchangeissCoxetal.1993,page311TheAPATaskForceonWomeninAcademestatesthatBecauseofthepotentialforbias,APAhasman-datedthateditorsofAPAjournalsoffermaskedreviewasanoptionhowever,mandatorymaskedreviewofar-ticlesshouldbeinstitutedaspolicyyFouadetal.2000,page3.Thereportincludesinitsrecommendations,ImplementapolicyofmandatorymaskedreviewforallAPApeer-reviewedpublicationspublicationsibid,page.TheAPAGuidestates,MostAPAjournals,likethemajorityofotherprofessionalpublications,practiceanonymous,ormaskedreviewssCalfeeValencia200ftheAPAjournalsthatacceptpapersubmissions,almosthalfuiremaskingandmostoftherestallowitonreuest.Thereisapatternhere.Itisthereforeonlytobepectedthatseniorestablishedresearcherswilltendtoseekthestatusuo,beinglessinclinedtowanttomovetodouble-blindrefereeing,whilenew(andalsowomenandresearchersinlowerstatusnamedinstitutions)researcherswilltendtopreferthatdouble-blindrefereeingbeintro-ro-Billard1993,page3228PrevalenceDisciplinesvarywidelyintheiruseofsingle-anddouble-blindreview,butthehistoricaltrendisclear.Anon-randomizedsurveyin1989showedthatchemistry,physics,math,andpsychology,therespond-ingjournalsindicatethattheyuseasingle-blindre-viewingsystem....Biologyappearstohavebothsingle-blindanddouble-blindjournals,asdoeshistoryandanthropology.Political-scienceandsociologyjour-nalsreportuniformlyusingdouble-blindreviewingmeth--Blank1991,pp.10verall,considering37journalsinninedisciplines,79weresingle-blindandweredouble-blind.Foursurveysofthefreuencyofbindinghavebeenpublished,butnoneisbasedonarandomsamplingofjournals,andtheirsizeandresponseratesoftenleavesomethingtobedesired.Theirresultssuggestthatthemajorityofscienticjournalsdonotpracticeblindre-viewandthatblindingmaybemorecommoninthesocialsci
encesthaninthephysicalandmedicalsci--Yankauer1991,page8Amorerecentsurveyof3journalsselectedfromeighteendisciplinesrevealedthatDBRisincreas-inginprevalence,incomparisonwithpreviousstud-Acrossthedisciplines,themajorityofsurveyedjournalsuseddouble-blindreviews(),37ployedsingle-blind,andonlymadeuseofopenre-viewwBachandSawallis2003,page.Withincom-puterscience(29journals,1responding),single-blindandweredouble-blind.WithinACM,alljournalsaresingle-blindandaboutofACMconferencesaresingle-blind(asofofSnodgrass2000Similarly,thedatabaseeldhastraditionallyreliedonsingle-blindreviewing.Untilrecently,allofitscon-ferencesandjournalshavebeensingle-blind.TheACMSIGMDconferenceadopteddouble-blindreviewinginnibid9SummaryThereisalongtraditionattachedtothepeerreviewsys-tem.Asusersofscience,wealldependonitourpro-fessionalrealizationsarebasedupontheworkofothers,andwecountonjournal(andbook)editorstoseparatethewheatfromthetares.Althoughthereisnosuchthingasperfection,itwouldbeadisservicetotheprofessioniftoomanyscienticwritingsaddressedirrelevantissuesor Blank1991R.M.Blank,TheEffectsofDouble-BlindversusSingle-BlindReviewingperimentalEvidencefromTheAmericanEconomicReview,AmericanEconomicReview7,December1991.1991.Borts197G.Borts,ReportoftheManagingEditor,AmericanEconomicReview64:482,May197197Campanario1998aJ.M.Campanario,PeerReviewforJournalsatItStandsTodayPart1,ScienceCommunica-tion211,March1998.1998.Campanario1998bJ.M.Campanario,PeerReviewforJournalsatItStandsTodayPart2,ScienceCommunica-tion,June1998.1998.Carlandetal.1992J.A.Carland,J.W.Carland,andC.D.Aby,Jr.,ProposedCodicationofEthicacyinthePublicationProcess,JournalofBusinessEthics,1992.1992.CeciPeters198S.J.CeciandD.P.Peters,HowblindisblindreviewAmericanPsychologisttChoetal.1998M.K.Cho,A.C.Justice,M.A.Winker,J.A.Berlin,J.F.Waecklerle,M.L.Callaham,andD.Ren-nie,MaskingauthoridentityinpeerreviewWhatfactorsinuencemaskingsuccessJournaloftheAmericanMedicalAssociation,July1,1998.1998.Coxetal.1993D.Co,L.Gleser,M.Perlman,N.Reid,andK.Roeder,ReportoftheAdHocCommitteeofDouble-BlindRefereeing,StatisticalScience317,August1993.1993.Dalton199M.Dalton,RefereeingofScholarlyWorksforPrimaryPublishing,AnnualReviewofInforma-tionScienceandTechno
logy(ARIST),Volume30,M.E.Williams(ed.),AmericanSocietyofInfor-mationScience,199199Evansetal.1990A.T.Evans,R.A.McNutt,R.H.Fletcher,andS.W.Fletcher,TheEffectsofBlindingontheualityofPeerReviewARandomizedTrial,ClinicalResearch38(2),1990.1990.Fisheretal.199M.Fisher,S.B.Friedman,andB.Strauss,Theeffectsofblindingonacceptanceofresearchpapersbypeerreview,JournaloftheAmericanMedicalAssociation,July13,199199Fletcher&Fletcher1997R.H.FletcherandS.W.Fletcher,EvidencefortheEffectivenessofPeerReview,Sci-enceandEngineeringEthics0,1997.1997.Fouadetal.2000N.Fouad,S.Brehm,C.I.Hall,M.E.Kite,J.S.Hyde,andN.F.Russo,WomeninAcademe:TwoStepsForward,OneStepBack,ReportoftheTaskForceonWomeninAcademe,AmericanPsycholog-icalAssociation,2000.http://www.apa.org/pi/wpo/academe/repthome.html,viewedJune21,200200Franzini1987L.R.Franzini,EditorsAreNotBlind,AmericanPsychologist,page10,January1987.1987.Garfunkeletal.199J.M.Garfunkel,M.H.Ulshen,H.J.Hamrick,andE.E.Lawon,EffectofInstitutionalPrestigeonReviewersRecommendationsandEditorialDecisions,JournaloftheAmericanMedicalAssociation138,July13,199199Genest1993C.Genest,Comment,StatisticalScience327,August,1993.1993.Goldberg19P.Goldberg,AresomewomenprejudicedagainstwomenTransaction30,April,1919Gordon1980M.D.Gordon,Theroleofrefereesinscienticcommunication,ThePsychologyofWrittenCommunication:SelectedReadings,J.Hartley(ed.),London,EnglandKoganPage,pp.22Harnad1982S.Harnad,Peercommentaryonpeerreview,BehavioralandBrainSciences,1982.1982.Hill&Provost2003S.HillandF.Provost,TheMythoftheDouble-BlindReview,SIGKDDExplorations,December2003. Tobiasibrin1978S.TobiasandM.ibrin,DoesBlindReviewingMakeaDifferenceEducationalRe-searcher,January1978.1978.Tung200A.K.H.Tung,ImpactofDoubleBlindReviewingonSIGMDPublicationAMoreDetailAnalysis,2pages,July200200vanRooyen1999S.vanRooyen,F.Godlee,S.Evans,R.Smith,andN.Black,EffectofBlindingandUnmaskingontheualityofPeerReviewARandomizedTrial,JournalofGeneralInternalMedicinectober1999.1999.Yankauer1991A.Yankauer,HowBlindisBlindReviewAmericanJournalofPublicHealthJuly1991.1991.uckermanMerton1971uckermanandR.K.Merton,PatternsofEvaluationinScienceInstitutionaliza-tion,StructureandFunctionsoftheRefereeSystem,Minerva:66–100,Ja