Symposium September symposium speaks to the purposes of PS Political Sci ence Politics - PDF document

the purposes & many ends. It is a politics striving cessible analyses and on current issues and gogic journal within teaching, train- and curriculum development. Replication, Replication ~ing,' Haward University Political science is process by the analysis adequate docu- often have later. Since mation is usually others, even with their quantitative and qualitative such as a forum scientists from every type recodes formed? Precisely which used? What were the exact rules used for the content analysis? When your study were they cho- What statistical used? What method you choose? Producing a such questions prove conse- quential, is virtually impossible. this reason, quantitative ana- Altman, Pasteur as Liar Who Stole Ideas," The sec. Geison, Gerald The Private University Press. mation exists with which evaluate, and upon a a third Keo- hane, and Verba 1994), is also real- world phenomena real world essentially descriptive quantitative and qualitative these data. phases are with only to be at the conclusions. In should always at- PS: Political Science & Politics Replication, Replication tempted. In standard can malung that one's description is sufficiently the data and much less may not be possible collection phase, inasmuch have changed time a future researcher undertakes excellent example Thursby, and Anderson the authors Monq, After accepting a revealing their extensive but largely failed to replicate articles. Their that inad- vertent errors published empiri- cal articles articles rare occurrence." Even when they found no errors, replication was often impossible the authors not provided. More portant, when standard, they that the their programs for submission reveals sights which undetected." Since collect far relying primarily existing from government a replica- tion policy even more substantial than striking example demonstrates, the ure to standard poses serious its most an article book cannot what use to the conclusions? What an article protection should keep researchers from wasting their time reading these works. At worst, vastly more expand, extend, has no More generally, understand and research, and select developing their agendas. Without complete about where data come from and how from it, truly understand wise requires at least book editors extremely important ther development most common and productive method precise path taken ensures that second researcher researcher's hard academics refer articles and with citations, acknowledgments, promotions, and when the Reproducing and high-quality existing also an extremely useful pedagogi- tool, albeit science students only infrequently interested motivation: an article read less often, frequently, and and pride an article book, only profession, up by other researchers, not used to upon for other contexts. Moreover, applauded, cited favorably, criticized, equally strong you are debate (see citing Diamond and Leimer and that the cal science is cited (Hamilton 1994)! smaller fraction articles stimu- This problem greatly collective knowledge ment and Academia is enterprise that is usually researchers compete contributing toward com- contrast, when the same problem from measurement techniques and meth- most important, rather than peatedly reinventing Proposed Solutions empirical political formal rules also needed. Academics, administrators, review- editors can read, understood, taken seriously (It is be able own work from memory.) involve putting about the and reported. Unfortu- nately, journals and ally will this properly. More- material neces- on paper. archives, described below, can readily be Video Archive University and Consortium for Research at is to replication data replicate empirical ers, these recodes, extracts of existing publi- cally data (or directions for obtain exactly the same used), and "read-me" that de- scribes what included and reproduce the information avail- able, only the subset actually used can be analogously vis-a-vis qualitative research. A replication data set for qualitative projects should include detailed descriptions rules followed, inter- views conducted, a replication data set. qualitative re- and sometimes Replication data followed. But detailed qualitative not adhering is still worth the also be worthwhile discuss collectively standard (see Ragin a replication been created, should be made publicly available reference to original publication One approach make the on request, but be inconvenient rarely profes- archivists. Their complicates self-distribution, earlier pub- not remain using professional entails rou- off-site dupli- a different building technology changes), frequent the author's remain widely methods for ing this information asks, and functions for Public Affairs University and "Publication-Related Archive" and Social Research data sets. PAVA is the more up-to-date archive. Staff will available within such servers "anonymous FTP" "Mosaic." world, with an In- has free, these data. ICPSR is and bet- the two staff will keep and data via other classes is willing provide additional documentation; additional advertising. has some ICPSR is known institution options. More- either archive via disk Video Archive, Pur- due University, 1000 Liberal Arts Building, West Lafayette, IN 47907-1000) and/or PS: Political & Politics easier and self-ex- tracting archive (with a PKZIP the DOS system, TAR Unix, or Stu£fIt for the MacIntosh) and sub- mit the data via anonymous FTP; citation, and a graph describing the contents also be send electronic the ICPSR, publincoming send electronic a replication the archive, it will through their publications and also appear to the Tenure and Review Committees contributions to community. Adherence to the standard should this standard are more have their research better be taken more seriously scholarly peers. addition, how- ever, candidates submit their a national nized for that scholars an extra sec- curriculum vitae contribution would more clearly noticed, and should be committee, departmental, university decisions tenure. Outside letter-writ- also make graduate programs can also encourage adher- the replication strengthen stu- and ultimately professional "pub- lication" for Ph.D. dissertation. This is intended be an knowledge and scholarly community. work, students submit replica- for their Graduate programs can also adopt rules that require dissertation submit replication appropriate. In graduate programs cialize and professionalize into the PAVA will acceDt re~lication L L a period Government at Harvard University, quantitative and, applicable, qualitative disser- tations must submit replication a requirement Ph.D. is important that the stu- and submit data set the data the student the stu- a head since our been adopted, have opted short embargo or none the results existing articles be an Many economics graduate Ph.D. replicate a for their second-year paper. This practice more widely Books and journals and university and commercial presses work scholarship that important contributions reasons described above, standard are more meet these their journal or book series a replication publication indicating archive they replicate their numerical the date some authors, ment explaining this rule, it, could In this evaluative process editor accordingly. their informants, and for However, these ment, because editors or the footnote, set has been submitted the results can be replication claims and should the convenience and editorial policy like sample text quantitative articles articles books at this press] must indicate archive they deposit the information replicate their numerical date when be submitted. information deposited should include items programs, lists recodes, that describes what included and explains to re- the exact the published work. Authors may the Public at Purdue sity or the Consortium for Political and Research (ICPSR) at the University places to deposit their data. Statements ex- the inappropriateness for a work (or indeterminate periods the data or portions requirement. Peer review- be asked to assess general evalua- upon qualitative data to sub- comparable footnote that the names stricter requirements. may be appropriate not usu- that the data public archive. submission at and have that the be prepared a maxi- Sciences even mends that data available during review process. experiment with asking ate student an editorial to the accepted but not articles. Reviewers make suggestions the authors incorporated before clearer for future scholars. experiments would be very and future scholars. The exact requirement should individual jour- nals and in politi- cal science less restrictive the norms Important Exceptions presumption should that authors provide free data, the combination with the author, always have Exceptions are essen- first publication and for other reasons. these exceptions probably not apply in not normally names and addresses respondents. In and related cases, authors comply with the rep- requirement by releasing graduate admissions, process (King, Bruce, In fact, paper, since for prospective expected grades the expected mission would also have unpopular decision at general counsel's cases such the excep- some rare tiality could publicly available. interviews among a small other surveys population, potentially pose this problem. In cases, the be able the data restrict their gously, specific analysis rules have been avoid revealing individual cross-tabu- lations with fewer important exceptions, but too cover comparatively few because they as survey Center. However, be re- distributed by other authors some way, such recodes. Rights of First Publication an em- bargo for specified period Political Science & Politics one to only the data actu- a publication. you conducted 10 for an only those a five-category the categories, you could provide recoded with only the two (perhaps by the South- states), you only the Then you remaining information You certainly could provide which would probably make your more valuable to the schol- community, but would remain with cases, authors embargo the subset clean, docu- archive the data set from which extracted. This more con- investigator, and future research- the more comprehensive version the data. the data since the this proposal thors' expectations, the cur- ensure that can be usually taking some steps tions are important, but they articles published. Support for These Policies Replication Policy Adoptions Formal support year, the Meier, and require foot- notes about sets to included with editors have encountered no resis- time and also reports that 70% empirical articles he author.) The Weale, editors), and Desai and C. Shelley 11) have adopted similar policies, der the editorship the process and law is "to have a cation data is archived (and it will lute requirement and Decisions," North, has recently this policy. The Free HarperCollins done the same. Many and editorial have indicated for a replication the process Science Program's regarding replication clearly stated "All with the assistance this award force this recently adopted several new and book funds must include a which public archive replicate their numeri- it will submitted (or an be). Second, a grant awarded, the officer will ask the pro- verify that data- requirement. Third, within a principal investigators science pro- data have been deposited. officials will consider confor- their data-archiving an important renewals and new computer-read- adequate documen- connection with the (Sie- National Statistics" policies similar Fienberg et are many national differences sharing, related policies and recommendations have been international organi- zations, academic and societies, granting agencies, and scholarly Boruch and Cordray the APSA Section (edited by Alvarez and announced that ars' data-set reference, a citation the associated article, and a subsequent issues. the data stantive focuses; Laura Brown), Lee Epstein), editor), the Section Newsletter Science Newsletter (Dave Robertson, editor), the APSA interest groups Replication Policy Discussions Replication policies science community recent years. enthusiastic and three most recent annual Po- Group summer also been devoted this issue. tion unanimously passed in 1994 The APSA annual conven- tion. After on the comparativists have executive committee committee subsequently footnote. This pro- now being distributed newsletter for general comment from also been discipline's editorial and section Arneri- forums, support issues have been for implementing replication proposals discussed this paper Questions and Answers versations about raised and with the most common large and dificult Investigators receive for collecting and making scholarly commu- recognition is the data articles, acknowledge- help, and promotion, tenure, and raises. data set can only thor receives. not hav- one's ideas stolen, a familiar unfounded fear have experienced while writing having one's ignored. Submitting data set can discussed above, gathered but not used need not data set; only those variables and observa- tions necessary happens to tions submitted, In most cases, nearly two years elapse from not sufficient instance, the author can the data at a specified future date. dard will much more available through an unintended consequence this proposal their time an- alyzing existing data rather lecting new data, spending the computer lab rather Experience suggests publicly available However, content that many more data were during this time that increasing other scholars past research, made much Wouldn't it all jour- book series adopted exactly policy at the coordination, but instance argues a different style manu- a dif- articles. It should not slightly different policies regarding adopted them replication standard, won't or not publish articles save their book manuscript may be not been adopted this policy, and & Politics and explained authors, they it is their articles far more more dam- to the my data, isn't out that science moves for- ourselves vul- being wrong. pursuing a deeper understanding ment and others the wrong. Although being always pleasant, is unambiguous making a Again, being political sci- the much more serious Shouldn't editors collect replica- have been submitted? is a possibility, but no more than are authors. Editors the ICPSR. I also necessary, since future researchers. eveiyone starts submitting sets, won't archives rap- with junk? unlikely. Unlike most data future researchers. archives can harness process to value into their collections. for a byte of hard disk and a would easily replication data is in the interest of individual need journals We shouldn't need laws when cus- but experience and most that this goods problem cannot be Dew- ald, Thursby, Boruch and Cordray and Fienberg et worrying might be misting more important question actual same measure- are applied new substan- Good science requires that existing numeri- same methods the former. scholars to pursue more certain present results. existing re- able easier new contexts. Moreover, dard practice change at able to relate changes to from innumerable on previous Alt, Neal Beck, DiIulio, John Green, Matthew Holden, Gary Klass, David Laitin, Malcolm Litchfield, Rockwell, thank the National Science Foundation for SBR-9321212 and and the Guggen- Memorial Foundation for cases, the running the same data the same should probably actually involves called "replica- to the world from which the same measurements, such numerical re- due to time, place, not expect duplicate the results exactly; however, cedure confers the scientific benefit whether the substantive are systematic features the world this article, I follow common current work on social sciences, ethical considerations, the investigators, and cerns, see Sieber and Walker Fienberg et and Feigenbaum and mentioning that research. It probably exists to every other and area it is anything but been widely adopted disci- other procedures can be by teachers methods classes. term papers about through the (usually with data but few along with disk containing These are then given other stu- next week's cases, the replicator and learn a about the occasion the can be the peer review process. manuscripts also should verify which ar- a replication has been Since the footnote the contribution the schol- reviewer's responsibility. Moreover, to authors that they this footnote, and deposit their public archive, help remind scholarly contribution. also could the footnote when they send out their request Symposium References Boruch, Robert F., Cordray. Sharing." In Stephen E. Fienberg, Mar- Academy Press. and Public American Psychologist (April):414-23. William G., 1986. Empirical Economics: Journal of Money, Credit, and Banking (Sep- tember):587-603. the Appraisal al., eds. Kluwer Market for (1r)reproducible comments and Social Epistemology (July-September):215-92. Fienberg, Stephen E., Margaret and Miron Straf, eds., Washington, DC: Charles C. Ragin. sis," a 23, 1. David P. "Research Papers: Uncited Now?" p. 25. 1991. "On Political Analysis Gary, John and Michael Science Graduate Political Science & Politics 4 (Decem- ber):772-78. Gary, Robert 0. Designing Social Scientific Inference D. R., Security and Private Journal of Political Economy 90:60&29. the Social New- bury Park, Sage Publications. Herrnson,' promote research and develop community within community and way to promote research. 1995) is that these their labor without compensation. fication policy is mistakenly re- and would harm studies, which independent data tion, and fostering arrangements shared data that data and those seek to reanalyze them. interesting, and Replication" mis- states the physical and science. Replica- secondaly studies the same prob- that investigated investigator; the same data base that used by may not be used. independently collected data study the same is called same data is called study one to study Sec- Information.) Telephone conversa- Sieber, Joan E., Science Data: Newbury Sage Publi- Harvard Uni- Brad Palmquist, the ICPSR!) large collection merged precinct-level election and King can reached at Littauer Center, North Yard, Cambridge, Massachusetts gking@harvard.edu; (617) 495-2027. The current version this manuscript via gopher or anonymous FTP from haavelmo.harvard.edu. analysis frequently, depends on the use datasets. Data essential for verifications and be involved Re~lication. verification. and sec- ondab analysis used for ent purposes and require cluding independent It enables whether data study were supported under one set tions for an empirical generalizations Political Science & Politics