But how do researchers decide what to believe and what to trust when choosing which statistical methods to use How do they decide the credibility of methods Statisticians and statistical practitioners seem to rely on a sense of anecdotal evidence ba ID: 43744
Download Pdf The PPT/PDF document "Convincing Evidence Andrew Gelman Keith ..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
![[Ethics and Statistics]](https://thumbs.docslides.com/447196/ethics-and-statistics.jpg)
![[Ethics and Statistics]](https://thumbs.docslides.com/148817/ethics-and-statistics-575.jpg)
2.Computersimulations(forexample,demonstratingapproximatecoverageofintervalestimatesundersomerangeofdeviationsfromanassumedmodel);3.Solutionstotoyproblems(forexample,thecomparisonofRubin(1981)ofapartialpoolingestimateforatest-preparationprogramineightschoolstothenopoolingorcompletepoolingestimates);4.Improvedperformanceonbenchmarkproblems(forexample,gettingbetterpredictionsfortheBostonHousingData(HarrisonandRubinfeld,1978),anexamplemuchbelovedoftextbookwritersinstatisticsandcomputerscience);5.Cross-validationandexternalvalidationofpredictions(seeVehtariandOjanen,2012),ascanbedoneinvariousexamplesrangingfromeducationtobusinesstoelectionforecasting;6.Successasrecognizedinaeldofapplication(forexample,astatisticalmethodthatisusedandrespectedbybiologists,oreconomists,orpoliticalscientists);7.Successinthemarketplaceofsoftwareortextbooks(underthetheorythatifpeoplearewillingtopayforsomething,itislikelytohavesomethingtoo er);8.Facevalidity:whetherthemethodseemsreasonable.Thiscanbeaminimumrequirementforconsideringanewmethod.AsnotedbyGelman(2013),\Noneoftheseisenoughonitsown.Theoryandsimulationsareonlyasgoodastheirassumptions;resultsfromtoyproblemsandbenchmarksdon'tnecessarilygeneralizetoapplicationsofinterest;cross-validationandexternalvalidationcanworkforsomesortsofpredictionsbutnotothers;andsubject-matterexpertsandpayingcustomerscanbefooled.Theveryimperfectionsofeachofthesesortsofevidenceandhowtheyapplytodi erentuserpopulationsandsettingsgivesaclueastowhyitmakessensetocareaboutallofthem.Wecan'tknowforsuresoitmakessensetohavemanywaysofknowing."Informalheuristicreasoningisimportanteveninpuremathematics(Polya,1941).Thereisalsotheconcernthatastatisticalmethodwillbeuseddi erentlyintheeldthaninthelab,sotospeak|or,togivethisproblemapharmaceuticalspin,thatanewmethod,approvedforsomeparticularclassofproblems,willbeused\o -label"insomeothersetting.Rubin(1984)discussesconcernsofrecommendingmethodsofanalysisforrepeatedusebythose(andoftenourselves)withlimitedstatisticalexpertise,limitedresources,andlimitedtime.Tofurthercomplicatethis,therehaslongbeenexperimentalevidencethatoptimalmethodsofinformationprocessingdonotalwaysnotleadtooptimalhumanperformance,andthisvariesbylevelofskill,incentivesandtimepressure(DriverandStreufert,1967).Wemayalsowishtoconsiderhowweshouldchoosebetweenmethodsingivenapplicationsforourselves,torecommendtocolleaguesofsimilarordi erentlevelsoftechnicalskill,andtocommunitiesofuserswhoarenotfull-timestatisticiansorquantitativeanalysts.Thatis,howshouldwegoaboutapprovingstatisticalmethodsforuseinvariousapplicationsbyvarioususers,makingreasoned,criticalchoices.Todothisweleanonbackgroundmaterial,againfollowingthemodelofthechoiceofmedicaltreatmentsforusebyvariousprofessionalsorendusers.Ourprimaryobjectiveistomaximizetherateoflearningabouttheempiricalapplicationwhileminimizingtherateandmagnitudeofmistakes.Thesegoalsrequireasortofmeta-evidencethatisnotcapturedbyanysinglesortofinquiry.Moregenerally,wehavearguedthatstories,totheextentthattheyareanomalousandimmutable,arecentraltobuildingunderstandinginsocialscience(GelmanandBasb ll,2013).2 level,accesstosoftware,budgetlimitationsofclient)thananyputativeevidence.Aregulatoryperspectivewouldattempttosetthisaside,atleastuntilthebenetsandharmsofmethodshavebeenassessedalongwiththeirrelativevalue/importanceandtherealuncertaintiesabouttheseareclaried.Wearenotliterallysuggestingthatstatisticalmethodsbesubjecttoregulatoryapprovalbutratherthatthisperspectivecanhelpusmakesenseofthemixofinformationavailableaboutthee ectivenessofdi erentresearchmethods.Morerecently,ithasbeenarguedthattruthcomesfrom\bigdata"(seeHardy,2013,foracontraryview).Weagreewiththesaying\datatrumpsanalysis"butinpracticeitcanbeeasiertoworkwithsmalldatasetsweunderstandthanwithlargedatasetswithunknownselectionbiases.Forexample,inouranalysisofhomeradonlevels(Linetal.,1999),weusednationalandstate-levelrandomsamplesurveysofabout80,000homes(whichsoundslikealotbutisnotthatmuchwhenyouconsiderthatradonconcentrationsvaryspatially,andthereareover3000countiesintheUnitedStates),ignoringmillionsofmeasurementsthatwerecollectedbyindividualhomeowners,buyers,andsellers,outsideofanysurveycontext.Wesuspectthatwerewetohavereadyaccesstothelargesetofself-selecteddata,itwouldbepossibletoperformsomeanalysistocalibratewithrespecttothemorecarefullygatheredmeasurementsandgetthebestofbothworlds.Inthatparticularexample,however,wedoubtthiswilleverhappenbecausethereisnotsuchasenseofurgencyabouttheproblem;ourimpressionisthatjustabouteveryonewhoisworriedaboutradonhasalreadyhadtheirhousemeasured.Forotherproblemssuchasmedicaltreatments(or,inthebusinessworld,socialadvertising),wesuspectthatmuchcanbelearned(indeed,isalreadybeinglearned)bycombiningexperimentalmeasurementswithlargeravailableobservationaldata(see,forexample,Kaizar,2011,andChen,Owen,andShi,2013).Thisarticleisappearinginavolumewhosegoalistoconsider\afutureagendafortheoreticalormethodologicalresearchonauthorship,functionalroles,reputation,andcredibilityonsocialmedia."Wedonothaveaclearsenseofwhatthisagendashouldbe,butwethinkitimportanttorecognizethedisconnectbetweenourocialandunocialmodesofreasoninginstatistics,andthemanydi erentsourcesofpracticalevidenceweusetomakeprofessionaldecisions.Afruitfuldirectionoffutureresearchcouldbetheformalizationofsomeofourinformalrules,muchinthewaythatRosenbaum(2010)formalizedandcritiquedthewell-knownrulesofHill(1965)inepidemiology.ReferencesChen,A.,Owen,A.B.,andShi,M.(2013).Dataenrichedlinearregression.http://arxiv.org/abs/1304.1837Driver,M.J.,andStreufert,S.(1969).Integrativecomplexity:anapproachtoindividualsandgroupsasinformation-processingsystems.AdministrativeScienceQuarterly14,272{285.Gelman,A.(2006).Priordistributionsforvarianceparametersinhierarchicalmodels.BayesianAnalysis1,515{533.Gelman,A.(2013).Howdowechooseourdefaultmethods?FortheCommitteeofPresidentsofStatisticalSocieties50thanniversaryvolume.Gelman,A.,andBasb ll,T.(2013).Whendostorieswork?Evidenceandillustrationinthesocialsciences.Technicalreport,DepartmentofStatistics,ColumbiaUniversity.Ghitza,Y.,andGelman,A.(2013).DeepinteractionswithMRP:Electionturnoutandvotingpatternsamongsmallelectoralsubgroups.AmericanJournalofPoliticalScience.Gelman,A.,andLoken,E.(2012).Statisticians:Whenweteach,wedon'tpracticewhatwepreach.Chance25(1),47{48.4