utahedu Abstract Compilers should be correct To improve the quality of C compilers we created Csmith a randomized testcase generation tool and spent three years using it to 64257nd compiler bugs During this period we reported more than 325 previously ID: 23024
Download Pdf The PPT/PDF document "Finding and Understanding Bugs in C Comp..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
pageoftheWebsiteforGMP,theGNUMultiplePrecisionArith-meticLibrary,states,“MostproblemswithcompilingGMPthesedaysareduetoproblemsnotinGMP,butwiththecompiler.”ImprovingthecorrectnessofCcompilersisaworthygoal:Ccodeispartofthetrustedcomputingbaseforalmosteverymoderncomputersystemincludingmission-criticalnancialserversandlife-criticalpacemakerrmware.Large-scalesource-codevericationeffortssuchastheseL4OSkernel[12]andAirbus'svericationofy-by-wiresoftware[24]canbeunderminedbyanincorrectCcompiler.TheneedforcorrectcompilersisampliedbecauseoperatingsystemsarealmostalwayswritteninCandbecauseCisusedasaportableassemblylanguage.Itistargetedbycodegeneratorsfromawidevarietyofhigh-levellanguagesincludingMatlab/Simulink,whichisusedtogeneratecodeforindustrialcontrolsystems.Despiterecentadvancesincompilerverication,testingisstillneeded.First,averiedcompilerisonlyasgoodasitsspecicationofthesourceandtargetlanguagesemantics,andthesespecicationsarethemselvescomplexanderror-prone.Second,formalvericationseldomprovidesend-to-endguarantees:“details”suchasparsers,libraries,andleI/Ousuallyremaininthetrustedcomputingbase.ThissecondpointisillustratedbyourexperienceintestingCompCert[14],averiedCcompiler.UsingCsmith,wefoundpreviouslyunknownbugsinunprovedpartsofCompCert—bugsthatcausethiscompilertosilentlyproduceincorrectcode.Ourgoalwastodiscoverserious,previouslyunknownbugs:inmainstreamCcompilerslikeGCCandLLVM;thatmanifestwhencompilingcorelanguageconstructssuchasarithmetic,arrays,loops,andfunctioncalls;targetingubiquitousarchitecturessuchasx86andx86-64;andusingmundaneoptimizationagssuchas–Oand–O2.Thispaperreportsourexperienceinachievingthisgoal.Ourrstcontributionistoadvancethestateoftheartincompilertest-casegeneration,nding—asfarasweknow—manymorepreviouslyunknowncompilerbugsthananysimilarefforthasfound.OursecondcontributionistoqualitativelyandquantitativelycharacterizethebugsfoundbyCsmith:Whatdotheylooklike?Inwhatpartsofthecompilersaretheyprimarilyfound?Howaretheydistributedacrossarangeofcompilerversions?2.CsmithCsmithbeganasaforkofRandprog[27],anexistingrandomCprogramgeneratorabout1,600lineslong.Inearlierwork,weextendedandadaptedRandprogtondbugsinCcompilers'translationofaccessestovolatile-qualiedobjects[6],resultingina7,000-lineprogram.Ourpreviouspapershowedthatinmanycases,thesebugscouldbeworkedaroundbyturningvolatile-objectaccessesintocallstohelperfunctions.Thekeyobservationwasthis:whiletherulesregardingtheaddition,elimination,andreorderingofaccessestovolatileobjectsarenotatallliketherulesgoverningordinaryvariableaccessesinC,theyarealmostidenticaltotherulesgoverningfunctioncalls.ForsometestprogramsgeneratedbyRandprog,ourrewritingprocedurewasinsufcienttocorrectadefectthatwehadfoundintheCcompiler.Ourhypothesiswasthatthiswasalwaysdueto“reg-ular”compilerbugsnotrelatedtothevolatilequalier.Toinvestigatethesecompilerdefects,weshiftedourresearchemphasistowardlookingforgenericwrong-codebugs.WeturnedRandprogintoCsmith,a40,000-lineC++programforrandomlygeneratingCpro-grams.ComparedtoRandprog,CsmithcangenerateCprogramsthatutilizeamuchwiderrangeofCfeaturesincludingcomplexcontrolowanddatastructuressuchaspointers,arrays,andstructs.MostofCsmith'scomplexityarisesfromtherequirementthatit Figure2.Findingbugsinthreecompilersusingrandomizeddiffer-entialtestinginterleavestaticanalysiswithcodegenerationinordertoproducemeaningfultestcases,asdescribedbelow.2.1RandomizedDifferentialTestingusingCsmithRandomtesting[9],alsocalledfuzzing[17],isablack-boxtestingmethodinwhichtestinputsaregeneratedrandomly.Randomizeddifferentialtesting[16]hastheadvantagethatnooraclefortestresultsisneeded.Itexploitstheideathatifonehasmultiple,deter-ministicimplementationsofthesamespecication,allimplementa-tionsmustproducethesameresultfromthesamevalidinput.Whentwoimplementationsproducedifferentoutputs,oneofthemmustbefaulty.Giventhreeormoreimplementations,atestercanusevotingtoheuristicallydeterminewhichimplementationsarewrong.Figure2showshowweusetheseideastondcompilerbugs.2.2DesignGoalsCsmithhastwomaindesigngoals.Firstandmostimportant,everygeneratedprogrammustbewellformedandhaveasinglemeaningaccordingtotheCstandard.ThemeaningofaCprogramisthesequenceofsideeffectsitperforms.TheprincipalsideeffectofaCsmith-generatedprogramistoprintavaluesummarizingthecom-putationperformedbytheprogram.1Thisvalueisachecksumoftheprogram'snon-pointerglobalvariablesattheendoftheprogram'sexecution.Thus,ifchangingthecompilerorcompileroptionscausesthechecksumemittedbyaCsmith-generatedprogramtochange,acompilerbughasbeenfound.TheC99language[11]has191undenedbehaviors—e.g.,dereferencinganullpointeroroverowingasignedinteger—thatdestroythemeaningofaprogram.Italsohas52unspeciedbehaviors—e.g.,theorderofevaluationofargumentstoafunction—whereacompilermaychoosefromasetofoptionswithnorequirementthatthechoicebemadeconsistently.ProgramsemittedbyCsmithmustavoidallofthesebehaviorsor,incertaincasessuchasargument-evaluationorder,beindependentofthechoicesthatwillbemadebythecompiler.Manyundenedandunspeciedbehaviorscanbeavoidedstructurallybygeneratingprogramsinsuchawaythatproblemsneverarise.However,anumberofimportantundenedandunspeciedbehaviorsarenoteasytoavoidinastructuralfashion.Inthesecases,Csmithsolvestheproblemusingstaticanalysisandbyaddingrun-timecheckstothegeneratedcode.Section2.4describesthehazardsthatCsmithmustavoidanditsstrategiesforavoidingthem.Csmith'sseconddesigngoalistomaximizeexpressivenesssubjecttoconstraintsimposedbytherstgoal.An“expressive”generatorsupportsmanylanguagefeaturesandcombinationsoffeatures.Ourhypothesisisthatexpressivenessiscorrelatedwithbug-ndingpower. 1AccessestovolatileobjectsarealsosideeffectsasdescribedintheCstandard.Wedonotdiscussthese“secondary”sideeffectsofCsmith-generatedprogramsfurtherinthispaper.2 Theglobalxpointanalysisisrunwhenaloopisclosedbyaddingitsback-edge.IfCsmithndsthattheprogramcontainsunsafestatements,itdeletescodestartingfromthetailoftheloopuntiltheprogrambecomesgloballysafe.Thisstrategyisaboutthreetimesfasterthanpessimisticallyrunningtheglobaldataowanalysisbeforeaddingeverypieceofcode.2.6DesignTrade-offsAllowimplementation-denedbehaviorAnideallyportabletestprogramwouldbe“strictlyconforming”totheClanguagestandard.Thismeansthattheprogram'soutputwouldbeindependentofallunspeciedandunspeciedbehaviorsand,inaddition,beindepen-dentofanyimplementation-denedbehavior.C99has114kindsofimplementation-denedbehavior,andtheyhavepervasiveimpactonthebehaviorofrealCprograms.Forexample,theresultofper-formingabitwiseoperationonasignedintegerisimplementation-dened,andoperandstoarithmeticoperationsareimplicitlycasttoint(whichhasimplementation-denedwidth)beforeperformingtheoperation.Webelieveitisimpossibletogeneraterealisticallyex-pressiveCcodethatretainsasingleinterpretationacrossallpossiblechoicesofimplementation-denedbehaviors.ProgramsgeneratedbyCsmithdonotgeneratethesameoutputacrosscompilersthatdifferin(1)thewidthandrepresentationofintegers,(2)behaviorwhencastingtoasignedintegertypewhenthevaluecannotberepresentedinanobjectofthetargettype,and(3)theresultsofbitwiseoperationsonsignedintegers.InpracticethereisnotmuchdiversityinhowCimplementationsdenethesebehaviors.Formainstreamdesktopandembeddedtargets,thereareroughlythreeequivalenceclassesofcompilertargets:thosewhereintis32bitsandlongis64bits(e.g.,x86-64),thosewhereintandlongare32bits(e.g.,x86,ARM,andPowerPC),andthosewhereintis16bitsandlongis32bits(e.g.,MSP430andAVR).UsingCsmith,wecanperformdifferentialtestingwithinanequivalenceclassbutnotacrossclasses.NogroundtruthCsmith'sprogramsarenotself-checking:weareunabletopredicttheiroutputswithoutrunningthem.ThisisnotaproblemwhenweuseCsmithforrandomizeddifferentialtesting.Wehaveneverseenan“interesting”splitvotewhererandomizeddifferentialtestingofacollectionofCcompilersfailstoproduceaclearconsensusanswer,norhaveweseenanycasesinwhichamajorityoftestedcompilersproducesthesameincorrectresult.(Wewouldcatchtheproblembyhandaspartofverifyingthefailure-inducingprogram.)Infact,wehavenotseeneventwounrelatedcompilersproducethesameincorrectoutputforaCsmith-generatedtestcase.Itthereforeseemsunlikelythatallcompilersundertestwouldproducethesameincorrectoutputforatestcase.Ofcourse,ifthatdidhappenwewouldnotdetectthatproblem;thisisaninherentlimitationofdifferentialtestingwithoutanoracle.Insummary,despitethefactthatKnightandLeveson[13]foundasubstantialnumberofcorrelatederrorsinanexperimentonN-versionprogramming,CsmithhasyieldednoevidenceofcorrelatedfailuresamongunrelatedCcompilers.OurhypothesisisthattheobservedlackofcorrelationstemsfromthefactthatmostcompilerbugsareinpassesthatoperateonanintermediaterepresentationandthereissubstantialdiversityamongIRs.NoguaranteeofterminationItisnotdifculttogeneraterandomprogramsthatalwaysterminate.However,wejudgedthatthiswouldlimitCsmith'sexpressivenesstoomuch:forexample,itwouldforceloopstobehighlystructured.Additionally,always-terminatingtestscannotndcompilerbugsthatwrongfullyterminateanon-terminatingprogram.(Wehavefoundbugsofthiskind.)About10%oftheprogramsgeneratedbyCsmithare(apparently)non-terminating.Inpractice,duringtesting,theyareeasytodealwithusingtimeouts.Targetmiddle-endbugsCommercialtestsuitesforCcompil-ers[1,19,20]areprimarilyaimedatcheckingstandardsconfor-mance.Csmith,ontheotherhand,ismainlyintendedtondbugsinthepartsofacompilerthatperformtransformationsonaninterme-diaterepresentation—theso-called“middleend”ofacompiler.Asaresult,wehavefoundlargenumbersofmiddle-endbugsmissedbyexistingtestingtechniques(Section3.6).Atthesametime,Csmithisratherpooratndinggapsinstandardsconformance.Forexample,itmakesnoattempttotestacompiler'shandlingoftrigraphs,longidentiernames,orvariadicfunctions.Targetingthemiddleendhasseveralaspects.First,allgeneratedprogramspassthelexer,parser,andtypechecker.Second,weper-formedsubstantialmanualtuningofthe80probabilitiesthatgovernCsmith'srandomchoices.Ourgoalwastomakethegeneratedpro-grams“lookright”—tocontainabalancedmixofarithmeticandbitwiseoperations,ofreferencestoscalarsandaggregates,ofloopsandstraight-linecode,ofsingle-levelandmulti-levelindirections,andsoon.Third,Csmithspecicallygeneratesidiomaticcode(e.g.,loopsthataccessallelementsofanarray)tostress-testpartsofthecompilerwebelievetobeerror-prone.Fourth,wedesignedCsmithwithaneyetowardgeneratingprogramsthatexercisetheconstructsofacompiler'sintermediaterepresentation,andwedecidedtoavoidgeneratingsource-leveldiversitythatisunlikelytoimprovethe“coverage”ofacompiler'sintermediaterepresentations.Forexam-ple,sinceadditionallevelsofparenthesesaroundexpressionsarestrippedawayearlyinthecompilationprocess,wedonotgeneratethem,nordowegenerateallofC'ssyntacticloopformssincetheyaretypicallyallloweredtothesameIRconstructs.Finally,Csmithwasdesignedtobefastenoughthatitcangenerateprogramsthatareafewtensofthousandsoflineslonginafewseconds.Largeprogramsarepreferredbecause(empirically—seeSection3.3)theyndmorebugs.Insummary,manyaspectsofCsmith'sdesignandimplementationwereinformedbyourunderstandingofhowmoderncompilersworkandhowtheybreak.3.ResultsWeconductedveexperimentsusingCsmith,ourrandomprogramgenerator.Thissectionsummarizesourndings.Ourrstexperimentwasuncontrolledandunstructured:overathree-yearperiod,weopportunisticallyfoundandreportedbugsinavarietyofCcompilers.Wefoundbugsinallthecompilerswetested—hundredsofdefects,manyclassiedashigh-prioritybugs.(§3.1)Inthesecondexperiment,wecompiledandranonemillionrandomprogramsusingseveralyears'worthofversionsofGCCandLLVM,tounderstandhowtheirrobustnessisevolvingovertime.AsmeasuredbyourtestsovertheprogramsthatCsmithproduces,thequalityofbothcompilersisgenerallyimproving.(§3.2)Third,weevaluatedCsmith'sbug-ndingpowerasafunctionofthesizeofthegeneratedCprograms.Thelargestnumberofbugsisfoundatasurprisinglylargeprogramsize:about81KB.(§3.3)Fourth,wecomparedCsmith'sbug-ndingpowertothatoffourpreviousrandomCprogramgenerators.Overaweek,Csmithwasabletondsignicantlymoredistinctcompilercrasherrorsthanpreviousprogramgeneratorscould.(§3.4)Finally,weinvestigatedtheeffectoftestingrandomprogramsonbranch,function,andlinecoverageoftheGCCandLLVMsourcecode.Wefoundthatthesemetricsdidnotsignicantlyimprovewhenweaddedrandomlygeneratedprogramstothecompilers'existingtestsuites.Nevertheless,asshownbyourotherresults,Csmith-generatedprogramsallowedustodiscoverbugsthataremissedbythecompilers'standardtestsuites.(§3.5)WeconcludethepresentationofresultsbyanalyzingsomeofthebugswefoundinGCCandLLVM.(§3.6,§3.7)5 Figure3.Distinctcrasherrorsfound,andratesofcrashandwrong-codeerrors,fromrecentLLVMandGCCversionsconsistencycheckfailed.Thesegraphsconservativelyestimatethenumberofdistinctfailuresinthesecompilers,sinceweencounteredmanysegmentationfaultscausedbyuseoffreememory,null-pointerdereferences,andsimilarproblems.Wedidnotincludethesefaultsinourgraphedresultsduetothedifcultyofmappingcrashesbacktodistinctcauses.Itisnotclearwhichofthesetwometricsofcrashinessispreferable.Therateofcrashesiseasytogame:wecanmakeitarbitrarilyhighbybiasingCsmithtogeneratecodetriggeringknownbugs,andcompilerwriterscanreduceittozerobyeliminatingerrormessagesandalwaysreturninga“success”statuscodetotheoperatingsystem.Thenumberofdistinctcrashes,ontheotherhand,suffersfromthedrawbackthatitdependsonthequantityandstyleofassertionsinthecompilerundertest.AlthoughGCChasmoretotalassertionsthanLLVM,LLVMhasahigherdensity:aboutoneassertionper100linesofcode,comparedtoonein250forGCC.Run-timefailuresThebottompairofgraphsinFigure3showstherateofwrong-codeerrorsinourexperiment.Unfortunately,we7 bepossibletoautomaticallypartitionapplicationsintosmallerpartssothatequivalencecheckingcanbedonepiecewise.FutureworkAugmentingCsmithwithwhite-boxtestingtech-niques,wherethestructureofthetestedsystemistakenintoaccountinarst-classway,wouldbeproductive.Thiswillbedifcultforseveralreasons.First,weanticipatesubstantialchallengesininte-gratingthenecessaryconstraint-solvingmachinerywithCsmith'sexistinglogicforgeneratingvalidCprograms.Itispossiblethatwewillneedtostartover,nexttimeengineeringaversionofCsmithinwhichallconstraintsareexplicitanddeclarative,ratherthanbeingburiedinasmallmountainofC++.Second,theinverseproblemsthatmustbesolvedtogenerateaninputbecomeprohibitivelydif-cultwheninputspassthroughaparser,particularlyiftheparsercontainshashtables.Godefroidetal.[8]showedawaytosolvethisproblembyintegratingaconstraintsolverwithagrammarforthelanguagebeinggenerated.However,duetoitsnon-localpointerandeffectanalyses,thevaliditydecisionproblemforprogramsinthesubsetofCthatCsmithgeneratesisfarharderthanthequestionofwhetheraprogramcanbegeneratedbytheJavaScriptgrammarusedbyGodefroidetal.5.RelatedWorkCompilershavebeentestedusingrandomizedmethodsfornearly50years.BoujarwahandSaleh[4]gaveagoodsurveyin1997.In1962,Sauder[22]testedthecorrectnessofCOBOLcompilersbyplacingrandomvariablesinprograms'datasections.In1970,Hanford[10]usedaPL/1grammartodrivethegenerationofrandomprograms.Thegrammarwasextensibleandwasaugmentedby“syntaxgenerators”thatcouldbeused,forexample,toensurethatvariablesweredeclaredbeforebeingused.In1972,Purdom[21]usedasyntax-directedmethodtogeneratetestsentencesforaparser.Hegaveanefcientalgorithmforgeneratingshortsentencesfromacontext-freegrammarsuchthateachproductionofthegrammarwasusedatleastonce,andhetestedLR(1)parsersusingthistechnique.BurgessandSaidi[5]designedanautomaticgeneratoroftestcasesforFORTRANcompilers.Thetestsweredesignedtobeself-checkingandtocontainfeaturesthatoptimizingcompilerswereknowntoexploit.Inordertopredicttestcases'results,thecodegeneratorrestrictedassignmentstatementstobeexecutedonlyonceduringtheexecutionofthesub-programormainprogram.ThesetestsfoundfourbugsintwoFORTRAN77compilers.In1998,McKeeman[16]coinedtheterm“differentialtesting.”HisworkresultedinDDT,afamilyofprogramgeneratorsthatconformtotheCstandardatvariouslevels,fromlevel1(randomcharacters)tolevel7(generatedcodeis“modelconforming,”incor-poratingsomehigh-levelstructure).DDTismoreexpressivethanCsmith(DDTiscapableofgeneratingalllegalCprograms)anditwasusedtondnumerousbugsinCcompilers.Toourknowledge,McKeeman'spapercontainstherstacknowledgmentthatitisim-portanttoavoidundenedbehavioringeneratedCprogramsusedforcompilertesting.However,DDTavoidedonlyasmallsubsetofallundenedbehaviors,andonlythenduringtest-casereduc-tion,notduringnormaltesting.Thus,itisnotasuitablebasisforautomaticbug-nding.Lindig[15]usedrandomlygeneratedCprogramstondseveralcompilerbugsrelatedtocallingconventions.Histool,calledQuest,wasspeciallytargeted:ratherthangeneratingcodewithcontrolowandarithmetic,Questgeneratescodethatcreatescomplexdatastructures,loadsthemwithconstantvalues,andpassesthemtoafunctionwhereassertionscheckthereceivedvalues.Becauseitstestsareself-checking,Questisnotbasedondifferentialtesting.Self-checkingtestsareconvenient,butthedrawbackisthatQuestisfarlessexpressivethanCsmith.LindigusedQuesttotestGCC,LCC,ICC,andafewothercompilersandfound13bugs.Sheridan[23]alsousedarandomgeneratortondbugsinCcompilers.Ascriptrotatedthroughalistofconstantsoftheprincipalarithmetictypes,producingasourcelethatappliedvariousoperatorstopairsofconstants.ThistoolfoundtwobugsinGCC,onebuginSUSELinux'sversionofGCC,andvebugsinCodeSourcery'sversionofGCCforARM.Sheridan'stoolproducesself-checkingtests.However,itislessexpressivethanCsmithanditfailstoavoidundenedbehaviorsuchassignedoverow.Zhaoetal.[32]createdanautomatedprogramgeneratorfortestinganembeddedC++compiler.Theirtoolallowsageneraltestrequirement,suchaswhichoptimizationtotest,tobespeciedinascript.Thegeneratorconstructsaprogramtemplatebasedonthetestrequirementandusesittodrivefurthercodegeneration.Zhaoetal.usedGCCasthereferencetocheckthecompilerundertest.Theyreportedgreatlyimprovedstatementcoverageinthetestedmodulesandfoundseveralnewcompilerbugs.6.ConclusionUsingrandomizeddifferentialtesting,wefoundandreportedhun-dredsofpreviouslyunknownbugsinwidelyusedCcompilers,bothcommercialandopensource.Manyofthebugswefoundcauseacompilertoemitincorrectcodewithoutanywarning.Mostofourre-porteddefectshavebeenxed,meaningthatcompilerimplementersfoundthemimportantenoughtotrackdown,and25ofthebugswereportedagainstGCCwereclassiedasrelease-blocking.Allofthisevidencesuggeststhatthereissubstantialroomforimprovementinthestateoftheartforcompilerqualityassurance.Tocreatearandomprogramgeneratorwithhighbug-ndingpower,thekeyproblemwesolvedwastheexpressivegenerationofCprogramsthatarefreeofundenedbehaviorandindependentofunspeciedbehavior.Csmith,ourprogramgenerator,usesbothstaticanalysisanddynamiccheckstoavoidthesehazards.Thereturnoninvestmentfromrandomtestingisgood.Ourroughestimate—includingfaculty,staff,andstudentsalaries,machinespurchased,anduniversityoverhead—isthateachofthemorethan325bugswereportedcostlessthan$1,000tond.Theincrementalcostofanewbugthatwendtodayismuchlower.SoftwareCsmithisopensourceandavailablefordownloadathttp://embed.cs.utah.edu/csmith/.AcknowledgmentsTheauthorswouldliketothankBruceChilders,DavidCoppit,ChuckyEllison,RobbyFindler,DavidGay,CaseyKlein,GerwinKlein,ChrisLattner,SorinLerner,XavierLeroy,BillMcKeeman,DiegoNovillo,AlastairReid,JulianSeward,ZachTatlock,ourshepherdAtanasRountev,andtheanonymousreviewersfortheirinvaluablefeedbackondraftsofthispaper.WealsothankHansBoehm,XavierLeroy,MichaelNorrish,BryanTurner,andtheGCCandLLVMdevelopmentteamsfortheirtechnicalassistanceinvariousaspectsofourwork.ThisresearchwasprimarilysupportedbyanawardfromDARPA'sComputerScienceStudyGroup.References[1]ACEAssociatedComputerExperts.SuperTestC/C++compilertestandvalidationsuite.http://www.ace.nl/compiler/supertest.html.[2]F.Bellard.TCC:TinyCcompiler,ver.0.9.25,May2009.http://bellard.org/tcc/.[3]C.L.Bife.UndenedbehaviorinGoogleNaCl,Jan.2010.http://code.google.com/p/nativeclient/issues/detail?id=245.11