tstorealsystemsandwehaefoundittoworkwreportexperiencepaperandpointoutpitfallstoapaperintroducestheconceptofwritingpropertiesandcthemusingSectionshowshowtodenetestdatageneratorsforuserdenedtypes ID: 371865
Download Pdf The PPT/PDF document "QuickCheck:ALightweightToolforRandomTest..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
QuickCheck:ALightweightToolforRandomTestingofHaskellProgramsKoenClaessenChalmersUniversityofTechnologykoen@cs.chalmers.seJohnHughesChalmersUniversityofTechnologyrjmh@cs.chalmers.seABSTRACTisatoolwhichaidstheHaskellprogrammerinulatingandtestingpropertiesofprograms Propertiesdescribedcallytestedonrandominputbutitisalsopossibletodenecustomtestdatagenerators epresentanberoftoolalsopointoutsomepitfallstoaRandomtestingisespeciallysuitableforfunctionalprogramsbecausepropertiescanbestatedatanegrain Whenafunctionisbuiltfromseparatelytestedcomponentsthenrandomtestingsucestoobtaingoodcoerageofthedenitionundertest 1.INTRODUCTIONensuringsoftItiserylabouruptothecostDespiteanecdotalevidencethatfunctionalprogramsrequiresomewhatlesstesting Onceittypecksitusuallyworks inpracticeitisstillamajorpartoffunctionalprogramdevThecostoftestingmotivatese ortstoautomateitwhollyorpartlyAutomatictestingtoolsenabletheprogrammertocompletetestinginashortertimeortotestmorethoroughlyintheaailabletimeandtheymakeiteasytorepeattestsaftereachmodicationtoaprogram InthispaperwdescribetoolelopedtestingHaskellprograms unctionalprogramsarewellsuitedtoautomatictesting Itisgenerallyacceptedthatpurefunctionsaremheasierbecausebeforeimperativelanguageevenifwholeprogramsareoftenpurefunctionsfrominputtooutputtheproceduresfromwhictheyarebuiltareusuallynot usrelativfunctionsaboundinHaskellonlycomputationsintheIOPermissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforprotorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitationontherstpage.Tocopyotherwise,torepublish,topostonserversortoredistributetolists,requirespriorspecipermissionand/orafee.ICFP00,Montreal,Canada.Copyright2000ACM1-58113-202-6/00/0009..monadarehardtoandsocanbeatanegrain Atestingtoolmustbeabletodeterminewhetheratestispassedorfailedthehumantestermustsupplyanautoablecriteriondoingso ehatouseformalspecicationsforthispurpose ehaedesignedasimpledomainspeciclanguageoftestablesphthetesterusestodeneexpectedpropertiesofthefunctionsundertest thencksthatthepropertiesholdinalargeberofThespecicationlanguageisembeddedinHaskellusingtheclasssystem Propertiesarenormallywritteninthesamemoduleasthefunctionstheytestwheretheyservealsoascabledoctationofthebehaviourofthecode Atestingtoolmustalsobeabletogeneratetestcasesauehahosenthesimplestmethodrandomcompetessystematicmethodsinpractice eritismeaninglesstotalkaboutrandomtestingwithoutdiscussingthedistributionoftestdata Randomtestingismoste ectivewhenthedistributionoftestdatafollowsthatofactualdatabutwhentestingreuseablecodeunitsasopposedtowholesyspossibledatainallsubsequentreusesisnotknoAuniformdishoosearandomclosedtermwithauniformdistributionforexampleehahosentoputdistributionundertheumantester salsoembeddedinHaskell andawytoobservdistributionoftestcases ByprogrammingathetestercannotonlythedistributioncomplexinAnimportantdesigngoalwasthatshouldbemoduleofabouthisinpracticemainlyusedfromtheHugsinehaealsowritlittleaboutHaskellsyntaxandconsequentlysupportsthefulllanguageanditsextensions ItisnotdependentonanparticularHaskellsystem Acostthatcomeswiththisdecisionisthatwecanonlytestpropertiesthatareexpressibleandobservmethodisindetectingfaults erwehaeusedinavyofapplicationsrangingfromsmallexper tstorealsystemsandwehaefoundittoworkwreportexperiencepaperandpointoutpitfallstoapaperintroducestheconceptofwritingpropertiesandcthemusingSectionshowshowtodenetestdatageneratorsforuserdenedtypes SectionbrieydiscussestheimplemenSectionpresentsanberofcasestudiesthatshowtheusefulnessofthetool Section2.DEFININGPROPERTIES2.1ASimpleExampleAsarstexamplewetakethestandardfunctionhreversesalist ThissatisesanberofusefullahasInfactthersttoofthesecNotethattheselawsholdonlyfor nitetotalpaperspecicallyeitmorelikelythatthepropertiesarecomputable InordertocktheselawsusingerepretthemasHaskellfunctions uswedenewifthesefunctionsreturnforeverypossibleargutthenthepropertieshold eloadthemintotheHugseHaskellinterpreterandcallforexampleThefunctionesalawasaparameterandappliesittoalargenberofrandomlygeneratedargumen infact reporting OK iftheresultiserycase reportsorexampleifwemistakenlydenethenckingthelawmightproducewherethecountermodelcanbeextractedbytakingand isaratherarbitrarynbersoourlibraryprovidesaytospecifythisasaparameter Infacttheprogrammermustprovidealittlemoreinforthefunctionisactuallyoerloadedinordertobeablehandlelawswithavaryingnberofvablesandtheoerloadingcannotberesolvedifthelawitselfhasapolymorphictypeasintheseexamples ThustheprogrammermustspecifyaxedtypeatwhichthelawistobeSowesimplygivypesignatureforeachlawforOfcoursethepropertholdspolymorphicallyspecifytobequiteimportantintheorexample isassociativeforthetypebutnotforarguethatapropertyholdspolymorphically2.2Functionsearealsoabletoformulatepropertiesthatquantifyokforexamplethatfunctioncompositionisassociativewerstdeneextensionalequalitandthenwrite ypesthatfunctioncompositioniscomme thenthefunctionvaluesareprintedjustas Inthiscaseediscoerthatthe law weareckingisfalsebutnot2.3ConditionalLawswswhicharesimpleequationsarecontlyrepretedbybooleanfunctionaswehaeseenbutingeneralylawsholdonlyundercertainconditions videsanimplicationcombinatortorepresentsuchconditionallaorexamplethelacanberepresentedbythedenitionthelaNotethattheresultypeofthepropertyischangedfrombecausepropertyforrandomtestcasesetryckingittestcasesacandidatetestcasedoesnotsatifytheconditionitisdiscardedandanewtestcaseistried kingthelaasusualbutsometimesckingaconditionallawproducestheoutput Ifthepreconditionofalawisseldomsatisedthenwemighgeneratemanytestcaseswithoutndinganywhereitholds Insuchcasesitishopelesstosearchforcasesinwhicthepreconditionholds Ratherthanallowtestcasegenerationtorunforeverwegenerateonlyalimitednberofcandidatetestcasesthedefaultis Ifwedonotndvalidtestcasesamongthosecandidatesthenwesimreportberofperform IntheexampleweknowthatthelawpassedthetestinItisthenuptotheprogrammertodecidewhetherthisisenoughorwhetheritshouldbetestedmorethoroughly2.4MonitoringTestDataitseemsthatwetestedthethoroughlyenoughtoestablishitscredibiliterwustbecareful Letusmodifyasfollokingthelawnowproducesthemessagebinatordoesnotchangethemeaningofawbutitclassiessomeofthetestcasesinthiscasethoseistheemptylistwereclassiedas trivial eseethatalargeproportionofthetestcasesonlytestedinsertionintoanemptylist ecangetmoreinformationthanjustlabellingsometestThecomgatherallvaluesthatarepassedtoitandprintoutahistogramofthesevorexampleifwewrite emightgetasaresult SoweseethatonlycasestestedinsertionintoalistwithfairlystrongevidencethatthelawholdsitisworryingthateryshortlistsdominatethetestcasessostronglyallitiseasytodeneanerroneousversionoferthelessworksforlistswithatmostoneelemenThereasonforthisbehaviourofcourseisthatthepreisorderedbutonlyofthelistsoflengthare testcaseswithlongerlistsaremorelikelytoberejectedb isHaskell sinxfunctionapplication theprecondition Thereisariskofthiskindofproblemeverytimeweuseconditionallawssoitisalwysimportanproportionactuallytested Thebestsolutionthoughistoreplacetheconditionwithacustomtestdatageneratorfororderedlists ewritespeciesthetestdatageneratorkingthelawnoes aswouldexpect videssupportfortheprogrammertodenehisorherowntestdatawithcontroloerthedistributionoftestdatawhicewilllookatinthenext2.5InniteStructuresellfunctionesanonemptylistreturnsalistthatrepeatsthecontsofthatlistinnitelywtakealookatthefollowinglawformulatedinAlthoughinelythesincewearecomparingtoinnitelistsusingcomputablewhichdoesnotterminate Insteadwecanreforulatethepropertyasalogicallyequivtonebyusingthattinnitelistsniteinitialtsareequal Anotherissuerelatedtoinnitestructuresisquanpropertiesthatforexampleholdforallinnitelistsbutingeneralitisnotclearhowtoformulateandexecutepropertiesaboutstructurescontainingbottom 3.DEFININGGENERATORS3.1ArbitraryThewegeneraterandomtestdataofcoursedependsype troducedypeofwhicypeisaninstanceifwecangeneratearbitraryelementsinit isanabstracttyperepresentingageneratorfortypeTheprogrammercaneitherusethegeneratorsbuiltinasinstancesofthisclassorsupplyacustomgeneratorusingthebinatorwhicesawintheprevioussection ornowwedenethetype Notethatleavingtheconditionoutresultsinanerrorbecauseisnotdenedforemptylists isarandomnberseedageneratorisjustafunctionwhichcanmanufactureaninapseudorandomButwewilltreatasanypesowedeneaprimitivegeneratortochoosearandomnberinaninalandweprogramothergeneratorsintermsofit fromsimpleronestodosowedeclaretobeaninstancemethodsofthetherstoneofwhichcontructsathesecondonebeingthemonadicsequencingoperatorwhicgeneratesanandpassesittoitssecondargumenttogendenitionofneedstorandomnberseedstoitstoargumentsandisonlypassedoneseedbutluckilytheHaskellrandomnberlibraryprovidesanoperationtosplitoneseedintotypeswhichisdenedintermsofeageneratorthatappliesthepairingoperatorhdeclarationsformostofHaskell spredenedtypes 3.2GeneratorsforUser-DenedTypesforeacypethenwustrelyontheusertoprovideinstancesforuserdenedtypes Inprinciplewcouldtrytogeneratetheseautomaticallyinapreprocessorpolytypicprogrammingbutwsteadtoleaethistasktotheuser Thisispartlybecausetobealighttooleasytoimpletandeasytouseinastandardprogrammingentwedon twttoobligeuserstoruntheirprogramsthroughapreprocessorbeteeneditingthemandtestingButanotherstrongreasonisthatitseemstobevhardtoconstructageneratorforatypewithoutknosomethingaboutthedesireddistributionoftestcases producingvidecombinatorstoenableaprogrammertodenehisogeneratorseasilyThesimplestcalledjustmakesahoiceamongalistofalternativegeneratorswithauniformforexampleifthetypeisdenedbthenasuitablegeneratorcanbedenedbAsanotherexamplewecouldgeneratearbitrarylistsusingwhereweusetoapplytheconsoperatortoanarbitraryheadandtail erthisdenitionisnotreallyitproduceslistswithanproducedspecifythefrequencywithwhicheachalternativeiscedenetochoosetheconscasefourtimesasoftenasthenilcaseleadingtoanaeragelistlengthoffourelemenmoregeneraldatatypesitturnsouteneedevennercontroloerthedistributionofgeneratedSupposeypettoaoidchoosingatoooftenhencetheusethisde nitiononlyhasachanceoftermiThereasonisthatforthegenerationofaegenerationsmustterminate Iftherstfewrecursionschooseesthengenerationterminatesonlyifverymanyrecursivegenerationsallterminateandthechanceofthisissmall enwhengenerationterttoaoidthis sinceweperformalargenberoftestswteachtesttobesmallandquicButthenotionofasizeishardeventodeneingeneralforypetiontypesanethereforegivetheresponsibilitforlimitingsizestotheprogrammerdeningthetestdatahangetherepresentationofgeneratorstowherethenewparameteristobeinterpretedassomekindofsizebound edeneanewcombound generatesanypassingthecurrentsizeboundtopretthesizeboundinsomereasonablewyduringtestdata boundberofnodesinthegeneratedtreeswhichisquitereasonable wthatwehaeintroducedthenotionofasizeboundecanuseitsensiblyinthegeneratorsforothertypessuctegersandlistssothattheabsolutevaluerespectivlengthisboundedbythesize Sothedenitionswepretedearlierneedtobemodiedaccordinglyestressthatthesizeboundissimplyanextraglobalparameterwhicheverytestdatageneratormayaccesseveryuseofseesthesameboundedoto dividethesizeboundamongthegenerators sothatforexamplealongergeneratedlistwouldhaesmallerelemeneepingtheoerallsizeofthestructurethesame Thereasonisthatwewishtoaoidcorrelationsbeteenthesizesofdi erentpartsofthetestdatawhichmightdistortthetestresults edovarythesizebeteendi erenttestcases ebeginpropertboundesforagreatervyoftestcaseswhichbothmaktestingmoree ectiveandimproesourchancesofndingenoughtestcasessatisfyingthepreconditionofconditionalproperties Italsomakesitmorelikelythatwewillndasmallcounterexampletoapropertifthereisone 3.3GeneratingFunctionspropertiesariablesthenwustbeabletogeneratearbitraryfuncderstandhownoticethatafunctiongeneratoroftypeisrepresentedbyafunctionoftypeIntRandabByreorderingparametersthisisequivttotheypeIntRandbhrepresenaGenecanthusdeneanduseittoproduceageneratorforafunctiontypeprovidedwecanconstructageneratorfortheresulttypewhicwdependsontheargumenetakecareofthisdependencebydeninganewclasswhosemethodmodiesageneratorinawdependingonitsrstparameter ewillthinkofproducingenthisclasswecandenehgeneratesanarbitaryfunctionthatusesitsargumentomodifythegenerationofitsresult Inordertodeneinstancesofeneedawtoconstructgeneratortransformers ethereforedenethe constructsageneratorwhichtransformstherandomnberseeditispassedinawydependingonbeforepassingittoThisfunctionmustbedenedvsothatallthegeneratorsweconstructusingitareindependenenanylistofincanconstructageneratortransformerrisetoindependentgeneratortransformerswithaveryhighecandeneinstancesofthatchoosebeteengeneratortransformersdependingontheargumenorexamplethebooleaninstancetransformsageneratorinindependenysforandforthegeneratorscoarbitraryTruegFalsegwillbeindependenInasimilarwecandenesuitableinstancesformanyothertypes orexampletheintegerinstancejustconertsitsintegerargumentinasequenceofbitswhicharethenusedasgeneratortransformersinturn Instancesofforrecursivedatatypescanbedenedaccordingtoastandardpattern orexamplethelistinstanceisjustthatdi erentlistsmappedtoindependenpingeachconstructortoanindependenttransformercomposingthesewithtransformersderivedfromeachcomponenypessamewSincetheprogrammerisresponsibleformakingthesedenitionsforuserdenedtypesitisimportantthattheybestraighecaneveninterpretfunctionsasgeneratortransformerswithaninstanceoftheformTheideaisthatweapplythegivenfunctiontoanarbitrarytandusetheresulttotransformthegivengenerInthiswofunctionswhicharedi erentwillgivrisetodi erentgeneratortransformers Notethatifwehadtriedtoaoidneedingtosplitrandomberseedsbydeningthemonadasastatetransformerontherandomseedratherthanastatereaderthenbeentionandwouldnothaebeenabletogeneraterandom4.IMPLEMENTINGQUICKCHECKpropertieswithavaryingnberofargumentsanddi ertresulttypes oimplementthisweintroducethetypeandwecreatethetypeclass yperepresentspredicatesthatcanbecytesting meansthatitneedstobeableaterandominputandnallyproductatestresult Soaisacomputationinthemonadendinginanabstracttypewhiceepstrackofthebooleanresultofthetestingtheclassicationsoftestdataandthetsusedinthetestcase LetustakealookatsomeinstancesofAneasyypetockisofcourseurtherfunctionsforwhicenthepropertypeitselfisaninstancesothatecannestpropertycomUsingthefunctionitbecomeseasytodenetheItstypeis Moredetailsoftheimplementationcanbefoundintheappendix 5.SOMECASESTUDIES5.1Unicussauncationalgorithmwhicehaedevelopedalongwithaspecication ThiswasquiterevbothasregardstheimpactthathasonprogrammingandthepitfallsthatmustbeaItistoolargetindetailjustdiscussthelessons5.1.1ImpactonTypeDeFirstofalltheuseofhadanimpactonthedesignofthetypesintheprogram edenedthetypeoftermstobeuniedasratherthantheequivouldprobablyhahosenotherwise Thetypeeuseddistinguishesbeteenastringusedasaconstructorandastringusedinotherconandbeteenanaturalnberusedasavariablenameandanintegerusedinotherconypesforexamplethanforstrings HadwstringsthenitisveryunlikelythatwouldevergenerateitissameconstructornametInsteadwhosetogenerateconstructornamesusinghgivesusagoodchancethatgeneratedtermswillbeatleastpartiallyuniable ewisewelimitedunicationariablesintestdatatoasmallset Ofcoursewecouldhaeusedthesecondypeaboandspeciedacustomtestdatageneratorwithanexplicitineachpropert Butitismoreconttolettestdatabeautomaticallygeneratedusingsooneisypes areotheradvtagestodoingsoalso itpermitsthetypeertodetectmoreerrors Sousingtroducingnewtypesinprograms 5.1.2CheckingFunctionalPropertiesAunierneedstomanagethecurrentsubstitutionandalsothepossibilityoffailuresinrecursivecalls Aconytodosoistouseamonad edenedaunicationrepresentedbyafunctionwithoperationstodenean extensionalequality operatoronmonadicaluesandckboththemonadlawsandpropertiessuc5.1.3ErrorsFoundreportfoundalargenberoferrorsinthisexample Infactnoerrorsatallwerefoundintheunieritself Thisisprobablymoreareectiononthenberoftimestheauthorshaprogrammedunierspreviouslythanonthee ectivenessofeknowhowtodoitquitesimplyOntheotherhandwedidnderrorsintheorexampleedenedasubstitutionfunctionhrepeatedlysubstitutesuntilnovariablesinthedomainofthesubstitutionremainandstatedtheobviouspropertealedthispropertytobefalse itholdsonlyforacyclicsubstitutionsotherwiseaninnitetermisgeneratedandtheequalitytestloops Thiserrorwusingthefunctionhprintsouttheartstoeverytestcasebeforeitcksit ereobligedtocorrectthespecicationtomadeusthinkharderaboutthepropertiesofourcodeanddocumentthemcorrectly OnthedoulatingthespecicationcorrectlyperhapswritingtheimplemenThiswaspartlybecausepredicatessuchasarenontrivialtodeneagoodsettheorylibrarywouldhaehelpedhere 5.1.4AFalseSenseofSecurityexperiasthefalsesenseofsecuritythatcanbeengenderedwhenone sprogrampassesalargenberoftestsintrivialehaealreadyproblemwhenwdiscussedconditionalpropertiesinthisexampleitbituspropertiestheformsinceourunierreturnswhenitfails Withalitfairlylikelytobeuniablesincevariablesoccurquiteoftenandifeithertermisavariablethenunicationwillalmostcertainlysucceed Ontheotherhandifneithertermisaariablethentheprobabilitthattheywillunifyissmall usthecasewhereonetermisavariableisheavilyotedamongthetestcasesthatsatisfytheprecondition wefoundthatoeroftestcaseshadthisproppropertecanhardlyconsiderthattheywerethoroughlyThesolutionwastouseacustomtestdatageneratoregeneratedygeneratingrandomtermandreplacingrandomsubtermsbintodi erenThisusuallygeneratesuniabletermsalthoughmayfailtowhenvariablesareusedinconsisteninthetoterms Withthismodicationtheproportionoftrivialcasesfelltoareasonable Thisexperienceunderlinestheimportanceofindistributionofpropertiesareused 5.2CircuitProperties5.2.1LavainaNutshell"isatooltodescribesimulateandformallyvifyhardwaisasocalleddlanguagewhicpropertiesexpressedinanexistingprogramminglanguageinthiscaseTheideaistoviewahardwarecircuitasafunctionfromvidesprimitivecircuitssuchasandMorecomplicatedcircuitsaredenedbycombiningthese videsinputsandtheoutputsarecalculated urthermoretheLaasystemdenescombinatorsforcircompositionpositionreplicatesitinacolumnofcircuitsconnectingthev5.2.2PropertiesinLavaPropertiesofcircuitscanbedenedinasimilarwexampletodenethepropertythatacertaincircuitiscomewesaypestainingsignalsinthiscaseapair PropertiescanbeformallyvedothisbyproinputstothecircuitorpropertandcalculatingaconcreteexpressioninaHaskelldatatyperepresenthecircuit externaltheoremproAllthisisdonebytheoafunctionHereishoecanuseittovthatasocalledhalfaddercomponentiscommTheLaasystemprovidesanberoffunctionsandcomnatorstocontlyexpresspropertiesandformallyvinLavaThoughweareabletoverifypropertiesaboutcircuitsinbenettoollikTherearetomainreasonsforthat rstonethatcallinganexternaltheoremproeryheatprocess WhenvatheoremprovingarequitebigandweoftenhaetowaitforalongtimetogetananswSoatypicaldevtcycleistowritedownthespecicationofthecircuitrstthenmakeanimplemenitforobviousbugsandlastlycalltheexternaltheoremproerforverifyingthecorrectness HereisanexampleofhowtouseinLaAddingthistestingmethodologytoLaaturnedouttobequitestraighardbecauseLaaalreadyhadanotionofproperties estingcanbedoneforallcircuitypestialcircuitscontaininglatcesimplyckthecircuitpropertyforasequenceofinputs 5.2.4HigherOrderTestingThesecondreasonforusingtestinginLaaissimplythatpropertiescanonlydealwithatmostrstorderlogicsandtheLasystemisonlyabletogenerateformulasofthattype Sometimeswouldliketoverifypropertiesaboutorexampleprovingthatdistributeso isveryhardtoverifyinLaallfactsuchpropertiesarehardtoverifyautomaticallyingeneralwecandoitforsmallxedsizeshoButsincethesekindofpropertiesforarbitrarycircuits Adrakisthatwehaetoxthetypesofthesecirpropertiesaboutthemarepolymorphicinthecircuits inputandoutputtypes 5.2.5ErrorsFoundTheauthorsusedthelibrarywhiledevarithmeticcircuits alreadyusedinthedevtprocessbutonlyinavlimitedandadhocwwmhmorethoroughtestingaspossible okinds efounderrorsthatourformalvtionmethodwouldhaefoundaswlogicalerrorsintheButsecondlyealsofounderrorsduetothefactthatrandominputalsomeansrandominputorexampleforanbitadderweonlyuseandformallyspecicinputsizes Randomtestingthatwehadforgottentodeneoneofthesecases!5.3PropositionalTheoremProvingorteachingpurposesweimplementedtodi erenmethodspropositionalmethodsvisPutnammethodwhichusesbackingtogenmodels methodmethodtonthedilemmaproofsystemtogatherinformationabouttheliteralsintheclauseset functiontakesanextraargumentanhisthesocalled saturationlevel aparameterwhiclimitsthedepthoftheproofsandusuallyliesbeteenandIftheresultofitmeansthatthereasaconIftheresultisitmeansthaterymodeloftheclausesetshouldhaasasubmodel ismhmorestraighardtoimpropertystatedaboNotethatwsomestatisticsdiction whentheresultwandthesizeofthecaseofealsoexpressedthatwedisqualifyatestcasewhenWiththehelpofthispropertfoundbugs!Thesebugswereduetoimplicitunjustiedassumptionswaboutbothrithmsassumedthatnoclausesintheinputcouldconthesameliteraltwiceandthefunctionassumedthatnoneoftheinputclauseswasempthniquesasinsectionpropertytookaboutandfromtheoutputwcouldseethedistributionofasabout$ 5.4PrettyPrintingAndyGillreportedaninterestingstoryaboutusingtous HeuseditindevelopingavtofWtedhisvtfunctionallyusingHaskstillusingHaskellheusedastatemonadwithexceptionstoelopanimperativeimplementationofthesamelibraryTheideawasthatthesecondimplementationmodelswhatgoesoninaJaaimplemenThenheexpressedtherelationshipbeteenthetodifproperties ThisquicklypointsoutwheremyrasoningisfaultyandprovidesgratteststocatchthecornersoftheimplemeneproblemswerefoundthethirdofwhichshowethatIhadmertwocinmyimplementationthatIshouldnothave urthermorehemadeanimprotinthewreportsexamplesfoundareverylargeanditisdiculttogobactothepropertyandunderstandwhyitisacounterexample erwhenthecounterexampleisanelementofatreeshapeddatatypetheproblemcanoftenbelocatedinoneofthesubtreesofthecounterexamplefound GillextendedclasswithanewmethodtoitstforInsomecasesmhsmallercounterexampleswtinglibrarywasportingthestateandexceptionmonadmodelinHaskelltoJaHethenusedtogenbercodeordertockthattheJaaimplementationwasequivtothetoHaskellmodels 5.5EdisonChrisOkisalibraryofecientdatastructuressuitableforimplementationanduseinfunctionalprogramminglanguages Hehasusedtostateandtestpropertiesofthelibraryerydatastructureinthebeenhasincludedseveralextramodulesespeciallyforformpropertiesaboutthesedatastructures Hereports Myexehasmostlybenthatofaverysatis eduser letsmeEdisonwithpr maybofthee ortofmyprevioustestsuiteanddoesamuchbjobtobasakialsomentionsadrakhavingtodowiththeellmodulesystem HeoftenusesonespecicationofAnaturalwytodothisistoplacethespecicationinonemoduleandeachimplementationinaseperatemodule Butsincethespecicationreferstotheimplementationthenthespecicationmoduleustimportthetationonetlyundertest asakiwasobligedtoeditthespecicationmodulebyhandbeforeeachtestsoastoimporttherightimplemenhpreferablewouldbetoparameterisethespecicationonanimplementationmoduleylefunctorswouldbereallyhelpfulhere!6.DISCUSSION6.1OnRandomTestingerynaiveapproacSystematicmethodsareoftenpreingeneralatestadequacycriterionisdenedandproceedsadequacycriterion orexampleasimplecriterionisthatwpathwithexceptionsforloops befolloedinatleastonetest Awidevyofadequacycriterahaebeenproposedarecentsurveyis ehahosennottobaseonsuchanadequacycriterion InpartthisisbecausemanycriteriawneedreinterpretationbeforetheycouldbeappliedtoHaskprogramsitismhlessclearforexamplewhataconwpathisinalanguagewithhigherorderfunctionsandlazyevInpartsuchacriterionwouldforceustousemhmoreheatmethodsevenmeasuringpatherageforexamplewouldrequirecompilermodicationsandthustietoaparticularimplementationofellnamelytheonewemodiedtocollectpathinforGeneratingtestdatatoexerciseaparticularpathrequiresconstraintsolving onemustndinputvalueswhicetheseriesoftestsalongthegivenpathproducespeciedresults WhilesuchconstraintsolvingmaybefeasibleforarithmeticdatafortherichsymbolicdatatypesfoundinHaskellprogramsitisadicultresearchprobleminitswnrigherapartfromthedicultyofautomatingsystematictestingmethodsforHaskellthereisnoclearreasontobelievbetterDuranandNtafoscomparedthefaultdetectionprobabilitofrandomtestingwithpartitiontestinganddiscoeredthatthedi erencesine ectivenessweresmall Hamletandylorrepeatedtheirstudymoreextensivandcorroboratedtheoriginalresults Althoughpartitiontestingistlymoree ectiveatexposingfaultstoquoteHamlet stsurveyBytaking morointsinardomtestanyadvantageartitiontestmighthavehadisdout orsmallprogramsinparticularitislikelythatrandomtestcaseswillindeedexerciseallpathsforexamplesothatgoodyankpropertiesofindividualfunctionsbutthefunctionstheycallaretestedindependen Soevenwhenisusedtotestalargeprogramwealwystestasmallpartatatime ThereforewemayexpectrandomtestingtowparticularlywhgreaterautomatingsystematictestingforHaskellourchoiceofrandomtestingisclear 6.2CorrectnessCriteriaTheproblemofdeterminingwhetheratestispassedornotisknownastheacleprOnesolutionistocompareprogramoutputwiththatofanotherversionoftheproperhapsanolderorperhapsabut obviouslycorrect vanexecutablespecicationmightplaythesamer%Thiskindoforaclecaneasilybeexpressedasapropertalthoughourpropertiesaremhmoregeneral eroftenonecanckthataprogram soutputiscorrectmhmoreecientlythanonecancomputetheoutBlumandKannanexploitthisintheirworkonprogramwhichclassiestheprogram soutputascorrectorwithahighprobabilityofclassifyingcorrectlyanddoessowithstrictlyloercomplexit TheydistinguishprofromprogramtheirproposalisthatprogramsshouldalwktheiroutputandindeedinfurtherworkBlumetal edhowprogramswhichusuallyproducecorrectanswerscanevwrongoutputinparticulardomains Ofcourseresultcerscanalsobeexpressedaspropertiesalthoughweusethemfortestingratherthanasapartofthenalprogram spropertylanguageishoermoregeneralthanresultcViaconditionalpropertiesorspecictestdatageneratorswecanexpresspropertieswhichholdonlyforasubsetofallpossibleinputs Thuswoidtestingfunctionsincaseswhichleadtoruntimeerrorsorcasesinedonotcareabouttheresult orexamplewedonottestinsertionintoanunorderedlist thereisnopoinindoingso etaresultcermustverifythataprogramproducesthe correct outputinallcaseseventhosewhicareuninpropertiesarenotlimitedtockingtheresultofanindividualfunctioncall thepropertythatanoperatorisassociativeforexamplecannotreallybesaidtocktheresultofanyindividualuseoftheoperatorbutstillexpressesauseful global propertthatcanbecedbytesting propertiesspecicationrectlywasusedintheDAISTSsystemfortestingabstractdatatypeswhichcompiledequationalpropertiesintestingcodealthoughtestcaseshadtobesuppliedbythekingautomatictestcasegenerationDAISTSdidnotneedequivtsofourconditionalandquantiedpropAlthoughthelanguageusedwasimperativeabstractdatatypeoperationshadtobeforbiddentosidee ecttheirtsthustheprogramstobetestedwereessenbefunctional yandHamletdescribeatechniquefortestingC classesagainstanalgebraicspecicationwhichiserthespecicationlanguagemustberestrictedinordertoguaranteethatspecicationsbeanimated Thereseemstobenopublishedworkonautomatictestingofprogramsagainstspecications esimply ethatfunctionalprogramsandpropertybasedspecicationsareaverygoodmatcecanusethegivenpropertiesdirectlyfortesting erembeddingthespecicationlanguageinHaskellpermitsustowriteverypoandexiblepropertiesumof6.3TestDataGenerationtoolslimiteddomainswiththegoalofmatchingthedistributionofactualdataforthesystemundertestthesocalledationalprInthiscasestatisticalinferencesaboutthemeantimebeteensystemfailurescanbedrawnfromthetestresults InordertomorecomplexdataitispopuhtoitwpressallthedesiredpropertiesoftestdataforexamplerandomprogramThereforethegrammarswereenhancedwithactionsorextendedtoattributegrammars ThisapproachasbeenmostusedfortestingcompilersalthoughMaurerarguesforitsuseinmanycontexts Grammarshaebeenusedforsystematictestingforexamplethegeneratedtestdataisrequiredtoexerciseproductionybeandnotedtheterminationproblemforrecursivegrammars HissolutionthoughwasjusttoincreasetheprobabilitiesofgeneratingleaessothatevtualterminationisguaranOurexperienceisthatthisresultsinfartoohighaproportionoftrivialtestcasesandthereforeinecienttestingustbeproperlyebelieveourmethodofcontrollingsizesismhsuperior ItseemsthattheneedtolearnacomplexlanguageofextendedgrammarshashinderedtheadoptionofthesemethodsbeddingainHaskellweprovideatleast thesamecapabilitiesbutsparetheprogrammertheneedtolearnmorethanafewnewoperators tthesametimeweprovideallthepoerandyneededtogeneratetestdatasatisfyingcomplexintsinalanguagetheprogrammeralreadyknolinkinggeneratorstotypesviaHaskell sclasssystemwereetheprogrammeroftheneedtospecifygeneratorsatallinmanycasesandwheretheymustbespeciedtheprogrammer sworkisusuallylimitedtospecifyinggeneratorsforhisorherownnewtypes 6.4OnRandomnessehaeencounteredsomeinterestingproblemsinreasoningaboutprogramswhichuserandomnbergeneration Inparticularthemonadwhicisbasedonisnotamonadatall!ConsidertherstmonadlaSinceourimplementationofbindsplitsitsrandomnberoperandseedsonthetosidesoftheequationandythereforeproducedi erentresults Sothelawsimplydoesnothold hoerweconsiderthelawtobetruebecausethetosidesproducethesameresultseveniftheresultsdi erforanyparticularseed Butwhatpreciselydowemeanby morally ecannotxtheproblemjustbyreinterpretingequalityfortheypeclaimingthetosidesarejustdi erentrepresengoodbecausewecanactuallyobservethedi erenceatothertypesysupplyingarandomnberseedsomethingwehaetobeabletodoiftheypeistobeuseful Insteadwehatoreinterpretwhatwemeanbyprogramequivalenceinthepresenceofrandomnbergeneration theimperativeprogramismorallyequivttothesameprogramwiththeassigntsreversedinthesamesensebutofcourseproducesatresult Thereissomeinterestingsemantictheorytobedonehere 6.5OnLazyEvaluationehaearguedinthepastthatlazyevaluationisaninaluableprogrammingtoolthatradicallychangesthewproperties aconicthereaboperfectlyinnitestructuresinspecicationsprovidedthepropertieseactuallytestarecomputableforexamplewecantestthatarbitrarilylongprexesofinnitelistsareequalratherthancomparingtheliststhemselvmonadhasabindoperationbecausewesplittherandomnberoperandthentheother andsowecanfreelydenegeneratorsthatproduceinniteresults Whatwecannotdoisterminationinatestresult SowecannottestforexamplethepropertOntheotherinaumantestercannotebeentestlazyprogramssatisfactorilybyhandsofarthenwearenotinaworsepositionifweuseetahtestercanobservethatproducesanerrormessagefromtheevaluationof withoutproducinganyotheroutputrstandcanthusinferthatthepropertyaboeholds TheproblemisthattheHaskellstandardprovidesnowyforatomakethesameobettherearevariousextensionsofHaskellwhicdoindeedmakethispossible SomeworkdonebyAndyGillhasshownthatgivensuchextensionswecouldformandckpropertiessuchastheoneaboeusing6.6SomeRemajoristhatitencouragesustoformulateformalspecicationsthusimprovingourunderstandingofouritisopentotheprogrammertodothisfewreallydoperhapsbecausethereislittleandperhapsbecausespecicationalueifthereisnockatallthatitcorrespondstothetedprogram addressesboththeseisitgivesusashorttermpao viaautomatedtesting andsomereasontobelievethatpropertiesstatedinamoduleactuallyhold enlybetspecicationrstcategoryisuselesstodiscoerexceptinsofarasithelpswithfurthertesting ittellsusnothingabouttheactualThethirdcategoryisobviouslyusefulinasensethesearetheerrorswetestinordertond Butthesecondcategoryisalsoimportant eveniftheydonotrevealamiseinthecodetheydorevealamisunderstandingaboutwhatitdoes Correctingsuchmisunderstandingsimproourabilitytomakeuseofthetestedcodecorrectlylater WhenformulatingspecicationsonerapidlydiscoerstheneedforalibraryoffunctionsthatimplementcommonmathearetionofnitesettheoryforusewithyoftheabstractionsinitaretooinecienttobeofmhuseinprogramsbutinspecicationswheretheobjectistostatepropertiesasclearlyandsimplyaspossibletheycomeintheiroBecauseofthisdi erenceinpurposethereisaneedforlibrariesspecicallytargetedatspecications majorthedistributionanddecidetlymanytestshaebeenrun AlthoughweproystocollectthisinformationwecannotcompeltheprogrammertoAprogrammerwhodoesnotgainingafalsesenseofsecurityfromalargenberofinadequatetests erhapswecoulddeneadequacymeasuresjustonthegeneratedtestdataandthuswarntheuseratleastinthiskindofsituation 7.CONCLUSIONSehaetakentorelativelyoldideasnamelyspecicationsasoraclesandrandomtestingandfoundwystomakthemeasilytoHaskvideanembeddedlanguageforwritingpropertiesingexpressivenesswithoutthelearningcost Thelanguagetainscontfeaturessuchasquantiersconditionalsandtestdatamonitors eprovidetypebaseddefaultrandomtestdatageneratorsincludingrandomfunctionsgreatlyreducingthee ortofspecifyingthem Thirdlyeprovideanembeddedlanguageforspecifyingcustomtestdatageneratorswhichcanbebasedonthedefaultgeneratorsgivinganercontroloertestdatadistribution alsointroduceanoelwyofcontrollingsizewhengeneratingrandomelementsofrecursivedatatypes oldtechniquesworksextremelywellforHaskThefunclocalpropertiessincealldependenciesofafunctionareexplicit Andpreciselyrandomtestingisknowntoworkverywellforsmallnegrainedprogramsandise ectiveinndingfaults thetoolislighandeasytouseandprovidesashorttermpao forexplicitlystatingpropertiesoffunctionsinaprogramwhichgreatlyincreasestheunderstandingoftheprogramfortheprogrammeraswellasfordocumentationpurposes ouldliketothankAndyGillChrisOkasakiandtheanonymousrefereesfortheirusefultsonthispaper 8.REFERENCESS AnyandR Hamlet Automaticallyckingantationagainstitsformalspecication InIrvineSoftwareSymppagesMarch RolandBackhousePatrikJanssonJohanJeuringandLambertMeertens GenericProgrammingAntroduction InenotesinComputerSciencolume BjesseK ClaessenM SheeranandS Singh areDesigninHaskell IneonFunctionalPrBaltimore AM BlumandS Kannan Designingprogramsthatktheirwork Inc stSymposiumontheoryofComputingpages" ACMMay M BlumM LubandR Rubinfeld Selftesting$correctingwithapplicationstonproblems Inc ndSymposiumontheTheoryofpages" ACMMay A CelentanoS C ReghizziP DellaVignaandC Ghezzi CompilertestingusingasenePreExp " K ClaessenandD Sands ObservableSharingforunctionalCircuitDescription InAsianComputereConferPhetThailand AM DavisandH Putnam AcomputingprocedureforticationtheoryJournaloftheAssociationforComputingMachinery" J DuranandS Ntafos AnevaluationofrandomansactionsonSoftwareEngine July J GannonR HamletandP McMullin Dataabstractionimplementationspecicationandtesting ansPrgLangandSystems D Hamlet Randomtesting InJ MarciniakeditordiaofSoftwareEnginepages"" R HamletandR Tylor Partitiontestingdoesnotinspirecondence ansactionsonSoftwar December J Hughes WhunctionalProgrammingMatters InD TurnereditorchTopicsinF AddisonW M P Jones TheHugsdistribution Currenailablefrom M Maurer Generatingtestdatawithenhancedtextfreegrammars IEEESoftwar" GunnarStk ASystemforDeterminingPropositionalLogicTheoremsbyApplyingValuesandRulestoTripletsthatareGeneratedfromaF SwedishPtNo ""approed U S PtNo "" EuropeanPNo PhilipWadler Theoremsforfree!eonFunctionalPrammingandComputerLondonSeptember PhilipWadler AprettierprinterMarch Draftpaper H ZhuP HallandJ Ma SoftareunittesterageandadequacyComputingSurveys "December" Appendix:ImplementationHereweshowtheimplementationofthecodeisaailablefrommoduleQuickCheckwhereimportMonad importRandomGennewtypeGenaGenIntRandachooseRandomaaaGenachooseboundsGennrfstrandomRboundsrvariantIntGenaGenavariantvGenmGennrmnrandsr v randsr r randsr wherer r splitr promoteaGenbGenabpromotefGennraletGenmfainmnrsizedIntGenaGenasizedfgenGennrletGenmfgenninmnrinstanceMonadGenwherereturnaGennraGenm kGennr letr r splitr Genm km nr inm nr elementsaGenaelementsxsxs liftMchoose lengthxs vectorArbitraryaIntGenavectornsequencearbitraryi noneofGenaGenaoneofgenselementsgensidfrequencyIntGenaGenafrequencyxschoose summapfstxspickxspicknkxxsnkotherwisepicknkxsArbitrary CoarbitraryclassArbitraryawherearbitraryGenainstanceArbitraryBoolwherearbitraryelementsTrueFalseinstanceArbitraryIntwherearbitrarysizednchoosenninstanceArbitraryaArbitrarybArbitraryabwherearbitraryliftM arbitraryarbitraryinstanceArbitraryaArbitraryawherearbitrarysizednchoose nvectorinstanceArbitraryaArbitrarybArbitraryabwherearbitrarypromotecoarbitraryarbitraryclassCoarbitraryawherecoarbitraryaGenbGenbinstanceCoarbitraryBoolwherecoarbitrarybvariantifbthen else instanceCoarbitraryIntwherecoarbitrarynn variant n variant coarbitrarynotherwisevariant coarbitraryndiv instanceCoarbitraryaCoarbitrarybCoarbitraryabwherecoarbitraryabcoarbitraryacoarbitrarybinstanceCoarbitraryaCoarbitraryawherecoarbitraryvariant coarbitraryaasvariant coarbitraryacoarbitraryasinstanceArbitraryaCoarbitrarybCoarbitraryabwherecoarbitraryfgenarbitrarycoarbitrarygenfPropertynewtypePropertyPropGenResultdataResultResultokMaybeBoolstampStringargumentsStringnothingResultnothingResultokNothingstampargumentsresultResultPropertyresultresPropreturnresclassTestableawherepropertyaPropertyinstanceTestableBoolwherepropertybresultnothingokJustbinstanceTestablePropertywherepropertyproppropinstanceArbitraryaShowaTestablebTestableabwherepropertyfforAllarbitraryfevaluateTestableaaGenResultevaluateagenwherePropgenpropertyaforAllShowaTestablebGenaabPropertyforAllgenbodyPropdoagenresevaluatebodyareturnargaresargaresresargumentsshowaargumentsresTestableaBoolaPropertyapropertyaFalsearesultnothinglabelTestableaStringaPropertysaPropaddfmapevaluateawhereaddresresstampsstampresclassifyTestableaBoolStringaPropertyclassifyTruenamelabelnameclassifyFalsepropertycollectShowaTestablebabPropertycollectvlabelshowv