tstorealsystemsandwehaefoundittoworkwreportexperiencepaperandpointoutpitfallstoapaperintroducestheconceptofwritingpropertiesandcthemusingSectionshowshowtodenetestdatageneratorsforuserdenedtypes ID: 371865
Download Pdf The PPT/PDF document "QuickCheck:ALightweightToolforRandomTest..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
QuickCheck:ALightweightToolforRandomTestingofHaskellProgramsKoenClaessenChalmersUniversityofTechnologykoen@cs.chalmers.seJohnHughesChalmersUniversityofTechnologyrjmh@cs.chalmers.seABSTRACTisatoolwhichaidstheHaskellprogrammerinulatingandtestingpropertiesofprograms Propertiesdescribedcallytestedonrandominputbutitisalsopossibletodenecustomtestdatagenerators epresentanberoftoolalsopointoutsomepitfallstoaRandomtestingisespeciallysuitableforfunctionalprogramsbecausepropertiescanbestatedatanegrain Whenafunctionisbuiltfromseparatelytestedcomponentsthenrandomtestingsucestoobtaingoodcoerageofthedenitionundertest 1.INTRODUCTIONensuringsoftItiserylabouruptothecostDespiteanecdotalevidencethatfunctionalprogramsrequiresomewhatlesstesting Onceittypecksitusuallyworks inpracticeitisstillamajorpartoffunctionalprogramdevThecostoftestingmotivateseortstoautomateitwhollyorpartlyAutomatictestingtoolsenabletheprogrammertocompletetestinginashortertimeortotestmorethoroughlyintheaailabletimeandtheymakeiteasytorepeattestsaftereachmodicationtoaprogram InthispaperwdescribetoolelopedtestingHaskellprograms unctionalprogramsarewellsuitedtoautomatictesting Itisgenerallyacceptedthatpurefunctionsaremheasierbecausebeforeimperativelanguageevenifwholeprogramsareoftenpurefunctionsfrominputtooutputtheproceduresfromwhictheyarebuiltareusuallynot usrelativfunctionsaboundinHaskellonlycomputationsintheIOPermissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforprotorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitationontherstpage.Tocopyotherwise,torepublish,topostonserversortoredistributetolists,requirespriorspecipermissionand/orafee.ICFP00,Montreal,Canada.Copyright2000ACM1-58113-202-6/00/0009..monadarehardtoandsocanbeatanegrain Atestingtoolmustbeabletodeterminewhetheratestispassedorfailedthehumantestermustsupplyanautoablecriteriondoingso ehatouseformalspecicationsforthispurpose ehaedesignedasimpledomainspeciclanguageoftestablesphthetesterusestodeneexpectedpropertiesofthefunctionsundertest thencksthatthepropertiesholdinalargeberofThespecicationlanguageisembeddedinHaskellusingtheclasssystem Propertiesarenormallywritteninthesamemoduleasthefunctionstheytestwheretheyservealsoascabledoctationofthebehaviourofthecode Atestingtoolmustalsobeabletogeneratetestcasesauehahosenthesimplestmethodrandomcompetessystematicmethodsinpractice eritismeaninglesstotalkaboutrandomtestingwithoutdiscussingthedistributionoftestdata RandomtestingismosteectivewhenthedistributionoftestdatafollowsthatofactualdatabutwhentestingreuseablecodeunitsasopposedtowholesyspossibledatainallsubsequentreusesisnotknoAuniformdishoosearandomclosedtermwithauniformdistributionforexampleehahosentoputdistributionundertheumantester salsoembeddedinHaskellandawytoobservdistributionoftestcases ByprogrammingathetestercannotonlythedistributioncomplexinAnimportantdesigngoalwasthatshouldbemoduleofabouthisinpracticemainlyusedfromtheHugsinehaealsowritlittleaboutHaskellsyntaxandconsequentlysupportsthefulllanguageanditsextensions ItisnotdependentonanparticularHaskellsystem Acostthatcomeswiththisdecisionisthatwecanonlytestpropertiesthatareexpressibleandobservmethodisindetectingfaults erwehaeusedinavyofapplicationsrangingfromsmallexper tstorealsystemsandwehaefoundittoworkwreportexperiencepaperandpointoutpitfallstoapaperintroducestheconceptofwritingpropertiesandcthemusingSectionshowshowtodenetestdatageneratorsforuserdenedtypes SectionbrieydiscussestheimplemenSectionpresentsanberofcasestudiesthatshowtheusefulnessofthetool Section2.DEFININGPROPERTIES2.1ASimpleExampleAsarstexamplewetakethestandardfunctionhreversesalist ThissatisesanberofusefullahasInfactthersttoofthesecNotethattheselawsholdonlyfor nitetotalpaperspecicallyeitmorelikelythatthepropertiesarecomputable InordertocktheselawsusingerepretthemasHaskellfunctions uswedenewifthesefunctionsreturnforeverypossibleargutthenthepropertieshold eloadthemintotheHugseHaskellinterpreterandcallforexampleThefunctionesalawasaparameterandappliesittoalargenberofrandomlygeneratedargumeninfactreportingOKiftheresultiserycase reportsorexampleifwemistakenlydenethenckingthelawmightproducewherethecountermodelcanbeextractedbytakingand isaratherarbitrarynbersoourlibraryprovidesaytospecifythisasaparameter Infacttheprogrammermustprovidealittlemoreinforthefunctionisactuallyoerloadedinordertobeablehandlelawswithavaryingnberofvablesandtheoerloadingcannotberesolvedifthelawitselfhasapolymorphictypeasintheseexamples ThustheprogrammermustspecifyaxedtypeatwhichthelawistobeSowesimplygivypesignatureforeachlawforOfcoursethepropertholdspolymorphicallyspecifytobequiteimportantintheorexample isassociativeforthetypebutnotforarguethatapropertyholdspolymorphically2.2FunctionsearealsoabletoformulatepropertiesthatquantifyokforexamplethatfunctioncompositionisassociativewerstdeneextensionalequalitandthenwriteypesthatfunctioncompositioniscommethenthefunctionvaluesareprintedjustasInthiscaseediscoerthatthe law weareckingisfalsebutnot2.3ConditionalLawswswhicharesimpleequationsarecontlyrepretedbybooleanfunctionaswehaeseenbutingeneralylawsholdonlyundercertainconditions videsanimplicationcombinatortorepresentsuchconditionallaorexamplethelacanberepresentedbythedenitionthelaNotethattheresultypeofthepropertyischangedfrombecausepropertyforrandomtestcasesetryckingittestcasesacandidatetestcasedoesnotsatifytheconditionitisdiscardedandanewtestcaseistried kingthelaasusualbutsometimesckingaconditionallawproducestheoutput Ifthepreconditionofalawisseldomsatisedthenwemighgeneratemanytestcaseswithoutndinganywhereitholds Insuchcasesitishopelesstosearchforcasesinwhicthepreconditionholds Ratherthanallowtestcasegenerationtorunforeverwegenerateonlyalimitednberofcandidatetestcasesthedefaultis Ifwedonotndvalidtestcasesamongthosecandidatesthenwesimreportberofperform IntheexampleweknowthatthelawpassedthetestinItisthenuptotheprogrammertodecidewhetherthisisenoughorwhetheritshouldbetestedmorethoroughly2.4MonitoringTestDataitseemsthatwetestedthethoroughlyenoughtoestablishitscredibiliterwustbecareful Letusmodifyasfollokingthelawnowproducesthemessagebinatordoesnotchangethemeaningofawbutitclassiessomeofthetestcasesinthiscasethoseistheemptylistwereclassiedastrivial eseethatalargeproportionofthetestcasesonlytestedinsertionintoanemptylist ecangetmoreinformationthanjustlabellingsometestThecomgatherallvaluesthatarepassedtoitandprintoutahistogramofthesevorexampleifwewriteemightgetasaresultSoweseethatonlycasestestedinsertionintoalistwithfairlystrongevidencethatthelawholdsitisworryingthateryshortlistsdominatethetestcasessostronglyallitiseasytodeneanerroneousversionoferthelessworksforlistswithatmostoneelemenThereasonforthisbehaviourofcourseisthatthepreisorderedbutonlyofthelistsoflengthare testcaseswithlongerlistsaremorelikelytoberejectedb isHaskell sinxfunctionapplication theprecondition Thereisariskofthiskindofproblemeverytimeweuseconditionallawssoitisalwysimportanproportionactuallytested Thebestsolutionthoughistoreplacetheconditionwithacustomtestdatageneratorfororderedlists ewritespeciesthetestdatageneratorkingthelawnoesaswouldexpect videssupportfortheprogrammertodenehisorherowntestdatawithcontroloerthedistributionoftestdatawhicewilllookatinthenext2.5InniteStructuresellfunctionesanonemptylistreturnsalistthatrepeatsthecontsofthatlistinnitelywtakealookatthefollowinglawformulatedinAlthoughinelythesincewearecomparingtoinnitelistsusingcomputablewhichdoesnotterminate Insteadwecanreforulatethepropertyasalogicallyequivtonebyusingthattinnitelistsniteinitialtsareequal Anotherissuerelatedtoinnitestructuresisquanpropertiesthatforexampleholdforallinnitelistsbutingeneralitisnotclearhowtoformulateandexecutepropertiesaboutstructurescontainingbottom 3.DEFININGGENERATORS3.1ArbitraryThewegeneraterandomtestdataofcoursedependsype troducedypeofwhicypeisaninstanceifwecangeneratearbitraryelementsinit isanabstracttyperepresentingageneratorfortypeTheprogrammercaneitherusethegeneratorsbuiltinasinstancesofthisclassorsupplyacustomgeneratorusingthebinatorwhicesawintheprevioussection ornowwedenethetype Notethatleavingtheconditionoutresultsinanerrorbecauseisnotdenedforemptylists isarandomnberseedageneratorisjustafunctionwhichcanmanufactureaninapseudorandomButwewilltreatasanypesowedeneaprimitivegeneratortochoosearandomnberinaninalandweprogramothergeneratorsintermsofit fromsimpleronestodosowedeclaretobeaninstancemethodsofthetherstoneofwhichcontructsathesecondonebeingthemonadicsequencingoperatorwhicgeneratesanandpassesittoitssecondargumenttogendenitionofneedstorandomnberseedstoitstoargumentsandisonlypassedoneseedbutluckilytheHaskellrandomnberlibraryprovidesanoperationtosplitoneseedintotypeswhichisdenedintermsofeageneratorthatappliesthepairingoperatorhdeclarationsformostofHaskell spredenedtypes 3.2GeneratorsforUser-DenedTypesforeacypethenwustrelyontheusertoprovideinstancesforuserdenedtypes Inprinciplewcouldtrytogeneratetheseautomaticallyinapreprocessorpolytypicprogrammingbutwsteadtoleaethistasktotheuser Thisispartlybecausetobealighttooleasytoimpletandeasytouseinastandardprogrammingentwedon twttoobligeuserstoruntheirprogramsthroughapreprocessorbeteeneditingthemandtestingButanotherstrongreasonisthatitseemstobevhardtoconstructageneratorforatypewithoutknosomethingaboutthedesireddistributionoftestcases producingvidecombinatorstoenableaprogrammertodenehisogeneratorseasilyThesimplestcalledjustmakesahoiceamongalistofalternativegeneratorswithauniformforexampleifthetypeisdenedbthenasuitablegeneratorcanbedenedbAsanotherexamplewecouldgeneratearbitrarylistsusingwhereweusetoapplytheconsoperatortoanarbitraryheadandtail erthisdenitionisnotreallyitproduceslistswithanproducedspecifythefrequencywithwhicheachalternativeiscedenetochoosetheconscasefourtimesasoftenasthenilcaseleadingtoanaeragelistlengthoffourelemenmoregeneraldatatypesitturnsouteneedevennercontroloerthedistributionofgeneratedSupposeypettoaoidchoosingatoooftenhencetheusethisde nitiononlyhasachanceoftermiThereasonisthatforthegenerationofaegenerationsmustterminate Iftherstfewrecursionschooseesthengenerationterminatesonlyifverymanyrecursivegenerationsallterminateandthechanceofthisissmall enwhengenerationterttoaoidthissinceweperformalargenberoftestswteachtesttobesmallandquicButthenotionofasizeishardeventodeneingeneralforypetiontypesanethereforegivetheresponsibilitforlimitingsizestotheprogrammerdeningthetestdatahangetherepresentationofgeneratorstowherethenewparameteristobeinterpretedassomekindofsizebound edeneanewcomboundgeneratesanypassingthecurrentsizeboundtopretthesizeboundinsomereasonablewyduringtestdata boundberofnodesinthegeneratedtreeswhichisquitereasonable wthatwehaeintroducedthenotionofasizeboundecanuseitsensiblyinthegeneratorsforothertypessuctegersandlistssothattheabsolutevaluerespectivlengthisboundedbythesize Sothedenitionswepretedearlierneedtobemodiedaccordinglyestressthatthesizeboundissimplyanextraglobalparameterwhicheverytestdatageneratormayaccesseveryuseofseesthesameboundedoto dividethesizeboundamongthegenerators sothatforexamplealongergeneratedlistwouldhaesmallerelemeneepingtheoerallsizeofthestructurethesame Thereasonisthatwewishtoaoidcorrelationsbeteenthesizesofdierentpartsofthetestdatawhichmightdistortthetestresults edovarythesizebeteendierenttestcasesebeginpropertboundesforagreatervyoftestcaseswhichbothmaktestingmoreeectiveandimproesourchancesofndingenoughtestcasessatisfyingthepreconditionofconditionalproperties Italsomakesitmorelikelythatwewillndasmallcounterexampletoapropertifthereisone 3.3GeneratingFunctionspropertiesariablesthenwustbeabletogeneratearbitraryfuncderstandhownoticethatafunctiongeneratoroftypeisrepresentedbyafunctionoftypeIntRandabByreorderingparametersthisisequivttotheypeIntRandbhrepresenaGenecanthusdeneanduseittoproduceageneratorforafunctiontypeprovidedwecanconstructageneratorfortheresulttypewhicwdependsontheargumenetakecareofthisdependencebydeninganewclasswhosemethodmodiesageneratorinawdependingonitsrstparameter ewillthinkofproducingenthisclasswecandenehgeneratesanarbitaryfunctionthatusesitsargumentomodifythegenerationofitsresult Inordertodeneinstancesofeneedawtoconstructgeneratortransformers ethereforedenethe constructsageneratorwhichtransformstherandomnberseeditispassedinawydependingonbeforepassingittoThisfunctionmustbedenedvsothatallthegeneratorsweconstructusingitareindependenenanylistofincanconstructageneratortransformerrisetoindependentgeneratortransformerswithaveryhighecandeneinstancesofthatchoosebeteengeneratortransformersdependingontheargumenorexamplethebooleaninstancetransformsageneratorinindependenysforandforthegeneratorscoarbitraryTruegFalsegwillbeindependenInasimilarwecandenesuitableinstancesformanyothertypes orexampletheintegerinstancejustconertsitsintegerargumentinasequenceofbitswhicharethenusedasgeneratortransformersinturn Instancesofforrecursivedatatypescanbedenedaccordingtoastandardpattern orexamplethelistinstanceisjustthatdierentlistsmappedtoindependenpingeachconstructortoanindependenttransformercomposingthesewithtransformersderivedfromeachcomponenypessamewSincetheprogrammerisresponsibleformakingthesedenitionsforuserdenedtypesitisimportantthattheybestraighecaneveninterpretfunctionsasgeneratortransformerswithaninstanceoftheformTheideaisthatweapplythegivenfunctiontoanarbitrarytandusetheresulttotransformthegivengenerInthiswofunctionswhicharedierentwillgivrisetodierentgeneratortransformers Notethatifwehadtriedtoaoidneedingtosplitrandomberseedsbydeningthemonadasastatetransformerontherandomseedratherthanastatereaderthenbeentionandwouldnothaebeenabletogeneraterandom4.IMPLEMENTINGQUICKCHECKpropertieswithavaryingnberofargumentsanddiertresulttypes oimplementthisweintroducethetypeandwecreatethetypeclass yperepresentspredicatesthatcanbecytesting meansthatitneedstobeableaterandominputandnallyproductatestresult Soaisacomputationinthemonadendinginanabstracttypewhiceepstrackofthebooleanresultofthetestingtheclassicationsoftestdataandthetsusedinthetestcase LetustakealookatsomeinstancesofAneasyypetockisofcourseurtherfunctionsforwhicenthepropertypeitselfisaninstancesothatecannestpropertycomUsingthefunctionitbecomeseasytodenetheItstypeisMoredetailsoftheimplementationcanbefoundintheappendix 5.SOMECASESTUDIES5.1Unicussauncationalgorithmwhicehaedevelopedalongwithaspecication ThiswasquiterevbothasregardstheimpactthathasonprogrammingandthepitfallsthatmustbeaItistoolargetindetailjustdiscussthelessons5.1.1ImpactonTypeDeFirstofalltheuseofhadanimpactonthedesignofthetypesintheprogram edenedthetypeoftermstobeuniedasratherthantheequivouldprobablyhahosenotherwise Thetypeeuseddistinguishesbeteenastringusedasaconstructorandastringusedinotherconandbeteenanaturalnberusedasavariablenameandanintegerusedinotherconypesforexamplethanforstrings HadwstringsthenitisveryunlikelythatwouldevergenerateitissameconstructornametInsteadwhosetogenerateconstructornamesusinghgivesusagoodchancethatgeneratedtermswillbeatleastpartiallyuniable ewisewelimitedunicationariablesintestdatatoasmallset Ofcoursewecouldhaeusedthesecondypeaboandspeciedacustomtestdatageneratorwithanexplicitineachpropert Butitismoreconttolettestdatabeautomaticallygeneratedusingsooneisypes areotheradvtagestodoingsoalsoitpermitsthetypeertodetectmoreerrors Sousingtroducingnewtypesinprograms 5.1.2CheckingFunctionalPropertiesAunierneedstomanagethecurrentsubstitutionandalsothepossibilityoffailuresinrecursivecalls Aconytodosoistouseamonad edenedaunicationrepresentedbyafunctionwithoperationstodenean extensionalequality operatoronmonadicaluesandckboththemonadlawsandpropertiessuc5.1.3ErrorsFoundreportfoundalargenberoferrorsinthisexample Infactnoerrorsatallwerefoundintheunieritself ThisisprobablymoreareectiononthenberoftimestheauthorshaprogrammedunierspreviouslythanontheeectivenessofeknowhowtodoitquitesimplyOntheotherhandwedidnderrorsintheorexampleedenedasubstitutionfunctionhrepeatedlysubstitutesuntilnovariablesinthedomainofthesubstitutionremainandstatedtheobviouspropertealedthispropertytobefalseitholdsonlyforacyclicsubstitutionsotherwiseaninnitetermisgeneratedandtheequalitytestloops Thiserrorwusingthefunctionhprintsouttheartstoeverytestcasebeforeitcksit ereobligedtocorrectthespecicationtomadeusthinkharderaboutthepropertiesofourcodeanddocumentthemcorrectly OnthedoulatingthespecicationcorrectlyperhapswritingtheimplemenThiswaspartlybecausepredicatessuchasarenontrivialtodeneagoodsettheorylibrarywouldhaehelpedhere 5.1.4AFalseSenseofSecurityexperiasthefalsesenseofsecuritythatcanbeengenderedwhenone sprogrampassesalargenberoftestsintrivialehaealreadyproblemwhenwdiscussedconditionalpropertiesinthisexampleitbituspropertiestheformsinceourunierreturnswhenitfails Withalitfairlylikelytobeuniablesincevariablesoccurquiteoftenandifeithertermisavariablethenunicationwillalmostcertainlysucceed Ontheotherhandifneithertermisaariablethentheprobabilitthattheywillunifyissmall usthecasewhereonetermisavariableisheavilyotedamongthetestcasesthatsatisfythepreconditionwefoundthatoeroftestcaseshadthisproppropertecanhardlyconsiderthattheywerethoroughlyThesolutionwastouseacustomtestdatageneratoregeneratedygeneratingrandomtermandreplacingrandomsubtermsbintodierenThisusuallygeneratesuniabletermsalthoughmayfailtowhenvariablesareusedinconsisteninthetoterms Withthismodicationtheproportionoftrivialcasesfelltoareasonable Thisexperienceunderlinestheimportanceofindistributionofpropertiesareused 5.2CircuitProperties5.2.1LavainaNutshell"isatooltodescribesimulateandformallyvifyhardwaisasocalleddlanguagewhicpropertiesexpressedinanexistingprogramminglanguageinthiscaseTheideaistoviewahardwarecircuitasafunctionfromvidesprimitivecircuitssuchasandMorecomplicatedcircuitsaredenedbycombiningthese videsinputsandtheoutputsarecalculated urthermoretheLaasystemdenescombinatorsforcircompositionpositionreplicatesitinacolumnofcircuitsconnectingthev5.2.2PropertiesinLavaPropertiesofcircuitscanbedenedinasimilarwexampletodenethepropertythatacertaincircuitiscomewesaypestainingsignalsinthiscaseapair PropertiescanbeformallyvedothisbyproinputstothecircuitorpropertandcalculatingaconcreteexpressioninaHaskelldatatyperepresenthecircuit externaltheoremproAllthisisdonebytheoafunctionHereishoecanuseittovthatasocalledhalfaddercomponentiscommTheLaasystemprovidesanberoffunctionsandcomnatorstocontlyexpresspropertiesandformallyvinLavaThoughweareabletoverifypropertiesaboutcircuitsinbenettoollikTherearetomainreasonsforthat rstonethatcallinganexternaltheoremproeryheatprocess WhenvatheoremprovingarequitebigandweoftenhaetowaitforalongtimetogetananswSoatypicaldevtcycleistowritedownthespecicationofthecircuitrstthenmakeanimplemenitforobviousbugsandlastlycalltheexternaltheoremproerforverifyingthecorrectness HereisanexampleofhowtouseinLaAddingthistestingmethodologytoLaaturnedouttobequitestraighardbecauseLaaalreadyhadanotionofproperties estingcanbedoneforallcircuitypestialcircuitscontaininglatcesimplyckthecircuitpropertyforasequenceofinputs 5.2.4HigherOrderTestingThesecondreasonforusingtestinginLaaissimplythatpropertiescanonlydealwithatmostrstorderlogicsandtheLasystemisonlyabletogenerateformulasofthattype Sometimeswouldliketoverifypropertiesaboutorexampleprovingthatdistributeso isveryhardtoverifyinLaallfactsuchpropertiesarehardtoverifyautomaticallyingeneralwecandoitforsmallxedsizeshoButsincethesekindofpropertiesforarbitrarycircuits Adrakisthatwehaetoxthetypesofthesecirpropertiesaboutthemarepolymorphicinthecircuits inputandoutputtypes 5.2.5ErrorsFoundTheauthorsusedthelibrarywhiledevarithmeticcircuits alreadyusedinthedevtprocessbutonlyinavlimitedandadhocwwmhmorethoroughtestingaspossible okinds efounderrorsthatourformalvtionmethodwouldhaefoundaswlogicalerrorsintheButsecondlyealsofounderrorsduetothefactthatrandominputalsomeansrandominputorexampleforanbitadderweonlyuseandformallyspecicinputsizes Randomtestingthatwehadforgottentodeneoneofthesecases!5.3PropositionalTheoremProvingorteachingpurposesweimplementedtodierenmethodspropositionalmethodsvisPutnammethodwhichusesbackingtogenmodels methodmethodtonthedilemmaproofsystemtogatherinformationabouttheliteralsintheclauseset functiontakesanextraargumentanhisthesocalledsaturationlevelaparameterwhiclimitsthedepthoftheproofsandusuallyliesbeteenandIftheresultofitmeansthatthereasaconIftheresultisitmeansthaterymodeloftheclausesetshouldhaasasubmodel ismhmorestraighardtoimpropertystatedaboNotethatwsomestatisticsdictionwhentheresultwandthesizeofthecaseofealsoexpressedthatwedisqualifyatestcasewhenWiththehelpofthispropertfoundbugs!Thesebugswereduetoimplicitunjustiedassumptionswaboutbothrithmsassumedthatnoclausesintheinputcouldconthesameliteraltwiceandthefunctionassumedthatnoneoftheinputclauseswasempthniquesasinsectionpropertytookaboutandfromtheoutputwcouldseethedistributionofasabout$ 5.4PrettyPrintingAndyGillreportedaninterestingstoryaboutusingtous HeuseditindevelopingavtofWtedhisvtfunctionallyusingHaskstillusingHaskellheusedastatemonadwithexceptionstoelopanimperativeimplementationofthesamelibraryTheideawasthatthesecondimplementationmodelswhatgoesoninaJaaimplemenThenheexpressedtherelationshipbeteenthetodifproperties ThisquicklypointsoutwheremyrasoningisfaultyandprovidesgratteststocatchthecornersoftheimplemeneproblemswerefoundthethirdofwhichshowethatIhadmertwocinmyimplementationthatIshouldnothave urthermorehemadeanimprotinthewreportsexamplesfoundareverylargeanditisdiculttogobactothepropertyandunderstandwhyitisacounterexample erwhenthecounterexampleisanelementofatreeshapeddatatypetheproblemcanoftenbelocatedinoneofthesubtreesofthecounterexamplefound GillextendedclasswithanewmethodtoitstforInsomecasesmhsmallercounterexampleswtinglibrarywasportingthestateandexceptionmonadmodelinHaskelltoJaHethenusedtogenbercodeordertockthattheJaaimplementationwasequivtothetoHaskellmodels 5.5EdisonChrisOkisalibraryofecientdatastructuressuitableforimplementationanduseinfunctionalprogramminglanguages Hehasusedtostateandtestpropertiesofthelibraryerydatastructureinthebeenhasincludedseveralextramodulesespeciallyforformpropertiesaboutthesedatastructures HereportsMyexehasmostlybenthatofaverysatis eduser letsmeEdisonwithpr maybofthee ortofmyprevioustestsuiteanddoesamuchbjobtobasakialsomentionsadrakhavingtodowiththeellmodulesystem HeoftenusesonespecicationofAnaturalwytodothisistoplacethespecicationinonemoduleandeachimplementationinaseperatemodule Butsincethespecicationreferstotheimplementationthenthespecicationmoduleustimportthetationonetlyundertest asakiwasobligedtoeditthespecicationmodulebyhandbeforeeachtestsoastoimporttherightimplemenhpreferablewouldbetoparameterisethespecicationonanimplementationmoduleylefunctorswouldbereallyhelpfulhere!6.DISCUSSION6.1OnRandomTestingerynaiveapproacSystematicmethodsareoftenpreingeneralatestadequacycriterionisdenedandproceedsadequacycriterion orexampleasimplecriterionisthatwpathwithexceptionsforloopsbefolloedinatleastonetest Awidevyofadequacycriterahaebeenproposedarecentsurveyis ehahosennottobaseonsuchanadequacycriterion InpartthisisbecausemanycriteriawneedreinterpretationbeforetheycouldbeappliedtoHaskprogramsitismhlessclearforexamplewhataconwpathisinalanguagewithhigherorderfunctionsandlazyevInpartsuchacriterionwouldforceustousemhmoreheatmethodsevenmeasuringpatherageforexamplewouldrequirecompilermodicationsandthustietoaparticularimplementationofellnamelytheonewemodiedtocollectpathinforGeneratingtestdatatoexerciseaparticularpathrequiresconstraintsolvingonemustndinputvalueswhicetheseriesoftestsalongthegivenpathproducespeciedresults WhilesuchconstraintsolvingmaybefeasibleforarithmeticdatafortherichsymbolicdatatypesfoundinHaskellprogramsitisadicultresearchprobleminitswnrigherapartfromthedicultyofautomatingsystematictestingmethodsforHaskellthereisnoclearreasontobelievbetterDuranandNtafoscomparedthefaultdetectionprobabilitofrandomtestingwithpartitiontestinganddiscoeredthatthedierencesineectivenessweresmall Hamletandylorrepeatedtheirstudymoreextensivandcorroboratedtheoriginalresults AlthoughpartitiontestingistlymoreeectiveatexposingfaultstoquoteHamlet stsurveyBytaking morointsinardomtestanyadvantageartitiontestmighthavehadisdout orsmallprogramsinparticularitislikelythatrandomtestcaseswillindeedexerciseallpathsforexamplesothatgoodyankpropertiesofindividualfunctionsbutthefunctionstheycallaretestedindependen Soevenwhenisusedtotestalargeprogramwealwystestasmallpartatatime ThereforewemayexpectrandomtestingtowparticularlywhgreaterautomatingsystematictestingforHaskellourchoiceofrandomtestingisclear 6.2CorrectnessCriteriaTheproblemofdeterminingwhetheratestispassedornotisknownastheacleprOnesolutionistocompareprogramoutputwiththatofanotherversionoftheproperhapsanolderorperhapsabut obviouslycorrect vanexecutablespecicationmightplaythesamer%Thiskindoforaclecaneasilybeexpressedasapropertalthoughourpropertiesaremhmoregeneral eroftenonecanckthataprogram soutputiscorrectmhmoreecientlythanonecancomputetheoutBlumandKannanexploitthisintheirworkonprogramwhichclassiestheprogram soutputascorrectorwithahighprobabilityofclassifyingcorrectlyanddoessowithstrictlyloercomplexit TheydistinguishprofromprogramtheirproposalisthatprogramsshouldalwktheiroutputandindeedinfurtherworkBlumetal edhowprogramswhichusuallyproducecorrectanswerscanevwrongoutputinparticulardomains Ofcourseresultcerscanalsobeexpressedaspropertiesalthoughweusethemfortestingratherthanasapartofthenalprogram spropertylanguageishoermoregeneralthanresultcViaconditionalpropertiesorspecictestdatageneratorswecanexpresspropertieswhichholdonlyforasubsetofallpossibleinputs Thuswoidtestingfunctionsincaseswhichleadtoruntimeerrorsorcasesinedonotcareabouttheresult orexamplewedonottestinsertionintoanunorderedlistthereisnopoinindoingso etaresultcermustverifythataprogramproducesthe correct outputinallcaseseventhosewhicareuninpropertiesarenotlimitedtockingtheresultofanindividualfunctioncallthepropertythatanoperatorisassociativeforexamplecannotreallybesaidtocktheresultofanyindividualuseoftheoperatorbutstillexpressesauseful global propertthatcanbecedbytesting propertiesspecicationrectlywasusedintheDAISTSsystemfortestingabstractdatatypeswhichcompiledequationalpropertiesintestingcodealthoughtestcaseshadtobesuppliedbythekingautomatictestcasegenerationDAISTSdidnotneedequivtsofourconditionalandquantiedpropAlthoughthelanguageusedwasimperativeabstractdatatypeoperationshadtobeforbiddentosideeecttheirtsthustheprogramstobetestedwereessenbefunctional yandHamletdescribeatechniquefortestingC classesagainstanalgebraicspecicationwhichiserthespecicationlanguagemustberestrictedinordertoguaranteethatspecicationsbeanimated Thereseemstobenopublishedworkonautomatictestingofprogramsagainstspecications esimply ethatfunctionalprogramsandpropertybasedspecicationsareaverygoodmatcecanusethegivenpropertiesdirectlyfortesting erembeddingthespecicationlanguageinHaskellpermitsustowriteverypoandexiblepropertiesumof6.3TestDataGenerationtoolslimiteddomainswiththegoalofmatchingthedistributionofactualdataforthesystemundertestthesocalledationalprInthiscasestatisticalinferencesaboutthemeantimebeteensystemfailurescanbedrawnfromthetestresults InordertomorecomplexdataitispopuhtoitwpressallthedesiredpropertiesoftestdataforexamplerandomprogramThereforethegrammarswereenhancedwithactionsorextendedtoattributegrammars ThisapproachasbeenmostusedfortestingcompilersalthoughMaurerarguesforitsuseinmanycontexts Grammarshaebeenusedforsystematictestingforexamplethegeneratedtestdataisrequiredtoexerciseproductionybeandnotedtheterminationproblemforrecursivegrammars HissolutionthoughwasjusttoincreasetheprobabilitiesofgeneratingleaessothatevtualterminationisguaranOurexperienceisthatthisresultsinfartoohighaproportionoftrivialtestcasesandthereforeinecienttestingustbeproperlyebelieveourmethodofcontrollingsizesismhsuperior ItseemsthattheneedtolearnacomplexlanguageofextendedgrammarshashinderedtheadoptionofthesemethodsbeddingainHaskellweprovideatleastthesamecapabilitiesbutsparetheprogrammertheneedtolearnmorethanafewnewoperators tthesametimeweprovideallthepoerandyneededtogeneratetestdatasatisfyingcomplexintsinalanguagetheprogrammeralreadyknolinkinggeneratorstotypesviaHaskell sclasssystemwereetheprogrammeroftheneedtospecifygeneratorsatallinmanycasesandwheretheymustbespeciedtheprogrammer sworkisusuallylimitedtospecifyinggeneratorsforhisorherownnewtypes 6.4OnRandomnessehaeencounteredsomeinterestingproblemsinreasoningaboutprogramswhichuserandomnbergeneration Inparticularthemonadwhicisbasedonisnotamonadatall!ConsidertherstmonadlaSinceourimplementationofbindsplitsitsrandomnberoperandseedsonthetosidesoftheequationandythereforeproducedierentresults Sothelawsimplydoesnothold hoerweconsiderthelawtobetruebecausethetosidesproducethesameresultseveniftheresultsdierforanyparticularseed Butwhatpreciselydowemeanby morally ecannotxtheproblemjustbyreinterpretingequalityfortheypeclaimingthetosidesarejustdierentrepresengoodbecausewecanactuallyobservethedierenceatothertypesysupplyingarandomnberseedsomethingwehaetobeabletodoiftheypeistobeuseful Insteadwehatoreinterpretwhatwemeanbyprogramequivalenceinthepresenceofrandomnbergeneration theimperativeprogramismorallyequivttothesameprogramwiththeassigntsreversedinthesamesensebutofcourseproducesatresult Thereissomeinterestingsemantictheorytobedonehere 6.5OnLazyEvaluationehaearguedinthepastthatlazyevaluationisaninaluableprogrammingtoolthatradicallychangesthewproperties aconicthereaboperfectlyinnitestructuresinspecicationsprovidedthepropertieseactuallytestarecomputableforexamplewecantestthatarbitrarilylongprexesofinnitelistsareequalratherthancomparingtheliststhemselvmonadhasabindoperationbecausewesplittherandomnberoperandthentheotherandsowecanfreelydenegeneratorsthatproduceinniteresults Whatwecannotdoisterminationinatestresult SowecannottestforexamplethepropertOntheotherinaumantestercannotebeentestlazyprogramssatisfactorilybyhandsofarthenwearenotinaworsepositionifweuseetahtestercanobservethatproducesanerrormessagefromtheevaluationofwithoutproducinganyotheroutputrstandcanthusinferthatthepropertyaboeholds TheproblemisthattheHaskellstandardprovidesnowyforatomakethesameobettherearevariousextensionsofHaskellwhicdoindeedmakethispossible SomeworkdonebyAndyGillhasshownthatgivensuchextensionswecouldformandckpropertiessuchastheoneaboeusing6.6SomeRemajoristhatitencouragesustoformulateformalspecicationsthusimprovingourunderstandingofouritisopentotheprogrammertodothisfewreallydoperhapsbecausethereislittleandperhapsbecausespecicationalueifthereisnockatallthatitcorrespondstothetedprogram addressesboththeseisitgivesusashorttermpaoviaautomatedtesting andsomereasontobelievethatpropertiesstatedinamoduleactuallyhold enlybetspecicationrstcategoryisuselesstodiscoerexceptinsofarasithelpswithfurthertestingittellsusnothingabouttheactualThethirdcategoryisobviouslyusefulinasensethesearetheerrorswetestinordertond Butthesecondcategoryisalsoimportanteveniftheydonotrevealamiseinthecodetheydorevealamisunderstandingaboutwhatitdoes Correctingsuchmisunderstandingsimproourabilitytomakeuseofthetestedcodecorrectlylater WhenformulatingspecicationsonerapidlydiscoerstheneedforalibraryoffunctionsthatimplementcommonmathearetionofnitesettheoryforusewithyoftheabstractionsinitaretooinecienttobeofmhuseinprogramsbutinspecicationswheretheobjectistostatepropertiesasclearlyandsimplyaspossibletheycomeintheiroBecauseofthisdierenceinpurposethereisaneedforlibrariesspecicallytargetedatspecications majorthedistributionanddecidetlymanytestshaebeenrun AlthoughweproystocollectthisinformationwecannotcompeltheprogrammertoAprogrammerwhodoesnotgainingafalsesenseofsecurityfromalargenberofinadequatetests erhapswecoulddeneadequacymeasuresjustonthegeneratedtestdataandthuswarntheuseratleastinthiskindofsituation 7.CONCLUSIONSehaetakentorelativelyoldideasnamelyspecicationsasoraclesandrandomtestingandfoundwystomakthemeasilytoHaskvideanembeddedlanguageforwritingpropertiesingexpressivenesswithoutthelearningcost Thelanguagetainscontfeaturessuchasquantiersconditionalsandtestdatamonitors eprovidetypebaseddefaultrandomtestdatageneratorsincludingrandomfunctionsgreatlyreducingtheeortofspecifyingthem Thirdlyeprovideanembeddedlanguageforspecifyingcustomtestdatageneratorswhichcanbebasedonthedefaultgeneratorsgivinganercontroloertestdatadistribution alsointroduceanoelwyofcontrollingsizewhengeneratingrandomelementsofrecursivedatatypes oldtechniquesworksextremelywellforHaskThefunclocalpropertiessincealldependenciesofafunctionareexplicit Andpreciselyrandomtestingisknowntoworkverywellforsmallnegrainedprogramsandiseectiveinndingfaults thetoolislighandeasytouseandprovidesashorttermpaoforexplicitlystatingpropertiesoffunctionsinaprogramwhichgreatlyincreasestheunderstandingoftheprogramfortheprogrammeraswellasfordocumentationpurposes ouldliketothankAndyGillChrisOkasakiandtheanonymousrefereesfortheirusefultsonthispaper 8.REFERENCESS AnyandR Hamlet Automaticallyckingantationagainstitsformalspecication InIrvineSoftwareSymppagesMarch RolandBackhousePatrikJanssonJohanJeuringandLambertMeertens GenericProgrammingAntroduction InenotesinComputerSciencolume BjesseK ClaessenM SheeranandS Singh areDesigninHaskell IneonFunctionalPrBaltimore AM BlumandS Kannan Designingprogramsthatktheirwork Inc stSymposiumontheoryofComputingpages" ACMMay M BlumM LubandR Rubinfeld Selftesting$correctingwithapplicationstonproblems Inc ndSymposiumontheTheoryofpages" ACMMay A CelentanoS C ReghizziP DellaVignaandC Ghezzi CompilertestingusingasenePreExp" K ClaessenandD Sands ObservableSharingforunctionalCircuitDescription InAsianComputereConferPhetThailand AM DavisandH Putnam AcomputingprocedureforticationtheoryJournaloftheAssociationforComputingMachinery" J DuranandS Ntafos AnevaluationofrandomansactionsonSoftwareEngineJuly J GannonR HamletandP McMullin Dataabstractionimplementationspecicationandtesting ansPrgLangandSystems D Hamlet Randomtesting InJ MarciniakeditordiaofSoftwareEnginepages"" R HamletandR Tylor Partitiontestingdoesnotinspirecondence ansactionsonSoftwarDecember J Hughes WhunctionalProgrammingMatters InD TurnereditorchTopicsinF AddisonW M P Jones TheHugsdistribution Currenailablefrom M Maurer Generatingtestdatawithenhancedtextfreegrammars IEEESoftwar"GunnarStk ASystemforDeterminingPropositionalLogicTheoremsbyApplyingValuesandRulestoTripletsthatareGeneratedfromaF SwedishPtNo ""approedU S PtNo ""EuropeanPNo PhilipWadler Theoremsforfree!eonFunctionalPrammingandComputerLondonSeptember PhilipWadler AprettierprinterMarch Draftpaper H ZhuP HallandJ Ma SoftareunittesterageandadequacyComputingSurveys"December" Appendix:ImplementationHereweshowtheimplementationofthecodeisaailablefrommoduleQuickCheckwhereimportMonad importRandomGennewtypeGenaGenIntRandachooseRandomaaaGenachooseboundsGennrfstrandomRboundsrvariantIntGenaGenavariantvGenmGennrmnrandsr v randsrrrandsr whererr splitrpromoteaGenbGenabpromotefGennraletGenmfainmnrsizedIntGenaGenasizedfgenGennrletGenmfgenninmnrinstanceMonadGenwherereturnaGennraGenmkGennrletrr splitrGenm kmnrinm nr elementsaGenaelementsxsxs liftMchooselengthxsvectorArbitraryaIntGenavectornsequencearbitraryinoneofGenaGenaoneofgenselementsgensidfrequencyIntGenaGenafrequencyxschoosesummapfstxspickxspicknkxxsnkotherwisepicknkxsArbitrary CoarbitraryclassArbitraryawherearbitraryGenainstanceArbitraryBoolwherearbitraryelementsTrueFalseinstanceArbitraryIntwherearbitrarysizednchoosenninstanceArbitraryaArbitrarybArbitraryabwherearbitraryliftM arbitraryarbitraryinstanceArbitraryaArbitraryawherearbitrarysizednchoosenvectorinstanceArbitraryaArbitrarybArbitraryabwherearbitrarypromotecoarbitraryarbitraryclassCoarbitraryawherecoarbitraryaGenbGenbinstanceCoarbitraryBoolwherecoarbitrarybvariantifbthenelseinstanceCoarbitraryIntwherecoarbitrarynnvariantnvariant coarbitrarynotherwisevariantcoarbitraryndiv instanceCoarbitraryaCoarbitrarybCoarbitraryabwherecoarbitraryabcoarbitraryacoarbitrarybinstanceCoarbitraryaCoarbitraryawherecoarbitraryvariantcoarbitraryaasvariantcoarbitraryacoarbitraryasinstanceArbitraryaCoarbitrarybCoarbitraryabwherecoarbitraryfgenarbitrarycoarbitrarygenfPropertynewtypePropertyPropGenResultdataResultResultokMaybeBoolstampStringargumentsStringnothingResultnothingResultokNothingstampargumentsresultResultPropertyresultresPropreturnresclassTestableawherepropertyaPropertyinstanceTestableBoolwherepropertybresultnothingokJustbinstanceTestablePropertywherepropertyproppropinstanceArbitraryaShowaTestablebTestableabwherepropertyfforAllarbitraryfevaluateTestableaaGenResultevaluateagenwherePropgenpropertyaforAllShowaTestablebGenaabPropertyforAllgenbodyPropdoagenresevaluatebodyareturnargaresargaresresargumentsshowaargumentsresTestableaBoolaPropertyapropertyaFalsearesultnothinglabelTestableaStringaPropertysaPropaddfmapevaluateawhereaddresresstampsstampresclassifyTestableaBoolStringaPropertyclassifyTruenamelabelnameclassifyFalsepropertycollectShowaTestablebabPropertycollectvlabelshowv