ThistestwillpassforcorrectimplementationsofthedashboardNextsupposewewanttobealittlebitmorepreciseItsohappensthattheDashboardshouldconvertthesensorspeedgiveninmstokmhieifreadSpeedreturns10u ID: 124004
Download Pdf The PPT/PDF document "mocks;generatingthemocksisextremelypainf..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
mocks;generatingthemocksisextremelypainfulifthemockingframeworkimposesarbitraryrestrictionsonwhatwecanwrite.Wewantthefreedomtocombinethefeaturesofthemockingframeworkhoweverwelike.ClearandunambiguousspecicationsInmostmockingframeworks,themeaningofaspecicationcanbequitesubtle,apointweillustrateinSection2.Forexample,theseframeworkshaverulesforresolvingambiguity,andtheusercanexploittheserulesinwritingspecications.Thisisconvenientbutmakesithardtosaywhatagivenspecicationmeans.OurAUTOSARmockingspecicationsare,bynecessity,sometimeslongandcomplex.Theyaretrickytogetright.Thelastthingwewantfromourframe-workisasubtlesemantics!Wewanteachmockingspecicationtohaveasimple,declarativemeaning.Likewise,wewantthemockingframeworktorejectambigu-ousspecications,ratherthanmakearbitrarychoices:thisreducesthenumberofpotentialpitfalls.Onemightexpectthatwecoulduseanambiguousspecicationtomockanondeterministiccomponent,iftheframeworkresolvesambiguityrandomly.Webelievethisisthewrongapproach,becauseitmakestestsunrepeatable.Instead,thetestsuiteitselfshouldchooseaparticulardeterministicinterpretation.Thispaperpresentsanewmockingframeworkwhichisexpressive,isbuiltfromasmallcoreoforthogonalfeatures,hasasimple,compositionalsemanticswhereeveryspecicationhasaclearmeaning,andwhichavoidsmakingarbitrarychoicesduringtestexecutionbyrejectingambiguousspecications.AlthoughourrequirementscamefromtheAUTOSARtestingproject,webelievethesefeaturesarecompellingintheirownright,andareespeciallyimportantwhentestinglargecomponents.Thecontributionsofthepaperareasfollows:{Wepresentanewframeworkformocking(Sections3{4).Theframeworkisgiventwosemantics,asimple,compositionaldenotationalsemanticsandasmall-stepoperationalsemantics.Thetwosemanticshavebeenprovedequivalent(seetheaccompanyingtechnicalreport[13]).{Weavoidmakingarbitrarychoicesduringtestexecutionbyrulingoutam-biguousmockingspecications.Specically,weprovideaproceduretovali-datespecications(Section5)whichrulesoutspecicationswhichaream-biguous.Thevalidationissoundwithrespecttothesemantics.Perhapssur-prisingly,itisalsocomplete,whichmeansthatifwerejectaspecication,itmustbeambiguous,andwecanmoreoverndatracethatdemonstratestheambiguity.Thesoundnessproof,asketchofthecompletenessproof,andalinktothefullformalizationarefoundinthetech.report[13]{Weextendourbasicframeworktomakeitpracticalanddescribehowtoimplementitinamemory-ecientway(Section6).{Wereportonourexperienceusinganearlierversionofthisframeworkinalargeindustrialcasestudywritingspecicationsfor,andthentestingimple-mentationsof,automotivesoftware(Section7). Thistestwillpassforcorrectimplementationsofthedashboard.Next,sup-posewewanttobealittlebitmoreprecise.ItsohappensthattheDashboardshouldconvertthesensorspeed,giveninm/s,tokm/h;i.e.ifreadSpeedreturns10,updateDisplayshouldbecalledwith36asitssecondargument.Wechangetheexpectedcallsto:EXPECT_CALL(mDisplay,updateDisplay(Display::SPEED,36)).Times(1);EXPECT_CALL(mDisplay,updateDisplay(Display::SPEED,_)).Times(2);Surprisingly,thistestfailseveniftheimplementationdoesthecorrectthing.Itturnsoutthatexpectationsareputonastack,soaretestedinthereverseorderthattheyaredened.Thus,thecorrectwaytospecifythiswouldbeEXPECT_CALL(mDisplay,updateDisplay(Display::SPEED,_)).Times(2);EXPECT_CALL(mDisplay,updateDisplay(Display::SPEED,36)).Times(1);eventhoughthecallreturning36happensrst.Andindeed,thistestpasses.NowsupposethatwechangethespecicationsothatthenalcalltoreadSpeedreturns10insteadof12:5:EXPECT_CALL(mSensor,readSpeed()).WillOnce(Return(10)).WillOnce(Return(6.7)).WillOnce(Return(10));EXPECT_CALL(mDisplay,updateDisplay(Display::SPEED,_)).Times(2);EXPECT_CALL(mDisplay,updateDisplay(Display::SPEED,36)).Times(1);Wemightexpectthistesttopass,butitdoesnot!Thereasonisthat(bydefault)expectationsarenotremovedfromthestackoncetheyarefullled.ThusassoonasthefunctionupdateDisplayiscalledwithargument36itremainsonthestackasbeingcalledonce.ThesecondtimeitiscalleditincreasesthecallcountoftheupdateDisplaywithargument36insteadofincreasingthecallcountofupdateDisplaywitharbitraryargument.Theabovemockingspecicationlooksambiguous,sinceasecondcallwithargument36canbehandledintwoways:itcanbeacceptedbytherstclauseorrejectedbythesecond.Themockingframeworkhasarbitrarilychosenthesecondway.ThewaytoxthistestinGoogleMockiseithertoexpect36twice,ortotellthesecondexpectationtoretireonceitisfullledwiththefeatureRetiresOnSaturation().Wechoosethesecondoptionandthetestnowpasses:EXPECT_CALL(mDisplay,updateDisplay(Display::SPEED,_)).Times(2);EXPECT_CALL(mDisplay,updateDisplay(Display::SPEED,36)).Times(1).RetiresOnSaturation();Whileconventionalmockingframeworkshaveaprecisesemantics,itisquitecomplicated.Therearesubtleinteractionsbetweenfeaturesbecausetheseman-ticsisnotcompositionalandambiguousspecicationsaregivenanarbitrary,thoughdocumented,semantics. readthelightsensoreachtimetheyupdateapartofthedisplay,whileothersonlyreaditonceperloop.Butwheneverthedashboardreadsthelightsensor,itmustthenupdatethedisplaybrightness.Thedashboardmayreadthelightsensoranynumberoftimesperloop,whichwecanmodelusingtheoperator:(read light7!6:light display7!())Thethreespecicationsabovecapturethreeaspectsofthedashboard.Tomockthedashboardasawhole,wecombinethethreespecicationswiththeparallelcompositionoperator\jj".Thissaysthatthedashboardmayinterleavetheexecutionofthethreespecications,butmustrespecttheorderofeventswithineachsinglespecication.Forexample,thedashboardmayreadthespeed,thenthelightsensor,thensetthedisplaybrightness,thenupdatethedisplay:(read speed7!5:833:update display(speed;21)7!())jj(read light7!6:light display7!())jj((read battery7!234:update display(battery;70)7!())+")Fromthisspecicationwecanautomaticallygeneratemocks.Ourmockscheckthatthecallsmadebythedashboardpreciselymatchthecallsinthespecication:noextracalls,nomissingcalls,andallcallsintherightorder.4AProcessCalculusforMockingWehavenowseenallofthefeaturesofourmockinglanguage,andbeginaformaltreatmentofitssemantics.Mockingspecicationsresembletermsinaprocesscalculus,andtheirsyntaxissummarisedbelow.Aneventa7!zdenotescallingthefunctionatogetresultz.Fornowwetreataandzabstractly;inSection6wewillbreathelifeintothecalculusbyallowingeventstoberealfunctioncalls.p::="ja7!zjp:qjpjjqjp+qjpWewanttoassignmeaningtomockingspecications.Wethereforedeneadenotationalsemanticsintermsoftraces;atraceisasequenceofevents.ThelanguageL(p)ofaprocessisthesetoftracesthattheprocessaccepts,i.e.thatsatisfythemockingspecication,andisdenedasfollows:L(p:q)=fstjs2L(p)^t2L(q)gL(p+q)=L(p)[L(q)L(pjjq)=fujs2L(p)^t2L(q)^uisaninterleavingofsandtgL(p)=fs1s2snjn2Nandforalli,si2L(p)gL(a7!z)=fa7!zgL(")=f"gThissemanticsiscompactandeasytounderstand,andidealforunderstandingthebehaviourofamockingspecication.However,itisoflittleuseforimple-mentingthemockingframework.Itacceptsorrejectswholeexecutiontraces, 5AmbiguitydetectionAsarguedintheIntroductionandSection2,wewanttoforbidambiguousspecications,becausetheyleadtocomplexsemantics,ortounrepeatabletestsifresolvedatrandom.Anexampleofanambiguousspecicationinourlanguageisa7!z1+a7!z2:iftheprogramcallsa,wedonotknowwhethertoreturnz1orz2.WewillseeinSection6.1thattheuserdoesnotdecidewhatvalueaneventwillreturnuntilthateventiscalled,sowemustalsorejecta7!z+a7!z|wehavenowayofknowingthatbotheventswillalwaysreturnthesamevalue.Thissuggeststhefollowingdenitionofambiguity:pisambiguousifforsomecalla,therearetwoapplicablereductionrulesforp!a.Aprocessisalsoambiguousifitreducestoanambiguousprocess.Ourprocessa7!z+a7!zisambiguousbecause,forthecalla,therules+Land+Rbothapply.Herearesomeexamplesofambiguousprocesses:{a7!z1+a7!z2isambiguous,asabove.Ingeneral,ifp!aandq!a,thenp+qisambiguous.{(a7!z1:b7!z2)jjb7!z3isambiguous:afteracalltoa,itreducestob7!z2jjb7!z3,inwhichtherearetwob-reductions.Ingeneral,ifpandqhaveoverlappingalphabets,thenpjjqisambiguous.{(a7!z1+"):a7!z2isambiguous:callinga,wecouldreturneitherz1orz2.{Alongthesamelines,a7!z1:(a7!z2+"):a7!z3isambiguous:afteracalltoa,weareleftwith(a7!z2+"):a7!z3,essentiallythepreviousexample.Theexamplesabovetellushowtodetectambiguity.Wewillstartwith+andjj.Notethatthetwoconstructsneeddi erentrules:thesecondexampleisambiguous,butreplacingjjby+itbecomesunambiguous.With+,therstcallneedstotelluswhichalternativetochoose,butwithjjeverycallneedstohavethisproperty.{Ifp!aandq!a,thenp+qisambiguousbecauserules+Land+Rbothapply.{Ifa2alphabet(p)\alphabet(q)thenpjjqisambiguousbecausewecanreachaprocessp0jjq0wherep0!aandq0!a;rulesjjLandjjRthenbothapply.(Thealphabetofaprocessissimplythesetofeventsthatappearsyntacticallyinit.)WewilldeneafunctionpXthatchecksthatpisunambiguous.Fornowweonlydenetheeasycases:p+qX=pX^qX^:9a(p!a^q!a)pjjqX=pX^qX^alphabet(p)\alphabet(q)=;a7!zX=true"X=trueSequentialcompositionistrickier.Lookingat(a7!z1+"):a7!z2,weseethatthereductionrulesThenLandThenRbothapply,therstbecause 6FromProcessCalculustoMockingFrameworkThegoalofthissectionistoturntheprocesscalculusintoafully- edgedmockingframework.Abasicimplementationissimple.Werstcheckthatthemockingspecicationpisunambiguous.Toexecutep,wewaitforthesystemundertesttomakeacalla.Wecheckifp!a;zqforsomeq;ifnot,thecalliserroneous.Otherwise,wereturntheresultztothecaller,andcontinuebyexecutingq.Finally,whenthetestnishes,wecheckthatthenalprocessisaccepting.6.1MatchingInourexamplessofar,aneventspeciesasingleconcretecallsuchasupdate display(speed;21)andaconcreteresultlike5:833.Inreality,wedonotalwaysknowthefunctionargumentssoprecisely,andneedarichereventlan-guage.Inourframework,aneventspeciesapatternoffunctioncalls.Forex-ample,wemaywriteupdate display(speed; ),wherethe\ "isawildcard;thismatchesanycalltoupdate displaywheretherstargumentisspeed.Apatternsimplystandsforanyoftheconcretecallswhichitmatches.Wealsoallowtheevent'sreturnvaluetodependonthecallarguments.Theusercanassociateanevaluationfunctionwitheachevent,whichisgiventhecall'sconcreteargumentsandcomputesthereturnvalue.4Notethateachoc-currenceofaneventinthemockingspecicationcanhaveadi erentevaluationfunction:thesamecallneednotalwaysreturnthesameresult.Aneventthatre-turnsaconstantresultisadegeneratecasewheretheevaluationfunctionignoresitsarguments.Weneedtobecarefulthatwecanstillexecutemockingspecicationsthatusepatternmatching,andcheckthemforambiguity.Executingthespecicationisnotaproblem:weonlyneedtobeabletocheckifaconcretecallmatchesaparticularevent.Givenaprocesspandacallc,wecheckifthereisaneventthatpcanconsumeandwhichmatchesc.Finally,weusetheevaluationfunctionassociatedwiththeeventtocalculatethereturnvalue,andreducep.Wecanalsocheckthespecicationforambiguity,aslongaswecantellwhetheranytwoeventsintersect.(Twoeventsintersectifthereisasinglecon-cretecallthatmatchesbothofthem.)Itwillhelptowriteouttheexistingrules,usingequalityexplicitlywheneverwecomparetheeventsoftwoprocesses:p+qX=pX^qX^:9a9b(p!a^q!b^a=b)pjjqX=pX^qX^:9a9b(a2alphabet(p)^b2alphabet(q)^a=b)a7!zX=true"X=truep:qX=pX^qX^:9a9b(p?a^q!b^a=b)pX=pX^:9a9b9c9q(p!aq^q?b^p!c^b=c) 4ThisiswhywecouldnottelliftwoeventshavethesamereturnvalueinSection5. {Wedonotneedtoaugmentpjjq,thoughpandqthemselvesareaugmented.Thereductionrulesarethesameasbefore.{Weaugmentasingleevent,a7!z,witha agthatindicateswhetherwehaveperformedtheevent.Ifthe agshowsthatwehavealreadyperformedtheevent,wemaynolongerperformit.Replicationisthetrickiestcase,becauseinexecutingpwemayexecutepanunlimitednumberoftimes.Tohandlethisweneedtobeabletoresetaterm,whichsetsits agsbacktotheirinitialstate.Wheneverpinpdoesnotacceptaneventa7!z,butdoesaccepttheemptytrace,weresetpandfeeda7!ztoit;thiscorrespondstounrollingpintheoriginalsemantics.Wealsoaugmentpwitha agthatrecordswhetherwehaveperformedanyreductionsonit;this agissetaftertheveryrstreduction,andallowsustomodelthefactthatpalwaysacceptstheemptytrace.6.3ExtensionsThemockinglanguagewehavepresentedsofarisquiteminimal.Whenwritingmockingspecicationsinpracticeweusealargerrepertoireofconstructs.Con-structswe'vefoundusefulincludepermutations,optionalbehavioursandniterepetition.Thepermutationconstructoperatesonalistofbehavioursandissimilartoparallelcompositionbutdoesn'tallowinterleavingofbehaviours:thebehavioursmustexecuteoneafteranother,butinanarbitraryorder.Constructslikethesearedenableinthelanguagewe'vealreadypresented.Forexample,anoptionalpissimplyp+".However,inourimplementationwe'veaddedthemasprimitivesforreasonsofeciency.Itisparticularlyimpor-tanttohavepermutationsbeaprimitiveintheimplementationsinceitsencodingintoourcalculuscausesanexponentialblow-upinthesizeoftheprocess.AsanexampleofusingpermutationsconsidertheexamplewithparallelcompositionfromSection3:(read speed7!5:833:update display(speed;21)7!())jj(read light7!6:light display7!())jj((read battery7!234:update display(battery;70)7!())+")Thisspecicationallowsallvaluestobereadbeforeanyupdatesareperformed.Thismightbeexactlythefreedomonewishestoexpress.However,supposethatwewishtoensurethatthecallstoread speedandupdate displayshouldhappeninimmediatesequencewithoutbeinginterruptedbyanyoftheothercalls,andlikewisewiththecallstoread batteryandupdate display.Wecanachievethisbyusingpermutationsinsteadofparallelcomposition,writingthepermutationofp,qandrasperm[p;q;r],asfollows:perm[read speed7!5:833:update display(speed;21)7!();(read light7!6:light display7!());(read battery7!234:update display(battery;70)7!())+"] SinceChasnoobjects,wewillsimplycomparetheexpressivenessofthetwoapproaches.Themaindi erenceisthatGoogleMockprovideslotsofdefaultbehaviour:expectationsareputinparallelbydefault,therearedefaultreturnvalues,etc.Thelanguagewedenehasnodefaultbehaviour|everythingisexplicit.Bothapproacheshavetheirmerits,buthiddendefaultsrequireawell-educateduser.Intermsofexpressiveness,wehaveobservedthreekeydi erences:{GoogleMockhasstate,i.e.oneactionmaysetavariablethatcanbereadbyalateraction.Thisisnotincludedinourlanguagesincewehavenothadtheneedforit.Itwouldbepossibletoextendourlanguagewithstate,butthemoreinterestingquestioniswhywehaven'thadtheneedforit.Webelievethereasonisthecompositionalityandexpressivenessofourmock-inglanguage.Comparetowritingaregularexpressionandimplementinganequivalentstatemachine.Regularexpressionsprovideadeclarativeandcompositionalinterfacewithouttheneedforstatewhichismuchsimplertousethanhavingtomaintainthestateofthestatemachineexplicitly.{GoogleMockonlydoesreplicationofsingleevents;itisnotpossibletorepeat,forexample,asequenceofcalls.Inourparticularusecase,Lisacentralingredient,thusnothavingitwouldhavepresentedaproblemtous.{Finally,theredoesnotseemtobeawaytoexpressp+qinGoogleMock.OnecouldsayatMost(1)forbothpandq,butthatwouldnotcatchthecasewhenneitherorbotharecalled.Againthisiscentraltoourusecase,butperhapsoneoftenmanageswithoutitinordinaryunittesting.Anareacloselyrelatedtomockingisruntimemonitoring.Inparticular,Jass[4]allowsmonitoringof"traceassertions"expressedinaCSP-likelanguage;ifthemonitoredcodeperformsaneventinthealphabetoftheprocessthatisnotpartofanytrace,thenanexceptionisraised.Thetraceassertionlanguageisdescribedbyexampleandformalpropertiesarenotstatedorproven.Ingen-eral,run-timemonitorscanallownon-determinisminthemonitor,becausethiscannotleadtonon-determinisminthetestoutcome.Becausemockingsuppliesreturnvaluestothecodeundertest,thennon-deterministicmockingwillleadtonon-deterministictestoutcomes.Similarly,modelcheckerscanallownon-deterministicenvironmentssincetheycanexplorebranchingexecutions,collectconstraints,andusesolverstondinterestingcases:sinceeachtestexecutioncanfollowonlyonebranchthenwedonotenjoythesamefreedom.OurmockinglanguagesharesmanysimilaritieswiththelanguagePSL[1],usedbythehardwarecommunityforspecifyingandverifyingcircuits.PSLisdividedintoseverallayersandoneoftheselayersisamodellinglayer,usedforspecifyingpartsofthedesignwhicharenotyetimplemented.Althoughsimilarinspirittoourlanguage,PSL'smockinglanguagenaturallydi ersonmanydetailsasittargetshardware,notsoftware.