TestOraclesLucianoBaresiDip.ElettronicaeInformazionePolitecnicodiMilan - PDF document

TestOraclesLucianoBaresiDip.ElettronicaeInformazionePolitecnicodiMilanoMilano,Italybaresi@elet.polimi.itMichalYoungDept.ofComputerScienceUniversityofOregonEugene,Oregon,USAmichal@cs.uoregon.edu4thAugust2001Allsoftwaretestingmethodsdependontheavailabilityofanoracle,thatis,somemethodforcheckingwhetherthesystemundertesthasbehavedcorrectlyonaparticularexecution.Anidealoraclewouldprovideanunerringpass/failjudgmentforanypossibleprogramexecution,judgedagainstanaturalspeciÞca-tionofintendedbehavior.Practicalapproachesmustmakecompromisestobal-ancetrade-offsandprovideusefulcapabilities.Thisreportsurveysproposedap-proachestotheoracleproblemthataregeneralinthesensethattheyrequireneitherpre-computedinput/outputpairsnorapreviousversionofthesystemundertest.Thesurveyisnotencyclopedic,butdiscussesrepresentativeexamplesofthemainapproachesandtacticsforsolvingcommonproblems. PartiallysupportedbytheItalianNationalResearchCouncil(CNR).ThisworkhasalsobeensupportedbytheDefenseAdvancedResearchProjectsAgencyandRomeLaboratory,AirForceMaterielCommand,USAF,underagreementnumberF30602-97-2-0034.TheU.S.GovernmentisauthorizedtoreproduceanddistributereprintsforGovernmentalpurposesnotwithstandinganycopyrightannotationthereon.Theviewsandconclusionscontainedhereinarethoseoftheauthorsandshouldnotbeinterpretedasnecessarilyrepre-sentingtheofÞcialpoliciesorendorsements,eitherexpressedorimplied,oftheDefenseAdvancedResearchProjectsAgency,RomeLaboratory,ortheU.S.Government. 1Introduction32OraclesforTransducers43EmbeddedAssertionLanguages63.1Anna........................................73.1.1Complexobjects..............................93.1.2VirtualText................................103.2CAssertionSystems................................113.2.1APP....................................113.2.2Nana....................................123.3Eiffel........................................143.4JavaAssertionSystems..............................153.4.1iContract.................................164ExtrinsicInterfaceContracts204.1ADL........................................204.1.1ADL/C..................................214.1.2ADL/Java.................................224.2TOG........................................244.3AlgebraicSpeciÞcations..............................264.3.1DAISTS..................................264.3.2Self-checkingADTs............................285PureSpeciÞcationLanguages315.1ZandObject-Z..................................315.1.1TestTemplates..............................345.2TemporalOracles.................................365.2.1TemporaloraclesfromGIL........................385.3SCR........................................395.4Multi-LanguageSpeciÞcations..........................416TraceChecking426.1ProtocolConformanceTesting..........................436.1.1Wp:Arepresentativeprotocolconformancetestmethod.........446.2OraclesforGUIs..................................467LogFileAnalysis488Discussion49 1IntroductionAllsoftwaretestingmethodsdependontheavailabilityofanoracle,thatis,somemethodforcheckingwhetherthesystemundertesthasbehavedcorrectlyonaparticu-larexecution.Inmuchoftheresearchliteratureonsoftwaretestcasegenerationortestsetadequacy,theavailabilityoforaclesiseitherexplicitlyortacitlyassumed,butap-plicableoraclesarenotdescribed.Inthecurrentindustrialpracticeofsoftwaretesting,theoracleisoftenahumanbeing.Relyingonahumantoassessprogrambehaviorshastwoevidentdrawbacks:accuracyandcost.WhilethehumanÒeyeballoracleÓhasanadvantageovermoretechnicalmeansininterpretingincomplete,natural-languagespeciÞcations,humansarepronetoerrorwhenassessingcomplexbehaviorsorde-tailed,precisespeciÞcations,andtheaccuracyoftheeyeballoracledropsprecipitouslywithincreasesinthenumberoftestrunstobeevaluated.Evenifitweremorede-pendable,theeyeballoracleisprohibitivelyexpensiveforlargevolumesoftestcases,andsomaybecomealimitingfactorwhenotherpartsoftestingareacceleratedwithautomation.AnidealtestoraclewouldsatisfydesirablepropertiesofprogramspeciÞcations,suchasbeingcompletebutavoidingover-speciÞcation,whilealsobeingefÞcientlycheckable.Thesepropertiesareinconßict,andmanyoftheinterestingissuesandtrade-offsinthedesignoftestoraclesystemscomeinvariouswaysthattensionsbe-tweendesirablepropertiesofspeciÞcationsandnecessarypropertiesofimplementa-tionsareresolved.AsanoraclesystemtakesonmoreofthecapabilitiesofaÒrealÓspeciÞcationlanguage,orprovidesmorepowerfulfacilitiesforderivingrun-timechecksfromexternalspeciÞcations,severalproblemsmustbesolved.ApproachestobridgingthegapusuallyinvolvesomecombinationofrestrictingthespeciÞcationlanguagetowhatcanbeeffectivelyorefÞcientlychecked(e.g.,disallowingquantiÞcationoverinÞ-nitesets),mappingimplementationentitiestospeciÞcation-levelentities,and/ortakingadvantageofthepeculiaritiesofparticularapplicationdomains.Theresearchliteratureontestoraclesisarelativelysmallpartoftheresearchliter-atureonsoftwaretesting.Someolderproposalsbasetheiranalysiseitherontheavail-abilityofpre-computedinput/outputpairs[Pan78,Ham77]oronapreviousversionofthesameprogram,whichispresumedtobecorrect[Cha82].Theformerhypothesisisusuallytoosimplistic:beingabletoderiveasigniÞcantsetofinput/outputpairswouldimplythecapabilityofanalyzingthesystemoutcome.Thelatterhypothesissometimesappliestoregressiontesting,butisnotsufÞcientinthegeneralcase.Weyukerhassetforthsomeofthebasicproblemsandarguedthattrulygeneraltestoraclesareoftenunobtainable[Wey82].Thisreportsurveysproposedapproachestoautomatedtestoraclesthataregeneralinthesensethattheyrequireneitherpre-computedinput/outputpairsnorapreviousversionofthesystemundertest.Thesurveyisthematicratherthanchronological,groupingsystemstocompareandcontrastrelatedapproachestovariantsofafewbasicproblemsanddesigntrade-offs. 2OraclesforTransducersManyprogramsaretransducersthatreadaninputsequenceandproduceanoutputsequence,maintainingalogicalcorrespondencebetweentheinputandoutputstruc-tures.Forexample,averylargenumberofprogramsinwebservicesaretransducersfromsomenativeÞleformattohypertextmarkuplanguage(HTML).ItisnoteasytoexpresstheintendedbehaviorofthesetransducersininternalassertionsorinterfacespeciÞcationsforprogrammodules;itispreferabletoexpressandchecktherelationbetweentheinputsequenceandoutputsequence.AspeciÞcationforsuchaprogram,andatestoraclederivedfromthatspeciÞcation,mustbebasedonadescriptionofthosestructures.Theprimarytechnologyfordescribingandrecognizinglogicalstructuresintex-tualinputisparsingwithcontext-freegrammars,soitshouldnotbesurprisingthatgrammarswouldplayapartinspecifyingandcheckingtransducers.DayandGan-non[DG85]havedescribedasystemthattranslatesaformalspeciÞcationofinputandoutputÞlesintoanautomatedoracle.TheprototypesystemdescribedbyDayandGannonisspeciÞctoprogramswritteninCFPascal,asimpliÞedversionofPascalwithonlyasprimitivedatatypes,butinprincipleitshouldbeappli-cabletootherlanguagesincludingthescriptinglanguages(Perl,Awk,Python,etal)commonlyusedtowritesimpletransducers.ThespeciÞcationsfromwhichDayandGannonextracttestoraclesaredividedintoasyntaxsectionandasemanticssection.ThesyntaxusestwoBNFgrammars(seeExample1)tospecifytheformatofinputandoutputÞles,respectively,atthecharacterlevel.ThesemanticsdeÞnesrules(seeExample2)thatspecifytherelationshiptheoutputmusthavewiththeinput.Example1AnexamplespeciÞcationtakenfrom[DG85],whichrequiresaÞle(line)textandaÞle(line)ofblanks,asinput,andproducesaÞle(line)asoutputthatcon-rightjustiÞedtothelengthof.Blanksthatseparatewordsshouldbedistributedequally.FileIn,FileOut,Width;FILE=Width;EOLN_TOKEN=ON;&#xWidt;&#xh000;::=lan;&#xk_s3;�.10;tringoln;lan;&#xk_st;&#xring;::=lan;&#xk000;lan;&#xk_st;&#xring;0.1;|FILE=FileIn;EOLN_TOKEN=ON;ile;&#xIn00;::=&#xWlis;&#xt000;lan;&#xk_st;&#xring;0.1;oln;|lan;&#xk_s3;�.10;tringoln;&#xWlis;&#xt000;::=&#xWlis;&#xt000;lan;&#xk000;&#xword;|&#xword;&#xword;::=&#xword;har;|har;lan;&#xk_st;&#xring;::=lan;&#xk000;lan;&#xk_st;&#xring;0.1;|&#xlamb;� FILE=FileOut;EOLN_TOKEN=ON;ile;&#xOut0;::=&#xWbli;&#xst00;oln;|&#xWbli;&#xst00;lan;&#xk_st;&#xrin3;�.10;goln;|lan;&#xk_s3;�.10;tringoln;&#xWbli;&#xst00;::=&#xWbli;&#xst00;lan;&#xk_st;&#xrin3;�.10;g&#xword;|&#xword;::=&#xword;har;|har;lan;&#xk_st;&#xring;::=lan;&#xk000;lan;&#xk_st;&#xring;0.1;|lan;&#xk000;TheÞrstsetofrulesdeÞnesÞle:Itcanbeasequenceofblanks,possiblyempty.ThesecondsetofrulesdeÞnesÞleasasequenceofwordsseparatedbyblanksistheemptytoken).ThethirdsetofrulessimilarlydeÞnesÞleSemanticrulescanbecomposedofboththestandardcomparisonsbetweenintegers,and),booleans(),andlists,bags,andsets(),anduser-deÞnedfunctions.Special-purposefunctionsareprovidedforoperatingontextualsequences:decomposeaÞleintolines,eachcontainingasinglecharacterorword,respectively.,andcreatelists,sets,andbagsfromÞlesthatcontaincomponentsondifferentlines.returnsthenumberofelements(lines)inaÞle.Theuser-deÞnedfunctionsarewrittenintheimplementationlanguageofthepro-gramundertest,inthiscasePascal.TheonlyinputmustbeatextÞle,whiletheoutputcanbeanothertextÞle,aboolean,oraninteger.Example2ThesemanticrulesforthepreviousBNFgrammars.BString_Lengths:File;EQ_Distw:Boolean;Number(Chars(FileIn))Number(Chars(Width))List(Words(FileOut))=List(Words(FileIn))Number(Chars(FileOut))=Number(Chars(Width))EQ_Dist(BString_Lengths(FileOut))=TRUEThissectioncomprisestwospecial-purposefunctionsandfourrules.Function returnsaÞlethatcontainsthelengthsofblankstringsseparatingthewordsonaÞle; returnsTRUEifthelengthsareequallydistributed,thatis,thedif-ferencebetweenthegreatestandlowestvaluesisone.TheÞrstrulerequiresthelengthoftheinputÞlebelessthanorequaltothelengthoftheÞleofblanks(.)ThesecondrulestatesthathelistsofwordsintheinputandoutputÞlesmustbethesame.Thethirdrulerequiresthatthenumberofcharactersinbethesame.Theforthrulerequirestheblankstobeequallydistributed.Thesyntaxandsemanticssectionsarecompiledtogethertoobtainanoraclepro-gramforcheckingconsistencyofanoutputtextwiththecorrespondinginputtext. 3EmbeddedAssertionLanguagesAssertionlanguagesallowexpressionsofintenttobeembeddeddirectlyinprogramsourcecode.Typicalembeddedassertionlanguagesstatepropertiestobecheckedataparticularcontrolpointintheprogram,directlyintermsofprogramminglanguageconstructsandentities.TheprototypicalembeddedassertionlanguageisthemacrooftheCprogramminglanguage,whichsimplyevaluatesabooleanexpressionandprintsanerrormessageiftheexpressiondoesnotevaluatetoEarlydevelopmentofembeddedassertionlanguageswasaimedatleastasmuchtowarddebuggingastesting,withnoparticularemphasisonrelatingembeddedas-sertionstoamoreabstractorglobalspeciÞcationofintent.[Tay83]and[Lv85]areamongtheÞrstassertionsystemsinwhichembeddedassertionsarecon-sideredasaformofprogramspeciÞcationintheirownright.ConsideringembeddedassertionsasspeciÞcations,andassertionsupportasawayofusingthosespeciÞcationsastestoracles,raisesseveralproblems,amongthem:Non-localassertions:Embeddedassertionsareevaluatedatparticularpointsintheexecutionofaprogram,typicallybytreatingtheassertionasaprogramstate-ment.SpeciÞcationssometimesdescribepropertiesthatshouldbeinvariantdur-ingacomputation,independentofcontrolpoint,butitwouldbeunwieldytoplacecorrespondingassertionsatallrelevantprogrampoints.Manyembed-dedassertionlanguagesprovideforprecondition/postconditionpairsassoci-atedwithaprocedureasawholetobeevaluatedatthebeginningandreturn(s)fromtheprocedure.Classinvariants,supportedbyassertionlanguagesforsev-eralobject-orientedlanguages,areessentiallypost-conditionsassociatedwiththeclassconstructorandeachmethod(oreachmethodthatmodiÞestheobjectstate).Muchlesscommonissupportforassertionsthatareevaluatedateachpointwhereaconstraintexpressedinanassertioncouldbeviolated.Statecaching:SpeciÞcationsoftenconstrainrelationsbetweenvaluesatdifferentpointsinexecution.Inparticular,procedurepostconditionassertionstypicallyrelatetheprogramstatebeforeandafterexecutionoftheprocedure.Evaluationofsuchas-sertionsrequiressavingacopyofpartsoralloftheÒbeforeÓvaluesmentionedintheassertion.Thiscanbeproblematicwhensomeofthosevaluesarelargeorcomplex,suchaswhentheprocedureundertestmanipulatesalinkeddataAuxiliaryvariables:InadditiontoÒbeforeÓvalues,programspeciÞcationsmayrefertootherentitiesthatdonotexistinnormalprogramevaluation.TheseareknownasÒghostÓorÒauxiliaryÓvariables.Severalassertionlanguagesprovideameans WewilltreatseparatelysystemsinwhichassertionsareusedprimarilytoassociateprogramstatewithanexternalmodelorspeciÞcation.WealsodonotconsideranotationtobeanÒembeddedassertionlanguageÓifitislargelydistinctfromtheunderlyingprogramminglanguage,regardlessofwhetheritisembeddedintheprogramminglanguage.IfanassertionappliestoeachinstanceofaclassthenaclassinvariantsufÞcesforevaluatingtheas-sertionateachpointwhereitcouldbeviolated.Classinvariantsdonotprovidethisfunctionalitywhentheassertionshouldapplyonlytoparticularobjectinstances,ortheassertionappliestosomethingoutsidetheclasssystem,e.g.,anvariableinJava. todeÞneanduseauxiliaryvariables.Ageneralruleisthatnormalprogramexecutionmustnotbeaffectedbyanycomputationonauxiliaryvariables.QuantiÞcation:SpeciÞcationsmakeheavyuseofuniversalandexistentialquantiÞ-cation.SometimesevaluationofquantiÞersisstraightforwardlymappedtopro-gramloops,butthisisnotalwaysanacceptablestrategy.InaspeciÞcationlan-guagedesignedfordescribingrequiredprogrambehaviorasclearlyandsuc-cinctlyaspossible,itisnaturaltomakefreeuseofquantiÞcationoverlargeandevenoverinÞnitesets.Forexample,itisperfectlyreasonabletostatethatalltheelementsofarrayAoccurinarrayBasA[i]=B[j]TheloopinterpretationofthequantiÞersisproblematicifAandBeachhave1000items,particularlyiftheassertionexpressesaninvariantthatispreservedbyswapsoftwoelementsofAwithinatightloop.Worse,thereisnothinginprinciplewrongwithassertingexceptthatevenverycleverevaluationstrategieswouldbeunlikelytoobtainadeÞnitiveanswerinahumanlifetime.Thefollowingsectionssurveyanumberofembeddedassertionlanguages,particu-larlywithrespecttothewaysinwhichtheyaddresstheseissues.3.1Anna(ANNotatedAda)[Lv85,LvHKBO87,SRN85,Luc90]isaspeciÞcationno-tationforAdaprograms.istheprimaryancestorofmanyofthemorerecentexecutableassertionlanguagesincluding,althoughsomeofthekeyfea-turesdiscussedhereappearedearlierinanassertionsystemfortheHAL/Slanguage[Tay80,Tay83].extendsthebaseAdalanguagewithconstructsintendedtoallowastyleofprogramminginwhichspeciÞcationandimplementationareacontinuousprocess.Whileourfocusinthisreportisasanassertionlanguageforproducingtestora-cles,itgrewoutofearlierresearchinformalveriÞcation,andwasintendedtobeusefulforstaticveriÞcationaswellasdynamictesting.Run-timecheckingwasprovidedformostbutnotallAnnaassertions.Programself-checksusableastestoraclesarecon-structedbytransformingspeciÞcationsintoAdacodeforprogramself-checks.ViolationofanassertedpropertycausesthepredeÞnedexception tobeannotationsarewrittendirectlyinthesourcecodeasÒformalcomments,Ói.e.,astextthatistreatedascommentsbytheAdacompilerbutfollowssyntacticand Theideaforthisexample(FermatÕslasttheorem)isduetoRichardN.Taylor.Ada83,theversionofAdacurrentwhentheprojectwasactive semanticrulesthatareinterpretedbythelanguageprocessor.FormalcommentsaredividedintoannotationsvirtualAdatextAssertionspersearegiveninannotations,markedasformalcommentsinwhicheachlinebeginswith.TheyaredeÞnedusingtheAdasyntax,extendedwithquantiÞers:forallx:T=�forallvaluesof(sub)type,ifisdeÞnedthenistrue.Eachannotation:(1)hasitsownscope,deÞnedbyapplyingAdascoperules;(2)canuseonlyÒvisibleÓentities,thatis,eitheractualorvirtualvariablesthatareavail-ablewithinthescope;(3)canbegeneric:atemplateforannotatingtheinstantiationsoftheunit;(4)cannothaveside-effectsontheactualprogram.providesdifferentkindsofannotationsforthedifferentAdaconstructs.jectannotations(Example3(a))constrainvaluesofobjectswithindeclarativeregions.Theyareequivalenttoasetofassertions:oneassertionforeachlineofcodethatcouldmodifytheobject.Typeandsubtypeannotations(Example3(b))constrainatypeorsubtype:theyextendtheAdaconcept,andlikeobjectannotationsareequlenttoasetofassertionsplacedateachpointatwhichanobjectofagiventypecanbemodiÞed.Example3annotations[Lv85]M,N:INTEGER:=0;--|N=M;a)TheobjectannotationrequiresthatbealwayslessthanorequaltosubtypeEVENisINTEGER;--|whereX:EVEN؂.;　=Xmod2=0;b)Thetypeannotationontyperequiresthateachvalueassignedtoavariableofbedivisibleby2.function"/"(NUMERATOR,DENOMINATOR:INTEGER)returnINTEGER;--|whereDENOMINATOR�0;c)Thesubprogramannotationenforcesapreconditionrequiringthatallcallstohavenon-zeroDENOMINATORprocedureBINARY_SEARCH(A:inARRAY_OF_INTEGER;KEY:inINTEGER;POSITION:outINTEGER);--|whereORDERED(A),--|out(A(POSITION)=KEY),--|raiseNOT_FOUND�=forallIinA’RANGE�=KEY�A(I); quantiÞersextendÞrstorderlogicquantiÞersbybeingapplicabletocollectionsinwhichisundeÞnedforsomevalues.VirtualvariablesarevariablesintroducedinvirtualAdatext,describedbelow.AÒgenericÓunitinAdaissimilartoaÒtemplateÓclassinC++,butsomewhatmoreßexibleinitsparameterization. d)Thekeywordintroducesapostconditionassertion.Thepropagationannotationensuresthat israisedonlywhenisorderedandisnotacomponentStatementannotationsspecifypropertiesofstatements:theirscopeisdeterminedbythecompoundstatement(block)inwhichtheyaredeclared.Subprogramannota-(Example3(c))extendtheAdaspeciÞcationpartofsubprogramsandprovideawaytoclearlystatethebehaviorofasubprogramindependentlyfromitsbody.Suchannotationsincludeconstraintsonformalparameters,resultsoffunctioncalls,andcon-ditionsunderwhichexceptionsshouldbepropagated.Exceptionpropagationannota-(Example3(d))specifyexceptionalbehaviors:theyannotateexceptionhandlers,statements,andunitsthatmaypropagateexceptions.Contextannotationslowprogrammerstospecifytheuseofnon-localvariableswithinaprogramunit.3.1.1ComplexobjectsComplexdatastructuresaretypicallyimplementedinAdausingpackages,muchasclassesareusedinotherobject-basedandobject-orientedlanguages.Anoperationonacomplexobjectisimplementedasaprocedureorfunctionofthepackage(whataC++orJavaprogrammerwouldcallaÒmethodÓ).AconstraintonacomplexdatastructureislikeastructuralinvariantinveriÞcationofanimplementationofanabstractdatatype,inthatitshouldbeapreconditionandpostconditionofeachcompleteoperationonthedatastructure,butneednotholdatintermediatepointsduringtheimplementationofeachoperation.constraintsoncomplextypesfollowthisapproach,evaluatingtheassertionsonlyuponreturnfromeachpackageprocedureorfunction.IngeneralitisnotsufÞcienttospecifyadataabstractionbydescribingtheeffectsofindividualoperations.Rather,thebehaviorofadataabstractionisoftenspeciÞedbydescribingtheobservableeffectsofsequencesofoperations,orbyrelatingtheresultsofdifferentsequencesofoperations,asinalgebraicspeciÞcations[GHW85,Gut77].ForthecaseinwhichthelocaldataencapsulatedinanAdapackageisusedtorepresentaninstanceofanabstractdatatype,providesawaytodenotethewholeinternalstateofapackageandtodenotethestateofapackageafterasequenceofoperations.Example4packageaxioms[Luc90,pg157].packageSTACKis--|axiom--|forallS:STACK’TYPE;X,Y:ITEM�=--|S[PUSH(X);POP(Y)]=S[POP(Y);PUSH(X)],--|S[PUSH(X);POP(Y)]=S;Inexample4,STACK’TYPEreferstothewholeinternalstateofthepackage.TheÞrstequationstatesthatPUSHandPOPoperationscommute(whenbothterminatesuccessfully;theaxiomdoesnotsayanythingaboutwhathappensifanexceptionisraisedduetostackunderßoworoverßow),andthesecondequationstatesthatPUSHandPOPareinverses.Adaallowsauser-writtenfunctiontooverridetheequalityoperationdenotedby,sothisassertioncouldcomparethestatesafterapplication ofanappropriateabstractionfunction,asisusualfordescribingcorrectnessconditionsofabstractdatatypeimplementations[Gut77].axiomnotation,togetherwithquantiÞers,isexpressiveenoughtostateconditionsforwhichastraightforwardtranslationintoarun-timecheckwouldbeun-acceptable.Example5UniversalquantiÞcationinanaxiom[Luc90,pg159].forallA,B,N:INTEGER�=AmodB=(A+N*B)modBAlthoughtheaxiominexample5isaconciseandclearstatementofapropertyofoperation,onewouldnotliketochecktheconditionbytranslatingittoatriplynestedloopoverallrepresentableintegers.3.1.2VirtualTextFormalreasoningaboutaprogramisoftenfacilitatedbytheadditionofvariablesthatarenotneededintheactualprogramcomputationbutareneededasÒbookkeepingÓforacorrectnessargument.ThesehavebeenvariouslycalledÒghostvariables,ÓÒdummyvariables,ÓorÒauxiliaryvariables.ÓIn,ghostvariablesandcomputationsareintroducedinvirtualAdatext(Example6),markedwith.Thetypes,functions,andvariablesintroducedinvirtualAdatextarevisibleinothervirtualAdatextandinannotations,butnotintheAdaprogram.allowsthebodiesofentitiesintroducedinvirtualtexttobedeÞnedusingeitherannotationsorvirtualtext,i.e.,AdacodethatdeÞnesthefunctionbody.Inthelattercase,virtualconceptsbecomeanÒexecutableÓmeansfortestingandanalyzingdeliveredprograms.VirtualtextmustbelegalAda,withafewadditionalrestrictions.VirtualtexttreatsactualAdaobjectsasread-onlyvaluesthatitcannotchange.VirtualtextalsocannothideentitiesoftheunderlyingAdaprogram,i.e.,thesamenamecannotrefertotwodifferent(virtualandactual)entities.Example6virtualtext[Lv85]packageSTACKis--:functionLENGTHreturnNATURAL;procedurePUSH(X:inITEM);--|whereinSTACK.LENGTHMAX,--|out(STACK.LENGHTH=inSTACK.LENGTH+1);procedurePOP(X:outITEM);endSTACK;packagebodySTACKistypeTABLEisarray(POSITIVERANGE�)ofITEM;SPACE:TABLE(1..MAX);INDEX:NATURALrange0..MAX:=0;--:functionLENGTHreturnNATURAL --|wherereturnINDEX;isseparate;endSTACK;Packageusesthevirtualfunctiontospecifythesemanticsofitspro-cedures;thefunctionisdeÞnedthroughanannotation.NoticealsotheuseofkeywordsintheannotationofproceduretodeÞnepre-andpostconditions,respectively.(TheisseparateclauseisstandardAda,indicatingthatthebodyofthefunctionappearsinadifferentsourceÞle.)3.2CAssertionSystemsTwoassertionsystemsforC,,canbeviewedaspartialre-implementationsfortheCprogramminglanguage.Thecontributionsofareprimarilymethodological,i.e.,inthestudyofhowanassertionsystemcanbeeffectivelyused(particularlyforsoftwaretesting),andwhatfeaturesarereallyneededforeffectiveuse,ratherthaninventionofnewfeatures.arealsorepresentativeoftwomarkedlydifferentimplementationapproachesforassertionsystems.isim-plementedasapre-processor,andfeaturessuchasvaluecachingareimplementedprimarilythroughtranslation.,incontrast,isintegratedwithaparticularprogramdebuggerandexploitsdebuggerfeaturestoimplementvaluecaching.3.2.1APP(AnnotationPre-Processor)[Ros92,Ros95],developedinandfortheUNIXenvi-ronment,hasthesamecommandinterfaceas,thestandardpre-processorofUNIXcompilers.ItextendswiththecapabilityofÒexpandingÓtheannotationsas-sociatedwithprograms.recognizesassertionswrittenasspecialcomments:theymustbeenclosedin/*@@*/(Example7).CommentscanbenestedinassertionsfollowingtheC++Õssyntax:acommentstartswithandendsattheendoftheline.Example7speciÞcationoffunction ([Ros95])intsquare_root(x)intx;assumex�=0;returnywherey�=0;returnywherey*yx&&x(y+1)*(y+1);Theassertionsstatethat,beforeexecutingthefunction,mustbenon-negative;afterexecution,mustbenon-negativeandmustbebetween+1),inclusive.AssertionsaredeÞnedusingaslightlymodiÞedversionofCÕsexpressionlanguage.forbidsassignments,butaddstheoperatoranditerators.Assignments,and TheexamplesherearetakenfromRosenblum[Ros95]andusetheolderK&RdialectofC,ratherthanthenewerANSIC. assignment-likeoperators(e.g.,),cannotbeusedtoavoidside-effects.As-sertionsshouldonlyevaluateprogramstatesandnotchangethem.Theoperatorusedtorequirethatanexpressionbeevaluatedintheentrystate(beforestate)ofthefunctionthatcontainstheexpression.AssertionscanbeclassiÞedbythepoint(s)atwhichtheyareevaluated.Pre-conditionsareintroducedbythekeyword,postconditionsbythekeyword,postconditionconstraintsonreturnedvaluesbythekeyword,andconstraintsonintermediatestatesarespeciÞedusingthekeyword.Thesecor-respondtothespeciÞcationsofassertionsonewouldencounterinaFloyd-orHoare-styleprogramcorrectnessargument[HK76].IteratorsareusedtoextendexpressionswithboundedquantiÞers(Example8).LikeCÕsloops,aniteratorhasaquantiÞedvariable,acondition,andanexpressionthatcomputesthenextvalueinthecollection.Example8speciÞcationoffunction([Ros95])int*sort(x,size)int*x;intsize;assumex&&size�0;returnSwhereS&&all(inti=0;iinsize-1;i=i+1)S[i]S[i+1]&&all(inti=0;iinsize;i=i+1)some(intj=0;jinsize;j=j+1)x[i]==S[j];Beforeexecution(clause),thearray()mustnotbenullandmustbepositive.Theexecution(clause)shouldproduceanarray:(1)notbeempty;(2)allitselementsshouldbeordered(i.e.,.,S[i+1]);(3)eachelementofshouldcorrespondtoanelementof,i.e.,shouldbeapermutationof.Noticethattheclauseusestheoperatortocomputethevalueofbeforeexecution.EvaluatingquantiÞersasprogramloopscansigniÞcantlyimpactexecutiontimewhencollectionsarelargeorquantiÞersarenested.Notethatthepostconditionas-sertionforinvolvesnestedquantiÞerstocheckthepermutationcondition.Thischeckrequiresanumberofcomparisonsquadraticinthesizeofthearray.Evenso,thepermutationcheckaswrittenissufÞcientonlyifthearraycontainsnoduplicateelements;theoutputarrayisrequiredonlytocontainatleastonerepresentativeofeachvalueintheinputarray.WhileonemightÞndamoreefÞcientcheckforthepermu-tationcondition,ingeneralitcanbedifÞculttodeviseassertionsthatarebothclearstatementsofintent(usefulasspeciÞcations)andalsoefÞcienttoevaluate.3.2.2Nana[Mak98]isalibraryforassertioncheckingandloggingfortheGNUC/C++environment.borrowsconceptsandideasfromotherprojects(Anna,and ),existingprogramminglanguages(Eiffel),andformalmethods(ZandVDM)todeliveranefÞcientsolutiontoprogrammingwithassertions.Quotingtheauthor,Òanicelittlelibraryimplementingsomeoldideasinahopefullyusefulform.Óhopeistopushtheseconceptstocommongoodpractice.Assertionscanrefertobothbeforeandafterstates.Universal()andexistential)quantiÞersÐtogetherwithothermacros(e.g.,)ÐaresupportedasdeÞnedinthe(Example9).alsosupportsC++iteratorsasprovidedbytheStandardTemplateLibrary[MS96].Example9speciÞcationoffunction([Mak98])voidqsort(intv[],intn){I(v!=NULL&&n�=0);L("qsort(%p,%d)\n",v,n);/*thesortingcode*/I(A(inti=1,in,i++,v[i-1]v[i]));sortstheelementsinin.Beforesorting,theÞrstchecksthevalidityofinputparametersandtheclauselogsthecurrentvaluesoftoacircularbuffer.Aftersorting,thesecondclauseveriÞesthatallelementselementsaresorted.Usingmoretraditionallogicnotation,thelastassertionwouldbe:be:i1]v[i].2AssertioncheckingcanbeprogrammedeitherusingsimpleCcodeorexploitingthedebuggingfacilitiesprovidedby(Example10).Auxiliaryvariablesandvalue-cachingforpostconditionscanbeimplementedusingconveniencevariables,dynami-callytypedglobalvariablesprovidedbythedebugger.Example10TwospeciÞcationsoffunction[Mak98]boolisempty(){DS($s=s);/*codetodotheoperation*/DI($s==s);a)Usingconveniencevariables,beforeexecution,thevalueofiscopiedintheconve-niencevariable.Afterexecution,thecurrentvaluesof,i.e.,thevalueofbeforeexecution,arecompared.boolisempty(){ID(intolds);IS(olds=s);/*codetodotheoperation*/I(olds==s);b)UsingpureC,beforeexecution,thevalueofiscopiedinthedummyvariableAfterexecution,thecurrentvaluesof,i.e.,thevalueofbeforeexecution,arecompared. 3.3EiffelEiffel[Mey97,Mey92]isthemostwell-knownprogramminglanguageinwhichas-sertionsareabuilt-inlanguagefeature.InthisreportwelimitourattentiontothesupportEiffelprovidestoprogrammingwithassertions,touchingonotherfeaturesofEiffelasanobject-orientedprogramminglanguageonlyinsofarastheyinteractwithassertionfeatures.Eiffelencouragesadesignbycontractmethodology,interpretingrelationsamongroutines(methods)ofasystemascontractsbetweenclients(callers)andsuppliers(routines).Assertionsprovideawayofpreciselystatingandcheckingthecontractsthatgoverncooperationamongclassesinasystem.Eiffelassertionscanbeusedasprogramdocumentation,encouragingtheiruseasaprimaryformofspeciÞcation.Classinterfaces,calledshortforms,canautomaticallybegeneratedbyremovingallnon-exportedfeaturesandimplementationdetailsfromthesourcecode.Shortformscontainonlypublicpropertiesandassertionsandareavalidmeanstounderstandaprogramwithoutreadingthewholecode.AfterdeÞningassertions,actualimplementationscanbeenclosedinclausesorelsepostponedusingkeyword.Inthelattercase,Eiffelbecomesa(low-level)speciÞ-cationlanguagewithwhichusersdeÞnethesemanticsoftheiroperations,omittingimplementationdetails.Eiffelassertionsservenaturallyastestoracles,andcanbeconsideredtobeoraclesderiveddirectlyfrominterfacespeciÞcations,totheextentthattheexpressivenessoftheassertionlanguageissufÞcienttocaptureinterfacespeciÞcations.Aswithotherassertionssystems,though,assertionsinEiffelreßecttrade-offsbetweenexpressive-nessandcost.Eiffelassertions(Example11)mustbebooleanexpressions.Toavoidperformanceproblems,Eiffelassertionscannotincludesets,sequences,orquantiÞers.Eiffelassertionscanrefertoclasses,singleroutines,andindividualstatements.Theinvariantkeywordintroducesassertionsthateachclassinstancemustalwayssat-isfy,i.e.,aconditionthatshouldbeestablishedbycreationofanobjectinstanceandmaintainedbyeachoperation.Thekeywordintroducespreconditions,i.e.,requirementsthatthecallers(clients)ofaroutinemustsatisfy.Similarly,keywordidentiÞespostconditions,theconditionsthattheroutine(provider)guarantiesonreturn.Keywords,andcanbeusedtostateassertionsonparticularpointsinexecution.AnassertionthatmustholdatagivenpointinthecodeisdeÞnedinaclause;theconditionthatmustbesatisÞedaslongasaloopisexecuted,andtheconditionforloopterminationaredeÞnedininvariantclauses,respectively.Example11AnEiffelclass([Eif])classACCOUNTbalance:INTEGER;withdraw(sum:INTEGER)issum�=0sumbalance balance=oldbalance-sumend--endWITHDRAWbalance�=0hasoneintegerattribute,,andoneroutine,withdrawTheclassinvariantimposesthatmustalwaysbenon-negativeforeachin-stanceoftheclass.Thepreconditionoftheroutinerequiresthatthetobewith-drawnbebetween,inclusive.ThepostconditionensuresthatthenewvalueofisitsoldvaluelessthewithdrawnSinceEiffelisanobject-orientedlanguage,itsassertionsub-languagemustberec-onciledwithinheritance,polymorphism,andlatebinding.Thebasicproblemistoensurethat,whereveranobjectofsubclassBcanbesubstitutedforanobjectofclassA,subclassBhonorsthecontractofclassA.TheinvariantofanEiffelsubclassistheconjunctionofitslocalinvariantwithalltheinvariantsofitssuperclasses.PreandpostconditionsmustalwaysberedeÞnedinawaythatensuressubclassesaresubstitutablefortheirrespectivesuperclasses.Ifisaroutine(method)ofclassitsredeÞnitionin,subclassofonlybeequaltoorweakerthan;dually,canonlybeequaltoorstronger.Thus,whenatrun-timeacalltobecomesacallto,thepreconditionisÒweakenoughÓtobesatisÞedbyanycallerthatsatisÞesthepreconditionof,andthepostconditionofisÒstrongenoughÓtosatisfyanycallerthatreliesonthepostconditiontoAssertionsareintegratedwiththeexceptionhandlingmechanismsofEiffel.Ifamonitoredassertionisviolatedatrun-time,itraisesanexceptionthatcouldstopexe-cutionortriggerarecoveryaction.Whenassertionsareusedastestoracles,usuallytheonlydesirableÒrecoveryÓisproductionofadiagnosticmessagebeforehaltingex-ecution,butthesamefacilitycanbeusedinaproductionversionofthesoftwaretoestablishastablestateafteranunanticipatedevent.InadditiontodisallowingquantiÞersinassertionstolimittheexpenseofrun-timechecking,Eiffelprovidestheprogrammercontroloverwhichassertionswillbemoni-toredforeachclass.Theprogrammermayspecifyatcompiletimeno-check,precon-ditionsonly,postconditionsonly,preandpostconditions,oreverything.Aswithotherassertionsystems,thisprovidesthepossibilityofusingassertionsastestoraclesanddebuggingaidswithoutincurringthefullexpenseofmonitoringindeliveredsystems.3.4JavaAssertionSystemsThewidediffusionofJavahasmotivatedseveraleffortstoprovideassertionfacilitiesfortheJavalanguage.Someaspectsofthesesystemsaddressidiosyncrasiesofthelanguage,butmanyaddressgeneralproblemsinadaptingassertionsystemstomod-ernobject-orientedlanguagesandprogrammingstyles.TheyrangefromsimpleJava packagesofferinganmethodtore-implementationsoftheEiffelassertionfa-cilities,andfrompre-processorstosystemsthatexploitlow-levelcommunicationwiththeJavavirtualmachinetoinsertassertionson-the-ßyduringexecution.Inadditiontofeaturesanddesignstrategy,thesesystemsalsovarygreatlyinmaturitylevel:somearerobust,usableproducts,whileothersareresearchprototypes(someonlypartiallyimplemented).SeveraldifferentsystemsaresummarizedinTable1.Themainvariationscanbesummarizedasfollows:NearlyallJavaassertionsystemsarebasedonpre-processorstotransformanno-tatedcodeintoJavasourcecode.Onlytwoarenot:Handshakeuseslow-levelservicestoaddon-the-ßyassertionstocodejustbeforebeingexecutedbythevirtualmachine,andtheJavaSpeciÞcationRequest(JSR)#41isaproposedex-tensiontotheJavalanguageitself.Manysystemsembedassertionsasspecial-purposecomments,whichthepre-processortransformsintoJavacode.jContractor,aloneamongthepre-processorimplementationsofassertionprocessors,interpretsspeciallynamedmethodsaspre-andpostconditionassertions.AfewprovidealibrarypackagewithmethodswhichcanbecalledlikeanormalJavamethod.AllthelistedassertionfacilitiessupportthedeÞnitionofpre-andpostconditionchecksandinvariants,atleastindirectly,butonlyafewalsosupportassertionchecksatarbitrarypointsincontrolßow(includingloopinvariants).Systemsthatprovideanmethodoftendonothavespecialsupportforpreandpostconditionsorobjectinvariants,insteadrequiringtheusertoplacecallstoatthepointwherethepre-orpostconditionshouldbeevaluated.ItisnaturaltorefertoÒpreviousÓvaluesinpostconditions,andalsotorefertothevaluereturnedbyafunction(whichisanonymousinJava),butonlyafewassertionfacilitiesprovideawaytodirectlyrefertothem.OnlyafewsystemssupportuniversalandexistentialquantiÞersdirectly.Inmanycases,theprogrammermustsimulatetheeffectofquantiÞersusingloops.AmongcurrentJavaassertionssystems,providesmostoftheimportantfeaturesintheirmosttypicalform(asspecialcommentstransformedbyapre-processorintoexecutableJava),anditisalsoamongthemostwidelyknownandusedsystems.Ratherthandescribingeachassertionsystemindividually,therefore,wepresentasarepresentativeofthewholefamilyofJavaassertionfacilities.3.4.1iContractprovidesanEiffel-likeassertionfacilityforJava.AsinAnna,assertionsareembeddedinsourcecodecomments,whicharetransformedbyapre-processorintoexecutableJavacode.Example12illustratesthegeneralformofpre-andpostconditions.ThekeywordspeciÞesclassandinterfaceinvariantsand Hereafter,classesandinterfacesarecollectivelyreferredtoas Table1:Javaassertionsystems Approach invasive pre-processor comments package pre/poststyle check quantiÞers oldvalue returnedvalue iContract([Kra98])    -  -    JaWA/Jass([jas])    -      Handshake([DH98]) - - - -  - - - - jContractor([KHB98])   - -  - -  - JMSAssert([JMS])    -  - -   JPP(IDebug)([KC98])    -  - - -  JML([Bho00])    -    - - corejavaAssert([ass])  - -    - - - JUnit([jUn]) - - -  ? ? - - - JSR([jsr]) L - - - ? ? - - - :directlysupported;:indirectlysupported;L:languageextensionThemeaningofthecolumnsisasfollows:Invasive:Wecategorizeasystemasinvasiveifassertionsareembeddedinthesourceprogram(e.g.,ascommentsorasmethodcalls).Assertionsarenon-invasiveiftheyareexternal,notpartoftheprogramtext.Pre-processor:Theassertionsystemisimplementedasapre-processorthattransformsanextendedprogramintoapureprogram.Assertionsareencodedinspecialcomments.Package:Assertionsareprovidedthroughalibrarypackageorclasseswithmethods.Pre-andpostcondition,invariant:Supportformethodpre-andpostconditions,andforclassinvariants.DirectsupportmeansthatspeciÞcationscanbeplacedtogetherasakindofexplicitÒcontractÓforaclassormodule.Indirectsupportmeansthattheprogrammermustplaceassertionswithinthecontrolßowofmethods,atthepointwheretheyshouldbeevaluated.Simpleassertionstatements,placedbytheprogrammerinthenormalcontrolßowofamethod.Assertionsmayquantifyuniversallyorexistentiallyoverelementsofacollection.Oldvalue:Post-conditionassertionscanrefertothevaluesofvariablesbeforeamethodcall(implyingthattheassertionsystemsavesacopyofrelevantvalues)aswellascurrentvariablevalues.Returnedvalue:Theresultreturnedbyamethodcanbereferencedinapost-conditionassertion. specifypreandpostconditionsformethods.Afterthekeyword,program-merscanwriteanyJavaexpressionthatreturnsaboolean.extendsJavaoperatorswith,and.TheÞrsttwooperatorsaretheex-istentialanduniversalquantiÞersandcaniterateoverjava.util.Enumerationjava.util.Collection(Java2),andjava.util.Vectordirectly(withouthavingtoconverteverythingtoEnumerationÞrst).Itisalsopossibletoiterateoverarraysofobjecttypesandprimitivetypesandoverrangesofintegersandotherprimi-tivetypes.Theoperatorisshorthand:istransformedintoCthencheckIMultipleinvariantsforthesameclass,ormultiplepre-orpostconditionsforthesamemethod,areconjoinedtoformasingleaggregateexpression.Aftereachex-pression,programmerscanalsospecifytheclassthatistobeusedtoconstructtheexceptionthatisthrowniftheassertiondoesnothold.IfnoclassisdeÞned,classRunTimeExceptionisused.Example12annotations([Kra98])*@prei�=0#ArrayIndexOutOfBoundsException*@preithis.SIZE#ArrayIndexOutOfBoundsExceptionStringgetEntry(inti)throwsArrayIndexOutOfBoundsException{//noneedtomanuallycheckindexboundsThepreconditionconstrainsthevalue()thatidentiÞesthepositionsofavailableen-triesinthearray.mustbegreaterthanorequalto,butitmustalsobelessthanthearraysize().Ifoneofthepreconditionsdoesnothold,classdexOutOfBoundsExceptionisusedtoconstructtheexception./**Appendanelementtotheargument*@postlist.size()==list.size()@pre+1;voidappend(Vectorlist,Objecto);Thesizeofthelistafterinsertionmustbeonemorethanthesizebeforeinsertion(thesavedvalueofwhichisavailableusingthekeyword/**Eachemployeemustbeintheemploymentlistofallhisemployers*@invariantemployees!=null*implies*forallEmployeeeinemployees.elements()|*existsEmployercine.getEmployers()|*c==thisclassEmployer{protectedVectoremployees;//ofEmployee TheinvariantofclassspeciÞesthateachemployeemustbepartofthelistofemployeesofallhisemployers.ThespeciÞcationusesquantiÞerstostatethatifthelistofemployees()isnotempty,thenforallemployeesinthelisttheremustexistanemployer,obtainedbycallingmethodgetEmployers,suchthatandtheÒownerÓofthelist()arethesameobject.Classinvariantscanaccessclassandinstancevariablesaswellasmethodsoftheirassociatedclasses.Interfaceinvariantshavealmostthesamescope,buttheycannotaccessinstancevariables.Preconditionscanaccessallpropertiesthatareinthescopeoftheirassociatedmethod.Besidesthis,postconditionscanrefertoapseudo-variable,whichidentiÞestheresultvalueofthemethod,andpre-invocationvalues,thatis,thevaluesthattheexpressionshadbeforeinvocation:appendedtoanyexpressionrepresentsthevalueofbeforeexecutingthemethod.Invariants,preandpostconditionshaveaccesstoboundvariablesofquantiÞedexpressions.Ifinvariant,preandpostconditionsrefertoinstancevariables,thevariablesmustnotbeprivateunlesstheclassisÞnal.managesallfourtypeextensionmechanisms(classandinterfaceexten-sion,interfaceimplementation,andinnerclasses)inthesameway.Iftypeextends,allinvariantsandconditionsdeÞnedinapplytoaswell(Example13).deÞnesÒlocalÓinvariantsandconditions,mergesinheritedandlocalconstraintsusingthefollowingrules:Theinvariantistheconjunctionofthelocalin-variantandallinheritedinvariants,becausesubtypesmustcomplytoallrestrictionsoftheirsupertypes.Postconditionsarelikewiseconjoined,becausereÞnedmethodsmustofferatleastthefunctionalityofinheritedmethods.Incontrast,thepreconditionisthedisjunctionofthelocalprecondition(ifany)andallinheritedpreconditions,becauseredeÞnedmethodsmustacceptatleasttheinputargumentsoftheinheritedmethod.Example13Samplepropagationofpreandpostconditionsbetweeninterface([Kra98])interfacePerson{*@postreturn�0intgetAge();*@preage�0voidsetAge(intage);classEmployimplementsPerson{protectedintage_;publicintgetAge(){returnage_; publicvoidsetAge(intage){age_=age;PreandpostconditionsdeÞnedontheinterfaceareimplicitlypropagatedtotheimplementationclassAtrun-time,requiresthatthepreconditionsassociatedwiththechosenconstructorholdbeforecreatinganobject.Bothclassinvariantsandconstructorpost-conditionsmustholdwhentheobjectexists.Iftheconstructorfailsandanexceptionisthrown,theclassinvariantsandpostconditionsdonothavetohold.Afterobjectcre-distinguishesbetweencallsonpublic,package,andprotectedmethodsandonprivatemethods.IntheÞrstcase,bothpreconditionsandinvariantsmusthold;inthesecondcase,invariantsarenotchecked.Toimproveßexibility,allowsprivatemethodstotemporarilyviolateclassinvariants.Itrequiresthatclassinvariantsholdbeforeexitingthepublicmethodtriggeredtheprivateones.Inbothcases,postconditionsarecheckedbeforeexitingthemethod.Iftheexecutionfailsduetoathrownexception,classinvariantsmuststillhold,butpostconditionsarenotenforced.doesnotrequireanyconstraintonobjectdestruction(i.e.,onmethod4ExtrinsicInterfaceContractsTheassertionlanguagesconsideredintheprevioussectionprovideawaytoembedsomecheckableinterfacespeciÞcations,andpossiblyotherchecking,withinthepro-gramtobetested.Inthissectionweconsidersystemswhichprovidecheckablespec-iÞcationsatasimilarlevelofdetail,andwhichcanbeusedastestoraclesinroughlysimilarcircumstances(forunitandsubsystemtesting,butnotforoverallsystemtest-ing)butwhichkeepspeciÞcationsseparatefromtheimplementation.AlthoughitcanbeconsideredatrivialvariationtokeepinterfacespeciÞcationsinaseparateÞleortoembedthemintosourcecode,theseparationistypicallycoupledwithmoresigniÞcantdifferences.Inparticular,extrinsicspeciÞcationsaretypicallywritteninnotationsthatarelesscloselytiedtothetargetprogramminglanguage,andcanevenbeprogramminglanguage-independent,andtheyaremorelikelytobetiedtoaparticularspeciÞcationlanguage.4.1ADL(AssertionDeÞnitionLanguage)[SH94]isalanguageframeworkdesignedfortestingsoftwarecomponents.Incontrasttoassertionlanguagesembeddedinpartic-ularprogramminglanguages,isameta-notation,i.e.,asetofgeneral-purposeconceptsthatcanberenderedintothesyntaxesofdifferentprogramminglanguages. Inthiscontext,JavaÕspackageandprotectedmethodsarealsoconsideredÒpublicÓforthepurposeofdeterminingwhenassertionsmustbechecked. speciÞcationsarenotinsertedintheprogramundertest,butareplacedinseparateunits.ThisÒnon-intrusiveÓapproachmakesitpossibletoassociateassertionswithpre-compiledcode,suchasexistinglibrariesoroperatingsystems.DevelopersmustdeÞnethebindingsbetweenthespeciÞcationsandthefunctionsintheprogram.Assertioncheckingfunctions,thatis,thetestoracles,arethengeneratedautomatically.Anassertioncheckingfunctionisawrapperaroundthefunctionundertest()that,be-sidescalling,evaluatestheassociatedspeciÞcationtodeterminewhetherexecutescorrectly.speciÞcationsdeÞnepost-conditionsontheircorrespondingprocedures(func-tions,methods,etc.).Theyarepartialinthesensethatdevelopersneedspecifyonlywhattheywanttotest;furtherdetailscanbeaddedtothespeciÞcationasinformalcomments.TheypredicateonlyonÒafterÓstates,thatis,thestatesreachedafterexe-cution.ÒBeforeÓstatescanbereferredtousingthecall-stateoperator()thatsuppliesthevaluesofvariablesatthetimetheprocedureiscalled(value-caching).AlthoughearlyversionsofdidnotsupportquantiÞers,currentimplementationsprovidebothuniversal()andexistential()quantiÞers.wasÞrstinstantiatedasfortheCprogramminglanguage[Mic93].Subsequently,ADL2[Pro]extendedtheoriginalproposaltocoverobject-orientednotationsandinterfacedescriptionlanguages.ViswandaandSankarpresentaprelimi-narydesignfor[VS96],laterexpandedbyObayashietal[OKMM98],whodescribethenewinstantiationsforC++,CORBAIDL,andJava,andhighlightinterest-ingpeculiaritiesofADL/Java.ThefollowingsectionsbrießydescribetheCandJavainstantiationsof4.1.1ADL/C[Mic93]tailorstheframeworkfortheClanguageinterfaces.ItallowsuserstodescribethebehaviorofClanguageinterfacesorinterfacesreadilycallablefromC,generatedocumentation,andautomaticallyderivetestimplementationsoftheinterfaces.speciÞcationconsistsofasetof(Example14).Eachmodulecancontaintypeconstituentsobjectconstituentsfunctionconstituents.Func-tionconstituentscontainsemanticdescriptions,whichareorganizedinbindings.BindingsdeÞneshortnamestorefertowholeexpressions.Assertionsde-Þnebooleanexpressionsthatmustbetrueattheendoftheexecution.Additionally,auxiliaryfunctionsdeÞneconceptsthatdonotbelongtotheprogramundertest,butareneededforthepurposeofspeciÞcation.Example14speciÞcationofmodulemodule)modulebank{interrno;intNEG_AMT,INS_FUND;typedefintacct_no; Thetermdoesnotrefertoobject-orientedtechnology.ItisusedinthesamesenseasinC. intbalance(inacct_noacct);intwithdraw(inacct_noacct,inintamt)semantics{exception:=(return==-1),normal:=!exception,negative_amount:=(errno==NEG_AMT),insufficient_funds:=(errno==INS_FUND),@(amt)&#x:000;negative_amount,@(amt&#x:000;balance(acct))&#x:000;insufficient_funds,exception&#x:000;--unchanged(balance(acct)),normally(balance(acct)==@balance(acct)-amt,return==balance(acct)ModuledeÞnesasimplesetofconstituentsforabankaccount: ,and manageerrorconditions.Type mapsaccountnumberstointegers.ThebehavioroffunctionisleftunspeciÞed.Itcanbeusedbyotherfunctions,butitsimplementationisnotautomaticallychecked.deÞnesfourbindingsandfourassertions.TheÞrsttwobindingsassociateexpressionstothespecialnamesexceptionandtheremainingtwobindingsdeÞnealiases amountandin-sufficient forparticularerrorconditions.Theexceptionoperatorisusedtospecifythat canbetrueafterexecutiononlyamt0istruebeforeexecution.Theseconduseofsimilarlyassociatesapreconditionwiththeinsufficient exception.ThethirdassertionusesthepredeÞnedfunctiontostatethatifthefunctionfails,i.e.,istrue,remainsunchanged.ThefourthassertionspeciÞesthebehaviorwhenistrue:Thefunctiondecrementsthebalanceofandreturnsthenewaccountbalance.AllconstituentnamesaredirectlyboundtoCelementswiththesamenames.4.1.2ADL/JavaADL/Javaextendstechnologytocopewithobject-orientedconceptslikeinher-itance,polymorphism,overloading,andlatebinding.ADL/JavaspeciÞcationsareor-ganizedinhierarchiesofadlclasses(Example15),withatmostoneforeachJavaclass.DevelopersmayprovidespeciÞcationsforasubsetoftheJavaclassesinaprogram.Thismeansthat:(1)graphsaresub-graphsofthecorrespondingJavaclasshierarchies;(2)AncanspecifymethodsthataredeÞnedinthecorrespondingJavaclass,butalsomethodsthatareoverriddeninorsimplyinheritedbytheJavaclass.RedeÞnedmethodscanbespeciÞedusingtheper.semanticsfeature,inwhichcasethedeÞnitionofthegivenmethodusesthepreviousdeÞnitionsofthesamemethodinallinheritedclass.Theapproachisrecur-sive:IfamethodhasalreadybeenspeciÞedtwiceinsuperclasses,then callingaredeÞnitionofinclass,subclassof,wouldevaluatetheassertionsof,thentheassertionsof,andthentheassertionsofExceptionsaredeÞnedthroughJava-likestatementsen-ableexceptioncatchingwhileevaluatingassertions;statementsdeÞnealternateassertiongroupstobeusedtoÒserveÓcaughtexceptions.Example15ADL/JavaspeciÞcationofclassclass)adlclassbank{BankAcctopen_acct(longamt)throwsnegAmtExc,bnkFullExc{semantics[abnormal=thrown(negAmtExc,bnkFullExc)]{amt0&#x:000;thrown(negAmtExc);@bankAux.bank_is_full(this)&#x:000;thrown(bnkFullExc);if(normal){return.get_balance()==amt;get_accts()==@get_accts()+1;bankAux.is_active(this,return.get_acct_num())==true;if(abnormal){unchanged(get_accts());Thesemanticdescriptionofmethod deÞnesonebindingandfourasser-Thebindingrelatesthespecialnameabnormaltotheoccurrenceofoneofthetwothrowableexceptions,i.e.,istrueifoneofthetwoexceptionsisthrown.TheÞrstassertionstatesthatif,thenthemethodfails(i.e.,throwsanexception).But,ifthemethodfailsandwasthrown,thenThesecondassertionstatesthatifthebankisfull,thenthemethodfails.Butifitfailsandwasthrown,thebankmustbefull.Thethirdassertionstatesthatnormalexecutionsrequirethat:(1)theamountofthenewlyopenedaccount,whichisidentiÞedbykeyword,beequalto;(2)thenumberofaccountsopenedatthebankbeincreasedbyone;(3)thenewaccountbeactive.Thefourthassertionstatesthatabnormalexecutionsshouldnotchangethenum-berofopenedaccounts.ADL/JavaassertionsallowalsoforinlinedeclarationsandauxiliarydeÞnitions.Inlinedeclarationsareordinarytextualmacros.AuxiliaryimperativedeÞnitionscanbeaddedusingclauses.TheseareblocksofpureJavacodethatdonotaffectthedeclarativestyleofassertions.Prologuesandepiloguescanbeglobal(belongtothecompilationunit)orlocal(belongtothesingleclass). 4.2TOGItispotentiallyadvantageoustouseanexistingspeciÞcationnotation,ratherthanin-ventinganewnotationjustforthepurposeofcreatingtestoracles.Ontheotherhand,derivingoraclesfromÒpureÓspeciÞcationlanguage(discussedbelowinSection5)ismademoredifÞcultbythegeneralityofnotationsthatwerenotdesignedforrun-timechecking.AmiddlegroundisoccupiedbyadaptationsofexistingspeciÞcationstylesandnotationstotheparticulartaskofproducinginterfacecontractsfromwhichtestoraclescanbeautomaticallyderived.WeconsiderÞrstanadaptationofSCR-styletab-ularspeciÞcations,andthentwoapproachestoadaptingalgebraicspeciÞcationstotestoraclegeneration.TOG(TestOracleGenerator)[PP98]generatesoraclesfromrelationalprogramdocumentationintheformoftabularexpressions[PMI94].Theprogram(function)undertestisspeciÞedbymeansofitssignature,theexternalvariablesituses,anditssemanticsintheformofaspeciÞcationrelationbetweeninitialandÞnalexecutionstates.AllthisinformationisgroupedinatablecalledprogramspeciÞcation(Exam-ple16).Thenotationanditssemanticsisbasedcloselyonthetabularnotationsdevel-opedforprogramspeciÞcation,andparticularlyforcontrolsystems,butisadaptedtospecifyingtheconcreteinterfaceofproceduresinaparticularprogramminglanguage.Somedetailsofthisnotationaredescribedbelow.Example16AtabularspeciÞcationoffunctionfunction)ProgramSpeciÞcation voidfind(intB[N],intx,int*j,bool*present) externalvariables: D true Rnd= (9i;i)^ (8i;bRi)) 8B[ix) :(8B[ix 0j 8B[j0 0= FALSE 8B;8x;B0;x0) AuxiliaryPredicateDeÞnitionsons;int8b;inta0[];intb0):=(8i;bRange(i))8a[i]=a0[i]^(8b=b0)InductivelyDeÞnedPredicatesUserDeÞnitions#include"defs.h"#defineN10/*sizeofarraytosearch*/ havetheusualmeaningsofthevaluebeforeandafterexecution,respectively. Thesignatureoffunctionisstraightforward:isthearraytosearchin,thevaluetobesearched,identiÞesthepositionof,andstatesifisin.Thefunctiondoesnotuseanyexternalvariables.ItsspeciÞcationrelationdoesnotimposeanyrestrictionontheinputdomainorontheÒcompetencesetÓportionofthedomainforwhichshouldterminate),sothesearerepresentedbythecharacteristicpredicatet isthecharacteristicpredicateofthesetofacceptableexecutionsummaries,pairswhereisthestartingstateandisacorrespondingstoppingstate.denotethevalueofavariableintheinitialandÞnalstate,respectively.Theallowedbehaviorisbrokenintotwocases,dependingonwhetheranelementequaltoappearsinthetable,andthesetwocasesaredescribedintwocolumnsofthetabularexpression.Thetabularexpressionisconjoinedwiththeauxiliarypredicate,whichprohibitschangestoothervariables.InductivelydeÞnedpredicateslikeformallydenotetheÞxedpointofasetequation,butcanbeunderstoodoperationallyasaniterator,e.g.,aClanguageloopoftheformfor(i=;i=SpeciÞcationrelationsarelimiteddomainrelations(LD-relations)describedbytheirdomain,competenceset,andcharacteristicpredicate.LD-relationsarebasedonalogicthatallowspartialfunctions,butensuresthatpredicatesaretotal[Par93].Primitiverelationsareifoneormoreoftheirargumenttermsisafunctionappli-cationwithargumentvaluesoutsidethefunctionÕsdomain.ThisensuresthatalwayshasaclearlydeÞnedvalue,either,regardlessofthevaluesoftheirargu-ments.But,thismeansalsothatthelogicsdoesnotincludetheaxiom,because,onlyifisdeÞnedat.ThedevelopersoftabularspeciÞcationsclaimthatthisleadstoclearerspeciÞcations.AccordingtothedeÞnitionsgivensofar,theexecutionsummaryofaterminatingprogrammustbeinshouldterminateforallstartingstates,butnotterminatefor.Moreover,ifdoesnotbelongto,theexecutionsummaryshouldnotbein.Tocheckthesestatements,TOGautomaticallygeneratestestoraclesasfourCfunctions:initializestheoracle,evaluatesonsummaryexecutions,inCompSetinDomainevaluate,respectively,onstartingstates.FunctioninRelationistheactualora-cle;thelasttwofunctionscanbeusedtoavoidcheckingnon-terminatingexecutions)andexecutionswithoutacceptableresults(SpeciÞcationrelationsdescribeactualprograminterfacesanddatastructures,whichraisessomeofthesamedifÞcultiesasinotherassertionlanguages.CachingofÒbe-foreÓvalues(inspeciÞcationexpressions)posestheusualproblemswhendealingwithpointersorcomplexdatastructures,andinterpretationofinductivepredicatesasloops(e.g.,evaluatingthepredicateintheexample)canbeunreasonablyex-pensive.ProblemsmorespeciÞctotheapproachofadaptingaspeciÞcationlanguagefortestoraclegenerationarisefromtheinabilitytoexpressimplementationdetails,suchasÒavalidblockofmemoryallocatedintheheap.ÓProcedureswithformalfunc-tionparametersarenotconsideredasÒprograms,Óbecausetheydonotdetermineasetofpossibleexecutions. Thesignaturein[PP98]isslightlydifferentduetoatypographicalerror. 4.3AlgebraicSpeciÞcationsTheinterfacespeciÞcationsconsideredsofarareintheformofassertionsthatcanbeevaluatedataparticularpointinexecution,orprecondition/postcondition.WhilebalancingexpressivepowerwithefÞcientrun-timeevaluationposesachallenge,theideaofusingthesespeciÞcationsastestoraclesbyevaluatingthematrun-timeisatleastfairlysimpleinprinciple.NotallformsofinterfacespeciÞcationcanbeadaptedinthissimplemanner.Inparticular,thealgebraicapproachtospecifyingthebehaviorofabstractdatatypespresentsachallenge,becausetheconditionsthatshouldbetrueaftersomeparticularoperationarenotgivendirectly,butratherbyequatingtheresultsofdifferentsequencesofoperations.Wenextconsidertwoapproachestousingderivingtestoraclesfromalgebraicab-stractdatatypespeciÞcations.IntheDAISTSapproach,axiomsequatingthedifferentsequencesofoperationsareuseddirectly,andtheymustbeusedtoderivetestcasesandthetestoracletogether(theyarenotfullygeneraloraclesforjudgingthecorrectnessofarbitrarytestexecutions).Inthesecond,morerecentapproach,adirectimplemen-tationofthealgebraicdeÞnitionofthedatatype(interpretingequationsoperationallyasrewriterules)isusedasanalternativeimplementation,whosebehaviorcanbecom-paredtothebehaviorofamoreconventionalimplementationundertest.4.3.1DAISTSDAISTS(Data-Abstraction,Implementation,SpeciÞcation,andTestingSystem)[GMH81]isanintegratedframeworkforimplementing,specifyingandtestingabstractdatatypes(ADTs)implementedintheobject-basedlanguageSIMPL-D,amemberoftheSIMPL[Bas76]familyoflanguages.TheSIMPL-Dimplementationisaugmentedwithasetofalgebraicaxiomsdescribingthetype,andasetoftestcasestovalidateit.TheimplementationisaSIMPL-Ddeclaration(Example17).ThedeÞnesasetofvariabledeclarations(i.e.,thetyperepresentation)andasetoffunctions(i.e.,thebody).Example17AnexcerptofaSIMPL-DdeÞnitionofthedatatype[GMH81]defineEltType=‘int’defineUndefined=‘0’classStack=Push,Pop,Top,Empty,NewStack,StackEqual,Depth,Limit,assigndefineStackSize=‘20’uniqueEltTypearrayValues(StackSize)uniqueintStackTopStackfuncNewStackStackResultResult.stackTop:=-1 WecallSIMPL-Dobject-basedratherthanobject-orientedbecause,althoughitdoesprovideencapsu-lateddatastructuresimilartoÒclassesÓinanobject-orientedlanguage,itdoesnotprovidesomeofthetypicalfacilitiesofamodernobject-orientedlanguage,particularlyinheritance. return(Result)BoolfuncEmpty(StackS)return(S.StackTop=-1)StackfuncPush(StackS,EltTypeElt)StackResultifS.StackTop+1=StackSizereturn(S)Result:=SResult.StackTop:=Result.StackTop+1Result.Values(Result.StackTop):=Eltreturn(Result)StackfuncPop(StackS)StackResultifEmpty(S)return(NewStack)Result:=SResult.StackTop:=Result.StackTop-1return(Result)endclassDAISTSaxiomcannotbeuseddirectlytocheckthecorrectnessofanarbitraryoperation(methodcall).However,anaxiomcanbeusedtoderiveatemplateofatestcasetogetherwithanoraclespeciÞctothatcase.EachaxiomofthespeciÞcation(Example18)isnamedandhasalistofnamesandtypesoftheusedfreevariables.DAISTSconvertsaxiomsintocodecallingontheimplementation:Thefreevariablesbecomeparameters,andtheoperationsbecomefunctioncalls.Example18Someaxiomsforelements([GMH81])Empty(NewStack)�True;onanewstackmustnotreturntrue.Empty2(StackS,EltTypeI):Empty(Push(S,I))=False;onastack,aftera,mustreturnfalse.Top(NewStack)=Undefined;onanewstackmustreturnundeÞned.Top2(StackS,EltTypeI):Top(Push(S,I))=ifDepth(S)=StackSizethenTop(S)elseI; onastack,aftera,mustreturneither,ifwasfull,orthepushed,otherwise.Pop(NewStack)=NewStack;onanewstackmustreturnanewstack.Pop2(StackS,EltTypeI):Pop(Push(S,I))=ifDepth(S)=StackSizethenPop(S)elseS;onastack,aftera,mustreturneither,ifwasfull,or,other-wise.Atrun-time,aDAISTSspeciÞcationactsasadriverprogramthatexecutesasetofThedriverapplieseachaxiomwith(separatelyprovided)testdata,andcom-parestheexecutionoftheleftandrightsidesoftheaxioms.Ifthetwoexecutionsdonotreturnvaluesthatareconsideredtobeequal(accordingtoaADT-speciÞcequalityfunction),adiagnosticmessageisprintedindicatingthattheaxiomfailed.DAISTSwasapioneeringandwell-regardeddemonstrationthattestoraclescouldbederivedfromformalspeciÞcationsevenwhentheyarenotalreadyinaprecondi-tion/postconditionform.However,theapproachhastwoimportantlimitations.First,sincetheaxiomaticspeciÞcationdirectlydescribesimplementationentitiesandop-erations,theimplementationsmustuseafunctionalstyle,ormoretothepoint,theapproachisnotapplicabletotypicalimplementationsofdatastructuresinproceduralandobject-orientedlanguages.Second,theapproachrequiresgenerationoftestcasesalongwithoracles,sinceeachoracleisspeciÞctotestcasesderivedfromaparticularaxiom.Itcannotbeusedtoderivegeneraloraclesforarbitrarytestcases.4.3.2Self-checkingADTsAmorerecentapproachtoextractingtestoraclesfromabstractdatatypes,describedbyAntoyandHamlet[AH00],treatsthealgebraicspeciÞcationessentiallyasanalter-nativeimplementationwhosebehaviorcanbecomparedtotheconventionalÒby-handÓimplementation.IncontrasttoDAISTS,thisapproachdoesnotextracttestcasesalongwiththetestoracles,butthederivedoraclesaregeneralinthesensethattheycanbeusedwithtestcasesobtainedbysomeothermethod.TheADTisspeciÞedthroughasetofoperationsandasetofaxioms(Example19).ThespeciÞcationisconvertedintoC++codeandbecomestheso-calleddirectim-plementationoftheADT(incontrasttoaconventionalÒby-handÓimplementation).ConstructorsbecomefunctionsthatallocatememoryfortherepresentationoftheADTandreturnapointertoit;axiomsaretransformedintofunctionsthatrepresenttheinstancesofADTsastermsandmanipulatethesetermsaccordingtorewriterules. DAISTSalsoprovidessomesummaryandcoverageinformationthatisnotdiscussedhere,sinceitisnotdirectlyrelevanttotestoracles;interestedreaderscanÞndthesedetailsintheprimarydescriptionofDAISTS[GMH81].ThetermÒconstructorÓasusedbyAntoyandHamletincludeswhatsomeauthorshavecalledÒmutatorsÓ(functionswhoseargumentsmayincludetheADTbeingdeÞned). Therewritingsystemensuresconßuenceandterminationtomaketherewritingpro-cesssimplerandmoreefÞcientthanitwouldbeotherwise,thoughpresumablystilllessefÞcientthanaconventionalimplementationoftheADT.Example19PartsofanalgebraicspeciÞcationfortheself-checkingADT([AH00]).empty(integer,integer)insert(integer,intset)createsanemptysetbyrequiringitssizeandtheupperboundforitselements.insertsanewintheempty(M,R)�-?:-MrRmustabort()if()eitherthesizeislessthanoneortheupperboundislessthanthesize.ThelastconstraintcomesfromthedeÞnitionof.Sincesetsdonotallowfornegativeandrepeatedelements,theycannothaveupperboundsthatarelessthantheirsizes.insert(E,empty(_,R))&#xM000;-?:-E&#xM000;Rmustabortiftheelementisgreaterthantheupperboundofthesetcreated,nomatterofthesetÕssize.Symbol identiÞesanonymousvariables.insert(E,insert(F,S))�-insert(F,S):-member(E,insert(F,S))Addinganelementtoasetdoesnotmodifythesetifalreadybelongstotheset.Theaxiomsforoperationareomittedherebutcanbefoundin[AH00].insert(E,insert(F,S))�-?:-notmember(E,insert(F,S))andcardinality(insert(F,S))�=maxsize(insert(F,S))mustabortwhentryingtoaddanewelementdoesnotbelongtotheset,butthesize()ofthesetisgreaterthanorequaltoitsmaximumsize).Themeaningsofinsert(E,insert(F,S))�-insert(F,insert(E,S)):-notmember(E,insert(F,S))andcardinality(insert(F,S))maxsize(insert(F,S))andE&#x-602;&#x.300;FInsertinganelementinaset,towhichelementhadbeenaddedpreviously,isthesameasinsertingintheset,towhichhadbeenaddedbefore.Thisistrueifdoesnotbelongtotheset,thecardinalityofthesetislessthanthemaximumsize,andisgreaterthan.Thelastconstraintmustholdtoensurethattherewritingsystemisconßuent.User-deÞnedimplementationsarecalledby-handimplementations.Adirectim-plementationandaby-handimplementation,togetherwitharepresentationmappingdeÞnetheself-checkingimplementation(Example20).Therepresentationmapping Forexample,aby-handimplementationofcanbefoundinastandardreferenceonC++[Str86],Section5.3.2. isakeycomponentoftheapproach:ItdeÞnesthecorrespondencesbetweenthedatastructuresoftheby-handimplementationandtheabstractionsofthealgebraicspeciÞ-Atrun-time,invocationofamethodoftheby-handimplementationbecomesanin-vocationofthemethodwiththesamenameoftheself-checkingimplementation.Boththeoriginalcodeandthecorrespondingrewriterulesareexecuted.Therepresentationmappingtransformsresultsontheby-handimplementationintotheirabstractrepresen-tations.Theself-checkingcodecomparestheseabstractresultswiththenormalformcomputedbytherewritingsystemandtreatsanydiscrepancyasananomaly.Example20Excerptsoftheself-checkingimplementationofclassclass)ClassdeÞnitionclassintset{abssetabstract;abssetconcr2abstr();//By-handimplementationintcursize,maxsize;int*x;intset(intm,intn);Theself-checkingimplementationofclassistheby-handimplementationwithtwoadditionalprivateentities:identiÞesthedirect-implementationandistherepresentationmapping.Theby-handimplementationdeÞnestwointegerattributestostorethecurrentandmaximumsizes,apointertoanintegerforthedatastructure,andasetofpublicmethods.Inthisexcerptweshowonlyaconstructorthatrequirestwointegers:,themaximumsize,and,theupperbound.MethoddeÞnitionintset::intset(intm,intn){if(m||n)error("illegalintsetsize")cursize=0;maxsize=m;x=newint[maxsize];//Additionalstatementsforself-checkingabstract=empty(m,n);Theself-checkingimplementationoftheconstructor,whichcorrespondtoconstructoroftheaxiomaticspeciÞcation,istheby-handimplementationwithtwoaddi-tionalstatements.TheÞrststatementcomputestheabstractresultbycallingfunctionofthedirect-implementation.Thesecondstatementchecksformutualconsis-tency.NoticethatisapredeÞnedmacrothatcompareswiththeresultfrommethodconcr2abstrand,ifneeded,printsdiagnosticmessages.Representationmappingabssetintset::concr2abstr(){absseth=empty(maxsize,MAXINT);for(inti=0;icursize;i++) h=::insert(x[i],h);Therepresentationmappingisstraightforward.Itcreatesanemptyabstractset,byofthedirect-implementation,andinsertsallconcreteelements(().2Whiletheself-checkingapproachhasbeendemonstratedwithC++andJava,itaddressesthedataabstractionfacilitiesofthoselanguagesandnototherfeaturesofobject-orientedlanguages,particularlyinheritanceorpolymorphism.Possiblyap-proachesdevelopedforassertionlanguagessuchascouldbeadoptedstraight-forwardly,butthishasnotbeenexplored.Perhapsalargerquestionistheextenttowhichproducinganexplicitrepresentationmappingcreatesanadditionalburdenfortheprogrammer,inadditiontotheburdenofcreatingalgebraicspeciÞcations.5PureSpeciÞcationLanguagesThetestoraclesconsideredinprevioussectionshavebeenexpressedinspeciÞcationlanguagesdesignedforrun-timechecking.TheyareinthissensenotÒpureÓspeciÞ-cations,i.e.,notspeciÞcationsthatonemightwriteÞrsttocommunicatetheintendedbehaviorofacomponentorsystemandonlyafterwarduseasthesourceoftestora-ThemainadditionalchallengeposedbyusingaspeciÞcationlanguageisthateffectiveproceduresforevaluatingthepredicatesorcarryingoutthecomputationstheydescribearenotgenerallyaconcerninthedesignoftheselanguages.Thischallengemaybeovercomethroughsomecombinationofrestrictingtheexpressionsthatmaybetranslatedtotestoracles,transformingsomeexpressionsintoother,equivalentexpres-sionsthataremoresuitableforrun-timeevaluation,andrequiringtheusertohand-codepartsofthecomputationthatcannotbederivedautomatically.ThesystemsforderivingtestoraclesfromZspeciÞcations,discussednext,useallofthesetactics.5.1ZandObject-ZZ[Spi89]anditsobject-orientedvariantObject-Z[CDD89]aremodel-basedspeciÞ-cationlanguagesthatdescribeintendedbehaviorusingfamiliarmathematicalobjects:sets,bags,functions,integers,etc.BeingÒpureÓspeciÞcationlanguages,theyarefreeoftheconstraintsoflanguagesdesignedforefÞcientcomputationalinterpretation.OnecandescribeorquantifyoverinÞnitesetsaseasilyasoverÞnitesets,onemaynegatearbitrarypredicates,andonemay(andconventionallydoes)leavemanyimplementa-tiondetails,includingconcretedatastructures,unspeciÞed.ThusitisnotimmediatelyobvioushowaZorObject-ZspeciÞcationcanbeinterpretedasatestoracle. OnecouldarguethatthetabularSCR-stylespeciÞcationofTOG(Section4.2)areÒpureÓspeciÞcationsinthissense.Wehavegroupedtheminsteadwithassertionsystemsbecausetheyreferdirectlytoprogramentitiesinthesyntaxoftheimplementationlanguage,butthereisnoclearlinebetweenderivingoraclesfromaÒpureÓspeciÞcationlanguageandadaptingaspeciÞcationlanguagetoserveasalanguagefordeÞningtestoracles. TheapproachesproposedbyMikk([Mik95])andMcDonaldetal.([MMS97,MS98])forderivingoraclesfromZandObject-ZbeginbyconstrainingspeciÞcationstoanÒexecutableÓsubset,whichcanbesemi-automaticallytranslatedintoCorC++.ExecutabilityrequiresthatdeÞnedtypesbeÞnite,allpredicatesbeevaluableusingaÞnitenumberofiterations,andtherangeofquantiÞedexpressionsbeÞniteortrans-formabletoaÞniteone.BoththeMikkÕsapproachandtheapproachofMcDonaldetal.startwithaso-optimizationphase,analyzingandtransformingtheoriginalZspeciÞcationsintotheexecutablesubset.Forexample,(wheredenotesthesetofnaturalnumbers)isnotintheexecutablesubset,sinceisaninÞniteset,butusingtherestrictionthequantiÞercanberestrictedtoaÞniterange.InMikkÕsapproach,asetofpredeÞnedrules(seeExample21)canbecomplementedwithhumanintervention.NonethelessitisnotalwayspossibletotranslatespeciÞcationsintotheexecutablesubset,andthereforenotallZorObject-ZspeciÞcationscanbeusedastestExample21ApredeÞnedrewriterule([Mik95]) NotOccur(y;P)]P(y)^y2MStraightforwardinterpretationofthetopformulainvolvesevaluatingforeveryele-mentofset.Therewriterule,whichisapplicableonlyifpredicatedoesnotcontainasfreevariable(),replacesthetopformulabyaconjunctionoftwotestsinvolvingMikkÕspredicatecompileralsoimposesfurtherrestrictions:Allactualparame-tersofagenericschemamustbeconstantexpressionstodetermineallitspossibleinstancesstatically.EachtransformableZschemabecomesaCfunction:Itsnamebe-comesthenameofthefunction,thedeclarationpartdeterminestheformalparameters,andthepredicatepartdeÞnesthebodyofthefunction.ImplementationsarebuiltbyreplacingZexpressionswiththeirvalues,propositionalcalculusformulaewithtruthtables,andquantiÞcationsandsetconstructionswithiterations.PartiallydeÞnedfunc-arecodedusingathree-valuedlogic(undeÞned)inastrictway(aswell).CodegenerationreliesonVDM-SLconstructsandVDMDC(VDMDomainCompiler,[SH91])formappingandconvertingtypes.Atruntime,eachfunctionactsasoraclebyevaluatingitspredicatesforstatespassedasparametersandreturnsabooleanvalueindicatingtheoutcomeoftheevaluation.McDonaldetal.adoptasimilarapproachandproposeseveralalternativetrans-formationofObject-ZspeciÞcationsintotestoraclesforcontainerclasses([MMS97,MS98]).AfteroptimizingthespeciÞcation,theyuseaspecial-purposeC++library,whichimplementssomeofthestandardtypesoftheZmathematicaltoolkit,totrans-formstandardZtypes(forexamplebecomesbecomes set&#xint2;.10; Agenericschemaisaschemaparameterizedwithrespecttoagiventype,likeaC++template,anAdageneric,orapolymorphictypeinML.DeÞnitiondomainsofpartialfunctionsmustbedecidabletoautomaticallygeneratethecorresponding whereZ isatemplateclassinstantiatedusingintegers).Theoracleclassandtheclassundertest(CUT)caneitherbeindependentclassesorrelatedthroughinheritance.Whentheyareindependent,thetestdriverappliestheoracletotheorigi-nalCUT.WhentheoracleclassextendsandinheritsfromtheCUT,itcanbeinvokedautomatically(asaÒwrapperÓ)tocheckresultsproducedbytheCUT.Example22showsanexcerptofanObject-ZspeciÞcationthatdescribestheremoveoperationforaclassIntSetandtheC++codegeneratedasoracle,usinginheritancetoÒwrapÓtheCUTwiththeoracle.EachaugmentedoperationprovidedbytheoracleclassevaluatestheobjectÕsstatebeforetherealoperation,performsitbycallingtheCUT,andthenevaluatesthepoststate.Bothpre-andpost-stateareevaluatedinthespeciÞcationdomainbymeansofauser-suppliedtranslationmethod(abstrac-tionfromconcretedomaintospeciÞcationdomain)andanothermethodcheckstheinvariantontheabstractstate.Theabstractstatesandoperationsareimple-mentedusingtheC++Zmathematicaltoolkitandareevaluatedusingsimplebooleanmacros.Example22ExcerptsfromthedeÞnitionofanclass([MS98]) IntSet (MAXSIZE;INIT;remove;removeAisMember; jMAXSIZE:N items: itemsMAXSIZE INIT items=? ... remove
(items) x x?62items!removeNotFoundExc items0=itemsnfx?g Theoracleisaclasswithamethodforeachoperation,alongwithaconstructor,whichcorrespondstotheschema,andtwoparticularmethods:.TheÞrstmethodmustbecodedbyhandanddeÞnesthetranslationfromconcretetoabstractvalues;thesecondmethodcodestheclassinvariant.classIntSetOracle{intMAXSIZE;IntSetOracle(constintmaxsize0);voidremove(constintx); voidabs(Z_Set&#xint0;state_items);voidinv(Z_Set&#xint0;state_items)const;voidcheck_remove(const&#xint0;Z_Set*pre_items,const&#xint0;Z_Set*post_items,constintx);voidIntSetOracle::inv(Z_Se&#xint0;tstate_items)const{CHECKVALBOOLEAN(1,state_items-&#xint0;size()MAXSIZE);voidInSetOracle::remove(constintx){&#xint0;Z_setpre_items=newZ_Set&#xint3;�.10;();abs(pre_items);inv(pre_items);IntSet::remove(x);&#xint0;Z_setpost_items=newZ_Set&#xin30;&#x.100;t();abs(post_items);inv(post_items);check_remove(pre_items,post_items,x);deletepre_items;deletepost_items;TheoracleimplementationoftheremovemethoddeÞnesasetofintegers,usesittostoretheabstractobjectÕsstate,andcheckstheinvariantagainstit.Then,itappliestheoriginalmethod,storesthepoststateinaspecial-purposesetandchecksitagainsttheinvariantoncemore.Attheend,itappliespre-andpost-stateandtheremovedvaluetotheoriginalspeciÞcationÐrenderedasaparticularmethodÐtoinvestigatetheircompliance.The methodsimplycallsamacrothattakestwobooleanvaluesandreturnsanerrormessageifthetwovaluesaredifferent.Inthiscase,thecomparisonisbetweentrue(inC++)andthefactthatthesetÕssizemustbelessthanorequaltothemaximumsize.Thus,themacrocheckswhetherthisholdstrue.5.1.1TestTemplatesStocksandCarringtonhavedescribedasomewhatdifferentapproachforderivingtestcasesandoraclesfromZspeciÞcations,consideringtestprogramspeciÞcationandtestcasespeciÞcationasasingleextendedtask.Theirapproach,calledtheTestTemplateFrameworkor,isconcernedprimarilywithtestcaseselection,whichisoutsidethescopeofthissurvey.Testoraclescanbeassociatedwithindividualtesttemplates(testcasespeciÞcations).Ahierarchicalarrangementoftesttemplatesallowstheoracletobeeithergeneral(nearthetopofthehierarchy)orspecializedtoparticulartestcases.TheapproachisillustratedwithtestoracletemplatesfortheZspeciÞcationinEx-ample23.Example23ExcerptsfromthedeÞnitionofablock-structuredsymboltable([SC96]) :SYM7!VAL 34 Thetableisrepresentedasapartialfunction()fromsymbolstovalues Update0
ST s v 0=stfs?7!v?g ThedeÞnitionofoperationsisdividedintobasicfunctionalityanderrorconditionsorsuccessmessages.deÞnesthebasicfunctionalityof.Itaddsanewmappingtothetable.Alloperationsonthesymboltabletaketheschemaasargu-representsastatechange.makesaddthenewmapping,ifisnotalreadyinthetable,orelsereplacethecurrentvalueassociatedwith Success repT rep simplyindicatessuccessfulcompletionofaanoperationbysettingthetoÒok.ÓThecompletedeÞnitionofisthentogetherwithwithst:SYM7!VAL;s?:SYM;v?:VAL]OSUpb=[st0:SYM7!VAL;rep!:REPORT]VISUpb=ISUpTheinputsetof)isthebeforestateof,andTheoutputsetistheÔafterÕvalueof,thatis,,andThevalidinputsetistheinputset.Anoracletemplateisadescriptionoftheexpectedoutputsgiventheinputsde-scrivedbythetesttemplate:TheoperationÕsinputisrestrictedtothatdeÞnedinthetesttemplateandthenprojectedontotheoutputspaceoftheoperation(Example24).OraclescandeÞnesetsofoutputswheneithertesttemplatesarenotsingle-instancetemplates,oroperationsarenon-deterministic.Example24Asampletestandoracletemplatesforoperation([SC96])Usingcharactersforthedatatype,andnaturalnumbersfor,wedeÞneatesttemplatetemplates?=ÔaÕ^v?=1] Themappingisasingle-instancetesttemplateand1)==(istheformaldeÞnitionoftheoracletemplateforoperationontesttemplateTheoracletemplateistheconjunctionofthetwoschemas,projectedovertheoutputspaceof.Thatis,1)==[Afterapplyingthesingletestcaserepresentedby,thesymboltableshouldcontainonlythemappingshouldbe5.2TemporalOraclesSpeciÞcationlanguagesareoftenspecializedtodescribingparticularaspectsofpro-grambehavior,andabstractingotheraspects(whichmaybedescribedinotherspeciÞ-cationlanguages).Inparticular,speciÞcationsbasedontemporalorintervallogicsareoftenusedtodescribeallowablesequencesofevents,whileelidingdetailsofprogramfunctionality.ThesearemostfamiliarfromstaticveriÞcationusingtemporallogicmodelcheckers[CES86,Hol97,CPG00],buttemporalspeciÞcationlanguageshavealsobeenproposedforprogramspeciÞcations.AnoracleforatemporalspeciÞcationlanguagejudgestheacceptabilityofa(partiallyorfullyordered)sequenceofevents.DillonandYupresentanapproachforderivingoraclesfromtemporallogicsusingatableaumethod[DR96,DY94].OraclesfortemporallogicformulaeareÞnite-stateacceptorsthatacceptsequencesofprogramstates,whereeachstateinthesequenceisrepresentedbyanassignmentofbooleanvaluestothepropositionalvariablesoftheformula.Transitionsintheautomatonarelabeledwith(non-temporal)logicalformulaethatcanbeevaluatedinindividualstates,sotheassignmentoftruthvaluesineachstatedetermineswhichtransitionistaken.WhenproducingaÞnite-stateautomaton,theoriginalspeciÞcationisassociatedwiththeinitialstate,i.e.,itiswhatmustbetruefromthebeginningofexecution.Thetableaumethodappliesasetofrewriterulestoconvertatemporalformulatoanequivalentformula.Rewriterulesareappliedrepeatedlyuntiltheoriginalformulaisbrokendownintosum-of-productsform.Attheconclusionofrewriting,eachtermoftheoveralldisjunctisaconjunctioninwhicheachindividualtermiseitheranatomicproposition(possiblynegated)whichcanbecheckeddirectly(i.e.,evaluatedwithaparticularassignmentofpropositionaltruthvaluesfromaprogramstate),orelseaformulabeginningwiththeÒnextstateÓconnective;thelatterarecalledÒdeferredÓformulae.Thedirectlycheckablepartsofatermbecomesthelabelonatransitionleadingtoastateassociatedwiththedeferredpartoftheterm(strippedoftheÒnext-stateÓpreÞx).Thisprocessisrepeatedinthenewstate,andsoonrepeatedly,togeneratealltransi-tionsandstatesoftheÞnite-stateacceptor.Becausethereareonlyaboundednumber Thechoiceisnon-deterministicifthelabelsofmorethanonetransitionmaybetrueinthesameprogramstate.DillonandYuprovideamethodfordeterminisingtheacceptors,whichcantheoreticallysufferfromacombinatorialblowupbutwhich,theyreport,performsreasonablywellinpractice. offormulaethatcanbegeneratedinthisway,eventuallytheprocessmustgenerateonlystatesandtransitionsthathavealreadybeengenerated,andthetableaualgorithmterminates.InprinciplethesizeofÞnitestateautomatacanbeexponentiallylargerthanthetemporalformulafromwhichtheyarederived,butexperiencesuggeststhatstandardsafetyandlivenesspropertiesusuallyproducesufÞcientlycompactautomata.Asimpleexamplemayhelpthereaderobtainanintuitionforhowatableaual-gorithmworks;amorecomplexexamplethatillustratesmanymoredetailsoftheal-gorithmcanbefoundinDillonandYu[DR96].SupposetheoriginalformulawasÒeventuallyandeventually,Ótypicallywritten(ÒeventuallyÓ)mayberewrittentoanequivalentformula,ornext-stateeventuallyÓ)i.e.,iseventuallytrueifitiseithertruealreadyorif,inthenextstate,itwilleventuallybetrue.Thesamerewriteruleisappliedtotheotherpartoftheformula,andaftersomerearrangingonearrivesatsum-of-productsform:Thefourpartsofthisformulabecomefourtransitions.Thewholetermisaformulacanbecheckeddirectly,soitlabelsatransitiontoastatelabeledÒtrue,Ói.e.,anacceptingstatewhichisreachedwhenthewholetemporalformulahasbeensatisÞedbyaprogramstateorsequenceofprogramstates.Thetermhasadirectlycheckablepart,whichlabelsatransitiontoastateassociatedwiththesub-formula,i.e.,ifweobserveaprogramstateinwhichistrue,thentheremainingobligationistoÞndanotherprogramstateinwhichistrue.Sofarthetableauconstructionproducesanon-deterministicautomaton.Notethatthetwotransitionsconsideredsofararenotexclusive(cancertainlybetrueiftrue).Thelastterm,moreover,ismadeupcompletelyofadeferredsub-formulae,andthereforeitproducesatransitionlabeledÒtrue.ÓWhileitwouldbepossibletointerpretanon-deterministicautomaton,keepingtrackofallthepossiblestatestheautomatonmayhavereachedafterallpossibletransitionsconsistentwitheachprogramstate,itisalsopossibletoÒdeterminiseÓtheautomaton.Forthesum-of-productsformulaabove,itisnotdifÞculttoseethatonecanobtainthesameultimateresultiftheformulaisrewrittenasTherewrittenformulaproducesadeterministicchoiceamongtransitions. Forthesakeofsimplicity,wehaveglossedoveranimportantdetail,viz.,determiningwhichautomatonstatesareaccepting(Þnal)states.InDillonÕsapproach,twoÒnextmomentÓmodalitiesareused,aÒweaknextÓindicatingthatthesub-formulaissatisÞediftheexecutionends,andaÒstrongnextÓwhichrequiresfurtherexecution.ThetableauforrequirestheÒstrongnext.Ó 5.2.1TemporaloraclesfromGILTableaumethodshavebeenusedforalongtimetoconstructautomatausefulintem-porallogicmodelcheckingofÞnite-statemodels[Wol85,VW94,GPVW95].DillonandYuobservedthattherequirementsofautomatausefulastestoraclesaresomewhatdifferent,sincemodelcheckinginvolvesreasoningaboutinÞniteexecutionsequences,whiletestoraclesdealonlywithÞniteexecutionsequences.Thisleadstoconstructionofsmallerautomata,whichcanbeinterpretedasordinaryÞnite-stateacceptorsratherthanB¬uchiautomata.TherestrictiontoÞnitesequencesalsohasconsequencesforthekindofspeciÞca-tionformulathatmakessense.DillonandYuadoptanintervallogic,inwhichonedoesnotstatewhatmusteventuallyhappen,butratherstatethatcertainthingsmusthappenwithinsomeboundedbetweentwootherstates[DY94].TheparticularintervallogicpresentedbyDillonandYuusesagraphicalsyntaxthatisdesignedtolookliketimingdiagrams.TheGILgraphicalsyntaxisratherunwieldyandisnotpresentedhere;theinterestedreadermayÞndadescriptionandexampleintheoriginalpaperspapers+94,DY94,DR96].Anexecutiontracecanbeconsideredasasequenceofeventsalternatingwithanabstractrepresentationofprogramstates.Forthepurposesofcheckingaprogramex-ecutionagainstatemporalorintervallogicspeciÞcation,programstatesareabstractedtoassignmentsofbooleanvaluestothepropositionalvariablesthatappearinspeciÞca-tionformulae,andeventsinclude(atleast)anyprogramactionthatchangesthevalueofoneormorepropositionalvariable.TheinputalphabetoftheÞnite-stateacceptoristhesequenceofprogramstatesalone(ignoringtheevents);theautomatoneitheracceptsorrejectsasequenceofabstractprogramstates.Inpractice,anautomatonisconstructedforthenegationofthespeciÞcationformula,soacceptingasequenceofabstractprogramstatesindicatesthatthesequenceviolatesthespeciÞedproperty.Example25ConversionofanintervallogicformulatoatestoracleintheformofaÞnite-stateacceptor(adaptedfrom[DY94]).Y94])..:p;.pk..:p))3mTheaboveformularequiresthatpropositionalvariablemustbetrueinsomeprogramstatewithineachintervalthatbeginswhenpropositionalvariablechangesvaluesfromfalsetotrue(),andendingwhenbecomesfalseagain.ThisspeciÞcationisnegatedtoobtainaformuladescribingaviolationoftheproperty:operty:..:p;..NegationshavebeenÒpushedintoÓtheformulauntiltheyareassociatedwithindividualpropositionalvariables,whichisapreconditionforconstructingtheautomatonwithatableaumethod.NegationchangesÒeventuallyÓtoÒalwaysÓandviceversa,andalsoaffectstheintervalconstructions.Thenegatedformulacanbereadasfollows:Thereissomeinterval(


)beginningwhenthetruthvalueofchangesfromfalsetotrue AB¬uchiautomaton[B60]islikeaÞnite-stateacceptor,butinsteadofacceptingaÞnitesequenceifitendsinanacceptingstate,aB¬uchiautomatonacceptsaninÞnitesequenceifitpassesthroughanaccept-ingstateaninÞnitenumberoftimes.InÞnitesequencescanberepresentedasloopingpathsinagraphrepresentation,whichishowmodelcheckerstestB¬uchiautomatonacceptanceinÞnitetime. ;..)andendingwhenbecomesfalseagain(),andwithinthatintervalremainsalwaysfalse(ApplyingthetableaumethodtothisnegatedformulatoformaÞnite-stateacceptor,andatthesamedeterminisingtheacceptor,wewouldobtain: 0 1 2 4 3 5 p :p :p p^m p p^m :p :p p :p p p Eachnodeoftheautomatonisassociatedwithaset(disjunction)oftemporalformulae.Node0isassociatedwiththeoverallformulaformula..:p;..istrueintheinitialprogramstate,wepasstonode1whichisassociatedwiththethe..:p;.....:p;..andsoforth.Ifweencounteraprogramstateinwhichisfalse,andthenastateinwhichistrueandisfalse,weenternode3oftheautomaton,inwhichtheendofexecutionorastateinwhichisfalsewillindicateaviolationofthetemporalspeciÞcation.5.3SCRThetemporaloraclesdescribedintheprevioussectionaregeneralinthesensethatthesameoraclecanbeusedforanyarbitraryexecution,i.e.,theoracleisdecoupledfromtestcaseselectionorgeneration.IfthespeciÞcationisalreadyintheformofastatemachine,orcanbeeasilyinterpretedasastatemachine,thenitmaybeusefultoderivetestcasesandcorrespondingoraclestogether.TheSCR(SoftwareCostReduction,[HKPS78])methodisarequirementsspeci-Þcationmethodologybasedonatabularnotationandonseveralaccompanyingtoolsforerrordetection.GargantiniandHeitmeyerhavedescribedanapproachforderivingfromSCRspeciÞcationmodelsofexternalsystembehavior,oraclespairedwithtest casesintheformoftestsequencesofinputsandexpectedoutputs.SincetheoutputsareÒcomputedÓfromthespeciÞcation,theSCRmodelactsastestoracle.AnSCRspeciÞcationdescribesboththesystemandtheenvironmentinwhichthesystemshouldoperate.Thesystemisrepresentedasanautomaton;theenvironmentasasetofcontrolledmonitoredvariables.Changesinthevaluesofcontrolledvari-ablesproduceinputeventstowhichthesystemreactsbychangingstateandpossiblyproducingoneormoreoutputevents,thatis,changesincontrolledquantities.SCRspeciÞcationsmayincludealsoauxiliaryvariablescalledmodeclasses,whosevalues,and.SCRspeciÞcationsareorganizedineventtables(Example26)conditiontables:theformeridentifyalleventsthesystemissensitiveto;thelatterdeÞnesystemresponses.Example26TheeventtablethatdeÞnesthemodeclass[GH99].TheSafetyInjectionSystem(SIS)isasimpliÞedsystemforsafetyinjectioninanuclearplant.ItsinputsarethemonitoredvariableswaterPres,and,andthesingleoutputisthecontrolledvariablesafetyInjection.ThespeciÞcationin-cludesalsoamodeclasswhoseeventtableis: Oldmode Event Newmode tooLow @T(waterPres permitted permitted @T(waterPres high @T(waterPres tooLow high @T(waterPres permitted ThenotationdenotesaÒtransitionÓintoastateinwhichthepredicatetrue,e.g.,describesastateinwhichisnottrue,butinwhichitbetrueinthenextstate.TheapproachofGargantiniandHeitmeyerusesmodelcheckingforconstructingthetestsequences.GivenapropertyandanSCRspeciÞcation,testsequencesaregeneratedasfollows:(a)Insteadof,theprocessstartsfromthenegationofthepremise,.(b)ThemodelcheckerisaskedtoÒverifyÓ.IfwereveriÞablytrue,thenwouldbeavacuousrequirement.SinceisnotveriÞablytrue,themodelcheckerproducesacounter-example.Thecounter-exampleisasequenceofeventsbeginningintheinitialstateandleadingtoastateinwhichistrue,whichisasuitablestatefortestingtheimplication.(c)ThetraceisusedtogenerateatestsequencebyÒexecutingÓitontheautomatonthatrepresentsthesystem(Example27.)Example27Anexampleoftestsequencegeneration[GH99]:Supposethefollowingproperty:low)WHENblock=Onreset=OffsafetyInjection’=Off InSection4.2wehavealreadydiscussedinterpretingtabularSCRinterfacespeciÞcationsasoracles.Wegroupedthatapproachwithothersthatdirectlydescribedprogramimplementationentities(variables,procedures,etc.),whereastheapproachdescribedinthissectionisbasedonobservableexternalsystembehavior.ThedistinctionbetweenÒmoduleÓandÒsystemÓistosomeextentarbitrary,andonecouldcertainlygroupapproachestotestoraclegenerationdifferently. TheexpressionstatesthatifwaterPresdropsbelowtheconstant,thensafetyInjectionmustbeIftheSCRspeciÞcationofSISsatisÞedtheproperty,wecouldusethepropertytoderiveatestsequence.Thenegationofthepremise,statedintemporallogic,iswaterPreswaterPresblockresetOffwhichisrenderedintotheinputsyntaxofthemodel-checkeras:AG!(EX(waterPreslow)&!(waterPreslow&block=On&reset=Off)Themodel-checkerproducesatracethatgeneratesthefollowingtestsequence: Step Monitoredvar.value Controlledvar.value Modeclassvalue 0 waterPress=2 safetyInjection=On pressure=tooLow block=Off reset=On 1 reset=Off 2 waterPress=5 3 waterPress=8 4 waterPress=10 safetyInjection=Off pressure=permitted 5 block=On 6 waterPress=8 pressure=tooLow Ateachstep,thetableshowsonlythevariablevalueswhichchangefromonesteptoanother.Step0correspondstotheinitialstateandonlyatstep4wehaveachangeintheoutput.Thesix-steptestsequencecanberepresentedmoreconciselyas:(r,off;-),(w,5;-),(w,8;-),(w,10;s,off),(b,on;-),(w,8;-)where,andrepresenttheinputvariableswaterPres,andrepresentstheoutputsafetyInjection;andindicatesnochanges.Theapproachreferstothepropertiesusedtogeneratetestsequencesastrapprop-andproposesamethodtoautomaticallyselectthemfromSCRtables.TrappropertiesaregeneratedbycoveringallpossiblesoftwarebehaviorsdescribedinthespeciÞcation.ThiswaytheselectionprocessdoesnotrequirehumaninterventionandthequalityofdeÞnedtestsequencesdoesnotdependonwhoselectstheproperties.Thedescriptionofhowtrappropertiesareinferredfromthetablesisoutofthescopeofthispaper;detailscanbefoundin[GH99].5.4Multi-LanguageSpeciÞcationsDifferentspeciÞcationlanguagesaresuitedtodescribingdifferentprogramproperties.Forexample,temporallogicsarewellsuitedtodescribingallowedsequencesofevents,particularlylivenessproperties,buttheyarepoorlysuitedtodescribingpropertiesofcomplexdatastructures.AsetofcomplementaryspeciÞcationsindifferentformalisms isoftenmoreconciseandunderstandable,andthereforelesspronetoerrorsinspeciÞ-cationorimplementation,thanaspeciÞcationinwhichallpropertieshavebeencoercedintoonespeciÞcationparadigm[ZJ96].OnemightsimplycreatetestoraclesfromeachofthemultiplenotationsusedinaprogramspeciÞcation.Iforaclesandtestcaseselectionwereentirelyindependent,thecostofindependenttestoraclegenerationwouldberedundancyinmonitoringprogramexecutions,whichmightbeacceptable.Inmanycases,though,testoraclesandtestcaseselectionarecoupled,andthenonewouldbelimitedtoapplyingonlyonenotation-speciÞcsetoftestoracleswitheachclassoftestcases.Richardsonetal.havedescribedamoreintegratedapproachtomanagingoraclesgeneratedfrommultipleformalspeciÞcationlanguages[RLAO92].Oracleinfor-mationisassociatedwithtestclasses,constraintsontestinputs(dataorsequencesofstimuli)atasufÞcientlyabstractlevelthatasingleconcretetestcasemaysatisfytestclassesderivedfromdifferentspeciÞcations.AnyoraclederivedfromaformalspeciÞcationdeÞnesordependsonarelationbetweenentitiesinthesemanticdomainofthespeciÞcation,andobservableentities(events,variablevalues,etc.)fromprogramexecutions.Whentherearemultiplespec-iÞcationnotations,thereisonerelationforeachnotation,butthoserelationsmaybepartlycombined.IntheapproachofRichardsonetal.,asingleexecutionmonitorsupportsmultiplemappingsofcontrolanddataintothesemanticdomainsofcomple-mentaryspeciÞcations.Themonitoressentiallygathersstateinformation(orrather,arecordofstatechanges)sufÞcienttointerprettheeventsandstatevariablesinallofthespeciÞcations,throughasetof(possiblydifferent)datamappings.ThisinformationisgatheredatpointsrelevanttoanyofthespeciÞcations,asdeÞnedbyacontrolmappingsomemonitoredinformationmaybeirrelevanttosomespeciÞcations.SpeciÞcationsareconjoinedbyapplyingeachoracletoeachapplicablecontrolpoint.6TraceCheckingFrequentlyapartialtraceofeventsiseitherdirectlyavailable(atinterfacesbetweenamoduleorsystemanditsenvironment)orcanbeobtainedthroughprograminstrumen-tation.SuchatracecanbecheckedbyanoraclederivedfromaformalspeciÞcationofexternallyobservablebehavior(e.g.,theGILspeciÞcationsdiscussedinSection5.2),oritmaybecheckedforconformancetoamoredetailed,operationalmodelofprogrambehavior.Testingtechniquesinwhichsequencesofinteractionsarecheckedagainstformaldesignmodelshavebeenmostthoroughlydevelopedinprotocolconformancetest-ing.CharacteristicsofsystemsinthisdomainÐconcurrency,physicaldistribution, TheapproachisillustratedwithZandareal-timeintervallogic.Sincethecontributionoftheworkisintacticsforcombiningoraclesmorethantheoraclegenerationmethodsfortheindividualnotations,andsinceinanycasetheindividualoraclegenerationmethodsaresimilartoapproachesdescribedelsewhereinthissurvey,weomittheexample.Inthecommunicationprotocoldomain,theseformaldesignmodelsareconventionallycalledÒspeciÞ-cations.ÓWewillfollowthatconventionwhendiscussingcommunicationprotocols,butwillreverttothetermÒmodelsÓfordomainsinwhichtheywouldbeadesigndetailthatisnotpartofthenormallyobservablesystembehavior. sensitivitytotimingÐundermineconventionaltestingtechniquesandmotivatedevel-opmentofmoresuitableandspecial-purposeones.Moreover,sincecommunicationprotocolsdeÞneinterfacesoverwhichotherwiseopaquecomponentsfromdifferentorganizationsmustcooperate,theyareasuitablycompleteandpreciserepresentationofacceptablebehavior.Oraclesbasedonsequencesofinteractionscanalsobeappliedtodistributedcomponent-basedsoftwareinotherdomains.Theapproachisparticularlyusefulfortestingsys-temsinwhichsourcecodeisunavailable(buttheeventtracecanbeobtainedatinter-facesbetweenthesystemundertestanditsenvironment).Icanbeusedinintegrationandsystemtesting,whereembeddedassertionsandspeciÞcationsofindividualinter-facesareofteninadequatetocapturebehaviorsofinterest.Forexample,anoraclethatcheckseventtracescancorrelateeventsinwidelyseparatedmodules(say,aninputandaresultingoutput)withouteitheradirectinterfacebetweenthetwomodules(noÒcon-tractÓagainstwhichinterfaceassertionscouldbemade)oracompletemodelofhowtheoneeventleadseventuallytotheother.TracecheckingcanbeappliedwhennocompletespeciÞcationofacceptablebehav-iormaybeavailable.Rather,onemayhaveonlyanumberofsmallproperties,eachofwhichisbelievedtobenecessarytocorrectfunctioningofthesystem,andwhichcanbecheckedindependently.Satisfyingeachofthesepropertiesdoesnotimplyoverallcorrectness,butviolationofanyoneindicateseitherafaultyprogramorfaultyunder-standingorformulationoftheproperties.Someofthepropertiesmayberelatedtooverallcorrectnesspropertiesofaprogramorsystem,andothersmaybebasedonamoredetailedoperationalunderstandingofthesystemisintendedtowork.6.1ProtocolConformanceTestingAutomaticderivationoftestoraclesfromformalspeciÞcationsbasedonstatema-chineswasdevelopedearliestandmostthoroughlyinthedomainofcommunica-tionprotocolconformancetesting,possiblyduetoseveralpeculiaritiesofthatdomain[vBDZ89,vBP94,FvBK91].EnablingfactorsfordevelopmentofprotocoltestingtechniquesincludewidespreadadoptionofasmallnumberofspeciÞcationformalisms,observabilitythroughwell-deÞnedinterfaces,andfunctionalrequirementsthatdonotextendtoofarbeyondwhatcanbemodeledusingÞnite-statemachines.UsuallythesameFSMspeciÞcationisusedbothasasourceoftestcasesandasasourceofinfor-mationforthetestoracle.CommunicationprotocolsareoftenspeciÞedusingcommunicatingÞnitestatema-chines,usuallywithsomeextensionsthatmakethemnottrulyÞnite-state.Forex-ample,astatemachinethatsimplyreceivesamessageononeportandthensendsthesamemessageonanotherportisnotreallyÞnite-stateunlessthesetofpossiblemessagesisÞnite.Fortunately,thenon-Þnite-statepartsofthespeciÞcationareoftensimpleenoughthatanFSMremainsausefulmodelfortestingaswellasspeciÞca-tion.Controlsystemssharesomeofthesecharacteristics,andtheapproachdescribedinSection5.3isessentiallysimilar,althoughthestate-machinemodelinthatcaseisderivedfromSCRspeciÞcations. 6.1.1Wp:Arepresentativeprotocolconformancetestmethod(partialWmethod)[FvBK91]isrepresentativeofconformancetestmethodsinwhichtestcoverageandanoracleareinterdependent,andinwhich(incon-trasttomostdynamictesttechniques)successfulexecutionofatestsuiteissufÞcienttomakestronginferencesofcorrectness.Ofcourse,theinferencedependsonstrictlimitationsonboththeformalrepresentationoftheprotocolanditsimplementation.TheybothmustbeÞnite,fullyspeciÞed,anddeterministic.Theymustsharethesameinputalphabetandthenumberofstatesintheimplementationmustbeboundbyaknowninteger.Allstatesinthetwoautomatamustbereachablefromtheinitialstates,andthetwoautomatamustprovidearesetoperationwhichreturnstheprotocoltotheinitialstate.Ifallthesehypotheseshold,themethodisabletoselect(basedonthespeciÞcation)asetoftestcasesthatareabletodetect(fromrunningtheimplemen-tationandcheckingitwithanoraclebasedonthespeciÞcation)allerrorsduetobothwrongoutputsproducedbystatetransitionsandtransfererrors,thatis,differentstatesreachedbycorrespondingtransitionsinExample28,takenfromFujiwaraetal[FvBK91],illustratestheWp-method.Theandtheimplementationmeettherequirementsofthemethod,butnotequivalentto.AnytestsuiteselectedbytheWp-methodisthereforeguaranteedtoexposethediscrepancy.OraclesintheWp-methodarebasedonsequencesofinputsandoutputsthatuniquelyidentifystatesinthespeciÞcationmachine.AnidentiÞcationsetisdeterminedforeachstateofspeciÞcation;theunionofallthecharacterizationsettheautomaton.ThegeneralstrategyistoÞrstusetocharacterizeeachstateintheimplementation,andafterwordusethesmallertocheckthatatransitionreachesaparticularstateExample28AsimpleprotocolspeciÞcationandincorrectimplementation(from[FvBK91]) S0 S1 S2 a/e a/f c/e c/e b/f b/f b/e c/f a/f I0 I1 I2 a/e a/f c/e c/e b/f b/f b/e c/f (a)SpeciÞcation(b)ImplementationSpeciÞcationandimplementationareclearlydifferent:hasaself-transitiononstate(i.e.,if),whilethesametransitioninisfromstate.Forthesakeofclarity,transitionsarenotdrawninthetwographs.asinputalphabetandasoutputalphabet.Byapplyingthe approach,weobtain:Phase1:Inputsequences:Outputsequences:TheÞrstphaseisnotenoughtodiscovertheerrorsincetheinputsequences,appliedonthetwoautomata,givethesameoutputs.Phase2:Inputsequences:Outputsequences:Wecanidentifythefaultybehavioroftheimplementationonlybyapplyingthesecondsetofinputsequences.Theoutputinboldshouldhavebeen,thishighlightsthenextstatefaultoftheimplementation.Intheexample,theoutputafterinputisenoughtodistinguishstate,so.AlthoughwemightchooseasasufÞcientsetofsequencestoidentify,we(arbitrarily)choose,which(thoughnotminimumamongallpossiblechoices)isalsominimal,i.e.,neitherofitssubsetswouldbesufÞ-cienttodistinguish.Inputwithoutputisenoughtoindicatethatweareineitherstate,whileinputwithoutputindicatesweareineither,sotryingbothisenoughtoconcludethatweareinstateTotrybothinputsfromstateitisnecessarytoreachonce,try,andlaterresettotheinitialstateandreachagain.Ingeneral,wewillneedasetofinputsequencessufÞcienttoreacheachstate,inadditiontoasetofsequencestodistinguisheachstatefromallothers.Inthiscase,thestatecoverissufÞcienttoreachallthestates,andthesequencesthatreachanddistinguishitfromotherstatesare.Itmightseemthatweneedonlytoreachandidentify,butinitiallywecharacterizeitwith(usingalltheelementsof).Thisisbecause,ifweusedjusttocharacterize,andjusttocharacterizewemightbefooledbyanimplementationinwhichasinglestatehastransitions.Afterhavingusedtheentiresettocharacterizeeachstateonce,wecanbesurethatthesmallersetisenoughtoidentifyeachstate.TheÞrstphaseoftheWpmethodcoverseachstate,butitmaynotcoveralltran-sitions.ThesecondphaseoftheWpmethodcoverseachoftheremainingtransitions(thetransitioncoversequenceslessthestatecoverset,whichwascoveredintheÞrstphase),usingonlythesetstochecktheendingstateofeachtransition. TheheavyconstraintsontheFSAsimposedin[FvBK91]havebeenrelaxedin[LPB93]forbothpartiallyspeciÞedandnondeterministicautomata.Partiallyspec-iÞedautomatacanbeinterpretedascompletespeciÞcationsusingeitherthedonÕtcareinterpretation,whichletstheimplementationdecidetheoutputforundeÞnedtransi-tions,ortheforbiddeninterpretation,whichconsidersunspeciÞedtransitionsasfor-biddenones,thatis,transitionsthatcannotbeexecuted.Nondeterministicautomatafurtherextendtheconformancerelationtobecomeaquasi-equivalencerelation:ThespeciÞcationanditsimplementationmustproducethesamesetofoutputsequencesforeveryinputsequencethatcanbeacceptedbythespeciÞcation.Theyrequirealsothecomplete-testingassumption(i.e.,aÞnitefairnessassumption):byapplyingaÞnitenumberoftimesagiveninputsequencetotheimplementation,itmustbepos-sibletoexercisealltheexecutionpathsoftheimplementationwhicharetraversedbytheinputsequence.Thenumberoftimesaninputsequencehastobeappliedhastobedeterminedusingstatisticalandoptimizationtechniques.OnecouldarguethattheÒoracleÓpartofprotocolconformancetestingismerelytheindividualcheckofanoutputproducedbyatransitionagainstthatpredictedbythespeciÞcation,andtherestofamethodlikeWpconcernstestcoverage.TheviewtakenhereisthattheÒoracleÓpartistheevidenceforconcludingthatthestatereachedintheimplementationcorrespondstothestateprescribedbythespeciÞcation,whichmayrequirecheckingmultipletransitionsÑe.g.,thecharacterizationofstatesintheÞrststageoftheWpmethod,andtheidentiÞcationofstatesinthesecondstage.Howeveronechoosestoviewit,anessentialcharacteristicofprotocolconformancetestingisthetightcouplingbetweentestselectionandoracle.6.2OraclesforGUIsGraphicaluserinterfaces(GUIs)arenotoriouslyexpensivetotest,sinceusuallythebe-haviorofanapplicationmustbejudgedacceptableornotbyahumantester.Thecur-rentlydominanttool-supportedapproachincommercialpracticeiscaptureandreplayofexecutionsequences,whichrequiresahumantesterinitiallybutgreatlyreducesthecostofrepeatingtestsasthesoftwareevolves.Asregardstestoracles,capture/replaywithautomatedcomparisontopastresultsisaninstanceofanoraclebasedonasetofpre-computedinput/outputpairs,whichisoutsidethescopeofthissurvey.Analternativetocapture/replaytestingofGUIsisautomatedtestingbasedonamodelofGUIoperation.WhenthemodelcanbeÒexecutedÓthroughexamplescenar-ios,automatedtestingbearssomeresemblancetoothertechniquesthatdriveasystemthroughsequencesofstimuliandjudgethecorrectnessoftheresultingsequencesofresponses.Primarydifferencesappearintheformofthemodelwhichservesasaspec-iÞcation,andinthewaythatresponsesareobserved.AnimportantenablingconditionisthattheGUIimplementationsubstrateprovideawayofqueryingasetofpropertiesthatdescribethestateofGUI,ratherthanrequiringinterpretationoftheactualdisplay.ThePlanningAssistedTesterforgrapHicaluserinterfaceSystems(PATHS[MPS00])isatestoracleforgraphicalinterfacesbasedonanexternalformalspeciÞcationoftheGUIundertest.AsetofpropertiesofinterestdeÞnesthestatespacetobeanalyzed;special-purposeoperatorsdeÞnehowthecurrentstatecanchange.Theinitialstatedependsonthechosentestcase,aswewillseebelow. Interfacescanbecharacterizedwitheitherareducedsetofproperties.Intheformercase,werelyonthegraphicaltoolkitusedtoimplementtheinterfacetoidentifytheproperties.Forexample,ifwedecidedtouseJavaanditspackage,thepropertiescouldbealltheinstancevariablesassociatedwithobjects(classes).Inthelattercase,designerscandeÞneasetofpropertiesthatpermitthemtoworkatahigherabstractionlevel.AÞrstsetofpropertiesforanotepad-likeinterfaceispresentedinExample29.Actions,thatis,statetransducers,areusedtocharacterizetheevolutionofthestateofaGUIovertime.Sincethestatespaceremainsimplicit,actionsarenotassociateddirectlywithtransitions,buttheyaremodeledasoperators:Anactiontogetherwithitspossibleparameters,thesetofpreconditionsthatmustbesatisÞedtoexecuteitanditseffects.Forexample,iftheactionwereset-background-color(window,,wherethebackgroundcolorisapropertyforallwindows,thepreconditioncouldrequirethatthewindowbeactiveandthecurrentcolordifferentfromthenewone.Theeffectcouldrequirethatthecolorbeactuallychanged(Seeexample29foracompleteoperator.)TestcasesdeÞnetheinitialstatesandthesequenceofoperatorsthatmustbeap-plied.Ateachstep,thenextstateisdeÞnedbyapplyingtheeffectsassociatedwiththeoperatortothecurrentstate.Example29Someproperties,apossibleinitialstate,andanoperatorforasimplenotepad-likeapplication(from[MPS00])in(file,text)FilecontainstextcontainsFile(dir,file)DircontainsÞlecurrentFont(font,style,size)DeÞnesthecurrentfont,style,andsizeforthedocumentfont(text,font,style,size)Textisinfont,style,andsizeonScreen(text)TextisdisplayedonthescreenApossibleinitialstate(deÞnedbyaparticulartestcase)isdeÞnedasfollows:containsÞle(samples,f4.doc)containsÞle(private,f1.doc)currentFont(TimesNormal12pt),TimesNormal12pt)ÒistheÓÒistheÓ,TimesNormal12pt),TimesNormal12pt)ThisexcerptoftheinitialstateidentiÞestwodirectories,samplesandprivate,andtwo Þlesinthesedirectories,f1.docandf4.doc.Then,itdeÞnesthecurrentfontusedforthedocumentandafewstringsthatareintheÞle(text)f1.doc,alongwiththeusedfont,type,andsize.AsimpleoperatortoopenaÞleofagivendirectory,isdeÞnedasfollows:OperatorNamePreconditionTheoperatorstatesthattheopenedÞlebecomesthecurrentÞle.Noobject(string)isdisplayedonthescreenwhileopeningtheÞle;attheendonlytheobjectsintheÞleareonthescreen.Theoracleconsistsoftwoprocesses:anexecutionmonitoranda.TheÞrstprocessisinchargeofextractingthecurrentvaluesforallpropertiesofinterest.TheveriÞercheckstheserealvaluesagainsttheexpectedones,thatis,theonescomputedontheexternalmodel.TheveriÞcationcanbetailoredtodifferentdegreesoftesting:TheveriÞercancompareonlythosepropertiesthatareexpectedtochange,thatis,thosepropertiesthatareinßuencedbytheeffectsoftheoperator.ItcancheckonlythepropertiesinthereducedsetdeÞnedbythedesigner,oritcancheckallproperties.7LogFileAnalysisInmostsoftwaredevelopment,theprimaryspeciÞcationsanddesignmodelsarenotcommunicatingstatemachines,andthebehaviorofinterestisnotlimitedtoexchangeofmessageswithanenvironment.Thetechniquesofprotocolconformancetestingarethereforenotcompletelyapplicable,butrelatedtechniquesmaystillbeapplicable.Andrews[And98,AZ00]hasdescribedtheapplicationofparallelstate-machinespeciÞcationstochecklogÞlesproducedbyapplicationsystems.Softwaredevelopersexplicitlyincludecommandstologeventsofpotentialinterest.Testoraclessimulateexecutionofeachindividualstatemachinereactingtoonlytheloggedeventsrelevanttothatstatemachine.TheLogFileAnalysisLanguage(LFAL)[And98]isanexplicitdescriptionofthestatesandtransitionsinastatemachinethatacceptsparticularsequencesofevents.Example30LFALstatemachinespeciÞcationdescribingapropertyofagraphicaluserinterface[And98].machineall_popups_get_closed;initialstatenone;fromnone,onopen_popup(Name),toopen(Name);fromopen(Name),onpopup_response(Name,X),toexp_close(Name);fromexp_close(Name),onclose_popup(Name), tonone;finalstatenone;Example30isaLFALspeciÞcationofasequencingpropertyofagraphicaluserinterface.Itrequiresthateachpop-upwindowiseventuallyclosed,butonlyafterauserhasrespondedtoit.Symbolsbeginningwithupper-caselettersarevariableswhichmustbeconsistentlyboundinthelogÞle,e.g.,theexamplespeciÞcationwouldmatchtheopen,response,closesequenceforpop-upinexample31,butitwouldfailforpop-upinwhichthewindowcloseeventisnotprecededbyauserresponse.Example31ApartiallogÞleforanLFALtestoracle.Anarbitrarysequenceofirrelevanteventrecordsmaybeinterleavedwiththeeventsshown.open_popupwp8018irrelevent_eventfoobar"sometext"24open-popupwp8019popup_responsewp8019exp_closewp8019menu_open"mightberelevanttosomeothermachine"popup_closewp8018Inpractice,manyprogrammersalreadycreateinstrumentationthatproducestracesofÒinterestingÓeventsfordebugging,buttheyoftendisableitorevenremoveitbeforesystemtesting.Andrewshasdemonstratedthatacollectionofsimplestate-machinespeciÞcationscanbeusedastestoraclesforlogÞlesinavarietyofapplicationdo-mains,notlimitedtothoseinwhichstatemachinesaretypicallyusedasspeciÞcations[AZ00].Feather[Fea98,FS01]hasappliedavariantoflogÞleanalysisinwhichthelogisloadedintoadatabaseandthespeciÞedpropertiesarestatedasdatabasequeries,usingthedatabasequeryengineinlieuofaspecial-purposeprogramforcheckingtestoracles.Insomecases,existingspeciÞcationsofapplicationfunctionalitycanbecreatedautomaticallyfromspeciÞcationsordesigndocumentationinadomain-speciÞcnotation[FS01].TheproblemofrelatingnamesinaspeciÞcationtolow-leveleventsislesssevereinlogÞleanalysis,sinceexplicitloggingactionsaredistinctfromprogramfunction-ingandcanusethevocabularyofthestate-machinespeciÞcations.Andrewsargues,though,thatthecontentoflogÞlesmustbecarefullydesignedtocapturetherelevantsemanticevents.EssentiallythedeveloperisrequiredtodesignthemappingbetweeneventsinspeciÞedpropertiesandimplementation-leveleventswhilecreatinginstru-mentedsourcecode.8DiscussionAnidealoraclesystemwouldderivetestoraclesfromthesamesoftwarespeciÞcationusedastheagreementbetweenclientandimplementer.ItwouldacceptÒnaturalÓspec-iÞcations,withoutimposingconstraintsthatmakethatspeciÞcationlessusefulasdoc-umentationandthecurrencyofnegotiation.Itwouldnonethelessprovideanunerring pass/failjudgmentforanypossibleprogramexecution,atreasonablecost.ProducinganykindofprogramspeciÞcationwhichiscomprehensive,precise,andunderstand-ableisdifÞcultenough,soitisnotsurprisingthataddingeffectivecomputabilitytothesetofconstraintsrendersthemunsolvable.Thusitisunlikelythattherewilleverbeanidealsystemforcreatingtestoracles.Instead,thereareavarietyofapproachesthatmakedifferentcompromisestoproducetestoraclesthat,thoughnotideal,balancethetrade-offstoprovideusefulcapabilities.Wehavegroupedoraclesystemsbasedonimplementationapproaches(e.g.,em-beddedassertions,executionloganalyzers)andonthekindsofspeciÞcationstheyaccept(e.g.,interfacespeciÞcations,designmodels,property-andmodel-basedspeci-Þcationsofexternallyvisiblebehavior.)ImperfectasthisclassiÞcationschememaybe,itdoestendtogathertogethersystemsthatfacesimilarproblems,andservesthereforetohighlightsomerecurringstrategiesandsomedifferencesintactics.Concretevs.abstractstateandbehavior:Someoftheoraclesystemspredicatedi-rectlyonimplementation-levelstateorobservablebehavior.TheseÒconcreteÓoraclesincludeembeddedrun-timeassertionsandinterfacecontractassertions,butalsologÞleanalysistools.Whenoraclesarebasedonmoreabstractdescrip-tionsofprogrambehavior,theymustbridgethegapbetweentheconcreteentitiesandspeciÞcationentities.Inonewayoranotherthisalwaysinvolvesprovidinganabstractionmappingfromconcretetoabstractentities.Partiality:OraclesystemsbasedonspeciÞcationswrittenforotherpurposes(whetherspeciÞcationsofexternalprogrambehavior,ormoduleinterfacespeciÞcations)typicallytrytocheckthosespeciÞcationsprecisely,acceptingexactlythebe-haviorsconsistentwiththespeciÞcation.WhenanoraclesystemusesitsownspeciÞcationnotation,distinctfromspeciÞcationsusedforotherpurposes,itisusuallyÒpartialÓinthesensethatonlysomeincorrectbehaviorsarerejected,andotherincorrectbehaviorsmayescapedetection.WhileacompleteoraclebasedonprogramormodulespeciÞcationsisattractive,partialityhasimportantpragmaticadvantages,includinglow-costincrementaladoption.QuantiÞcation:SpeciÞcationnotationsandprogramminglanguagesmakedifferenttrade-offsbetweenexpressivenessandefÞcientcomputability.InaspeciÞcationnotation,thereistypicallynoreasontoavoidquantifyingoverlargeorinÞnitesets.Programminglanguages,ontheotherhand,eitherdonotprovidethoseconstructsormaketheircostsapparent.Atestoraclesystem,likeso-calledÒex-ecutablespeciÞcationlanguages,Ómuststrikeacompromisebetweenexpressive-nessandefÞciency.Therangeoftacticsusedbyoraclesystems,rangingfromcompleteomissionofquantiÞerstotreatingquantiÞersasloopingconstructstoattemptstorewritespeciÞcationstoeliminatethem,indicatesthatthereisnoclearoptimumbalancenoranyfullysatisfactoryapproachtoaccommodating WeyukercallstheseÒpseudo-oracles,Óandnotesthatitissometimesmucheasiertodistinguishplausiblefromimplausibleresultsthantopreciselydistinguishcorrectfromincorrectresults[Wey82]. Oraclesandtestcaseselection:Inanidealoraclesystem,oracleswouldbeorthogo-naltotestcaseselection.Inpracticalsystems,itissometimesmorepracticaltodetermineacceptablebehaviorsforlimitedclassesoftestcases.Inparticular,onemustoftentradeexpressivenessofspeciÞcationsagainstgenerality.Often,butnotalways,thetrade-offispartlydeterminedbythenatureofthespeciÞcations.Model-orientedspeciÞcationsanddesignmodels,whichlendthemselvestosim-ulatedexecution,areoftenusedtoderivetestclassesandtest-class-speciÞctestoraclestogether.Property-orientedspeciÞcationsaremoreoftenusedtoderivetestoraclesthatareindependentoftestcases,althoughevenaproperty-orientedspeciÞcationmaysometimesbesymbolicallyevaluatedtoobtainasimplerormoreefÞcienttestoracleforalimitedsetoftestcases.Thesesametrade-offsaretosomeextentanindicationofareaswhereonemayexpectfutureresearchprogress,tiedontheonehandtoresearchinsoftwaretestingandontheothertoresearchinspeciÞcationlanguagesandmethodologies.Intheinterim,despitetheabsencenoworintheforeseeablefutureofanidealsystemforcreatingtestoracles,thestateoftheartandpracticeisalreadywellenoughdevelopedthattherecanbelittleexcuseforrelyingexclusivelyonthemostexpensiveandleastdependableoftestoracles,thehumaneye.References[AH00]SergioAntoyandRichardG.Hamlet.AutomaticallyCheckinganImplementa-tionagainstItsFormalSpeciÞcation.IEEETransactionsonSoftwareEngineer-,26(1):55Ð69,January2000.[And98]JamesH.Andrews.TestingusinglogÞleanalysis:Tools,methods,andissues.Proceedingsofthe13thIEEEInternationalConferenceonAdvancedSoftware,pages157Ð166,Honolulu,Hawaii,October1998.[ass]Packagecorejava.http://www.hio.hen.nl/java/corejava/[AZ00]JamesH.AndrewsandYingjunZhang.Broad-spectrumstudiesoflogÞleanaly-sis.InProceedingsofthe22ndInternationalConferenceonSoftwareEngineer-ing(ICSE2000),pages105Ð114,Limerick,Ireland,June2000.ACMPress.Press.¬60]J.R.B¬uchi.Onadecisionmethodinrestrictedsecondorderarithmetic.InProc.1960CongressonLogic,Methodology,andPhilosophyofScience,pages1Ð11.StanfordUniversityPress,1960.[Bas76]VictorR.Basili.TheDesignandImplementationofaFamilyofApplication-OrientedLanguages.InProceedingsofthe5thTexasConferenceonComputing,pages6Ð12,October1976.[Bho00]A.Bhorkar.ARun-timeAssertionCheckerforJavausingJML.TechnicalReport00-08,DepartmentofComputerScience,IowaStateUniversity,2000..+89]D.Carrington,D.Duke,R.Duke,P.King,G.Rose,andG.Smith.Object-Z:Anobject-orientedextensiontoZ.InFormalDescriptionTechniques(FORTEÕ89)pages281Ð296.North-HollandPublishingCo.,December1989.[CES86]EdmundClarke,E.AllenEmerson,andA.PrasadSistla.AutomaticveriÞca-tionofÞnite-stateconcurrentsystemsusingtemporallogicspeciÞcations.ACMTransactionsonProgrammingLanguagesandSystems,8(2):244Ð263,1986. [Cha82]D.Chapman.AProgramTestingAssistant.CommunicationsoftheACMSeptember1982.[CPG00]EdmundClarke,DoronPeled,andOrnaGrumberg.ModelChecking.MITPress,2000.[DG85]J.D.DayandJ.D.Gannon.AtestoraclebasedonformalspeciÞcations.InProc.SoftFair,ASecondConf.onSoftwareDevelopmentTools,Techniques,and,pages126Ð130,SanFrancisco,Dec1985.ACMPress.[DH98]A.DuncanandU.Hlzle.AddingContractstoJavawithHandshake.TechnicalReportTRCS98-32,UniversityofCalifornia,SantaBarbara,1998.1998.+94]L.K.Dillon,G.Kutty,L.E.Moser,P.M.Melliar-Smith,andY.S.Ramakrishna.AGraphicalIntervalLogicforSpecifyingConcurrentSystems.ACMTransac-tionsonSoftwareEngineeringandMethodology,3(2):131Ð165,April1994.[DR96]L.K.DillonandY.S.Ramakrishna.GeneratingOraclesfromYourFavoriteTem-poralLogicSpeciÞcations.InProceedingsoftheFourthACMSIGSOFTSym-posiumontheFoundationsofSoftwareEngineering,volume21(6)ofACMSoft-wareEngineeringNotes,pages106Ð117.ACMPress,October1996.[DY94]L.K.DillonandQ.Yu.OraclesforCheckingTemporalPropertiesofConcurrentSystems.InProceedingsoftheACMSIGSOFTÕ94SymposiumontheFounda-tionsofSoftwareEngineering,pages140Ð153,December1994.[Eif]Eiffelwebpage.http://www.eiffel.com[Fea98]MartinS.Feather.Rapidapplicationoflightweightformalmethodsforconsis-tencyanalysis.IEEETransactionsonSoftwareEngineering,24(11):949Ð959,November1998.[FS01]MartinS.FeatherandBenSmith.AutomaticgenerationoftestoraclesÑfrompilotstudiestoapplication.AutomatedSoftwareEngineeringJournal,8(1):31Ð62,2001.2001.+91]S.Fujiwara,G.v.Bochmann,F.Khendek,M.Amalou,andA.Ghedamsi.TestSelectionBasedonFiniteStateModels.IEEETransactionsonSoftwareEngi-,17(6):591Ð603,June1991.[GH99]AngeloGargantiniandConnieHeitmeyer.UsingModelCheckingtoGenerateTestsfromRequirementsSpeciÞcations.InProceedingsofthe7thEuropeanEngineeringConferenceandthe7thACMSIGSOFTSymposiumontheFoun-dationsofSoftwareEngeneering,volume24.6ofSoftwareEngineeringNotes,pages146Ð162.ACMPress,September6Ð101999.[GHW85]JohnV.Guttag,JamesJ.Horning,andJeanetteM.Wing.TheLarchFamilyofSpeciÞcationLanguages.IEEESoftware,8(3):24Ð36,September1985.[GMH81]JohnD.Gannon,PaulMcMullin,andRichardG.Hamlet.Data-AbstractionIm-plementation,SpeciÞcation,andTesting.ACMTransactionsonProgrammingLanguagesandSystems,3(3):211Ð223,July1981.[GPVW95]RobGerth,DoronPeled,MosheVardi,andPierreWolper.Simpleon-the-ßyautomaticveriÞcationoflineartemporallogic.InProtocolSpeciÞcationTestingandVeriÞcation,pages3Ð18,Warsaw,Poland,1995.Chapman&Hall.[Gut77]JohnGuttag.Abstractdatatypesandthedevelopmentofdatastructures.municationsoftheACM,20(6):396Ð404,June1977. [Ham77]RichardG.Hamlet.TestingProgramswiththeAidofaCompiler.IEEETrans-actionsonSoftwareEngineering,3(4):279Ð290,July1977.[HK76]SidneyL.HantlerandJohnC.King.Anintroductiontoprovingthecorrectnessofprograms.ACMComputingSurveys,8(3):331Ð353,September1976.[HKPS78]K.L.Heninger,J.Kallander,D.L.Parnas,andJ.E.Shore.SoftwareRequire-mentsfortheA-7EAircraft.NRLMemorandumReport3876,UnitedStatesNavalResearchLaboratory,November1978.[Hol97]GerardHolzmann.ThemodelcheckerSpin.IEEETransactionsonSoftware,23(5):279Ð295,1997.[jas]TheJassPage.http://semantik.informatik.uni-oldenburg.[JMS]UsingJMSAsserttoDesignbyContract.http://www.mmsindia.com/JMSAssert.html[jsr]Jsr41.http://java.sun.com/aboutJava/communityprocess/jsr/jsr_041_asrt.html[jUn]Junit.http://www.junit.org[KC98]J.R.KiniryandE.Cheong.JPP:AJavaPre-processor.TechnicalReportCS-TR-98-15,CaliforniaInstituteofTechnology,1998.[KHB98]M.Karaorman,U.Hlzle,andJ.Bruno.jContractor:ReßectiveJavaLibrarytoSupportDesign-by-Contract.TechnicalReportTRCS98-31,UniversityofCali-fornia,SantaBarbara,1998.[Kra98]R.Kramer.iContractÐTheJavaDesignbyContractTool.InProceedingsofTOOLS26:TechnologyofObject-OrientedLanguagesandSystems,pages295Ð307.IEEEComputerSociety,1998.[LPB93]G.Luo,A.Petrenko,andG.V.Bochmann.SelectingTestSequencesforPartially-SpeciÞedNondeterministicFiniteStateMachines.TechnicalReportIRO-864,Dept.dÕInformatiqueetdeRechercheOprationnelle,UniversitdeMontral,1993.[Luc90]DavidLuckham.ProgrammingwithSpeciÞcations:AnIntroductiontoANNA,ALanguageforSpecifyingAdaPrograms.Springer-Verlag,1990.[Lv85]D.C.LuckhamandF.W.vonHenke.OverviewofAnna,aSpeciÞcationLan-guageforAda.IEEESoftware,2(2):9Ð22,March1985.[LvHKBO87]D.C.Luckham,F.W.vonHenke,B.Krieg-Br¬uckner,andO.Owe.Anna-ALan-guageforAnnotatingAdaPrograms,volume260ofLectureNotesinComputer.Springer-Verlag,1987.[Mak98]P.J.Maker.GNUNanaÐUserÕsGuide(version2.4).Technicalreport,SchoolofInformationTechnologyÐNorthernTerritoryUniversity,July1998.[Mey92]BertrandMeyer.Eiffel:TheLanguage.Object-OrientedSeries.PrenticeHall,NewYork,N.Y.,1992.[Mey97]BertrandMeyer.Object-OrientedSoftwareConstruction,SecondEdition.TheObject-OrientedSeries.Prentice-Hall,1997.[Mic93]SunMicrosystems.ADLlanguagereferencemanual,1993. [Mik95]E.Mikk.CompilationofZSpeciÞcationsintoCforAutomaticTestResultEvalu-ation.InProceedingsofthe9thInternationalConferenceofZUsers,volume967LectureNotesinComputerScience,pages167Ð180.Springer-Verlag,1995.[MMS97]J.McDonald,L.Murray,andP.Strooper.TranslatingObject-ZspeciÞcationstoobject-orientedtestoracles.InProceedings:4thAsia-PaciÞcSoftwareEngi-neeringandInternationalComputerScienceConference,pages414Ð426.IEEEComputerSocietyPress,1997.[MPS00]AtifM.Memon,MarthaE.Pollack,andMaryLouSoffa.AutomatedtestoraclesforGUIs.InProceedingsoftheACMSIGSOFT8thInternationalSymposiumontheFoundationsofSoftwareEngineering(FSE-00),volume25,6ofACMSoftwareEngineeringNotes,pages30Ð39.ACMPress,November2000.[MS96]D.R.MusserandA.Saini.STLTutorialandReferenceGuide:C++Program-mingwiththeStandardTemplateLibrary.Addison-Wesley,1996.[MS98]J.McDonaldandP.Strooper.TranslatingObject-ZspeciÞcationstopassivetestoracles.InProceedingsofthe2ndInternationalConferenceonFormalEngineer-ingMethods(ICFEM98),pages165Ð174.IEEEComputerSocietyPress,1998.[OKMM98]M.Obayashi,H.Kubota,S.P.McCarron,andL.Mallet.TheAssertionBasedTestingToolforOOP:ADL2.http://adl.xopen.org/exgr/icse/,May1998.[Pan78]D.J.Panzl.AutomaticSoftwareTestDrivers.,11(4):44Ð50,April1978.[Par93]DavidL.Parnas.PredicateLogicforSoftwareEngineering.IEEETransactionsonSoftwareEngineering,19(9):856Ð862,September1993.[PMI94]D.L.Parnas,J.Madey,andM.Iglewski.PreciseDocumentationofWell-StructuredPrograms.IEEETransactionsonSoftwareEngineering,20(12):948Ð976,December1994.[PP98]DennisK.PetersandDavidL.Parnas.UsingTestOraclesGeneratedfromPro-gramDocumentation.IEEETransactionsonSoftwareEngineering,24(3):161Ð173,1998.[Pro]ADLProject.Futureplans:Adl2.http://adl.xopen.org/future/[RLAO92]DebraJ.Richardson,StephanieLeif-Aha,andT.OwenOMalley.SpeciÞcation-basedTestOraclesforReactiveSystems.InProceedingsofthe14thInternationalConferenceonSoftwareEngineering,pages105Ð118,May1992.[Ros92]DavidS.Rosenblum.TowardsaMethodofProgrammingwithAssertions.Proceedingsofthe14thInternationalConferenceonSoftwareEngineeringpages92Ð104,May1992.[Ros95]DavidS.Rosenblum.APracticalApproachtoProgrammingWithAssertions.IEEETransactionsonSoftwareEngineering,21(1):19Ð31,January1995.[SC96]PaulStocksandDavidCarrington.AFrameworkforSpeciÞcation-BasedTest-IEEETransactionsonSoftwareEngineering,22(11):777Ð793,1996.[SH91]U.SchmidtandH.Horcher.TheVDMDomainCompiler:AVDMClassLibraryGenerator.InProceedingsofthe4thInternationalSymposiumofVDMEuropevolume551ofLectureNotesinComputerScience,pages675Ð687.Springer-Verlag,1991. [SH94]S.SankarandR.Hayes.ADL:AnInterfaceDeÞnitionLanguageforSpecifyingandTestingSoftware.ACMSIGPLANNotices,29(8):13Ð21,August1994.[Spi89]JohnM.Spivey.TheZNotation:AReferenceManual.Prentice-Hall,1989.[SRN85]S.Sankar,D.Rosenblum,andR.Neff.AnImplementationofAnna.InProceed-ingsoftheAdaInternationalConferenceonAdainUse,pages285Ð296.ACM,CambridgeUniversityPress,May1985.[Str86]BjorneStroustrup.TheC++ProgrammingLanguage.Addison-Wesley,1986.[Tay80]RichardN.Taylor.Assertionsinprogramminglanguages.ACMSIGPLANNo-,15(1):105Ð114,1980.[Tay83]RichardN.Taylor.AnintegratedveriÞcationandtestingenvironment.SoftwareÑPracticeandExperience,13:697Ð713,1983.[vBDZ89]G.v.Bochman,R.Dssouli,andJ.R.Zhao.Traceanalysisforconformanceandarbitrationtesting.IEEETransactionsonSoftwareEngineering,15(11):1347Ð1356,November1989.[vBP94]G.v.BochmannandA.Petrenko.ProtocolTesting:ReviewofMethodsandRel-evanceforSoftwareTesting.TechnicalReportIRO-923,Dept.dÕInformatiqueetdeRechercheOprationnelle,UniversitdeMontral,1994.[VS96]S.R.ViswanadhaandS.Sankar.PreliminaryDesignofADL/C++ÑASpec-iÞcationLangaugeforC++.InProceedingsofthe2ndConferenceonObject-OrientedTechnologiesandSystems(COOTS),Toronto,Canada,June1996.[VW94]MosheVardyandPierreWolper.ReasoningaboutinÞnitecomputations.Infor-mationandComputation,115:1Ð37,1994.[Wey82]ElaineJ.Weyuker.Ontestingnon-testableprograms.CompuerJournal,25:465Ð470,1982.[Wol85]PierreWolper.Thetableaumethodfortemporallogic:Anoverview.Logiqueet,110-111:119Ð136,1985.[ZJ96]PamelaZaveandMichaelJackson.WhereDoOperationsComeFrom?AMulti-paradigmSpeciÞcationTechnique.IEEETransactionsonSoftwareEngineering22(7):508Ð528,1996.