APracticalApproachtoMeasuringandImprovingtheFunctionalVericationofEmb - PDF document

APracticalApproachtoMeasuringandImprovingtheFunctionalVericationofEmbeddedSoftwareStéphaneBouvier,NicolasSauzèdeSTMicroelectronics,HomeEntertainment&Displays12rueJulesHorowitz,Grenoble,Francestephane.bouvier,nicolas.sauzede@st.comFlorianLetombe,JulienTorrèsSpringSoft,LogicVericationGroup6placeRobertSchuman,Grenoble,France°orian letombe,julien torres@springsoft.comABSTRACTWeproposeinthispapertoapplyfunctionalquali¯cation-basedonthetheoryofmutationanalysis-to¯rmwareco-veri¯cationenvironments,byintegratingtheGNUProjectDebugger()remoteserialprotocol(RSP)withthefunc-tionalquali¯cationengine.Morespeci¯cally,theCertitudefunctionalquali¯cationtoolisappliedtotheveri¯cationofanSTMicroelectronicsvideoIP.Inthissystem,thehard-warepartisgeneratedbyHighLevelSynthesis°owtoolsmainlyimplementingthedata-°owpart,whilethe¯rmwarepart,writteninC,isrunningonanSTMicroelectronicsem-beddedprocessormodeldevotedtothecontrolpart.Bothhardwareand¯rmwarepartsaresimulatedonaTransactionLevelModeling(TLM)platformwithRSPaccess,modelingavirtualsystem-on-chip.Inthisparticularcontext,theembeddedsoftwareisonlyac-cessiblethroughtheRSP.Wedescribehowourfunctionalquali¯cationtoolisextendedtoworkinthisenvironment.Wepresenthowtheuseofthistechniquealloweddeadcodeinthe¯rmwaretobeidenti¯edandtopointoutcriticalweaknessesintheveri¯cationenvironment.Weshowthat¯xingtheseissuesledtoreducingthememoryconsumptionofthe¯rmwareandto¯ndingcriticalbugsinthehard-ware/¯rmware.ThestandardnatureoftheRSPmeansthatthiscouldbeaglobaltechniquetoapplymutationanalysistoembeddedsoftware.CategoriesandSubjectDescriptorsB.1.4[ControlStructuresandMicroprogramming]:MicroprogramDesignAids|Firmwareengineering,Veri-¯cation;I.6.4[ComputingMethodologies]:SimulationandModeling|ModelValidationandAnalysisGeneralTermsVeri¯cation,Design,ExperimentationKeywordsFunctionalQuali¯cation;Co-veri¯cation;MutationAnaly-sis;TLM/SystemCVirtualSoC;EmbeddedSoftware;remoteserialprotocol.1.INTRODUCTIONWiththeincreasingcomplexityandreducedtime-to-marketoftoday'sembeddedsystemsthereisatrendtowardsdevel-opingmoreofthesysteminsoftware[1].Debuggingembed-dedsoftwarecanbetime-consumingand¯ndingbugslateinthedevelopmentcyclecanbeverycostly.Veri¯cationofthesoftwareandespeciallythehardware-dependentsoft-ware(¯rmware)canhelpto¯ndhardwarebugsandavoidre-spins[2].Asaresultitisnecessarytodohardware/softwareco-veri¯cationforthesesystemsto¯ndand¯xproblemsasearlyaspossible.Measuringhowwellthesesystemsareveri-¯edisbecomingmoreimportantsinceitcanhelptoestimateandlimittheriskof¯ndingcostlybugs.Forthepurposeofmeasuringveri¯cation,codecoveragetoolsexist,butcanbeshowntobeinsu±cientinthequali¯-cationofatestingenvironment[2].Mutationanalysisbasedonfaultinjection[3]isbroadlyacceptedasastrongermet-ricformeasuringthequalityofveri¯cation,andhasbeenbroadlyadoptedforHardwareDescriptionLanguages.Un-fortunately,acodecoveragetooloramutationanalysisen-gineisnotnecessarilywell-suitedtotheveri¯cationofthe¯rmwarebecauseofitsembeddednatureandthedi±cultytocommunicatewiththesetools.Thispaperpresentsaninnovativemethodtoperformem-beddedsoftwarefunctionalquali¯cation.Itshowshowamutationanalysisenginehasbeenextendedtobeabletocommunicatewiththe¯rmware.Thepaperisorganizedasfollows.Section2presentsgeneralconceptsaboutdevelopinganddebuggingembeddedsoft-ware.Section3de¯nescodeandfunctionalcoverageandsummarizesthemainmutationanalysisandfunctionalqual-i¯cationaspects.Section4presentsfunctionalquali¯cationappliedtoembeddedsoftware.Section5showsthee®ective-nessoftheproposedmethodologyinmeasuringthequalityofembeddedsoftwareveri¯cationenvironmentsofanindus-trialplatform.Finally,thepaperconcludesandopensonfutureapplicationsinSection6.2.DEVELOPINGANDDEBUGGINGEM-BEDDEDSOFTWAREWITHGDBInthissection,aquickoverviewofembeddedsoftwarede-velopmentanddebuggingisproposed,focusingontheGNUDebuggertool.Embeddedsoftwaredevelopmentisusuallybasedonahost-targetapproach:theembeddedsoftwareisdevelopedonahostmachine,thentransferredtothetargetmachinefortestanddebugpurposes.Fortheremainderofthispaper,theembeddedsoftwareiscalledthetarget,thesystemitislaunchedon(namelytheplatform)isthetargetmachine, andthemachinefromwhichtheplatformislaunchedisthehostmachine.Inthecontextofembeddedsoftwaredevel-opment,debuggingisacrucialaspect.S.SchneiderandL.Fraleigh[4]claimthat80%ofdevelopmente®ortisspentondebugging.TheGNUDebugger,orforshort[5],isthestandarddebuggerfortheGNUsoftwaresystem.Itisaportablede-buggerthatrunsonmanyUnixsystemsandworksformanyprogramminglanguages,includingC.providesextensivefeaturesfortracing,monitoring,andmodifyingtheexecu-tionofprograms.Forexample,theinternalvariablesoftheprogramcanbecheckedandmodi¯ed,theexecutionoftheprogramcanbeinterruptedviabreakpoints,etc.Forthepurposeofembeddedsoftwaredebugging,theGNUsoftwaresystemprovidesatoolcalledthatal-lowstosupportremotedebugging.runsonthehostmachinewhileisexecutedonthetargetma-chine.Theexecutionoftheprogramcanbecontroledfromcommandsexactlyasifithadbeenrundirectlyonthehostmachine.andcommunicateviaeitheraseriallinkoraTCPconnection,usingthestandardremoteserialprotocol(RSP).NotethatonlyTCPconnec-tionsareconsideredinthispaper.Theprotocolispubliclyavailable[6].Thisinformationisessentialfortheportabilityofthetechniques.3.TESTMETHODSSeveralcoveragetechniquesareconsideredinthissection,andade¯nitionoffunctionalquali¯cation,alongwithitsdi®erenceswithmutationanalysisareproposed.Finally,theCertitudetoolispresented.3.1CodeCoverageandFunctionalCoverageSoftwareveri¯cationprimarilyutilizescodecoveragetocheckiftheveri¯cationiscomplete.ToolslikeGNUgcovandoth-ersprovidestatementcoverage,callcoverage,andbranchcoverage.Variouscompanieshavestandardsforcodecover-agethatmustbemetbeforesoftwareisshipped.Themainadvantageofcodecoverageissimplicity,andresultscanbegeneratedwithlittlee®ort.However,codecoveragetoolsrequiresupportbythetargetenvironmentwhereembeddedsoftwareisveri¯ed.Functionalcoverageisadi®erenttechniquefromcodecover-age,andismoregeneral[7].Basically,functionalcoverageisthedeterminationofhowmuchfunctionalityofthedesignhasbeenexercisedbytheveri¯cationenvironment.Thistechniqueistypicallyusedwithpseudo-randomizedtestcasegeneration[8].Ifpseudo-randomizationisbeingused,func-tionalcoverageprovidesane®ectivefeedbackmechanismfortestscenariosastheyaredeveloped[9].Again,thetargetenvironmentneedstobeadaptedtosup-portfunctionalcoveragetoolssuchasFoCuSfromIBM[10],BullsEye[11],orComet[12].3.2FunctionalQualication3.2.1MutationAnalysisvs.FunctionalQualica-tionThemainweaknessofcoveragemetricsisthattheydonotconsiderthecheckingofoutputbehaviorofthedesignunderveri¯cation(DUV).Indeed,itispossibleforthesemetricstogivehighscoreseveniftheoutputbehavioroftheDUVisnotcompletelychecked.Toaddressthoseproblems,muta-tionanalysisandmutationtesting[13]havegainedpopular-ityinrecentdecades[14].Suchtestingapproachesrelyonthecreationofseveralversionsoftheprogramtobetested,\mutated"byintroducingsyntacticallycorrectfunctionalchanges.Thesemutatedversionsoftheprogramarecalled\mutants".Thepurposeofsuchmutationsistochangetheprogramtocheckifthetestsuiteisabletodetectthebe-havioraldi®erencebetweentheoriginalprogramandthemutatedversions.Morespeci¯cally,theoutputoftheDUViscomparedwithandwithoutthemutation[15].Ifthereisadi®erenceobservedintheoutputthenthemutantisconsideredtohavebeen\killed"[3].Thee®ectivenessofthetestsuiteisthenmeasuredbycomputingthepercentageofdetectedmutations.Similarconceptsarealsoappliedinhardwaretestingtoprovidemoree®ectivetestsuitesfortheDUV:veri¯cationengineersusehigh-levelfaultsimulationtomeasurethequalityoftestbenches[16],andtestpatterngenerationtoimprovefaultcoverage.Inthiscase,muta-tionsintroducedinthehardwaredescriptionsarereferredto\faults"[16].Functionalquali¯cationperformedbytheCertitudetool(¯rstintroducedin2009[17])anddiscussedinthispaperisdi®erent.Amutantisconsideredtohavebeenkilledwhenatestcasefails.Asintraditionalmutationanalysis,outputsofthedesignarestillmonitored.IfallthetestcasespassonamutatedversionoftheDUV,andadi®erenceisobservedintheoutput,thismeansthatcheckersaremissingintheveri¯cationenvironment.Functionalquali¯cationhighlightsthesemissingchecks.Suchcheckscanincludethecompari-sonofexpectedoutputbehaviorandassertionsmonitoringtheprogram'sinternalorexternalbehavior.Theabilityoftheveri¯cationenvironmenttodetectpotentialbugsisbeingmeasuredwhereasintraditionalmutationanalysisonlytheabilityoftheinputsequencestopropagatepotentialbugstooutputsismeasured.Thetermfunctionalquali¯cationhasthusbeenintroducedtocapturethisconceptofmeasuringthebugdetectionability.Veri¯cationisrequiredtoensurethequalityofthedesigncodeandthisactivityoftenconsumesaround70%ofthetotaldesignresources[18].Alargeamountofcodemustalsobecreatedtoimplementtheveri¯cationenvironmentanderrorsmayoccurintheimplementation.Errorsintheveri¯cationenvironmentcanresultinoneofthreesituations:Thetestcasefails:inthissimplestcase,theerrorintheveri¯cationcanbefound,assumingthatthedesigniscorrect;Thetestcasepasses:inthiscasethetestcasemayhidearealdesignbug;Thetestcaseismissing:typicallyduetoamistakeinthetestplan.Functionalquali¯cationisauniquetechnologythatidenti-¯espassingtestcasesaspotentiallyhidingrealdesignbugs. Itcanalsoidentifyawiderangeofmissingtestcasesthatothertechniquescannot.Forexample,complextemporalsequencesmaybemissing,preventingthee®ectsofhiddenbugsfrompropagatingtooutputswheretheycanbede-tected.Tobee®ective,functionalveri¯cationmustensurethattheDUVareshippedwithoutcriticalbugs.To¯ndadesignbug,threethingsmustoccurduringtheexecutionoftheveri¯cationenvironment:1.Thebugmustbeactivated;i.e.thecodecontainingthebugisexercised.2.Thebugmustbepropagatedtoanobservablepoint;e.g.theoutputsofthedesign.3.Thebugmustbedetected;i.e.behaviorischeckedandafailureindicated.TraditionalEDAtechnologieshavefocusedonitem1,ac-tivatingthebug.Techniquessuchascodecoverageandfunctionalcoveragecanhelpensurethatdesigncodeiswell-activated.Buttheycanneitherguaranteethatdesignbugswillbepropagated,northatthebugswillbedetectedbythecheckers,suchasassertionsorcomparisonagainstaref-erencemodel.3.2.2Certitude:afunctionalqualicationtoolCertitudeisafunctionalquali¯cationtoolcommercializedbySpringSoft[19].Itprovidesvariousfront-endsatdi®er-entlevelsofabstraction,includingtheVHDL,Verilog,andClanguages.Itsoveralloperationissummarizedinthissection.Certitudeautomaticallyinsertsbugs(alsocalledfaults)intothehardwareorsoftwaremodels.Thefaultmodelcontainsvarioustypesoffaultssuchasoperatorchanges,deadassign-ments,forcedconditions,etc.Asanexample,thefollowingoriginalcode=j;generatestwodi®erentkindsofmutations:=&;;ThenCertitudedetermineswhethertheveri¯cationenviron-mentcanactivatethefaultycode,propagatethee®ectstoanobservablepoint,anddetectthepresenceofthefault.Aknownfaultthatcannotbedetectedpointstoaveri¯cationweakness.Ifafaultcannotbedetected,thereisevidencethatactualdesignbugswouldalsonotbedetectedbytheco-veri¯cationenvironment.Certitudeoperationfallsintothreephases:1.Themodelanalysisphaseanalyzesthedesignandgen-eratesamodi¯edsourcecodewithfaultsinjected(in-strumentedcode);2.Theactivationphaserunsacompleteregressionandanalyzesthebehavioroftheveri¯cationenvironmentwithrespecttothefaults;3.Thedetectionphaserunsselectedtestsfromtheveri-¯cationenvironmenttomeasuretheabilityofthever-i¯cationenvironmenttodetectthefaults.Attheendofthequali¯cation,faultsareclassi¯edwiththefollowingstatuses:non-activated:thefaulthasnotbeenexercisedbythetestsuite;non-propagated:thefaulthasbeenactivated,butdidnotimpacttheoutputsofthedesign;non-detected:thefaulthasbeenpropagated,butcheckersdidnotnoticethebehaviorchange;detected:thefaulthasbeenpropagated,andatleastonecheckernoticedabehaviorchange;disabledbyuser:theveri¯erdecidedthatpartofthefunc-tionalitydidnotneedtobeveri¯ed.Subsequently,Certitudeprovidesuserswithacompletere-portoftheresultsinHTMLformatthathighlightstheprob-lemareas.Thisisusedtoexposeshortcomingsandguideimprovementsintheenvironmenttoensurethatbugsdonotslipthroughtheprocess.WhenintegratingCertitudeintoaprojectenvironment,itisimportanttounderstandthatitworksontopofthesim-ulationframework,andcanmakeuseofabatchsystem.Certitudeisapointtoolthatdoesnotrequirechangestotheprojectenvironmentitself.Onlyminormodi¯cationstosomescriptsmaybenecessary.ToadaptCertitudetotheveri¯cationenvironment,itneedstohavethefollowingin-formationandcontrol:alistofallsoftwaremodels¯lesthatmakeupthesystem,theabilitytorecompilethe(instru-mented)sourcecode,alistoftestcasenames,ascriptthatcanexecuteatestcaseandreturnapassorfailresult.4.APPLYINGFUNCTIONALQUALIFICA-TIONTOEMBEDDEDSOFTWARE:APRACTICALCASEThisSectiondescribestheRSP-enabledTransactionLevelModeling(TLM)developmentplatform,theCertitudetoolandtheproposedapproachtofunctionalquali¯cationofem-beddedsoftware.4.1TheRSP-enabledTLMdevelopmentplat-formThemainpurposeofaTLMplatformistoraisetheab-stractionlevelofatypicalSystemCplatform{whichworksatthesignal-level{tothetransaction-level.Thegoalistoabstracttheimplementationdetailsoftheinterconnectbymanipulatingabstractdatastructuresthatrepresentthepayload.ThepowerofTLMisthatitallowstheabstractionleveltobeadjustedbyprovidingthe°exibilitytochoosethedatagranularity:forexample,betweenasimpledatawordofthesizeofthebus,toavideomacro-blockorevenafullimage. TheTLMdevelopmentplatformusedforthisexperimentisasetofinterconnectedSystemCcomponentssuchasmem-ories,buscontrollers,CPUs,etc.ItcanbeusedtomodelaVirtualSystem-on-a-ChipthatsimulatesatthesametimethehardwareIntellectualProperty(IP)andtheembeddedsoftwarerunningonit.ATLMplatformcanembedasmanyCPU"cores"asneededbyinstantiatingthecorrespondingnumberofTLMproces-sormodels.TheTLMprocessormodelsareSystemCcom-ponentsthatprovideaparticularCPUkind(e.g.:STxP70,ST40,ARMCA9,PowerPC,etc.).TheplatformweusedintegratesanSTxP70(STMicroelec-tronicsproprietarymicro-controller)processormodelthatusesatechnologythatservesasahelpertoacceleratetheembeddedsoftwareexecution.ItconsistsofcompilingtherealembeddedsoftwaresourcecodeandexecutingitwithinthecontextoftherunningTLMsimulation,i.e.withadi-rectaccesstothesimulatedhardwareresources,suchasIPregisters,memories,interrupts,etc.AveryimportantfeatureoftheprocessormodeltechnologyusedintheTLMplatform,isthatitpresentstothe"outside"astandardRSPconnectionfeature,toletastandarddebugclientmanagetheembeddedsoftwareexecution.Thisisthekeypointthatenablesthenovelapproachdescribedinthispapertoactuallyperformthefunctionalveri¯cationoftheembeddedsoftware.4.2UsingCertitudeonembeddedsoftwareSTMicroelectronicshasbeenusingCertitudesinceitsmar-ketintroduction[20],andtodayrunsfunctionalquali¯cationonmorethan80percentofitsinternalIPdesigns.Inthiscontextofhistoricalcollaboration,theCertitudeteamde-velopscustomer-speci¯cfeaturesondemandwhenperspec-tivesofsuchfeaturesforfuturedeploymentarepromising.SincetheCertitudetooliswidelyusedwithinthecompanyforRTLandCdesigns,STMicroelectronicsnaturallyaskedtheCertitudedevelopmentteamtoadaptthetoolfortheirembeddedsoftwarerequirements.Inthecurrentcon¯guration,Certitudeinjectsfaultsinthedesignandcommunicateswiththesimulationviacontrol¯les.Allfaultsareinjectedtogetherinthedesigntoavoidre-compilations,andcontrol¯lesareusedtosetthesimula-tionparameters:faulttoexercise,resultofthesimulation,etc.,asdescribedin[21].Thismethodcan'tbedirectlyap-pliedtoembeddedsoftwaretestingenvironmentssincethetargetmachinedoesnotnecessarilyhavea¯lesystem,andthecodeinjectedintothesoftwarecontrolsitsbehavior,andrequiresseveralLinuxsystemcallsalsonotlikelytobesup-portedbythetargetsystem.Asaresult,usingCertitudewithitsdefaultbehaviorwouldresultincompilationerrorsandaninabilitytocommunicatebetweenthetoolandthetarget.AtraditionaluseofaTLMplatformthatembedsaprocessormodelwhicho®ersastandardRSPinterfaceistoconnectaclienttothetarget¯rmwarethatrunsinsidetheplat-form.Here,forthepurposeoffunctionalquali¯cation,wereusedthisstandardRSPinterfacetoconnecttheCertitudetooldirectlytothe¯rmwareexecution.Theframeworkhas Figure1:FrameworkSetupbeensetupasshowninFigure1.Thenewusagepathisdrawnwithdottedlines.TheCertitudetoolbehavesasastandardclient,bene¯t-ingfromthesamedebuggingfunctionality,suchas:control-lingexecution(stopandresume),inspecting/alteringmem-orycontents,settingbreakpoints,etc.Asdescribedinsec-tion2,theRSPisgenerallyusedfordebuggingpurposes,andtothebestofourknowledge,ithasneverbeenusedbeforeinthecontextofmutationanalysis.Asstatedabove,quali¯cationofembeddedsoftwaregivesrisetotwomainissues;namely(1)communication,and(2)compilation.The¯rstissueissolvedbythemechanismdescribedinFig-ure1.ThefaulttobeexercisedforasimulationcanbesetbysendingtheinformationviatheRSPinterface,andthesimulationresultcanbecheckedthesameway.Thesecondissueistackledbysimplifyingtheinstrumenta-tionofthedesign.Onlystandardvariabletypesareused,andnosystemcallisinvoked,noristhereany¯leaccessfromtheinstrumentedcode.Thisensuresfullcompatibilitywithanytargetsystem.Finally,a¯nitestatemachinehasbeensetuptowraptheplatformexecutionandcontrolthesimulationstatus.Itsop-erationconsistsof:launchingtheplatformandestablishingcommunication,parsingthetargetExecutableandLinkableFormat(ELF)code,settingthefaulttobeexercised(forthedetectionphase),authorizethetargettobelaunched,checkitsbehavior,getresultsandclosecommunication.Thiscom-plexmechanismguaranteescorrectexecutionandtermina-tionofsimulations.5.EXPERIMENTSThissectiondescribesthemachinesandtheusecasecon-sideredforexperiments.Wegivedetailsofexperimentalresults,alongwithimprovementscarriedoutontheconsid-eredIP.5.1MethodologyTheexperimenttofunctionallyqualifytheembeddedsoft-waretestingenvironmenthasbeendoneonaHighQual-ityVideoDisplayIPusedintheset-top-boxes.ThisIP getssomevideostreamsfrompreviousdesignstages(suchasH264videodecoder)andsendsthemtothenextstages(picturecomposition).Theinputstreamsareprocessedus-ingde-interlacing,rescale,andImageQualityImprovementsalgorithms.TheHighQualityVideoDisplayIPusedismadeoftwoparts.First,ahardwarepartisgeneratedbyanHLS(HighLevelSynthesis)°owtoolmainlyimplementingthedata-°owpart;secondasoftwarepartrunningonanSTxP70coreisdevotedtothecontrolpart.Thiscodeisknownastheembeddedsoftwarethatwewanttoinstrumentandcoverinthispaper.Thisembeddedcodeismainlyacontrolpartofthehardwaredesign.Itdealswiththestartconditions,readsacommandinmemoryandprocessesit.Itthencon¯gurestheIPin-ternalregistersandcadencesthehardwareblockinputsandoutputs.Thisleadstohighnumberofinteractionswiththehardware.Usually,thehardestpartstoverifyinadesignarethestatemachines.Accordingtothisnewarchitecture,thesestatemachinesaremainlylocatedintheembeddedsoftware.Hav-inggoodcoverageontheembeddedsoftwarethatimple-mentsasigni¯cantpartofthecontroliskeyto¯ndingveri¯-cationholesthatmayhidesomehardwarebugs.Beforethisquali¯cationbyerrorinjectionintheembeddedsoftware,therewasalmostnocoverageanalysisperformedonthesesoftwarecontrolstatemachines.Indeed,theonlypossibilitywehadwastotracetheprogramcounteroftheSTxP70coreandanalyzetheneverreachedvalues.Theplatformenvironmentusedtoperformthefunctionalveri¯cationisaSystemCTLMvirtualplatformwherebothCPU's(hostandembeddedmicro-controller)arereplacedbyaprocessormodelallowingexecutionofthesoftwareundertestonaLinuxPC.ThetestsuiteusedfortheHighQualityVideoDisplayIPismadeupof10,434testcases.ThistestsuiterunsinTLMinabout4hoursusing200LinuxCPUs,andhasbeenusedwiththeCertitudetool.Theempiricalresultspresentedinthissectionwereobtainedonthe200LinuxCPUs.ThehardwarehasbeenmodeledinSystemCTLMtoo.TheerrorinjectionusingCertitudehasalreadybeenmadeonthisSystemCTLMmodel.Thecurrentexperimentisex-pandingtheconcepttotheembeddedsoftware.5.2ResultsThisembeddedsoftwareismadeupof27¯les.TheCerti-tudemodelphaseinjected6781faults.Wedisabled592oftheinjectedfaults.TheTLMplatformwhichusesaprocessormodeltoemulatetheSTxP70micro-controllerisapureSystemCplatformwithuntimedmodels.AstheIPisdesignedtocopewithreal-timeconstraints,theembeddedsoftwarecontainssomededicatedcodetodealwithreal-timeconstraints.AstheSystemCplatformisanuntimedexecutionone,thisplatformcan'texercisereal-timerelatedmechanisms.Toreachthem,weneedatimedviewofthehardware.Thisisdonelaterintheveri¯cationprocess.Forthisexperiment,wedisableittoavoiduninterestingnon-detectedfaults.TheCertitudeactivationphasehasbeenrunusingthiscom-pletetestsuite.Thisleadstothe¯rstresultsinaboutonenight:59faultsnon-activatedand116non-propagated.This59non-activatedfaultsresultisequivalenttolinecoverage:suchfaultsrepresentlinesofcodeneverreachedbyanyofthetests.Thenon-propagatedfaultsresultgivesusaddi-tionalinformation.Forexample,detailsareprovidedonthe"if"statementspartiallyreached(\if"conditionsalwaystrueoralwaysfalse)andonthewritestovariablesthatdon'tchangetheirvalues.RunningafulldetectionphaseoftheCertitudetoolwiththisfulltestsuitewouldbetoolong.Foreachexercisedfault,thetoolwilllauncheverytestthatactivatesthisfaultuntilitdetectsitoralltestshavebeenrun.Intheworstcase,thismayleadtoabout6000timestheregressionwhichisbeyondthetimeavailablefortheproject.Togetanoverallideaofthedetectionresults,we¯rstranastatisticalmetricprovidedbytheCertitudetool.Itran-domlyselectsasampleoffaultsandrunsrandomlychosentestsoneachfaultofthesample.Thisresultsinanestima-tion(withamarginoferror)ofthenumberofdetectedandnon-detectedfaultsinamuchshortertimethanafullde-tection.Afterrunningforonenight,thisstatisticalmetricgaveanestimatednumberofdetectedfaultsofabout5200or,putanotherway,about700non-detectedfaults.Thestatisticalmetriconlygivesanestimateofthenumberofnon-detectedfaults,butitcannot¯ndwhichfaultsarenotdetected.Tooptimizetheruntimeofthedetectionphase,withSpringsoftsupport,weidenti¯edasmallsub-setoftestsbasedontheactivationresults.Inourcase,thissub-setoftestsonlycontains40testcases.Wethusranthedetectionphasewiththisshorttestsuiteandthengota¯rstresultof713non-detectedfaultsinaboutonenight.Thisresultcanbeconsidered"pessimistic"becauseanon-detectedfaultmayhavebeendetectedbyanothertestcasethatisnotintheshorttestsuite.Consequently,weneededabouttwoadditionalweekstoextendthesub-setoftestcasesandthusdetectsomeoftheseremainingfaults.Addingthesetestsreducedthenumberofnon-detectedfaultsto694.Tosummarize,afterallthesesteps,weobtainedthefollow-ingresultstatus:59faultsnon-activated(0.9%),116faultsnon-propagated(1.7%),694faultsnon-detected(10.2%),5320faultsdetected(78.5%),and592faultsdisabledbytheveri¯er(8.7%).Atthisstage,westartedtoanalyzewhythesefaultswerenotdetected,keepinginmindthatwehavenotrunallthetestsonthenon-detectedfaults.SoweanalyzedtheCertitudereportfromtheactivationanddetectionphases.Activation ¯rstshowedthattherewassomedeadcodethatcannotbeactivated.Thiscodehasbeenremovedanditsavedabout2%ofroomintheprogrammemory.Thisisaverygoodresultsincetheembeddedcontrolcodeneedstobeoptimizedto¯tonsmallmemories.Activationshowedusthatsometestsweremissing.Wethenwrote4additionalteststocoversomelines.Thesetestsaremainlycheckingfunctionalitythatwasmissedbytheprevioustestsuite.Oneofthesetestsshowedahardwarebuginacornercaseandavoidedare-spinofthesilicon.ThishardwarebugconcernedonemodeofourHighQualityVideoDisplayIPcalledpanoramicmodeand,morepre-cisely,onespeci¯coptionofthismode.The¯rstspeci¯ca-tionsoftheIPdidnotmentionthispossibilitywhichap-pearedlater.Unfortunately,ithadneverbeenaddedtothetestplan.Thetestcasegeneratorwasabletogeneratethiscasebutnoneofthetestswereusingthispossibility.Theveri¯cationholewasfoundanalyzingafaultinan\else"branchofan\if".Whenthenewtestsimplementingthisfeaturewerewritten,wefoundthatthedevelopmentver-sionoftheRTLpartofthedesignbehavedincorrectly.Detectionalsohighlightedmissingtests.Wethenwrote6newteststodetectthenon-detectedfaults.Thesetestsdidnotshownewbugs.Thetestswrittenafterthedetec-tionphasearemainlycheckinginitializationoftheembed-dedsoftwarebetweentwocommandsinsomecornercases.ThankstothesenewteststhefunctionalityoftheIPisbet-terveri¯edandsofarnobugshavebeenfoundinthepro-ductionversionoftheIP.6.CONCLUSIONANDFUTUREWORKInthispaper,wehavepresentedaframeworkforthefunc-tionalquali¯cationofembeddedsoftwaretestingenviron-ments,andshoweditsapplicationonanindustrialdesign.Nowadays,(i)thecloseintegrationbetweenhardwareandsoftwarepartsinmodernembeddedsystems,(ii)thede-velopmentofhigh-levellanguagessuitedformodelinghard-wareandsoftware,(iii)theneedfordevelopingveri¯cationstrategiestobeappliedearlyinthedesign°ow,requirethede¯nitionofmutationanalysis-basedstrategiesthatworkatsystemlevel,wherehardwareandsoftwarefunctionalityarenotcompletelyindependentandseparate.WealreadyknewthatCertitudewasaverye±cienttooltoqualifyveri¯cationenvironments(thistoolisalreadyde-ployedonhardwareveri¯cationprojects)whenrunningonRTLorontheCstandalonemodelsusedintheHLSdesign°ow.Extendingtheusageofthistooltohardwarecontrolimplementedbysoftwareisamusttoavoidcriticalhardwarebugs.WehaverunCertitudeontheembeddedsoftwareofourHighQualityVideoDisplayIP.Ithasallowedustoremovesomedeadcodeandtoaddsomemissingtests.Someofthemhaveuncoveredahardwarebugandpreventedare-spinofthesilicon.IthenceallowedSTMicroelectronicstokeepdeliveringquality-productstothemarket.ExperimentsfocusedontheGNUremoteserialproto-col,foritssimplicityandthefactthatitisaproven,de-factoindustrystandard.However,theprincipledescribedinthispapercouldinfactbedeployedwithanyembeddedsystemincludinganexternaldebuginterface(forexampleARMCycleAccurateDebugInterface(CADI)[22],Lauter-bachTRACE32[23],PowerStandardforCommonDebugInterface(CDI)[24],etc.),providedthatthedebugproto-colisstandard/open.Theonlyrequirementsaretobeablesomehowtomanagetheembeddedsoftwareexecution,andaccessthetargetmemory,registers,andbreakpoints.ThisisfutureworktoextendtheCertitudetoolinthisdirection.7.REFERENCES[1]E.A.Lee,\Embeddedsoftware,"AdvancesinComputers,vol.56,pp.56{97,2002.[2]G.D.Guglielmo,F.Fummi,G.Pravadelli,M.Hampton,andF.Letombe,\Onthefunctionalquali¯cationofaplatformmodel,"inProc.ofthe24thIEEEInternationalSymposiumonDefectandFaultToleranceinVLSISystems,2009,pp.182{190.[3]A.J.O®utt,\Apracticalsystemformutationtesting:Helpforthecommonprogrammer,"inProcofIEEEInternationalTestConference,1994,pp.824{830.[4]S.SchneiderandL.Fraleigh,\Thetensecretsofembeddeddebugging,"EETimesDesign,2004,http://www.embedded.com/showArticle.jhtml?articleID=47208538.[5]\GDB:TheGNUProjectDebugger,"http://www.gnu.org/software/gdb.[6]GNURemoteSerialProtocol,http://sourceware.org/gdb/current/onlinedocs/gdb/Remote-Protocol.html.[7]G.J.Myers,TheArtofSoftwareTesting.Wiley-Interscience,1999.[8]P.MishraandN.Dutt,\Functionalcoveragedriventestgenerationforvalidationofpipelinedprocessors,"inProc.oftheconferenceonDesign,AutomationandTestinEurope,2005,pp.678{683.[9]M.Hampton,Functionalquali¯cation:atechnicalbrief,2009,http://www.edadesignline.com/215600203.[10]Focus,http://www.alphaworks.ibm.com.[11]BullsEye,http://www.bullseye.com.[12]R.Grinwald,E.Harel,M.Orgad,S.Ur,andA.Ziv,\Userde¯nedcoverage-atoolsupportedmethodologyfordesignveri¯cation,"inProc.ofthe35thConferenceonDesignAutomation(DAC'98),1998,pp.158{163.[13]R.A.DeMillo,R.J.Lipton,andF.G.Sayward,\Hintsontestdataselection:Helpforthepracticingprogrammer,"IEEEcomputer,vol.11,no.4,pp.34{41,1978.[14]D.HyunsookandG.Rothermel,\OntheUseofMutationFaultsinEmpiricalAssessmentsofTestCasePrioritizationTechniques,"IEEETransactiononSoftwareEngineering,vol.32,no.9,pp.733{752,2006.[15]R.Guderlei,R.Just,C.Schneckenburger,andF.Schweiggert,\BenchmarkingTestingStrategieswithToolsfromMutationAnalysis,"inIEEEInternationalConferenceonSoftwareTestingVeri¯cationandValidationWorkshop,2008,pp.360{364.[16]M.Abramovici,M.Breuer,andA.Friedman,DigitalSystemsTestingandTestableDesign.NewYork:ComputerSciencePress,1990.[17]N.Bombieri,F.Fummi,G.Pravadelli,M.Hampton, andF.Letombe,\FunctionalQuali¯cationofTLMVeri¯cation,"inProc.oftheconferenceonDesign,AutomationandTestinEurope,2009,pp.190{195.[18]J.Bergeron,WritingTestbenchs:FunctionalVeri¯cationofHDLModels.KluwerAcademic,2000.[19]CertitudefromSpringSoft,http://www.springsoft.com/products/functional-quali¯cation/certitude.[20]O.Haller,\DeployingFunctionalQuali¯cationatSTMicroelectronics,Methodologies&CaseStudies,"IP&Design,FunctionalVeri¯cationGroupoftheConferenceonDesignAutomation,2008.[21]M.Hampton,\Proc¶ed¶eetSystµemed'EvaluationdeTestsd'unProgrammed'OrdinateurparAnalysedeMutations,"FrenchPatentFR2873832forCertessSARL,2006.[22]CADI,http://infocenter.arm.com/help/topic/com.arm.doc.dui0444d/index.html.[23]TRACE32,http://www.lauterbach.com/tutorial.pdf.[24]CDI,https://www.power.org/resources/downloads.