Model-BasedTestsofTruismsTimMenziesLaneDept.ofCom.Sci.UniversityofWest - PDF document

Model-BasedTestsofTruismsTimMenziesLaneDept.ofCom.Sci.UniversityofWestVirginiaPOBox6109,MorgantownWV,26506-6109,USA;tim@menzies.comDavidRaffo,Siri-onSetamanitPortlandStateUniversitySchoolofBusinessPOBox751PortlandOR97207USAdavid@sba.pdx.edussetaman@yahoo.comYingHu,SinaTootoonianDeptElec.&Comp.Eng.Vancouver,BritishColumbiaCanadaV6T1Z4huying_ca@yahoo.comachillesofpersis@hotmail.comAbstractSoftwareengineering(SE)truismscapturebroadly-applicableprinciplesofsoftwareconstruction.Thetroublewithtruismsisthatsuchgeneralprinciplesmaynotapplyinspeciccases.ThispaperteststhespecicityoftwoSEtruisms:(a)increasingsoftwareprocesslevelisadesirablegoal;and(b)itisbesttoremoveerrorsduringtheearlypartsofasoftwarelifecycle.Ourtestsarebasedontwowell-establishedSEmodels:(1)Boehmet.al.'sCOCOMOIIcostestimationmodel;and(2)Raffo'sdiscreteeventsoftwareprocessmodelofasoft-wareprojectlifecycle.Afterextensivesimulationsofthesemodels,theTAR2treatmentlearnerwasappliedtondthemodelparametersthatmostimprovedthepotentialperfor-manceofthereal-worldsystemsbeingmodelled.Thecasestudiespresentedhereshowedthatthesetru-ismsareclearlysub-optimalforcertainprojectssinceotherfactorsprovedtobefarmorecritical.Hence,weadviseagainsttruism-basedprocessimprovement.Thispaperof-fersageneralalternativeframeworkformodel-basedas-sessmentofmethodstoimprovesoftwarequality:modelling+validation+simulation+sensitivity.Thatis,afterrecord-ingwhatisknowninamodel,thatmodelshouldbevali-dated,exploredusingsimulations,thensummarizedtondthekeyfactorsthatmostimprovemodelbehavior.1IntroductionWhatisthebestmethodforimprovingsoftwarequal-ity?Softwareengineering(SE)researchhasgeneratedabewilderingrangeoftoolsforimprovingquality.Thelistoftechniquesisvery,verylongandincludesmodel0ASE,2002,http://ase.cs.ucl.ac.uk/cfp.htmlchecking[9],staticcodeanalysis[21],runtimeverica-tion[20],lightweightlanguagesforrequirementsengineer-ing[36],model-basedmethodsforoptimizingsoftwarepro-cesses[24],justtomentionafew.Howcananindustrialpractitionerhopetochoosetherighttoolfromthebewilderingsetofavailablealternatives?TheSEliteraturecontainsmanytruismsthatanindustrialpractitionermightusetoselecttherighttool.Forexample,Truism1:Itisbesttoremoveerrorsduringtheearlypartsofasoftwarelifecycle.[36].Truism2:Increasinganorganization'ssoftwareprocesslevelisadesirablegoal[3].Ifourpractitionerbelievesinthersttruism,thentheymightinvestigateMenzies'lightweightrequirementsengi-neeringlanguages[36].Alternatively,iftheybelieveinthesecondtruism,theymightrejectallautomaticsoftwareen-gineeringmethodsandjustexploresoftwareprocessim-provementtechniquessuchasCMMi[3].Theproblemwithtruismsisthattheymaybemisleadingforparticularprojects,orjustplainwrong.TheliteratureisfullofevidencethatchallengesmanySEtruisms.Forexam-ple,adubioustruismofvisualprogrammingisthat“visualrepresentationsareinherentlysuperiortomeretextualrep-resentations”.AreviewbyMenziessuggeststhattheavail-ableevidenceforthisclaimishardlyconclusive[30].Othercommonlycitedtruismsarejustasdubious.Forexample,despiteclaimsthatobject-oriented(OO)encapsulationwillreduceerrorratesinsoftware[37],empiricalresultssuggestthatdebugginganOOprogramismanytimesharderandlongerthandebuggingastandardproceduralprogram[19].Inotherwork,Fenton&Neil[13,14]offerascathingcri-tiqueofthetruismthat“pre-releasefaultratesforsoftwareareapredictorforpost-releasefailures”(asclaimedby[10],amongstothers).Forthesystemsdescribedin[15],they showthatsoftwaremodulesthatwerehighlyfault-pronepriortoreleaserevealedveryfewfaultsafterrelease.Giventhishistoryoffailedtruisms,dowehaveanyguid-ancetoofferourpractitionerstryingtoselectwhichtooltoapply?Onemethodistostudytheinherentpropertiesofthetools.Forexample,Lowryet.al.[27]andMenzies&Cukic[32]contrastthecostsanddefectdetectiondecayratesofformalmethods,whiteboxtesting,andblackboxtesting.Whilesuchananalysisisuseful,ourbeliefisthattoolselectionmustbemadeonaproject-specicbasis.Thisbeliefresultsfromtheexperimentsdescribedinthispaper.Usingmodel-basedsimulationsofsoftwareprocesses,wehaveidentiedthechangesthatmostimprovethequalityofparticularsoftwareprojects.ThesechangescontradictTru-ismOneandTruismTwo(shownabove).Itisneithernovelnorinterestingtosaythatparticularcasescancontradictgeneralprinciples.However,twoas-pectsofourworkarebothverynovelandveryinterest-ing.Firstly,ourcounter-examplestoTruismsOneandTruismsTwowerefoundintheveryrstcasestudiesweexplored.Theeasewithwhichwefoundthesecounter-examplesmakesusverysuspiciousofthesetruisms.Sec-ondly,whenthetruismsgeneratednon-optimumadvice,wefoundwecouldidentifyaminimalsetofchangesthatmostimprovedsoftwareprojects.Further,usingourmethod,theseminimalrecommendationsarehighlyspecializedtothespecicsoftwareprojectbeingstudied.OurmethodisbasedontheTAR2treatmentlearner,describedbelow.Givenunlimitedresources,asoftwareprojectmanagermayelecttoimplementTAR2'sminimalrecommendationsaswellastrying(e.g.)earlierdetectremovalorincreas-ingtheirsoftwareprocess.However,inthemoreusualcaseofresource-boundsoftwaredevelopment,softwareprojectmanagersmaynditusefultorestrictthemselvestojustTAR2'srecommendationssincethesearethemini-mumchangesthatmostimprovetheirprojects.Therestofthispaperisstructuredasfollows.SectionTwodescribesourgeneralmethodsfortestingtruismsandndingalternatekeyfactorsifthetruismsfail.Thatap-proachcanbesummarizedasfollows:decisions=modelling+validation+simulations+sensitivityThatis,afterrecordingwhatisknowninamodel,thatmodelshouldbevalidated,exploredusingsimulations,thensummarizedtondthekeyfactorsthatmostimprovemodelbehavior.SectionsThreeandFourapplythismethodtoTruismOneandTruismTwo.Methodsforimprovingsoftwareprojectswerefoundthatweredemonstrablybetterthantheadviceofferedbythetruisms.Finally,inSectionFive,wediscusshowthiskindofanalysismightbeappliedtoassessingtherelativemeritsof(e.g.)runtimevericationvsmodelchecking.Beforebeginning,itisimportanttostressthatitwouldbeamistaketoviewourresultsasnewtruismsthatreplaceexistingtruisms.Forexample,SectionThreeisnotanar-gumentthatthereisneveranyvalueinsoftwareprocessimprovement.Similarly,SectionFourisnotanargumentthatthereisneveranyvalueinearlylifecycleerrorremoval.Earlylifecycledefectremovalandincreasingsoftwarepro-cesslevelscanhavesignicantimpactonaproject.How-ever,whatourresultsaresayingisthat,fortheparticularprojectsstudiedhere,otherfactorsweremoresignicant.Further,thesemoresignicantfactorscouldbefoundverysimply,usingtreatmentlearning.2DecisionsUsingModel-BasedSimulationsThissectiondiscussesgeneralprinciplesformodelling+validation+simulations+sensitivity.Manygeneraltoolsandmethodologiesexistformod-ellingsuchasdistributedagent-basedsimulations[8],discrete-eventsimulation[18,25,26]1,continuoussimu-lation(alsocalledsystemdynamics)[1,45],state-basedsimulation(whichincludespetrinetanddataowap-proaches)[4,17,29],logic-basedandqualitative-basedmethods[5,chapter20][23],andrule-basedsimula-tions[38].Notethatforsoftwareprocessprogramming,elaboratenewmodellingparadigmsmaynotberequired.Forexample,theLittle-JILprocessprogramminglan-guage[6,46]justusesstandardprogrammingconstructssuchaspre-conditions,post-conditions,exceptionhandlers,andatop-downdecompositiontreeshowingsub-tasksin-sidetasks.Simulationscanbebasedonnominaloroff-nominalval-ues.Nominalsimulationsdrawtheirinputsfromknownoperationalprolesofsysteminputs[39].Off-nominalmonte-carlo(alsocalledstochastic)simulations,wherein-putsareselectedatrandom,cancheckforunanticipatedsituations[16].Stochasticsimulationhasbeenextensivelyappliedtomodelsofsoftwareprocess[42].Itisstrangetoreport,butexamplesofexecutionofLittle-JILarerare(aninterpreterexistsforLittle-JILmodelsbutexamplesofitsexecutionarenotmentioned(i.e.in[6,46]).Inasensitivityanalysis,thekeyfactorsthatmostinu-enceamodelareisolated.Also,recommendedsettingsforthosekeyfactorsaregenerated.Wetakecaretodistinguishsensitivityanalysisfromtraditionaloptimizationmethods.Inourexperience,therealsystemswedealwitharesocom-plexthattheydonotalwaystinto(e.g.)alinearopti-mizationframework.Studyingdatagrownfromsimulatorsletsusinvestigatecomplex,non-linearsystemsusingava-rietyofdatadrivendistributions.Thesemodelscancapturecomplexfeedbackandreworkloopswhicharenotpossiblefortraditionaloptimizationmethods.Ourexperienceisthatsimulationmodelscanlookatprocessesindetailaswellas1Seealsothehttp://www.imaginethatinc.comwebsite.2 atahighlevelofabstractionwhichiswherethemorean-alyticmodelsmustreside.Finally,simulationmodelscancapturemultipleperformancemeasuresnotabletobeex-ploredusingtheseoptimizationformulations.Thisisnottosaythattraditionaloptimizationmodelsarenotuseful.Forcertainquestions,traditionaloptimizationformulationsprovidethebesttforthequestionthatistryingtobean-swered.However,formanyquestions,thestandardopti-mizationmodelsarenotthebestchoiceandsomethingliketreatmentlearningmaybemoreuseful.2.1TreatmentLearningTreatmentlearningisourpreferredgeneralmethodformulti-dimensionalsensitivityanalysis[12,22,31,33–35].Treatmentlearnersndthefewestfactorsthatmostinu-enceasimulationmodel.Inthecaseofsimulationsgener-atedfrommodelsofsoftwareprocess,thesesummariescanreadasadviceonhowtobestoptimizeasoftwareprocess.TheTAR2treatmentlearninghasbeensuccessfullyap-pliedeitherinasinglebatchmode[22,33–35],orincre-mentallyinordertoencourageasystemtowardssomede-siredgoal[12,31].Thealgorithmmakesnoassumptionaboutcontinuityofvariablesandhencemaysucceedwherestandardregressionandlinearoptimizationmayfail.Con-ceptually,givenasetofinputattributesandoutputclassi-cations,thealgorithmsearchesthroughallcombinationsofattributerangesthatareunderconsiderationtondthosewhichleadtothemostdesiredoutputsandtheleastundesir-ableoutputs.Thissearchisclearlyintractablesinceacom-pletesearchofallsubsetsoftheinputattributerangeswouldtakeexponentialtime.TAR2onlyworkssincealinear-timeheuristicscoringmechanismcanquicklyndwhichattributerangescanbeignored.Asexplainedin[35],thealgorithmisaminimalcontrastsetassociationrulelearnerwithweightedclasses.Thatis,whilestandardmachinelearnersndpossiblycomplexde-scriptionsoftheclasseswithinadomain,TAR2ndstheminimaldifferencebetweenclasses(thecontrastset).Thealgorithmusesaheuristicclassweightinordertoreporttheleastchangethatselectsforthemostpreferredclasses.ThestandardintroductoryexampleforTAR2(e.g.asshownin[31])isthelogofgolfplayingbehaviorofFig-ure1.Thislogcontainsfourattributesand3classes.TAR2accessesascoreforeachclass.Foragolfer,theclassesinFigure1couldbescoredasnone=2(i.e.worst),some=4,lots=8(i.e.best).TAR2seeksoutstandingattributeranges;i.e.thosethatoccurfarmorefrequentlyinthehighlyscoredclassesthaninthelowerscoredclasses.Treatmentsareformedfromsubsetsoftheoutstandingattributerangesandappliedtotheexampledata.Exam-plesthatcontradicttheproposedtreatmentarerejected.Theworthofatreatmentisassessedbycomparingthebaselineoutlooktemp(oF)humiditywindy?classsunny8586falsenonesunny8090truenonesunny7295falsenonerain6570truenonerain7196truenonerain7096falsesomerain6880falsesomerain7580falsesomesunny6970falselotssunny7570truelotsovercast8388falselotsovercast6465truelotsovercast7290truelotsovercast8175falselotsFigure1.Alogofsomegolf­playingbehavior.baselineoutlook=overcasthumidity90024653602460040246311LEGEND:nonesomelotsFigure2.Treatmentsthatcanchangegolfplayingbehaviorfromthebaseline.classdistributiontothetreateddistribution.Forexample,Figure2showsTAR2'sanalysisofthegolfdata.Theleft-hand-sidehistogramofFigure2showsthebaselineclassfrequencyinFigure1.Notethatinthebaseline,weonlyplaygolflotsoftimesin65+3+6=43%ofcases.Themid-dlehistogramofFigure2showsthebestactionfoundbyTAR2:withtherestrictionthatoutlook=overcast,thenweplaygolflotsoftimesin100%ofcases.Theright-hand-sidehistogramofFigure2showstheworstactionfoundbyTAR2.Suchaworst-casescenariocanbegeneratedbyre-versingtheclassscores.ThisreversalmakesTAR2seektheworstpossibletreatment.Inthecaseofourgolfexample,withtherestrictionthathumidity90thenweplaylotsofgolfin13+1+1=20%ofcases.Insummary,tomaximizegolfplayingbehavior,weshouldselectaholidaylocationwithanovercastoutlook.Whileonholidays,weneedonlymonitorthehumidity(nottemperatureoroutlookorwind)andbecomealarmedifthehumidityincreasesover90%.TAR2'streatmentsshouldbeassessedonexamplesnotseenduringtraining.Therearetwostandardmethodsfordoingso:n-waycrossvalidationandre-simulation.Inthe3 former,thetrainingsetisdividedintoNbuckets.Foreachbucketinturn,atreatmentislearnedontheotherN1bucketsthentestedonthebucketputaside.AtreatmentisdeemedstableifitworksinthemajorityofallNturns.TheresultsofourrstcasestudywillbeassessedviaN-waycrossvalidation.Inare-simulationstudy,thetreat-mentsrecommendedbyTAR2areimposedonthesimula-tor.Thesimulatoristhenrunagain.AtreatmentisdeemedpredictiveifthepredicteddistributionisrealizedinoutputofasimulatorconstrainedtoTAR2'sproposedtreatment.Theresultsofoursecondcasestudywillbeassessedviare-simulation.CaseStudy1:“Increasinganorganizationssoftwareprocesslevelisadesirablegoal”Inthissection,weapplyourmodelling+validation+simulation+sensitivityapproachtothetruism:“increas-ingsoftwareprocesslevelisdesirable”.TheSoftwareEngineeringInstitute's(SEI)capabilitymaturitymodel(CMM[40])categorizessoftwareorgani-zationsintooneofveprocesslevels.BelowCMM5,thereisnouseofmeasurementstooptimizeacompany'ssoft-wareprocess.Also,belowCMM4,thereisnosystematicdatacollection.Lastly,belowCMM3,acompany'ssoft-wareprocessisnotevenwrittendown.TheCMM5movementhasmanyleaders(e.g.[3])butfewfollowers.In1996,lessthan12%ofonesampleofcertainsoftwareorganizationswereabovelevel32.Further,thecurrentindustryaverageseemstobelessthanCMM23.Inordertoassessthemeritsofincreasingsoftwarepro-cess,weappliedtheformulamodelling+validation+simulation+sensitivitytooneNASAproject.Elevenchangestothatprojectwereconsidered,includinganin-creasetothesoftwareprocesslevel.Countertothetruismofthissection,TAR2foundthatthebestactionincludedalowsoftwareprocesslevel.Therestofthissectiondescribesthatstudy.3.1ModellingFigure3showsaNASAsoftwareprojectscoredonthe22parametersoftheCOCOMO-IIsoftwarecostestimationmodel[2]4.ThecoreintuitionofCOCOMO-IIisthattheeffortrequiredtodevelopsoftwareincreasesexponentiallyasthatsoftwaregrowsinsize;i.e.:2http://www.telcordia.com/newsroom/mediaclips/telossource/telesrcanalyst.html3PersonalcommunicationwithSEIresearchers.4Foraprecisedenitionoftheseparameters,seehttp://sunset.usc.edu/research/COCOMOII/expert_cocomo/KC-1rangesnowchangesprec=0..5precedentness0,1ex=0..5developmentexibility1,2,41Scaleversresl=0..5architecturalanalysisorriskresolution0,22team=0..5teamcohesion1,22pmat=0..5processmaturity0,1,33rely=0..4requiredreliability4Productdata=1..4databasesize2attributescplx=0..5productcomplexity4,5ruse=1..5levelofreuse1,2,33docu=0..4documentationrequire-ments1,33time=2..5executiontimecon-straints?Platformutesstor=2..5mainmemorystorage2,3,42pvol=1..4platformvolatility1acap=0..4analystcapability1,22pcap=0..4programmercapability2Personnelutespcon=0..4programmercontinuity1,22aexp=0..4analystexperience1,2pexp=0..4platformexperience2ltex=0..4experiencewithlan-guageandtools1,33Projecttool=0..4useofsoftwaretools1,2attributessite=0..5multi-sitedevelopment2sced=0..4timebeforedelivery0,1,22Figure3.COCOMO­IIparameters.Scaledriversarelistedrst.Thecostdriversareunionoftheproduct,platform,personnel,andprojectattributes.LasttwocolumnsshowvaluesknownwithinoneNASAsoft­wareproject.aKSLOC(1:01+P5=1SFi)017Yj=1EMj1Thisexpressioncomputespersonmonthsforaproject;i.e.152hoursofeffort(andincludesdevelopmentandmanage-menthours).Inthisexpression,aisadomain-specicpa-rameter;KSLOCisestimateddirectlyorcomputedfromafunctionpointanalysis;SFiarethescaledrivers(e.g.driverssuchas“havewebuiltthiskindofsystembefore?”);andEMjarethecostdrivers(e.g.requiredlevelofreliabil-ity).Figure3liststheCOCOMO-IIscaledriversandcostdrivers.COCOMO-IIalsodenesasimilarequationfortherecommendednumberofpersonmonthsfortheproject.WeelectedtoreporttheCOCOMO-IIoutputexpressedasthenumberofstaffrequiredtogetpersonmonthsofworkdoneintherecommendednumberofmonths:staff=personmonthsmonths4 rely=rely=rely=rely=rely=verylownominalhighverylowhighsced=verylow00012sced=low00001sced=nominal00000sced=high00000sced=veryhigh00000Figure4.AMadachytable.From[28].Thistablereadsasfollows.Intheexceptionalcaseofhighreliabilitysystemsandverytightschedulepressure(i.e.sced=loworverylowandrely=highorveryhigh),addsomeincre­mentstothebuilt­inparameters(incrementsshowntop­right).Otherwise,inthenon­exceptionalcase,addnothingtothebuilt­inparameters.ThecolumnlabellednowinFigure3showsthecurrentsituationofaparticularNASAproject.Theanalystsin-terviewedforthiscasestudyknewsomeuncertaintiesex-istedintheirunderstandingofthisproject.Wheresome-whatuncertain,theyusedranges;e.g.itwasunclearifdevelopershadneverseenthiskindofapplicationbeforesoprec=f0;1g.Whentotallyuncertain,theyjustusedaquestionmark;e.g.noknowledgeaboutexecutiontimeconstraintswasavailablesotime=?=f2;3;4;5gwheref2;3;4;5gisthecompleterangeofpossiblevaluesfortime.Inthisstudy,inputssuchasFigure3areprocessedbytheCOCOMO-IImodelaswellastheMadachymodelofsoftwareprojectmanagementissues[28].WhiletheCOCOMO-IImodelestimatesdevelopmenteffort,theMadachymodeloutputsanumericindexrepresentinghowconcernedanexperiencedanalystmightbeaboutaparticu-larsoftwareproject.Themodelcontains94tablesthatim-plementacontext-dependentmodicationtointernalCO-COMOparameters.Figure4operationalizesoneofthe94Madachyheuristics:i.e.softwarethatmustbehighlyre-liableshouldnotbedevelopedunderexcessiveschedulepressure.WhileMadachycallshisworka“risk”model,hisdenitionissodifferenttothestandarddenitionofrisk=severityfrequencythatwerenameittoa“wor-ries”model.COCOMO-IIassessesprocessmaturity,orpmat,viaan18-pointquestionnairethatexploresthekeyprocessareasofaproject.SincemostprojectsscoreverylowontheCMMscale,COCOMO-IIdividesCMM1intotwozones.Hence,theCOCOMO-II'spmatrangeisf0,1,2,3,4,5gwhere0,1denotethelowerandupperhalfofCMM1(respectively).ObservethevaluesforprocessmaturityinFigure3.Ouranalystssaidthatpmat=f0;1;2;3g;i.e.thisparticularproject'sprocesslevelwaslessthanCMM3.ThecolumnlabelledchangesinFigure3showselevenproposedchangestothecurrentsituation.NotethatitisproposedtochangeprocessleveluptoCOCOMO-IIpmat=3.Thegoalofoursensitivitystudywillbetoas-sessifpmat=3isthepreferredprocesslevel.3.2ValidationBeforetrustingdecisionsfrommodelling+validation+simulation+sensitivity,itisimpor-tanttovalidatethemodel.TheCOCOMOteamhaspublishednumerousvalidationstudiesofCOCOMOmodels.Forexample,intentrialswithdatanotusedduringmodeltuning,theCOCOMO-IIeffortpredictioncamewithin25%oftheactualeffortin69%(onaverage)ofcases(thisisdenotedasPRED(25)=69)[7].InothervalidationworkfromtheCOCOMOteam,Madachyhasreportedstudies[28]withhismodelandtheCOCOMO-Iprojectdatabase.ThosestudiesshowedagoodcorrelationbetweentheMadachy“worries”indexandmonthsKDSI(whereKDSIisthousandsofdeliveredsourcelinesofcode).Thatis,theMadachymodel'soutputisameasureofthedangerthataprojectwilltakelongerthanplannedtobuild.3.3SimulationCOCOMOmodelsrequireaninternaltableofnumericvaluesthatmap(e.g.)pmat=2intotheeffortandscheduleequations.ThisstudyusedthelatestpublishedCOCOMO-IItableasshownin[7].Modelinputswereselectedatrandom,10,000times,fromtheparameterrangesshowninFigure3'snowcolumn.Inthisstudy,someuncertaintyexistedinthesizeestimatesandsotheSLOC(sourcelinesofcode)estimatewastakentobe75K,100K,125K.Hence,themodelwasrun30,000times(10,000timesforeachSLOCvalue).ModeloutputsforstaffandworrieswerecomputedusingCOCOMO-IIandtheMadachymodel(respectively).Fig-ure5showstheresultsasapercentilematrix;i.e.itshowswhatpercentageofthe30,000runsfallsintoaparticularrange.Thepercentilesmatrixiscolor-coded:thedarkerthecell,thelargethepercentageoftherunsinthatcell.AsmightbeexpectedfromthelargerangeofpossibleinputvaluesshowninFigure3,thereisalargevarianceintheresults.SensitivityTAR2wasusedtondwhichsubsetoftheproposedchangeshadthemostimpactontheproject;i.e.mostde-creasedstafflevelsandthe“worries”index.Treatmentsthatdidnotincluderangesoutsideofthechangescolumnof5 WorriesStaff0714212835Totals2001801601401201121001416801951166014153324052953920235Totals75630133100Figure5.SimulationoutputsusinginputsspeciedinFigure3.Eachcellshowsthepercentageoftherunsthatfallintoacertainrange.WorriesStaff0714212835Totals20018016014012010080602323401145562014721Totals2575100Figure6.SensitivityresultsforFigure5.Figure3wererejected.Afterexploringallsubsetsoftheproposedchanges,TAR2foundthefollowingtreatment:pmat=1^acap=2^sced=2Furtherexperimentationshowedthatnoothertreatmenthadabetterimpact.Thatis,thebesttreatmentisthiscasewasacombinationof(i)increasingthetimetodeliveryto100%ofthetimeproposedbytheproject-i.e.nopres-sureforanearlydelivery;(ii)usinganalystswithamiddle-rangeofability(fallbetweenthe45thto65thpercentile);and(iii)ensuringthattheprojectwasatleastintheupper-halfofCMM1,butdon'tgotoCMMprocesslevel2.Thistreatmentwastestedviare-simulation.Theinputrangesforpmat,acap,scedweresetasabove,andtherestoftheinputswereleftasbefore;i.e.abletorangeoverallthevaluesofFigure3.TheresultsareshowninFigure6.WhencomparedtoFigure5,itcanbeseenthattheproposedtreat-mentsgreatlyreducedthevarianceinthemodel'sbehaviorwhileimprovingthemeanvalues(decreasedstafngleveland“worries”).Notethat,contrarytothetruismthatprocesslevelim-provementisaninvaluablemethodofimprovingsoftwaredevelopment,thisparticularprojectneededonlyaverymin-imallevelofCMMprocess(upperhalfofCMM1).4CaseStudy2:“Itisbesttoremoveerrorsduringtheearlysoftwarelifecycle”Inthissection,weapplyourmodelling+validation+simulation+sensitivityapproachtothetruism:“Itisbesttofocusonerrorremovalduringearlylifecycle”.Manyresearchershavepredicatedtheirworkonthetru-ismthatcatchingerrorsduringearlysoftwarelifecycleisveryimportant.Forexample,therstauthorhaswrit-ten[36]:Thecaseformoreformalityin(earlylifecycle)isoverwhelming.Manyerrorsinsoftwarecanbetracedbacktoerrorsintherequirements[43].Of-ten,theconceptionofasystemisimprovedasadirectresultofthediscoveryofinadequaciesinthecurrentconception.Theearliersuchinade-quaciesarefound,thebetter,sincethecostofre-movingerrorsattheearlierstagecanbeordersofmagnitudecheaperthanthecostofremovingerrorsinthenalsystem[44].However,asweshallseeinthefollowingstudy,removingearlylifecycleerrorsforaparticularprojectwasnotnearlyasimportantasselectionoftheinspectionmethods4.1ModellingInthissectionwewilldescribeaparticularsimulationmodeltunedtoaparticularcompany,aswellastheprojectanddevelopmentcontextofthatmodel.Ahigh-levelblockdiagramofthismodelisshowninFigure7.ThemodelisfarmorecomplexthatsuggestedbyFigure7sinceeachblockreferencesmanyvariablessharedbyallotherblocks.Thismodelwasoriginallydevelopedin1995[41]andwassubsequentlytailoredtoaspeciclarge-scaledevelop-mentprojectataleadingsoftwaredevelopmentrmwiththefollowingproperties:Alargecompanywithanumberofdevelopmentsitesthroughouttheworld.Between10and20majorprojectsbeingconductedatonetimeatthestudysite.Workscopeincludedworld-widesystemsupportofmostoftheproductsbeingdevelopedatthesite(in-cludingtheonebeingstudied).Supportprofessionalsareactuallyexperienceddeveloperswhohavebeenas-signedtocorrectelddetecteddefects.Whenthesesupportprofessionalshavetimeavailable,theycarryoutlimiteddevelopmenttasks.6 projectsapprovedfunctionalspecificationhigh-leveldesignFS inspectionlow-leveldesignHLD inspectioncodedevLLDinspectioncodeinspectionunittestdevelopmentcompletefunctionaltestunit testcompletesystemtestfield support andmaintenancerelease tocustomersFigure7.High­levelblockdiagramofadis­creteeventmodelofonecompany'ssoftwareprocess.ThesiteachievedISOcerticationandwasassessedataLevel2usingaCMM-SPAduringthestudyperiod.Theproductstudiedhadcompleted5successivemajorreleaseswhenthestudybegan.Ineachrelease,majornewfunctionalitywasaddedandsubstantialrevisionsofexistingfunctionalityoccurred.Atpeakdevelopmentperiodstheprojectinvolvedover70people.Thesoftwareprocessstudiedatthiscompanyessen-tiallyfollowedawaterfallprocessmodelincludingthema-jorphasesoffunctionalspecication(FS),high-leveldesign(HLD),low-leveldesign(LLD),coding(CODE),unittest(UT),functionalvericationtest(FVT),andsystemveri-cationtest(SVT).Inspectionsofthefunctionalspecica-tion,high-leveldesign,low-leveldesign,andcodewerealsoconducted.AfterSVTandFVTwerecompletedtheprod-uctwasreleasedtothepublic.Thesephases,aswellasaperioddevotedtoeldsupportandmaintenancearecap-turedbythismodel.Inaddition,theprocesssegmentsfordevelopingtestplansandwritingtestcasesforfunctionaltestandsystemtestwerealsoincluded.Wedevelopedtwomodelsofthisprocess-astate-basedsimulationmodelusingtheStatemateMagnumtoolbyi-Logix5andadiscreteeventmodelusingtheExtendsimu-lationlanguage6.Thediscreteeventmodelcontained30+processstepswithtwolevelsofhierarchy.Themainperfor-mancemeasuresofdevelopmentcost,productqualityandprojectschedulewerecomputedbythemodel.Theseper-formancemeasurescouldalsoberecordedforanyindivid-ualprocessstepasdesired.Someoftheinputstothesim-ulationmodelincludedproductivityratesforvariouspro-cesses;thevolumeofwork(i.e.KSLOC);defectdetectionandinjectionratesforallphases;effortallocationpercent-agesacrossallphasesoftheproject;reworkcostsacrossallphases;parametersforprocessoverlap;theamount/effectoftrainingprovided;andresourceconstraints.Actualdatawereusedformodelparameterswherepos-sible.Forexample,inspectiondatawascollectedfromin-dividualinspectionformsforthetwopastreleasesoftheproduct.Distributionsfordefectdetectionratesandinspec-tioneffectivenesswheredevelopedfromtheseindividualinspectionreports.Also,effortandscheduledatawerecol-lectedfromthecorporateprojectmanagementtrackingsys-tem.Lastly,seniordevelopersandprojectmanagersweresurveyedandinterviewedtoobtainvaluesforotherprojectparameterswhenharddatawerenotavailable.Modelsweredevelopedfromthisdatausingmultiplere-gressiontopredictdefectratesandtaskeffort.Theresultwasamodelthatpredictedthethreemainperformancemea-suresofcost,quality,andschedule.Alistofallofthepro-cessmodicationsupportedbythemodelaretoonumeroustolisthere.Sufcetosaythatsmalltomediumscopepro-cesschangescouldbeeasilyincorporatedandtestedusingthemodel.4.1.1InspectionTypesForeachphaseofthemodelledprocess,therewasachoiceofeitherhavingoneoffourinspectiontypes;e.g.none,orafullfagan,orabaselineinspection,orawalkthrough.Thesetermsareexplainedbelow.Afullfaganinspection[11]isawell-researchedmanualinspectionmethod.Suchinspectionsarepreciselydened,includingasevenstepprocesspluspre-determinedrolesforinspectionparticipants.Faganinspectionscanndmany5StatemateandI-LogixareregisteredtrademarksrofI-LogixInc.(3RiversideDrive;Andover,Massachusetts01810USA).6ExtendandImagineThatareregisteredtrademarksrofImagineThat,Inc.(6830ViaDelOro,Suite230,SanJose,California,95119USA).7 errorsinasoftwareproduct.Forexample,forthecompanystudiedhere,thedefectdetectioncapability7oftheirfullFaganinspectionswasTR(0:35;0:50;0:65)8.Inthisstudy,afullFaganinspectionusedbetween4and6staff,plustheauthoroftheartifactbeinginspected.Abaselineinspectionwasacontinuationofcurrentprac-ticeatthecompanyunderstudy.ThebaselineinspectionatthiscompanywasessentiallyapoorlyperformedFaganinspection,ThedistinctionbetweenaproperFaganinspec-tionandthebaselineisthatstaffwouldreceivenewtrain-ing,checklistsandsupportinordertosignicantlyimprovetheeffectivenessoftheinspections.Thedatashowedthatbaselineinspectionshadvaryingdefectdetectioncapabili-tiesrangingfromaminimumof0.13,amaximumof0.30andanaverageof0.21(thesegureswereobtainedfromactualinspectionrecords).Walkthroughinspectionswereconductedbyonebytheauthoroftheartifactbeinginspectedinarelativelyinformalatmosphere.Processexpertsestimatedtheamountoftimeanddefectdetectioncapabilityforthistypeofinspection.ThoseestimateswereTR(0:07;0:15;0:23).4.1.2SummarizingModelOutputTheoutputsofthemodelareassessedviaamulti-attributeutilityfunctionutility=40(14quality)+320(70expense)+640(24duration)(1)wherequality,expenseanddurationaredenedasfollows.Qualityisthenumberofmajordefects(i.e.severity1and2)estimatedtoremainintheproductwhenreleasedtocus-tomers.Expenseisthenumberofperson-monthsofeffortusedtoperformtheworkontheprojectandtoimplementthechangestotheprocessthatwerestudied.Durationisthenumberofcalendarmonthsfortheprojectfromthebe-ginningoffunctionalspecicationuntiltheproductwasre-leasedtocustomers.Thejusticationforthisstyleofutilityfunctionisdis-cussedindetailin[41].Insummary,thisfunctionwascreatedafterextensivedebriengofthebusinessuserswhofundedthedevelopmentofthismodel.4.2ValidationThebaselinesimulationmodelofthelifecycledevelop-mentprocessseeninFigure7wasvalidatedinanumberofways.Themostimportantofwhichwereasfollows.In7Defectdetectioncapabilityisthepercentageofdefectsthatarelatentintheartifactthatisbeinginspectedthataredetected.8TR(a;b;c)denotesatriangulardistributionwithminimum,mode,meanofa;b;crespectively.500075001000012500150001750004000800012000utilitybestbaselineFigure8.Sortedutilitiesgeneratedincasestudy2.facevaliditystudies,processdiagrams,modelinputs,modelparametersandoutputswerereviewedbymembersofthesoftwareengineeringprocessgroupaswellasseniordevel-opersandmanagersfortheirdelitytotheactual.Inoutputvaliditystudies,themodelwasusedtoaccuratelypredicttheperformanceofseveralpastreleasesoftheproject.Fi-nally,inspecialcasestudies,themodelwasusedtopre-dictunanticipatedspecialcases.Specically,whenpredict-ingtheimpactofdevelopingoverlycomplexfunctionality,themodelpredictedthatdevelopmentwouldtakeapproxi-matelydoublethenormaldevelopmentschedule.Thisre-sultwasnotacceptedinitiallybymanagementasitwastoolong,however,uponfurtherinvestigationitwasfoundthatthemodelpredictionscorrespondedquiteaccuratelywiththiscompany'sactualexperience.4.3SimulationThesimulationmodeldescribedabovecontainsfourphasesofdevelopmentandfourinspectiontypesateachphase.Thefourphaseswerefunctionalspecication(FS);highleveldesign(HLD);lowleveldesign(LLD);andcod-ing(CODE).ThefourinspectiontypeswerefullFagan,baseline,walkthrough,andnone.Thisresultsin44dif-ferentcongurations.Eachcongurationwasexecuted50times,resultingin5044=12800runs.Eachrunwassum-marizedusingEquation1.TheutilitiesgeneratedbythismethodareshownsortedasthebaselineplotofFigure8.Notethehugerangeofoutputvalues:5,000to15,000.4.4SensitivityTAR2wasusedtoexplorethesimulationdata.Thebesttreatmentfoundwastousethefollowingconguration:8 Nofunctionalspecicationsinspections;Baselineinspectionsforhighleveldesign;i.e.nochangefromcurrentpractice;FullFaganforlowleveldesignandcodereviews.Furtherexperimentationshowedthatnolargertreatmenthadagreaterimpact.Thatis,otherfactorsintheFigure7modelwerenotasinuentialastheselectionofinspectiontype.Giventhisconguration,TAR2predictedthedistribu-tionofutilityvaluesseenasthebestplotofFigure8.Whencomparedtothebaselineplot,weseethatthistreatmentre-sultsinasignicantimprovementinmodelbehavior.Also,thevarianceintheutilitieshasbeengreatlyreduced.Thistreatmentwasassessedvia10-waycrossvalidation.ThebestplotofFigure8wasobservedtobetheaverageimprovementseenundercrossvalidation.Notethatcontrarytothetruismthatearlylifecyclede-tectionanderrorremovalisvaluable,thisparticularprojectwasmostimpactedbyspendinglesseffortonearlylifecycleinspectionsandmoreeffortonlatelifecycleinspections.5DiscussionOpponentsofourapproachmightarguethatitisnotassimpleasitappears.Forexample,ourapproachneedsamodelofthesoftwareprojectandbuildingsuchmodelscanbeanon-trivialtask.Therearetworepliestothisobjec-tion.Firstly,ourapproachdoesnotalwaysrequireelaboratemodelling.Forexample,themodelusedinSectionThreewasanoff-the-shelfopen-sourcemodel;i.e.itdidnotre-quireanyefforttodevelop.Hence,ourrecommendationisthatifresourcespermit,thendetailedmodelsshouldbebuilt.Otherwise,publicdomainmodelscansufce-atleastforaninitialstudy.Secondly,thisworkshowsthattruism-basedchangestosoftwareprocessesmaynotachievethebenetstheypromise.Theextramodellingcostrequiredbyourmethodmustbecomparedtothecostofdesigningandimplementinguselessprocesschangesthatdonotdeliverthepromisedbenets.Ourparadigmofdecisions=modelling+validation+simulations+sensitivitygivesustheabil-itytoexaminetheconditionsunderwhichtheperformanceofagivensystemmaybedramaticallyimproved.Moreover,thisapproachprovidesprescriptiveguidancebyidentifyingtherangestowhichvariousinputparameters(suchasin-spectionefciencyandthelike)muchbemovedinordertoachievethesedesiredlevelsofperformance.Ideally,wewouldliketoextendthecurrentstudytoassessingstandardautomatedsoftwareengineeringmethods.Forexample,inthefollowingscenariowewouldbeabletoassessmethodssuchasruntimeverication,modelchecking,orstaticanalysis:Inaparticularproject,itisdeterminedthataparticularrangeofdefectdetectioninsourcecodeisessentialformostimprovingoverallsoftwarequality.Itisdeterminedthatrangeistoohighformanualmeth-odssuchasFaganinspections.Usinghistoricaldata,itispossibletoselecttheau-tomatedsoftwareengineeringmethod(s)thatachievesthetargetdefectdetectionrange.Theselectedautomatedsoftwareengineeringmethodsareassessedbycheckingwhichischeapesttodeployinthecurrentproject.Notethatthisstudywouldrequireinformationonthedetectdetectioncapabilityofthedifferentautomaticsoft-wareengineeringtools.Tothebestofourknowledge,suchinformationisunavailable.Hence,atthistime,theabovescenariocan'tyetbeperformed.References[1]T.Abdel-HamidandS.Madnick.SoftwareProjectDynam-ics:AnIntegratedApproach.Prentice-HallSoftwareSeries,1991.[2]C.Abts,B.Clark,S.Devnani-Chulani,E.Horowitz,R.Madachy,D.Reifer,R.Selby,andB.Steece.COCOMOIImodeldenitionmanual.Technicalreport,CenterforSoft-wareEngineering,USC,,1998.http://sunset.usc.edu/COCOMOII/cocomox.html#downloads.[3]D.Ahern,A.Clouse,andR.Turner.CMMIDistilled.Addison-Wesley,2001.[4]M.AkhaviandW.Wilson.Dynamicsimulationofsoftwareprocessmodels.InProceedingsofthe5thSoftwareEngineer-ingProcessGroupNationalMeeting(HeldatCostaMesa,California,April26-29).SoftwareengineeringInstitute,CarnegieMellonUniversity,1993.[5]I.Bratko.PrologProgrammingforArticialIntelligence.(thirdedition).Addison-Wesley,2001.[6]A.Cass,B.S.Lerner,E.McCall,L.Osterweil,S.M.S.Jr.,andA.Wise.Little-jil/juliette:Aprocessdenitionlan-guageandinterpreter.InProceedingsofthe22ndInter-nationalConferenceonSoftwareEngineering(ICSE2000),pages754–757,June2000.[7]S.Chulani,B.Boehm,andB.Steece.Bayesiananalysisofempiricalsoftwareengineeringcostmodels.IEEETransac-tiononSoftwareEngineerining,25(4),July/August1999.[8]W.Clancey,P.Sachs,M.Sierhuis,andR.vanHoof.Brahms:Simulatingpracticeforworksystemsdesign.InP.Compton,R.Mizoguchi,H.Motoda,andT.Menzies,editors,Proceed-ingsPKAW'96:PacicKnowledgeAcquisitionWorkshop.DepartmentofArticialIntelligence,1996.[9]E.Clarke,E.Emerson,andA.Sistla.Automaticvericationofnite-stateconcurrentsystemsusingtemporallogicspec-ications.ACMTransactionsonProgrammingLanguagesandSystems,8(2):244–263,April1986.[10]H.Dunsmore.Evidencesupportssometruisms,beliesoth-ers.(someempiricalresultsconcerningsoftwaredevelop-ment).IEEESoftware,pages96–99,May1988.[11]M.Fagan.Advancesinsoftwareinspections.IEEETrans.onSoftwareEngineering,pages744–751,July1986.9 [12]M.FeatherandT.Menzies.Convergingontheoptimalat-tainmentofrequirements.InIEEEJointConferenceOnRe-quirementsEngineeringICRE'02andRE'02,9-13thSeptem-ber,UniversityofEssen,Germany,2002.Availablefromhttp://tim.menzies.com/pdf/02re02.pdf.[13]N.E.FentonandM.Neil.Acritiqueofsoftwaredefectpredictionmodels.IEEETransactionsonSoftwareEngineering,25(5):675–689,1999.Avail-ablefromhttp://citeseer.nj.nec.com/fenton99critique.html.[14]N.E.FentonandM.Neil.Softwaremetrics:Aroadmap.InA.Finkelstein,editor,Softwaremet-rics:Aroadmap.ACMPress,NewYork,2000.Availablefromhttp://citeseer.nj.nec.com/fenton00software.html.[15]N.E.FentonandS.Peeger.SoftwareMetrics:ARigor-ous&PracticalApproach(secondedition).InternationalThompsonPress,1995.[16]W.Gutjhar.Partitionvs.randomtesting:Theinuenceofuncertainty.IEEETransactionsonSoftwareEngineering,25(5):661–674,September/October1999.[17]D.Harel.Statemate:Aworkingenvironmentforthedevel-opmentofcomplexreactivesystems.IEEETransactionsonSoftwareEngineering,16(4):403–414,April1990.[18]H.Harrell,L.Ghosh,andS.Bowden.SimulationUsingPro-Model.McGraw-Hill,2000.[19]L.Hatton.Doesoosyncwithhowwethink?IEEESoftware,pages46–54,May/June1998.[20]K.HavelundandT.Pressburger.Modelcheckingjavaprogramsusingjavapathnder.InternationalJournalonSoftwareToolsforTechnologyTransfer,2(4),April2000.Availablefromhttp://ase.arc.nasa.gov/visser/jpf/jpf1.ps.gz.[21]S.Horwitz,T.Reps,andD.Binkley.Interproceduralslicingusingdependencegraphs.ACMTransactionsonProgram-mingLanguagesandSystems,1(12):26–60,January1990.[22]Y.Hu.Bettertreatmentlearning,2003.MastersThesis,DepartmentofElectricalEngineering,UniversityofBritishColumbia,inpreperation.[23]Y.Iwasaki.Qualitativephysics.InP.C.A.BarrandE.Feigenbaum,editors,TheHandbookofArticialIntelli-gence,volume4,pages323–413.AddisonWesley,1989.[24]M.Kellner,R.Madachy,andD.Raffo.Softwareprocessmodelingandsimulation:Why,what,how,.JournalofSys-temsandSoftware,46(2/3),1999.[25]D.Kelton,R.Sadowski,andD.Sadowski.SimulationwithArena,secondedition.McGraw-Hill,2002.[26]A.LawandB.Kelton.SimulationModelingandAnalysis.McGrawHill,2000.[27]M.Lowry,M.Boyd,andD.Kulkarni.Towardsatheoryforintegrationofmathematicalvericationandempiricaltest-ing.InProceedings,ASE'98:AutomatedSoftwareEngineer-ing,pages322–331,1998.[28]R.Madachy.Heuristicriskassessmentusingcostfactors.IEEESoftware,14(3):51–59,May1997.[29]R.MartinandD.M.Raffo.Amodelofthesoftwaredevel-opmentprocessusingbothcontinuousanddiscretemodels.InternationalJournalofSoftwareProcessImprovementandPractice,June/July2000.[30]T.Menzies.Evaluationissuesforvisualprogramminglan-guages.InS.Chang,editor,HandbookofSoftwareEn-gineeringandKnowledgeEngineering,VolumeII.World-Scientic,2002.Availablefromhttp://tim.menzies.com/pdf/00vp.pdf.[31]T.Menzies,E.Chiang,M.Feather,Y.Hu,andJ.Kiper.Condensinguncertaintyviaincrementaltreatmentlearn-ing.InAnnalsofSoftwareEngineering(submitted),2002.Availablefromhttp://tim.menzies.com/pdf/02itar2.pdf.[32]T.MenziesandB.Cukic.Howmanytestsareenough?InS.Chang,editor,HandbookofSoftwareEngineeringandKnowledgeEngineering,VolumeII,2002.Availablefromhttp://tim.menzies.com/pdf/00ntests.pdf.[33]T.MenziesandY.Hu.Reusingmodelsforrequirementsen-gineering.InFirstInternationalWorkshoponModel-basedRequirementsEngineering,2001.Availablefromhttp://tim.menzies.com/pdf/01reusere.pdf.[34]T.MenziesandY.Hu.Agentsinawildworld.InC.Rouff,editor,FormalApproachestoAgent-BasedSystems,bookchapter,2002.Availablefromhttp://tim.menzies.com/pdf/01agents.pdf.[35]T.MenziesandY.Hu.Justenoughlearning(ofassociationrules).InWVUCSEEtechreport,2002.Availablefromhttp://tim.menzies.com/pdf/02tar2.pdf.[36]T.Menzies,J.Powell,andM.E.Houle.Fastformalanalysisofrequirementsvia'topoidiagrams'.InICSE2001,2001.Availablefromhttp://tim.menzies.com/pdf/00fastre.pdf.[37]B.Meyer.Object-orientedSoftwareConstruction.Prentice-Hall,HemelHemstead,1988.[38]P.MiandW.Scacchi.Aknowledge-basedenvironmentformodelingandsimulationsoftwareengineeringprocesses.IEEETransactionsonKnowledgeandDataEngineering,pages283–294,September1990.[39]J.Musa.SoftwareReliabilityEngineeredTesting.McGraw-Hill,1998.[40]M.Paulk,B.Curtis,M.Chrissis,andC.Weber.Capabilitymaturitymodel,version1.1.IEEESoftware,10(4):18–27,July1993.[41]D.Raffo.Modelingsoftwareprocessesquantitativelyandassessingtheimpactofpotentialprocesschangesofprocessperformance,1995.Ph.D.thesis,ManufacturingandOpera-tionsSystems,CarnegieMellonUniversity.[42]D.M.Raffo,J.V.Vandeville,andR.Martin.Softwarepro-cesssimulationtoachievehighercmmlevels.JournalofSys-temsandSoftware,46(2/3),April1999.[43]D.Reifer.Softwarefailuremodesandeffectsanalysis.IEEETransactionsonReliability,pages247–249,1979.[44]F.Schneider,S.Easterbrook,J.Callahan,G.Holzmann,W.Reinholtz,A.Ko,andM.Shahabuddin.Validatingre-quirementsforfaulttolerantsystemsusingmodelchecking.In3rdIEEEInternationalConferenceOnRequirementsEn-gineering,1998.[45]H.Sterman.BusinessDynamics:SystemsThinkingandModelingforaComplexWorld.IrwinMcGraw-Hill,2000.[46]A.Wise,A.Cass,B.S.Lerner,E.McCall,L.Os-terweil,andJ.S.M.Sutton.Usinglittle-jiltocoor-dinateagentsinsoftwareengineering.InProceedingsoftheAutomatedSoftwareEngineeringConference(ASE2000)Grenoble,France.,September2000.Availablefromftp://ftp.cs.umass.edu/pub/techrept/techreport/2000/UM-CS-2000-045.ps.10