Postmortem reviews purpose and approaches in software engineering Torgeir Dingsyr SINTEF - PDF document

Postmortemreviews:purposeandapproachesinsoftwareengineeringTorgeirDingsøyrSINTEFInformationandCommunicationTechnology,SPAndersensvei15b,7465Trondheim,NorwayReceived21June2004;revised17August2004;accepted25August2004 Conductingpostmortemsisasimpleandpracticalmethodfororganisationallearning.Yet,notmanycompanieshaveimplementedsuch 1.Introduction InformationandSoftwareTechnology47(2005)293–303www.elsevier.com/locate/infsof *Tel.:4773592979;fax:4773592977.E-mailaddress:torgeir.dingsoyr@sintef.no. 1.1.KnowledgemanagementArecentimprovementtrendhasbeenknowledgemanagement,whichisrelatedtocreating‘learningorganisations’,insoftwareengineering:‘learningsoftwareorganisations’.Alearningorganisationis‘anorganisationskilledatcreating,acquiring,andtransferringknowledge,andatmodifyingitsbehaviourtoreectnewknowledgeandinsight’’.GeorgeHubergivessomeadviceonwhatmanagerscandotomaketheirorganisationsmore‘learning’ing’:
Learnfromexperience—systematicallycapture,store,interpretanddistributerelevantexperiencegatheredfromprojects;andalsotoinvestigatenewideasbycarryingoutexperiments.Usingacomputer-basedorganisationalmemory—tocaptureknowledgeobtainedfromexpertstospreaditthroughtheorganisation.Aresearchareathatislinkedtoorganisationallearningisresearchon‘communitiesofpractise’asabasisforlearning.EtienneWengerwrites:“learningisanissueofsustainingtheinterconnectedcommunitiesofpractisethroughwhichanorganizationknowswhatitknows”s”.Inthemuch-citedbookonlearningorganisations,TheFifthDisciplineine,wendfurthercharacteristicsoflearningorganisations:theabilityof‘systemsthinking’—toseemorethanjustpartsofasystem.Thisoftenmeanstoinvolvepeopleinanorganisationtodevelopa‘sharedvision’,somecommongroundsthatmaketheworkmeaningful,andalsoservetoexplainaspectsthatyouyourselfdonothavehands-onexperiencein.Anotherwayofimprovingcommunicationinanorganisationistoworkon‘mentalmodels’thatsupportaction,‘personalmastery’;thatpeoplemakeuseoftheircreativityandabilities.Andnally,‘grouplearning’,toenhancedialogueandopennessintheorganisation.1.2.LearningTheprocessoftransferringknowledgebetweenpeopleisusuallyreferredtoas‘learning’.Webster’ster’sdeneslearningas‘toacquireknowledgeoforskillinbystudy,instruction,orexperience,tobecomeinformedoforacquaintedwith’or‘tomemorize’.Inorganisationalliterature,itisoftendenedasa‘purposefullychangeofaction’.Whatdoesitmeantosaythatanorganisationasawholelearns?Thisdiffersfromindividuallearningintworespectsects:rst,itoccursthroughsharedinsight,knowledgeandsharedmodels.Second,itisnotonlybasedonthememoryoftheparticipantsintheorganisation,butalsoon‘institutionalmechanisms’likepolicies,strategies,explicitmodelsanddenedprocesses.Wecancallthisthe‘culture’oftheorganisation.Thesemechanismsmaychangeovertime,whichisaformoflearning.ArgyrisandSchondistinguishbetweenwhattheycallsingleanddouble-looplearning[10]inorganisations.Single-looplearningimpliesabetterunderstandingofhowtochange(or‘tune’),sayaprocess,toremoveanerrorfromaproduct.Itisasinglefeedback—loopfromobservedeffectstomakingsomechanges(renements)thatinuencetheeffects.Doublelooplearning,ontheotherhand,iswhenyouunderstandthefactorsthatinuencetheeffects,andthenatureofthisinuence,whattheycallthe‘governingvalues’alues’.Insoftwareengineering,a‘learningsoftwareorganis-ation’hasbeendenedasanorganisationthathasto‘createaculturethatpromotescontinuouslearningandfosterstheexchangeofexperience’ience’.Dybaputsmoreemphasisonactioninhisdenition:“Asoftwareorganisationthatpromotesimprovedactionsthroughbetterknowledgeandunderstanding”g”.Inthefollowingsections,wewillpresenttwomodelsfromtheliteratureonhowknowledgeistransferredbetweenindividualsinorganisations,whatwecandescribeas‘learning’onanindividuallevel,and‘organizationallearning’foracommunity.Wedonotaimtocoverthewholerangeoftheoriesoflearning,butwillfocusontwoapproachesthatweconsiderinteresting,andhasbeenusedintheknowledgemanagementeld,namely:Learningthroughparticipation:communitiesofpractise.Learningasaconversionprocessbetweentacitandexplicitknowledge.1.2.1.Learningthroughparticipation:communitiesofpractiseThetraditionalviewoflearninghasbeenthatitbesttakesplaceinasettingwhereyouisolateandabstractknowledgeandthen‘teach’itto‘students’inroomsfreeofcontext.EtienneWengerdescribesthisviewoflearningasanindivi-dualprocesswhereforexamplecollaborationisconsideredakindofcheatingcheating.Inhisbookaboutcommunitiesofpractise,hedescribesacompletelydifferentview:learningasasocialphenomenon.Acommunityofpractisedevelopsitsown‘practises,routines,rituals,artifacts,symbols,conven-tions,storiesandhistories’.Thisisoftendifferentfromwhatyoundinworkinstructions,manualsandthelike.Inthiscontext,Wengerdeneslearningas:Forindividuals:learningtakesplaceinengaginginandcontributingtoacommunity.Forcommunities:learningistorenethepractise.Fororganisations:learningistosustaininterconnectedcommunitiesofpractise.Wendcommunitiesofpractiseeverywhere:atwork,athome,involunteerwork.AnditcanbeachallengetoT.Dingsøyr/InformationandSoftwareTechnology47(2005)293–303 sustainsuchnetworksofpeople,forexampleinturbulentorganisationsthatundergoreorganisationprocesses.Theworkoncommunitiesofpractiseiscloselylinkedtoworkonsituatedlearninglearning.1.2.2.LearningasaconversionprocessbetweentacitandexplicitknowledgeInthemuch-citedbook‘TheKnowledge-CreatingCompany’,whereNonakaandTakeuchiexplainsthesuccessofJapanesecompaniesbytheireffortat‘organizationalknowledgecreation’.Theyalsoofferamodelofhowknowledgeistransformedandconvertedinanorganisationisation.Whenwediscussedtheword‘knowledge’,wedividedbetweentacitandexplicitknowledge.NonakaandTakeuchiclaimsthatknowledgeisconstantlyconvertedfromtacittoexplicitandbackagainasitpassesthroughanorganisation.Theysaythatknowledgecanbeconvertedfromtacittotacit,fromtacittoexplicit,orfromexplicittoeithertacitorexplicitknowledgeasshowninFig.1Wenowdescribeeachofthesefourmodesofconversion:Socializationmeanstotransfertacitknowledgetotacitthroughobservation,imitationandpractice,whathasbeenreferredtoas‘onthejob’training.Craftsmanshiphasusuallybeenlearnedinthisway,whereoralcommunicationiseithernotusedorjustplaysaminorInternalisationistotakeexternalisedknowledgeandconvertitintoindividualtacitknowledgeintheformofmentalmodelsortechnicalknow-how.‘Documentsandmanualsfacilitatethetransferofexplicitknowledgetootherpeople,therebyhelpingthemexperiencetheexperiencesofothersindirectly(i.e.‘re-experience’them)’.Externalisationmeanstogofromtacittoexplicitknowledge.Explicitknowledgecan‘taketheshapesofmetaphors,analogies,concepts,hypothesesormodels’.Thisconversionisusuallytriggeredbydialogueorcollectivereection,butcanalsobetheresultofindividualreection,forexampleinawritingprocess.Combinationistogofromexplicittoexplicitknowledge,thatis,tocombineandsystemizeknowledgefromdifferentsourcessuchasdocuments,meetings,telephoneconferencesandbulletinboards.Systematisingthiskindofexplicitknowledgeistorecongureitbysorting,adding,combiningorcategorisingtheknowledge.AccordingtoNonakaandTakeuchi,knowledgepassesthroughdifferentmodesofconversioninaspiralwhichmakestheknowledgemorerened,andalsospreadsitacrossdifferentlayersinanorganisation.Hansenetal.al.discussestwostrategiesforknowledgemanagement,onerelyingoncodication,theotherrelyingonsharingtacitknowledge,whattheycallpersonalisation.1.3.TheprojectasalearningarenaInsoftwareengineering,toreuselifecycleexperience,processesandproductsforsoftwaredevelopmentisoftenreferredtoashavingan‘ExperienceFactory’[3]—aseparateorganisationalentitywithresponsibilityforcapturingandreusingexperience.Thisapproachhasbeenmuchcitedinthesoftwareengineeringeld.Experienceiscollectedfromsoftwaredevelopmentprojects,andpack-agedandstoredinanexperiencebase.Bypackaging,wemeangeneralising,tailoringandformalisingexperiencesothatitiseasytoreuse.TheExperienceFactoryorganisationassistssoftwaredevelopingprojectswithearlierexperiencebothinupstartandduringexecution,andcansuggestimprovementsinthedevelopmentprocesses,basedoncollectedexperience.2.PostmortemreviewsWerstdenewhatwemeanbya‘postmortem’.Then,wedescribepostmortemreviewsfromthesoftwareengineeringliterature,beforepresentingsomemethodsforconductingpostmortemreviews.2.1.Whatisa‘postmortem’?Byapostmortem,wemeanacollectivelearningactivitywhichcanbeorganisedforprojectseitherwhentheyendaphaseorareterminated.Themainmotivationistoreectonwhathappenedintheprojectinordertoimprovefuturepractise—fortheindividualsthathaveparticipatedintheprojectandfortheorganisationasawhole.Thephysicaloutcomeofameetingisapostmortemreport.Thistypeofprocesseshasalsobeenreferredtoas‘projectretrospectives’,‘postmortemanalysis’, Fig.1.ConversionofknowledgeaccordingtoNonakaandTakeuchi.Wecanimagineknowledgegoingthroughallconversionprocessesinaspiralformasidevelopsinanorganisation.T.Dingsøyr/InformationandSoftwareTechnology47(2005)293–303 ‘postprojectreview’,‘projectanalysisreview’,‘qualityimprovementreview’,‘autopsyreview’,‘Santayanareview’,‘afteractionreviews’and‘touch-downmeetings’.Researchersinorganisationallearningsometimesusetheterm‘reectivepractice’,whichcanbedenedas‘thepracticeofperiodicallysteppingbacktoponderonthemeaningtoselfandothersinone’simmediateenvironmentaboutwhathasrecentlytranspired.Itilluminateswhathasbeenexperiencedbybothselfandothers,providingabasisforfutureaction’[17].Thisinvolvesuncoveringandmakingexplicitresultsofplanning,observationandachievedpractice.Itcanleadtounderstandingofexperi-encesthathavebeenoverlookedinpractice.ThetwotheoriesoflearningthatwepresentedinSection1.2putdifferentemphasisonthiskindoflearning.InthemodelofNonakaandTakeuchi,postmortemsareacombinationoflearningthroughsocialisationandthroughexternalisation.Inlisteningtoothersyouemploysocialisa-tionandinreectingandsharingyourownexperienceyouexternaliseyourtacitknowledge.Postmortemsarealsoamethodforleveragingknowledgefromtheindividualleveltotheorganisationallevel.Inacommunityofpractiseview,postmortemsareanarenafortheindividualtocontributewithknowledgetothecommunity,andalsoforthecommunitytodiscusschangesofpractiseonkeyareas.Inasurveyonessentialpractisesinresearchanddevelopment-companies,‘learningfrompost-projectaudits’areseenasoneofthemostpromisingpractisesthatcouldyieldcompetitiveadvantageadvantage.Asurveyonpost-projectreviewsinresearchanddevelopmentcompaniesshowthatonlyoneoutofveprojectsreceivedapost-projectreviewew.Also,thereviewstendtofocusontechnicaloutputandbureaucraticmeasurements.Process-relatedfactorsarerarelydiscussed.Asaknowledgemanagementandsoftwareprocessimprovementtool,postmortemreviewsaresimpletoorganise.Theprocessfocusesondialogueanddiscussionwhichisacentralelementinknowledgetransfer.VonKroghetal.writesthat“itisquiteironicthatwhileexecutivesandknowledgeofcerspersistinfocusingonexpensiveinformation-technologysystems,quantiabledatabases,andmeasurementtools,oneofthebestmeansforknowledgesharingandcreatingknowledgealreadyexistswithintheircompanies.Wecannotemphasizeenoughtheimportantpartconversationsplay”play”.Anexampleofpostmortemreviewsare‘afteractionreviews’conductedbytheUSarmysinceaftertheVietnamwar,focusingona‘professionaldiscussionofanevent’toprovideinsight,feedbackanddetailsabouttheeventevent.Kransdorfffcriticizespostmortemsbecausepeopleparticipatingdonothaveanaccuratememory,whichcanleadtodisputes.Hesuggestscollectingdataduringtheproject,forexamplethroughshortinterviews,inanefforttogetmoreobjectivematerial.2.2.PostmortemreviewsinsoftwareengineeringThereareseveralwaystoperformPostmortemReviews.Applehasusedamethodmethodwhichincludesdesigningaprojectsurvey,collectingobjectiveprojectinformation,conductingadebriengmeeting,a‘projecthistoryday’andnallypublishingtheresults.AtMicrosofttheyalsoputmucheffortintowriting‘Postmortemreports’.Thesecontaindiscussionon‘whatworkedwellinthelastproject,whatdidnotworkwell,andwhatthegroupshoulddotoimproveinthenextproject’’.Thesizeoftheresultingdocumentisquitelarge,‘groupsgenerallytake3–6monthstoputapostmortemdocumenttogether.Thedocumentshaverangedfromunder10tomorethan100pages,andhavetendedtogrowinlength’.Inabookaboutteamsoftwaredevelopment,WattsHumphreysuggestsawaytodopostmortemsto‘learnwhatwentrightandwrong,andtoseehowtodothejobbetterthenexttime’time’.Adescriptionofanotherlightweightapproachwhichseekstoelicitexperienceusinginterviews,andnotagroupprocess,isdescribedbySchneiderSchneider.NormanKerthlistsatotalof19techniquestobeusedinpostmortemsms,somefocusingoncreatinganatmospherefordiscussionintheproject,someforreviewingthepastproject,someforhelpingateamidentifyandembracechangeduringtheirnextproject,andsomefordealingwiththeuniqueeffectsofafailedproject.Kerthrecommendsusingthreedaysinordertoeffectalastingchangeinthecompany.Tiedemanmansuggeststhreetypesofpostmortems,relatedtoawaterfallmodelofsoftwaredevelopment,onefor‘planning’,onefor‘design/verication’andone‘eldpostmortem’toprovidefeedbackafterthedevelopedsystemhasbeeninuseforsometime.TheGameDevelopermagazinepublishespostmortemsongamedevelopmentprojectsinmostissues,seeforexampleapostmortemonthegame‘AggressiveInline’Inline’.Thearticlescontainabriefdescriptionaboutthegamedevelopedandtheprojectorganisation,andthenusuallyveissuesthat‘wentright’andveissuesthat‘wentwrong’.2.3.MethodsforconductingpostmortemreviewsWenowpresentthreemethodsforconductingpost-mortemreviewsfromtheliterature.Wehaveselectedthreemethodsthatcanbeperformedinshorttime,andarethussuitableevenforsmallandmedium-sizecompanies.Theycanalsobeagoodstartforcompanieswantingmorein-depthmethodslater.NealWhittensuggeststhefollowingprocessforconductingpostprojectreviewsreviews:(1)Declareintent—theprojectheadshouldstatehisorherintentiontohaveapostprojectreviewatthecompletionT.Dingsøyr/InformationandSoftwareTechnology47(2005)293–303 oftheproject,byalettertoallprojectparticipants.Thelettershoulddescribethepostprojectreviewprocess.Selectparticipants—participantsfromeachmajorparticipatingorganisationsshouldbeselected:fromplanning,development,test,publications,performance,usability,modulebuildgroup,etc.Managersshouldnotparticipateinthepostprojectreviewteam,astheyarealsoevaluatingtheperformanceofpeople,andthismighthindertopicsfromsurfacingintheprocess.Prepareforworkshop—participantsareaskedtodohomeworkbeforetheworkshop:torespondtoasetofquestions,like‘Whatlevelofproductivitywasachievedforyourtasks?Howdiditcomparewithwhatyouexpected?’.Manyquestionscanbeaskedfromvariousareaslikestafng,missionobjectives,educationandtraining,tools,qualitytosupportfromoutsidegroups.Conductworkshop—theworkshopcanlastfromhalfadaytotwodays,andinclude:(a)10–30minpresenta-tionsoffeedbackonthequestionsfromeachpartici-pant.(b)Constructionofathingsthat‘wentright’listwiththemostbenecialitemsplacedatthetop.(c)Constructionofa‘wentwrong’listinpriorityorder.(d)Developproposalsthataddresstheproblems-eitheringroupsorcollectively.Presentresults—resultsoftheworkshoparerstpresentedtotheprojectleadership.Firstandsecondleveloftheprojectleadershipshouldatleastbeinvited.Secondly,theresultsarepresentedtoallparticipantsinameeting.Adoptrecommendations—apostprojectreviewreportiscompleted,whichincludesinformationfromthework-shopandrecommendationsfromtheprojectleadership.Thereportiseitherdistributedtoprojectleadersortoallpersonnel.Theprojectleadershipisresponsibleforactingonthecommittedrecommendations.CollisonandParcellParcellsuggestthefollowingstepsfororganisingaretrospectmeeting:Callthemeeting—holdthemeetingassoonaspossibleaftertheprojectends,andmakethemeetingaphysicalmeetingratherthanavideoconference.Invitetherightpeople—ifasimilarprojectisunder-way,invitethenewprojectteamalso.Theprojectleaderneedstoattend,aswellaskeymembersoftheproject.Inthecalltoattendees,announcethepurposeasto‘makefutureprojectsrunmoresmoothly,byidentifyingthelearningpointsfromthisproject’.Appointafacilitator—appointonethatisnotcloselyinvolvedintheproject,butwhoisoutsidetheline-managementstructure,asthemeetingistobeclearlyseparatefromanypersonalperformanceassessment.Revisittheobjectivesanddeliverablesoftheprojectndtheoriginalcriteriaforsuccess,andaskwhethertheprojectdeliveredthese.Revisittheprojectplanorprocess—incomplexprojects,itcanbeusefultoconstructaowchartofwhathappenedtoidentifytasks,deliverablesanddecisionpoints.Ask‘Whatwentwell’—ask“whatwerethesuccessfulstepstowardsachievingyourobjective?Whatwentreallywellintheproject?”.Ask“why”severaltimestoFindoutwhytheseaspectswentwell,andexpressthelearningasadviceforthefuture—identifythesuccessfactorsandbasefuturerecommendationsonagreedfacts.Thefacilitatorshouldpressforspecic,repeatableadvice.Thefacilitatorcaneitherorganizeaconversationthroughprobingquestions,oridentifyissuesandthenworkoneachasateam.Ask“whatcouldhavegonebetter”?—ask“whatweretheaspectsthatstoppedyoufromdeliveringevenmore?”.Startbyaskingtheprojectleader,thengoroundtheroom.Findoutwhatthedifcultieswere—identifystumblingblocksandpitfallstobeavoidedinthefuture.Ask“giventheinformationandknowledgewehavetoday,whatcouldwehavedonebetter?”Ensurethattheparticipantsleavethemeetingwiththeirfeelingsacknowledged—askpeopletoratetheproject:“lookingback,howsatisedwereyouwiththisproject,marksoutoften“.Followupbyasking“whatcouldhavemadeitatenforyou?”.‘Whatnext’—iftheteamisgoingstraightintoanewproject,itisusefultofollowtheretrospectwithaplanningsessionforthis.Recordingthemeeting—awell-structuredaccountofthemeetingcancontain:(a)guidelinesforthefuture,(b)historyfromtheprojecttoillustratetheguidelines,(c)namesofthepeopleinvolved,forfuturereference,and(d)anykeyartifacts(documents,projectplans).Usedirectquotestocapturethedepthoffeelingandtocreateasummarythatiseasilyread.Birketal.haveusedPostmortemReviewsasagroupprocess[27,32–35],wheremostoftheworkisdoneinonemeetinglastinghalfaday.Theytrytogetasmanyaspossibleofthepersonswhohavebeenworkingintheprojecttoparticipate,togetherwithtwoprocessconsultants,oneinchargeofthePostmortemprocess,theotheractingasasecretary.Thegoalofthismeetingistocollectinformationfromtheparticipants,makethemdiscussthewaytheprojectwascarriedout,andalsotoanalysecausesforwhythingsworkedoutwellordidnotworkout.Afurtherdescriptionofthismethodcanbefoundinthe‘results’section.The‘requirements’forthisprocessisthatitshouldnottakemuchtimefortheprojectteamtoparticipate,anditshouldprovideaforumfordiscussingmostimportantexperiencefromtheproject,togetherwithananalysisofthisexperience.Themainndingsaredocumentedinareport.T.Dingsøyr/InformationandSoftwareTechnology47(2005)293–303 Allparticipantsinaprojectareinvitedtoahalf-daypostmortemmeetingwithoutanyrequirementsforpre-paration.Birketal.usetwotechniquestocarryoutthePostmortemReview.Forafocusedbrainstormonwhathappenedintheproject,atechniquenamedafteraJapaneseethnologist,JiroKawakitaita—called‘theKJMethod’isused.Foreachofthesesessions,theparticipantsaregivenasetofpost-itnotes,andaskedtowriteone‘issue’oneachnote.Fivenotesarehandedouttoeachperson.Aftersomeminutes,theparticipantsareaskedtoattachonenotetoawhiteboardandsaywhythisissuewasimportant.Thenthenextpersonpresentsanoteandsoonuntilallthenotesareonthewhiteboard.Thenotesarethengrouped,andeachgroupisgivenanewname.RootCauseAnalysis,alsocalledIshikawaorshbone-diagramsareusedtoanalysethecausesofimportantissues.Theprocessleaderdrawsanarrowonawhiteboardindicatingtheissuebeingdiscussed,andattachotherarrowstothisonelikeinashbonewithissuestheparticipantsthinkarecausingtherstissue.Sometimes,alsounderlyingreasonsforsomeofthemaincausesareattachedaswell.Thepostmortemmeetinghasfollowingsteps:.First,theconsultantsintroducedtheagendaofthedayandthepurposeofthepostmortemKJsession1.Consultantshandoutpost-itnotesandaskpeopletowritedownwhatwentwellintheproject,hearpresentations,grouptheissuesonthewhiteboard,andgivethempriorities.KJsession2.Consultantshandoutpost-itnotesandaskedpeopletowritedownproblemsthatappearedintheproject,hearpresentations,grouptheissuesonthewhiteboard,andgivethempriorities.Rootcauseanalysis.Theprocessconsultantleadingthemeetingdrawssh-bonediagramsforthemainissues,bothfromthethingsthatwentwellandthethingsthatwereproblematic.Birketal.useataperecorderduringthepresentations,andtranscribeeverythingthatissaid.Theconsultantswriteapostmortemreportabouttheproject,whichcontainanintroduction,ashortdescriptionoftheprojectanalysed,howtheanalysiswascarriedout,andtheresultsoftheanalysis.Theresultisaprioritisedlistofproblemsandsuccessesintheproject.Statementsfromthemeetingareusedtopresentwhatwassaidabouttheissueswithhighestpriority,togetherwithashbonediagramtoshowtheircauses.Inanappendix,everythingthatwaswrittendownonpost-itnotesduringtheKJsessionisincluded,aswellasatranscriptionofthepresentationoftheissuesthatwereusedonthepost-itnotes.Suchreportsareusuallybetween10and15pagesinlength.Thedayafterthemeeting,theconsultantspresentthereporttothepeopleinvolvedintheprojecttogatherfeedbackanddominorcorrections.3.Case:postmorteminamedium-sizedcompanyAbove,wehaveseendifferentapproachestoconductingpostmortemreviews.Inordertogetatbetterunderstandingofsuchreviews,wenowpresentresultsfromonereview.First,wepresentthecompany,thentheprojectonwherethereviewwascarriedout,andnallyextractsfromthepostmortemreport.Thecasereportedherewasselectedbecauseofawidedatacollectionasapartofanactionresearchhprojectonsoftwareprocessimprovement.Allwrittenmaterialfromthepostmortemmeetingwasphotographed,anddiscussionswererecordedontapeandtranscribed.Intheproject,researchersandindustryparticipantscollectivelydiscussedproblems,identiedpossiblesolutions,triedoutasolutionandtogetherreectedontheresults.3.1.AsatellitesoftwarecompanyThecompanymakessoftwareandhardwareforstationsreceivingdatafrommeteorologicalandEarthobservationsatellites.Sincethecompanywasfoundedin1984,theyhavedeliveredturnkeygroundstationsystems,consultancyser-vices,feasibilitystudies,systemengineering,training,andsupport.Thecompanyhasbeenworkingwithlargedevelop-mentprojects,bothasaprimecontractorandasasubcon-tractor.Thecompanypossessastableandhighlyskilledstaff,manywithmaster’sdegreesinComputerScience,Math-ematicsorPhysics,andhavean‘engineeringculture’.Approximately60peopleareworkinginthecompany,andthemajorityisworkingwithsoftwaredevelopment.ProjectsaremanagedinaccordancewiththeEuropeanSpaceAgencyPSS-05standards,andareusuallyxedpriceprojects.Thecompanyhadproblemswithestimatingthesizeofnewsoftwareprojects.Manypeopleinthecompanyalsofeltthattheydidnottransferenoughexperiencebetweentheirsoftwaredevelopmentprojects.Everyprojectwrotean‘experiencereport’,butthesewereseldomconsideredinteresting,andwerenotreadveryoften.Toimprovethis,thecompanydecidedtotrypostmortemreviewsattheendofprojects.3.2.TheprojectWeorganisedapostmorteminoneproject,whichhaddevelopedasoftwaresystemforasatellitethatwasrecordingenvironmentaldata.Theprojecthaddevelopedamodulethatwastoanalysedatafromthissatellite,fromEuropeanSpaceAgencyspecications.Thiswasacriticalprojectforthecompany,asitwastherstinalineofnewservices.Theprojectlasted36months,andemployedfourpeopleintheanalysisphase,8–12peopleinthedesignphase,and5–9peopleintesting.Theprojectspentatotalof47,000workhours.Thevepeopleinacore-teamparticipatedinthepostmortemreview,includingtheprojectmanager.Thiswasthersttimethepeopleintheprojecthadparticipatedinapostmortemmeeting.T.Dingsøyr/InformationandSoftwareTechnology47(2005)293–303 3.3.ThepostmortemWeorganisedthepostmortemasdescribedbyBirketal.inSection2.3,andwillnowpresentsomeoftheresults.Becausetheparticipantsinthepostmortemmeetingkneweachotherwell,westartetwithabriefintroduction,followedbyaKJbrainstormsessiontoidentifyissuesthatwentwell.OneresultfromtheKJsessiononproblemsthatappearedintheproject,wasthreepost-itnotesgroupedtogetherandnamed‘changingrequirements’.TheyareshownintheupperleftcornerofFig.2.Whenpresentingthesenotes,participantsgavethefollowingstatementsfortwoofthenotes:“Anotherthingwaschangesofrequirementsduringtheproject:frommypointofview—whoimplementedthings,itwasdifculttodecide:whenaretherequire-mentschangedsomuchthatthingshavetobemadefromscratch?Somewrongdecisionsweretakenthatreducedthequalityofthesoftware”. Fig.2.Post-itnotesshowingsomeoftheproblemsinasoftwaredevelopmentproject,afteraKJsession.Thenotesaregroupedthematically.Eachgroupwaslatergivenanewname,forexamplethethreenotesintheupperleftcornerwerenamed‘changingrequirements’. Fig.3.Ishikawadiagramshowingmainandsub-causedfor‘changingRequirements’.Forexample,participantsinthepostmortemmeetingthoughtthatchangingrequirementswaspartlyaproblembecauseofapoororiginalspecicationfromthecutstomer.Thespecicationwaspoorbecauserequirementswereincomplete,containedfaults,werevagueanduntestable.T.Dingsøyr/InformationandSoftwareTechnology47(2005)293–303 “Unclearcustomerrequirements—whichmadeususealotoftimeindiscussionsandmeetingswiththecustomertogetthingsright,whichmadeusspendalotoftimebecausethecustomerdidnotdogoodenoughwork.”Whenwelaterbroughtthisissueupagaininordertondsomeoftherootcausesfor‘changingrequirements’,weendedupwiththeshbonediagraminFig.3Therootcausesforthechangingrequirements,asthepeopleparticipatingintheanalysissawit,wasthattherequirementswerepoorlyspeciedbythecustomer,therewere‘newrequirements’duringtheproject,andthecompanyknewlittleofwhatthecustomerwasdoing.Anotherreasonforthisproblemwasthatdocumentsrelatedtorequirementsweremanagedpoorlywithinthecompany.Fig.3,wehavealsolistedsomesubcauses.Afterthepostmortemmeetingwasnished,weaskedpeopletostatewhattheythoughtoftheprocess.Allparticipantshadgotnewinsightsontheproject—wereabletoseeissuesfromnewperspectives.Also,manystatedthatthewayofconductingpostmortemwasmotivatinginitselfbecauseitwasunusual(theirnormalworkdaywouldbetodevelopsoftwareincellofcesandattendnormalmeetings).Giventhetimerestrictionstouseonlyhalfaday,wedidnotgiverecommendationstomanagementinthecompany,otherthanstatingthesuccessesandproblemsintheproject.Thereportwaslaterdiscussedinameetingwhereallprojectmanagersinthecompanywereinvited,wheretheydiscussedchangesinhowprojectswerecarriedoutbasedonwhatwaslearnedinthisproject.4.SummaryanddiscussionInastudyin19companiesacrossEuropeinindustriessuchasmanagementconsulting,engineering,constructionandtelecommunicationsonproject-basedlearningprac-tices,KeeganandTurnerTurnerfoundthat‘projectteammembersfrequentlydonothavethetimeformeetings,orforsessionstoreviewlessonslearned.Often,projectteammembersareimmediatelyreassignedtonewprojectsbeforetheyhavehadtimeforlessonslearnedsessionsorafteractionreviews’.Theydidnotndasinglecompanywhereemployeesexpressedsatisfactionwiththepostmortemprocess.KeeganandTurnerdonotdiscusswhatkindofpostmortemprocessesexistedinthecompanies,butthemainndingwasthattheprocesseswereseldomusedinpractice.Wethinkthereisaneedforhelpingsoftwarecompanieschoosingsimpleandpracticalmethodsforconductingpostmortems,tomakeiteasiertoperformpostmortemstoahigherdegree.Thebenetofconductingpostmortemreviewsaremainlythatitprovidesalearningforumwherediscussionsarerelevanttotheprojectandtothecompany.Itcanalsobeawayformanagementtoshowthattheylistentowhattheemployeessay,andarewillingtodiscussimprovementefforts.Wewillnowdiscussthethreeapproachesdescribed(someofthediscussionpointsaresummarizedinTable1moreindepth.Inthedicsussion,weusethematerialfromthemedium-sizedsatellitesoftwarecompanyforexamples.4.1.RequirementsforagoodpostmortemprocessOpenness,patience,theabilitytolisten,experimentationwithnewwordsandconcepts,politeness,theformationofapersuasiveargumentandcouragearesomeingredientsforagooddiscussionssion.Inapostmortemthisissoughtbyhavingaskilledprocessleaderwhoencouragesopendialogue,andshouldpreventcritiqueofindividualsandthatdominatingpeoplegetthemostofthemeetingtime.NormanKerthKerthemphasizestheimportanceofagoodatmospherethroughthe‘primedirective’:“Regardlessofwhatwediscover,wemustunderstandandtrulybelievethateveryonedidthebestjobheorshecould,givenwhatwasknownatthetime,hisorherskillsandabilities,theresourcesavailable,andthesituationathand”.Forlongerpostmortems,exercisessuchas‘createsafety’and‘under-understandingdifferencesinpreferences’nces’canbeusedtofurtherfocusoncreatingagoodatmosphere.Inthesatellitesoftwarecase,weonlyusedashortintroduction,asteammemberskneweachotherwellfromworkingcloselyforalongperiodoftime.Participantsspent5heach,atotalof25h.Twofacilitatorsspenttenhourseach,givingatotaltimeexpenseof45h,whichislessthan0.1%ofthetotaltimespentontheproject.Itisdifculttogiveadviceonwhatisthe‘optimal’timeusageforapostmortem.Moretimewillallowmoreissuestobediscusseddeeper,thusincreasingthelearningeffect.The Table1SummaryofselecteddifferencesbetweenthreemethodsforconductingpostmortemreviewsWhittenCollisonandParcellBirketal.Whotoinvite?FromeachmajorparticipatingAllprojectmembers,possiblynewprojectAllprojectmembersHomework?YesNoNoTypeofdiscussion?OpenOpenStructuredOutput?RecommendationsGuidelinesHistoriesNamesofpeopleKeyStructuredreportwithissuesthatwentwellandcouldbebetterT.Dingsøyr/InformationandSoftwareTechnology47(2005)293–303 timeusedforapostmortemshoulddependonwhatkindofstrategyacompanyhas—codicationrequiresmoreworkthanpersonalisation.Itshouldalsodependonthesizeoftheproject,asthereshouldbemoreissuestodiscussinalargerprojectthaninasmall.4.2.WhotoinviteThethreemethodsdescribedallarguethatoneshouldinviteabroadaudienceforapostmortem.Whittenmentionsparticipantsfromplanning,development,test,usabilityandmodulebuildasexamplerolestoinvite.CollisonandParcellsuggeststhatpeoplefromsimilarprojectsthatareunderwayshouldbeinvitedaswellaskeymembersfromtheproject.ThemethodofBirketal.recommendsgetting‘asmanypeopleaspossible’fromtheprojecttoparticipate.Lookingbackatthemethodsforknowledgesharing,itseemsreasonablethatallparticipantsinaprojectcancontributewithknowledgethatisrelevantforfutureprojectsthroughsocialisation.Invitingmanypeoplecanalsobroadenexistingcommunitiesofpractisewithinanorganisation,especiallyifpeoplefromnewprojectsarealsoinvited.Ifthepostmortemisconductedasalightweightprocess,thecostwillnotbehigh.Invitingexternalstakeholderssuchasacustomerwillmovethefocusfrominternaleventstostakeholderrelations.Thismakesitdifculttoblamestakeholdersthatarenotpresentinthepostmortemmeeting,likethepeopleinthesatellitesoftwarecompanypartlyblamethecustomerforapoorrequirementspecicationandmanagementforgivingintocustomerdemandstooeasily.4.3.Withorwithouthomework?Shouldapostmorteminclude‘homework’fortheattendants?Whittenrecommendsthatallattendeesgothroughasetofquestionstopreparethemselvesfortheworkshop.CollisonandParcelldonotputemphasisonhomework,neitherdoBirketal.Areasonfordoinghomeworkisthatthelearningprocessistakenoveralongerperiodoftime.Peoplewhopreparecanalsoeasiercontributeinanopenmeetingsession.InthemethodsuggestedbyBirketal.,allparticipantsaregiventimeforreectionduringtheworkshop,toidentifymainsuccessesandproblems.Givenahighnumberofparticipants,thereisahighprobabilitythatthemostimportantissuesaredealtwith,evenwithouthomework.Buthomeworkcanstimulateindividualreection,externalisation,butwillalsorequiremoretime.Anotherquestioniswhetherthefacilitatorshoulddohomework.WiththemethodofBirk.etal.,thefacilitatordoesnotneedtoknowmuchabouttheproject,asthemainintentionistousetechniquestogettheparticipantstoreect.However,ifthefacilitatorstyleismoreintrusive,askingquestionsthatistostimulatereection,preparationisnecessary.Inthesatellitesoftwarecase,theattendantsdidnotdoanyhomework,andthefacilitatorshadlittleinformationabouttheproject—onlyashortdiscussionwiththeprojectmanagerbeforethepostmortemreviewmeeting.4.4.FacilitatorAllmethodsrecommendusingafacilitatorforthemeeting.Thequestioniswhatkindofpersonistherighttouse.Theprojectmanagercanbeoneoption,butthispersonissomuchinvolvedintheprojectthatitcanbedifculttoalloweveryonetoexpressopinionswithoutcommenting.Also,issuesthatpeoplethinkcanbesensitivetotheprojectmanagermightnotappear.Itisprobablywisetousesomeonefromoutsideoftheproject,whomtheparticipantstrust.Itcanalsobesomeonewhoisexternaltothecompany.Abenetofusinganexternalpersonisthatparticipantshavetoexplainissuestothispersonmorethoroughlythantheywouldtoaninsider.Thiscancausedifferentinterpretationswithintheprojecttobeuncovered.Thefacilitatorshouldalsobeproperlytrainedinordertofollow-upwhenstatementsfrompeopleareunclear.4.5.Openorstructureddiscussion?Anotherquestioniswhethertohaveanopenorstructureddiscussionoftheexperiencefromtheproject.AnopendialogueassuggestedbyWhittenisseenasacentrallearninginstrumentintheworksofSenge.However,thisformcaneasilytakealotoftime,andmightfocusonalimitednumberofissues.Itmightalsobethattheseissuesareonlyinterestingtothemostdominantpeopletakingpartinthemeeting.Birketal.areusingtheKJmethodinordertogiveeachparticipantthepossibilitytoinuenceequallyonthetopics.TheKJmethodisequallystrongwhethertheparticipantspreferthinkingaboutideasthroughquietintrospectionorinteractivebrainstorming.AdrawbackwiththeKJmethodcanbethatittakestimetoreachconsensusonnewnamesforgroupsofissues.Inthecasewiththesatellitesoftwarecompany,however,thiswasnottime-consuming.Thismightbebecausewedidnotencouragediscussionaboutwhetheranissuewasaproblemornot,wefocusedondiscussing‘whatwouldbeapropernameforasetofissuesthatsomeparticipantsfeltwereimportant’.4.6.Withorwithoutmanagement?Shouldthemanagementortheprojectmanagertakepartduringapostmortem?Wedonotthinkthemanagementshouldtakepartinthepostmortem,astheintentionistofocusonlearning,andmanagementalsohasaroleofevaluatingemployees.Thiscanbeaproblemaswesawinthesatellitesoftwareexample,wheremanagementwasblamedforsomeoftheproblemsintheproject.ButthisT.Dingsøyr/InformationandSoftwareTechnology47(2005)293–303 kindofproblemscanbediscussedwithmanagementafterthepostmortemmeetingisover.Theprojectmanagerisveryusefultoincludebecausethispersonhasamoreoverallviewoftheprojectthantherestoftheparticipants.Butthispersoncanalsobequicktodefendalldecisionstakenduringtheproject,andmakeitdifculttohaveafreeexchangeofideasonhowtoimprovethenextproject.Givenastrongfacilitatorthatisawareofthepossibleproblemswiththeprojectmanager,wethinkaprojectmanagershouldbeinvitedtogetamorecompleteoverviewinthepostmortem.4.7.Whatshouldbeoutput?Whatshouldtheoutputofapostmortembe?Whittendescribesalistofrecommendationsthataregiventothecompanymanagementinordertoensurelearninginotherprojects.CollisonandParcellalsomentionsuchguidelinesforthefuture,butalsomentionhistoriestoillustratetheguidelines,namesofpeopleinvolvedandkeyartifacts.Theyalsorecommendusingdirectquotestocapturethedepthoffeelingandtocreateasummarythatiseasilyread.Birketal.suggestswritingareportwhichdescribestheproject,whatwentwell,whatwentwrong,andthecausesofwhatwentwellandwrong.Theyalsotranscribemuchofwhatissaidduringthemeetinginordertogivemorecontextforfuturereaders.Iftheintentionofthepostmortemmainlyistocomeupwithimprovementsuggestions,probablythemethoddescribedbyWhittenissufcient.Butiftheintentionistotransferknowledgealsotopeoplewhodidnottakepartinthepostmortem,themethodofBirketal.ismoreappropriate.Therearemanyexamplesofpostmortemreportsnotbeingused.KerthKertharguesthattheparticipantsinthepostmortemmeetingshouldwritethereport,otherwisetheyloosecommitmenttothecontent.TheCross-Afnitytyproducesproposalsforchange,whichidentiespeoplewillingtoworkonthechange.4.8.Learningfocus:tacitorexplicitknowledge?Anarearelatedtothepreviousdiscussioniswhatkindofknowledgetransferisintendedfromthepostmortems.IfwegobacktothetwostrategiessuggestedbyHansenetal.,wecanviewpostmortemsassupportingpersonalisationinthatitprovidesanarenafor‘reectivepractice’whereparticipantscandiscusspastevents.Fromacommunityofpractice-view,apostmortemcanbeonearenatoengageinandtocontributetothecommunity.Themainaimofthepostmortemistodiscusschangesthatwillleadtorenedpractice.Wecanalsoseepostmortemsasanattempttocodifyknowledgefromprojects,wherethemainoutputisthereport,whichshouldprovideinsighttootherprojectteams(asapartofsystematicallycapturing,storing,interpretinganddistributingrelevantexperiencefromprojectsasseenasanimportantlearningmechanismbyHuberHuber).Howpostmortemsareusedshoulddependonwhatstrategythecompanyhas.Smallercompaniesshouldfocusonsharingtacitknowledge,asacodicationstrategyisexpensive.Largercompaniesaremoredependentofcodiedknowledge,andshouldinvestmoreinthedocumentation.5.ConclusionandfutureworkWehaveinvestigatedpostmortemreviewsfromaknowledgemanagementperspective,andpresentedthreemethodsforconductingpostmortemsfromtheliterature.Wehavealsopresentedexampleresultsfromapostmortemreport.Themethodsvaryinseveraldimensions.Theyputdifferentemphasisonwhotoinvite,howtoprepare,howtofacilitatethepostmortemmeeting,howtostructurediscus-sions,andwhatthewrittenoutputofthepostmortemistobe.Companieswantingtoconductpostmortemsshoulddecideonthemethodtouseafterwhatgeneralstrategytheyhaveforknowledgemanagement.Theyshouldalsodecidewhethertheywanttofocuspurelyoninternalprojectaffairs,oralsotoincluderelationstoprojectstakeholders.Ageneraladviceistousepeoplewhoarenotdirectlyinvolvedintheprojecttofacilitatethepostmortemmeeting.AcknowledgementsIamgratefultomanypeopleforhavingdiscussionsonpostmortemreviews,particularlytheresearchgroupatSINTEFICT:NilsBredeMoe,ToreDyba,GeirKjetilHanssen,HansWesterheimandTorErlendFægri.IwouldfurtherliketothankTorStalhaneattheNorwegianUniversityofScienceandTechnology,KevinDesouzafromtheUniversityofIllinois,MaxvonZedtwitzfromTsinghuaUniversityandespeciallyNormanKerthforcommentsonthisarticle.ThisworkwasconductedthroughtheSoftwareProcessImprovementbasedonKnowledgeandExperience(SPIKE)project,supportedbytheResearchCouncilofNorway.References[1]V.R.Basili,Quantitativeevaluationofsoftwareengineeringmethod-ology,ProceedingsoftheFirstPanPacicComputerConference,[2]M.C.Paulk,C.V.Weber,B.Curtis,M.B.Chrissis,TheCapabilityMaturityModel:GuidelinesforImprovingtheSoftwareProcess,Addison-Wesley,Boston,1995.T.Dingsøyr/InformationandSoftwareTechnology47(2005)293–303 [3]V.R.Basili,G.Caldiera,H.D.Rombach,Theexperiencefactoryin:J.J.Marciniak(Ed.),EncyclopediaofSoftwareEngineering,Wiley,NewYork,1994,pp.469–476.[4]D.Garvin,Buildingalearningorganization,HarvardBusinessReview1993;78–91.[5]G.Huber,Organizationallearning:aguideforexecutivesintechnology-criticalorganizations,InternationalJournalonTechnol-ogyManagement,SpecialIssueonUnlearningandLearningforTechnologicalInnovation11(1996)821–832.[6]E.Wenger,CommunitiesofPractise:Learning,MeaningandIdentity,CambridgeUniversityPress,Cambridge,UK,1998.[7]P.M.Senge,TheFifthDiscipline:theArtandPractiseofTheLearningOrganisation,CenturyBusiness,1990.[8]Webster’sEncyclopedicUnabridgedDictionaryoftheEnglishLanguage,GramercyBooks,NewYork,1989.[9]R.Stata,Organizationallearning:thekeytomanagementinnovationin:K.Starkey(Ed.),HowOrganizationsLearn,ThomsonBusinessPress,London,1996,pp.316–334.[10]C.Argyris,D.A.Schon,OrganizationalLearningII:Theory,MethodandPractise,AddisonWesley,Reading,MA,1996.[11]C.Argyris,OvercomingOrganizationalDefences:FacilitatingOrganizationalLearning,PrenticeHall,EnglewoodCliffs,NJ,1990.[12]R.L.Feldmann,K.-D.Althoff,Onthestatusoflearningsoftwareorganisationsintheyear2001,LearningSoftwareOrganizationsWorkshop,2001,pp.2–6.[13]T.Dyba,Enablingsoftwareprocessimprovement:aninvestigationontheimportanceoforganizationalissues,Dr.ingthesis,DepartmentofComputerandInformationScience,NorwegianUniversityofScienceandTechnology,Trondheim,2001,pp.332,ISBN82-471-5371-8.[14]J.Lave,E.Wenger,SituatedLearning,CambridgeUniversityPress,Cambridge,NewYork,1991.[15]I.Nonaka,H.Takeuchi,TheKnowledge-CreatingCompany,OxfordUniversityPress,NewYork,1995.[16]M.T.Hansen,N.Nohria,T.Tierney,What’sYourStrategyforManagingKnowledge?,in:HarvardBusinessReviewonOrganiz-ationalLearning,HarvardBusinessSchoolPress,Boston,1994,pp.[17]J.A.Raelin,Publicreectionasthebasisoflearning,ManagementLearning32(2001)11–30.[18]M.M.Menke,ManagingR&Dforcompetitiveadvantage,ResearchTechnologyManagement40(1997)40–42.[19]M.Zedtwitz,Organizationallearningthroughpost-projectreviewsinR&D,R&DManagement32(2002)255–268.[20]G.v.Krogh,K.Ichijo,I.Nonaka,EnablingKnowledgeCreation,OxfordUniversityPress,NewYork,2000.[21]P.L.Townsend,J.E.Gebhart,HowOrganizationsLearn,CrispPublications,1999.[22]A.Kransdorff,Usingthebenetsofhindsight—theroleofpost-projectanalysis,TheLearningOrganization3(1996)11–15.[23]B.Collier,T.DeMarco,P.Fearey,Adenedprocessforprojectpostmortemreview,IEEESoftware13(1996)65–72.[24]M.A.Cusomano,R.W.Selby,MicrosoftSecrets—HowtheWorld’sMostPowerfulSoftwareCompanyCreatesTechnology,ShapesMarkets,andManagesPeople,TheFreePress,1995.[25]W.S.Humphrey,Thepostmortem,in:IntroductiontotheTeamSoftwareProcess,AddisonWesleyLongman,Reading,MA,1999,pp.185–196.[26]K.Schneider,LIDs:alight-weightapproachtoexperienceelicitationandreuse,SecondInternationalConferenceonProductFocusedSoftwareProcessImprovement,PROFES2000,2000,pp.407–424.[27]N.L.Kerth,Projectretrospectives:ahandbookforteamreviews,DorsetHousePublishing,NewYork,2001.[28]M.J.Tiedeman,Post-mortems—methodologyandexperiences,IEEEJournalofonSelectedAreasinCommunications8(1990).[29]R.Condon,Postmortem:-axis’saggressiveinline,GameDeveloper2002;42–49.[30]N.Whitten,ManagingSoftwareDevelopmentProjects:FormulaforSuccess,Wiley,NewYork,1995.[31]C.Collison,G.Parcell,LearningtoFly:PracticalLessonsfromoneoftheWorld’sLeadingKnowledgeCompanies,CapstonePublication,[32]A.Birk,T.Dingsøyr,T.Stalhane,Postmortem:Neverleaveaprojectwithoutit,IEEESoftware,SpecialIssueonKnowledgeManagementinSoftwareEngineering19(2002)43–45.[33]T.Dingsøyr,N.B.Moe,Ø.Nytrø,Augmentingexperiencereportswithlightweightpostmortemreviewsin:F.Bomarius,S.Komi-Sirvio(Eds.),ThirdInternationalConferenceonProductFocusedSoftwareProcessImprovement,Springer,Kaiserslautern,Germany,2001,pp.167–181.[34]T.Stalhane,T.Dingsøyr,N.B.Moe,G.K.Hanssen,Postmortem—anassessmentoftwoapproaches,EuroSPI,2001,pp.[35]T.Dyba,T.Dingsøyr,N.B.Moe,ProcessImprovementinPractice—aHandbookforitCompanies,Kluwer,Boston,2004.[36]R.Scupin,TheKJMethod:atechniqueforanalyzingdataderivedfromJapaneseethnology,HumanOrganization56(1997)233–237.[37]D.J.Greenwood,M.Levin,IntroductiontoActionResearch,Sage,BeverlyHills,CA,1998.[38]A.Keegan,J.R.Turner,Quantityversusqualityinproject-basedlearningpractises,ManagementLearning32(2001)77–98.[39]G.P.Huber,Organizationallearning:thecontributingprocessesandtheliteratures,OrganizationalScience2(1991)88–115.T.Dingsøyr/InformationandSoftwareTechnology47(2005)293–303