Efcient Data Structures for TamperEvident Logging Scot - PDF document

EfcientDataStructuresforTamper-EvidentLoggingScottA.CrosbyDanS.Wallachscrosby@cs.rice.edudwallach@cs.rice.eduDepartmentofComputerScience,RiceUniversityAbstractManyreal-worldapplicationswishtocollecttamper-evidentlogsforforensicpurposes.Thispaperconsidersthecaseofanuntrustedlogger,servinganumberofclientswhowishtostoretheireventsinthelog,andkepthonestbyanumberofauditorswhowillchallengetheloggertoproveitscorrectbehavior.Weproposesemanticsoftamper-evidentlogsintermsofthisauditingprocess.Theloggermustbeabletoprovethatindividualloggedeventsarestillpresent,andthatthelog,asseennow,isconsistentwithhowitwasseeninthepast.Toaccomplishthisefciently,wedescribeatree-baseddatastructurethatcangeneratesuchproofswithlogarithmicsizeandspace,improvingoverpreviouslinearcon-structions.Whereaclassichashchainmightrequirean800MBtracetoprovethatarandomlychoseneventisinalogwith80millionevents,ourprototypereturnsa3KBproofwiththesamesemantics.Wealsopresentaexiblemechanismforthelogservertopresentauthenticatedandtamper-evidentsearchresultsforalleventsmatchingapredicate.Thiscanallowlarge-scalelogserverstoselectivelydeleteoldevents,inanagreed-uponfashion,whilegeneratingefcientproofsthatnoinappropriateeventsweredeleted.Wedescribeaprototypeimple-mentationandmeasureitsperformanceonan80millioneventsyslogtraceat1,750eventspersecondusingasingleCPUcore.Performanceimprovesto10,500eventspersecondifcryptographicsignaturesareofoaded,correspondingto1.1TBofloggingthroughputperweek.1IntroductionThereareover10,000U.S.regulationsthatgovernthestorageandmanagementofdata[22,58].Manycountrieshavelegal,nancial,medical,educationalandprivacyregulationsthatrequirebusinessestoretainavarietyofrecords.Loggingsystemsarethereforeinwideuse(albeitmanywithoutmuchinthewayofsecurityfeatures).Auditlogsareusefulforavarietyofforensicpurposes,suchastracingdatabasetampering[59]orbuildingaversionedlesystemwithveriableaudittrails[52].Tamper-evidentlogshavealsobeenusedtobuildByzan-tinefault-tolerantsystems[35]andprotocols[15],aswellastodetectmisbehavinghostsindistributedsystems[28].Ensuringalog'sintegrityisacriticalcomponentinthesecurityofalargersystem.Malicioususers,includingin-siderswithhigh-levelaccessandtheabilitytosubverttheloggingsystem,maywanttoperformunloggedactivitiesortamperwiththerecordedhistory.Whiletamper-resistanceforsuchasystemmightbeimpossible,tamper-detectionshouldbeguaranteedinastrongfashion.Avarietyofhashdatastructureshavebeenproposedintheliteratureforstoringdatainatamper-evidentfashion,suchastrees[34,49],RSAaccumulators[5,11],skiplists[24],orgeneralauthenticatedDAGs.Thesestructureshavebeenusedtobuildcerticaterevocationlists[49],tobuildtamper-evidentgraphandgeometricsearching[25],andauthenticatedresponsestoXMLqueries[19].Allofthesestorestaticdata,createdbyatrustedauthorwhosesignatureisusedasaroot-of-trustforauthenticatingresponsesofalookupqueries.Whileauthenticateddatastructureshavebeenadaptedfordynamicdata[2],theycontinuetoassumeatrustedauthor,andthustheyhavenoneedtodetectinconsis-tenciesacrossversions.Forinstance,inSUNDR[36],atrustednetworklesystemisimplementedonuntrustedstorage.Althoughversionvectors[16]areusedtodetectwhentheserverpresentsforking-inconsistentviewstoclients,onlytrustedclientssignupdatesforthelesystem.Tamper-evidentlogsarefundamentallydifferent:Anuntrustedloggeristhesoleauthorofthelogandisrespon-sibleforbothbuildingandsigningit.Alogisadynamicdatastructure,withtheauthorsigningastreamofcommit-ments,anewcommitmenteachtimeaneweventisaddedtothelog.Eachcommitmentsnapshotstheentireloguptothatpoint.Ifeachsignedcommitmentistherootofanauthenticateddatastructure,well-knownauthenticateddictionarytechniques[62,42,20]candetecttamperingwithineachsnapshot.However,withoutadditionalmech-anismstopreventit,anuntrustedloggerisfreetohavedif-ferentsnapshotsmakeinconsistentclaimsaboutthepast.Tobesecure,atamper-evidentlogsystemmustbothde-tecttamperingwithineachsignedloganddetectwhendifferentinstancesofthelogmakeinconsistentclaims.Currentsolutionsfordetectingwhenanuntrustedserverismakinginconsistentclaimsovertimerequirelinearspaceandtime.Forinstance,topreventundetectedtampering,existingtamperevidentlogs[56,17,57]whichrelyuponahashchainrequireauditorsexamineeveryintermediateeventbetweensnapshots.Onepro-posal[43]foratamper-evidentlogwasbasedonaskiplist.Ithaslogarithmiclookuptimes,assumingthelog isknowntobeinternallyconsistent.However,provinginternalconsistencyrequiresscanningthefullcontentsofthelog.(SeeSection3.4forfurtheranalysisofthis.)Inthesamemanner,CATS[63],anetwork-storageservicewithstrongaccountabilityproperties,snapshotstheinternalstate,andonlyprobabilisticallydetectstamperingbyauditingasubsetofobjectsforcorrectnessbetweensnapshots.PavlouandSnodgrass[51]showhowtointegratetamper-evidenceintoarelationaldatabase,andcanprovetheexistenceoftampering,ifsuspected.Auditingthesesystemsforconsistencyisexpensive,requiringeachauditorvisiteachsnapshottoconrmthatanychangesbetweensnapshotsareauthorized.Ifanuntrustedloggerknowsthatajust-addedeventorreturnedcommitmentwillnotbeaudited,thenanytamperingwiththeaddedeventortheeventsxedbythatcommitmentwillbeundiscovered,and,bydenition,thelogisnottamper-evident.Topreventthis,atamper-evidentlogrequiresfrequentauditing.Tothisend,weproposeatree-basedhistorydatastructure,logarithmicforallauditingandlookupoperations.Eventsmaybeaddedtothelog,commitmentsgenerated,andauditsmaybeperformedindependentlyofoneanotherandatanytime.Nobatchingisused.Unlikepastdesigns,weexplicitlyfocusonhowtamperingwillbediscovered,throughauditing,andweoptimizethecostsoftheseaudits.Ourhistorytreeallowsloggerstoefcientlyprovethatthesequenceofindividuallogscommittedto,overtime,makeconsistentclaimsaboutthepast.InSection2wepresentbackgroundmaterialandpro-posesemanticsfortamper-evidentlogging.InSection3wepresentthehistorytree.InSection4wedescribeMerkleaggregation,awaytoannotateeventswithattributeswhichcanthenbeusedtoperformtamper-evidentqueriesoverthelogandsafedeletionofevents,allowingunneededeventstoberemovedin-place,withnoadditionaltrustedparty,whilestillbeingabletoprovethatnoeventswereimproperlypurged.Section5describesaprototypeimplementationfortamper-evidentloggingofsyslogdatatraces.Section6discussesapproachesforscalingthelogger'sperformance.RelatedworkispresentedinSection7.FutureworkandconclusionsappearinSection8.2SecurityModelInthispaper,wemaketheusualcryptographicassump-tionsthatanattackercannotforgedigitalsignaturesorndcollisionsincryptographichashfunctions.Further-morewearenotconcernedwithprotectingthesecrecyoftheloggedevents;thiscanbeaddressedwithexternaltechniques,mostlikelysomeformofencryption[50,26,54].Forsimplicity,weassumeasinglemonolithiclogonasinglehostcomputer.Ourgoalistodetecttampering.ItisimpracticaltopreventthedestructionoralterationofdigitalrecordsthatareinthecustodyofaByzantinelog-ger.Replicationstrategies,outsidethescopeofthispaper,canhelpensureavailabilityofthedigitalrecords[44].Tamper-evidencerequiresauditing.Ifthelogisneverexamined,thentamperingcannotbedetected.Tothisend,wedividealoggingsystemintothreelogicalentities—manyclientswhichgenerateeventsforappendingtoalogorhistory,managedonacentralizedbuttotallyuntrustedlogger,whichisultimatelyauditedbyoneormoretrustedauditors.Weassumeclientsandauditorshaveverylimitedstoragecapacitywhileloggersareassumedtohaveunlimitedstorage.Byauditingthepublishedcommitmentsanddemandingproofs,auditorscanbeconvincedthatthelog'sintegrityhasbeenmaintained.Atleastoneauditorisassumedtobeincorruptible.Inoursystem,wedistinguishbetweenclientsandauditors,whileasinglehostcould,infact,performbothroles.Wemusttrustclientstobehavecorrectlywhiletheyarefollowingtheeventinsertionprotocol,butwetrustclientsnowhereelse.Ofcourse,amaliciousclientcouldinsertgarbage,butwewishtoensurethatanevent,oncecorrectlyinserted,cannotbeundetectablyhiddenormod-ied,eveniftheoriginalclientissubsequentlycolludingwiththeloggerinanattempttotamperwitholddata.Toensurethesesemantics,anuntrustedloggermustregularlyproveitscorrectbehaviortoauditorsandclients.Incrementalproofs,demandedofthelogger,provethatcurrentcommitmentandpriorcommitmentmakeconsistentclaimsaboutpastevents.Membershipproofsasktheloggertoreturnaparticulareventfromthelogalongwithaproofthattheeventisconsistentwiththecurrentcommitment.Membershipproofsmaybedemandedbyclientsafteraddingeventsorbyauditorsverifyingthatoldereventsremaincorrectlystoredbythelogger.Thesetwostylesofproofsaresufcienttoyieldtamper-evidence.Asanyvanillalookupoperationmaybefollowedbyarequestforproof,theloggermustbehavefaithfullyorriskitsmisbehaviorbeingdiscovered.2.1SemanticsofatamperevidenthistoryWenowformalizeourdesiredsemanticsforsecurehistories.EachtimeaneventXissenttothelogger,itassignsanindexiandappendsittothelog,generatingaversion-icommitmentCithatdependsonalloftheeventsto-date,X0:::Xi.ThecommitmentCiisboundtoitsversionnumberi,signed,andpublished.Althoughthestreamofhistoriesthataloggercommitsto(C0:::Ci;Ci+1;Ci+2:::)aresupposedtobemutually-consistent,eachcommitmentxesanindependenthistory.Becausehistoriesarenotknown,apriori,tobeconsistentwithoneother,wewilluseprimes(0)todistinguishbetweendifferenthistoriesandtheeventscontainedwithinthem.Inotherwords,theeventsinlogCi(i.e.,thosecommittedbycommitmentCi)areX0:::Xi andtheeventsinlogC0jareX00:::X0j,andwewillneedtoprovetheircorrespondence.2.1.1MembershipauditingMembershipauditingisperformedbothbyclients,verifyingthatneweventsarecorrectlyinserted,andbyauditors,investigatingthatoldeventsarestillpresentandunaltered.TheloggerisgivenaneventindexiandacommitmentCj,ijandisrequiredtoreturntheithelementinthelog,Xi,andaproofthatCjimpliesXiistheitheventinthelog.2.1.2IncrementalauditingWhileaveriedmembershipproofshowsthataneventwasloggedcorrectlyinsomelog,representedbyitscommitmentCj,additionalworkisnecessarytoverifythatthesequenceoflogscommittedbytheloggerisconsistentovertime.Inincrementalauditing,theloggerisgiventwocommitmentsCjandC0k,wherejk,andisrequiredtoprovethatthetwocommitmentsmakecon-sistentclaimsaboutpastevents.AveriedincrementalproofdemonstratesthatXa=X0aforalla2[0;j].Onceveried,theauditorknowsthatCjandC0kcommittothesamesharedhistory,andtheauditorcansafelydiscardCj.Adishonestloggermayattempttotamperwithitshistorybyrollingbackthelog,creatinganewforkonwhichitinsertsnewevents,andabandoningtheoldfork.Suchtamperingwillbecaughtiftheloggingsystemsatiseshistoricalconsistency(seeSection2.3)andbyalogger'sinabilitytogenerateanincrementalproofbetweencommitmentsondifferent(andinconsistent)forkswhenchallenged.2.2ClientinsertionprotocolOnceclientsreceivecommitmentsfromtheloggeraf-terinsertinganevent,theymustimmediatelyredistributethemtoauditors.Thispreventstheclientsfromsubse-quentlycolludingwiththeloggertorollbackormodifytheirevents.Tothisend,weneedamechanism,suchasagossipprotocol,todistributethesignedcommitmentsfromclientstomultipleauditors.It'sunnecessaryforeveryauditortoauditeverycommitment,solongassomeauditorauditseverycommitment.(WefurtherdiscusstradeoffswithotherauditingstrategiesinSection3.1.)Inaddition,inordertodealwiththeloggerpresentingdifferentviewsofthelogtodifferentauditorsandclients,auditorsmustobtainandreconcilecommitmentsreceivedfrommultipleclientsorauditors,perhapswiththegossipprotocolmentionedabove.Alternativelytheloggermaypublishitscommitmentinapublicfashionsothatallauditorsreceivethesamecommitment[27].Allthatmattersisthatauditorshaveaccesstoadiversecollectionofcommitmentsanddemandincrementalproofstoverifythattheloggerispresentingaconsistentview.2.3Denition:tamperevidenthistoryWenowdeneatamper-evidenthistorysystemasave-tupleofalgorithms:H:ADD(X)!Cj.GivenaneventX,appendsittothehistory,returninganewcommitment.H:INCR.GEN(Ci;Cj)!P.GeneratesanincrementalproofbetweenCiandCj,whereij.H:MEMBERSHIP.GEN(i;Cj)!(P;Xi).GeneratesamembershipproofforeventifromcommitmentCj,whereij.Alsoreturnstheevent,Xi.P:INCR.VF(C0i;Cj)!f�;?g.ChecksthatPprovesthatCjxeseveryentryxedbyC0i(whereij).Outputs�ifnodivergencehasbeendetected.P:MEMBERSHIP.VF(i;Cj;X0i)!f�;?g.ChecksthatPprovesthateventX0iisthei'theventinthelogdenedbyCj(whereij).Outputs�iftrue.TherstthreealgorithmsrunontheloggerandareusedtoappendtothelogHandtogenerateproofsP.AuditorsorclientsverifytheproofswithalgorithmsfINCR.VF,MEMBERSHIP.VFg.Ideally,theproofPsenttotheau-ditorismoreconcisethanretransmittingthefullhistoryH.Onlycommitmentsneedtobesignedbythelog-ger.Proofsdonotrequiredigitalsignatures;eithertheydemonstrateconsistencyofthecommitmentsandthecon-tentsofaneventortheydon't.Withtheseveoperations,wenowdene“tamperevidence”asasystemsatisfying:HistoricalConsistencyIfwehaveavalidincrementalproofbetweentwocommitmentsCjandCk,wherejk,(P:INCR.VF(Cj;Ck)!�),andwehaveavalidmembershipproofP0fortheeventX0i,whereij,inthelogxedbyCj(i.e.,P0:MEMBERSHIP.VF(i;Cj;X0i)!�)andavalidmembershipproofforX00iinthelogxedbyCk(i.e.,P00:MEMBERSHIP.VF(i;Ck;X00i)!�),thenX0imustequalX00i.(Inotherwords,iftwocommitmentscommitconsistenthistories,thentheymustbothxthesameeventsfortheirsharedpast.)2.4OtherthreatmodelsForwardintegrityClassictamper-evidentloggingusesadifferentthreatmodel,forwardintegrity[4].Theforwardintegritythreatmodelhastwoentities:clientswhoarefullytrustedbuthavelimitedstorage,andloggerswhoareassumedtobehonestuntilsufferingaByzantinefailure.Inthisthreatmodel,theloggermustbepreventedfromundetectablytamperingwitheventsloggedpriortotheByzantinefailure,butisallowedtoundetectablytamperwitheventsloggedaftertheByzantinefailure.Althoughwefeelourthreatmodelbettercharacterizesthethreatsfacedbytamper-evidentlogging,ourhistory treeandthesemanticsfortamper-evidentloggingareapplicabletothisalternativethreatmodelwithonlyminorchanges.Underthesemanticsofforward-integrity,membershipauditingjust-addedeventsisunnecessarybecausetamper-evidenceonlyappliestoeventsoccurringbeforetheByzantinefailure.Auditingajust-addedeventisunneedediftheByzantinefailurehasn'thappenedandirrelevantafterwards.Incrementalauditingisstillnec-essary.Aclientmustincrementallyauditreceivedcom-mitmentstopreventaloggerfromtamperingwitheventsoccurringbeforeaByzantinefailurebyrollingbackthelogandcreatinganewfork.Membershipauditingisrequiredtolookupandexamineoldeventsinthelog.Itkis[31]hasasimilarthreatmodel.HisdesignexploitedthefactthatifaByzantineloggerattemptstorollbackitshistorytobeforetheByzantinefailure,thehistorymustforkintotwoparallelhistories.HeproposedaprocedurethattestedtwocommitmentstodetectdivergencewithoutonlineinteractionwiththeloggerandprovedanO(n)lowerboundonthecommitmentsize.Weachieveatighterboundbyvirtueoftheloggercooperatinginthegenerationoftheseproofs.TrustedhardwareRatherthanrelyingonauditing,analternativemodelistorelyonthelogger'shardwareitselftobetamper-resistant[58,1].Naturally,thesecurityofthesesystemsrestsonprotectingthetrustedhardwareandtheloggingsystemagainsttamperingbyanattackerwithcompletephysicalaccess.Althoughourdesigncouldcer-tainlyusetrustedhardwareasanauditor,cryptographicschemeslikeoursrestonsimplerassumptions,namelytheloggercanandmustproveitisoperatingcorrectly.3HistorytreeWenowpresentournewdatastructureforrepresentingatamper-evidenthistory.WestartwithaMerkletree[46],whichhasalonghistoryofusesforauthenticatingstaticdata.InaMerkletree,dataisstoredattheleavesandthehashattherootisatamper-evidentsummaryofthecon-tents.Merkletreessupportlogarithmicpathlengthsfromtheroottotheleaves,permittingefcientrandomaccess.AlthoughMerkletreesareawell-knowntamper-evidentdatastructureandouruseisstraightforward,thenov-eltyinourdesignisinusingaversionedcomputationofhashesovertheMerkletreetoefcientlyprovethatdiffer-entlogsnapshots,representedbyMerkletrees,withdis-tinctroothashes,makeconsistentclaimsaboutthepast.AlledhistorytreeofdepthdisabinaryMerklehashtree,storing2deventsontheleaves.Interiornodes,Ii;rareidentiedbytheirindexiandlayerr.EachleafnodeIi;0,atlayer0,storeseventXi.InteriornodeIi;rhasleftchildIi;r1andrightchildIi+2r1;r1.(Figures1through3demonstratethisnumberingscheme.)Whenatreeisnotfull,subtreescontainingnoeventsareI00;3I00;2 I00;1 X00 X01 I02;1 X02 Figure1:Aversion2historywithcommitmentC02=I00;3.I000;3I000;2 I000;1 X000 X001 I002;1 X002 X003 I004;2 I004;1 X004 X005 I006;1 X006 Figure2:Aversion6historywithcommitmentC006=I000;3:I0;3I0;2 I0;1 I2;1 X2 X3 I4;2 I4;1 I6;1 X6 Figure3:AnincrementalproofPbetweenaversion2andversion6commitment.Hashesforthecirclednodesareincludedintheproof.Otherhashescanbederivedfromtheirchildren.CirclednodesinFigures1and2mustbeshowntobeequaltothecorrespondingcirclednodeshere.representedas.ThiscanbeseenstartinginFigure1,aversion-2treehavingthreeevents.Figure2showsaversion-6tree,addingfouradditionalevents.Althoughthetreesinourgureshaveadepthof3andcanstoreupto8leaves,ourdesignclearlyextendstotreeswithgreaterdepthandmoreleaves.Eachnodeinthehistorytreeislabeledwithacrypto-graphichashwhich,likeaMerkletree,xesthecontentsofthesubtreerootedatthatnode.Foraleafnode,thelabelisthehashoftheevent;foraninteriornode,thelabelisthehashoftheconcatenationofthelabelsofitschildren.Aninterestingpropertyofthehistorytreeistheabilitytoefcientlyreconstructoldversionsorviewsofthetree.ConsiderthehistorytreegiveninFigure2.TheloggercouldreconstructC002analogoustotheversion-2treeinFigure1bypretendingthatnodesI004;2andX003wereandthenrecomputingthehashesfortheinteriornodesandtheroot.IfthereconstructedC002matchedapreviouslyadvertisedcommitmentC02,thenbothtreesmusthavethesamecontentsandcommitthesameevents. b X0 X1 X2 X3 b X4 X5 b X6 Figure4:GraphicalnotationforahistorytreeanalogoustotheproofinFigure3.Soliddiscsrepresenthashesincludedintheproof.Othernodesarenotincluded.Dotsandopencirclesrepresentvaluesthatcanberecomputedfromthevaluesbelowthem;dotsmaychangeasneweventsareaddedwhileopencir-cleswillnot.Greycirclenodesareunnecessaryfortheproof.ThisformstheintuitionofhowtheloggergeneratesanincrementalproofPbetweentwocommitments,C02andC006.Initially,theauditoronlypossessescommitmentsC02andC006;itdoesnotknowtheunderlyingMerkletreesthatthesecommitmentsx.Theloggermustshowthatbothhistoriescommitthesameevents,i.e.,X000=X00;X001=X01,andX002=X02.Todothis,theloggersendsaprunedtreePtotheauditor,showninFigure3.ThisprunedtreeincludesjustenoughofthefullhistorytreetocomputethecommitmentsC2andC6.Unnecessarysubtreesareelidedoutandreplacedwithstubs.Eventscanbeeitherincludedinthetreeorreplacedbyastubcontainingtheirhash.Becauseanincrementalproofinvolvesthreehistorytrees,thetreescommittedbyC02andC006withunknowncontentsandtheprunedtreeP,wedistinguishthembyusingadifferentnumberofprimes(0).FromP,showninFigure3,wereconstructthecorre-spondingrootcommitmentforaversion-6tree,C6.Were-computethehashesofinteriornodesbasedonthehashesoftheirchildrenuntilwecomputethehashfornodeI0;3,whichwillbethecommitmentC6.IfC006=C6thenthecor-respondingnodes,circledinFigures2and3,intheprunedtreePandtheimplicittreecommittedbyC006mustmatch.Similarly,fromP,showninFigure3,wecanrecon-structtheversion-2commitmentC2bypretendingthatthenodesX3andI4;2areand,asbefore,recomputingthehashesforinteriornodesuptotheroot.IfC02=C2,thenthecorrespondingnodes,circledinFigures1and3,intheprunedtreePandtheimplicittreecommittedbyC02mustmatch,orI00;1=I0;1andX02=X2.IftheeventscommittedbyC02andC006arethesameastheeventscommittedbyP,thentheymustbeequal;wecanthenconcludethatthetreecommittedbyC006isconsistentwiththetreecommittedbyC02.BythiswemeanthatthehistorytreescommittedbyC02andC006bothcommitthesameevents,orX000=X00,X001=X01,andX002=X02,eventhoughtheeventsX000=X00,X001=X01,X004,andX005areunknowntotheauditor.3.1Isitsafetoskipnodesduringanaudit?IntheprunedtreeinFigure3,weomittheeventsxedbyI0;1,yetwestillpreservethesemanticsofatamper-evidentlog.Eventhoughtheseearliereventsmaynotbesenttotheauditor,theyarestillxedbytheunchangedhashesabovetheminthetree.Anyattemptedtamperingwillbediscoveredinfutureincrementalormembershipauditsoftheskippedevents.Withthehistorytree,auditorsonlyreceivetheportionsofthehistorytheyneedtoaudittheeventstheyhavechosentoaudit.Skippingeventsmakesitpossibletoconductavarietyofselectiveauditsandoffersmoreexibilityindesigningauditingpolicies.Existingtamper-evidentlogdesignsbasedonaclassichash-chainhavetheformCi=H(Ci1kXi),C1=anddonotpermiteventstobeskipped.Withahashchain,anincrementalormembershipproofbetweentwocom-mitmentsorbetweenaneventandacommitmentmustincludeeveryintermediateeventinthelog.Inaddition,becauseintermediateeventscannotbeskipped,eachaudi-tor,orclientactingasanauditor,musteventuallyreceiveeveryeventinthelog.Hashchainingschemes,assuch,areonlyfeasiblewithloweventvolumesorinsituationswhereeveryauditorisalreadyreceivingeveryevent.Whenmembershipproofsareusedtoinvestigateoldevents,theabilitytoskipnodescanleadtodramaticreductionsinproofsize.Forexample,inourprototypedescribedinSection5,inalogof80millionevents,ourhistorytreecanreturnacompleteproofforanyrandomlychoseneventin3100bytes.Inahashchain,whereintermediateeventscannotbeskipped,anaverageof40millionhasheswouldbesent.AuditingstrategiesInmanysettings,itispossiblethatnoteveryauditorwillbeinterestedineveryloggedevent.Clientsmaynotbeinterestedinauditingeventsinsertedorcommitmentsreceivedbyotherclients.Onecouldeasilyimaginescenarioswhereasingleloggerissharedacrossmanyorganizations,eachonlyincentivizedtoauditthein-tegrityofitsowndata.Theseorganizationscouldruntheirownauditors,focusingtheirattentiononcommitmentsfromtheirownclients,andonlyoccasionallyexchangingcommitmentswithotherorganizationstoensurenofork-inghasoccurred.Onecanalsoimaginescenarioswhereindependentaccountingrmsoperateauditingsystemsthatrunagainsttheircorporatecustomers'logservers.Thelogremainstamper-evidentifclientsgossiptheirreceivedcommitmentsfromtheloggertoatleastonehon-estauditorwhousesitwhendemandinganincrementalproof.Bynotrequiringthateverycommitmentbeauditedbyeveryauditor,thetotalauditingoverheadacrossallauditorscanbeproportionaltothetotalnumberofeventsinthelog—farcheaperthanthenumberofeventstimesthenumberofauditorsaswemightotherwiserequire. Avi;0=nH(0kXi)ifvi(1)Avi;r=(H(1kAvi;r1k)ifvi+2r1H(1kAvi;r1kAvi+2r1;r1)ifvi+2r1(2)Cn=An0;d(3)Avi;rFHi;rwhenevervi+2r1(4)Figure5:Recurrenceforcomputinghashes.Skippingnodesoffersothertime-securitytradeoffs.Auditorsmayconductauditsprobabilistically,selectingonlyasubsetofincomingcommitmentsforauditing.Ifaloggerweretoregularlytamperwiththelog,itsoddsofremainingundetectedwouldbecomevanishinglysmall.3.2ConstructionofthehistorytreeNowthatwehaveanexampleofhowtouseatree-basedhistory,wewillformallydeneitsconstructionandsemantics.Aversion-nhistorytreestoresn+1events,X0:::Xn.Hashesarecomputedoverthehistorytreeinamannerthatpermitsthereconstructionofthehashesofinteriornodesofolderversionsorviews.WedenotethehashonnodeIi;rbyAvi;rwhichisparametrizedbythenode'sindex,layerandviewbeingcomputed.Aversion-vviewonaversion-nhistorytreereconstructsthehashesoninteriornodesforaversion-vhistorytreethatonlyincludedeventsX0:::Xv.Whenv=n,thereconstructedrootcommitmentisCn.ThehashesarecomputedwiththerecurrencedenedinFigure5.Ahistorytreecansupportarbitrarysizelogsbyincreasingthedepthwhenthetreells(i.e.,n=2d1)anddeningd=dlog2(n+1)e.Thenewroot,onelevelup,iscreatedwiththeoldtreeasitsleftchildandanemptyrightchildwhereneweventscanbeadded.Forsimplicityinourillustrationsandproofs,weassumeatreewithxeddepthd.Onceagivensubtreeinthehistorytreeiscompleteandhasnomoreslotstoaddevents,thehashfortherootnodeofthatsubtreeisfrozenandwillnotchangeasfutureeventsareaddedtothelog.Theloggercachesthesefrozenhashes(i.e.,thehashesoffrozennodes)intoFHi;rtoavoidtheneedtorecomputethem.Byexploitingthefrozenhashcache,theloggercanrecomputeAvi;rforanynodewithatmostO(d)operations.Inaversion-ntree,nodeIi;risfrozenwhenni+2r1.Wheninsertinganeweventintothelog,O(1)expectedcaseandO(d)worsecasenodeswillbecomefrozen.(InFigure1,nodeI00;1isfrozen.IfeventX3isadded,nodesI02;1andI00;2willbecomefrozen.)Nowthatwehavedenedthehistorytree,wewilldescribetheincrementalproofsgeneratedbythelogger.Figure4abstractlyillustratesaprunedtreeequivalentto X0 X1 X2 X3 b X4 X5 b X6 Figure6:Aproofskeletonforaversion-6historytree.theproofgiveninFigure3,representinganincrementalprooffromC2toC6.Dotsrepresentunfrozennodeswhosehashesarecomputedfromtheirchildren.Opencirclesrepresentfrozennodeswhicharenotincludedintheproofbecausetheirhashescanberecomputedfromtheirchildren.Soliddiscsrepresentfrozennodeswhoseinclusionisnecessarybybeingleavesorstubs.Grayedoutnodesrepresentelidedsubtreesthatarenotincludedintheprunedtree.Fromthisprunedtreeandequations(1)-(4)(showninFigure5)wecancomputeC6=A60;3andacommitmentfromanearlierversion-2view,A20;3.Thisprunedtreeisincrementallybuiltfromaproofskeleton,seeninFigure6—theminimumprunedtreeofaversion-6treeconsistingonlyoffrozennodes.Theproofskeletonforaversion-ntreeconsistsoffrozenhashesfortheleftsiblingsforthepathfromXntotheroot.Fromtheincludedhashesandusingequations(1)-(4),thisproofskeletonsufcestocomputeC6=A60;3.FromFigure6theloggerincrementallybuildsFigure4bysplittingfrozeninteriornodes.Anodeissplitbyincludingitschildren'shashesintheprunedtreeinsteadofitself.Byrecursivelysplittingnodesonthepathtoaleaf,theloggercanincludethatleafintheprunedtree.Inthisexample,wesplitnodesI0;2andI2;1.ForeachcommitmentCithatistobereconstructableinanincrementalprooftheprunedtreePmustincludeapathtotheeventXi.ThesamealgorithmisusedtogeneratethemembershipproofforaneventXi.Giventheseconstraints,wecannowdenethevehistoryoperationsintermsoftheequationsinFigure5.H:ADD(X)!Cn.Eventisassignedthenextfreeslot,n.Cniscomputedbyequations(1)-(4).H:INCR.GEN(Ci;Cj)!P.TheprunedtreePisaversion-jproofskeletonincludingapathtoXi.H:MEMBERSHIP.GEN(i;Cj)!(P;Xi).TheprunedtreePisaversion-jproofskeletonincludingapathtoXi.P:INCR.VF(C00i;C0j)!f�;?g.FromPapplyequations(1)-(4)tocomputeAi0;dandAj0;d.ThiscanonlybedoneifPincludesapathtotheleafXi.Return�ifC00i=Ai0;dandC0j=Aj0;d. P:MEMBERSHIP.VF(i;C0j;X0i)!f�;?g.FromPapplyequations(1)-(4)tocomputeAj0;d.AlsoextractXifromtheprunedtreeP,whichcanonlybedoneifPincludesapathtoeventXi.Return�ifC0j=Aj0;dandXi=X0i.Althoughincrementalandmembershipproofshavedif-ferentsemantics,theybothfollowanidenticaltreestruc-tureandcanbebuiltandauditedbyacommonimplemen-tation.Inaddition,asingleprunedtreePcanembedpathstoseveralleavestosatisfymultipleauditingrequests.Whatisthesizeofaprunedtreeusedasaproof?Theprunedtreenecessaryforsatisfyingaself-containedin-crementalproofbetweenCiandCjoramembershipproofforiinCjrequiresthattheprunedtreeincludeapathtonodesXiandXj.Thisresultingprunedtreecontainsatmost2dfrozennodes,logarithmicinthesizeofthelog.Inarealimplementation,thelogmayhavemovedontoalaterversion,k.IftheauditorrequestedanincrementalproofbetweenCiandCj,theloggerwouldreturnthelatestcommitmentCk,andaprunedtreeofatmost3dnodes,basedaroundaversion-ktreeincludingpathstoXiandXj.Moretypically,weexpectauditorswillrequestanincrementalproofbetweenacommitmentCiandthelatestcommitment.TheloggercanreplywiththelatestcommitmentCkandprunedtreeofatmost2dnodesthatincludedapathtoXi.ThefrozenhashcacheInourdescriptionofthehistorytree,wedescribedthefullrepresentationwhenwestatedthattheloggerstoresfrozenhashesforallfrozeninteriornodesinthehistorytree.Thiscacheisredundantwheneveranode'shashcanberecomputedfromitschildren.Weexpectthatloggerimplementations,whichbuildprunedtreesforauditsandqueries,willmaintainandusethecachetoimproveefciency.Whengeneratingmembershipproofs,incrementalproofs,andquerylookupresults,thereisnoneedfortheresultingprunedtreetoincluderedundanthashesoninteriornodeswhentheycanberecomputedfromtheirchildren.Weassumethatprunedtreesusedasproofswillusethisminimumrepresentation,containingfrozenhashesonlyforstubs,toreducecommunicationcosts.Canoverheadsbereducedbyexploitingredundancybetweenproofs?Ifanauditorisinregularcommu-nicationwiththelogger,demandingincrementalproofsbetweenthepreviouslyseencommitmentandthelatestcommitment,thereisredundancybetweentheprunedsubtreesonsuccessivequeries.IfanauditorpreviouslyrequestedanincrementalproofbetweenCiandCjandlaterrequestsanincrementalproofPbetweenCjandCn,thetwoproofswillsharehashesonthepathtoleafXj.Theloggermaysendapartialproofthatomitsthesecommonhashes,andonlycontainstheexpectedO(log2(nj))frozenhashesthatarenotsharedbetweenthepathstoXjandXn.ThisdevolvestoO(1)ifaproofisrequestedaftereveryinsertion.Theauditorneedonlycachedfrozenhashestomakethiswork.Treehistorytime-stampingserviceOurhistorytreecanbeadaptedtoimplementaround-basedtime-stampingservice.Aftereveryround,theloggerpublishesthelastcommitmentinpublicmediumsuchasanews-paper.LetCibethecommitmentfromthepriorroundandCkbethecommitmentoftheroundaclientrequeststhatitsdocumentXjbetimestamped.AclientcanrequestaprunedtreeincludingapathtoleavesXi;Xj;Xk.TheprunedtreecanbeveriedagainstthepublishedcommitmentstoprovethatXjwassubmittedintheroundanditsorderwithinthatround,withoutthecooperationofthelogger.Ifaseparatehistorytreeisbuiltforeachround,ourhis-torytreeisequivalenttothethreadedauthenticationtreeproposedbyBuldasetal.[10]fortime-stampingsystems.3.3StoringthelogonsecondarystorageOurhistorytreeoffersacuriousproperty:itcanbeeasilymappedontowrite-onceappend-onlystorage.Oncenodesbecomefrozen,theybecomeimmutable,andarethussafetooutput.Thisorderingispredetermined,startingwith(X0),(X1;I0;1),(X2),(X3;I2;1;I0;2),(X4):::.ParenthesesdenotethenodeswrittenbyeachADDtrans-action.Ifnodeswithineachgrouparefurtherorderedbytheirlayerinthetree,thisorderissimplyapost-ordertraversalofthebinarytree.Datawritteninthislinearfashionwillminimizediskseekoverhead,improvingthedisk'swriteperformance.Giventhislayout,andassumingalleventsarethesamesizeondisk,convertingfroman(index;layer)tothebyteindexusedtostorethatnodetakesO(logn)arithmeticoperations,permittingefcientdirectaccess.Inordertohandlevariable-lengthevents,eventdatacanbestoredinaseparatewrite-onceappend-onlyvaluestore,whiletheleavesofthehistorytreecontainoffsetsintothevaluestorewheretheeventcontentsmaybefound.Decouplingthehistorytreefromthevaluestorealsoallowsmanychoicesforhoweventsarestored,suchasdatabases,compressedles,orstandardatformats.3.4ComparingtoothersystemsInthissection,weevaluatethetimeandspacetradeoffsbetweenourhistorytreeandearlierhashchainandskipliststructures.Inallthreedesigns,membershipproofshavethesamestructureandsizeasincrementalproofs,andproofsaregeneratedintimeproportionaltotheirsize.ManiatisandBaker[43]presentatamper-evidentlogusingadeterministicvariantofaskiplist[53].Theskiplisthistoryislikeahash-chainincorporatingextraskiplinksthathopovermanynodes,allowingforlogarithmiclookups. HashchainSkiplistHistorytree ADDTimeO(1)O(1)O(log2n)INCR.GENproofsizetoCkO(nk)O(n)O(log2n)MEMBERSHIP.GENproofsizeforXkO(nk)O(n)O(log2n) Cachesize-O(log2n)O(log2n)INCR.GENpartialproofsize-O(nj)O(log2(nj))MEMBERSHIP.GENpartialproofsize-O(log2(ni))O(log2(ni))Table1:Wecharacterizethetimetoaddaneventtothelogandthesizeoffullandpartialproofsgeneratedintermsofn,thenumberofeventsinthelog.Forpartialproofsaudits,jdenotesthenumberofeventsinthelogatthetimeofthelastauditandidenotestheindexoftheeventbeingmembership-audited.InTable1wecomparethethreedesigns.AllthreedesignshaveO(1)storagepereventandO(1)com-mitmentsize.Forskiplisthistoriesandtreehistories,whichsupportpartialproofs(describedinSection3.2),wepresentthecachesizeandtheexpectedproofsizesintermsofthenumberofeventsinthelog,n,andtheindex,j,ofthepriorcontactwiththeloggerortheindexioftheeventbeinglookedup.Ourtree-basedhistorystrictlydominatesbothhashchainsandskiplistsinproofgenerationtimeandproofsizes,particularlywhenindividualclientsandauditorsonlyauditasubsetofthecommitmentsorwhenpartialproofsareused.CanonicalrepresentationAhashchainhistoryandourhistorytreehaveacanonicalrepresentationofboththehistoryandofproofswithinthehistory.Inparticular,fromagivencommitmentCn,thereexistsoneuniquepathtoeacheventXi.Whentherearemultiplepathsauditingismorecomplexbecausethealternativepathsmustbecheckedforconsistencywithoneanother,bothwithinasinglehistory,andbetweenthestreamofhistoriesCi;Ci+1;:::committedbythelogger.Extrapathsmayimprovetheefciencyoflookinguppastevents,suchasinaskiplist,oroffermorefunctionality[17],butcannotbetrustedbyauditorsandmustbechecked.ManiatisandBaker[43]claimtosupportlogarithmic-sizedproofs,howevertheysufferfromthismulti-pathproblem.Toverifyinternalconsistency,anauditorwithnopriorcontactwiththeloggermustreceiveeveryeventinthelogineveryincrementalormembershipproof.EfciencyimprovesforauditorsinregularcontactwiththeloggerthatusepartialproofsandcacheO(log2n)statebetweenincrementalaudits.Ifanauditorhaspreviouslyveriedthelogger'sinternalconsistencyuptoCj,theauditorwillbeabletoverifythelogger'sinternalconsis-tencyuptoafuturecommitmentCnwiththereceiptofeventsXj+1:::XnOnceanauditorknowsthattheskiplistisinternallyconsistentthelinksthatallowforlogarithmiclookupscanbetrustedandsubsequentmembershipproofsonoldeventswillruninO(log2n)time.Skiplisthistoriesweredesignedtofunctioninthismode,witheachauditoreventuallyreceivingeveryeventinthelog.AuditingisrequiredHashchainsandskiplistsonlyofferacomplexityadvantageoverthehistorytreewhenaddingnewevents,butthisadvantageiseeting.Iftheloggerknowsthatagivencommitmentwillneverbeaudited,itisfreetotamperwiththeeventsxedbythatcommitment,andthelogisnolongerprovablytamperevident.Everycommitmentreturnedbytheloggermusthaveanon-zerochanceofbeingauditedandanyevaluationoftamper-evidentloggingmustincludethecostsofthisunavoidableauditing.Withmultipleauditors,auditingoverheadisfurthermultiplied.Afterinsertinganevent,hashchainsandskiplistssufferanO(nj)disadvantagethemomenttheydoincrementalauditsbetweenthereturnedcommitmentandpriorcommitments.Theycannotreducethisoverheadby,forexample,onlyauditingarandomsubsetofcommitments.Evenifthethreatmodelisweakenedfromouralways-untrustedloggertotheforward-integritythreatmodel(SeeSection2.4),hashchainsandskiplistsarelessefcientthanthehistorytree.Clientscanforgoauditingjust-addedevents,butarestillrequiredtodoincrementalauditstopriorcommitments,whichareexpensivewithhashchainsorskiplists.4MerkleaggregationOurhistorytreepermitsO(log2n)accesstoarbitraryevents,giventheirindex.Inthissection,weextendourhistorytreetosupportefcient,tamper-evidentcontentsearchesthroughafeaturewecallMerkleaggregation,whichencodesauxiliaryinformationintothehistorytree.Merkleaggregationpermitstheloggertoperformauthorizedpurgesofthelogwhiledetectingunauthorizeddeletions,afeaturewecallsafedeletion.Asanexample,imaginethataclientagscertaineventsinthelogas“important”whenitstoresthem.Inthehistorytree,theloggerpropagatestheseagstointeriornodes,settingtheagwhenevereitherchildisagged.Toensurethatthetaggedhistoryistamper-evident,thisagcanbeincorporatedintothehashlabelofanodeandcheckedduringauditing.Asclientsareassumedtobetrustedwheninsertingintothelog,weassumeclientswillproperlyannotatetheirevents.Membershipauditingwilldetectiftheloggerincorrectlystoredaleafwiththewrongagorimproperlypropagatedtheag.Incrementalauditswoulddetecttamperingifanyfrozen nodehaditsagaltered.Now,whenanauditorrequestsalistofonlyaggedevents,theloggercangeneratethatlistalongwithaproofthatthelistiscomplete.Iftherearerelativelyfew“important”events,thequeryresultscanskipoverlargechunksofthehistory.Togenerateaproofthatthelistofaggedeventsiscomplete,theloggertraversesthefullhistorytreeH,pruninganysubtreeswithouttheagset,andreturnsaprunedtreePcontainingonlythevisitednodes.TheauditorcanensurethatnoaggednodeswereomittedinPbyperformingitsownrecursivetraversalonPandverifyingthateverystubisunagged.Figure7showstheprunedtreeforaqueryagainstaversion-5historywitheventsX2andX5agged.InteriornodesinthepathfromX2andX5totherootwillalsobeagged.Forsubtreescontainingnomatchingevents,suchastheparentofX0andX1,weonlyneedtoretaintherootofthesubtreetovouchthatitschildrenareunagged.4.1GeneralattributesBooleanagsareonlyonewaywemayaglogeventsforlaterqueries.Ratherthanenumerateeverypossiblevariation,weabstractanaggregationstrategyoverattributesintoa3-tuple,(t;;G).trepresentsthetypeofattributeorattributesthataneventhas.isadeterministicfunctionusedtocomputetheattributesonaninteriornodeinthehistorytreebyaggregatingtheattributesofthenode'schildren.Gisadeterministicfunctionthatmapsaneventtoitsattributes.Inourexampleofclient-aggedevents,theaggregationstrategyis(t:=BOOL;:=_;G(x):=x:isFlagged).Forexample,inabankingapplication,anattributecouldbethedollarvalueofatransaction,aggregatedwiththeMAXfunction,permittingqueriestondalltransactionsoveraparticulardollarvalueanddetectiftheloggertamperswiththeresults.Thiscorrespondsto(t:=INT;:=MAX;G(x):=x:value).Or,considereventshav-inginternaltimestamps,generatedbytheclient,arrivingattheloggeroutoforder.Ifweattributeeachnodeinthetreewiththeearliestandlatesttimestampfoundamongitschildren,wecannowquerytheloggerforallnodeswithinagiventimerange,regardlessoftheorderofeventarrival.ThereareatleastthreedifferentwaystoimplementkeywordsearchingacrosslogsusingMerkleaggregation.Ifthenumberofkeywordsisxedinadvance,thentheattributetforeventscanbeabit-vectororsparsebit-vectorcombinedwith:=_.Ifthenumberofkeywordsisunknown,butlikelytobesmall,tcanbeasortedlistofkeywords,with:=[(setunion).Ifthenumberofkeywordsisunknownandpotentiallyunbounded,thenaBloomlter[8]maybeusedtorepresentthem,withtbeingabit-vectorand:=_.Ofcourse,theBloomlterwouldthenhavethepotentialofreturningfalsepositivestoaquery,buttherewouldbenofalsenegatives. b X0 X1 X2 X3 b X4 X5 Figure7:DemonstrationofMerkleaggregationwithsomeeventsaggedasimportant(highlighted).Frozennodesthatwouldbeincludedinaqueryarerepresentedassoliddiscs.MerkleaggregationisextremelyexiblebecauseGcanbeanydeterministiccomputablefunction.However,oncealoghasbeencreated,(t;;G)arexedforthatlog,andthesetofqueriesthatcanbemadeisrestrictedbasedontheaggregationstrategychosen.InSection5wedescribehowwewereabletoapplytheseconceptstothemetadatausedinSysloglogs.4.2FormaldescriptionTomakeattributestamper-evidentinhistorytrees,wemodifythecomputationofhashesoverthetreetoincludethem.EachnodenowhasahashlabeldenotedbyAvi;r:HandanannotationdenotedbyAvi;r:Aforstoringattributes.Togethertheseformthenodedatathatisattachedtoeachnodeinthehistorytree.Notethatthehashlabelofnode,Avi;r:H,doesnotxitsownattributes,Avi;r:A.Instead,wedeneasubtreeauthenticatorAvi;r:=H(Avi;r:HkAvi;r:A)thatxestheattributesandhashofanode,andrecursivelyxeseveryhashandattributeinitssubtree.FrozenhashesFHi;r:AandFHi;r:HandFHi;r:aredenedanalogouslytothenon-Merkle-aggregationcase.Wecouldhavedenedthisrecursioninseveraldiffer-entways.Thisrepresentationallowsustoelideunwantedsubtreeswithasmallstub,containingonehashandonesetofattributes,whileexposingtheattributesinawaythatmakesitpossibletolocallydetectiftheattributeswereimproperlyaggregated.Ournewmechanismforcomputinghashandaggre-gatesforanodeisgiveninequations(5)-(10)inFigure8.ThereisastrongcorrespondencebetweenthisrecurrenceandthepreviousoneinFigure5.Equations(6)and(7)extractthehashandattributesofanevent,analogoustoequation(1).Equation(9)handlesaggregationofattributesbetweenanodeanditschildren.Equation(8)computesthehashofanodeintermsofthesubtreeauthenticatorsofitschildren.INCR.GENandMEMBERSHIP.GENoperatethesameaswithanordinaryhistorytree,exceptthatwhereverafrozenhashwasincludedintheproof(FHi;r),wenowincludeboththehashofthenode,FHi;r:H,anditsattributesFHi;r:A.BotharerequiredforrecomputingAvi;r:AandAvi;r:Hfortheparentnode.ADD;INCR.VF, Avi;r:=H(Avi;r:HkAvi;r:A)(5)Avi;0:H=nH(0kXi)ifvi(6)Avi;0:A=nG(Xi)ifvi(7)Avi;r:H=(H(1kAvi;r1:k)ifvi+2r1H(1kAvi;r1:kAvi+2r1;r1:)ifvi+2r1(8)Avi;r:A=(Avi;r1:Aifvi+2r1Avi;r1:AAvi+2r1;r1:Aifvi+2r1(9)Cn=An0;d:(10)Figure8:HashcomputationsforMerkleaggregationandMEMBERSHIP.VFarethesameasbeforeexceptforusingtheequations(5)-(10)forcomputinghashesandpropagatingattributes.Merkleaggregationinatesthestorageandproofsizesbyafactorof(A+B)=AwhereAisthesizeofahashandBisthesizeoftheattributes.4.2.1QueriesoverattributesInMerkleaggregationqueries,wepermitqueryresultstocontainfalsepositives,i.e.,eventsthatdonotmatchthequeryQ.Extrafalsepositiveeventsintheresultonlyimpactperformance,notcorrectness,astheymaybelteredbytheauditor.Weforbidfalsenegatives;everyeventmatchingQwillbeincludedintheresult.Unfortunately,Merkleaggregationqueriescanonlymatchattributes,notevents.Consequently,wemustconservativelytransformaqueryQovereventsintoapredicateQGoverattributesandrequirethatitbestable,withthefollowingproperties:IfQmatchesaneventthenQGmatchestheattributesofthatevent(i.e.,8xQ(x))QG(G(x))).Furthermore,ifQGistrueforeitherchildofanode,itmustbetrueforthenodeitself(i.e.,8x;yQG(x)_QG(y))QG(xy)and8xQG(x)_QG())QG(x)).Stablepredicatescanfalselymatchnodesoreventsfortworeasons:events'attributesmaymatchQGwithouttheeventsmatchingQ,ornodesmayoccurwhere(QG(x)_QG(y))isfalse,butQG(xy)istrue.WecallapredicateQexactiftherecanbenofalsematches.ThisoccurswhenQ(x),QG(G(x))andQG(x)_QG(y),QG(xy).Exactqueriesaremoreefcientbecauseaqueryresultdoesnotincludefalselymatchingeventsandthecorrespondingprunedtreeprovingthecorrectnessofthequeryresultdoesnotrequireextranodes.Giventheseproperties,wecannowdenetheaddi-tionaloperationsforperformingauthenticatedqueriesonthelogforeventsmatchingapredicateQG.H:QUERY(Cj;QG)!PGivenapredicateQGoverattributest,returnsaprunedtreewhereeveryelidedsubtreesdoesnotmatchQG.P:QUERY.VF(C0j;QG)!f&#x-5.1;ä¡£;?gCheckstheprunedtreePandreturns&#x-5.1;ä¡£ifeverystubinPdoesnotmatchQGandthereconstructedcommitmentCjisthesameasC0j.BuildingaprunedtreecontainingalleventsmatchingapredicateQGissimilartobuildingtheprunedtreesformembershiporincrementalauditing.Theloggerstartswithaproofskeletonthenrecursivelytraversesit,splittinginteriornodeswhenQG(FHi;r:A)istrue.BecausethepredicateQGisstable,noeventinanyelidedsubtreecanmatchthepredicate.IfthereareteventsmatchingthepredicateQG,theprunedtreeisofsizeatmostO((1+t)log2n)(i.e.,tleaveswithlog2ninteriortreenodesonthepathstotheroot).ToverifythatPincludesalleventsmatchingQG,theauditordoesarecursivetraversaloverP.IftheauditorndsaninteriorstubwhereQG(FHi;r:A)istrue,thever-icationfailsbecausetheauditorfoundanodethatwassupposedtohavebeensplit.(UnfrozennodeswillalwaysbesplitastheycomposetheproofskeletonandonlyoccuronthepathfromXjtotheroot.)TheauditormustalsoverifythatprunedtreePcommitsthesameeventsasthecommitmentC0jbyreconstructingtherootcommitmentCjusingtheequations(5)-(10)andcheckingthatCj=C0j.Aswithanordinaryhistorytree,aMerkleaggregatingtreerequiresauditingfortamper-detection.Ifaneventisneveraudited,thenthereisnoguaranteethatitsattributeshavebeenproperlyincluded.Also,adishonestloggerorclientcoulddeliberatelyinsertfalselogentrieswhoseattributesareaggregatedupthetreetotheroot,causinggarbageresultstobeincludedinqueries.Evenso,ifQisstable,amaliciousloggercannothidematchingeventsfromqueryresultswithoutdetection.4.3ApplicationsSafedeletionMerkleaggregationcanbeusedforexpiringoldandobsoleteeventsthatdonotsatisfysomepredicateandprovethatnoothereventsweredeletedinappropriately.WhileMerkleaggregationqueriesprovethatnomatchingeventisexcludedfromaqueryresult,safedeletionrequiresthecontrapositive:provingtoanauditorthateachpurgedeventwaslegitimatelypurgedbecauseitdidnotmatchthepredicate.LetQ(x)beastablequerythatistrueforalleventsthattheloggermustkeep.LetQG(x)bethecorrespondingpredicateoverattributes.TheloggerstoresaprunedtreethatincludesallnodesandleafeventswhereQG(x)istrue.Theremainingnodesmaybeelidedandreplacedwithstubs.WhenaloggercannotgenerateapathtoapreviouslydeletedeventXi,itinsteadsuppliesaprunedtreethatincludesapathtoanancestornodeAofXiwhereQG(A)isfalse.BecauseQisstable,ifQG(A)isfalse,thenQG(G(Xi))andQ(Xi)mustalsobefalse. SafedeletionandauditingpoliciesmusttakeintoaccountthatifasubtreecontainingeventsXi:::Xjispurged,theloggerisunabletogenerateincrementalormembershipproofsinvolvingcommitmentsCi:::Cj.Theauditingpolicymustrequirethatanyauditsusingthosecommitmentsbeperformedbeforethecorrespondingeventsaredeleted,whichmaybeassimpleasrequiringthatclientsperiodicallyrequestanincrementalprooftoalaterorlong-livedcommitment.Safedeletionwillnotsavespacewhenusingtheappend-onlystoragedescribedinSection3.3.However,ifdata-destructionpoliciesrequiredestroyingasubsetofeventsinthelog,safedeletionmaybeusedtoprovethatnounauthorizedlogeventsweredestroyed.“Private”searchMerkleaggregationenablesaweakvariantofprivateinformationretrieval[14],permittingclientstohaveprivacyforthespeciccontentsoftheirevents.Toaggregatetheattributesofanevent,theloggeronlyneedstheattributesofanevent,G(Xi),nottheeventitself.Toverifythataggregationisdonecorrectlyalsoonlyrequirestheattributesofanevent.Ifclientsencrypttheireventsanddigitallysigntheirpublicattributes,auditorsmayverifythataggregationisdonecorrectlywhileclientspreservetheireventprivacyfromtheloggerandotherclientsandauditors.Bloomlters,inadditiontoprovidingacompactandapproximatewaytorepresentthepresenceorabsenceofalargenumberofkeywords,canalsoenableprivateindexing(see,e.g.,Goh[23]).TheloggerhasnoideawhattheindividualkeywordsarewithintheBloomlter;manykeywordscouldmaptothesamebit.Thisallowsforprivatekeywordsthatarestillprotectedbytheintegritymechanismsofthetree.5SyslogprototypeimplementationSyslogisthestandardUnix-basedloggingsystem[38],storingeventswithmanyattributes.Todemonstratetheeffectivenessofourhistorytree,webuiltanimplementa-tioncapableofstoringandsearchingsyslogevents.Usingeventsfromsyslogtraces,capturedfromourdepartmentalservers,weevaluatedthestorageandperformancecostsoftamper-evidentloggingandsecuredeletion.Eachsyslogeventincludesatimestamp,thehostgener-atingtheevent,oneof24facilitiesorsubsystemthatgen-eratedtheevent,oneof8logginglevels,andthemessage.Mosteventsalsoincludeatagindicatingtheprogramgeneratingtheevent.Solutionsforauthentication,man-agement,andreliabledeliveryofsyslogeventsoverthenetworkhavealreadybeenproposed[48]andareintheprocessofbeingstandardized[32],butnoneofthisworkaddressestheloggingsemanticsthatwewishtoprovide.OurprototypeimplementationwaswritteninahybridofPython2.5.2andC++andwasbenchmarkedonanIntelCore2Duo2.4GHzCPUwith4GBofRAMin64-bitmodeunderLinux.Ourpresentimplementationissingle-threaded,sothesecondCPUcoreisunderutilized.OurimplementationusesSHA-1hashesand1024-bitDSAsignatures,borrowedfromtheOpenSSLlibrary.Inourimplementation,weusethearray-basedpost-ordertraversalrepresentationdiscussedinSection3.3.Thevaluestoreandhistorytreearestoredinseparatewrite-onceappend-onlylesandmappedintomemory.Nodesinthehistorytreeuseaxednumberofbytes,permittingdirectaccess.GeneratingmembershipandincrementalproofsrequiresRAMproportionaltothesizeoftheproof,whichislogarithmicinthenumberofeventsinthelog.MerkleaggregationqueryresultsizesarepresentlylimitedtothosewhichcantinRAM,approximately4millionevents.Thestorageoverheadsofourtamper-evidenthistorytreearemodest.Ourprototypestoresveattributesforeachevent.Tagsandhostnamesareencodedas2-of-32bitBloomlters.Facilitiesandhostsareencodedasbit-vectors.Topermitrangequeriestondeveryeventinaparticularrangeoftime,anintervalisusedtoencodethemessagetimestamp.Alltogether,therearetwentybytesofattributesandtwentybytesforaSHA-1hashforeachnodeinthehistorytree.Leaveshaveanadditionaltwelvebytestostoretheoffsetandlengthoftheeventcontentsinthevaluestore.Werananumberofsimulationsofourprototypetodeterminetheprocessingtimeandspaceoverheadsofthehistorytree.Tothisend,wecollectedatraceoffourmillioneventsfromthirteenofourdepartmentalserverhostsover106hours.Weobserved9facilities,6levels,and52distincttags.88.1%oftheeventsarefromthemailserverand11.5%arefrom98,743failedsshconnectionattempts.Only.393%oftheloglinesarefromothersources.Intestingourhistorytree,wereplaythistrace20timestoinsert80millionevents.Oursyslogtrace,afterthereplay,occupies14.0GB,whilethehistorytreeaddsanadditional13.6GB.5.1PerformanceoftheloggerTheloggeristheonlycentralizedhostinourdesignandmaybeabottleneck.Theperformanceofarealworldloggerwilldependontheauditingpolicyandrelativefrequencybetweeninsertingeventsandrequestingaudits.Ratherthansummarizetheperformanceoftheloggerforoneparticularauditingpolicy,webenchmarkthecostsofthevarioustasksperformedbythelogger.Ourcapturedsyslogtracesaveragedonlyteneventspersecond.Ourprototypecaninserteventsatarateof1,750eventspersecond,includingDSAsignaturegeneration.Insertinganeventrequiresfoursteps,showninTable2,withthenalstep,signingtheresultingcommitment,responsibleformostoftheprocessingtime.Throughput Step Task %ofCPURate (events/sec) A Parsesyslogmessage 2.4%81,000 B Inserteventintolog 2.6%66,000 C Generatecommitment 11.8%15,000 D Signcommitment 83.3%2,100 Membershipproofs -8,600 (withlocality) Membershipproofs -32 (nolocality) Table2:Performanceoftheloggerineachofthefourstepsre-quiredtoinsertaneventandsigntheresultingcommitmentandingeneratingmembershipproofs.Ratesaregivenassumingnothingotherthanthespeciedstepisbeingperformed.wouldincreaseto10,500eventspersecondiftheDSAsignatureswerecomputedelsewhere(e.g.,leveragingmultipleCPUcores).(Section6discussesscalabilityinmoredetail.)Thiscorrespondsto1.9MB/secofuncompressedsyslogdata(1.1TBperweek).Wealsomeasuredtherateatwhichourprototypecangeneratemembershipandincrementalproofs.Thesizeofanincrementalproofbetweentwocommitmentsdependsuponthedistancebetweenthetwocommitments.Asthedistancevariesfromaroundtwototwomillionevents,thesizeofaself-containedproofvariesfrom1200bytesto2500bytes.Thespeedforgeneratingtheseproofsvariesfrom10,500proofs/secto18,000proofs/sec,withshorterdistanceshavingsmallerproofsizesandfasterperformancethanlongerdistances.Forbothincrementalandmembershipproofs,compressingbygzip[18]halvesthesizeoftheproofs,butalsohalvestherateatwhichproofscanbegenerated.Afterinserting80millioneventsintothehistorytree,thehistorytreeandvaluestorerequire27GB,severaltimeslargerthanourtestmachine'sRAMcapacity.Table2presentsourresultsfortwomembershipauditingscenarios.Inourrstscenariowerequestedmembershipproofsforrandomeventschosenamongthemostrecent5millioneventsinserted.Ourprototypegenerated8,600self-containedmembershipproofspersecond,averaging2,400byteseach.Inthishigh-localityscenario,themostrecent5millioneventswerealreadysittinginRAM.Oursecondscenarioexaminedthesituationwhenauditre-questshadlowlocalitybyrequestingmembershipproofsforrandomeventsanywhereinthelog.Thelogger'sperformancewaslimitedtoourdisk'sseeklatency.Proofsizeaveraged3,100bytesandperformancedegradedto32membershipproofspersecond.(WediscusshowthismightbeovercomeinSection6.2.)Totestthescalabilityofthehistorytree,webench-markedinsertperformanceandauditingperformanceonouroriginal4millioneventsyslogeventtrace,withoutreplication,andthe80millioneventtraceafter20xreplication.Eventinsertionandincrementalauditingareroughly10%sloweronthelargerlog.5.2PerformanceofauditorsandclientsThehistorytreeplacesfewdemandsuponauditorsorclients.Auditorsandclientsmustverifythelogger'scommitmentsignaturesandmustverifythecorrectnessofprunedtreerepliestoauditingrequests.Ourmachinecanverify1,900DSA-1024signaturespersecond.OurcurrenttreeparseriswritteninPythonandisratherslow.Itcanonlyparse480prunedtreespersecond.Oncetheprunedtreehasbeenparsed,ourmachinecanverify9,000incrementalormembershipproofspersecond.Presently,oneauditorcannotverifyproofsasfastastheloggercangeneratethem,butauditorscanclearlyoperateindependentlyofoneanother,inparallel,allowingforexceptionalscaling,ifdesired.5.3MerkleaggregationresultsInthissubsection,wedescribethebenetsofMerkleaggregationingeneratingqueryresultsandinsafedeletion.Inourexperiments,duetolimitationsofourimplementationingeneratinglargeprunedtrees,ourMerkleaggregationexperimentsusedthesmallerfourmillioneventlog.Weused86differentpredicatestoinvestigatethebenetsofsafedeletionandtheoverheadsofMerkleaggregationqueries.Weused52predicates,eachmatch-ingonetag,13predicates,eachmatchingonehost,9predicates,eachmatchingonefacility,6predicates,onematchingeachlevel,and6predicates,eachmatchingthekhighestlogginglevels.ThepredicatesmatchingtagsandhostsuseBloomlters,areinexact,andmayhavefalsepositives.Thiscauses34ofthe65Bloomlterqueryresultstoincludemorenodesthanour“worstcase”expectationforexactpredicates.ByusinglargerBloomlters,wereducethechancesofspuriousmatches.Whena4-of-64Bloomlterisusedfortagsandhostnames,prunedtreesresultingfromsearchqueriesaverage15%fewernodes,atthecostofanextra64bitsofattributesforeachnodeinthehistorytree.Inarealimplementation,theexactparametersoftheBloomlterwouldbestbetunedtomatchasampleoftheeventsbeinglogged.MerkleaggregationandsafedeletionSafedeletionallowsthepurgingofunwantedeventsfromthelog.Auditorsdeneastablepredicateovertheattributesofeventsindicatingwhicheventsmustbekept,andtheloggerkeepsaprunedtreeofonlythosematchingevents.Inourrsttest,wesimulatedthedeletionofalleventsexceptthosefromaparticularhost.Theprunedtreewasgeneratedin14seconds,containing1.92%oftheeventsinthefulllogandserializedto2.29%ofthesizeofthefulltree.Although98.08%oftheeventswerepurged,theloggerwasonlyabletopurge95.1%ofthenodesinthe 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 Fraction of annotations keptFraction of events keptNon-bloom Bloom, 2-of-32 bits Bloom, 4-of-64 bits Worst Case Best Case Figure9:Safedeletionoverhead.Foravarietyofqueries,weplotthefractionofhashesandattributeskeptafterdeletionversusthefractionofeventskept.historytreebecausetheloggermustkeepthehashlabelandattributesfortherootnodesofelidedsubtrees.Whenmeasuringthesizeofaprunedhistorytreegeneratedbysafedeletion,weassumetheloggercacheshashesandattributesforallinteriornodesinordertobeabletoquicklygenerateproofs.Foreachpredicate,wemeasurethekeptratio,thenumberofinteriornodeorstubsinaprunedtreeofallnodesmatchingthepredicatedividedbythenumberofinteriornodesinthefullhistorytree.InFigure9foreachpredicateweplotthekeptratioversusthefractionofeventsmatchingthepredicate.Wealsoplottheanalyticbest-caseandworst-casebounds,basedonacontinuousapproximation.Theminimumoverheadoccurswhenthematchingeventsarecontiguousinthelog.Theworst-caseoccurswheneventsaremax-imallyseparatedinthelog.OurBloom-lterqueriesdoworsethanthe“worst-case”boundbecauseBloomltermatchesareinexactandwillthustriggerfalsepositivematchesoninteriornodes,forcingthemtobekeptintheresultingprunedtree.AlthoughmanyBloomltersdidfarworsethanthe“worst-case,”amongtheBloomltersthatmatchedfewerthan1%oftheeventsinthelog,theloggerisstillabletopurgeover90%ofthenodesinthehistorytreeandoftendidmuchbetterthanthat.MerkleaggregationandauthenticatedqueryresultsInoursecondtest,weexaminetheoverheadsforMerkleaggregationquerylookupresults.Whentheloggergeneratestheresultstoaquery,theresultingprunedtreewillcontainbothmatchingeventsandhistorytreeoverhead,intheformofhashesandattributesforanystubs.Foreachpredicate,wemeasurethequeryoverheadratio—thenumberofstubsandinteriornodesinaprunedtreedividedbythenumberofeventsintheprunedtree.InFigure10weplotthequeryoverheadratioversusthefractionofeventsmatchingthequeryforeachofour86predicates.Thisplotshows,foreacheventmatchingapredicate,proportionallyhowmuchextraoverheadisin- 0.1 1 10 100 1000 10000 100000 1e+06 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 Average annotations in proof per eventFraction of events in the query resultNon-bloom Bloom, 2-of-32 bits Bloom, 4-of-64 bits Worst case Best case Figure10:Queryoverheadperevent.Weplottheratiobe-tweenthenumberofhashesandmatchingeventsintheresultofeachqueryversusthefractionofeventsmatchingthequery.curred,perevent,forauthenticationinformation.Wealsoplottheanalyticbest-caseandworst-casebounds,basedonacontinuousapproximation.Theminimumoverheadoccurswhenthematchingeventsarecontiguousinthelog.Theworst-caseoccurswheneventsaremaximallyseparatedinthelog.Withexactpredicates,theoverheadofauthenticatedqueryresultsisverymodest,andagain,inexactBloomlterquerieswillsometimesdoworsethanthe“worstcase.”6Scalingatamper-evidentlogInthissection,wediscusstechniquestoimprovetheinsertthroughputofthehistorytreebyusingconcurrency,andtoimprovetheauditingthroughputwithreplication.Wealsodiscussatechniquetoamortizetheoverheadofadigitalsignatureoverseveralevents.6.1FasterinsertsviaconcurrencyOurtamper-evidentlogoffersmanyopportunitiestoleverageconcurrencytoincreasethroughput.Perhapsthesimplestapproachistoofoadsignaturegeneration.FromTable2,signaturesaccountforover80%oftheruntimecostofaninsert.Signaturesarenotincludedinanyotherhashesandtherearenointerdependenciesbetweensignaturecomputations.Furthermore,signingacommitmentdoesnotrequireknowinganythingotherthantherootcommitmentofthehistorytree.Conse-quently,it'seasytoofoadsignaturecomputationsontoadditionalCPUcores,additionalhosts,orhardwarecryptoacceleratorstoimprovethroughput.Itispossibleforaloggertoalsogeneratecommitmentsconcurrently.IfweexamineTable2,parsingandinsertingeventsinthelogisabouttwotimesfasterthangeneratingcommitments.Likesignatures,commitmentshavenointerdependenciesononeother;theydependonlyonthehistorytreecontents.AssoonaseventXjisinsertedintothetreeandO(1)frozenhashesarecomputedandstored, aneweventmaybeimmediatelylogged.ComputingthecommitmentCjonlyrequiresread-onlyaccesstothehistorytree,allowingittobecomputedconcurrentlybyanotherCPUcorewithoutinterferingwithsubsequentevents.Byusingsharedmemoryandtakingadvantageoftheappend-onlywrite-oncesemanticsofthehistorytree,wewouldexpectconcurrencyoverheadtobelow.Wehaveexperimentallyveriedthemaximumrateatwhichourprototypeimplementation,describedinSection5,caninsertsyslogeventsintothelogat38,000eventspersecondusingonlyoneCPUcoreoncommodityhardware.Thisisthemaximumthroughputourhardwarecouldpotentiallysupport.Inthismodeweassumethatdigitalsignatures,commitmentgeneration,andauditrequestsaredelegatedtoadditionalCPUcoresorhosts.Withmultiplehosts,eachhostmustbuildareplicaofthehistorytreewhichcanbedoneatleastasfastasourmaximuminsertrateof38,000eventspersecond.AdditionalCPUcoresonthesehostscanthenbeusedforgeneratingcommitmentsorhandlingauditrequests.Forsomeapplications,38,000eventspersecondmaystillnotbefastenough.Scalingbeyondthiswouldrequirefragmentingtheeventinsertionandstoragetasksacrossmultiplelogs.Tobreakinterdependenciesbetweenthem,thefundamentalhistorytreedatastructurewepresentlyusewouldneedtoevolve,perhapsintodisjointlogsthatoccasionallyentanglewithoneanotherasintimelineentanglement[43].Designingandevaluatingsuchastructureisfuturework.6.2LogslargerthanRAMForexceptionallylargeauditsorqueries,wheretheworkingsetsizedoesnottintoRAM,weobservedthatthroughputwaslimitedtodiskseeklatency.Similarissuesoccurinanydatabasequerysystemthatusessecondarystorage,andthesamesoftwareandhardwaretechniquesusedbydatabasestospeedupqueriesmaybeused,includingfasterorhigherthroughputstoragesystemsorpartitioningthedataandstoringitin-memoryacrossaclusterofmachines.Asinglelargequerycanthenbeissuedtotheclusternodemanagingeachsub-tree.Theresultswouldthenbemergedbeforetransmittingtheresultstotheauditor.Becauseeachsub-treewouldtinitshost'sRAM,sub-querieswouldrunquickly.6.3SigningbatchesofeventsWhenlargecomputerclustersareunavailableandtheperformancecostofDSAsignaturesisthelimitingfactorinthelogger'sthroughput,wemayimproveperformanceoftheloggerbyallowingmultipleupdatestobehandledwithonesignaturecomputation.Normally,whenaclientrequestsaneventXtobeinserted,theloggerassignsitanindexi,generatesthecommitmentCi,signsit,andreturnstheresult.IftheloggerhasinsufcientCPUtosigneverycommitment,theloggercouldinsteaddelayreturningCiuntilithasasignatureforsomelatercommitmentCj(ji).Thislatersignedcommitmentcouldthenbesenttotheclientexpectinganearlierone.ToensurethattheeventXiinthelogcommittedbyCjwasX,theclientmayrequestamembershipprooffromcommitmentCjtoeventiandverifythatXi=X.Thisissafeduetothetamper-evidenceofourstructure.IftheloggerwereevertolatersignaCiinconsistentwithCj,itwouldfailanincrementalproof.Inourprototype,insertingeventsintothelogistwentytimesfasterthangeneratingandsigningcommitments.Theloggermayamortizethecostsofgeneratingasignedcommitmentovermanyinsertedevents.Thenumberofeventspersignedcommitmentcouldvarydynamicallywiththeloadonthelogger.Underlightload,theloggercouldsigneverycommitmentandinsert1,750eventspersecond.Withincreasingload,theloggermightsignoneinevery16commitmentstoobtainanestimatedinsertrateof17,000eventspersecond.Clientswillstillreceivesignedcommitmentswithinafractionofasecond,butseveralclientscannowreceivethesamecommitment.Notethatthisanalysisonlyconsidersthemaximuminsertratefortheloganddoesnotincludethecostsofreplyingtoaudits.Theoverallperformanceimprovementsdependonhowoftenclientsrequestincrementalandmembershipproofs.7RelatedworkTherehasbeenrecentinterestincreatingappend-onlydatabasesforregulatorycompliance.Thesedatabasespermittheabilitytoaccessoldversionsandtracetam-pering[51].Avarietyofdifferentdatastructuresareused,includingaB-tree[64]andafulltextindex[47].Thesecurityofthesesystemsdependsonawrite-oncesemanticsoftheunderlyingstoragethatcannotbeindependentlyveriedbyaremoteauditor.Forward-securedigitalsignatureschemes[3]orstreamauthentication[21]canbeusedforsigningcommitmentsinourschemeoranyotherloggingscheme.Entriesinthelogmaybeencryptedbyclientsforprivacy.KelseyandSchneier[57]havetheloggerencryptentrieswithakeydestroyedafteruse,preventinganattackerfromreadingpastlogentries.Ahashfunctionisiteratedtogeneratetheencryptionkeys.Theinitialhashissenttoatrustedauditorsothatitmaydecryptevents.Logcrypt[29]extendsthistopublickeycryptography.MaandTsudik[41]considertamper-evidentlogsbuiltusingforward-securesequentialaggregatingsignatureschemes[39,40].Theirdesignisround-based.Withineachround,theloggerevolvesitssignature,combininganeweventwiththeexistingsignaturetogenerateanewsignature,andalsoevolvestheauthenticationkey.Attheendofaround,thenalsignaturecanauthenticateanyeventinserted. Daviset.al.[17]permitskeywordsearchinginalogbytrustingtheloggertobuildparallelhashchainsforeachkeyword.Techniqueshavealsobeendesignedforkeywordsearchingencryptedlogs[60,61].Atamper-evidentstoreforvotingmachineshasbeenproposed,basedonappend-onlysignatures[33],butthesignaturesizesgrowwiththenumberofsignedmessages[6].Manytimestampingserviceshavebeenproposedintheliterature.HaberandStornetta[27]introduceatime-stampingservicebasedonhashchains,whichinuencedthedesignofSurety,acommercialtimestampingservicethatpublishestheirheadcommitmentinanewspaperonceaweek.Chronosisadigitaltimestampingserviceinspiredbyaskiplist,butwithahashingstructuresimilartoourhistorytree[7].Thisandothertimestampingdesigns[9,10]areround-based.Ineachround,theloggercollectsasetofeventsandstorestheeventswithinthatroundinatree,skiplist,orDAG.Attheendoftheroundtheloggerpubliclybroadcasts(e.g.,inanewspaper)thecommitmentforthatround.Clientsthenobtainalogarithmically-sized,tamper-evidentproofthattheireventsarestoredwithinthatroundandareconsistentwiththepublishedcommitment.Efcientalgorithmshavebeenconstructedforoutputtingtimestampau-thenticationinformationforsuccessiveeventswithinaroundinastreamingfashion,withminimalstorageontheserver[37].Unlikethesesystems,ourhistorytreeallowseventstobeaddedtothelog,commitmentsgenerated,andauditstobeperformedatanytime.ManiatisandBaker[43]introducedtheideaoftimelineentanglement,whereeveryparticipantinadistributedsystemmaintainsalog.Everytimeamessageisreceived,itisaddedtothelog,andeverymessagetransmittedcontainsthehashoftheloghead.Thisprocessspreadscommitmentsthroughoutthenetwork,makingitharderformaliciousnodestodivergefromthecanonicaltime-linewithouttherebeingevidencesomewherethatcouldbeusedinanaudittodetecttampering.Auditorium[55]usesthispropertytocreateashared“bulletinboard”thatcandetecttamperingevenwhenN1systemsarefaulty.Secureaggregationhasbeeninvestigatedasadis-tributedprotocolinsensornetworksforcomputingsums,medians,andotheraggregatevalueswhenthehostdoingtheaggregationisnottrusted.Techniquesincludetradingoffapproximateresultsinreturnforsublinearcommunicationcomplexity[12],orusingMACcodestodetectone-hoperrorsincomputingaggregates[30].OtheraggregationprotocolshavebeenbasedaroundhashtreestructuressimilartotheoneswedevelopedforMerkleaggregation.Thesestructurescombineaggrega-tionandcryptographichashing,andincludedistributedsensor-networkaggregationprotocolsforcomputingau-thenticatedsums[13]andgenericaggregation[45].Thesensornetworkaggregationprotocolsinteractivelygener-ateasecureaggregateofasetofmeasurements.InMerkleaggregation,weuseintermediateaggregatesasatoolforperformingefcientqueries.Also,ourMerkleaggre-gationconstructionismoreefcientthanthesedesigns,requiringfewercryptographichashestoverifyanevent.8ConclusionsInthisworkwehaveshownthatregularandcontinousauditingisacriticaloperationforanytamper-evidentlogsystem,forwithoutauditing,clientscannotdetectifaByzantineloggerismisbehavingbynotloggingevents,removingunauditedevents,orforkingthelog.Fromthisrequirementwehavedevelopedanewtamper-evidentlogdesign,basedonanewMerkletreedatastructurethatpermitsaloggertoproduceconciseproofsofitscorrectbehavior.Oursystemeliminatesanyneedtotrustthelogger,insteadallowingclientsandauditorsoftheloggertoefcientlyverifyitscorrectbehaviorwithonlyaconstantamountoflocalstate.Bysharingcommitmentsamongclientsandauditors,ourdesignisresistanteventosophisticatedforkingorrollbackattacks,evenincaseswhereaclientmightchangeitsmindandtrytorepudiateeventsthatithadloggedearlier.WealsoproposedMerkleaggregation,aexiblemechanismforencodingauxiliaryattributesintoaMerkletreethatallowstheseattributestobeaggregatedfromtheleavesuptotherootofthetreeinaveriablefashion.Thistechniquepermitsawiderangeofefcient,tamper-evidentqueries,aswellasenablingveriable,safedeletionof“expired”eventsfromthelog.Ourprototypeimplementationsupportsthousandsofeventspersecond,andcaneasilyscaletoverylargelogs.WealsodemonstratedtheeffectivenessofBloomlterstoenableabroadrangeofqueries.Byvirtueofitsconciseproofsandscalabledesign,ourtechniquescanbeappliedinavarietyofdomainswherehighvolumesofloggedeventsmightotherwiseprecludetheuseoftamper-evidentlogs.AcknowledgementsTheauthorsgratefullyacknowledgeFarinazKoushan-far,DanielSandler,andMosheVardiformanyhelpfulcommentsanddiscussionsonthisproject.TheauthorsalsothanktheanonymousrefereesandMicahSherr,ourshepherd,fortheirassistance.Thisworkwassupported,inpart,byNSFgrantsCNS-0524211andCNS-0509297.References[1]ACCORSI,R.,ANDHOHL,A.Delegatingsecurelogginginpervasivecomputingsystems.InSecurityinPervasiveComputing(York,UK,Apr.2006),pp.58–72.[2]ANAGNOSTOPOULOS,A.,GOODRICH,M.T.,ANDTAMASSIA,R.Persistentauthenticateddictionariesandtheirapplications.InInternationalConferenceon InformationSecurity(ISC)(Seoul,Korea,Dec.2001),pp.379–393.[3]BELLARE,M.,ANDMINER,S.K.Aforward-securedigitalsignaturescheme.InCRYPTO'99(SantaBarbara,CA,Aug.1999),pp.431–448.[4]BELLARE,M.,ANDYEE,B.S.Forwardintegrityforsecureauditlogs.Tech.rep.,UniversityofCaliforniaatSanDiego,Nov.1997.[5]BENALOH,J.,ANDDEMARE,M.One-wayaccumulators:adecentralizedalternativetodigitalsignatures.InWorkshopontheTheoryandApplicationofCryptographicTechniquesonAdvancesinCryptology(EuroCrypt'93)(Lofthus,Norway,May1993),pp.274–285.[6]BETHENCOURT,J.,BONEH,D.,ANDWATERS,B.Cryptographicmethodsforstoringballotsonavotingmachine.InNetworkandDistributedSystemSecuritySymposium(NDSS)(SanDiego,CA,Feb.2007).[7]BLIBECH,K.,ANDGABILLON,A.CHRONOS:Anauthenticateddictionarybasedonskiplistsfortimestampingsystems.InWorkshoponSecureWebServices(Fairfax,VA,Nov.2005),pp.84–90.[8]BLOOM,B.H.Space/timetrade-offsinhashcodingwithallowableerrors.CommunicationsoftheACM13,7(1970),422–426.[9]BULDAS,A.,LAUD,P.,LIPMAA,H.,ANDWILLEMSON,J.Time-stampingwithbinarylinkingschemes.InCRYPTO'98(SantaBarbara,CA,Aug.1998),pp.486–501.[10]BULDAS,A.,LIPMAA,H.,ANDSCHOENMAKERS,B.Optimallyefcientaccountabletime-stamping.InInternationalWorkshoponPracticeandTheoryinPublicKeyCryptography(PKC)(Melbourne,Victoria,Australia,Jan.2000),pp.293–305.[11]CAMENISCH,J.,ANDLYSYANSKAYA,A.Dynamicaccumulatorsandapplicationtoefcientrevocationofanonymouscredentials.InCRYPTO'02(SantaBarbara,CA,Aug.2002),pp.61–76.[12]CHAN,H.,PERRIG,A.,PRZYDATEK,B.,ANDSONG,D.SIA:Secureinformationaggregationinsensornetworks.JournalComputerSecurity15,1(2007),69–102.[13]CHAN,H.,PERRIG,A.,ANDSONG,D.Securehierarchicalin-networkaggregationinsensornetworks.InACMConferenceonComputerandCommunicationsSecurity(CCS'06)(Alexandria,VA,Oct.2006),pp.278–287.[14]CHOR,B.,GOLDREICH,O.,KUSHILEVITZ,E.,ANDSUDAN,M.Privateinformationretrieval.InAnnualSymposiumonFoundationsofComputerScience(Milwaukee,WI,Oct.1995),pp.41–50.[15]CHUN,B.-G.,MANIATIS,P.,SHENKER,S.,ANDKUBIATOWICZ,J.Attestedappend-onlymemory:Makingadversariessticktotheirword.InSOSP'07(Stevenson,WA,Oct.2007),pp.189–204.[16]D.S.PARKER,J.,POPEK,G.J.,RUDISIN,G.,STOUGHTON,A.,WALKER,B.J.,WALTON,E.,CHOW,J.M.,EDWARDS,D.,KISER,S.,ANDKLINE,C.Detectionofmutualinconsistencyindistributedsystems.IEEETransactionsonSoftwareEngineering9,3(1983),240–247.[17]DAVIS,D.,MONROSE,F.,ANDREITER,M.K.Time-scopedsearchingofencryptedauditlogs.InInformationandCommunicationsSecurityConference(Malaga,Spain,Oct.2004),pp.532–545.[18]DEUTSCH,P.Gzipleformatspecicationversion4.3.RFC1952,May1996.http://www.ietf.org/rfc/rfc1952.txt.[19]DEVANBU,P.,GERTZ,M.,KWONG,A.,MARTEL,C.,NUCKOLLS,G.,ANDSTUBBLEBINE,S.G.FlexibleauthenticationofXMLdocuments.JournalofComputerSecurity12,6(2004),841–864.[20]DEVANBU,P.,GERTZ,M.,MARTEL,C.,ANDSTUBBLEBINE,S.G.Authenticdatapublicationovertheinternet.JournalComputerSecurity11,3(2003),291–314.[21]GENNARO,R.,ANDROHATGI,P.Howtosigndigitalstreams.InCRYPTO'97(SantaBarbara,CA,Aug.1997),pp.180–197.[22]GERR,P.A.,BABINEAU,B.,ANDGORDON,P.C.Compliance:Theeffectoninformationmanagementandthestorageindustry.TheEnterpriseStorageGroup,May2003.http://searchstorage.techtarget.com/tip/0,289483,sid5 gci906152,00.html.[23]GOH,E.-J.Secureindexes.CryptologyePrintArchive,Report2003/216,2003.http://eprint.iacr.org/2003/216/Seealsohttp://eujingoh.com/papers/secureindex/.[24]GOODRICH,M.,TAMASSIA,R.,ANDSCHWERIN,A.Implementationofanauthenticateddictionarywithskiplistsandcommutativehashing.InDARPAInformationSurvivabilityConference&ExpositionII(DISCEXII)(Anaheim,CA,June2001),pp.68–82.[25]GOODRICH,M.T.,TAMASSIA,R.,TRIANDOPOULOS,N.,ANDCOHEN,R.F.Authenticateddatastructuresforgraphandgeometricsearching.InTopicsinCryptology,TheCryptographers'TrackattheRSAConference(CT-RSA)(SanFrancisco,CA,Apr.2003),pp.295–313.[26]GOYAL,V.,PANDEY,O.,SAHAI,A.,ANDWATERS,B.Attribute-basedencryptionforne-grainedaccesscontrolofencrypteddata.InACMConferenceonComputerandCommunicationsSecurity(CCS'06)(Alexandria,Virginia,Oct.2006),pp.89–98.[27]HABER,S.,ANDSTORNETTA,W.S.Howtotime-stampadigitaldocument.InCRYPTO'98(SantaBarbara,CA,1990),pp.437–455.[28]HAEBERLEN,A.,KOUZNETSOV,P.,ANDDRUSCHEL,P.PeerReview:Practicalaccountabilityfordistributedsystems.InSOSP'07(Stevenson,WA,Oct.2007).[29]HOLT,J.E.Logcrypt:Forwardsecurityandpublicvericationforsecureauditlogs.InAustralasianWorkshopsonGridComputingandE-research(Hobart,Tasmania,Australia,2006).[30]HU,L.,ANDEVANS,D.Secureaggregationforwirelessnetworks.InSymposiumonApplicationsandtheInternetWorkshops(SAINT)(Orlando,FL,July2003),p.384.[31]ITKIS,G.Cryptographictamperevidence.InACMConferenceonComputerandCommunicationsSecurity(CCS'03)(WashingtonD.C.,Oct.2003),pp.355–364.[32]KELSEY,J.,CALLAS,J.,ANDCLEMM,A.SignedSyslogmessages.http://tools.ietf.org/id/draft-ietf-syslog-sign-23.txt(workinprogress),Sept.2007.[33]KILTZ,E.,MITYAGIN,A.,PANJWANI,S.,ANDRAGHAVAN,B.Append-onlysignatures.InInternationalColloquiumonAutomata,LanguagesandProgramming(Lisboa,Portugal,July2005).[34]KOCHER,P.C.Oncerticaterevocationandvalidation.InInternationalConferenceonFinancialCryptography (FC'98)(Anguilla,BritishWestIndies,Feb.1998),pp.172–177.[35]KOTLA,R.,ALVISI,L.,DAHLIN,M.,CLEMENT,A.,ANDWONG,E.Zyzzyva:Speculativebyzantinefaulttolerance.InSOSP'07(Stevenson,WA,Oct.2007),pp.45–58.[36]LI,J.,KROHN,M.,MAZIERES,D.,ANDSHASHA,D.Secureuntrusteddatarepository(SUNDR).InOperatingSystemsDesign&Implementation(OSDI)(SanFrancisco,CA,Dec.2004).[37]LIPMAA,H.Onoptimalhashtreetraversalforintervaltime-stamping.InProceedingsofthe5thInternationalConferenceonInformationSecurity(ISC02)(Seoul,Korea,Nov.2002),pp.357–371.[38]LONVICK,C.TheBSDSyslogprotocol.RFC3164,Aug.2001.http://www.ietf.org/rfc/rfc3164.txt.[39]MA,D.Practicalforwardsecuresequentialaggregatesignatures.InProceedingsofthe2008ACMsymposiumonInformation,computerandcommunicationssecurity(ASIACCS'08)(Tokyo,Japan,Mar.2008),pp.341–352.[40]MA,D.,ANDTSUDIK,G.Forward-securesequentialaggregateauthentication.InProceedingsofthe2007IEEESymposiumonSecurityandPrivacy(Oakland,CA,May2007),IEEEComputerSociety,pp.86–91.[41]MA,D.,ANDTSUDIK,G.Anewapproachtosecurelogging.TransactionsonStorage5,1(2009),1–21.[42]MANIATIS,P.,ANDBAKER,M.Enablingthearchivalstorageofsigneddocuments.InFAST'02:Proceedingsofthe1stUSENIXConferenceonFileandStorageTechnologies(Monterey,CA,2002).[43]MANIATIS,P.,ANDBAKER,M.Securehistorypreservationthroughtimelineentanglement.InUSENIXSecuritySymposium(SanFrancisco,CA,Aug.2002).[44]MANIATIS,P.,ROUSSOPOULOS,M.,GIULI,T.J.,ROSENTHAL,D.S.H.,ANDBAKER,M.TheLOCKSSpeer-to-peerdigitalpreservationsystem.ACMTransactionsonComputerSystems23,1(2005),2–50.[45]MANULIS,M.,ANDSCHWENK,J.Provablysecureframeworkforinformationaggregationinsensornetworks.InComputationalScienceandItsApplications(ICCSA)(KualaLumpur,Malaysia,Aug.2007),pp.603–621.[46]MERKLE,R.C.Adigitalsignaturebasedonaconventionalencryptionfunction.InCRYPTO'88(1988),pp.369–378.[47]MITRA,S.,HSU,W.W.,ANDWINSLETT,M.Trustworthykeywordsearchforregulatory-compliantrecordsretention.InInternationalConferenceonVeryLargeDatabases(VLDB)(Seoul,Korea,Sept.2006),pp.1001–1012.[48]MONTEIRO,S.D.S.,ANDERBACHER,R.F.ExemplifyingattackidenticationandanalysisinanovelforensicallyviableSyslogmodel.InWorkshoponSystematicApproachestoDigitalForensicEngineering(Oakland,CA,May2008),pp.57–68.[49]NAOR,M.,ANDNISSIM,K.Certicaterevocationandcerticateupdate.InUSENIXSecuritySymposium(SanAntonio,TX,Jan.1998).[50]OSTROVSKY,R.,SAHAI,A.,ANDWATERS,B.Attribute-basedencryptionwithnon-monotonicaccessstructures.InACMConferenceonComputerandCommunicationsSecurity(CCS'07)(Alexandria,VA,Oct.2007),pp.195–203.[51]PAVLOU,K.,ANDSNODGRASS,R.T.Forensicanalysisofdatabasetampering.InACMSIGMODInternationalConferenceonManagementofData(Chicago,IL,June2006),pp.109–120.[52]PETERSON,Z.N.J.,BURNS,R.,ATENIESE,G.,ANDBONO,S.Designandimplementationofveriableaudittrailsforaversioninglesystem.InUSENIXConferenceonFileandStorageTechnologies(SanJose,CA,Feb.2007).[53]PUGH,W.Skiplists:Aprobabilisticalternativetobalancedtrees.InWorkshoponAlgorithmsandDataStructures(1989),pp.437–449.[54]SAHAI,A.,ANDWATERS,B.Fuzzyidentitybasedencryption.InWorkshopontheTheoryandApplicationofCryptographicTechniquesonAdvancesinCryptology(EuroCrypt'05)(May2005),vol.3494,pp.457–473.[55]SANDLER,D.,ANDWALLACH,D.S.CastingvotesintheAuditorium.InUSENIX/ACCURATEElectronicVotingTechnologyWorkshop(EVT'07)(Boston,MA,Aug.2007).[56]SCHNEIER,B.,ANDKELSEY,J.Automaticevent-streamnotarizationusingdigitalsignatures.InSecurityProtocolsWorkshop(Cambridge,UK,Apr.1996),pp.155–169.[57]SCHNEIER,B.,ANDKELSEY,J.Secureauditlogstosupportcomputerforensics.ACMTransactionsonInformationandSystemSecurity1,3(1999).[58]SION,R.StrongWORM.InInternationalConferenceonDistributedComputingSystems(Beijing,China,May2008),pp.69–76.[59]SNODGRASS,R.T.,YAO,S.S.,ANDCOLLBERG,C.Tamperdetectioninauditlogs.InConferenceonVeryLargeDataBases(VLDB)(Toronto,Canada,Aug.2004),pp.504–515.[60]SONG,D.X.,WAGNER,D.,ANDPERRIG,A.Practicaltechniquesforsearchesonencrypteddata.InIEEESymposiumonSecurityandPrivacy(Berkeley,CA,May2000),pp.44–55.[61]WATERS,B.R.,BALFANZ,D.,DURFEE,G.,ANDSMETTERS,D.K.Buildinganencryptedandsearchableauditlog.InNetworkandDistributedSystemSecuritySymposium(NDSS)(SanDiego,CA,Feb.2004).[62]WEATHERSPOON,H.,WELLS,C.,ANDKUBIATOWICZ,J.Namingandintegrity:Self-verifyingdatainpeer-to-peersystems.InFutureDirectionsinDistributedComputing(2003),vol.2584ofLectureNotesinComputerScience,pp.142–147.[63]YUMEREFENDI,A.R.,ANDCHASE,J.S.Strongaccountabilityfornetworkstorage.ACMTransactionsonStorage3,3(2007).[64]ZHU,Q.,ANDHSU,W.W.Fossilizedindex:Thelinchpinoftrustworthynon-alterableelectronicrecords.InACMSIGMODInternationalConferenceonManagementofData(Baltimore,MD,June2005),pp.395–406.