weneedaneffectiveandefcientmechanismtodiscovernewhookingbehaviorsandunderstandtheirhookingmechanismsUnfortunatelytheexistingmalwareanalysisprocedureispainstakingmostlymanualanderrorproneVario ID: 291325
Download Pdf The PPT/PDF document "HookFinder:IdentifyingandUnderstandingMa..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
HookFinder:IdentifyingandUnderstandingMalwareHookingBehaviorsHengYinyzZhenkaiLiangyDawnSongxyhyin@ece.cmu.eduzliang@cmu.edudawnsong@cs.berkeley.eduyCarnegieMellonUniversity,Pittsburgh,PA,USAzCollegeofWilliamandMary,Williamsburg,VA,USAxUCBerkeley,Berkeley,CA,USAAbstractInstallingvarioushooksintothevictimsystemisanimportantattackingstrategyemployedbymalware,in-cludingspyware,rootkits,stealthbackdoors,andothers.Inordertodefeatexistinghookdetectors,malwarewrit-erskeepexploringnewhookingmechanisms.However,thecurrentmalwareanalysisprocedureispainstak-ing,mostlymanualanderror-prone.Inthispaper,weproposetherstsystematicapproachforautomat-icallyidentifyinghooksandextractinghookingmech-anisms.Weproposeauniedapproach,ne-grainedimpactanalysis,toidentifymalwarehookingbehaviors.Ourapproachdoesnotrelyonanypriorknowledgeofhookingmechanisms,andthuscanidentifynovelhook-ingmechanisms.Moreover,wedeviseamethodusingsemantics-awareimpactdependencyanalysistoprovideasuccinctandintuitivegraphrepresentationtoillus-tratehookingmechanisms.Wehavedevelopedaproto-type,HookFinder,andconductedextensiveexperimentsusingrepresentativemalwaresamplesfromvariouscat-egories.WehavedemonstratedthatHookFindercancorrectlyidentifythehookingbehaviorsofallsamples,andprovideaccurateinsightsabouttheirhookingmech-anisms.1IntroductionThearmsracebetweenmalwarewritersandmalwaredefendersisescalating.Inordertoevademalwarede-fensetechniques,malwarewritersarealwaysstrivingtoexplorenovelattackingtechniques.Inresponse,mal-waredefendersmustaccuratelyandresponsivelyunder-standmalware'sattackingvectorstogainanupperhand.Oneimportantmalwareattackingvectorisitshook-ingmechanism.Maliciousprogramsimplanthooksformanydifferentpurposes.Spywaremayimplanthookstogetnotiedofthearrivalofnewsensitivedata.Forex-ample,keyloggersmayinstallhookstointerceptusers'keystrokes;passwordthievesmayinstallhookstogetnotiedoftheinputofusers'passwords;networksnif-fersmayinstallhookstoeavesdroponincomingnet-worktrafc;andBHO-basedadwaremayalsoinstallhookstocaptureURLsandothersensitiveinformationfromincomingwebpages.Inaddition,rootkitsmayimplanthookstointerceptandtamperwithcriticalsys-teminformationtoconcealtheirpresenceinthesystem.Malwarewithastealthbackdoormayalsoplacehooksonthenetworkstacktoestablishastealthycommunica-tionchannelwithremoteattackers.Severaltools[4,13,24]detecthookingbehaviorsbycheckingknownmemoryregionsforsuspiciousentries.However,theyneedpriorknowledgeofhowexistingmalwareimplantshooks.Therefore,theybecomefu-tilewhenmalwareusesnewhookingmechanisms.Thisconcernisnothypothetical.Recently,newstealthyker-nelbackdoors(deepdoor[26]anduay[30])arereportedtoemployanovelhookingmechanismforinterceptingthenetworkstack.Tosetuphooks,theyoverwriteonlyasmallportioninNDISdatablock.Withoutknowingthisparticularhookingmechanism,wecanhardlynoticethiskindofhooks.Infact,allexistinghookdetectionmeth-odshavefailedtodetectthiskindofhooks.Inresponsetorapidlyevolvingmalwaretechniques, weneedaneffectiveandefcientmechanism,todis-covernewhookingbehaviorsandunderstandtheirhook-ingmechanisms.Unfortunately,theexistingmalwareanalysisprocedureispainstaking,mostlymanualanderror-prone.Variouscodeobfuscationtechniquesusedinmalwaremakethismanualprocessevenmoredif-cult.Inthispaper,weproposetherstsystematicap-proachtothisresearchproblem.Inparticular,givenanunknownmaliciousbinary,weaimtoidentifyifthiscodeinstallsanyhooksintothesystem,andifso,pro-videdetailedinformationabouthowitinstallsthehooks.Theintuitionofourapproachisthatahookimplantedbyapieceofmaliciouscodeisoneoftheimpacts(intermsofmemoryandregisters)thatthemaliciouscodehasmadetothewholesystem,andthisimpacteven-tuallyaffectstheexecutionowofthesystemtojumpintothemaliciouscode.Inordertocapturethisdistinctbehavior,weproposeanovelapproach,ne-grainedim-pactanalysis.Itworksbyidentifyingalltheimpactsmadebythemaliciouscode,andkeepingtrackoftheimpactsowingacrossthewholesystem.Ifthecon-trolowisaffectedbyoneoftheseimpactstojumpintothemaliciouscode,thenwedeterminethatthistransi-tioniscausedbyahook,whichisinstalledbythemali-ciouscode.Tounderstandhowthishookisimplanted,wedeviseasemantics-awareimpactdependencyanaly-sismechanism.Itperformsdependencyanalysisonthehistoryofimpactpropagation,leveragedwithOS-levelsemantics.WehaveprototypedourapproachintoatoolcalledHookFinder,andevaluateditwitheightmalwarepro-grams.Intheexperiment,HookFinderidentiedhook-ingbehaviorsofeachmalwaresamplewithinminutes.Foreachidentiedhookingbehavior,HookFindergavevaluableinsightsanddetailsabouttheunderlyinghook-ingmechanism.TheefciencyandeffectivenessofHookFindermakesitpossibletoautomaticallycatego-rizehookingbehaviorsofthelargevolumeofmalwaresamplesreceivedbyanti-viruscompanieseveryday,andinstantlyrealizeandrespondtonovelhookingmecha-nisms.Insummary,thispapermakesthefollowingcontri-butions:Weproposene-grainedimpactanalysisasauni-edapproachtoidentifyingthehookingbehaviorofmaliciouscode.Sinceitdoesnotrelyonanypriorknowledgeofhookingmechanisms,ourap-proachiswellsuitedforidentifyingnovelhookingmechanisms.Inordertoprovidevaluableinsightsabouthowmalwareimplantshooks,wedeviseasemantics-awareimpactdependencyanalysismethod,whichprovidesasuccinctandintuitivegraphicalrepre-sentationtohelpmalwareanalystsunderstandthehookingmechanismemployedbyapieceofmal-ware.WehavedesignedanddevelopedHookFindertodemonstratethefeasibilityofourapproach.Wehaveconductedextensiveexperimentswithrepre-sentativemalwaresamplesfromvariouscategories,anddemonstratedthatHookFindercouldcorrectlyidentifytheirhookingbehaviors,andprovideaccu-rateinsightsabouttheirhookingmechanisms.Thepaperisstructuredasfollows.Thenextsectiongivesanoverviewofourapproach.Section 3 describesdetailsonthedesignandimplementationofHookFinder.Section 4 presentstheexperimentalresults.Section 5 discussessomerelatedissues.Section 6 surveysrelatedworkandSection 7 concludesthepaper.2ProblemStatementandOurApproachInthissection,weformalizetheproblemofhook-ingbehaviordetectionandanalysis,andgiveabriefoverviewofourapproach.2.1ProblemStatementGivenamalwaresample,ourapproachrstdeter-mineswhetheritcontainshookingbehaviors.Ahookingbehaviorcanbeformalizedasfollows.Amaliciouspro-gramCattemptstochangeamemorylocationLoftheoperatingsystem,toimplantahookH.Whenacertaineventhappens,theoperatingsystemwillloadthehookH,andthenstartstoexecutemaliciouscodeFinpro-gramC.WerefertotheaddressofFashookentry,andLashooksite.Figure 1 (a)showsapieceofpseudocodethathooksanentryintheSystemServiceDescriptorTa-ble(SSDT)ofWindowssystem.Thishookingmech-anismisusedinmanykernel-modemalwaresamples,suchastheSonyRootkit[27].Inthisexample,thehookentryFisNewZwOpenKey,andthehooksiteListheentryforZwOpenKeyintheservicedescriptortable,andthehookHistheaddressofNewZwOpenKey,asillustratedinFigure 1 (b).Ifourapproachdetectshookingbehaviorsinamal-waresample,itoutputsagraphrepresentationofthehookingmechanism,hookgraph.Thehookgraphtellsustwomaincharacteristicsofahookingmechanism:hooktypeandimplantingmechanism. #defineSYSTEMSERVICE(_function)\KeServiceDescriptorTable.ServiceTableBase\[*(PULONG)((PUCHAR)_function+1)]voidHookSyscalls(){...OldZwOpenKey=SYSTEMSERVICE(ZwOpenKey);SYSTEMSERVICE(ZwOpenKey)=NewZwOpenKey;...} SSDTL: Hook SiteC: Malicious ProgramF: NewZwOpenKeyH: HookZwOpenKey (a)(b)Figure1.AnSSDTHookingExample.ThiscodeattemptstohookZwOpenKey,bywritingtheaddressofitsownfunctionNewZwOpenKeyintothecorrespondingentryoftheSSDTKeServiceDescirptorTable.HookTypeDependinghowitisinterpretedbytheCPU,ahookHcanbeeitheradatahookoracodehook.AdatahookisinterpretedasdatabytheCPU,andisusedasthedestinationaddressofsomecontroltransferinstructiontojumpintothehookentryF.Forexample,thehookinFigure 1 isadatahook,becauseitistheaddressofthehookentry,andisinterpretedasthejumptarget.AcodehookisinterpretedascodebytheCPU.Acodehookcontainsajump-likeinstruction(suchasjmpandcall),andisinjectedtooverwritesomesystemcode(suchaskernelmodulesandcommonDLLs).Whenacodehookisactivated,theexecutionisredirectedintothemaliciouscodeF.Weneedtodetecthookingbehaviorsinbothcases,andweshouldbeabletotellwhatkindofhookitiswhenwedetectone.Aswewillseelater,thepoliciesusedtodetecthookingbe-haviorsaredifferentbetweenthesetwocategoriesduetotheirdifferentnature.ImplantingMechanismMalwarehastwochoicestoinstallHintoL.First,itmaydirectlywriteHintoLusingitsowncode.Second,itmaycallafunctiontoachieveitonitsbehalf.Windowssys-temprovidesseveralAPIsforapplicationstoregis-tervariouseventhandlers(i.e.,hooks).Forexample,SetWindowsHookExallowsanapplicationtoregis-terahookforcertainWindowsevent,suchaskeystrokeevents.Wheneverakeystrokeisenteredintothesys-tem,Windowswillcallthehookfunctionprovidedbythisapplication.Inaddition,functionslikememcpyandWriteProcessMemorycanoverwriteamemoryre-giononbehalfoftheircallers.Thus,onceweidentifyahook,weneedtodeterminewhichmethodthemalwareusedtoregisterthehook.IfthemalwaredirectlymodiesLtoinstallH,weneedtounderstandwhereLis,andhowthemalwaresampleobtainsL.SinceLisusuallynotlocatedinaxedplace,malwarehastonditfromsomestaticpoint.Thisstaticpointcanbeaglobalsystemsym-bol,ortheresultofafunctioncall.Afterobtainingthisstaticpoint,malwaremaywalkthroughthedatastructuresreferencedbyittoeventuallylocateL.TheexampleinFigure 1 makesuseofthismethod,andthehooksiteLiscalculatedfromaglobalsymbolKeServiceDescriptorTable.Forthistypeofimplantingmechanism,thehookgraphanswersthefol-lowingquestions:Whereisthestaticpoint?Howdoesthemalwareobtainthestaticpoint?HowdoesitinferthenallocationLfromthestaticpoint?Ifthemalwareinvokesanexternalfunctiontoreg-isterH,weneedtoidentifythefunction'saddressandname.Inaddition,weneedtoknowtheactualargu-mentsthatareusedtocallthisfunction.Thefunctioncallanditsargumentlistcangivesemanticinformationabouthowthehookandwhatkindofhookisregistered.Forexample,ifweidentifythatamaliciousprogramcallsSetWindowsHookExtoregisterahook,weareabletotellfromtherstargumentwhattypeofhookisregistered.Forthistypeofimplantingmechanism,thehookgraphanswersthefollowingquestions:Whatistheexternalfunction,includingitsentryaddressanditsname? Whatargumentsdoesthemalwareusetoinvokethisfunction?2.2OurApproachSincemostmalwareprogramsareequippedwithvar-iouscodeobfuscationtechniquestofoilstaticanalysis,ourapproachisbasedondynamicanalysis.Thatis,weactuallymonitortheexecutionofthemalwareinaspe-cialenvironment,andusetheobtainedinformationtoderivehowitimplantsthehook,andhowthehookisactivatedbytheoperatingsystem.Notethatourap-proachisdesignedforanalysis,noton-linedetection.Ourapproachisdividedintotwosteps:hookdetectionandhookingmechanismanalysis.HookDetection:Fine-grainedImpactAnalysisOurapproachisbasedonthefollowingintuition.Maliciouscodemakeschanges,includingmemoryandtheothermachinestatechanges,totheexecutionenvironmentasitruns.Wecallthesechangesimpacts.Obviously,ahookHisoneoftheimpactsmadebythemaliciouscode,andthisimpactnallyredirectstheexecutioncon-trolowintothemaliciouscode.Hence,ifweareabletoidentifyalltheimpactsofthemaliciouscode,andob-serveoneoftheimpactsbeingusedtocausetheexecu-tiontoberedirectedintothemaliciouscode,wecande-termineahookinstalledbythemaliciouscode.Further-more,wearealsointerestedinhowanimpactisformu-lated,forthepurposeofunderstandinghookingmecha-nism.Therefore,weidentifyinitialimpacts,thenewlyintroducedimpactsbythemaliciouscode,andthenkeeptrackoftheimpactspropagatingoverthesystem.Basedonthisintuition,weproposene-grainedim-pactanalysis.Wemarkalltheinitialimpactsmadebythemaliciouscodeatbytelevel.Theinitialimpactsin-cludedatawrittendirectlybythemaliciouscode,anddatawrittenbytheexternalcode(throughfunctioncalls)onitsbehalf.Thenwekeeptrackoftheimpactspropa-gatingthroughthewholesystem.Duringtheexecution,ifweobservethattheinstructionpointer(i.e.,EIPinx86CPUs)isloadedwithamarkedimpact,andtheexe-cutionjumpsimmediatelyintothemaliciouscode,thenweidentifyahook.Furthermore,inthiscase,wehavedeterminedthatthejumptargetisthehookentryF,thememorylocationthattheinstructionpointerisloadedfromisthehooksiteL,andthecontentwithinListhehookH.HookingMechanismAnalysis:Semantics-awareIm-pactDependencyAnalysisOnceidentifyingahookH,wewanttounderstandthehookingmechanism.Dur-ingtheimpactpropagation,werecordintoatracethede-tailsabouthowtheimpactsarepropagatedinthesystem.Therefore,fromthetraceentrycorrespondingtothede-tectedhookH,wecanperformbackwarddependencyanalysisonthetrace.TheresultgiveshowthehookHisformulatedandinstalledintothehooksiteL.However,sucharesultisdifculttounderstand,becauseitonlyprovideshardware-levelinformationandsometimescanbeenormous.WecombineOS-levelsemanticsinforma-tionwiththeresult,andperformseveraloptimizationstohideunnecessarydetails.Thenaloutputisasuccinctandintuitivegraphicalrepresentation,assistingmalwareanalyststounderstanditshookingmechanism.Notethatourapproachwouldcatchnormalhook-ingbehaviors.WindowsprovidesanumberofAPIs,suchasCreateThreadandCreateWindow,forap-plicationstoregistertheircallbackfunctions.Windowswillinvokethesecallbacksoncertainevents.Thesefunctioncallsthatregisternormalhookscanbecom-piledintoawhite-list.Thenifoneofthesenormalhooksiscapturedbyourdetectionstep,wecanclas-sifyitasnormal,byextractingitshookingmechanismandcomparingitwiththewhite-list.Inpractice,wendthiswhite-listingapproachveryeffective.Notethatnormalhooksarenotconsideredfalsepositivesinourcase,sinceourgoalistoextractandanalyzeanyhook-ingmechanismwhichmaybeemployedbythesampleofinterest.3SystemDesignandImplementationTodemonstratethefeasibilityofourapproach,wedesignandimplementasystem,HookFinder,toidentifythehookingbehaviorandunderstandthehookingmech-anism.Inthissection,wegiveanoverviewofHook-Finderanddescribeitscomponents.3.1SystemOverviewTheoverviewofHookFinderisillustratedinFig-ure 2 .HookFinderisbasedonawhole-systememula-tor.Itemulatesanx86computerandrunsaWindowsguestsystemontopofit.ThemalwaretobeanalyzedisexecutedintheWindowsguestsystem.Therearetworeasonswhyweemployawhole-systememulator.First,itfacilitatesinstrumentingCPUinstructionsinane-grainedmanner.Inparticular,weareabletoinstru-menteveryCPUinstructionexecutedintheWindowsguestsystem.Second,itprovidesanexcellentprotectionlinebetweentheanalysisenvironmentandthemalware. WindowsWhole-system EmulatorMalwareDetectorExtractorAnalyzerSemanticsHookImpactImpact TraceHook Hook Graph andHooking Mechanism Figure2.SystemOverviewTherefore,itisrelativelymoredifcultformaliciouscodetointerferewithourdetectionandanalysispro-cedureandaffecttheanalysisresults.Intheimplemen-tation,wedevelopHookFinderontopofTEMU[29],whichisthedynamicanalysiscomponentintheBit-Blazeproject[2].Withintheemulator,webuildthreecomponents:im-pactanalysisengine,semanticsextractor,andhookde-tector.Theimpactanalysisengineisacentralcompo-nent,whichperformsne-grainedimpactanalysis.Itmarkstheimpactsmadebythemalware,andkeepstrackofimpactspropagatingoverthewholesystem.Awhole-systememulatoronlyprovidesahardware-levelviewofthesystem,suchasthestatesofCPUregisters,physicalmemory,andI/Odevices.However,malwareanalystsneedtounderstandthemalwareandsystembehaviorsattheoperating-systemlevel.Thesemanticsextractorim-plementsthefunctionalityofextractingOS-levelseman-ticsinformationfromtheemulatedenvironment.Forex-ample,itprovidesprocessandmoduleinformationofthecurrentinstructionexecuted.Itcanalsoprovidein-formationaboutexternalfunctioncalls.Thehookdetec-torbehaveslikeacontroller,cooperatingwiththeimpactanalysisengineandthesemanticsextractortoidentifyhooks.Toanalyzehookingmechanisms,theimpactpropa-gationevents,aswellasnecessaryOS-levelsemanticsinformation,arerecordedintoatrace,calledtheimpacttrace.Thehookanalyzeranalyzestheimpacttraceandgeneratesasuccinctandintuitivegraphicalrepresenta-tion,hookgraph.Thehookgraphconveysessentialin-formationformalwareanalyststoeasilyunderstandthehookingmechanism.3.2ImpactAnalysisEngineTheimpactanalysisengineperformsne-grainedim-pactanalysis,andiscomposedoftwosub-components:impactmarkerandimpacttracker.Theimpactmarkerisresponsibleformarkingtheinitialimpactsmadebythemaliciouscode,andtheimpacttrackerkeepstrackoftheimpactspropagation.ImpactMarkerIntheimpactmarker,weaimtoiden-tifyalltheinitialimpactsthatcanbeusedtoinstallhooks.Thisisimportant,becauseifwefailtomarksomeinitialimpacts,malwarewritersmayexploitthisfacttoevadeourdetection.First,weconsiderthataninstructionfrommaliciouscodedirectlymakesanimpact.Whenanexecutablebi-naryisloadedintothesystem,amodulespaceisallo-catedforit,andthecodeanddatasegmentsfromthebinaryarecopiedintothismodulespaceandinitialized.ThesemanticsextractormentionedinSection 3.3 isabletotellwhichmodulespacebelongstothesampleunderanalysis.Then,foraninstructionlocatedinthatmod-ule,weneedtomarkitsimpactaccordingly.Thatis,wemarkthedestinationoperand,eitheramemorylocationoraCPUregister,ifitisnotmarkedalready.Inaddition,weconsiderthatmaliciouscodemaymakeanimpactbycallinganexternalfunction.Forexample,itmaycallReadFiletoobtaintheaddressofthehookentryFfromacongurationle,andtheninstallitasthehookHintothehooksiteLbycallingmemcpy.Ifwedonotconsiderthissituation,Hwillnotbemarked.Therefore,weneedtomarktheoutputofthatexternalfunctiontoo.Again,wewilldiscussinSection 3.3 howthesemanticsextractordeterminesifaninstructionisexecutedunderthecontextofanexternalfunctioncall.Toidentifytheimpactsmadeinanexternalfunction,wetreatmemorywritesandregisterwritesdifferently.Formemorywrites,wemarkamemorylocationifitiswrittenunderthecontextoftheexternalfunctioncall,anditisnotalocalvariableonthestack.Todeterminealocalvariable,weobtainthestackrangeforthecur- rentthreadfromthesemanticsextractor,andcomparethememorylocationwiththevalueofESPontheen-tryoftheexternalfunctioncall:ifthememorylocationissmallerthanthevalueofESPandwithinthestackrange,thenitisalocalvariable.Forregisterwrites,weonlyneedtoconsiderEAX.Accordingtothefunc-tioncallingconventions(i.e., cdecland stdcall)inWindows,EAXcontainsthereturnvaluewhenappli-cable,whiletheothergeneral-purposeregisters(exceptthestackpointerESP)remainunchanged.NowweneedtodetermineifEAXcontainsthereturnvalueandmarkitaccordingly.WesavethevalueofEAXontheentryofanexternalfunctioncall,andthenontheexitofthefunction,checkifEAXischanged.Ifso,wemarkthisEAX.Furthermore,malwaremaydynamicallygeneratenewcode.Sinceself-generatedcodeisalsopartofim-pactsmadebythemaliciouscode,thememoryregionoccupiedbyitmusthavealreadybeenmarked.Thus,wecandetermineifaninstructionisgeneratedfromtheoriginalmaliciousbinarybysimplycheckingifthememoryregionoccupiedbythatinstructionismarked.Ifso,wealsotreatthatcoderegionasmaliciouscode,andmarktheinputstakenbytheself-generatedcodetoo.ImpactTrackerTheimpacttrackerkeepstrackoftheimpactspropagatingthroughoutthesystem.Ittracksdatadependenciesbetweensourceanddestina-tionoperands.Thatis,ifanybyteofanysourceoperandismarked,thedestinationoperandisalsomarked.Inaddition,foramemorysourceoperand,ifitsaddressbecomesmarked,wealsomarkthedestinationoperand.Thispolicyenablesustotrackhowthemaliciouscodewalksthroughadatastructure,startingfromamarkedpointertothedatastructure.Thesetwopoliciesaresim-ilartothoseinthedynamictaintanalysissystems[7,10,11,22,33].Notethattheimpacttrackerkeepstrackofimpactspropagatingoverthewholesystem,includ-ingthedisk.Itstillkeepstrackoftheimpactsthatareswappedouttodisk,orwrittentotheregistryandlesystem.Therefore,HookFinderisabletodetectthehooksthatareregisteredthroughtheregistryandlesys-tem.Whatmakestheimpacttrackerdifferentfromdy-namictaintanalysisisthewayitchecksimmediateoperands.Thatis,ifaninstructionhasanimmediateoperand,theimpacttrackerchecksifthememoryre-gionoccupiedbythisimmediateismarkedandifso,propagatestheimpactaccordingly.Incontrast,thedy-namictaintanalysissystemstreatimmediateoperandsasclean.Inourscenario,themaliciouscodemayover-writethesystemcodewithmanipulatedimmediatenum-bersintheinstructions.Forexample,inthecodehookcase,themaliciouscodemayinjectintothesystemcodeajumpinstructionwithahard-codedtargetaddress,toredirecttheexecutiontothemaliciouscode.Thisim-mediateoperandisacrucialimpactthatisdeliberatelyinjectedbythemaliciouscodetosetupahook.There-fore,weneedtocheckimmediateoperands.Toenabledependencyanalysis,theimpacttrackerperformsanextraoperationduringtheimpactpropa-gation.Thatis,weassignauniqueidentiertoeachmarkedbyteofthedestinationoperand.WerefertothisidentierasdependencyID.Thenforeachinstruc-tionthatcreatesorpropagatesthemarkeddata,wewritearecordintotheimpacttrace.Therecordcon-tainstherelationshipsbetweenthedependencyIDsofmarkedsourceandthedestinationoperand,associatedwithotherdetailedinformationaboutthatinstruction.3.3SemanticsExtractorThesemanticsextractorbridgesthesemanticgapbe-tweenthehardware-levelviewandthesoftware-levelview.Specically,thepurposesofthesemanticsextrac-torarethree-fold:(1)determinetheprocess,thread,andmoduleinformationforthecurrentinstruction;(2)deter-mineifaninstructionisexecutedinthecontextofanex-ternalfunctioncall,andifso,resolveitsfunctionnameandarguments;and(3)determinethesymbolnameifamemoryreadistoasymbol.Severalprevioussystems[10,1416,33]havedis-cussedextractingOS-levelsemanticsfromavirtualma-chinemonitororawhole-systememulator.Therearemainlytwotypesofapproaches.First,wecandirectlyexaminetheguestsystemstatesfromoutside,withcom-pleteknowledgeofcrucialdatastructures[10,14,15].Second,wecaninsertakernelmoduleintotheguestsystemtocollectthenecessaryinformation[16,33].OurimplementationisbasedonTEMU,whichcombinesthesetwoapproaches.Process,Thread,andModuleInformationTheothertwosystems[16,33]thatarealsobasedonTEMUhavedescribedhowweextractprocess,threadandmod-uleinformation.Tosummarize,thekernelmoduleloadedintheguestsystemregistersseveralcallbackrou-tines.Wheneveraprocessiscreatedordeletedoramod-uleisloadedintoaprocessmemoryspace,thecorre-spondingcallbackroutineisinvoked.Thecallbackrou-tinesgathertheinformationsuchasthevalueofCR3foreachprocessandthememoryregionforeachmod- ule,andthenpassittotheunderneathemulatorviaapredenedI/Oport.Obtainingthreadinformationisfairlystraightforward,asthedatastructureforthecur-rentthreadismappedintoawell-knownvirtualaddressinWindows.Wecansimplyreadthethreadinformation,suchasthethreadIDandstackbaseandsize,directlyfromoutside.ExternalFunctionCallPrevioussystems[10,33]havealsodiscussedhowtodetermineexternalfunctionscalledbythemaliciouscode,bycomparingthestackpointers.Theintuitionisthatthemaliciouscodehastopushtheargumentsandthereturnaddressontothestacktocallanexternalfunction.Thusbycomparingthestackpointerwhentheexecutionentersthemaliciouscode,andtheonewhentheexecutionleaves,wecandeter-mineiftheexecutionjumpingoutofthemaliciouscodeisbecauseofanexternalfunctioncall.Thengiventheentryaddressofanexternalfunction,wewanttoresolveitsfunctionname.WeachievethisbyparsingthePEheaderofamodulewheneveritisloadedintothesystem.EachbinaryinthePEformatcontainsanexporttablethatforeachofitsexportedfunctionsmapsitsnamewithitsoffsetwithinthebinary.Combin-ingtheoffsetwiththebaseaddressthatthemoduleisactuallyloadedin,wecaninfertheactualaddressofanexternalfunction.SymbolNameWhenaninstructionreadsamemorylocation,wewanttodetermineifitisreadingasym-bol,andifso,resolvethesymbolname.ThisisusefulingeneratinganOS-levelhookgraph.Similarlytore-solvingexternalfunctionname,weparsethePEheaderofamodulewheneveritisloadedintothesystem.WeextractsymbolnameswiththeiroffsetsinbothexporttableandimportTable,andinfertheactualaddressofasymbolusingthemodulebaseaddressanditsoffset.3.4HookDetectorThehookdetectorworksbycheckingifthecon-trolowisaffectedbysomemarkedvalue,whichredi-rectstheexecutionintothemaliciouscode.Morepre-cisely,weobservewhethertheinstructionpointerEIPismarked,andtheexecutionjumpsimmediatelyfromthesystemcodeintothemaliciouscoderegion,orthecoderegiongeneratedfromthemaliciouscode.Iftheconditionsaresatised,weidentifyahook:thejumptargetisthehookentryF,thememorylocationthatEIPisloadedfromisL,andthecontentinLisH.Theabovepolicyfunctionsproperlyforidentifyingdatahooks,butisproblematicforcodehooks.Thisisbecauseacodehookisapieceofcodegeneratedbythemaliciouscode,andthusistreatedasmaliciouscodebytheabovepolicy.Thereforewhenthecodehookredi-rectstheexecutiontothemaliciouscode,theabovepol-icywillnotraiseanalarmbecauseitseestheexecu-tionbeingtransferredfrommaliciouscodetomaliciouscode.Tosolvethisproblem,weextendtheabovepol-icysuchthattheexecutiontransitionsfromacodehookregionintomaliciouscodewillraiseanalert.Thenthequestionishowtodistinguishcodehookregionswithotherself-generatedcoderegions.Self-generatedcodeusuallyremainsinthemodulespaceofthemaliciouscode,orstaysinaregionthatisnotoccu-piedbyanymodule(suchasinheap),whereasacodehookregionisapieceofcodethatoverwritesacodere-gioninadifferentmodule.Therefore,duringexecution,ifthecurrentlyexecutedbasicblockismarkedandfromadifferentmodule,andEIPismarkedandjumpsintothemaliciouscode,weidentifyitasacodehook.3.5HookAnalyzerOnceasuspicioushookisidentied,thehookana-lyzerisabletoextractessentialinformationaboutitshookingmechanismbyperformingsemantics-awarede-pendencyanalysisontheimpacttrace.Theprocedureconsistsofthefollowingthreesteps:(1)fromthehookH,performbackwarddependencyanalysisontheim-pacttrace,andgeneratehardware-levelhookgraph;(2)withtheOS-levelsemanticsinformation,transformthehardware-levelhookgraphintoanOS-levelhookgraph;and(3)ifnecessary,simplifythehookgraphbyhidingunnecessarydetailsandmergingsimilarnodes.Wede-tailthesestepsrespectively.Hardware-levelHookGraphAhookgraphrepre-sentsdependenciesamongmalware'sinstructionsthatareusedtoimplantahook.Anodeofahookgraphcor-respondstoaninstructioninvolvinghookingbehavior;anedgeofahookgraphpointsfromaninstructionset-tinganoperandtoaninstructionusingtheoperandassource.Recallthateachrecordintheimpacttracehasde-pendencyinformation.WiththehookHidentiedbyourhookdetector,wecreatetherstnodeinourhookgraph,representingtheinstructionthatactivatesH.Wethenobtainthehook'sdependencyIDIDh,andlocatetherecordthatdenesIDhintheimpacttrace.Finally,wesearchbackwardsintheimpacttracetoadddepen- f8ab1ee6: mov 0xf8ab20a0, %ediM[0xf8ab20a0]=0x804dd6e3 f8ab1f56: mov 0x1(%edi), %eax Impacted Address f8ab1f59: mov 0xf8ab20b4, %ecxM[0xf8ab20b4]=0x80559b80 f8ab1f5f: mov (%ecx), %ecx f8ab1f61: movl $0xf8ab166e, (%ecx,%eax,4)M[0x804e2efc]=0xf8ab166e 804df051: mov (%edi,%eax,4), %ebx 804df069: call *%ebx aries.sys+ee6: mov ZwOpenKey, %edi[aries.sys+10a0]=0x804dd6e3 aries.sys+f56: mov 0x1(%edi), %eax Impacted Address aries.sys+f59: mov KeServiceDescriptorTable, %ecxM[aries.sys+10b4]=0x80559b80 aries.sys+f5f: mov (%ecx), %ecx aries.sys+f61: movl aries.sys+66e, (%ecx,%eax,4)M[ntoskrnl.exe+e2efc]=0xf8ab166e ntoskrnl.exe+8051: mov (%edi,%eax,4), %ebx ntoskrnl.exe+8069: call *%ebx (a)Hardware-levelhookgraph(b)OS-levelhookgraphFigure3.HardwarelevelandOSlevelhookgraphsforahookinSonyRootkit.dencyinformation.Specically,foreachrecordRintheimpacttrace,ifitcreatesanewdependencyIDidthatisusedinthehookgraph,weaddedanodeNrepresentingtheinstructioncorrespondingR,andaddedgesfromNtoothernodesthatusesidassourceoperandsintheircorrespondinginstructions.Weperformthisbackwardsearchrecursivelyuntilwereachthebeginningofthetrace.Besidesthedependencyinformation,eachrecordcontainsdetailedinformationaboutaninstruction,suchasitsaddressandthevaluesofitsoperands.Ifthein-structionisexecutedunderthecontextofanexternalfunction,therecordalsocontainstheentryaddressofthatexternalfunction,andthevalueofESPontheentryofcall.Wealsoputthesedetailsintothecorrespondingnodes.Theresultantgraphisthehardware-levelhookgraph.Figure 3 (a)showsahardware-levelhookgraphbuiltfromahookinSonyRootkit[27],whichemploysthesamehookingmechanismasthesampleshowninFig-ure 1 .Arectanglenodedenotesaninstructionpropagat-ingmalware'simpacts.Adiamondnodedenotesthatitssuccessor'sdestinationaddressisaffectedbythemal-ware'simpacts.Notethattosavespace,weonlydis-playreallyimportantinformationforeachnode,suchastheinstructionaddressandthedisassembledinstruc-tion.Foreachmemoryoperand,weshowitsaddressandvalue.Iftheinstructionisexecutedunderthecontextofanexternalfunctioncall,wealsoshowtheentryofthefunctioncallandtheESPvalueontheentry.OS-levelHookGraphWiththeOS-levelsemanticsinformationprovidedbythesemanticsextractor,wecantransformahardware-levelhookgraphintoanOS-levelhookgraph.Giventheaddressofaninstruc-tion,wecanshowwhichmoduleitbelongstoanditsoffsettothemodulebase.Similarlyformemoryaccess,wecandetermineifitfallsintoanymodulespace.Ifthememoryaccessistoasymbol,wecanevenresolveitssymbolname.Giventheentryad-dressofanexternalfunction,wecanresolveitsfunctionname.Then,theresultinggraphisanOS-levelhookgraph.Figure 3 (b)illustratestheOS-levelhookgraphtransformedfromFigure 3 (a).WecanseethatFig-ure 3 (b)correctlyreectsthehookregistrationproce-dureshowninFigure 1 .Thatis,symbolsZwOpenKeyandKeServiceDescriptorTableareusedtocal-culatethehooksiteL(showninthediamond-shapednode),andanaddress(aries.sys+66e)iswrittenintoL.ThisisthehookH,theaddressofthehookentryF.Inadditiontoresolvingfunctionnames,HookFinderalsoextractsfunctionargumentsfromanimpacttrace.Sincepushingargumentsontothestackisalsopartoftheimpactsmadebyamalwaresample,theinformationabouttheseargumentsisalreadyrecordedintheimpacttrace.Toextractafunction'sarguments,HookFinderlo-catestherstrecordRoftheactivationofthefunction.TherecordsprecedingRcontainfunctionarguments,butmayalsocontainothernon-argumentimpactsmadebythemalware.AstheimpactstracehasinformationaboutthevalueofregisterESPatthebeginningofthefunction'sactivation,weonlyincludetheimpactswithin acertaindistancetothevalueofESP.Inthecurrentim-plementation,wesearchforupto10four-bytewordsfollowingthelocationofESPasarguments.GraphSimplicationTheresultinghookgraphcanbeverycomplexinsomecases.Forbetterreadabilityandclarity,wesimplifyitusingthefollowingcriteria:(1)iftwoadjacentnodesbelongtothesameexternalfunctioncall,wemergethemintoasinglevirtualnode;(2)iftwoadjacentnodesaredirect-copyinstructions,suchasmov,push,andpop,wemergethemintoasin-glenode,becausetheseinstructionspropagatethesamevaluewithoutmodication.Weapplythesetwocrite-riarepeatedlyonourhookgraphuntilnonodescanbemerged.Theresultisoftenagraphmuchclearertobeinterpreted.4EvaluationInthissection,wepresentdetailsontheexperimentalresultsofHookFinder,byevaluatingitwithreal-worldmalwaresamples.Werstgiveasummaryoftheex-perimentalresultsoverthesesamples,andthenpresentdetailsontwoofthem.Inallourexperiments,werunHookFinderonaLinuxmachinewithadual-core3.2GHzPentiumCPUand2GBRAM.OntopofHook-Finder,weinstallWindowsXPProfessionalSP2with512MofallocatedRAMastheguestoperatingsystem.4.1OverviewOursamplesetconsistsofeightmalwaresamples,whichareobtainedfrompublicresources(suchas[20,23])andcollaborativeresearchers.InTable 1 ,wechar-acterizethesesamplesaccordingtowhethertheyarepacked,whethertheyarekerneloruserthreats,andwhichcategoriestheybelongto.WeincludeUayback-doortoverifythecapabilityofHookFinderinidentify-ingnovelhooks 1 .Intheexperiment,HookFinderhassuccessfullyiden-tiedhooksforallthesamples.Wesummarizethere-sultsinTable 2 .InthesecondcolumnofTable 2 ,welisttheelapsedtimeforeachsample.Itbreaksdownintotwoparts:theruntimeforrunningthesampleintheem-ulatedenvironment(shownastherstnumber),andtheruntimeforgeneratinghookgraphs(asthesecondnum-ber).Afterexecutingasample,wewaitfor2-3minutestomakesureithasfullystarted.Inordertotriggerpo-tentialhookbehavior,wethenperformaseriesofsimple 1Sincedeepdoorisnotreleasedbyitsauthor,wecannotincludeitinourexperiment.interactionswiththeemulatedsystem,includinglistingadirectory,andpingingaremotehost,whichmaycostanother2or3minutes.Theruntimeforgeneratinghookgraphsvariesfrom2secondsto33minutes,dependingonthetracesize,thenumberofhooks,andotherfactors.Intotal,HookFinderspendsupto39minutesonasam-pleduringtheevaluation,whichisefcientcomparedtomanualmalwareanalysisthatcanlasthoursordays.Thethirdcolumnliststhesizeoftheimpacttraceforeachsample.Aswecansee,themaximumsizeinthetableis14G,whichisacceptableforacomplexprogramexecutingmillionsofinstructions.Thefourthandfthcolumnshowsthenumberofsus-picioushooksandthetotalnumberofidentiedhooks,foreachsample.Wefoundsomenormalhooksregis-teredbythefollowingfunctions:EVENT SINK AddRef,FltDoCompleteProcessingWhenSafe,StartServiceDis-patcherA,CreateThread,CreateRemoteThread,andPsCreateSystemThread.Notethatourapproachdoesnotdistinguishtheintentofahookingbehavior.Thus,wewillidentifyallhooksintherstplace;thenwechecknormalhooksbycomparingthemwithourwhite-list.Thelastcolumngivesessentialinformationaboutthehookingmechanism.Wefoundthatthreesam-plesinstalledcodehooks.AllthreesamplesderivethehooksitesbycallingGetProcAddress.Vanquishdirectlywritesthehooksintothehooksites,whereasAFXRootkitandHackerDefendercallWriteProcess-MemoryandNtWriteVirtualMemoryrespectivelytoachieveit.Theothersixsamplesinstalleddatahooks,fourofwhichcallexternalfunctionstoinstallthehooks.Inparticular,CFSDcallsFltRegisterFilter,andTrojan/Keylogg-LFandTroj/ThiefcallSetWindow-sHookEx.Wealsoextractedargumentsforthesefunc-tioncalls,andwefoundthatTrojan/Keylogg-LFin-stalledaWH KEYBOARD LLhook,andTrojan/ThiefinstalledaWH CALLWINDPROChook.Theremain-ingtwosamplesdirectlywritehooksintohooksites.ThestaticpointsareKeServiceDescriptorTableandNdisRegisterProtocolforSonyRootkitandUayBack-door,respectively.4.2DetailedAnalysisHerewepresentdetailedresultsfortwomalwaresamples:UayBackdoorandVanquish.UaybackdoorHookFinderidentiedvedatahooksintotalforthissample.Wereviewedthegeneratedhookgraphs,andwefoundthatthreeofthemwereinstalledbyPsCreateSystemThread.Thiskernelfunctioncreates Sample Size Packed? Kernel/User Category Troj/Keylogg-LF 64KB Y User Keylogger Troj/Thief 334KB N User PasswordThief AFXRootkit[1] 24KB Y User Rootkit CFSD[6] 28KB N Kernel Rootkit SonyRootkit[27] 5.6KB N Kernel Rootkit Vanquish[31] 110KB N User Rootkit HackerDefender[12] 96KB N Both Rootkit UayBackdoor[30] 212KB N Kernel Backdoor Table1.MalwareSamplesinOurExperiment Sample Runtime Trace Hooks HookingMechanism Total Mal Troj/Keylogg-LF 6m+9m 3.7G 2 1 Data,Call:SetWindowsHookEx(WH KEYBOARD LL,...) Troj/Thief 4m+3s 143M 1 1 Data,Call:SetWindowsHookEx(WH CALLWINDPROC,...) AFXRootkit 6m+33m 14G 4 3 Code,Call:WriteProcessMemory CFSD 4m+2m 2.8G 5 4 Data,Call:FltRegisterFilter SonyRootkit 4m+2s 25M 4 4 Data,Direct,StaticPoint:KeServiceDescriptorTable Vanquish 6m+12m 4.4G 11 11 Code,Direct,StaticPoint:GetProcAddress HackerDefender 5m+27m 7.4G 4 1 Code,Call:NtWriteVirtualMemory Uaybackdoor 4m+25s 117M 5 2 Data,Direct,StaticPoint:NdisRegisterProtocol Table2.Summarizedexperimentalresultsasystemthreadwiththethreadentryprovidedbythecaller.Thus,thesethreehooksarenormalhooks.Theothertwoaresuspicious,andtheirhookgraphsaresim-ilar.WeshowonegraphinFigure 4 .Wealsoshowtheoriginalhardware-levelgraphinFigure 6 intheAp-pendix.AswecanseeinFigure 4 ,therearetwobranchesatthebottom.TheleftbranchdescribeshowthehooksiteLwasinferred,andtherightbranchpresentshowthehookHwasformulated.Fromthetopoftherightbranch,wecanseethatHoriginatedfromtheoutputofafunctioncallNdisAllocateMemoryWithTag.Thisker-nelfunctionisusedtoallocateamemoryregioninthekernelspace.Accordingtothefunction'ssemantics,thisoutputhastobetheaddressoftheallocatedmemoryre-gion.ThisaddressisnallyimplantedintothehooksiteL.Fromthetopoftheleftbranch,weobservethatLisderivedfromtheoutputofafunctioncallNdisReg-isterProtocol.Thiskernelfunctionregistersanetworkprotocol.Accordingtothefunctionsemantics,webe-lievethisoutputistheprotocolhandleinthesecondar-gument.ThishandlerpointstoaninternaldatastructuremaintainedbytheWindowskernel.Thenwecanseetheinstruction(atuay.sys+1695)readsaeldwiththeoff-set0x10inthisdatastructure.Theobtainedvalue(v1)isthenusedasapointertoreadanothervalue(v2)fromtheoffset0x10inthedatastructurepointedbyv1,inthesubsequentinstruction(atuay.sys+16a0).Then,thein-struction(atuay.sys+1589)addsv2with0x40,andtheresultingvalueiseventuallyusedasthehooksiteL.WebelievethatthissampleactuallywalksintothisinternaldatastructurethatitobtainsfromNdisRegisterProtocol,andlocatesthedesignatedhooksiteL.Interestingly,thedenitionofthedatastructurefortheprotocolhandlecreatedfromNdisRegisterProtocolisnotreleasedinanydocumentationfromMicrosoft,butthismalwaresampleseemstobeabletounderstandthisdatastructure,andknowshowtolocatethedesiredhooksitefromit.Thehookgraphforanothersuspicioushookisverysimilartothisone,exceptthatitaddsv2with0x10.Withtheknowledgeofhowthisinternalstructureisdened,wewouldbeabletotellwhichtwofunctionsthismal-waresampleactuallyhooked.ByanalyzingthissampleusingHookFinder,weareabletounveilanovelmechanismforinterceptingthe NDIS.sys+829a: mov %ecx,0x10(%ebx)Call: NdisRegisterProtocol[0x81dd0f38]=0x81e95ca8 uay.sys+1695: mov 0x10(%eax), %esi uay.sys+16a0: mov 0x10(%esi), %esi NDIS.sys+22faa: call *0x40(%eax) uay.sys+1589: lea 0x40(%esi), %eax Simple Propagation Impacted Address NDIS.sys+115b: mov %eax, (%ecx)Call: NdisAllocateMemoryWithTag[0xf56f2cc4]=0x81e563a8 Simple Propagation uay.sys+fcd: mov %eax, (%esi)[0x81ed3548]=0x81e563a8 Simple Propagation NDIS.sys+827f: mov 0xc(%ebp), %eaxCall: NdisRegisterProtocol Impacted Address NDIS.sys+828c: mov %ebx, (%eax)Call: NdisRegisterProtocol[0xf56f2d68]=0x81dd0f28 uay.sys+168d: mov 0xfffffffc(%ebp), %eax Figure4.HookGraphforUaynetworkstackemployedbymalware.Thatis,malwarecantamperwiththefunctionpointersinsomekerneldatastructuresassociatedwithregisterednetworkpro-tocols.Withthisimportantunderstanding,wecanver-ifyandprotecttheintegrityofthesedatastructures,todefendagainstthiskindofhookingmechanism.VanquishHookFinderidentied11codehooksinto-talforVanquish.Afterreviewingthehookgraphs,wefoundthatVanquishhookedfouruniqueAPIs:Reg-CloseKey,LoadLibraryExW,RegEnumKeyWandRe-gEnumKeyExW.Thus,multiplehooksmaycorrespondtooneAPIhooking,becauseVanquishinstallsonehookperprocessforthatAPI.WeshowahookgraphforhookingRegCloseKeyinFigure 5 .Theotherhookgraphsaresimilar.First,wecanseethebottomnode.Thisistheactualinstruc- kernel32.dll+119ab4: mov %eax, (%esi)Call: GetProcAddress[0x61f81c]=0x77dd6bf0 kernel32.dll+ac81: mov 0xc(%ebp), %eaxCall: GetProcAddress Simple Propagation vanquish.dll+2170: sub 0x8(%ebp), %edx vanquish.dll+1ea7: add $0x1,%eax vanquish.dll+2834: push $0x1ae4c22[0x61f824]=0x1ae4c22 vanquish.dll+216d: mov 0xc(%ebp), %edx advapi32.dll+6bf0: jmp 0x89d0e032 vanquish.dll+2173: sub $0x5, %edx vanquish.dll+2176: mov %edx, 0xfffffff8(%ebp) [0x61f810]=0x89d0e02d vanquish.dll+217f: mov 0xfffffff8(%ebp), %ecx vanquish.dll+2182: and $0xff, %ecx vanquish.dll+218b: mov %cl, 0x1(%edx) [0x1ae928d]=0x2d vanquish.dll+1ea0: mov (%ecx), %dl Simple Propagation Impacted Address vanquish.dll+1ea2: mov %dl, (%eax) [0x77dd6bf1]=0x2d Figure5.HookGraphforVanquishtionVanquishinjectedintothesystemcodetosetupthehook.Itisajmpinstruction,anditsaddressistheentrypointofRegCloseKey.Therestofthegraphshowshowthejumptargetofthisinstructionisformulated.Heretheaddressofthisjumptarget(i.e.,0x77dd6bf1)isthehooksiteL,andthecontentinLisH(i.e.,0x89d0e032).Again,theleftbranchrepresentshowLwasinferred,andtherightbranchindicateshowHwasformulated.TheleftbranchstartswiththeoutputoffunctioncallGetProcAddress.Thisfunctionreturnstheactualfunc-tionaddress,givenafunctionname.Therefore,thesourceoftheleftbranchistheaddressofafunctioncall,andtheactualvalueis0x77dd6bf0,whichistheaddressforRegCloseKey.Aswefollowthelinksdown,wecanseethisaddressisaddedby1andusedasL.Obviously,theoffset1isfortheopcodeofjmp.Now fortherightbranch,wecanseethatitoriginatesfromanimmediate(0x1ae4c22)pushedontothestack.ThisvalueisrstsubtractedbytheaddressforRegCloseKey,andthensubtractedby5.Thenthevalueisandwith0xfftogetthelowestbyte,andthisbyteiswrittentothehooksiteLdirectly.Obviously,thesestepsareusedtocalculatetherelativeaddressforthejmpinstruction.5DiscussionInthissection,wediscusstheresilienceofoursystemtovariousevasiontechniquesthatmalwarewritersmayexploit.ExploitingControlDependencyThebasisofourap-proachistoidentifyallimpactsmadebythemaliciouscode,andkeeptrackoftheimpactpropagationviadatadependency.Itisnaturalformalwarewriterstothinkofexploitingcontroldependency,toevadeourdetection.Forexample,themaliciouscodemayembedacomplexswitchstatementlikebelowtocutthedatadependencybetweenaandb. switch(a){case1:b=1;break;case2:b=2;break;...} Thisevasionisnotviable.Thisisbecauseintheim-pactmarker,wethoroughlymarkalltheinitialimpacts(i.e.,memoryandregisterwrites)madebythemaliciouscode.Thus,theoutputbwillbemarkedanyway.NotExhibitingHookingBehaviorsWhenTestedMalwaremaynotexhibithookingbehaviorduringourdynamicanalysis.Itmaydetectthatitisrunninginouranalysisenvironmentandstayinactiveifindeed.Forexample,itmayrunaredpilltest[25],observeconsid-erableslowdownonperformance,orperformmoreso-phisticatedmethodstodeterminethisfact.Moreover,somemalwareonlyperformsmaliciousbehaviorundercertainconditions,suchasonaspecicdate.Thisisacommonshortcomingofdynamicanalysis.ThecurrentimplementationofHookFindercandealwithsomecom-mondetectionmethods.Wespeciallyinstrumentseveralinstructionslikesidttoreturndeceitfulresultstomal-ware,inordertobypasstheredpilltest.WealsoslowdownthefrequencyofthePITtimerinQEMUtodis-guisetheperformanceslowdownofouremulatedsys-tem.Amorecomprehensivesolutiontothisproblemwouldbetoexploremultipleexecutionpathsthatde-penduponcertainconditions.Someresearchworkhasbeendoneinthisdirection.Moseretal.[17]andBrum-leyetal.[3]alsousedQEMUtobuildmalwareanalysissystems,whichareabletouncoverhiddenbehaviorsofmalwarebyexploringmultipleexecutionpaths.WewillleaveincorporatingthesetechniquesintoHookFinderasfuturework.Evadingthroughreturn-into-libcInthispaper,weconsiderthatmalwareregistersafunctioninitsowncodeasahook.Potentially,malwaremaynotneces-sarilyregisteritsownfunction.Itcanputtheaddressofcertainfunctioninsystemcodeintothehooksite,exploitingthefunctionalityofthatfunctiontoperformsometasks,withoutbeingdetectedbyHookFinder.Thispotentialevasionresemblesreturn-into-libcinbufferoverowattacks[18].WedonotconsiderthiskindofevasioninthecurrentimplementationofHookFinder,asitisgenerallydifculttorealize,intermsofndinggoodcandidatefunctionsandpreparingcompatiblestacklay-out.Wewouldliketoextendourdetectionstrategytocopewiththispotentialevasioninourfuturework.SubvertingorMisleadingHookFinderBuiltontopofanemulator,HookFinderprovidesstrongisolationsuchthatitisunlikelyforthemalwarerunninginsidetointerferewithHookFinderandthehostsystem.How-ever,somestudyshowsthepossibilityofsubvertingtheentireemulatedenvironmentbyexploitingbufferover-owsandintegerbugs[21].Thisproblemcanbead-dressedbyxingthesebugs.HookFindermayalsobemisled.HookFinderidentiesandanalyzeshooksbyexaminingbothhardware-levelandOS-levelinforma-tion.Hardware-levelinformationcanbetrustworthy,becausetheunderlyinghardwarereliesonittoruntheguestsystem.However,OS-levelinformationcanbespurious.Malwarecanndnumerousmethodstohijackthesemanticsextractor.Especially,thekernelmoduleinsertedintotheguestsystemcanbeanobvioustar-get.InthefuturereleaseofHookFinder,wearegoingtodevelopamorerobustandsecuresemanticsextractor.Morespecically,wewillreasonaboutOS-levelseman-ticscompletelyfromoutside,usingreliableandfaithfulstatesoftheemulatedsystem.6RelatedWorkHookDetectionResearchershavedevelopedseveraltools,suchasVICE[4],SystemVirginityVerier[24],andIceSword[13],todetecttheexistenceofhooksinthesystem.Withpriorknowledgehowmaliciouscodeusuallysethooks,thesetoolsexamineknownmemory regionsforsuspiciousentries.Thecommonexaminedplacesaresystemservicedescriptortable(i.e.,SSDT)exportedbytheOSkernel,interruptdescriptortable(i.e.,IDT)thatstoresinterrupthandlers,importaddresstables(i.e.,IAT)andexportaddresstables(i.e.,EAT)ofimportantsystemmodules.Assumingthatimportantsystemmodulesdonotmodifytheircode(withafewex-ceptions),SystemVirginityVerierchecksifcodesec-tionsofimportantsystemDLLsanddriversremainthesameinmemoryasthoseinthecorrespondingbinariesondisk.Innature,thesetoolsfallintomisusedetection,andthuscannotdetecthooksinpreviouslyunknownmemoryregions.Incomparison,ourapproachcapturestheintrinsiccharacteristicsofhookingbehaviors:oneofthemalware'simpactshastobeusedtoredirectthesys-temexecutionintothemaliciouscode.Therefore,itcanidentifyunknownhookingbehaviors.Moreover,italsoprovidesinsightsaboutthehookingmechanisms.DynamicTaintAnalysisThene-grainedimpactanalysisresemblesthedynamictaintanalysistechnique,whichisproposedtosolveandanalyzemanyothersecu-rityrelatedproblems.Manysystems[8,9,19,22,28]de-tectexploitsbytrackingthedatafromuntrustedsourcessuchasthenetworkbeingmisusedtoalterthecontrolow.Othersystems[7,10,33]makeuseofthistech-niquetoanalyzehowsensitiveinformationisprocessedbythesystem.Chowetal.appliesdynamictaintanal-ysistounderstandthelifetimeofsensitiveinformation(suchaspassword)inoperatingsystemsandlargepro-grams[7].Egeleetal.utilizethistechniquetoana-lyzeBHO-basedspywarebehavior[10].Yinetal.alsomakeuseofdynamictaintanalysistodetectandanalyzeprivacy-breachingmalware[33].Moreover,dynamictaintanalysisisusedforotherapplications,suchasau-tomaticallyextractingprotocolmessageformats[5],andpreventingcross-sitescriptingattacks[32].7ConclusionInthispaper,wepresentedanoveldynamicanaly-sisapproach,ne-grainedimpactanalysis,toidentifymalwarehookingbehaviors.Thisapproachcharacter-izesmalware'simpactsonitssystemenvironment,andobservesifoneoftheimpactsisusedtoredirectthesystemexecutionintothemaliciouscode.Sinceitcap-turestheintrinsiccharacteristicsofhookingbehavior,thistechniqueisabletoidentifynovelhooks.Moreover,wedevisedasemantics-awareimpactdependencyanal-ysismethodtoextracttheessentialinformationaboutthehookingmechanisms,whichisrepresentedashookgraphs.Wedevelopedaprototype,HookFinder,andconductedextensiveexperimentsusingrepresentativemalwaresamplesfromvariouscategories.Theexper-imentalresultsdemonstratedthatHookFindercancor-rectlyidentifythehookingbehaviorsforallthesamples,andthegeneratedhookgraphsprovideaccurateinsightsabouttheirhookingmechanisms.8AcknowledgementWewouldliketothankourshepherd,NielsProvos,andtheanonymousreviewersfortheirdetailedsugges-tionsandinsightfulcomments.ThismaterialisbaseduponworkpartiallysupportedbytheNationalScienceFoundationunderGrantsNo.0311808,No.0433540,No.0448452,No.0627511,andCCF-0424422.PartialsupportisalsoprovidedbytheU.S.ArmyResearchOfceundertheCyber-TARe-searchGrantNo.W911NF-06-1-0316,andundergrantDAAD19-02-1-0389throughCyLabatCarnegieMel-lon.Moreover,thisworkisalsosupportedinpartbytheKoreanMinistryofInformationandCommunication(IMC)andtheKoreanInstituteforInformationTechnol-ogyAdvancement(IITA)underprogram[2005-S-606-02,NextGenerationPredictionandResponseTechnol-ogyforComputerandNetworkSecurityIncidents].Anyopinions,ndings,andconclusionsorrecommendationsexpressedinthismaterialarethoseoftheauthor(s)anddonotnecessarilyreecttheviewsoftheNationalSci-enceFoundation.References[1]Afxrootkit. http://www.rootkit.com/project.php?id=23 .[2]BitBlaze:BinaryanalysisforCOTSprotectionandmaliciouscodedefense. http://bitblaze.cs.berkeley.edu/ .[3]D.Brumley,C.Hartwig,Z.Liang,J.Newsome,D.Song,andH.Yin.BotnetAnalysis,chapterAutomaticallyIdentifyingTrigger-basedBehaviorinMalware.2007.[4]J.ButlerandG.Hoglund.VICEcatchthehook-ers!InBlackHatUSA,July2004. http://www.blackhat.com/presentations/bh-usa-04/bh-us-04-butler/bh-us-04-butler.pdf .[5]J.Caballero,H.Yin,Z.Liang,andD.Song.Polyglot:Automaticextractionofprotocolmessageformatusingdynamicbinaryanalysis.InProceedingsofthe14thACMConferencesonComputerandCommunicationSe-curity(CCS'07),October2007.[6]Clandestinelesystemdriver. http://www.rootkit.com/vault/merlvingian/cfsd.zip . [7]J.Chow,B.Pfaff,T.Garnkel,K.Christopher,andM.Rosenblum.Understandingdatalifetimeviawholesystemsimulation.InProceedingsofthe13thUSENIXSecuritySymposium(Security'04),August2004.[8]M.Costa.Vigilante:End-to-endcontainmentofinternetworms.InProceedingsofthe20thACMSymposiumonOperatingSystemsPrinciples(SOSP'05),October2005.[9]J.R.CrandallandF.T.Chong.Minos:Controldataat-tackpreventionorthogonaltomemorymodel.InPro-ceedingsofthe37thInternationalSymposiumonMi-croarchitecture(MICRO'04),December2004.[10]M.Egele,C.Kruegel,E.Kirda,H.Yin,andD.Song.DynamicSpywareAnalysis.InProceedingsofthe2007UsenixAnnualConference(Usenix'07),June2007.[11]A.Ho,M.Fetterman,C.Clark,A.Wateld,andS.Hand.Practicaltaint-basedprotectionusingdemandemulation.InEuroSys2006,April2006.[12]Hackerdefender. http://www.rootkit.com/project.php?id=5 .[13]IceSword. http://www.antirootkit.com/software/IceSword.htm .[14]X.Jiang,X.Wang,andD.Xu.Stealthymalwarede-tectionthroughvmm-basedout-of-the-boxsemanticviewreconstruction.InProceedingsofthe14thACMconferenceonComputerandCommunicationsSecurity(CCS'07),October2007.[15]S.T.Jones,A.C.Arpaci-Dusseau,andR.H.Arpaci-Dusseau.Antfarm:Trackingprocessesinavirtualma-chineenvironment.InUSENIXAnnualTechnicalCon-ference,GeneralTrack,2006.[16]M.G.Kang,P.Poosankam,andH.Yin.Renovo:Ahid-dencodeextractorforpackedexecutables.InProceed-ingsofthe5thACMWorkshoponRecurringMalcode(WORM'07),Oct.2007.[17]A.Moser,C.Kruegel,andE.Kirda.Exploringmulti-pleexecutionpathsformalwareanalysis.InProceed-ingsofthe2007IEEESymposiumonSecurityandPri-vacy(Oakland'07),May2007.[18]Nergal.Theadvancedreturn-into-lib(c)exploits(PaXcasestudy). http://www.phrack.org/archives/58/p58-0x04 .[19]J.NewsomeandD.Song.Dynamictaintanalysisforau-tomaticdetection,analysis,andsignaturegenerationofexploitsoncommoditysoftware.InProceedingsofthe12thAnnualNetworkandDistributedSystemSecuritySymposium(NDSS'05),February2005.[20]Offensivecomputing. http://www.offensivecomputing.net/ .[21]T.Ormandy.AnEmpiricalStudyintotheSecu-rityExposuretoHostofHostileVirtualizedEnvi-ronments. http://taviso.decsystem.org/virtsec.pdf .[22]G.Portokalidis,A.Slowinska,andH.Bos.Argos:anemulatorforngerprintingzero-dayattacks.InEuroSys2006,April2006.[23]rootkit.com. http://www.rootkit.com/ .[24]J.Rutkowska.Systemvirginityverier:Deningtheroadmapformalwaredetectiononwindowssystems.InHackInTheBoxSecurityConference,September2005. http://www.invisiblethings.org/papers/hitb05_virginity_verifier.ppt .[25]J.Rutkowska.RedPill...OrHowToDetectVMMUsing(Almost)OneCPUInstruction. http://invisiblethings.org/papers/redpill.html ,2006.[26]J.Rutkowska.Rootkithuntingvs.compromisedetection.InBlackHatFederal,January2006. http://www.invisiblethings.org/papers/rutkowska_bhfederal2006.ppt .[27]Sony'sDRMRootkit:TheRealStory. http://www.schneier.com/blog/archives/2005/11/sonys_drm_rootk.html .[28]G.E.Suh,J.W.Lee,D.Zhang,andS.Devadas.Secureprogramexecutionviadynamicinformationowtrack-ing.InProceedingsofthe11thInternationalConferenceonArchitecturalSupportforProgrammingLanguagesandOperatingSystems(ASPLOS'04),October2004.[29]TEMU:TheBitBlazedynamicanalysiscomponent. http://bitblaze.cs.berkeley.edu/temu.html .[30]UAYkernel-modebackdoor. http://uty.512j.com/uay.rar .[31]Vanquish. https://www.rootkit.com/vault/xshadow/vanquish-0.2.1.zip .[32]P.Vogt,F.Nentwich,N.Jovanovic,E.Kirda,C.Kruegel,andG.Vigna.Cross-SiteScriptingPreventionwithDy-namicDataTaintingandStaticAnalysis.InProceedingoftheNetworkandDistributedSystemSecuritySympo-sium(NDSS'07),February2007.[33]H.Yin,D.Song,E.Manuel,C.Kruegel,andE.Kirda.Panorama:Capturingsystem-wideinformationowformalwaredetectionandanalysis.InProceedingsofthe14thACMConferencesonComputerandCommunica-tionSecurity(CCS'07),October2007. Appendix:Hardware-levelHookGraphs f83e2faa: call *0x40(%eax)f5b5c583: mov (%edi), %esif5b5c589: lea 0x40(%esi), %eaxf5b5c58c: push %eax[0xf56f2cc4]=0x81ed3548f5b5bf13: mov 0xc(%ebp), %esi806f447f: push %esi Callee: 0x0 ESP: 0xf56f2ca8804df993: pop %esi Callee: 0x0 ESP: 0xf56f2ca88054b051: push %esi Callee: 0xf83c1145 ESP: 0xf56f2c9c[0xf56f2c38]=0x81ed35488054b12f: pop %esi Callee: 0xf83c1145 ESP: 0xf56f2c9cf5b5bfcd: mov %eax, (%esi)f83c115b: mov %eax, (%ecx) Callee: 0xf83c1145 ESP: 0xf56f2c9c[0xf56f2cc4]=0x81e563a8f5b5bfc5: mov 0xc(%ebp), %eaxf5b5bfcd: mov %eax, (%esi)[0x81ed3548]=0x81e563a8f5b5bf44: push %edi f5b5bfcf: pop %edi[0xf56f2ca0]=0xf56f2cc4f83c1158: mov 0x8(%ebp), %ecx Callee: 0xf83c1145 ESP: 0xf56f2c9cf5b5bf4f: push %eaxf83c115b: mov %eax, (%ecx) Callee: 0xf83c1145 ESP: 0xf56f2c9c[0xf56f2c8c]=0x81ed3548Callee: 0xf83c817d ESP: 0xf56f2cdcCallee: 0xf83c817d ESP: 0xf56f2cdc[0xf56f2cec]=0x81ef2218Callee: 0x0 ESP: 0xf56f2cd4[0xf56f2cac]=0x81ef2218f5b5c60d: push %eax[0xf56f2ce4]=0xf56f2d68f83c827f: mov 0xc(%ebp), %eax f83c828c: mov %ebx, (%eax) Callee: 0xf83c817d ESP: 0xf56f2cdcf83c828c: mov %ebx, (%eax) Callee: 0xf83c817d ESP: 0xf56f2cdc[0xf56f2d68]=0x81dd0f28f83c829a: mov %ecx, 0x10(%ebx) [0x81dd0f38]=0x81e95ca8f5b5c68d: mov 0xfffffffc(%ebp), %eaxf5b5c695: mov 0x10(%eax), %esif5b5c511: push %esi[0xf56f2cd8]=0x81e95ca8f5b5c5bc: pop %esif5b5c6a0: mov 0x10(%esi), %esif5b5c511: push %esi[0xf56f2cd8]=0x81e68d50f5b5c69a: push %esi f5b5c513: mov 0x8(%ebp), %edi8056c91a: mov %edi, %edi8056c938: push %edi [0xf56f2ca8]=0x81ef22188056c987: pop %edi Callee: 0x0 ESP: 0xf56f2cd4 Figure6.HardwarelevelhookgraphforUaybackdoor 7c919ab4: mov %eax,(%esi) Callee: 0x7c80ac28 ESP: 0x61f814[0x61f81c]=0x77dd6bf0 7c80ac81: mov 0xc(%ebp),%eax Callee: 0x7c80ac28 ESP: 0x61f814 1ae12e0: mov %eax,0xfffffffc(%ebp) [0x61f820]=0x77dd6bf0 1ae12e3: mov 0xfffffffc(%ebp),%eax 1ae2823: mov %eax,0x1ae92d0[0x1ae92d0]=0x77dd6bf0 1ae2839: mov 0x1ae92d0,%edx 1ae282a: push $0x1ae928c[0x61f82c]=0x1ae928c 1ae2188: mov 0x14(%ebp),%edx 1ae21d7: mov 0x14(%ebp),%ecx 1ae2834: push $0x1ae4c22[0x61f824]=0x1ae4c22 1ae216d: mov 0xc(%ebp),%edx 1ae283f: push %edx [0x61f820]=0x77dd6bf0 1ae2170: sub 0x8(%ebp),%edx 1ae21db: mov 0x8(%ebp),%edx 77dd6bf0: jmp 0x89d0e032 1ae2173: sub $0x5,%edx 1ae2176: mov %edx,0xfffffff8(%ebp) [0x61f810]=0x89d0e02d 1ae217f: mov 0xfffffff8(%ebp),%ecx 1ae2182: and $0xff,%ecx 1ae218b: mov %cl,0x1(%edx) [0x1ae928d]=0x2d 1ae218b: mov %cl,0x1(%edx) 1ae1ea0: mov (%ecx),%dl 1ae21da: push %ecx [0x61f800]=0x1ae928c 1ae1ead: mov 0xc(%ebp),%ecx 1ae21de: push %edx [0x61f7fc]=0x77dd6bf0 1ae1ea4: mov 0x8(%ebp),%eax 1ae1ea7: add $0x1,%eax 1ae1eaa: mov %eax,0x8(%ebp)[0x61f7fc]=0x77dd6bf1 1ae1e9a: mov 0x8(%ebp),%eax 1ae1eb0: add $0x1,%ecx 1ae1eb3: mov %ecx,0xc(%ebp) [0x61f800]=0x1ae928d 1ae1e9d: mov 0xc(%ebp),%ecx 1ae1ea2: mov %dl,(%eax) 1ae1ea2: mov %dl,(%eax) [0x77dd6bf1]=0x2d Figure7.HardwarelevelhookgraphforVanquish