Acti Mapping Resisting NIDS Ev asion ithout Altering r - PDF document

ActiveMapping:ResistingNIDSEvasionWithoutAlteringTrafcUmeshShankarushankar@cs.berkeley.eduUniversityofCaliforniaatBerkeleyVernPaxsonvern@icir.org,ee.lbl.govICSICenterforInternetResearchandLawrenceBerkeleyNationalLaboratoryAbstractAcriticalproblemfacedbyaNetworkIntrusionDe-tectionSystem(NIDS)isthatofambiguity.TheNIDScannotalwaysdeterminewhattrafcreachesagivenhostnorhowthathostwillinterpretthetrafc,andat-tackersmayexploitthisambiguitytoavoiddetectionorcausemisleadingalarms.Wepresentalightweightso-lution,ActiveMapping,whicheliminatesTCP/IP-basedambiguityinaNIDS'analysiswithminimalruntimecost.ActiveMappingefcientlybuildsprolesofthenetworktopologyandtheTCP/IPpoliciesofhostsonthenetwork;aNIDSmaythenusethehostprolestodisambiguatetheinterpretationofthenetworktrafconaper-hostbasis.ActiveMappingavoidsthesemanticandperformanceproblemsoftrafcnormalization,inwhichtrafcstreamsaremodiedtoremoveambigui-ties.WehavedevelopedaprototypeimplementationofActiveMappingandmodiedaNIDStousetheActiveMapping-generatedproledatabaseinourtests.Wefoundwidevariationacrossoperatingsystems'TCP/IPstackpoliciesinreal-worldtests(about6,700hosts),un-derscoringtheneedforthissortofdisambiguation.1IntroductionANetworkIntrusionDetectionSystem(NIDS)pas-sivelymonitorsnetworktrafconalink,lookingforsus-piciousactivityasdenedbyitsprotocolanalyzers(seeFigure1).Inordertocorrectlyanalyzeastreamoftrafc,theNIDSmustrstdeterminewhichpacketsreachthetar-gethostitismonitoringandthen,forthosethatdo,in-ResearchsupportedinpartbyanNDSEGfellowshipandanequipmentdonationfromIntel.SiteFirewall:NIDSAlert::ISP1ISP2Figure1.Adiagramofatypicalsite'snetworkwithaNIDSterpretthemexactlyasthetargethostdoes.TheproblemisthereforeequivalenttoNIDSbeingabletoperformacompleteandprecisesimulationofthenetworkandthehostmachines;inthispaperwerestrictourdiscussiontotheNIDS'abilitytosimulatethenetworkandtransportlayers.Thedominantobstacletoachievingthisgoalisambiguity:thewidevarietyofnetworktopologiesandTCP/IP-stackpoliciesmakesitimpossiblefortheNIDStoknowthecorrectinterpretationoftrafcwithoutad-ditionalcontext.Theresultisadivergencebetweenhowahostinter-pretsasequenceofpacketsandhowtheNIDSbelievesthesequencehasbeeninterpreted.TheNIDScanbetrickedbyanattackerintobelievingthatnoattackoc-curredormaybeconfusedbyamultitudeofpossibleinterpretations,someofwhichareattacksandsomeofwhicharenot.Theevasionsarenotjusttheoretically possible:PtacekandNewsham[PN98]describeanum-berofspecicmethodsforexploitingthissortofam-biguityattheTCP/IPlayer.Furthermore,toolkitshavebeendevelopedwhichautomatetheiruse[So02,Mc98].Thus,itisofconsiderablepracticalconcernthatwendawaytoresolveTCP/IP-basedambiguities.Inthispaperweexploreanovelapproachtoelimi-natingTCP/IPambiguity,calledActiveMapping.Thekeyideaistoacquiresufcientknowledgeaboutthein-tranetbeingmonitoredthat,usingit,theNIDScantellwhichofthosepacketswillarriveattheirpurportedre-cipient,and,ifso,howtheywillbeinterpreted.ActiveMappingdoesthisbybuildingupaproledatabaseofthekeypropertiesofthehostsbeingmonitoredandthetopologythatconnectsthem.Prolesareconstructedbysendingspeciallycraftedpacketstoeachhostandinter-pretingtheresponsestodeterminepathpropertiesandTCP/IPpolicies(seeSection3andtheAppendixforde-tails).UsingActiveMappingprolesmakesaNIDScontext-sensitive.Somemeasureofcontext-sensitivity—awarenessofthehoststhemonitoristryingtoprotect—isnecessary;writingmoredetailedanalyzersisofnousewhenwedon'tknowhowtodisambiguatethetrafcweareanalyzing.NoamountofcarefulcodingintheNIDScanremovecontext-relatedambiguity.Thus,somethinglikeourapproach—gatheringhost-andnetwork-specicinformationandusingitintheNIDS—isinevitableifwearetomakeinroadsagainsttheproblemofambiguityinapassivemonitor.Theinformation-gatheringmaybedoneinotherways,e.g.,passively,buttheprincipleremainsthesame.PreviousworkproposestoeliminateambiguityinNIDSanalysisbyusingatrafcnormalizer[HKP01].Thenormalizer,whichsitsintheforwardingpathbeforetheNIDS,rewritesincomingtrafcintowell-formedstreamsthatpresumablyadmitonlyoneinterpretationonallreasonableTCP/IPimplementations.ThustheNIDS,withasinglepolicyset,canunambiguouslyana-lyzethetrafcforintrusionattemptsonanyofthehostsoftheprotectednetwork.Thoughitsucceedsinreducingambiguity,anormal-izer,likeanyactive(trafc-altering)element,hasanum-berofdrawbacks.Oneisperformance:thenormalizermustbeabletoreconstructeveryTCPstreaminreal-time.Anotherisrobustness:sincethenormalizerisintheforwardingpathofeverypacket,itmustbeex-tremelyreliableeveninthefaceofresourceexhaustion;italsomustberesistanttostateholdingandCPUattacksonitself.Normalizationalsopotentiallychangesthesemanticsofastream.Asdetailedin[HKP01],thesechangescanbreaksomemechanisms,liketracerouteandPathMTUdiscovery.Bycontrast,aNIDSarmedwithaproledatabasecanresolveambiguitiesinatrafcstreamitobserveswithouthavingtointerceptormodifythestream,whichhasmajoroperationalandsemanticadvantages.WestressthatmakingcontextualinformationavailabletotheNIDSistheonlywaytodocorrectdisambiguationofastreamwithoutmodifyingit,soemployingsome-thinglikeActiveMappingisessential.Next,letusconsideranexampleevasion.Figure2detailsanevasionbasedonuncertaintyaboutthenum-berofhopsbetweentheNIDSandatargethost.IfanattackermanipulatestheTTLeldofpacketstocon-fusetheNIDS,itcannotknowwhichofmanypossiblepacketsequenceswasactuallyreceivedandacceptedbythehost.Ontheotherhand,iftheNIDShasinformationaboutthenetworkpathtothehost,thenitcanelimi-natetheambiguity.ItisjustthisinformationthatActiveMappinggathersandsuppliestotheNIDS.Withit,theNIDScanignorepacketsthatwillnotreachthehost,en-ablingcorrectanalysis.Itmaybetemptingtotrytosi-multaneouslyanalyzeallpossibleinterpretationsofthepacketstream;however,thespaceofpossiblenetworktopologiesandTCP/IPpoliciesissolargeastomaketheproblemintractable(seeFigure2andtheAppendixforexamples).WehaveimplementedaprototypeofActiveMappingandrunitonanetworkofabout6,700hosts.OurtestsshowedthattheincreasedprecisioninanalysisdoesnotcomewithanysignicantperformancecostatruntimefortheNIDS.Theincreasedmemorycostwasminimalaswell.WepresentresultstothiseffectinSection5.Theorganizationofthispaperisasfollows.InSec-tion2,wediscussamodelofoperationofthemap-per.InSection3,wediscusstheabilitiesandlimita-tionsofActiveMapping,examiningselectedtestsinde-tail.Themapper'simplementationisdescribedinSec-tion4;theresultsofmappingreal-worldnetworksandNIDSintegrationtestsarepresentedinSection5alongwithadiscussionofperformanceandndings.WegiveanoverviewofrelatedworkinSection6,includingthepotentiallysymbioticrelationshipbetweenActiveMap-pingandnormalization,andconcludewithasummaryofourndingsinSection7.IntheAppendix,wemakeanefforttocoverthecompletespectrumofTCP/IPmap-pings.Design2.1AssumptionsInordertoperformmappingefciently,wemakecer-tainassumptionsaboutthenatureofthenetworkbeing2 Figure2.EvadingaNIDSbymanipulatingtheTTLeld[HKP01].TheNIDSis15hopsfromthesender,butthereceiveris20hopsfromthesender.PacketswithaninitialTTLgreaterthan15butlessthan20willbeseenbytheNIDSbutwillbedroppedbeforereachingthereceiver.Sincetheretransmittedsegmentsareinconsistent,theNIDSdoesnotknowthecorrectinterpretation.monitored:Networktopologyisrelativelystable.Wediscusshowoftenmappingmaybeperformed(basedontheprototypemapper'sperformance)inSections5.3and5.6.Theattackerisoutsidethenetwork;ifthereiscol-lusionwithauserontheinside,thereislittleanysystemcando.Maliciousinsidersworkingaloneareassumedtobeunabletochangeordroppartic-ularpackets.Thislatterassumptionismorelikelytobetrueforswitchednetworks.Thereisarewallthatcanbeusedforsim-plepacket-levelltering,especiallyaddress-basedingressandegresslteringtopreventspoong.Also,weassumetheNIDS'tapisinsidethere-wall.Hosts'TCP/IPstacksbehaveconsistentlywithinordinaryparameters:thatis,iftheyexhibitunusualbehavior,itwillbeatboundarycases.Wedonot,forexample,runeveryTCPmappingtestateverypossiblesequencenumber.2.2DesignGoalsWehavebeenguidedbyanumberofdesignprinci-plesinconstructingoursystem:Comparableruntimeperformance.TheuseofActiveMappingprolesshouldnotappreciablyslowdowntheNIDSnorsignicantlyincreaseitsmemoryrequirements.Mappingshouldbelightweight.Thebandwidthconsumedbymappingpacketsshouldbesmallenoughnottodisruptordinarytrafconthenet-worknordisrupttheoperationofthehostbeingmapped.Theprocessofmappingshouldalsobecompletedinamodestamountofwall-clocktime.Avoidharmingthehosts.Whilenointention-allymaliciouspacketsaresent,bugsinahost'sTCP/IPimplementationmightbetriggeredbythe(unusual)mappingpackets.Eachtestshouldbecheckedagainstknownvulnerabilitiesbeforebeingdeployed.2.3ArchitectureOuroverallstrategyisasfollows:independentoftheNIDS,anActiveMappingmachinescanseachhoston3 Host ProfileTableNIDSprobepacketsInternetTrafficFirewallMapperHost......Dotted lines indicate Figure3.InteractionbetweentheNIDSandtheMapper.TheActiveMappingsystemsendsspe­ciallycraftedpacketstoeachhosttodeterminethehopcount,PathMTU,andTCP/IPstackpolicies.Theresultsarecombinedintoaprole.TheNIDSusestheprolestocorrectlyinterpreteachpacketgoingtooneofthehosts.theinternalnetwork,buildingforeachhostaproleofthenetworkpathandTCP/IPstackproperties.Theseprolesarestoredinadatabase.Atruntime,aNIDScanusethedatabasetoresolvepotentialambiguities(seeFigure3).Forexample,theNIDScanusethehostpro-lestodecidewhethertoacceptordiscardSYNdatainTCPpackets,dependingonthepolicyofthehostforwhichthepacketisdestined.Wenotethatalthoughourmethodallowsmanyambiguitiestoberesolved,some-timestheverypresenceofambiguitymaybeindicativeofnoteworthybehavior(anattackattempt,forexample).ThusaNIDSmaywanttoretaintheabilitytonotifytheadministratorofsuspiciousbehavior,evenwhenthein-terpretationofthatbehaviorisclear.OurmappingtoolisintendedtoberunonamachinewhosenetworkpointistopologicallyequivalenttothelinktheNIDSiswatching(seeFigure3).Typicallythisisbetweentherewallandanyinternalrouters.Itisim-portantthatthemapperbeabletosendpacketsonthislink,sincethemappingresultsshouldreectthepathofactualpacketsfromthepointofviewoftheNIDS.Inor-dertokeeptheNIDSfrominterpretingmappingtrafcasattacksoninternalhosts,theNIDSshouldbecong-uredtoignoretrafctoandfromthemappingmachine.ThemapperbeginsmappingahostbyperformingservicediscoveryandhopcountandPathMTU(PMTU)determination.ItinitiatesTCPconnectionstoasetofTCPservicesandsendsICMPechopacketstodeter-minewhichservicesareavailable.Subsequenttestingisabstractedwithinthemappertousethoseservices.Forexample,sometestsrequiredeterminingwhethersomeparticularTCPdataisacceptedbythereceiver.Todoso,wecanuseanyserviceforwhichwecantellbyitsresponsewhetheritreceivedthespecicdata.Todate,wehavedevelopedconcreteimplementationsofSSHandHTTPbeneaththisabstraction.Othertestsre-quiresimplerabstractinterfaces.PMTUdetermination,forexample,isdonewithICMPechoifavailable,oranyTCPservicefailingthat.ThehopcountandPMTUofthepathtoeachhostaredeterminednext(Section3.2).Oncethesebasicpropertieshavebeenestablished,weconductavarietyofIPandTCPtests(Section3.2),gen-eratingthehostproles.Eachtestisrepeatedmultipletimestoconrmtheresultandaccountfortimeoutsandlateorduplicatedpackets.Toreducecorrelatedfailures,weneverrunmorethanoneinstanceofa(host,test)pairatthesametime.3ActiveMapping:DetailsandLimita-tionsTothoroughlyanalyzetheapplicabilityofActiveMappingtoresolvingpossibleambiguities,wefollowthe“header-walking”techniqueusedin[HKP01].Wefeelthisisagoodplacetostart,sincetheauthorsofthatpaperusedasystematicapproachtoenumeratepossibleambiguitiesinTCP/IPstreams.Weexamineeachambi-guitytoseeifitcanberesolvedviaActiveMapping,or,ifitisverysimple,ifitcanbehandledbystatelessre-wallrules(seeSection3.1).Thus,weprovideareason-ablycompletepictureoftheabilityofActiveMappingtoeliminateTCP/IPambiguity.4 AsummaryoftheActiveMappingapproachtoeverynormalizationin[HKP01]isintheAppendix.Todate,wehaveimplementedaselectedsubsetofthesemap-pings,spanninganumberofdifferenttypesofambigui-ties.WediscussthemindetailinSection3.2.Manyofthemappingsinthefulllistarestraightforwardadditionsgiventhetypeswehavealreadyimplemented.ActiveMappinghassomeadditionalconcernsbe-yondthosefornormalization(mostlyregardingtiming,since,e.g.,normalizedtrafcisgenerallynotsubjecttoIPfragmenttimeouts).WediscusstheseandothercasesthatarenoteasilytackledusingourapproachinSec-tion3.3.AdiscussionofsomepracticalconcernssuchasNATsandDHCPfollowsinSection3.5.3.1FirewallFiltersCertainsimplecasesshouldbehandledbystatelesspacketlteringattherewall:VerifyingIPheaderchecksumsIngressandegressltering,i.e.,acceptingexternalandinternalsourceaddressesonlyonthoserespec-tiveinterfacesBlockingpacketstobroadcastaddressesortore-servedprivateaddressspacesRejectingpacketswhoseIPheaderlengtheldistoosmallortoolargeInshort,therewallshouldrejectpacketsthatcouldnotbepartoflegitimatetrafcorthataresomalformedastobeuselesstoanendhost.3.2SelectedMappingsForacompletelistofnormalizationsandtheActiveMappingapproachtoeach,pleaseseetheAppendix.HopCount.KnowingthenumberofhopstoanendhostallowsustoresisttheevasioninFigure2.Theeas-iestwaytodeterminehopcountistousethetracerouteutility.However,itsstrategyofsendingthreepacketsperhop,increasingthesearchradiusonehopatatime,isquitetime-consuming.Tospeedthisup,weinsteaduseaknownserviceonthesystemandsendapackettoitthatisexpectedtoelicitaresponse.MosthostswillsettheinitialTTLvalueto2Nor2N1for5N8.ThusfromtheresponsewecanmakeagoodrstguessofthenumberofhopsGbysubtractingtheTTLintheresponsepacketfromthenexthighestvalueof2N+1.Thisguesscouldbewrongiftheroutingisasymmetricorthehosthappenstouseadifferentinitialvalue.OursearchstrategyisthereforetotesttherangeG4toG,thenifthatfailstoyieldaresult,performabinarysearch..KnowingthePathMTU—theMaximumTransmissionUnitoverthepathtothehost—isimpor-tantbecausepacketsgreaterthanthissizewiththeDF(Don'tFragment)bitsetarediscarded.Todetermineit,wesendpacketsofvarioussizestoknownservicesonthehostswiththeDFbitsetandwatchforresponses.1Aswithhopcountdetermination,weoptimizeforthecommoncase.SincemanyinternalnetworksrunonEth-ernet,whichhasanMTUsizeof1500bytes,wetestthisrst,thendoabinarysearchon[0,1500],sincethemap-per'sMTUwillgenerallybelimitedbyitslocalEthernetinterfaceto1500.TCPRSTAcceptance.RFC793[Po81c]speciesthataTCPRSTpacketistobeacceptedifandonlyifitiswithinthereceiver'swindow.Anon-compliantTCPcouldcreateproblemsforaNIDS,whichwouldnotknowiftheconnectionhadbeenterminatedornot.Thealgorithmweuseisasfollows.RepeatthefollowingstepswiththeoffsetOsetequalto0(insequence),1(inthewindow),andW+smallcon-stant(outsidethewindow):SendaTCPSYNpacketatsequencenumberS.ReceiveaSYN/ACKpacket,includingawindowsizeW.SendanACKpackettoestablishtheconnection.SendRSTpacketatS+O.SendFINpacketinsequence,i.e.,atS.Receiveoneof:ACKofFINpacket!RSTnotaccepted;RSTornothing2!RSTaccepted.OverlappingandInconsistentIPFragments.RFC791[Po81a]states,“Inthecasethattwoormorefrag-mentscontainthesamedataeitheridenticallyorthroughapartialoverlap,this[suggested]procedurewillusethemorerecentlyarrivedcopyinthedatabufferand1Inprinciple,wecouldalsowatchforICMPNeedsFragmentationresponses,someofwhichindicatethelimitingMTUsizeattheroutergeneratingtheresponse.ButitissimplerforustodirectlyassessPMTUend-to-end.2NotethatalthoughaRSTshouldbegeneratedinresponsetotheFINiftheinitialRSTwasaccepted,somehostshaverewallingsoft-warethatwillnotrespondtopacketsnotsenttoopenconnections(soastoleakaslittleinformationaspossible).Thusweequatereceivingnoresponse(within5seconds)toanacceptanceoftheRSTwesent.5 datagramdelivered.”Itdoesnottalkaboutinconsistentoroverlappingfragments.Furthermore,employingthesuggestedpolicyhassecurityimplications:rewallsandotherpacketltersmustreassemblepacketsbeforemak-inganydecisionsaboutthem,sinceatanypoint,anewfragmentcanoverwritedatafromanoldone.Itisthere-forenosurprisethattherearemanydifferentimplemen-tationsinuse.WeperformfragmentreassemblytestingusingICMPechopackets;inprinciplethetestcouldbeperformedwithTCPpacketsaswell.Wesendasequenceoffrag-ments,eachonecontainingamultiple-of-eightbytepay-load(sincetheIPoffseteldisinthoseunits).Thedia-grambelowshowseachofthesixfragments,numberedbytheorderinwhichtheyweresent;theirpayloadscon-sistedofthatnumberreplicatedamultiple-of-eightnum-beroftimes.Forexample,thethirdfragmentwassentatanIPoffsetof6(correspondingtothe48thoctetintheoverallpacket)andhada24-bytepayloadofthere-peatingcharacter`3'.EachfragmentbutthelasthadtheMF(MoreFragments)bitset.Thefragments'offsetsandthehost'spossibleinterpretationsaregivenbelow,alongwiththenamesofthepoliciestowhichtheycor-respond:311012345678901�--higherIPOffsetDataSent11122333(Fragments1,2,3)4444555666(Fragments4,5,6)DataReceived111442333666BSDpolicy144422555666BSD-rightpolicy111442555666Linuxpolicy111422333666Firstpolicy144442555666Last/RFC791policyThefollowingisadescriptionofthepolicieswehaveobservedsofar:BSD.Thispolicyleft-trimsanincomingfragmenttoexistingfragmentswithalowerorequaloffset,discardingitifitisoverlappedentirelybyexist-ingfragments.Allremainingoctetsareaccepted;overlappingfragmentswithagreateroffsetaredis-cardedortrimmedaccordingly.Thispolicyisdoc-umentedmorethoroughlyinWrightandStevens[WS95],pp.293-296.3OneunfortunateproblemisthatfortheICMPchecksumtobecorrect,wemustcalculateitassumingaparticularreassemblypolicy!Thuswemustsendallthefragments(withadifferentchecksum)onceforeachpolicy.BSD-right.ThispolicyissimilartoBSD,exceptfragmentsareright-trimmed(newfragmentstakeprecedenceoverthosewithalowerorequaloffset).Linux.TheLinuxpolicyisalmostthesameastheBSDpolicy,exceptthatincomingfragmentsaretrimmedonlytoexistingfragmentswithastrictlyloweroff-set;thatis,existingfragmentswiththesameoffsetwillbeoverwritten,atleastinpart.First.Alwaysaccepttherstvaluereceivedforeachoffsetinthepacket.Last/RFC791.Alwaystakethelastvaluereceivedforeachoffsetinthepacket.4Other.Threeotherpossiblepoliciesaretestedfor,butnonehaveyetbeenobservedinpractice.OverlappingandInconsistentTCPsegments.ThisproblemissimilartothatofIPfragmentreassembly.RFC793[Po81c]statesthatanimplementationshouldtrimsegments“tocontainonlynewdata”,whichimpliesa“First”policy.Theprinciplefortestingislikewisesimilartoevaluatingfragmentreassemblyambiguities,andwecoulddothemappingusinganyTCPserviceforwhichwecanconductanapplication-leveldialog.IdeallywewouldusetheTCPEchoservice,butthisisrarelysupported;weusedSSHandHTTPintesting.WediscussithereasimplementedforSSH.Uponconnecting,anSSHserversendsaversionstringoftheform&#xmajo;&#xr000;&#xmino;&#xr000;omm;nts;SSH-nrnn[Yl02].Theclientisexpectedtosendasimilarstringofitsown;ifthestringiswell-formed,theserverrespondswithad-ditionalparametersfornegotiation.Ifnotwell-formed,theserverclosestheconnection,optionallysendinganerrormessage.Ourtestmakesthewell-formednessoftheversionstringdependentonthereassemblypolicy.Bysendingdifferentcombinationsofsegments,wecandeducethepolicyfromthevariedresponses.Foreachofthefollow-ingtwotests,somehostswillreassemblethefollowinglegalversionstringSSH-2.0-blahnrnn5andsomewillreassembleanillegalversionstring,uponwhichtheywillendtheconnection.Thuswecantellby4Intesting,someCiscorouters(whichemployedtheLastpolicy)sentbackICMPechoresponseswithseveraladditionaltrailingNULcharacters.5Actually,theversionstringthemappersendstotheserveristhesameastheonetheserverinitiallysendstothemapper.Thisprevents“protocolmismatch”errors.6 thesuccessorfailureoftheconnectionwhetheralegalstringwasreassembledornot.Thersttestsendsthefollowingthreesegments.Onlypoliciesthatdonotleft-trim(orindiscriminatelytrim)toearlierdatawillfail.012346789012TCPSeq.OffsetSH-(Firstsegment)X2.0-blah\r\n(Secondsegment)S(Thirdsegment)Notethattheinitial`S'issentlasttopreventreassem-blyuntilitissent.Thesecondsendsfoursegments;thistesttriestofur-therdiscriminateamongpoliciesthatsucceededonthersttest.Policieswhichneverdiscardalready-receiveddatawillfailthistest.012346789012TCPSeq.OffsetSH(Firstsegment)+(Secondsegment)X-2.0-blah\r\n(Thirdsegment)S(Fourthsegment)Heretherearethreeobservedpolicies,characterizedbythesuccess(connection)ofthe(rst,second)test.TheyarethesameasforIPfragments:BSD(yes,yes),First(yes,no),andLast(no,no).Thefourthpossibil-ity(no,yes),hasnotyetbeendetectedinourtesting.ObservedresultsbyoperatingsystemmaybefoundinSection5.3.3DifcultorIntractableCasesThesuccessofActiveMappingdependsuponhosts'behavinginaconsistentandpredictableway.Thisisgenerallyagoodassumption,sincemostprotocolstacksaredeterministicandobeyrelativelysimplerulesforambiguityresolution.Thereare,however,atleastthreesourcesofnondeterminismthatcanmakeitdif-culttoperformprecisesimulationintheNIDS,evenwithActiveMapping:user-controlledparametersintheTCPstack,newsemantics,andnon-deterministicpacketdrops.pplication-levelParameters.Userscanchangecer-tainparametersthataffecttheTCP/IPstack.Oneex-ample,asnotedin[HKP01],istheuseoftheTCP“ur-gent”pointer,whichmarkssomepartofthesequencespaceascontainingimportantdatathatshouldbepro-cessedwithoutdelay.Dependingontheimplementationanduser-setparameters,thisdatamaybedeliveredviaasignalorinlinetotheuserprocess.ThereisnowayfortheNIDStodetermineunambiguouslythereconstructedbytestreamasseenbytheapplicationwithouthelpfromthehostorhardcodingoftheapplication'sinterpretationofurgentdata.Newsemantics.ANIDSmustunderstandthein-tendedsemanticsofastreamifitistointerpretitcor-rectly.UnknownTCPoptions,forexample,canbeig-noredifthetargethostdoesnotindicatesupportforthem.ThebesttheNIDScandoingeneralistobeup-datedregularlywithsupportfornewoptionsashostsontheinternalnetworksupportthem.Ifpartialnormaliza-tion(seeSection6.1)isavailable,unsupportedoptionscanbelteredout.NondeterministicPacketDrops.Perhapsthemostcommonreasonforpacketdropsisafullincomingpacketbufferataninternalrouterorendhost.Thusifroutersinternaltoasitebecomesaturated,orifapar-ticularhostisfacingveryhightrafcvolumes,pack-etsmaybedropped.Ifanattackercancausepacketstobedroppedinaveryprecisewayduringmapping,thatcouldaffectmappingresults;lesspreciseinterferenceislikelytobecaughtasaninconsistencybetweenmultipleruns.DroppingmayalsobedonebyrouterstomeetQual-ityofServiceguarantees.MechanismslikeDiffserv[B+99]thatimplementQoSbutwhoseexactworkingsaresite-specicarehardtopredict,sinceexternalandinternaltrafcmaybemingled,eachcontributingtopacketdropsfortheother.AmitigatingfactoristhatsuchQoSpoliciestendtobeimplementedeitherattheboundaryrouters(whichlterbeforetheNIDS)oratanexternalaggregationpoint.TheNIDSmustalsoknowwhenahostwilltimeoutanIPfragmentorTCPsegment.Withoutthisknowl-edge,anattackercanlaterretransmitthefragmentorsegmentwithdifferentdata:theNIDScannotknowwhichwasaccepted,evenwithknowledgeaboutwhichwouldbeacceptediftherstdidnottimeout.ThoughActiveMappingcantrytodeducethetimeoutvalue,theneedforprecisioninthetimeoutdeterminationmakesthisdifcult.3.4DealingwithTimeoutsandPacketDropsTheNIDScannotbenotiedofeveryrouterorendhostpacketdrop.Thehostbeingmonitoreddoes,how-ever,givesomeimplicitdropinformation,intheformofacknowledgmentsandresponsestorequestsorlackthereof.Whencombinedwithtemporalcausality,thiscanallowpartialreconstructionofthehost'sstate.IfweseeanacknowledgmentofaTCPsegmentoraresponsetoaUDPorICMPrequest,wecaninferthat7 therequestmusthavebeenacceptedusingonlypacketsthatprecededtheresponse.Furthermore,ifnoresponseissentwhenoneisexpected,wecaninferthatpacketshavebeendropped.IftheNIDScansendpacketsinrealtime,itcansenda“keep-alive”TCPpacket,onethatisoutofsequence.ThisshouldelicitanACKthatshowsthecurrentsequencenumber.TheNIDScanalsowatchforICMPmessagesin-dicatingtimeouts(“FragmentReassemblyTimeEx-ceeded,”per[Po81b]).Notallhostssendthesenoti-cations,andtheymightleakinformationtoanattacker.AcompromisemightbetocongurehoststogenerateinformativeICMPmessagesthatarelteredbythere-wall(butarestillseenbytheNIDS).3.5PracticalConsiderationsThereareadditionalconcernsthatariseinmappingrealnetworks.Ourinitialprototypedoesnothandleallthesecasesandtherearelikelytobeothers.Wediscusspossibleapproachestocommonreal-worldscenariosbe-low.WepointoutthatActiveMappingdoesnotrequireacompleteproleforeachhosttobeuseful:atbest,manyambiguitiesareeliminated;atworst,thedefaultbehavioristhatoftheoriginalNIDS.ThusActiveMap-pingmaybeincrementallydeployedevenwhilesomepracticalhurdlesarebeingsurmounted.NATSofarourdiscussionofmappinghasassumedthateachIPaddresscorrespondedtoexactlyonema-chine(andasinglesetofpolicies).IfaNAT[EF94]isrunninginsidethemonitoredsite(sothattheNIDSdoesnotseetheprivateaddresses),however,weneedaddi-tionalstrategies.TohandleserversbehindaNAT,wecouldmapeachportasthoughitbelongedtoaseparatemachine,checkingforallrelevantpoliciesoneachport.ItishardertodealwithclientsbehindaNAT,thoughthisisonlyrelevantinthecaseofoutsideserversattackinginternalclientsinaclientOS-specicway.ItcanbedifculttodetectwhenaNATisbeingused,thoughrecentworkbyBellovin[Be02]suggeststhatitispossibleinsomecases.IfnotallNATIPsarenotknowntosystemadministrators,themappercouldmapmulti-pleportsindependentlyorsamplethemfordifferences,whichwouldindicateaNAT'spresence.DHCPTheDynamicHostCongurationProtocol(DHCP)[Dr97]dynamicallyassignsIPaddressestoclients.ADHCPserverleasesoutaddresseswhenclientsrequestthem;leasesexpireperiodically.Deal-ingwithDHCPrequiressomeintegration:themappercouldbetriggereduponseeingDHCPrequests(ifthebroadcastdoesnotmakeittothemappingmachine,theDHCPservercanbesetuptonotifyit).TheproledatabasecouldincludeMACaddresses,sothemapperwouldknowwhenitalreadyhasaproleforagivenma-chine(perhapsgatheredpreviouslyunderadifferentIPaddress).IfintegrationwithaDHCPserverisnotpossi-ble,determiningMACaddressesmightbenontrivial;itisanareaforfuturework.TCPWrappers(Host-BasedAccessControl)SomehostsuseTCPWrapperstorestrictaccesstoservicestoasetofhostsdeterminedbyanAccessControlList.IftheActiveMappingmachineisnotgrantedaccess,sometestsrequiringmeaningfulinteractionwithaparticularTCPservicewillfail.Asimplesolutionistoallowadesignatedmappingmachineaccesstorelevantservices.AttacksontheActiveMapperAnaturalconcerniswhetherornotanattackercouldsubvertthemappingprocess,causingfalseresults,byattackingthemappingmachineortryingtochangemappingtrafc.Prevent-ingoutsiderattacksonthemapperdirectlyisstraight-forward:simplyhavetherewallrejectalltrafcdes-tinedforthemapper.Thereisnolegitimateneedfordirectaccesstoamappingmachinefromtheoutside.Agreaterconcernwouldbedirectattacksfrominsidema-chinesthathavebeencompromised;thethreatcouldbemitigatedbyonlyallowingaccesstowell-knownportsfromarestrictedsetofadministrativemachinesatthemapper'slocalrouter.Ofcourse,onceanattackerhascompromisedaninternalmachinemanyothertypesofattacksarepossible.4PrototypeImplementationWeimplementedActiveMappinginabout2,000linesofPerlandhaveportedittotheLinuxandFreeBSDoperatingsystems.ItrequiresaTCP/IPrewallcapability,thelibpcappacketcapturelibrary[MLJ94],andrawsocketsupport.Usingthesefeaturesgenerallyrequiressuperuseraccess.ICMPandTCPpacketsaresentdirectlyusingrawsockets.APcaplterissetuptocaptureresponses.Ouruser-levelTCPimplementationfollowsastrategysimilartothatofTbit[PF01],aTCPbehavior-inferencetool.LikeTbit,werewalloffhigh-numberedTCPportsforuseasephemeralsourceports(topreventthekernelfromrespondingtoincomingtrafctothoseportsbysendingRSTs).UnlikeTbit,whichdynamicallyin-stallsandremovesrewalllters,werequiretheusertoallocateandrewalloffarangeofportsinadvance;thisreducestheamountofsystem-dependentcodeinthemapperattheexpenseoftransparency.OurTCPimple-mentationisrudimentary;currentlyweperformneither8 reassemblynorimplementcongestioncontrol,forexam-ple.Nonetheless,ithasprovedadequatethusfarfortheshort-livedconnectionsneededformapping,especiallysinceserverstendtosendbackwell-formedrepliestoouroftenmalformedqueries.Themapperconductstestsinparallelwithrespecttomachinesbeingmappedandwithrespecttoeachindi-vidualtest.ThedegreeofparallelismisdeterminedbythenumberofavailableTCPsourceports,thesizeofthepacketbuffers,and(dueinparticulartoourunoptimizedimplementation)theCPUspeed.Eachtestisrepeatedacongurablenumberoftimes(three,intesting)andalltheresultsarerecorded.Thisisimportanttoaccountfordroppedpacketsandtimeouts.Currently,wehaveimplementednetworktopologyandservicediscoveryaswellasthespecictestsde-scribedinSection3.2.WemodiedtheBroNIDS[Pa98]touseActiveMappingprolestoproperlyinterprettrafc.(WenotethatthisintegrationmaybedonewithanyNIDSwhichdoesTCP/IPstreamreconstruction,sinceitwillincludeallthenecessarydecisionpoints.)Theintegrationwasstraightforward;afewhundredlinesofC++codewereneeded.Theperformanceimpactofthemodicationsisdiscussedinthefollowingsection.5ExperimentsandResults5.1ObservedActiveMappingProlesWerantheprototypeActiveMapperattheLawrenceBerkeleyNationalLaboratory.Theexactnumberofac-tivehostsduringourscanisnotknown,butwasesti-matedbasedonotherscanstobearound6,700.Weob-tainednontrivial,consistentdata(identicalresultsoverthreetrialsforsomethingotherthanthehostname)forjustover4,800hosts.ManyoftheIPsforwhichwedidnotobtainresultsareinDHCPblocks(hostsmaynotalwaysbepresent);inaddition,manyusersemployhost-basedrewallswhichwouldpreventourscanning.Wearecurrentlyworkingongettingmoreprecisedatahere(wenotethatrewallsarelikelytopreventtheat-tackstheNIDSislookingforinanycase!).Itissigni-cantthatweobtainedresultsforvirtuallyeverymachineforwhichOSdatawereknown;presumablymostothermachinesaremoretransientorarerewalledenoughtostopOSdetection.WepresentActiveMappingprolesbyoperatingsysteminFigure4.Sometestsdidnotyieldresultsduetoservices'beingprotectedwithTCPWrap-pers.Weexpectthislimitationcanbeovercomeinprac-ticebyaddingthemappingmachinetothehosts'ACLsasneeded.Theamountofobserveddiversityinpolicyisremark-able,giventhatweonlyranvetests.Whilehostswithagivenoperatingsystemversionexhibitedthesamepoli-cies,itisinterestingnotehowpolicieschangedfordif-ferentversionsofthesameoperatingsystem.Linuxinparticularseemstohaveundergoneanumberofpol-icychanges,evenduringminorrevisionsofthekernel.Wealsonotethatindividualuserscanalterpoliciesbyinstalling“hardening”orotherpatches.Itispreciselythisdiversity(borneoutbyourexperiments)thatunder-scorestheneedtodisambiguatetrafcdestinedforeachhostbaseditsparticularobservedpolicy.6For173hosts,wewereunabletogetresultsthatwereconsistent(denedasgettingidenticalresultsforthreetrials).Thisislesssurprising,perhaps,inlightofthefactthatallbut29ofthemwerefoundtobeprinters,routers,orscanners;manyoftheremaining29hadunknownoperatingsystems.Furthermore,allbut36ofthe173hostsgaveconsistentresultsforthetrialswhichcom-pleted,buthadoneormoretrialswhichdidnotcom-plete.Thiscouldbeduetocongestion.Inall,only10machineswhichwerenotknowntobespecial-purposedevicesyieldedresultswithconictinganswers.5.2StabilityofResultsWeperformedanothermappingofthehostsatLBNLabout5monthsaftertheoriginalstudywhoseresultsarepresentedabove.Thegoalwastoseewhatthe“churnrate”waslike,i.e.,howmanyIPaddresseshadcomeandgoneandwhetherornotproleshadstayedconstant.Ideally,onemightperformsuchananalysisasmallertimescales.First,letusexaminethedifferencesbetweenthesetsofIPaddressesinthetworuns.Intheoriginalmapping,4,882hostsprovidednontrivial,consistentresults;inthesecondmapping4,733hostsdidso.1,122IPswereintherstset,butnotthesecond;ofthese880wereinDHCPblocks.973IPswereinthesecondsetbutnottherst;669wereinDHCPblocks.ThelargefractioninDHCPblocksisimportantbecausethesetofmachinesintheseblocksmayhave“IPchurn”without“machinechurn”,whichismoresignicantforus,sinceitseemsfeasibletoinformtheNIDSaboutDHCPleaseupdates.Weestimatethe“machinechurn”inDHCPblocksbycomparingthedistributionsofprolesamongtheDHCPmachines.AswecanseefromFigure5,thedistributionisrelativelystable;thusweexpectthatmachinechurnshouldbemanageable.6Wenotethatarst-orderapproximationmightbeobtainedbyus-ingknownOSversioninformationwithalookuptable;itmayevenmakesensetorunActiveMappingandtheninfertheOSfromitsre-sults.Weplantoinvestigatethisrelationshipinthefuture.9 OSIPFragTCPSegRSTinwndRSToutsidewndAIX2BSDBSDYesNoAIX4.38.9.3BSDBSDYesNoCiscoIOSLastBSDYesNoFreeBSDBSDBSDYesNoHPJetDirect(printer)BSD-rightBSDYesNoHP-UXB.10.20BSDBSDYesNoHP-UX11.00FirstBSDYesYesIRIX4.0.5FBSDNoresultYesNoIRIX6.2BSDNoresultYesNoIRIX6.3BSDBSDYesNoIRIX646.4BSDBSDYesNoLinux2.2.10linuxNoresultNoNoLinux2.2.14-5.0linuxBSDYesNoLinux2.2.16-3linuxBSDNoNoLinux2.2.19-6.2.10smplinuxBSDNoNoLinux2.4.7-10linuxBSDYesNoLinux2.4.9-31SGIXFS1.0.2smplinuxBSDYesNoLinux2.4(RedHat7.1-7.3)linuxBSDYesNoMacOS(versionunknown)FirstBSDYesYesnetappunknownNoresultNoresultNoNonetappunknownNoresultNoresultYesNoNCDThinClients(noservicesexported)BSDNoresultNoresultNoresultOpenBSD(versionunknown)linuxBSDYesNoOpenBSD(versionunknown)linuxBSDNoNoOpenVMS7.1BSDBSDYesNoOS/2(versionunknown)BSDNoresultYesYesOS/2(versionunknown)NoresultNoresultNoNoOSF1V3.0BSDBSDYesNoOSF1V3.2BSDNoresultYesNoOSF1V4.0,5.0,5.1BSDBSDYesNoSunOS4.1.4BSDBSDYesNoSunOS5.5.1,5.6,5.7,5.8FirstLastYesNoTektronixPhaserPrinter(unknownmodel)LastNoresultNoNoTektronixPhaserPrinter(unknownmodel)FirstBSDYesYesTru64UnixV5.0A,V5.1BSDBSDYesNoVax/VMSBSDBSDYesNoWindows(95/98/NT4/W2K/XP)FirstBSDYesNoFigure4.SelectedObservedActiveMappingProles.ActiveMappingprolesobserved,byoperat­ingsystemofthehost.Testsreportedcorrespondtothosedescribedinsection3.2.Operatingsystemdatawerenotavailableforallmappinghosts,sotheabovetableisnotcompletewithrespecttoourtestset;insomecases,versionnumberswerenotknown.SomeentrieswithidenticalresultsacrossmanyversionsofanOShavebeensummarizedinoneline;someverysimilarOSversionswithidenticalresultshavebeenomittedforbrevity.Avalueof“NoResult”isduemostlytotheuseofTCPWrappers;insomecasesthemappedhostdidnotsupporttheservicerequiredtoperformmapping.SinceeverymachineacceptedaRSTinsequence,resultsforthattestarenotgiven.10 0 200 400 600 800 1000 1200 1400 1600Number of machinesProfileOriginal MappingSecond MappingFigure5.ThedistributionofAMprolesofDHCPclientsfortworunsseparatedbyvemonths.Thelargespikeinthegraphcorre­spondstoWindowsmachinesnotrunninganypublicservices.Amongthe1,618machineswithxedIPs,only35showedanydifferenceinAMprolebetweenthetworuns.AlthoughwedidnothavecompleteOSinforma-tionforalltheseIPsforeachrun,ourmanualcompar-isonwherepossibleshowedthatthesewerealmostallduetoOSupgrades.Thus,itdoesnotappearthatthereisaconsiderableamountofturnoverinthisgroup.5.3MappingTimeThetimesmeasuredaredependentonthepoliciesfound:sincemanytests'resultsaredeterminedbythepresenceorabsenceofaresponsefromthehostwithinacertaintime,somepoliciesgeneratemoretimeoutsthanothers.Mosttimeoutsareontheorderof5–10seconds;wefoundthisintervaltobesufcienttoaccountforde-laysatthehostsandinthenetwork.Mappingasinglehostrequiresapproximately37sec-onds.Thisminimumisduetothefacteachofthemap-pingtestsisrepeatedthreetimes,andasingletestre-quirestworoundsofcommunication.Wall-clocktimerisessublinearlythrough64hosts,duetoincreasedparallelism.Formorethan64hosts,though,timesarelikelytoscalelinearlysincethemapperimplementsrate-limitingtoavoidpacket-bufferoverows(aproblemwewereabletoalleviateinpartbyusinglarger-than-normalpacketcapturebuffers).In-deed,mapping101hoststook532seconds,or5.3sec-ondsperhost;for64hosts,thetimewas5.7secondsperhostandfor16hosts,ittook10.1secondsperhost.Ourprototypeimplementation'sinefciencyresultedinusertimeincreasesattherateofsomewhatlessthantwosecondsperhostbeingmapped.Asaresult,paral-lelismwaslimited,allowingsteady-stateratesofabout5secondsperactivehostonthefull-sitemappingwiththousandsofhosts.Weexpectthatthisgurecouldbeimprovedconsiderablywithabetterimplementation.5.4MappingTrafcWemeasuredbidirectionalnetworktrafcgeneratedduringmapping.Duringascanofasubnetwith101livehosts,werecordedstatistics(takenoverthreetrials)re-latingtothenumberofbytesandpacketsgeneratedbyscanning,bothtoandfromthemapper.TheresultsareinFigure6.ICMPpacketswereduetoICMPservicedis-covery,PMTUandhopcountdetermination,andsomeIPmappings.TCPpacketswereduetoTCPservicedis-covery,PMTUandhopcountdetermination(ifICMPwasnotsupported),andTCPmappings.TotalPerhostTotalbytes1.9MB49KB19KBTotalpackets32,893345326ICMPpackets21,7632215TCPpackets10,5887105Packets/sec.3.30.0Bytes/sec.1915Figure6.Trafcgeneratedbymapping101hostsonasinglesubnet.Threetrialswereconducted.5.5NIDSIntegrationTestsWemodiedtheBroNIDSbyaddingsupportfordisambiguationbasedonActiveMappingproles;westressthatthechoiceofNIDSwasforconveniencesinceourtechniqueswouldapplyequallytoanyTCP/IP-analyzingNIDS.Ourgoalsintestingweretwofold:rst,toensurethatusingActiveMappingwouldindeedre-sultincorrectinterpretationofnetworktrafc;second,tocheckthatusingActiveMappingwouldnotincuranysignicantruntimecost.Accordingly,werantwosetoftests:rst,asynthetictestwithambiguoustrafc;sec-ond,acomparisonoftheoriginalandActiveMapping-modiedNIDSonreal-worldtraces.(WeexpectthatresultswouldbesubstantiallythesamewithanyotherNIDSintegration.)11 5.5.1SyntheticTestsInordertotestthecorrectnessofthemodiedNIDS(itsabilitytodisambiguatetrafccorrectly,wegener-atedHTTPattacktrafcto8hostswithevasionmea-suresaddedusingfragroute[So02]tomodiedtraf-cto2hosts.Fragrouteautomaticallytransformedtherequeststreamtoincludeoverlappingandinconsis-tentIPfragmentsandTCPsegments.Theinconsistencyfavoredoneoftwopolicies(inourparlance,a“rst”policyanda“BSD”policy);thedatanotexpectedtobeacceptedwerechosenrandomly.Forthetwomachinesreceivingmodiedtrafc,weusedActiveMappingpro-leswhichwouldallowthetrafctobeproperlyinter-preted.WefoundthattheunmodiedNIDSbelievedtheHTTPrequesttobe:GET/msadcTpo6EGKEY./../..%bTMmzyQaL/system32/fipGNdDg++dir+c:nratherthan:GET/msadc/../../../../../../winnt/system32/cmd.exe?/c+dir+c:nwhichwastheactualrequestURL.Itisclearthattheun-modiedNIDS,whichhadnowaytoproperlyresolvetheambiguousoverlaps,chosethewrongdatatouseinreassembly.ThemodiedNIDSperformedreassemblycorrectly.TomeasuretheimpactofActiveMappingontheNIDS'performanceinthepresenceofarelativelyhighproportionofambiguoustrafc,weusedtwotracesof500connectionstothe8hosts.Intherst,wherenoneoftheconnectionsweremodiedbyfragroute,timeswereessentiallyidenticaloverthreetrials.Inthesecond,whereconnectionstotwoofthemachinesweremodiedbyfragroute,theActiveMapping-enabledNIDSwasactuallyabout15%faster,sinceitwasabletodiscardmoredata.Inpracticeweexpectthiseffecttobesmall,sinceitisonlyrelevantwhenthereareoverlap-pingIPfragmentsorTCPsegments(orthelike);suchoccurrencesareuncommon.5.5.2Real-worldTestsTogetapictureoftheperformanceimpactonalarger,morerealisticdataset,weusedtworeal-worldtraces.Therstwasofawidevarietyofnon-HTTPtrafc(mostlyjustSYN/FIN/RSTpackets,thedatalteredout)gatheredbyaone-hourcaptureatabusysite(100.2MBdata,1.2Mpackets,273Kconnections).Thesec-ondwasoftwohoursofHTTPtrafc(withfulldata)atanothersite(137MB,197Kpackets,6,379connec-tions).Inbothcases,theresultswerethesame:withActiveMappingon,executiontimewasessentiallyiden-tical7.Memoryusagewasapproximately200KhigherwithAM(specicproleswereusedforabout4,800hosts;adefaultonefortherest),asmallfractionofthe68MBusedoverall.WearecurrentlyworkingondeployinganActiveMapping-enabledNIDSoperationallytogetmoredataontheimpactofusingAMprolesonperformanceandprecision.ConclusionsandRecommendationsThetestresultssuggestthatmappingcanbeper-formedquitefrequently.AfullclassCsubnetcanbescannedinabout20minutes,sodailyscansduringoff-peaktimesarecertainlyfeasible.Importantly,withasteady-staterateofabout5secondsperhost(usingourunoptimizedprototype),itisfeasibletocompletelyremapevenlargesites—say,thousandsofhosts—onaweeklybasisduringoff-peakhours.Certaintestswhoseresultsweexpectnottochangeoften(e.g.,thoserelatedtonetworktopology)canbeperformedlessfrequently.Themapping-inducedtrafcofabout19KBperhostmappedisquitelowanditsimpactduringoff-peakhoursislikelytobenegligible.Remappingcanalsobetriggeredbyanyinconsis-tencybetweenthestoredpolicyandanobservedone.Forexample,ifahostsendsanICMPNeedsFragmenta-tionmessageforapacketsmallerthanthestoredPMTU,thenthehostshouldberemapped.Externalinformation,e.g.,OSngerprintresults,canbeusedtodetectchangesinthestatusofamachineaswell.On-the-ymapping—mappingwhentherstpackettoahostisseen—isprobablynotpossible,becausemanyteststakeseveralseconds.Inanycase,hostpol-icychangesaremostlikelytobetriggeredbyinfrequentoperatingsystemupgrades.MorefrequentchangestothepolicydatabasearethoseinitiatedbyDHCP.Aswehavenoted,wecanstorepoliciesbyMACaddressandsimplyupdateatablewhentheNIDSseesaDHCPre-quest(orisinformedofanewleasebytheDHCPserveritself).Fornewhosts—say,alaptopattachedforthersttimetothenetwork—mappingcanbeperformedinun-deroneminute(mappingasinglehosttakesontheorderof30seconds).Thisperiodofuncertaintyisunlikelytobeproblematic,sinceitisrarethatDHCPclientsexportpublicservices.RuntimeperformanceintheNIDSwasnotnegativelyaffectedbytheadditionofActiveMapping-baseddis-ambiguation.Infact,sinceusingActiveMappingre-sultsallowstheNIDStodiscardadditionalpackets,per-formanceinsomecaseswasactuallyimproved.Thead-7WithAMon,itwasslightlyfaster(lessthan1%).Wearenotsurewhythisisthecase;itcouldbeduetominorcacheorcompilereffects12 ditionalmemoryfootprintwasapproximately100bytesperhost.Weexpectwithallmappingsimplementeditwouldbeontheorderofafewhundredbytes.ThemodiedNIDSwasalsocapableofcorrectlyin-terpretingtrafcinawaythattheoriginalonewasnot,detectingpreciseattacksthattheoriginalcouldonlyhintatthroughwarningsaboutinconsistentretransmission.Westressthatnoamountofcareinthedesignoftheoriginalcouldhavechangeditsbehaviorinthisrespect:sincehosts'behaviorvaries,anysinglepolicyemployedbytheNIDSwillinevitablyfailforhoststhatemployadifferentone.6RelatedWork6.1NormalizationAspreviouslydiscussed,trafcnormalizationseekstoeliminatenetworktrafcambiguitiesbyalteringthetrafcstream[HKP01].Thenormalizerliesinthefor-wardingpathofpacketsintoasite.ItreassemblesIPfragmentsandTCPstreamsandstatefullymodiesorltersoutnonconformingtrafcbeforesendingpacketsontotheinternalnetwork.ItsefcacyinimprovingtheprecisionoftheNIDSreliesonitsoutputbeinginter-pretedinthesamewaybyallthehostsonthenetwork.Itlargelysucceedsatachievingthisgoal;thepaperalsodiscussessomeexceptions.Therearedisadvantagestonormalization,however.Anormalizerperformsthesamesortsoftasksasare-wall,butisdoingmorework:itdealswithTCPstreamsratherthanjustindividualpackets.Twomainconcernsarisingfromthiscomplexityareperformanceandro-bustness.Sincethenormalizerisintheforwardingpath,itmustbeabletoprocesseverypacketasitarrives,eveninthepresenceofstateholdingattacksonitself.Fur-ther,itmustbeextremelyreliable;ifitisnot,theentiresitemayloseInternetconnectivity.Anadditionalcon-cernisthatthenormalizerchangesthesemanticsofthestreamsitrewrites.Thiscanblockusefultrafc,causeunintendedproblemswithnewerprotocols,ordecreaseefciency.ItappearsthatActiveMappingcanreplacemanyofthenormalizations(seetheAppendixforafulllist).Still,therearecasesinwhichsomeamountofnormal-izationcanconfersignicantbenets:itsabilitytore-moveagsandoptionscanbeusedtoeliminateanyun-certaintyastotheiruse.Accordingly,itmaysometimesworkbesttousein-formedpartialnormalization,thatis,toperformalim-itedsetofnormalizationsthateliminateambiguitiesthatActiveMappingcannot.Ifthehostprolesindicatethatcertainkindsofnoncompliantpacketsareneverac-ceptedbyanyhost,orifadministratorswantanaddi-tionallayerofsafety,suchpacketsmaybelteredoutattherewall.6.2MappingToolsActiveMapping'stacticofsendingspeciallycraftedpacketsandinterpretingresponsestoinferhostproper-tieshasbeenemployedinavarietyoftools.Themostcommonpurposeforsuchtoolsistode-terminetheoperatingsystemofahost.Nmap[Fyo01]usesportscanscombinedwithIPandTCPoptionsinresponsestoguessahost'soperatingsystem.Queso[Sa98]takesasimilartack,sendingTCPpacketswithillegalagcombinations.BymatchinginitialTTLval-ues,advertisedTCPwindows,initialsequencenumbers,nonconformingresponsestopacketssenttoclosedports,andsoforth,thesetoolscandetectalargenumberofop-eratingsystemversions.Neitherprovidesuswithenoughpreciseinformationonthelonglistofpolicychoicesandparametersweneed.SincedoingOSdetectiontakesapproximatelyaslongasActiveMapping,thereseemslittleadvantagetodoingOSdetectioninsteadforthispurpose;however,knowingthehostOScanbeveryusefulineliminat-ingfalsepositives(i.e.,couldaparticularattackactu-allysucceed?).Wenotethat,especiallyinlightofthefactthatoperatingsystemsmaybeuser-modied,theactuallyobservedbehavioristheonlyrelevantthingforcorrectinterpretation:theOSversionisatbestaproxyforthisinformation.Nonetheless,thereisacertainsynergybetweenthetwo.IfOSdataisknown,itcanserveasaquickproxyformappingcharacteristicswhencoupledtoadatabasecontainingcanonicalmappingsbyOStypeandversion.Conversely,knownmappingscangiveatleastaroughestimationoftheOSahostisrunning.Thiscanbeuse-fulforalertltering:ifaparticularattackonlyworksonLinuxandthemappingdatasuggestaWindowsma-chine,thenwecanlteroutirrelevantalertswithoutknowingmorepreciselytheOSversions.TheNtopNIDShasbeensupplementedwithnetworkinformationinferredfrompassivemonitoring[DS00];thisinformationappearstobelimitedtoguessingthehopcountandguringoutwhichIPaddressescorre-spondtorouters.Tbit[PF01]triestolearntheTCPbehaviorofHTTPserversinregardstocongestioncontrol.ItonlysendslegitimateTCPpackets,relyingonTCPoptions,adver-tisedwindows,andtiminginformationtodeducetheserver'sTCPconguration(orbugstherein).Weuseitsschemeforimplementingouruser-levelTCP.13 7SummaryAmbiguityintheinterpretationofnetworktrafcisacriticaldifcultyforNetworkIntrusionDetection.Thisambiguitytakesmanyforms.Sometypesmaybere-solvedbycarefulconstructionoftheNIDS.Othertypesarefundamentallymoredifculttoresolve,andrequireadditionalinformationaboutthenetworkandthehostsbeingmonitored.Inthispaper,wehavepresentedAc-tiveMapping,amethodofeliminatingnetwork-andtransport-layerambiguitiesbyinformingtheNIDSofrelevantnetworkandhostTCP/IPstackpolicies.WestressthattheambiguitiesthatActiveMappingseekstoaddressarereadilyexploitable;systemshavebeende-signedfordoingjustthat[So02,Mc98].ActiveMappingrunsseparatelyfromtheNIDS(typ-icallyduringoff-peakhours)andworksbysendingspe-ciallycraftedpacketstoeachhostandinferringpolicyfromtheresponsesitreceives(orlackthereof).Itdoesnotrequireanyreal-timemanipulationoftheincomingtrafcstreambytheNIDS.InourtestswithaNIDSmodiedtouseActiveMapping-generatedproles,wefoundthattherewasessentiallynocostintermsofspeedormemoryusetogettheincreasedprecisioninanalysis;weexpectthiswillholdtrueforanyNIDS.Inaddition,wehaveshownthatActiveMappingitselfisefcientintermsoftime,networkbandwidthconsumed,andout-putsize.Preliminarymappingresultsshowconsiderablevariationinpolicyamonghosts'TCP/IPstacks,under-scoringtheneedfortheprecisesimulationthatActiveMappingenables.Futureworkinthisareamightin-cludeexploringusingpassivemonitoringtodeterminewhentoremapahost,aswellasanimplementationofmappingofDHCPclientsandimplementationofmoreofthemappingsdescribedintheAppendix.Finally,wenotethattheproblemofambiguoustraf-cisnotconnedtothenetworkandtransportlayers.Italsooccursattheapplicationlayer—forexample,ex-actlyhowwillaparticularURLbeinterpreted?—anddealingwithallpossibleambiguitiesappearsessentiallyintractable.ActiveMappingprolesmightbeabletohelplowerfalsepositivesbyallowingtheNIDStocon-sideronlyplatform-relevantattacks,butanalysisofthispotentialbenetisbeyondthescopeofthispaper.Thuswedonotclaimtohave“solved”theNIDSevasionproblem.However,webelievethatthegeneralproblemofambiguityresolutionisbestaddressedinasystem-atic,layeredfashion,andActiveMappingrepresentsasteptowardeliminatingambiguityatthebottomlayers.AcknowledgmentsWewouldliketothankMarkHandleyatICIRandParthaBanerjee,MarkDedlow,JayKrous,andCraigLeresintheSNSgroupatLBNLwhohelpeduswithtestingtheActiveMapperandingatheringdataaboutthehostsontheLBNLnetwork.WewouldalsoliketothankNikitaBorisov,MarkHandley,RobJohnson,ChrisKarlof,andDavidWagnerfortheirinsightfulandfocusingcommentsonthispaper.References[Be02]StevenM.Bellovin,“ATechniqueforCount-ingNATtedHosts.”ProceedingsoftheSec-ondInternetMeasurementWorkshop,Novem-ber2002.[B+99]S.Blakeetal,“AnArchitectureforDifferenti-atedServices,”RFC2475,Dec.1998.[Dr97]R.Dromsetal.,“DynamicHostCongurationProtocol,”RFC2131,Mar.1997.[DS00]L.DeriandS.Suin,“ImprovingNetworkSe-curityUsingNtop,”Proc.ThirdInternationalWorkshopontheRecentAdvancesinIntrusionDetection(RAID2000),2000.[EF94]K.EgevangandP.Francis,“TheIPNetworkAddressTranslator(NAT),”RFC1631,1994.[Fyo01]Fyodor.nmap,2001.http://www.insecure.org/nmap/.[HKP01]MarkHandley,ChristianKreibichandVernPaxson,“NetworkIntrusionDetection:Evasion,TrafcNormalization,”Proc.10thUSENIXSecuritySymposium,2001.[Mc98]JohnMcDonald.“DefeatingSniffersandIn-trusionDetectionSystems,”PhrackMagazine,8(54),Dec25th,1998.[MLJ94]S.McCanne,C.LeresandV.Ja-cobson,libpcap,availableathttp://www.tcpdump.org,1994.[Pa98]VernPaxson,“Bro:ASystemforDetectingNetworkIntrudersinReal-Time,”ComputerNetworks,31(23-24),pp.2435–2463,14Dec.1999.[PF01]JitendraPadhyeandSallyFloyd,“IdentifyingtheTCPBehaviorofWebServers,”Proc.ACMSIGCOMM,Aug.2001.14 [PN98]T.H.PtacekandT.N.Newsham,“Inser-tion,EvasionandDenialofService:Elud-ingNetworkIntrusionDetection”,SecureNet-works,Inc.,Jan.1998.http://www.icir.org/vern/Ptacek-Newsham-Evasion-98.ps[Po80]J.Postel,“UserDatagramProtocol,”RFC768,August1980.[Po81a]J.Postel,“InternetProtocol,”RFC791,September1981.[Po81b]J.Postel,“InternetControlMessageProtocol,”RFC792,September1981.[Po81c]J.Postel,“TransmissionControlProtocol,”RFC793,September1981.[Sa98]Savage,“QueSO,”savage@apostols.org,1998.http://www.backupcentral.com/cgi-bin/redirect?url=ftp://contrib.redhat.com/pub/contrib/libc6/SRPMS/queso-980922b-1.src.rpm.[So02]DugSong,fragroute.Availableathttp://www.monkey.org/˜dugsong/fragroute/.[WS95]GaryR.WrightandW.RichardStevens,TCP/IPIllustrated:TheImplementation,Addison-Wesley,1995.[Yl02]T.Ylonenetal,“SSHTransportLayerProto-col,”InternetDraft,workinprogress,2002.15 Appendix:FullListofActiveMappingsIn[HKP01],theauthorsadopteda“header-walking”technique—inspectionofeachTCPandIPheadereld—inanattempttoenumerateallambiguities(whichwouldthenberesolvedusinganormalizer).InouranalysisofActiveMappingasanapproachtothesameproblem,weborrowthatwork'slistofnormalizations,notingforeachhowittsintotheActiveMappingframework.TheideaistotrytogetacompletepictureofhowActiveMappingcan(orcan't)eliminatepossibleTCP/IPambiguitiesbylookingateach,thenstatingwhatsortofmappingtechniquewouldwork.Thereaderisreferredto[HKP01]formorethoroughexplanationsofsomeofthenormalizations.Wenotethatwehavenotimplementedallofthemappingssuggestedbelow;nonetheless,mostarestraightforwardgiventheonesthatwehaveimplementedandtested.TheDispositioncolumninthetablesbelowwillusuallycontainoneofthreemainapproaches,sometimescoupledwithashortexplanation:Drop.Thestatelessrewallshouldbeconguredtodropthispacket.Map.Wecansendchosenprobepacketstothehosttodetermineitspolicy.Themostcommoncase,“Mapfordrop,”indicatesthatthepacketshouldbesenttoahost—usuallyaspartofanopenconnection—toseewhetheritisdroppedoracknowledged.Ignore.Wedonotneedtoperformanymappingforthistest.Thereisatradeoffbetweenacceptingmalformedpacketsthatmightbeusefulandallowinginmalicioustrafc.Forsomenormalizations,achoiceshouldbemadeaboutwhethertheanomalyinquestionmight(orempiricallydoes)ariseinnormaltrafc.Ifitisdecidedthattheanomalouspacketwouldnotarisenormally,itmaybedroppedbyarewallorapartialnormalizerrunninginfrontoftheNIDS.IPNormalizations#IPFieldNormalizationPerformedDisposition1VersionNon-IPv4packetsdropped.DropifNIDSisnotIPv6-aware,elseIgnore.2HeaderLenDropifhdrlentoosmall.Drop.3HeaderLenDropifhdrlentoolarge.Drop.4DiffservCleareld.Ignoreifinternalroutersdon'tsupport;addDiffservpol-icytoNIDSotherwise5ECTCleareld.Mapfordrop.6TotalLenDropiftotlen�linklayerlen.Drop.7TotalLenTrimiftotlenlinklayerlen.Ignore.8IPIdentierEncryptID.Ignore.9ProtocolEnforcespecicprotocols.IgnoreunlesstheNIDSisawareofanyotherprotocol.–ProtocolPasspackettoTCP,UDP,ICMPhandlers.N/A(donebyNIDS)10FragoffsetReassemblefragmentedpackets.Map(seex3.2).11FragoffsetDropifoffset+len�64KB.Maptoseeifdata�64kisacceptedortrimmedoff,butdon'ttriggerknownbugs.12DFClearDF.MapPMTU(seex3.2).13DFDropifDFsetandoffset�0.Mapfordrop.Oneplausibleinterpretationis:donotfur-therfragmentthispacket.SomeSolarismachinesgener-atethesepackets;itisnotdisallowedbyRFC791[Po81a].14ZeroagClear.Firewallshouldclearifpossible;otherwiseMaptoseeifpacketswithzeroagsetaredropped.15SrcaddrDropifclassDorE.Drop.16SrcaddrDropifMSByte=127or0.Drop.17SrcaddrDropif255.255.255.255.Drop.16 #IPFieldNormalizationPerformedDisposition18DstaddrDropifclassE.Drop.19DstaddrDropifMSByte=127or0.Drop.20DstaddrDropif255.255.255.255.Drop.21TTLRaiseTTLtoconguredvalue.Map(seex3.2).22ChecksumVerify,dropifincorrect.DroporoptionallyMapfordrop.23IPoptionsRemoveIPoptions.Mapfordrop(esp.sourceroute/recordroute);addsupportforIPoptionstopacketprocessingonNIDS.Optionallyhaverouterorpartialnormalizerclearunsupportedoptions(packetsalreadytakingslowpathonrouter).24IPoptionsZeropaddingbytes.Ignore.Optionallyhaverouterclearpaddingbytes.UDPNormalizations#UDPFieldNormalizationPerformedDisposition1LengthDropifdoesn'tmatchlengthasindicatedbyIPtotallength.Map:assumeminimumofUDPorIPlengthtaken.Alsomapfordrop.Optionallydropifroutersupportsit.2ChecksumVerify,dropifincorrect.Mapfordrop.OptionallyjustDropifroutersupportsit.TCPNormalizations#TCPFieldNormalizationPerformedDisposition1SeqNumEnforcedataconsistencyinre-transmittedsegments.Map(seex3.2).2SeqNumTrimdatatowindow.Map:sendout-of-windowsegment,thensegmentsinre-versetostartofwindowtopreventstreamreassemblyun-tilallsegmentshavebeenreceived;checkACKsequencepoint.3SeqNumCold-start:trimtokeep-alive.IfNIDScansendpackets,sendkeep-alive(incorrectACKdesignedtoelicitthecurrentsequencepointinanACKfromtheinternalhost).OtherwiseIgnore:thisisacold-startproblem.4AckNumDropACKabovesequencehole.MaptoseeiftheACKisaccepted.5SYNRemovedataifSYN=1.Mapfordrop;ifnot,seeifdataisACKed.6SYNIfSYN=1&RST=1,drop.MaptoseeifRSTacceptedduringopenconnection;MaptoseeifSYNacceptedifnoconnectionestablished.7SYNIfSYN=1&FIN=1,clearFIN.SeeifFINisACKed;thesendercouldplausiblysay,“Iwanttoinitiateaconnection,buthavenothingtosend,”makingtheconnectionhalf-openrightaway.8SYNIfSYN=0&ACK=0&RST=0,drop.MapfordroporoptionallyDrop.9RSTRemovedataifRST=1.Ignore:therearenoknownexploits.Optionallyusenor-malizertoremovedata.10RSTMakeRSTreliable.Ifpossible,haveNIDSsend-keepalivetoensurethatRSTwasaccepted(reliableRST).11RSTDropifnotinwindow.Map(seeSection3.2)12FINIfFIN=1&ACK=0,drop.Mapfordrop.13PUSHIfPUSH=1&ACK=0,drop.Mapfordrop.14HeaderLenDropiflessthan5.Mapfordrop.15HeaderLenDropifbeyondendofpacket.Mapfordrop.16ReservedClear.IgnoreoroptionallyMapfordrop.17 #TCPFieldNormalizationPerformedDisposition17ECE,CWROptionallyclear.Ignore.18ECE,CWRClearifnotnegotiated.Ignore.19WindowRemovewindowwithdrawals.Mapfordrop.20ChecksumVerify,dropifincorrect.Mapfordrop.21URG,urgentZerourgentifURGnotset.Ignore.Optionallyuseapp-levelhostinformation(e.g.,particularHTTPserver)tointerpreturgentdata.22URG,urgentZeroifurgent�endofpacket.Asabove.Notethatitislegalfortheurgentpointertopointbeyondofthepacketcontainingit.23URGIfURG=1&ACK=0,drop.Mapfordrop.24MSSoptionIfSYN=0,removeoption.MaptoseeiftheoptionactuallychangestheMSSinthiscase.25MSSoptionCacheoption,trimdatatoMSS.TheNIDSshoulddothecaching.26WSoptionIfSYN=0,removeoption.Ignore:Windowscalingpresentsacold-startproblem;ifdesired,partialnormalizationcanremovetheoptionorelsetheNIDScantrytoinferitssuccessfromsubsequentACKs.27SACKNormalizations27-31regardingSACKIgnore:SACKsareadvisory,soshouldnotaffectthese-manticstheNIDSuses.32T/TCPoptsRemoveifNIDSdoesn'tsup-port.Mapfordrop.33T/TCPoptsRemoveifunderattack.N/A34TSoptionRemovefromnon-SYNifnotnegotiatedinSYN.Mapfordrop.35TSoptionIfpacketfailsPAWStest,drop.Mapfordrop.36TSoptionIfechoedtimestampwasn'tpre-viouslysent,drop.Mapfordrop.37MD5optionIfMD5usedinSYN,dropnon-SYNpacketswithoutit.MapfordropwhenoptionnotsetinSYN.Ifnotdropped,dosamethinginNIDSasinnormalizer,butthiscausesacold-startproblem.38otheroptsRemoveoptionsIgnore:optionallyremovewithapartialnormalizer.(SeeSection3.3).ICMPNormalizations#ICMPTypeNormalizationPerformedDisposition1EchorequestDropifdestinationisamulticastorbroadcastaddress.OptionallyDrop.2EchorequestOptionallydropifpingcheck-sumincorrect.OptionallyDrop.3EchorequestZero“code”eld.Mapfordrop.4EchoreplyOptionallydropifpingcheck-sumincorrect.Optionallydrop.5EchoreplyDropifnomatchingrequest.Ignore.6EchoreplyZero“code”eld.Mapfordrop.7SourcequenchOptionallydroptopreventDoS.OptionallyDrop.8DestinationUnreachableUnscrambleembeddedscram-bledIPidentier.Ignore:IPidentiersnotscrambledwithoutnormalizer.9otherDrop.OptionallyDropdependingonNIDSpolicy.18