washingtonedu Computer Science and Engineering University of Washington Seattle WA 981952350 ABSTRACT To date realistic ISP topologies have not been accessible to the re search community leaving work that depends on topology on an uncertain footing I ID: 31499
Download Pdf The PPT/PDF document "Measuring ISP Topologies with Rocketfuel..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Figure1:ISPnetworksarecomposedofPOPsandbackbones.SoliddotsinsidethecloudrepresentPOPs.APOPconsistsofbackboneandaccessrouters(inset).EachtracerouteacrosstheISPdiscoversthepathfromthesourcetothedestination.fuel,isdescribedinSection4.WepresentsampleISPmapsinSection5.InSection6,weevaluateourmapsforcompleteness,andourtechniquesfortheirmeasurementefciencyandaccuracy.WeanalyzepropertiesoftheinferredmapsinSection7,presentrelatedworkinSection8,andconcludeinSection9.2.PROBLEMANDAPPROACHThegoalofourworkistoobtainrealistic,router-levelmapsofISPnetworks.Inthissection,wedescribewhatwemeanbyanISPmapandthekeymeasurementchallengesthatwemustaddress.AnISPnetworkiscomposedofmultiplepointsofpresenceorPOPs,asshowninFigure1.EachPOPisaphysicallocationwheretheISPhousesacollectionofrouters.TheISPbackboneconnectsthesePOPs,andtheroutersattachedtointer-POPlinksarecalledbackboneorcorerouters.WithineveryPOP,accessrouterspro-videanintermediatelayerbetweentheISPbackboneandroutersinneighboringnetworks.TheseneighborroutersincludebothBGPspeakersandnon-BGPspeakers,withmostofthembeingnon-BGPspeakingsmallorganizations.OuraimistodiscoverISPmapsthatconsistofbackbone,access,anddirectlyconnectedneighboringdomainroutersalongwiththeIP-levelinterconnectionsbetweenthem.Thisconstitutestheinte-riorroutingregionoftheISP,plusinformationaboutitsbound-aries.ISPsareusuallyassociatedwiththeirBGPautonomoussys-temnumbers(ASNs).Themapwecollectdoesnotpreciselycorre-spondtotheaddressspaceadvertisedbytheASassociatedwithanISP.Ourmaps,asdenedabove,excludeportionsoftheaddressspacethatrepresentnon-BGPspeakingneighboringnetworks,orconsumerbroadbandordialupaccess,buttheydoincludethebound-arywiththeneighbors.Inthepaper,weuseISPnamesandtheirASnumbersinterchangeably,unlessthedistinctionisimportant.LikeearlierInternetmappingefforts[3,6,8],wediscoverISPmapsusingtraceroutes1.ThisprocessisillustratedinFigure1.Eachtracerouteyieldsthepaththroughthenetworktraversedfromthetraceroutesourcetothedestination.Traceroutepathsfrommul-tiplesourcestomultipledestinationsarethenmergedtoobtainan 1Usingtraceroutehasinherent,wellunderstoodlimitationsinstudyingnetworktopology.Forexample,traceroutedoesnotseebackuplinksinanetwork,anddoesnotexposelink-layerredun-dancyordependency(multipleIPlinksoverthesameber),ormulti-accesslinks.ISPmap.Weusepubliclyavailabletracerouteserversassources.Eachtracerouteserverprovidesoneormorevantagepoints:uniquetraceroutesourcesthatmayberouterswithintheAS,orthetracer-outeserveritself.ThekeychallengethatwefaceistobuildaccurateISPmapsus-ingfewmeasurements.Wecannotburdenpublictracerouteserverswithexcessiveload,sothenumberoftracerouteswecancollectfromeachserverislimited.Abrute-forceapproachtoInternetmappingwouldcollecttraceroutesfromeveryvantagepointtoeachofthe120,000allocatedprexesintheBGPtable.Ifpublictracer-outeserversarequeriedatmostonceevery1.5minutes,2thebrute-forceapproachwilltakeatleast125daystocompleteamap,aperiodoverwhichtheInternetcouldundergosignicanttopolog-icalchanges.Anotherbrute-forceapproachistotraceroutetoallIPaddressesownedbytheISP.Eventhisapproachisnotfeasi-blebecauseISPaddressspacecanincludemillionsofaddresses,forexampleAT&T's12.0.0.0/8alonehasmorethan16millionad-dresses.Ourdesignphilosophyistochoosetraceroutesthatwillcon-tributethemostinformationtothemapandomitthosethatarelikelytoberedundant.Ourinsightisthatexpectedroutingpathsprovideavaluablemeanstoguidethisselection.Thisapproachtradesaccuracyforefciency,thoughwewillseeaspartoftheevaluationofthesetechniquesthatthelossofaccuracyismuchsmallerthanthegaininefciency.Thatis,wemakeaworthwhileengineeringtradeoff.Evenaftertheconnectivityinformationhasbeenobtainedthroughtraceroutes,twodifcultiesremain.First,eachtracerouteconsistsofalistofIPaddressesthatrepresentrouterinterfaces.Foranac-curatemap,theIPaddressesthatbelongtothesamerouter,calledaliases,mustberesolved.Whenwestartedtoconstructmaps,wefoundthatpriortechniquesforaliasresolutionwereineffectiveatresolvingobviousaliases.Toaddressthisproblem,wedevelopanew,pair-wisetestforaliasesthatusesavarietyofrouteridenti-cationhintssuchastheIPidentier,rate-limiting,andTTLvalues.Second,tobeabletoanalyzethevariousstructuralpropertiesofthecollectedmaps,weneedtoidentifythegeographicallocationoftherouteranditsroleinthetopology.Followingthesuccessofrecentgeographicalmappingwork[14],weleveragelocationhintsthataretypicallyembeddedintheDNSnametoextractthebackboneandthePOPsfromtheISPmap.3.MAPPINGTECHNIQUESInthissection,wepresentourmappingtechniques.Theyaredividedintothreecategories:selectingmeasurements,resolvingIPaddressesaliases,andidentifyingISProuters,theirrole,andgeographicalinformationfromthetracerouteoutput.3.1SelectingMeasurementsBasedontwoobservations,weusetwoclassesoftechniquestoreducetherequirednumberofmeasurements.First,onlytracer-outesexpectedtotransittheISPneedtobetaken.Weuseatech-niquecalleddirectedprobingthatemploysBGPtablestoidentifyrelevanttraceroutesandprunetheremainder.Second,weareinter-estedonlyinthepartofthetraceroutethattransitstheISP.There-fore,onlyonetracerouteneedstobetakenwhentwotraceroutesenterandleavetheISPnetworkatthesamepoint.Weuseasetoftechniquescalledpathreductionstoidentifyredundanttraceroutes. 2Thislimitwasprovidedbytheadministratorofonetracerouteserver,butisstillaggressive.Traceroutestounresponsivedestina-tionsmaytakemuchlonger. 1.2.3.0/24134256910511754.5.0.0/1637878Figure2:AsampleBGPtablesnippet.Destinationprexesareontheleft,AS-pathsontheright.ASesclosertothedestinationaretotherightofthepath.3.1.1DirectedProbingDirectedprobingaimstoidentifytraceroutesthatwilltransittheISPnetwork.Ideally,ifwehadtheBGProutingtablecorrespond-ingtoeachvantagepoint,wewouldknowthepathsthattransitedtheISPbeingmapped.Sincethesetablesarenotavailable,weuseRouteViewsasanapproximation[13].ItprovidesBGPviewsfrom60vantagepoints.ABGPtablemapsdestinationIPaddressprexestoasetofAS-pathsthatcanbeusedtoreachthatdestination.EachAS-pathrep-resentsthelistofASesthatwillbetraversedtoreachthedestina-tionswithintheprex.WenowshowhowtoidentifythreeclassesoftraceroutesthatshouldtransittheISPnetwork.Inthisexample,weusetheBGPtablesnippetinFigure2tomapASnumber7.Traceroutestodependentprexes:Wecallprexesorigi-natedbytheISPoroneofitssingly-homedcustomersde-pendentprexes.AlltraceroutestotheseprexesfromanyvantagepointshouldtransittheISP.DependentprexescanbereadilyidentiedfromtheBGPtable:allAS-pathsfortheprexwouldcontainthenumberoftheASbeingmapped.4:5:0:0=16isadependentprex.Traceroutesfrominsiders:Wecallatracerouteserverlo-catedinadependentprexaninsider.Traceroutesfromin-siderstoanyprexshouldtransittheISP.TraceroutesthatarelikelytotransittheISPbasedonsomeAS-patharecalledup/downtraces.AtraceroutefromaserverinAS11to1:2:3:0=24isanup/downtraceinFigure2.Directedprobingcapitalizesontheroutinginformationtoskipunnecessarytraceroutes.However,incompleteinformationinBGPtables,dynamicroutingchanges,andmultiplepossiblepathsleadtotwokindsoferrors.ExecutedtraceroutesthatdonottraversetheISP(falsepositives)sacricespeed,butnotaccuracy.TraceroutesthattransittheISPnetwork,butareskippedbecauseourlimitedBGPdatadidnotincludethetruepath(falsenegatives),mayrep-resentalossinaccuracy,whichisthepricewepayforspeed.Inourevaluationsection,weestimatethelevelofboththesetypesoferrors.3.1.2PathReductionsNotallthetracerouteprobesidentiedbydirectedprobingwilltakeuniquepathsinsidetheISP.Thenumberofmeasurementsre-quiredcanbereducedfurtherbyidentifyingprobesthatarelikelytohaveidenticalpathsinsidetheISP.Welistthreedifferenttech-niquesherethatexploitcommonpropertiesofIProutingtocutdownonredundantmeasurements.Althoughdescribedseparately,thesetechniquescomposetobringaboutanevengreaterreductioninthenumberofrequiredmeasurements.IngressReduction.Thepathtakenbyapacketthroughanet-workisusuallydestination-specic.Whentraceroutesfromtwo Figure3:Pathreductions.(a)Onlyonetracerouteneedstobetakenperdestinationwhentwoservers(T's)shareaningress.(b)Onlyonetraceneedstobetakenwhentwodependentpre-xes(P's)shareanegressrouter.(c)Onlyonetraceneedstobetakeniftwoprexeshavethesamenext-hopASnumber. Figure4:Aliasresolution.Boxesrepresentroutersandcir-clesrepresentinterfaces.Traceroutelistsinputinterfacead-dressesfrompaths(left).Aliasresolutionclustersinterfacesintorouterstorevealthetruetopology.InterfacesÊandËarealiases(right).differentvantagepointstothesamedestinationentertheISPatthesamepoint,thepaththroughtheISPislikelytobethesame.ThisisillustratedinFigure3a.SincethetraceroutefromT2tothedes-tinationwouldberedundantwiththetraceroutefromT1,onlyoneisneeded.Thisredundancycanalsobeexploitedtobalanceloadbetweentracerouteservers.EgressReduction.Similarly,tracesfromthesameingresstoanyprexbehindthesameegressroutershouldtraversethesamepath.Suchtracesareredundant,soonlyoneneedstobecollected.ThisisillustratedinFigure3b.Next-hopASReduction.ThepaththroughanISPusuallyde-pendsonlyonthenext-hopAS,notonthespecicdestinationpre-x.Thismeansthatonlyonetracefromingressroutertonext-hopASislikelytobevaluable,asillustratedinFigure3c.Unlikeegressreduction,Next-hopASreductiondoesnotassumethatthereisonlyoneegressperdestination:thenext-hopASmaypeerinsev-eralplaces,whilethereislikelyonlyoneegressforacustomer'sprex.Ingressandegresspredictionsremovelikelyduplicatessothatmorevaluabletracescanbetakeninsteadwithoutsacricing-delity.Ifwendthatouringress-routerpredictionwasincorrect,werepeatthetraceusingotherserversthatsharethepredictedingress.3.2AliasResolutionTracerouteliststhesourceaddressesofthe“Timeexceeded”ICMPmessages;theseaddressesrepresenttheinterfacesthatre- ceivedtracerouteprobepackets.Asignicantprobleminrecov-eringanetworkmapfromtraceroutesisaliasresolution,ordeter-miningwhichinterfaceIPaddressesbelongtothesamerouter.TheproblemisillustratedinFigure4.Ifthedifferentaddressesthatrepresentthesameroutercannotberesolved,wegetadifferenttopologywithmoreroutersandlinksthantherealone.ThestandardtechniqueforaliasresolutionwasdevelopedasparttheMercatorproject[8].Itdetectsaliasesbysendingtraceroute-likeprobe(toahigh-numberedUDPportbutwithaTTLof255)directlytothepotentiallyaliasedIPaddress.Itreliesonroutersbeingconguredtosendthe“UDPportunreachable”responsewiththeaddressoftheoutgoinginterfaceasthesourceaddress:twoaliaseswillrespondwiththesamesource.Mercator'stechniqueisefcientinthatitrequiresonlyonemessagetoeachIPaddress,butwefoundthatitmissedmanyaliases.OurapproachtoaliasresolutioncombinesseveraltechniquesthatidentifypeculiarsimilaritiesbetweenresponsestopacketssenttodifferentIPaddresses.WeincludeMercator'sIPaddress-basedmethod,whichdetectsanaliaswhenbothresponseshavethesamesourceaddress.WecomparetheTTLsinresponsestoaddcon-dencetoanaliasmatch,aswellastochoosecandidateaddresspairstotest,althoughcomparingTTLsisnotaccuratebyitself.WetestforICMPratelimiting,asdescribedbelow.3However,noneofthesetechniqueswereassuccessfulascomparingtheIPidentiereldoftheresponses.TheIPidentierisintendedtohelpinuniquelyidentifyingapacketforreassemblyafterfragmentation.Assuch,itiscommonlyimplementedusingacounterthatisincrementedaftersendingapacket.Thisimpliesthatpacketssentconsecutivelywillhavecon-secutiveIPidentiers.4 Figure5:AliasresolutionbyIPidentiers.Asolidarrowrep-resentsmessagestoandfromoneIP,thedottedarrowtheother.TheprocedureforresolvingaliasesbyIPidentierisshowninFigure5.Ourtoolforaliasresolution,Ally,sendsaprobepacketsimilartoMercator'stothetwopotentialaliases.Theportunreach-ableresponsesincludetheIPidentiersxandy.Allysendsathirdpackettotheaddressthatrespondedrst.Ifxyz,andz�xissmall,theaddressesarelikelyaliases.Inpractice,sometoler-anceisallowedforreorderinginthenetwork.Asanoptimization,ifjx�yj&#x-285;200,thealiasesaredisqualiedandthethirdpacketisnotsent.Thisestablishesarange:in-orderIPidentierssuggestasinglecounter,whichimpliesthattheaddressesarelikelyaliases.Thisthree-packetapproachcompensatesforroutersthatincrementtheIPidcounteratvaryingspeeds,andreducesthelikelihoodofafalsepositive.Someroutersareconguredtorate-limitportunreachablemes-sages.Ifonlytherstprobepacketsolicitsaresponse,theprobedestinationsarereorderedandtwoprobesaresentagainafterve 3Wefoundthatrate-limitingroutersgenerallyrepliedwiththesamesourceaddressandwouldbedetectedbyMercator.4Wehavenotobservedanyroutersthatuserandomidentiersorimplementthecounterinleast-signicant-byteorder,thoughsomedonotsettheidentiereldatall.seconds.Ifagainonlytherstprobepacketsolicitsaresponse,thistimetothepacketfortheotheraddress,therate-limitingheuris-ticdetectsamatch.Whentwoaddressesappeartoberate-limitedaliases,theIPidentiertechniquealsodetectsamatchwhentheidentiersdifferbylessthan1000.Thereisasmallprobabilitythattworesponsepacketswillhavenearbyidentiers,withouttheIPaddressesactuallybeingaliases.Toremovefalsepositives,werepeatthealiasresolutiontestonunveriedaliasesatalatertime.3.3RouterIdenticationandAnnotationInthissectionwedescribehowwedeterminewhichroutersinthetracerouteoutputbelongtotheISPbeingmapped,theirgeo-graphicallocation,andtheirroleinthetopology.WerelyontheDNStoidentifyroutersthatbelongtotheISP.TheDNSnamesprovideamoreaccuratecharacterizationthantheaddressspaceadvertisedbytheASforthreereasons.First,routersofnon-BGPspeakingneighborsareoftennumberedfromtheAS'saddressspaceitself.Inthiscase,theDNSnameshelptoaccu-ratelylocatetheISPnetworkedgebecausetheneighboringdomainroutersarenotnamedintheISPsdomain(att.net,sprintlink.net,etc.).Insomecases,thedirectlyconnectedneighboringdomainroutershaveaspecialnamingconventionthathelpslocatethenet-workedge.Forinstance,smallneighbors(customerorganizations)ofSprintarenamedsl-neighborname.sprintlink.net,whichisdifferentfromSprint'sinternalrouternamingconvention.Sec-ond,edgelinksbetweentwonetworkscouldbenumberedfromeitherAS'saddressspace.Again,DNSnameshelptoidentifythenetworkedgecorrectlyiftheyareassignedcorrectly.Finally,DNSnamesareeffectiveinpruningoutcablemodems,DSL,anddialupmodempoolsbelongingtothesameorganizationastheISP,andhencenumberedfromthesameaddressspace.Weresorttothead-dressspacecriterionforrouterswithnoDNSnames(weobservedveryfewofthese),withtheconstraintthatallroutersbelongingtotheISPwouldbecontiguousinthetracerouteoutput.OneofourgoalswastounderstandthestructureofISPmaps,whichincludestheirbackboneandPOPs.Todothisweidentifytheroleofeachrouteraswellasitslocation.WeagainusetheinformationembeddedintheDNSnamesforthispurpose.MostISPshaveanamingconventionfortheirrouters.Forexample,sl-bb11-nyc-3-0.sprintlink.netisaSprintbackbone(bb11)routerinNewYorkCity(nyc),andp4-0-0-0.r01.miamfl01.us.bb.verio.netisaVeriobackbone(bb)routerinMiami,Florida(miam01).WediscoverthenamingconventionoftheISPbybrowsingthroughthelistofrouternameswegather.ForsomeISPs,westartedwithcitycodesfromthedatabasein[14].SomeroutershavenoDNSnamesortheirnameslacklocationinformation.Weinferthelocationofsuchroutersfromthatofitsneighbors.4.ROCKETFUELInthissection,wedescribeRocketfuel,ourmappingenginethatinfersmapsusingtheabovetechniques.ThearchitectureofRock-etfuelisshowninFigure6.APostgreSQLdatabasestoresallinfor-mationintheblackboardarchitecture:thedatabaseprovidesbothpersistentstorageofmeasurementresultsandasubstrateforinter-processcommunicationbetweenasynchronouslyrunningprocesses.TheuseofadatabaseenablesustorunSQLqueriesforsimplequestions,andintegratenewanalysismoduleseasily.Weused294publictracerouteserverslistedbythetraceroute.orgWebpage[9],representing784vantagepointsallacrosstheworld.Onetracerouteservercanpotentiallygeneratetraceroutesfrommanyroutersinthesameautonomoussystem.Forexample,oxide.sprintlink.netcangeneratetraceroutesfrom30differentvan- Figure6:ArchitectureofRocketfuel.TheDatabasebecomestheinter-processcommunicationsubstrate.tagepoints.277ofthepublictracerouteserversonlysupportonesource.TheBGPtablesaretakenfromRouteViews[13].WenowdescribeeachmoduleinFigure6inturn.First,egressdiscoveryistheprocessofndingtheegressroutersfordependentprexes.Thisinformationisusedforegressreduction.Tondtheegressrouters,wetraceroutetoeachdependentprexfromalocalmachine.BecausedependentprexesadvertisedbytheISPmaybeaggregated,webreaktheseprexesinto/24's(prexesoflength24,or,equivalently,256IPaddresses)beforeprobing.Weassumethatbreakingdownto/24sissufcienttodiscoverallegressesfordependentprexes.ThetasklistgenerationmoduleusesBGPtablestogeneratealistofdirectedprobes.Thedependentprexesinthedirectedprobesarereplacedwiththeir(possiblymultiple)egresses,andthedupli-catesareremoved.Thisenablesustotracejusttotheegresses,andnotbeyond.Pathreductionstakethetasklistfromthedatabase,applyingressandnext-hopASreductions,andgeneratejobsforexecution.In-formationabouttraceroutesexecutedinthepastisusedbythepathreductionsmoduletoperformthereductions.Forexample,pasttracerouteswouldtelluswhichingressisusedbyavantagepoint.Afteratracerouteistaken,thismodulealsocheckswhetherthepredictedingressoregresswasused.Ifso,thejobiscompleteandcanbetakenoffthelist.Otherwise,anothervantagepointthatislikelytotakethatpathistried.Theexecutionenginehandlesthecomplexitiesofusingpubliclyavailabletracerouteservers:load-limiting,load-balancing,anddif-ferentformatsoftracerouteoutput.Loaddistributionacrossdesti-nationsisachievedbyrandomizingthejoblist,whichisdonebysortingtheMD5hash[19]ofthejobs.Weenforceaveminutepausebetweenaccessestothesametracerouteservertoavoidover-loadingit.Traceroutestothesamedestinationprexarenotexe-cutedsimultaneouslytoavoidhot-spots.ThetracerouteparserextractsIPaddressesthatrepresentrouterinterfacesandpairsofIPaddressesthatrepresentlinksfromtheoutputoftracerouteservers.Oftenthisoutputincludespresentationmark-uplikeheaders,tablesandgraphics.AliasresolutionusingtheIPidentiertechniqueinSection3.2requiressomeengineeringtokeepfromtestingeverypairofIPad-dresses.Wereducethesearchspacewiththreesimpleheuristics.First,andmosteffectively,weexploitthehierarchyembeddedinDNSnamesbysortingrouterIPaddressesbytheir(piecewise)re-versedname.Forexample,nameslikechi-sea-oc12.chicago.isp.netandchi-sfo-oc48.chicago.isp.netarelexigraphi-callyadjacent,andadjacentpairsaretested.Second,routerIPswhosereplieshavenearbyreturnTTLsmayalsobealiases.IPsaregroupedbytheTTLoftheirlastresponse,andpairswithnearbyTTLaretested,startingwiththoseofequalTTL,thenthosewithin1,etc.Ofthe16,000aliaseswefound,94%matchedthereturnTTL,whileonly80%matchedtheoutgoingTTL.Third,“isanaliasfor”isatransitiverelation,sodemonstratingthatIP1isanaliasforIP2,alsodemonstratesthatallaliasesforIP1arealiasesforanyofIP2'saliases.AliasresolutioniscompletewhenalllikelypairsofIPaddressesareresolvedeitherasaliases,notaliases,orunresponsive.5.ISPMAPSWeranRocketfueltomaptendiverseISPs.Inthissection,wepresentsummarymapinformation,andsamplesofbackboneandPOPtopology.Thefullmapset,withimagesofthebackbonesandallthePOPsofthetenISPs,isavailableat[20].5.1SummaryInformationThenamesandaggregatestatisticsforalltenmappedISPsareshowninTable1.WeseealargerangeinthesizesoftheISPs,withthebiggestnetworks,AT&T,Sprint,andVeriodependingonthemetric,100timeslargerthanthesmallestnetworks.5.2BackbonesItisevidentthatthestyleofbackbonedesignvarieswidelybe-tweenISPs.Figure7showsthreesamplebackbonesoverlaidonamapoftheUnitedStates[23].WeseethattheAT&T'sbackbonenetworktopologyincludeshubsinmajorcitiesandspokesthatfanouttosmallerper-citysatellitePOPs.Incontrast,Sprint'snetworkhasonly20POPsintheUSA,allinmajorcities,andwellcon-nectedtoeachother,implyingthattheirsmallercitycustomersarebackhauledintothesemajorhubs.MostmajorprovidersstillhavetheAT&Ttypenetwork,andareinvariousstagesoftransitiontothisnewerdesign[4].Level3representsyetanotherparadigminbackbonedesign.Itshighlyconnectedbackboneismostlikelytheresultofusingacircuittechnology,suchasMPLS,ATMorframerelayPVCs,totunnelbetweenPOPs.5.3POPsUnlikethebackbonedesigns,wefoundPOPdesignstoberel-ativelysimilar.AgenericPOPhasafewbackboneroutersinadenselyconnectedmesh.InlargePOPs,backboneroutersmaynotbeconnectedinafullmesh.BackboneroutersalsoconnecttobackboneroutersinotherPOPs.Eachaccessrouterconnectstooneormoreroutersfromtheneighboringdomainandtotwoback-boneroutersforredundancy.Itisnotnecessarythatallneighbor-ingroutersareconnectedtotheaccessrouterusingapoint-to-pointlink.Instead,alayer2devicesuchasabridge,oramulti-accessmediumsuchasaLANmayaggregateneighboringroutersthatconnecttoanaccessrouter.Onecannotdifferentiatethesescenar-iosfrompoint-to-pointconnectionsusingtraceroute.Asanexampleofacommonpattern,Figure8showsourmapofSprint'sPOPinSpringeld,IL.ThisisasmallPOP;largePOPsaretoocomplextoshowhereindetail.Inthegure,namesofthealiasesarelistedtogether.Thethreebackbonenodesareshownontop,withtheaccessroutersbelow.Sprint'snamingconventionisapparent:sl-bbnnamesbackbonerouters,andsl-gwnnamestheiraccessrouters.Mostdirectlyconnectedneighboringrouters(notshown)arenamedassl-neighborname.sprintlink.net.ThesearemainlysmallorganizationsforwhichSprintprovidestransit.ThevalueofDNSnamesforunderstandingtheroleofroutersinthetopologyisclearfromthisnamingpractice. AS Name ISP withcustomer&peer POPs Routers Links Routers Links 1221 Telstra(Australia) 355 700 2,796 3,000 61 1239 Sprintlink(US) 547 1,600 8,355 9,500 43 1755 Ebone(Europe) 163 300 596 500 25 2914 Verio(US) 1,018 2,300 7,336 6,800 121 3257 Tiscali(Europe) 276 400 865 700 50 3356 Level3(US) 624 5,300 3,446 6,700 52 3967 Exodus(US) 338 800 900 1,100 23 4755 VSNL(India) 11 12 121 69 10 6461 Abovenet(US) 367 1,000 2,259 1,400 21 7018 AT&T(US) 733 2,300 10,214 12,500 108 Table1:Thenumberofrouters,links,andPOPsforalltenISPs.ISProutersincludebackboneandgatewayrouters.Withcustomerandpeerroutersaddsdirectlyconnectedcustomeraccessandpeerrouters.Linksincludeonlyinterconnectionsbetweenthesesetsofrouters,andareroundedtothenearesthundreds.POPsareidentiedbydistinctlocationtagsintheISP'snamingconvention. Figure7:BackbonetopologiesofAT&T(top),Sprint(middle),andLevel3(bottom).Multiplelinksmaybepresentbetweentwocities;onlyonelinkisshownforclarity.Shadedreliefback-groundimagec 1995RaySterner,JohnsHopkinsUniversityAppliedPhysicsLaboratory,usedwithpermission.6.EVALUATIONInthissectionweevaluatetheeffectivenessofourtechniquesalongtwoaxes:thedelityoftheresultingmapsandtheefciencywithwhichtheywereconstructed. Figure8:AsamplePOPtopologyfromSprintinSpringeld,Illinois.Thenamesareprexesofthefullnames,withoutsprintlink.net.MostPOPsinSprintarelargerandtoocom-plextoshow,butretainthesamedesign.6.1CompletenessWeusefourindependentteststoestimatetheaccuracyandcom-pletenessofourmaps.First,weasktheISPswemappedtohelpwithvalidation.Second,wedeviseanewtechniquetoestimatethecompletenessofanISPmapusingIPaddresscoverage.Third,wecomparetheBGPpeeringswefoundtothosepresentatRoute-Views.Finally,wecompareourmapswiththoseobtainedbySkit-ter[6],anon-goingInternetmappingeffortatCAIDA.6.1.1ValidatingwithISPsThreeoutoftenISPsassisteduswithapartialvalidationoftheirmaps.WedonotidentifytheISPsbecausethevalidationwascon-dential.Belowwelistthequestionsweaskedandtheanswerswereceived.1.DidwemissanyPOP?AllthreeISPssaidNo.Inonecase,theISPpointedoutamislocatedrouter;therouter'scitycodewasnotinourdatabase.2.DidwemissanylinksbetweenPOPs?AllthreesaidNo,though,intwocaseswehadaspuriouslinkinourmap. Thiscouldbecausedbybrokentracerouteoutputoraroutingchangewhilethetraceroutewasbeingtaken.3.UsingarandomsampleofPOPs,whatfractionofaccessroutersdidwemiss?OneISPcouldnotspotobviousmisses;anothersaidallbackbonerouterswerepresent,butsomeac-cessroutersweremissing;andthethirdsaidwehadincludedroutersfromanafliatedAS.4.Whatfractionofcustomerroutersdidwemiss?NoneoftheISPswerewillingtoanswerthisquestion.Twoclaimedthattheyhadnowaytocheckthisinformation.5.Overall,doyourateourmaps:poor,fair,good,verygood,orexcellent?Wereceivedthreeresponses:“Good,”“Verygood,”and“Verygoodtoexcellent.”Wefoundtheseresultsencouraging,astheysuggestthatwehaveanearlyaccuratebackboneandreasonablePOPs.ThissurveyandourownvalidationattemptsusingpublicISPmapsalsoconrmstousthatthepublicmapsarenotauthoritativesourcesoftopology.TheyoftenhavemissingPOPs,optimisticdeploymentprojections,andshowpartsofpartnernetworksmanagedbyotherISPs.6.1.2IPaddressspaceAsanothercompletenessestimate,wesearchedprexesoftheISP'saddressspaceforadditionalresponsiveIPaddresses.Newroutersfoundbyscanningaddressspacewouldtellusthatourtracerouteshavenotcoveredsomepartsofthetopology.Weran-domlyselected60/24prexesfromeachISPthatincludedatleasttworouterstosearchfornewrouters.Weselectonlyprexeswithatleasttworouters,becausemanyprexesusedtoconnectISPswillhaveonlyonerouterfromthemappedISP:ourcoverageofsuchaprexwouldbe100%,providinglittleinformation.NewroutersarethoseIPaddressesthatbothrespondtopingandhavenamesthatfollowtheISP'srouternamingconvention,thoughtheymaynotparticipateinforwarding.Prexeswerechosentomakesurethatbothbackboneandaccessrouterswererepresented.Table2showstheestimatedpercentagecoverageforeachISP.ThispercentageiscalculatedasthenumberofknownIPaddressesrelativetothetotalnumberofaddressesseeninthesubnets,notcountingadditionalaliasesofknownrouters.IftheISPhasacon-sistentnamingconventionforbackboneroutersandaccessrouters,thetotalisbrokendownintoseparatecolumns,otherwisen/aisshown.Thetablesuggeststhatwendfrom64%-96%oftheISPbackbonerouters.Theaccessroutercoverageisfair,andingenerallessthanbackbonecoverage.WeplantoinvestigatethedifferencesbetweentheroutersfoundbyRocketfuelandaddressrangescan-ning.6.1.3ComparisonwithRouteViewsAnotherestimateforcompletenessisthenumberofBGPadja-cenciesseeninourmapscomparedtothenumberobservedintheBGPtablesfromRouteViews[13].ForeachadjacencyintheBGPtable,acomplete,router-levelmapshouldincludeatleastonelinkfromarouterinthemappedAStooneintheneighboringAS.Figure9comparesthenumberofadjacenciesseenbyRocketfuelandRouteViews.TheworstcaseforRocketfuelisSprint(1239),wherewestillndmorethan70%oftheneighbors.Itisinterest-ingthatRocketfueldiscoversneighborsthatarenotpresentintheBGPtable,aresultconsistentwith[5].Westudiedtheadjacen-ciesfoundbybothapproaches,andfoundthattheBGPtablesndmoresmall(lowdegreeintheAS-graph)neighbors,whileRocket-fuelndsmorelargeneighbors. AS Backbone Access Total Telstra(1221) 64.4% 78.1% 48.6% Sprint(1239) 90.1% 35.0% 61.3% Ebone(1755) 78.8% 55.1% 65.2% Verio(2914) 75.1% 60.6% 57.5% Tiscali(3257) 89.1% n/a 41.5% Level3(3356) 78.6% 77.4% 55.6% Exodus(3967) 95.4% 59.8% 53.6% VSNL(4755) n/a n/a 48.4% Abovenet(6461) 83.6% n/a 76.0% AT&T(7018) 65.4% 91.6% 78.9% Table2:EstimateofRocketfuel'scoverageofrouter-likenamedIPaddresses.Aliasesofknownroutersarenotcounted.“n/a”impliesthattheISP'snamingconventiondoesn'tdiffer-entiatebetweenbackboneandaccessrouters. Figure9:ComparisonbetweenBGPadjacenciesseeninourmapsandthoseseenintheBGPtablesfromRouteViews. Figure10:ComparisonbetweenRocketfuelandSkitterforeachISP.6.1.4ComparisonwithSkitterSkitterisatraceroute-basedmappingprojectrunbyCAIDA[6].WeanalyzeSkitterdatathatwascollectedon11-27-01and11-28-01.WecomparetheIPaddresses,routersafteraliasresolution,andlinksseenbySkitterandRocketfuelforeachmappedAS.Wealsocountthenumberofroutersandlinksseeninonlyonedataset.TheIPaddressstatisticsarepresentedforeachASinFigure10andallthreestatisticsaresummarizedinTable3.Rocketfuelndsroughlyseventimesasmanylinks,IPsandroutersinitsareaoffocus.SomeroutersandlinkswereonlyfoundbySkitter.Whilesomeofthisdifferenceisduetothedifferenttimesofmapcollection(Skitterwas11/01,andRocketfuelwas1/02),mostofitcorrespondstoroutersmissedbyRocketfuel.Weinvestigatedandfoundthatthebulkofthesewereneighboringdo- mainrouters,andsomewereaccessrouters.ThatbothtoolsnddifferentroutersandlinksunderscoresthecomplexityofInternetmapping.6.2ImpactofReductionsThissectionevaluatesthedirectedprobingandpathreductionsdescribedinSection3.Weevaluatethesetechniquesforboththeextentofreductioninmeasurementsthattheybring,andpotentiallossofaccuracythattheycause.MostoftheresultspresentedhereareaggregatedoverallthetenISP'swemap;individualresultswerelargelysimilar.6.2.1DirectedProbingWeconsiderthreeaspectsofdirectedprobing:thefractionoftracesitisabletoprune;theamountofprunedtracesthatwouldhavetransitedtheISPandshouldhavebeenkept;andthetracesthatshouldnothavebeentakenbecausetheydidnotinfacttransittheISP.TheeffectivenessofdirectedprobingisshowninTable4.Thebrute-forcesearchfromallvantagepointstoallBGP-advertisedprexesplusthebrokendownISPprexes(/24s)requires90-150milliontraceroutes.Withdirectedprobingonlybetween1-8%ofthesetracesarerequired.Toestimatehowmanyusefultraces,whichwouldhavetraversedtheISP,werepruned,werananexperimentusingSkitterdata.As-sumingthattheonlyvantagepointsavailabletouswerethoseusedbySkitter,wecalculatethetracesthatwouldhavebeenselectedbydirectedprobing.UsingtheseandtheSkittertraces,whicharecollectedthroughbrute-forcemapping,wecalculatethefractionofSkittertracesthattraversedtheISPbutwerenotidentiedbydirectedprobing.WendthatthisfractionofusefulbutprunedtracesvariesbyISPfrom0.1to7%.Itislowfornon-USISPslikeVSNL(4755)andTiscali(3257),andhighforthebigUSISPslikeAT&TandSprint.Thisvariationcanbeattributedtothediffer-enceinthelikelihoodofatracefromavantagepointtoarandomlyselecteddestinationtraversingtheISP.Evenwhenthefractionofusefultracesis7%,itmeansthatinabsenceofextrainformationsuchasBGPtablesofthetracerouteserveritself,wewouldhavetocarryout100extrameasurementstoget7potentiallyusefulones.Wedidnotexplorehowmanyofthesepotentiallyusefultraceswereactuallyusefulinthattheyyieldedpathsnotalreadyseenbythechosentraces.Todeterminehowmanytraceswetookthatwereunnecessary,wewereabletotallydirectlyfromourmeasurementdatabase.Ofallthetraceswetook,roughly6%didnottransittheISP.Thesenumbersareveryencouragingbecausetakentogethertheymeanthatnotonlydoesdirectedprobinghelpcutthenumberoftracesto1-8%,butthatthereisverylittleusefulworkinwhatisprunedout,andverylittleuselessworkinwhatisdone.6.2.2IngressReductionInthissection,weevaluateingressreductionforitseffectivenessindiscardingunnecessarytraces.Overall,ingressreductionkeptonly12%ofthetraceschosenbydirectedprobing.ForVSNL,ingressreductionkeptonly2%astherewereonlyafewingressesforourmanyvantagepoints.ThenumberofvantagepointsthatshareaningressisgiveninFigure11.Thenumberofvantagepointssharinganingressissortedindecreasingorder,andplottedonalog-logscale.Fromtherightsideofthecurve,weseethattheapproachofusingpublictracerouteserversprovidesalargenumberofdistinctingressesintothemappedASes.Attheleft,manyvantagepointsshareasmallnumberofingresses,whichimpliesthatingressreductionsignif- Figure11:Thenumberofvantagepointsthatshareaningress,byrank,aggregatedacrossASes.232vantagepointssharethesameingressatleft,while247vantagepointshaveuniquein-gresses. Figure12:Thenumberofdependentprexesthatshareanegress,byrank,andaggregatedacrossallASes./24srefertobrokendownISPprexes,andallalsoincludesthedependentprexesnotoriginatedbytheISP.icantlyreducestheamountofworknecessary,evenafterdirectedprobing.6.2.3EgressReductionOverall,egressreductionkeptonly18%ofthedependentprextraceschosenbydirectedprobing.Figure12showsthenumberofdependentprexesthatshareanegressrouter.Thex-axisrepre-sentseachegressrouter,andthey-axisrepresentsthenumberofprexesthatsharethategress.Theleftpartofthecurvedepictsegressessharedbymultipleprexes,anddemonstratestheeffectivenessofegressreduction.Therightpartshowsthatmanyprexeshaduniqueegresses,evenforbrokendown/24s.ThisshowsthenecessityofbreakingdownlargeCIDRprexesintosmallerunits;mappingusingoneaddressperprexfromBGPtables,asperformedbyexistingBGPbasedmappingtechniques,wouldmissoutonmanyroutersinsidetheISP.Totestourhypothesisof/24beingasufcientlynegranu-larityforegressdiscovery,werandomlychose100/24s(halfofthesewereISPprexes)fromthesetofdependentprexesandbrokethemdowninto/30s.Wethentracedtoeach/30froma Links IPs Routers Total Unique Total Unique Total Unique Rocketfuel 92723 84317 57528 49389 50787 45720 Skitter 12643 4237 9533 1392 7245 1323 Table3:ComparisonbetweenRocketfuelandSkitteraggregatedoverall10ISPs. ASN Name Brute Directed Remote Egress Force Probes Traceroutes Discovery 1221 Telstra(Australia) 105M 1.5M 20K 20K 1239 Sprintlink(US) 132M 10.3M 144K 54K 1755 Ebone(Europe) 91M 15.3M 16K 1K 2914 Verio(US) 118M 1.6M 241K 36K 3257 Tiscali(Europe) 92M 0.2M 6K 2K 3356 Level3(US) 98M 5.0M 305K 10K 3967 Exodus(US) 91M 1.2M 24K 1K 4755 VSNL(India) 92M 0.5M 5K 2K 6461 Abovenet(US) 92M 0.7M 111K 3K 7018 AT&T(US) 152M 4.5M 150K 80K Table4:Theeffectivenessofdirectedprobing,alongwithasummaryofthenumberoftraceroutestaken,includingthosefromremote,publictracerouteservers,andthosetakenlocallytodetermineegressesfordependentprexes.localmachine.Theratioofpreviouslyunseenegressestotheto-taldiscoveredisanestimateofaccuracylossinexploringtheISPboundariesduetonotbreakingdownmorenely.Overall,0-20%oftheegressesdiscoveredduringthisprocesswerepreviouslyun-seen,withthemedianat8%.Thewiderangeinfractionofnewlydiscoveredegressessuggeststhatourassumption,whilevalidforsomeISPs(twoofthemhad0%newegresses),isnotuniversallyapplicable.Thisisperhapsbecausetheminimumcustomeralloca-tionunitusedbysomeISPsissmallerthana/24.Inthefuture,weintendtodynamicallyexplorethelengthtowhicheachdependentprexshouldbebrokendowntodiscoverallegresses.Techniquessuchasbinarysearchcanbeusedeffectivelyforthispurpose.6.2.4NextHopASReductionNext-hopASreductionreducesthenumberofup/downandin-sidertraces(thosethatleavetheISPtoenteranotherAS)to5%ofthosechosenbydirectedprobing.InFigure13,weshowthenumberofprexeschosenforeachvantagepoint(theupperline),andthenumberofnext-hopASesthatrepresentjobsafterreduc-tion.Next-hopreductioniseffectivebecausethenumberofnext-hopASesisconsistentlymuchsmallerthanthenumberofprexes.Itisparticularlyvaluableforinsiderswho,withdirectedprobing,wouldotherwisetraceroutetoall120,000prexesintheBGPta-ble.Next-hopASreductionallowsinsiderstoinsteadtracetoonly1,000externaldestinations.Wealsoevaluatehowcommonlytheearlyexitassumptionun-derlyingthereductionisviolated.WeusedVerioasatestcasebyconducting600Ktraceswithoutthereduction.Inall,thetracescontained2500(ingress,next-hopAS)pairs.Wefoundthatinonly7%ofthecasesdidaningressseemorethanoneegresswhencross-ingovertothesameAS.ItislikelythatdifferentISPshavedifferentpoliciesregardingearly-exitrouting,butneverthelessthisresultisencouraging.6.2.5OverallImpactOurreductionsaremostlyorthogonaltoeachotherandtheycomposetogivemultiplicativebenet.TheoverallimpactofthereductionscanbeseeninTable4,whichshowsthetotalnumberoftraceroutesthatwetooktoinferthemaps.Weexecutedlessthan0.1%ofthetracesrequiredbyabrute-forcetechnique,areductionofthreeordersinmagnitude.TheindividualreductionsforISPsvariedbetween0.3%(Level3)to0.05%(VSNL). Figure13:Thenumberofprexesanduniquenext-hopASesforvantagepoints.Ourmappingtechniquesalsoscalewiththenumberofvantagepoints.Ifwearegivenmorevantagepoints,wewouldbeabletogeneratebettermapsmorequickly,butwouldnotincreasethenumberoftraceroutesunnecessarily.Extravantagepointsbringoneofthetwothingstothetable–speedoraccuracy.Thespeedisimprovedwhenthenewvantagepointsharesaningresswithanexistingvantagepoint.AccuracyisimprovedifthenewvantagepointhasauniqueingresstotheISP.6.3AliasResolutionTheeffectivenessofboththeIPaddressbasedapproachandournewapproachtoaliasresolutionisshowninTable5.Thetableshowshowmanyaliases,whichareadditionalIPaddressesforthesamerouterbeyondtherst,werefoundbyeachtechnique.Allyndsalmostthreetimesmorealiasesthantheearlieraddress-basedapproach.Moreover,wefoundaliasesresolvedbyAllytobeasu-persetofthoseresolvedbyanaddress-basedtechnique.ThismeansthatusingonlyAllysufcesforaliasresolution.Asatesttobuildcondencethattheresolvedaliaseswerecor-rectandcomplete,wecomparethealiasesfoundbyAllytothosepredictedbyDNSnames.WechosetwoISPs,EboneandSprint,thatnamemanyoftheirrouterswitheasilyrecognizeduniqueiden- Tool AliasesRecognized Mercator 2,656 Ally 7,423 Table5:Ally'sIPidentier-basedtechniquendsalmostthreetimesasmanyaliasesasanaddress-basedtechnique. Figure14:ThenumberofaliasesobservedforrouterswithinthemappedISPs.tiers.Thisprovidesareferenceforestimatinghowmanyaliasesourtechniquemissed.OftheDNSpredictedaliases,forSprint,240backboneandgatewayrouterswerecorrectlyresolved.63routersdidnotresolvecorrectly:30oftheseroutershadatleastonein-terfaceaddressthatneverresponded.Wecorrectlyresolved119of139Ebonerouters,5ofwhichfailedfromunresponsiveaddresses.Thissuggeststhataproblemforeventhemosteffectivealiasre-solverishowtohandleunresponsiveIPaddresses.Outof56,000IPaddressesinourmaps,wefoundnearly6,000thatneverrespondedtoouraliasresolutionqueries.Weplantoinvestigatewhytherewere30Sprintand15Eboneroutersthatwereresponsive,butwerenotcompletelyandcorrectlyresolved.Potentialcausesincludeimplementationawsinouraliasresolutiontool,routersthatweretemporarilyunresponsive,consideringstaleDNSentriesauthoritative,androuterswithmul-tipleIPstacks(andthustwoIPidentiercounters).Figure14plotsaCDFofhowmanyaliaseswesawforrouterswithintheISPswemapped.ItshowsthatwesawonlyoneIPfor70%oftherouters,and2IPsforanother10%ofthem.Themaximumnumberofaliasesobservedwas24,foranAT&TrouterinNewYork.ThisgraphisanunderestimateofthenumberofaliasesroutershavesinceitislikelythatwedonotseeallpossibleIPsforarouter.7.ANALYSISTodemonstratetheutilityofourmaps,wealsoincludeinthispaperapreliminaryanalysisoftheirproperties.Wereportonrouteroutdegreedistributions,repeatingtheanalysisin[7]overourmoredetaileddata,andpresentPOPandpeeringstatistics,notpreviouslyreported,asmaybeusefulforsynthetictopologygeneration.7.1POPsizesThedistributionofPOPsizesisshowninFigure15forLevel3,AT&T,andSprint,bothasacumulativefractionofthetotalnumberofPOPs(top)andthetotalnumberofrouters(bottom).Todeter-minePOPsize,wecountonlybackboneandaccessrouters,andexcludethecustomerandpeerrouters.Thedistributionsareallskewed,thoughtheirdetailsdiffer,withAT&ThavingboththelargestPOPsandmostskeweddistribution,andLevel3havingtheleastskewandvariationinPOPsize.MostoftheroutersarepresentintenlargestPOPsforallthreenetworks,andallthreeISPshaveasignicantnumberofsmallPOPs.Forexample,morethan60%ofSprintPOPshavefewerthan15routersandthesePOPsaccountforlessthan20%ofallSprintrouters. Figure15:ThedistributionofPOPsizeforAT&T,Sprint,andLevel3byPOPs(top)andbyrouters(bottom)ThelargestPOPsareChicagoandNewYorkforAT&T,FortWorthandChicagoforSprint,andNewYorkandSanJoseforLevel3.ThesmallestPOPsforthesenetworksareinEurope,main-tainedbytheISPsfortrans-continentalconnectivity,orsmallercitiesintheUnitedStates.SmallPOPsmaybecalledbyothernameswithintheISP;wedonotdistinguishbetweenbackbonenet-worknodes,datacentersorprivatepeeringpoints.7.2RouterDegreeDistributionTodescribethedistributionofrouteroutdegreeintheISPnet-worksweusethecomplementarycumulativedistributionfunction(ccdf).Thisplotstheprobabilitythattheobservedvaluesaregreaterthantheordinate.Weconsiderallrouters,regardlessoftheirroleintheISP.Thecomplementarycumulativedistributionfunction(ccdf)ofrouterisshownintheaggregateoverallISPsinFigure16andforindividualISPsinFigure18(attheendofthepaper).WetthetailsofthesedistributionsusingPareto(“power-law”),Weibull,andlognormaldistributions.The parameterfortheParetotisestimatedovertherighthalfofthegraphtofocusonthetailofthedistribution.TheWeibullscaleandshapeparametersareestimatedusingalinearregressionoveraWeibullplot.Thelognormallineisbasedonthemeanandvarianceofthelogofthedistribution. Figure16:Routeroutdegreeccdf.TheParetotisonlyappliedtothetail.Weobservethat,unlikethemeasureddegreeinASgraphs,routeroutdegreehasasmallrangeinourdata;itcoversonlytwoor-dersofmagnitudeoverthetenISPs.Thisisdespitethefactthat,whilephysicalsizeandpowerconstraintsnaturallylimittheunder-lyingrouteroutdegree,ourdatacanincludeundetectedlayertwoswitchesthatwouldinatetheobservedrouteroutdegree,perhapssubstantially.TheccdfsofrouteroutdegreefromdifferentISPsaretbestbydifferentdistributions:sometWeibull,afewtthesimplerlog-normal,andmosthavetailsthatareconsistentwiththeParetot.ItappearsthatWeibulloftentsbothtailandbodyofthedistribution.Althoughthesedistributionshavesignicanttails,the parameterishighforaheavy-taileddistribution.7.3PeeringStructureSinceourmapsarecollectedusingtraceroutesthatenterandexitthemappedISPfromvariousingressesandegresses,theygiveustheuniqueopportunitytostudythepeeringstructurebetweenASes.FrompathsinBGPtables,onecanonlyinferfromanadja-cencybetweentwoASesintheAS-levelgraphthatthetwoASesconnectsomewhere.However,usingRocketfueltopologies,wecaninferwhereandinhowmanyplacesthesetwoASesexchangetrafc.Forexample,whileBGPtablesexposethefactthatSprintandAT&Tpeer,theydonotshowthedifferentlocationsatwhichthetwoISPsexchangetrafc.Bypeeringstructure,werefertothisimportantlevelofdetailnotpresentintheAS-levelgraphscol-lectedfromBGPtables.WesummarizethepeeringstructurebyshowingthenumberoflocationsatwhichthemappedISPexchangestrafcwitheachotherAS.TheotherASesmayrepresentotherISPs,whetherinatransitorpeerrelationship,aswellascustomersrunningBGP,e.g.,formultihoming.Weusethesameccdfplotstyleforsimplicity.Figure17plotsthisccdf,aggregatedoverthemappedISPs,whileFigure19(attheendofthepaper)showsplotsforindividualISPs.BothguresincludePareto,lognormalandWeibulltscalculatedasbefore.WeseethatthedataishighlyskewedforalloftheISPs.EachISPislikelytopeerwidelywithafewotherISPs,andtopeerinonlyafewplaceswithmanyotherISPs.TheserelationshipsareperhapsnotsurprisinggiventhatthedistributionofASsizeandASdegreeareheavytailed[24]. Figure17:Accdfofthenumberofrouter-leveladjacenciesseenforeachAS-leveladjacency.ASadjacenciesincludebothpeer-ingswithotherISPsandpeeringswithcustomersthatmanagetheirownAS.Wealsoseethatthedatahasasmallrange,coveringonlyonetotwoordersofmagnitude.Someofthe“peers”withmanyrouter-leveladjacenciesareactuallydifferentASeswithinthesameorga-nization:AS7018peerswithAS2386in69locationsandwithAS5074in45locations,butallthreerepresentAT&T.Discount-ingtheseoutliers,thegraphsshowthatitisrareforISPstopeerinmorethanthirtylocations.8.RELATEDWORKAnearlyattempt[15]toinferarouter-levelmapoftheInter-netstartedwithalistof5000destinations,andusedtraceroutesfromasinglenetworknode.Mercator[8]isalsoamapcollectiontoolrunfromasinglehost.Insteadofalistofhosts,itusesin-formedrandomaddressprobingtonddestinations.Boththeseef-fortsexploretheuseofsource-routingtodiscovercross-linkstoim-provethequalityofthenetworkmap.BurchandCheswick[3]useBGPtablestonddestinationprexes.Theysourcetheirtracer-outefromasinglemachine,butimprovetheircoveragebyusingtunnelstosomeothermachinesonthenetwork,similarineffecttosource-routing.Skitter[6]usesBGPtablesandadatabaseofWebservers.Skittermonitorsprobethenetworkfromabout20differentlocationsworldwide.Ourmappingtechniquediffersfun-damentallyfromalloftheseefforts.InsteadoftryingtocollecttherouterlevelmapofthewholeInternet,wedofocusedprobingofanISPtorecoveritsmap.TheresultisanISPmapthatismorecompletethanthatobtainedbyothermappingefforts.In[1],theauthorsanalyzethemarginalutilityofaddingvantagepointsanddestinationstodiscovertheInternetbackbonetopology.Ourworkissimilarinthatwealsotrytominimizethenumberofmeasurementsneeded,butweuseroutingknowledgeinsteadofreducingthenumberofvantagepoints.9.CONCLUSIONSANDFUTUREWORKInthispaper,wehavepresentednewtechniquesformappingtherouter-leveltopologyoffocusedportionsoftheInternet,suchasanISP,usingonlyend-to-endmeasurements.Wehaveshownthatroutinginformationcanbeexploitedinseveralwaystoselectonlythosetraceroutesthatareexpectedtobeuseful.Theresultis thatweareabletoreducethemappingworkloadbythreeordersofmagnitudecomparedtoabrute-forceall-to-allapproachwhilelosinglittleinthewayofaccuracy.Thisinturnenabledustousethemorethan750publictracerouteserversforourmeasurementsources,providinguswithmanymorevantagepointsthanothermappingefforts.Wehavealsopresentedanewaliasresolutiontechniquethatdiscoveredthreetimesmorealiasesthanthecurrentapproachbasedonreturnaddresses.Thisincreasestheaccuracyofourmapscomparedtoearlierefforts.WeusedournewtechniquestomaptendiverseISPs,andarere-leasingboththecompositemapsandrawdatatothecommunityat[20].Toevaluatethemaps,wecomparedthemwithi)thetruemapasunderstoodbytheISPoperators;ii)thetotalnumberofroutersfoundbyscanningsampledsubnets;iii)thepeeringsknowntoex-istfromBGPtables;andiv)previousmapsextractedfromSkit-ter.Ourmapsstackupwellinthesecomparisons.TheycontainroughlyseventimesasmanynodesandlinksintheareaoffocusasSkitter,andaresufcientlycompletebytheothermetricsthatwebelievetheyarerepresentativemodelsforISPnetworks.Theabovenotwithstanding,itiscleartousthatthisworkhasonlyscratchedthesurfaceofISPmapconstructionandanalysis.Itcanbereadilybeextendedinseveraldimensions.First,thedatawearereleasingcanbeusedtostudypropertiesofInternettopology.WereportednewresultsforthedistributionofPOPsizesandthenumberoftimesthatanISPconnectswithothernetworks,nd-ingthatbothdistributionshavesignicanttailsthoughsamplesaresmall.Second,wecanextractfurtherpropertiesfromthetracer-outesthatcanbeusedtoannotatethemapsandimprovetheiruse-fulness.Asanexample,wehaverecentlydevisedamethodforinferringapproximatelinkweightstocharacterizetheroutesthataretakenovertheunderlyingtopology.Finally,itseemslikelythatwecanimproveourbasictechniquesfurther,perhapssubstantially,leadingtomappingthatisbothmoreefcientandmoreaccurate.10.ACKNOWLEDGEMENTSWethankTomAndersonforhisguidance.Wearegratefultotheadministratorsofthetracerouteserverswhosepublicserviceenabledourwork,andoperatorswhoprovidedfeedbackonthequalityofourmaps.WethankLakshminarayananSubramanianforscriptsfrom[14],andCAIDAforskitterdata.RameshGovin-danprovidedindependentvericationofouraliasresolutiontech-nique,andhelpfulmappingadvice.WealsothankSteveBellovin,ChristopheDiot,andRandyBushforearlyinsightsintoISPback-boneandPOPtopologies.AllenDowneyprovidedlognormaldis-tributionanalysistoolsandguidance.WalterWillingerprovidedhelpfulfeedbackontheimplicationsofouranalysisresults.ThisworkwassupportedbyDARPAundergrantno.F30602-00-2-0565.11.REFERENCES[1]P.Barford,A.Bestavros,J.Byers,andM.Crovella.OntheMarginalUtilityofNetworkTopologyMeasurements.InACMSIGCOMMInternetMeasurementWorkshop,November2001.[2]A.BasuandJ.Riecke.StabilityissuesinOSPFrouting.InACMSIGCOMM,August2001.[3]H.BurchandB.Cheswick.MappingtheInternet.IEEEComputer,32(4):97–98,102,1999.[4]R.Bush.Privatecommunication,November2001.[5]H.Chang,R.Govindan,S.Jamin,S.Shenker,andW.Willinger.OnInferringAS-LevelConnectivityfromBGPRoutingTables.TechnicalReportUM-CSE-TR-454-02,2002.http://topology.eecs.umich.edu/.[6]k.claffy,T.E.Monk,andD.McRobb.Internettomography.InNature,January1999.[7]M.Faloutsos,P.Faloutsos,andC.Faloutsos.Onpower-lawrelationshipsoftheInternettopology.InACMSIGCOMM,1999.[8]R.GovindanandH.Tangmunarunkit.HeuristicsforInternetmapdiscovery.InIEEEINFOCOM,2000.[9]T.Kernen.traceroute.org.http://www.traceroute.org.[10]C.Labovitz,A.Ahuja,A.Bose,andF.Jahanian.DelayedInternetRoutingConvergence.InACMSIGCOMM,September2000.[11]R.Mahajan,S.M.Bellovin,S.Floyd,J.Ioannidis,V.Paxson,andS.Shenker.Controllinghigh-bandwidthaggregatesinthenetwork(extendedversion).http://www.aciri.org/pushback/,July2001.[12]A.Medina,I.Matta,andJ.Byers.BRITE:AexiblegeneratorofInternettoplogies.TechnicalReportBU-CS-TR-2000-005,BostonUniversity,2000.[13]D.Meyer.RouteViewsProject.http://www.routeviews.org.[14]V.N.PadmanabhanandL.Subramanian.AninvestigationofgeographicmappingtechniquesforInternethosts.InACMSIGCOMM,August2001.[15]J.PansiotandD.Grad.OnroutesandmulticasttreesintheInternet.InACMSIGCOMMComputerCommunicationReview,pages41–50,1997.[16]K.ParkandH.Lee.Ontheeffectivenessofroute-basedpacketlteringfordistributedDoSattackpreventioninpower-lawinternets.InACMSIGCOMM,August2001.[17]G.Philips,S.Shenker,andH.Tangmunarunkit.Scalingofmulticasttrees:CommentsontheChuang-Sirbuscalinglaw.InACMSIGCOMM,August1999.[18]P.Radoslavov,H.Tangmunarunkit,H.Yu,R.Govindan,S.Shenker,andD.Estrin.Oncharacterizingnetworktopologiesandanalyzingtheirimpactonprotocoldesign.TechnicalReportCS-00-731,USC,2000.[19]R.Rivest.TheMD5message-digestalgorithm,1992.NetworkingWorkingGroupRequestsforComment,MITLaboratoryforComputerScienceandRSADataSecurity,Inc.,RFC-1321.[20]Rocketfuelmapsanddata.http://www.cs.washington.edu/research/networking/rocketfuel/.[21]S.Savage,D.Wetherall,A.Karlin,andT.Anderson.PracticalnetworksupportforIPtraceback.InACMSIGCOMM,August2000.[22]A.C.Snoeren,C.Partridge,L.A.Sanchez,C.E.Jones,F.Tchakountio,S.T.Kent,andW.T.Strayer.Hash-basedIPtraceback.InACMSIGCOMM,August2001.[23]R.Sterner.ColorlandformatlasoftheUnitedStates.http://fermi.jhuapl.edu/states/.[24]H.Tangmunarunkit,J.Doyle,R.Govindan,S.Jamin,S.Shenker,andW.Willinger.DoesASSizeDetermineDegreeinASTopology?InACMComputerCommunicationReview.October2001.[25]H.Tangmunarunkit,R.Govindan,S.Jamin,S.Shenker,andW.Willinger.Networktopologygenerators:Degree-basedvsstructural.InACMSIGCOMM,August2002.[26]E.W.Zegura,K.Calvert,andS.Bhattacharjee.Howtomodelaninternetwork.InProceedingsofIEEEINFOCOM,1996.