IEEE JOURNAL ON SELECTED AREAS IN COMMUNICATIONS VOL - PDF document

IEEEJOURNALONSELECTEDAREASINCOMMUNICATIONS,VOL.21,NO.6,AUGUST2003879EvaluationandCharacterizationofAvailableBandwidthProbingTechniquesNingningHu,StudentMember,IEEE,andPeterSteenkiste,SeniorMember,IEEEThepacketpairmechanismhasbeenshowntobeareliablemethodtomeasurethebottlenecklinkcapacityonanetworkpath,butitsuseformeasuringavailablebandwidthismorechallenging.Inthispaper,weusemodeling,measurements,andsimulationstobettercharacterizetheinteractionbetweenprobingpacketsandthecompetingnetworktraffic.Wefirstconstructasimplemodeltounderstandhowcompetingtrafficchangestheprobingpacketgapforasingle-hopnetwork.Thegapmodelshowsthattheinitialprobinggapisacriticalparameterwhenusingpacketpairstoestimateavailablebandwidth.Basedonthisinsight,wepresenttwoavailablebandwidthmeasurementtechniques,theinitialgapincreasing(IGI)methodandthepackettransmissionrate(PTR)method.WeuseextensiveInternetmeasurementstoshowthatthesetechniquesestimateavailablebandwidthfasterthanexistingtechniquessuchasPathload,withcomparableaccuracy.Finally,usingbothInternetmeasurementssimulations,weexplorehowthemeasurementaccuracyofactiveprobingisaffectedbyfactorssuchastheprobingpacketsize,thelengthofprobingpackettrain,andthecompetingtrafficonlinksotherthanthetightlink.IndexTerms—Activeprobing,availablebandwidth,Internet,networkmeasurement.I.INTRODUCTIONHARACTERIZINGtheend-to-endnetworkavailablebandwidthisaproblemthatisbothintellectuallyin-triguingandofpracticalimportance.However,thescaleoftheInternet,trafficvolumeandthediversityofnetworktechnologiesmakeitaverychallengingtask.Furthermore,regularInternetusersdonothaveaccesstonetworkinternals,addingtothecomplexityofunderstanding,characterizing,andmodelingtheperformanceoftheInternet.Whiletheproblemsofcharacterizingend-to-endlatencyandbottlenecklinkca-pacityhavereceivedalotofattention[1]–[6],thequestionthatisofmostinteresttoapplicationsishowmuchbandwidthisavailabletothemalonganend-to-endInternetpath.Whataregoodtechniquesforestimatingavailablebandwidthandwhatfactorsaffectthemeasurementaccuracyarestillopenquestions.Thosequestionsarethefocusofthispaper.Networkmeasurementtechniquescanbeclassifiedintotwocategories:passivemeasurement[7],[8]andactiveprobingManuscriptreceivedAugust19,2002;revisedFebrury26,2003.ThispaperwassupportedinpartbytheDefenseAdvancedResearchProjectAgencyunderContractF30602-99-1-0518,monitoredbyAFRL/IFGA,Rome,NY,andunderContractF30602-96-1-0287,monitoredbyRomeLaboratory,AirForceMa-terielCommand,USAF.N.HuiswiththeComputerScienceDepartment,CarnegieMellonUniversity,Pittsburgh,PA15213-3890USA(e-mail:hnn+@cs.cmu.edu). links .Theircapacitiesare ,andthetrafficloadsontheselinksare .Wedefinethebottlenecklink ,where tightlinkisdefinedas ,where Inthefirstpartofthispaper,weassumethatthetightlinkisthebottlenecklink;weconsiderthecasewherethetwoaredif-ferentinSectionVIII.Theunusedbandwidthonthetightlink, ,iscalledtheavailablebandwidthofthepath.Theavailablebandwidthdefinedhere,generally,doesnotequaltheachievablebandwidthforanapplication.Applicationsoftencannotfullyutilizetheunusedbandwidthduetofactorssuchasasmallreceivesocketbufferandpacketreordering,whichmaylimittransmissioncontrolprotocol(TCP)throughput.Thispapermakesthefollowingfourcontributions.First,wedevelopasingle-hopgapmodelthatcapturestherelationshipbetweenthecompetingtrafficthroughputandthechangeofthepacketpairgapforasingle-hopnetwork.Weusethisgapmodeltohelpunderstandtheinteractionbetweentheprobingpacketsandthecompetingtraffic,andtoidentifytheconditionsunderwhichthepacketpairgapcanbeusedtoaccuratelycharacterizethecompetingtraffic.Second,basedontheinsightsgainedfromthegapmodel,wedeveloptwopacketpairtechniques—initialgapincreasing(IGI)andpackettransmissionrate(PTR)—tocharacterizetheavail-ablebandwidthonanetworkpath.Thetwotechniquesexper-imentallydetermineaninitialpacketpairgapthatwillyieldahighcorrelationbetweenthecompetingtrafficthroughputonthebottlenecklinkandthepacketgapatthedestination.By0733-8716/03$17.00©2003IEEE 880IEEEJOURNALONSELECTEDAREASINCOMMUNICATIONS,VOL.21,NO.6,AUGUST2003comparingtheavailablebandwidthandTCPthroughput,weshowthattherelativemeasurementerror,intermsofTCPper-formance,isgenerallylessthan30%.WealsoshowthatthemeasurementtechniquesdiscussedinthispaperaremuchfasterthanexistingmethodssuchasPathload[9],[10],withcompa-rablemeasurementaccuracy.Third,weexplorehowpackettrainparameterscanaffectthemeasurementaccuracy.UsingInternetmeasurements,weshowthataprobingpacketsizearound700Byteresultsinthebestaccuracy.Wealsoshowthatthelengthoftheprobingpackettrainshouldbeadjustedbasedontheburstinessofthecom-petingtraffic.Furthermore,westudythepotentialofIGIandPTRtodetecttherelativeburstinessofInternetbackgroundtraffic,whichisanotherimportantmetricthatmaybeofinteresttoapplications.Finally,weusesimulationstoquantifyhowvariousfactorsimpacttheaccuracyofthealgorithmsinmultihopnetworks.Specifically,welookatnetworkpathswherethetightlinkisnotthebottlenecklink,andpathswherelinksotherthanthetightlinkcarrysignificantamountoftraffic.Weshowthatwhiletheseeffectscanreducetheaccuracyofthealgorithms,theirimpactislikelytobeminimalinthecurrentInternet.Thispaperisorganizedasfollows.WefirstdiscusstherelatedworkinSectionII.InSectionIII,weintroducethegapmodel.TheIGIandPTRalgorithmsareintroducedinSec-tionIV.WepresentourperformanceevaluationmethodologyinSectionV.Theevaluationincludesthreeparts:SectionVIstudiestheperformancepropertiesofIGIandPTR,whichincludeacomparisonwithTCPthroughputandPathloadmea-surement.SectionVIIstudiestheimpactoftwoIGIandPTRparameters—theprobingpacketsizeandtheprobingpackettrainlength—ontheaccuracyofthealgorithms.SectionVIIIusessimulationtoquantifypossiblesourcesoferrorforIGIandPTRinmultihopnetwork.WeconcludeinSectionIX.II.RELATEDTheproblemofestimatingthebottlenecklinkbandwidthusingactiveprobingiswellstudied.Theworkin[11]classifiesthetoolsintosinglepacketmethodsandpacketpairmethods.Singlepacketmethodsestimatethelinkcapacitybymeasuringthetimedifferencebetweentheround-triptime(RTT)tooneendofanindividuallinkandthattotheotherendofthesamelink.Thisrequiresalargenumbersofprobingpacketstofilterouttheeffectofotherfactorssuchasqueueingdelay.Singlepackettoolsincludepathchar[4],clink[3],andpchar[6].Packetpairmethodssendgroupsofback-to-backpackets,i.e.,packetpairs,toaserverwhichechosthembacktothesender.AspointedoutinanearlierstudyonTCPdynamics[12],thespacingbetweenpacketpairsisdeterminedbythebottlenecklinkandispreservedbythelinkswithhigherbandwidth.ExampletoolsincludeNetDynprobes[13],packetpairs[14],bprobe[5],[15],andnettimer[1].Mostofthesetoolsusestatisticalmethodstoestimatethebandwidth,basedontheassumptionthatthemostcommonvalueforthepacketpairgapcapturesthebottlenecklinktransmissiondelay.Inpractice,interpretingthepacketpairmeasurementsisdifficult[16].Recentworkonpathrate[2]addressesthesechallenges Fig.1.Interleavingofcompetingtrafficandprobingpackets. istheinitialgap. istheprobingpacketlengthontheoutputlink. isthegapafterinterleavingwiththecompetingtraffic. isthecompetingtrafficthroughput.Also,refertoFig.2forthesymbols’definition.byexplicitlyanalyzingthemultimodalnatureofthepacketgapdistribution.Characterizingtheavailablebandwidthismoredifficultsinceitisadynamicpropertyanddependsonmorefactors.Becauseofthedynamicnatureoftheavailablebandwidth,itmustbeaveragedoveratimeinterval.Therefore,activemeasurementtechniquesoftenusepackettrains,i.e.,longersequencesofpackets.Atypicalexampleisthepacketbunchmode(PBM)method[16].Itextendsthepacketpairtechniquebyusingdifferent-sizedgroupsofback-to-backpackets.Ifroutersinthenetworkimplementfairqueueing,thebandwidthindicatedbytheback-to-backpacketprobesisanaccurateestimateforthe“fairshare”ofthebottlenecklink’sbandwidth[14].Anotherexample,cprobe[5],sendsashortsequenceofechopacketsbetweentwohosts.Byassumingthat“almost-fair”queueingoccursduringtheshortpacketsequence,cprobeprovidesanestimatefortheavailablebandwidthalongthepathbetweenthehosts.Treno[17]usesTCP-likeflowcontrolandcongestioncontrolalgorithmstoestimateavailablebandwidth.Theworkin[2]mentionsatechniqueforestimatingtheavailableband-widthbasedontheasymptoticdispersionrate(ADR)method.PartofourworkisrelatedtoADR,andwesharetheviewthattheADRreflectstheeffectofallthecompetingsourcesalongthetransmissionpath.However,wealsoidentifytheinitialprobingpacketgapasacriticalparameterthatmustbeselecteddynamicallyinordertoachievegoodaccuracy.Pathload[9],[10]characterizestherelationshipbetweenprobingrateandavailablebandwidthbymeasuringtheonewaydelayofprobingpackets.Bytryingdifferentprobingrates,areasonableestimatefortheavailablebandwidthcanbefound.TheworkclosesttooursistheTOPPmethod[18].Thismethodprovidesatheoreticalmodelfortherelationshipbetweenavailablebandwidthandprobingpacketspacingatbothendpoints.Simulationsareusedtovalidatethemethod.Bothofthesemethodsanalyzetherelationshipbetweenprobingtrainsandavailablebandwidth,buttheiranalysisdoesnotcapturethefine-graininteractionsbetweenprobesandcompetingtraffic.Thisisuseful,forexample,tounderstandthelimitationsoftheIII.STheideabehindusingpacketpairstomeasureavailableband-widthistohavetheprobinghostsendapairofpacketsinquicksuccessionandtomeasurehowthepacketpairgapischanged(Fig.1).Astheprobingpacketstravelthroughthenetwork,packetsbelongingtothecompetingtrafficmaybeinsertedbe-tweenthem,thusincreasingthegap.Asaresult,thegapvalueatthedestinationmaybeafunctionofthecompetingtrafficrate,makingitpossibletoestimatetheamountofcompetingtraffic. HUANDSTEENKISTE:EVALUATIONANDCHARACTERIZATIONOFAVAILABLEBANDWIDTHPROBINGTECHNIQUES881 Fig.2.Single-hopgapmodel.Theoutputgap isnotaffectedby intheDQR,whileintheJQR, isproportionalto .(Transmissiondelayisdefinedasthetimeforapackettobeplacedonalinkbyasender.)Inpractice,thewaythatthecompetingtrafficaffectsthepacketpairgapismuchmorecomplexthanwhatissuggestedabove.Inthissection,wedescribeandevaluateasimplemodelthatcapturesmoreaccuratelytherelationshipbetweenthegapvalueandthecompetingtrafficloadonasingle-hopnetwork.A.Single-HopGapModelThethree-dimensional(3-D)graphinFig.2showstheoutputgapvalue asafunctionofthequeuesize andthecompetingtrafficthroughput .Thismodelassumesthattheroutersusefirst-infirst-out(FIFO)queueingandthatallprobingpacketshavethesamesize.ItalsoassumesthatthecompetingtrafficisconstantintheintervalbetweenthearrivalofpacketP1andP2;giventhatthisintervalisontheorderof1ms,thisisareasonableassumption.Themodelhastworegions.Asdescribedbelow,thekeydif-ferencebetweenthesetworegionsiswhetherornotthetwopacketsP1andP2fallinthesamequeueingperiod.Aisdefinedtobethetimesegmentduringwhichthequeueisnotempty,i.e.,twoconsecutivequeueingperiodsaresepa-ratedbyatimesegmentinwhichthequeueisempty.Forthisreason,wecallthetworegionsinthemodelthedisjointqueuingregion(DQR)andthejointqueuingregion(JQR).IfthequeuebecomesemptyafterP1leavestherouterandbe-foreP2arrives,then,sinceweareassumingthat isconstantinthis(short)interval,P2willfindanemptyqueue.ThismeansthatthetheoutputgapwillbetheinitialgapminusthequeueingdelayforP1,i.e., UnderwhatconditionswillthequeuebeemptywhenP2ar-rives?BeforeP2arrives,therouterneedstofinishthreetasks:processingthequeue ( ),processingP1( ),andpro-cessingthecompetingtrafficthatarrivesbetweentheprobingpackets( ).Therouterhas timetocompletethesethreeoperations,sotheconditionis ,whichcorrespondstothetriangularDQRinFig.2.Inthisregion,theoutputgap isindependentofthecom-petingtrafficthroughput .Wewillrefertotheabove(1)astheDQRequation.Underalltheotherconditions,i.e.,inJQR,whenP2arrivesattherouter,thequeuewillnotbeempty.Sinceweassume isconstant,thismeansthatP1andP2areinthesamequeueingperiod.Theoutputgapconsistsoftwotimesegments:thetimetoprocessP1( ),andthetimetoprocessthecompetingtrafficthatarrivesbetweenthetwoprobingpackets( Therefore,inthisregion,theoutputgapwillbe Thatis,inthisregion,theoutputgap increaseslinearlywiththecompetingtrafficthroughput .Equation(2)isreferredtoastheJQRequation.Thismodelclearlyidentifiesthechallengeinusingpacketpairsforestimatingthecompetingtrafficthroughput.IfthepacketpairhappenstooperateintheDQRofthebottleneckrouter,theoutputgapwillbearnorelationshipwiththecom-petingtraffic,andusingtheJQRequation(sincetheuserdoesnotknowwhichregionapplies)willyieldanincorrectresult.Furthermore,theestimateobtainedusingasinglepacketpairwillonlyprovidetheaveragecompetingtrafficover ,whichisaveryshortperiod.Sincethecompetingtrafficislikelytofluctuate,oneingeneralwillwanttoaveragetheresultsofmultiplesamples,correspondingtoindependentpacketpairs.ThisofcourseincreasesthechancethatsomeofthesampleswillfallintheDQR.B.ProbingPacketTrainsEquation(2)showsthatintheJQR,wecanestimatethecompetingtrafficthroughput basedontheinitialgap theoutputgap ,andthebottleneckgap .However,thesingle-hopgapmodelassumesthatthecompetingtrafficisasmoothpacketstream.Inpractice,thecompetingtrafficflowwillbeburstyandasinglepairofprobingpacketswillnotcapturetheaveragethroughputofthecompetingtraffic.Todealwiththisproblem,peopleuseapackettrain[2],[16],i.e.,alongersequenceofevenlyspacedpackets.Theconclusionsfromthesingle-hopgapmodeldonotdi-rectlyapplytoapackettrain.Themainproblemisthatthe“pairs”thatmakeupapackettrainarenotindependent.Forex-ample,ifonepacketpairinthetraincapturesaburstofpackets 882IEEEJOURNALONSELECTEDAREASINCOMMUNICATIONS,VOL.21,NO.6,AUGUST2003fromthecompetingflow,itishighlylikelythatadjacentpairswillnotseeanycompetingtrafficandwill,thus,seeadecreaseintheirpacketgap.Intuitively,ifwewanttoestimatetheamountofcompetingtraffic,weshouldfocusontheincreasedgapsinaprobingpackettrainsincetheycapturethecompetingtraffic,whiledecreasedgapssawlittleornocompetingtraffic.NotethatthisobservationonlyapplieswhentheprobingpackettrainoperatesintheJQR.Moreprecisely,assumeaprobingtraininwhich gapsareincreased, areunchanged,and aredecreased.Ifwenowapply(2)toalltheincreasedgaps,wegetthefollowingestimateforthecompetingtrafficload: Here,thegapvalues , ,and denotethegapsthatareincreased,unchanged,anddecreased,respectively.Inthisformula, istheamountofcompetingtrafficthatarriveatrouterR1duringtheprobingperiod.Ideally, isthetotalprobingtime.Inpractice,weexcludegapvaluesthatinvolvelostorreorderedpackets,soinsuchcases,thedenominatormaybesmallerthanthetotalprobingtime.ThismethodofcalculatingcompetingtrafficloadwillbeusedbytheIGIalgorithminSectionIV,andwecallittheIGIformula.Anumberofgroupshaveproposedmethodstoestimatetheavailablebandwidthalonganetworkpath[5],[9],[10].Usingthesamenotationasusedabove,theequationusedin[5]is (4)Here, istheprobingpacketsize.Thisformularepresentstheaveragetransmissionrateofthepackettrain,measuredatthedestination.WewillalsousethisformulainthePTRalgorithmdescribedinSectionIV,andwecallitthePTRformula.Incon-trast,Pathload[9],[10]usestherateofthepackettrainssentbythesource.ThegapmodelshowsthattheIGIformulaonlyappliesintheJQR,andwewillshowlaterthatthePTRformulaisalsoonlyvalidundersimilarconditions.Notethatthemainparameterthatisunderourcontrolinthesingle-hopgapmodelis .IthasalargeimpactonthesizeoftheDQRand,thus,ontheregioninwhichthepackettrainoperates.Therefore,thekeytoanaccurateavailablebandwidthmeasurementalgorithmistofinda valuesothattheprobingpackettrainoperatesintheBeforediscussingthedetailsofthealgorithmsusedtoachievethat,wefirstuseseveralsimpletestbedexperimentstoillustratetheintuitionbehindthesingle-hopgapmodelandtoshowhowtheDQRsandJQRsaffecttheestimatesoftheIGIandPTR Fig.3.Testbedconfiguration. Fig.4.EffectofJQR.InitialandoutputgapforroutersR1(top)andR2C.TestbedIllustrationWerunexperimentsonanisolatedtestbed.ThetopologyisshowninFig.3.Inthisfigure,PsandPdaretheprobingsourceanddestination,andCsandCdareusedtogeneratecompetingtraffic.R1andR2areFreeBSD-basedroutersthatrunonallrelevantinterfacestorecordpackettimestampinforma-tion.Pssendsoutaseriesofevenlyspaced100-Bpackets,eachconsecutivepairofwhichcanserveasaprobingpair.Thecom-petingtrafficisgeneratedusingIperf[19],whichallowsustosimulatetypicalTCPtrafficsuchasFTPtraffic.WecontrolthecompetingtrafficthroughputbyadjustingtheTCPwindowsize.1)EffectofJQR:CapturingCompetingTraffic:Inthisex-periment,weuse1024probingpacketsofsize100Byte,sothebottleneckgapis0.08ms.Theinitialgapissetto0.31ms,andweuseacompetingtrafficloadof7.2Mb/s.AtypicalsetofexperimentalresultsisshowninFig.4:thetopgraphshowstheinitialandoutputgapsmeasuredonR1,andthebottomgraphshowsthecorrespondinggapsonR2.TheincreaseingapvaluesonR1iscausedbycompetingtrafficonthebottlenecklink.TheincreasedgapvaluesinthetopgraphofFig.4fallintothreeclusters:1.2(thetransmissiondelayofa1500Bytecom-petingpacketona10Mb/slink),2.3,and3.8ms.Theclusterscorrespondtoprobingpairsthatareseparatedbyexactlyone,two,andthreecompetingpackets.ThefactthatatmostthreepacketsareinsertedinaprobinggapshouldnotbeasurpriseWeusesuchalargenumberinordertogetalargeenoughprobingperiod.Onourtestbed,itdoesnotcausepacketdropsanditdoesnotsignificantlyaffectthecompetingflow’sthroughput. HUANDSTEENKISTE:EVALUATIONANDCHARACTERIZATIONOFAVAILABLEBANDWIDTHPROBINGTECHNIQUES883 Fig.5.BurstsofcompetingpacketsattheinputandoutputinterfaceofR1. isthetransmissiondelayofthethreecompetingpacketsonR1’sinputlink, +t istheirtransmissiondelayontheoutputlink, istheintervalbetweenthetimewhenthefirstthreepacketsfinishtransmission,andthetimewhenthesecondthreepacketsarrive.sincetheinitialgapis0.31ms,andthetransmissiontimeofthecompetingpacketsontheirinputlinkis0.12ms(notethattheinputlinksaretentimesfasterthanthebottlenecklink).Besidestheincreasedgapvalues,mostoftheothergapvaluesaredecreasedto0.08ms,whichisthetransmissiondelayoftheprobingpacketsonthe10Mb/sbottlenecklink.ThebottomgraphshowsthattheincreasedgapvaluesaremaintainedthroughrouterR2,becauseR2hasahigheroutputratethaninputrate.Thechangesinthegapvaluesarethedirectresultoftheburstycompetingtraffic.ThetraceonrouterR1showsthatinsomecases,thesourceCssendsoutthree1500Bytepacketsback-to-back( periodinFig.5).ThisbuildsupthequeueinrouterR1,andthequeuewilldrainduringtheperiod .Afterperiod ,morecompetingtrafficarrives.Apacketpairthatoverlapswithperiod willseeanincreasedgap;thegapvaluedependsonwhetherone,two,orthreecompetingpacketsareinsertedbetweenthepacketpair.Apacketpairthatfallsin willseeitsgapreducedto0.08ms.Inourexperiment,becausetheinputlinkcapacityistentimestheoutputlinkca-pacity, ismuchlongerthan ,somorepacketpairgapsarereducedthanincreased.Packetpairscanalsostraddlethe and periods.Inthatcase,thegapisreducedtoavaluebetween and (0.31ms).ThiseffectcorrespondstotheDQR,andinthisexample,itisnotverysignificant.UsingtheIGIformula,wecanobtainanestimatedcompetingtrafficthroughputof7.3Mb/s,andthePTRformulaestimatestheavailablebandwidthas2.4Mb/s.Bothestimatesareagoodmatch,giventhatIperfreportsanaveragethroughputof7.2Mb/s.2)EffectofDQR:LosingCompetingTraffic:Wenowre-ducethecompetingtrafficbysettingthesourceTCPsocketbuffersizeto512BytesandthedestinationTCPsocketbuffersizeto128Bytes.Thisforcesthecompetingtrafficsourcetosendroughlyone128Byte-packeteachRTT.Theparametersoftheprobingpackettrainarekeptthesame.Fig.6showsthattheincreasedgapvaluesarenolongerclusteredaroundasmallsetofdiscretevalues.WhenweapplytheIGIformulatothisexperiment,weobtainacompetingtrafficthroughputof3.8Mb/s,andPTRestimatestheavailablebandwidthas2.5Mb/s(correspondingto7.5Mb/scompetingtrafficthroughput).Botharehigherthantherealcompetingtrafficthroughputof1.4Mb/s.Toexplainthisresult,Fig.7showsadetailedsnapshotofthestartingandendingtimeoftwocompetingpackets(AandB)andsixprobingpackets(1–6)forboththeinputandoutputin-terfacesofrouterR1.Thelinesshowthetransmissiondelaysof Fig.6.EffectofDQR.Thechangesinthegapvaluesarerandom. Fig.7.Snapshotoftheinterleavingbetweentwocompetingpacketsandsixprobingpackets.thepackets.Giventhenatureofthecompetingtraffic,probingpacketswillalwaysencounteranemptyorveryshortqueue.Asaresult,itislikelythattwoconsecutiveprobingpacketswillfallindifferentqueueingperiods,andthechangesingapvaluesarefairlyrandomandnotstronglycorrelatedtothecompetingtrafficload.Insomecases,weseeagapincreasebecauseP2isdelayed,e.g.,thepair(2,3),whichhasitsgapincreasedto0.414ms.Inothercases,P1isdelayed,andweendupinregionDQR;anexampleisthepair(3,4).D.DiscussionThesingle-hopgapmodelandourexperimentsshowthechal-lengesassociatedwithusingpacketpairsandpackettrainstoestimatecompetingtrafficonthebottlenecklink.Towhatde-greethemeasuredgapatthedestinationreflectsthecompetingtrafficloaddependsonwhatregionthebottleneckrouterisop-eratingin.ThegoodnewsisthatwhenweareoperatingintheThetimestamprecordedbytcpdumpisthetimethelastbitofapacketpassesthroughthenetworkinterface.ThismeansthatforthesegmentsinFig.7,onlytherightendpointsaremeasuredtracedata.Theleftendpointsarecalculatedbasedonthepacketlengthandthecorrespondinginterface’stransmissionrate.Thesmalloverlapsbetween“3”and“A,”and“5”and“B”arenotpossibleandareprobablyduetothetimingerroroftcpdump. 884IEEEJOURNALONSELECTEDAREASINCOMMUNICATIONS,VOL.21,NO.6,AUGUST2003 Fig.8.Impactoftheinitialgaponavailablebandwidthmeasurements.Thearrowspointoutthemeasurementsattheturningpoint,thesmallestinitialgapwheretheaverageoutputgapequalstheaverageinitialgap.JQR,thereisaproportionalrelationshipbetweentheoutputgapandthecompetingtraffic.Thatisthestartingpointforthealgo-rithmsintroducedinthenextsection.IV.IGIPTRAInthissection,wedescribehowweusetheIGIandPTRformulasasthebasisfortwoavailablebandwidthestimationalgorithms.Themeasurementspresentedintheprevioussectionclearlyshowthattheinitialgap hasalargeimpactontheusefulnessoftheIGIandPTRformulas,sowefirststudytheroleoftheinitialgapmorecarefully.A.ImpactofInputGap Accordingtothesingle-hopgapmodel,ifweareintheJQR,theoutputgapofapacketpairortraincangiveusanestimateofthecompetingtrafficonthebottlenecklink.However,intheDQR,outputgapisindependentofthecompetingtraffic.WealsoseethatincreasingtheinitialgapwillincreasetheDQRarea.Thisarguesforusingsmallinitialgaps.Infact,if i.e.,iftheinitialgapissmallerthantheprobingpackettransmis-siondelayonthebottlenecklink,theDQRareadoesnotevenexist.However,withsmallinitialgaps,suchas ,wearefloodingthebottlenecklink,whichmaycausepacketlossesanddisrupttraffic.InordertobetterunderstandtheimpactoftheinitialprobinggapontheaccuracyoftheIGIandPTRformulas,wedesignthefollowingexperiment.WesendanIperfTCPcompetingtrafficflowof3.6Mb/sovera10-Mb/sbottlenecklink.Wethenprobethenetworkusingasetofpackettrains;thepackettrainlengthis256andtheprobingpacketsizeis750Byte.Westartwithaninitialprobinggapof0.022ms,whichisthesmallestgapthatwecangetonthetestbed,andgraduallyincreasetheinitialgap.Fig.8showstheaveragegapdifference(averagedoutputgapminustheaveragedinitialgap),thecompetingtrafficthroughputestimatedusingtheIGIformula,andtheavailablebandwidthestimatedusingthePTRformula.Weseethatforsmallinitialgaps(smallerthan whichisthetransmissiontimeonthebottlenecklink),wearefloodingthenetworkandthemeasurementsunderestimatethecompetingtrafficthroughput.Notethatforminimalinitialgaps,thePTRformulaissimilartotheformulausedtoestimatethebottlenecklinkcapacitybytoolssuchasbprobe[5],andinfact,thePTRestimateforsmallinitialgapsiscloseto10Mb/s,whichisthebottlenecklinkcapacity.Whentheinitialgapreaches ,theDQReffectstartstoap-pear.Notethat,unlessthenetworkisidle,wearestillfloodingthebottlenecklink.Sofar,theaverageoutputgapatthedesti-nationislargerthantheinitialgap.Whenwefurtherincreasetheinitialprobinggap,atsomepoint(0.84msinthefigure),theoutputgapequalstheinitialgap;wewillcallthisthe.Atthispoint,theprobingpacketsinterleavenicelywiththecompetingtraffic,andtheaveragerateofthepackettrainequalstheavailablebandwidthonthebottlenecklink.Inthisexperiment,theIGIestimateforthecompetingtrafficattheturningpointis3.2Mb/sandthePTRestimatefortheavail-ablebandwidthis7.1Mb/s;bothmatchtheactualcompetingtraffic(3.6Mb/s)quitewell.Aswecontinuetoincreasetheini-tialprobinggap,theoutputgapremainsequaltotheinitialgapsinceallthepacketsonaverageexperiencethesamedelay.Webelievethatthepointwheretheaverageoutputgapequalstotheinitialgap,i.e.,theturningpointshowninFig.8,isthecor-rectpointtomeasuretheavailablebandwidth.Theturningpointcorrespondstothesmallestinitialgapvaluewithwhichwearenotfloodingthebottlenecklink.Withrespecttothesingle-hopgapmodelinFig.2onwhichtheIGIformulaisbased,thisinitialgapwillresultinapackettrainthatkeepsthequeueasfullaspossiblewithoutoverflowingit;themodelshowsthatthisputsusintheJQR.WithrespecttothePTRformula,theinitialgapattheturningpointcorrespondstothepackettrans-missionratewherethepackettrainsconsumealltheavailablebandwidthwithoutsignificantinterferencewiththecompetingtraffic.Inotherwords,thepackettrainbehaveslikeanaggres-sive,butwellbehaved(i.e.,congestioncontrolled)applicationflow,soitsrateisagoodestimateoftheavailablebandwidth.TheIGIandPTRalgorithmsdiscussedbelowarebasedonpackettrainsthatoperateattheturningpoint.B.IGIandPTRAlgorithmsTheIGIandPTRalgorithmssendasequenceofpackettrainswithincreasinginitialgapfromthesourcetothedestinationhost.Theymonitorthedifferencebetweentheaveragesource(initial)anddestination(output)gapandtheyterminatewhenitbecomeszero.Atthatpoint,thepackettrainisoperatingattheturningpoint.WethenusetheIGIandPTRformulastocomputethefinalmeasurement.ThepseudocodefortheIGIalgorithmisshowninFig.9.Theavailablebandwidthisobtainedbysubtractingtheestimatedcompetingtrafficthroughputfromanestimateofthebottlenecklinkcapacity.Thebottlenecklinkcapacitycanbemeasuredusing,forexample,bprobe[5],nettimer[1],orpathrate[2].Notethaterrorsinthebottlenecklinkcapacitymeasurementwillaffecttheaccuracyoftheavailablebandwidthestimate,sincethebottlenecklinkcapacity isusedinthecalculationofthebottleneckgap ,thecompetingtrafficthroughput HUANDSTEENKISTE:EVALUATIONANDCHARACTERIZATIONOFAVAILABLEBANDWIDTHPROBINGTECHNIQUES885 Fig.9.IGI.SEND_PROBING_PACKETS()sendsoutprobe_numpacket_sizeprobingpacketswiththeinitialgapsettoGET_DST_GAPS()getsthedestination(output)gapvaluesandaddsthem;GET_INCREASED_GAPS()returnsthesumoftheinitialgapsthatarelargerthanthebottleneckgap;c_bw,b_bw,anddenotethecompetingtrafficthroughput,thebottlenecklinkcapacity,andtheavailablebandwidth,respectively. ,andtheavailablebandwidth .However,theanalysisoftheabovementionedtoolsandourexperienceshowthatthebottlenecklinkcapacitymeasurementisfairlyaccurate,sointhispaper,wedonotconsiderthisfactor.ThePTRalgorithmisalmostidenticaltotheIGIalgorithm.TheonlydifferenceisthatweneedtoreplacethelastthreelinesinFig.9by TheseformulasassumethatthereisnopacketlossorpacketInbothalgorithms,wetrytominimizethenumberofprobingphasesbycarefullyselectingthe and .InstepGET_GB(),wefirstprobeusingan thatisassmallaspossible.Thisallowsustoestimatethebottlenecklinkca-pacityand .Wethenset ,and .Anotherkeystepinbothalgorithmsistheautomaticdiscoveryoftheturningpoint.ThisisdoneintheprocedureGAP_EQUAL().Ittestswhetherthesourceanddestinationgapsare“equal,”whichisdefinedas Inourexperiments, issetto0.1.ThesetwostepsareakeydifferencebetweenPTRalgorithmandothertechniquesbasedon(4)sincetheyallowustoquicklyfindagoodinitialgap.WeevaluatehowfastthisalgorithmconvergesinSectionsVI-BandVII-B.Besidestheinitialgap,twootherparametersalsoaffecttheaccuracyoftheIGIandPTRalgorithms.Probingpacketsize.Measurementsusingsmallprobingpacketsareverysensitivetointerference.Theworkin[2]alsopointsoutsignificantpost-bottleneckeffectsforsmallpackets.Thisarguesforsendinglargerprobingpackets.Thenumberofprobingpackets.ItiswellknownthattheInternettrafficisbursty,soashortsnapshotcannotcap-turetheaveragetrafficload.Thatarguesforsendingafairlylargenumberofprobingpackets.However,sendingtoomanypacketscancausequeueoverflowandpacketlosses,increasetheloadonthenetwork,andlengthenthetimeittakestogetanestimate.Ourexperimentsshowthatthequalityoftheestimatesisnotverysensitivetotheprobingpacketsizeandthenumberofpackets,andthatthereisafairlylargerangeofgoodvaluesforthesetwoparameters.Forexample,a700-Bytepacketsizeand60packetspertrainworkwellontheInternet.WediscussthesensitivitytothesetwoparametersinmoredetailinSectionVII.V.EVALUATIONOurevaluationincludesthreeparts:1)InSectionVI,wecomparetheperformanceofIGI,PTR,andPathload,focusingonthemeasurementaccuracyandtheconvergencetime.2)InSectionVII,weanalyzehowtheprobingpacketsizeandthenumberofprobingpackets(packettrainlength)affectthemeasurementaccuracyofIGIandPTR.3)InSectionVIII,westudytheperformanceofIGIandPTRonanetworkpath,wherethetightlinkisnotthesameasthebottlenecklink.Wealsolookintoarelatedissueabouttheimpactofgaptimingerrors.ThefirsttwopartsarebasedonInternetmeasurements.Thelastpartisbasedonns2simulations,sinceweneedtocarefullycontrolthecompetingtrafficloadinthenetwork.ToevaluatetheaccuracyofthedifferentprobingalgorithmsontheInternet,weinterleaveprobingexperimentswithlargeapplicationdatatransfersthatshowhowmuchbandwidthisac-tuallyavailableandusableonthenetworkpath.However,itissometimeshardtodeterminetheactualavailablebandwidthonanInternetpath.Inprinciple,wewouldlikethedatatrans-ferstouseTCPsincemostapplications,especiallybulkdatatransferapplications,useTCP.Unfortunately,forhigh-band-widthpaths,wefindthatTCPisoftennotabletofullyutilizetheavailablebandwidth.Inmostcases,thereasonwassimplythatTCPend-to-endflowcontrolislimitingthethroughput,sinceourguestaccountsoftendonotallowustoincreasethesocketbufferstolargeenoughsizes.Onotherpaths,weob-serveasignificantamountofpacketreorderingorunexplained 886IEEEJOURNALONSELECTEDAREASINCOMMUNICATIONS,VOL.21,NO.6,AUGUST2003 Fig.10.ThroughputofparallelTCPflowsonthepathETHpacketlosses,bothofwhichcanhaveasignificantimpactonTCPperformance.Fortheabovereasons,weuseamixtureoftechniquestomeasurethe“true”availablebandwidth.Whenpossible,weuseasingleTCPflow.Whensmallwindowsizespreventusfromfillingthepipe,weuseanumberofparallelTCPflows.Thenumberofflowsisselectedonaperpathbasis.Atypicalexampleofhowtheend-to-endthroughputincreaseswiththenumberofflowsisshowninFig.10.Thethroughputincreasesinitiallyandthenflattensout.Typically,10oratmost20flowsaresufficienttofilltheavailablebandwidthpipe.Notethatthisapproachprovidesonlyaroughideaoftheac-curacyoftheprobingtechniques.Afirstproblemisthattheprobingandthedatatransferscannotberunatthesametime,sotheyseedifferenttrafficconditions,andweshouldexpectslightlydifferentresults.Moreover,becauseofthebandwidthsharingcharacteristicsofTCP,asingleTCPflowandmultipleparallelTCPflowsarenotequivalent.Ontheotherhand,ourapproachdoesmodelthewayapplicationswilltypicallyuseprobingtools,soourapproachcapturestheaccuracythatappli-cationswillperceive.OurexperiencewithtoolssuchasRemos[20]showsthatapplicationsingeneralonlyrequireroughesti-matesofpathproperties.TheimplementationoftheIGIandPTRalgorithmsneedsaccuratetimestampmeasurement.Asaresult,wewouldex-pectthebestresultswithkernelsupport,suchaslibpcap[21].However,formostoftheendhostsweuseforourexperiments,weonlyhaveguestaccounts,soalltheInternetmeasurementsarecollectedwithauser-levelimplementation.Theprobingpacketsareuser-definedprotocol(UDP)packets,andtimes-tampsaremeasuredwhentheclientorserverapplicationssendsorreceivestheUDPpackets.ThePathloadimplementationistakenfromhttp://www.cis.udel.edu/~dovrolis/pathload_1.0.2.tar.gz.Pathloadreturnsameasurementintervalthatshouldcontaintheactualavailablebandwidth.Inouranalysis,weusethecenteroftheintervalreturnedbyPathload.VI.COMPARATIVEVALUATIONInthissection,weanalyzetheperformanceofIGIandPTRalgorithmsusingexperimentsontheInternet.WealsocomparetheirperformancewiththatofPathload.TABLEIATHS A.NetworkPathsThedatapresentedinthissectioniscollectedusingaseriesofexperimentswhereeachexperimentmeasurestheavailablebandwidthusingthefollowingthreemethods:1)IGIandPTR:weusebothIGIandPTRalgorithmstoestimatetheavailablebandwidth.Theprobingpacketsizeissetto700Byteandtheprobingpacketnumberis60.WediscusswhywechoosethesetwovaluesinSectionVII.Pathload:Theresolutionparameterissetto2Mb/s.Bulkdatatransfer:WeuseoneormoreIperfTCPflowstoprobefortheactualavailablebandwidth.Thetransmis-siontimeis20seconds,andtheTCPwindowsizeatbothendsissetto128kB,whichissupportedonallmachineswehaveaccessto.Weseparatetheabovethreemeasurementsbya5secondssleepperiodtoavoidinterferencebetweenthemeasurements.Wesep-arateexperimentsby10minutesofidletime.Themeasurementsrunforanywherefrom6to40hours.Wecollectmeasurementsfor13Internetpaths,aslistedinTableI.Foreachpath,thefirstsiteisthesender,andthesecondsiteisthereceiver.Thecapacitiesinthethirdcolumndenotethebottlenecklinkcapacities,whichwewillalsorefertoasthepathcapacity.Thepathcapacitiesaremeasuredusingbprobe[5],andtheRTTsaremeasuredusingping.Thepathcapacitiesshowninthetableareobtainedby“rounding”themeasuredvaluestothenearestwell-knownphysicallinkcapacity.B.MeasurementAccuracyFig.11showstherelativemeasurementerrorofIGI,PTR,andPathload.Wedefinetherelativemeasurementerroras Here, canbe , ,and ,i.e.,theavailablebandwidthestimatesgeneratedbythedifferentCORNELL,CMU[1]–[3],NYU,ETH,andNCTUaremachinesinCor-nellUniversity,CarnegieMellonUniversity,NewYorkUniversity,ETHZurich(Switzerland),andNationalChiaoTungUniversity(Taiwan),respectively.MA,SLC[1],[2],SV,FC,SWEDEN,andNLaremachinesoncommercialnetworks,andtheyarelocatedinMassachusetts,SiliconValley,FosterCity,Sweden,andTheNetherlands,respectively.ThePathloadcodedoesnotapplytopathswithavailablebandwidthbelow1.5Mb/s(itreturnstheinterval[0,linkcapacity]),sowehavenoPathloadmea-surementsforPath1. HUANDSTEENKISTE:EVALUATIONANDCHARACTERIZATIONOFAVAILABLEBANDWIDTHPROBINGTECHNIQUES887 Fig.11.AvailablebandwidthmeasurementerrorfromIGI,PTR,andPathload.Eachbarshowsthemedianvalue,andthelineoneachbarshowsthe5%and95%percentilevalues. isthebulkdatatransmissionrate, isthebottlenecklinkcapacity.Forpaths1–10,weob-servethatthemeasurementerrorisbelow30%,andinmostcasestheerrorislessthan20%.Thatis,theestimatesproducedbytheIGI/PTRandthePathloadalgorithmsmatchtheTCPper-formancefairlywell.Forpaths11–13,therelativemeasurementerrorismuchhigher.Withoutinformationfromtheserviceproviders,itishardtotellwhatcausesthehighererrors.Becauseallthreemethodshavelowaccuracy,wehypothesizethatTCPhasdiffi-cultyusingtheavailablebandwidthduetobadpathproperties.Forexample,TableIshowsthattheRTTvariancesforpaths11and12arelargecomparedwiththosefortheotherpaths.Thismaybecausedbyrouteflaps,whichmaynegativelyinfluenceTCPsperformance.InFig.12,weshowamoredetailedcomparisonoftheband-widthestimatesforsixofthepaths.Wepickthree“good”pathswithdifferentpathproperties(paths1–3,seeTableI)andallthreeofthebadpaths(path11–13).ForpathsP1,P2,andP3,thegraphsconfirmthatallthreetechniquesprovidegoodestimatesoftheavailablebandwidth,asmeasuredbyIperf.Whichtechniqueismoreaccuratede-pendsonthepath.Forexample,IGIseemsmoreaccurateforP2andPathloadforP3.Onenotableexceptionistheperiodfromhour22tohour28forP1,wherebothIGIandPTRappeartounderestimatetheavailablebandwidth.Forthispath,thebot-tlenecklinkisadigitalsubscriberline(DSL),whichisingen-eralidle,asisshownbythehighavailablebandwidth.Duringthe22–28hourinterval,theDSLisused.SinceonlyoneorafewTCPconnectionsareactive,theyconsumeonlypartoftheavailablebandwidth.Thebulkdatatransfer,however,usesfiveparallelIperfflowsandappearstobegrabbingbandwidthfromtheotherflows.Thisillustratesthatthe“availablebandwidth”isnotnecessarilywell-definedanddependsonhowaggressivethesenderis.Notethatthisisasomewhatatypicalpath:onmostInternetpaths,individualsenderswillnotbeabletoaffectthebandwidthsharingaseasily.Forthethreepathswheretherelativemeasurementerrorishigh,weseetheavailablebandwidthestimatesproducedbyallthreemethodsaremuchhigherthanthebandwidthmeasuredusingIperf.Aswealreadysuggestedabove,thisprobablymeansthatTCP,asusedbyIperf,isnotabletofunctionwellbecauseofproblemssuchaswindowsize[22],lossrate,andvariableRTT[23].Notethatthethreebandwidthestimationtechniquesprovidefairlysimilarresults,exceptforpathP13,wherethePathloadestimatesareextremelyhigh.Formostpaths,theIGIandPTRestimatesarewithin10%ofeachother.OneexceptionisforpathP2[Fig.12(P2)],wheretheIGIestimateschangeoverawiderrangethanthosepro-videdbythePTRmethod.Webelievethisiscausedbytrafficonlinksotherthanthebottlenecklink.AswewilldiscussinSectionVIII,theIGImethodismoresensitivetocompetingtrafficfromnonbottlenecklinksthanthePTRmethod.C.ConvergenceTimesSofarourmeasurementshaveshownthatthethreealgorithmshavesimilaraccuracyintermsofpredictingavailableband-width.However,theIGIandPTRmethods,whichhavethesamemeasurementtime,aremuchfasterthanPathload,asisshowninTableII.Inthistable,weshowthepercentilevaluesofthemeasurementtimesat5%,50%(median),and95%foreachpathforboththeIGI/PTRandthePathloadtechniques.WeseethattheIGIandPTRmethodstypicallytakeabout1–2swhilePathloadtakesatleast12s[9].Wealsocomputetheratiobe-tweenPathloadandIGI/PTRforeachroundofmeasurements;themedianvaluesarelistedinthelastcolumnofthetable.Thegeometricmean[24]ofallratiosshowsthattheIGI/PTRmethodisonaveragemorethan20timesfasterthanPathloadforthe13pathsusedinthisstudy.ThelongmeasurementtimeforPathloadisduetoitscon-vergencealgorithm.Pathloadmonitorschangesintheone-waydelayoftheprobingpacketsinordertodeterminetherelation-shipbetweenprobingspeedandavailablebandwidth.Thiscanbedifficultifprobingpacketsexperiencedifferentlevelsofcon-gestion.ThiscanslowdowntheconvergenceprocessandcanresultinlongprobingtimesasshowninTableII.Incontrast,theconvergenceofIGI/PTRisdetermineddirectlybythepackettraindispersionatthesourceanddestination.Moreover,theIGIandPTRalgorithmsusethebottlenecklinkcapacity,whichisestimatedusingthesameprobingprocedure,toadjustsoastooptimizeconvergence.VII.IGIPTRAROPERTIESTheIGIandPTRalgorithmsselecttheappropriateinitialgapfortheprobingtrainsbysearchingfortheturningpoint,asdescribedinSectionIV.Inthissection,weuseInternetexperimentstostudytheimpactoftheothertwopackettrain 888IEEEJOURNALONSELECTEDAREASINCOMMUNICATIONS,VOL.21,NO.6,AUGUST2003 (P1)CORNELLMA(P11)SLCI (P2)SLCICMU2(P12)SLC2 (P3)NWUCMU1(P13)NCTUFig.12.AvailablebandwidthmeasurementsandthecorrespondingTCPperformance.ThenumberinthebracketsofIperfisthenumberofIperfTCPflowsuaxisistheclocktimevalue,anumberlargerthan24isthetimenextday.TABLEII parameters—theprobingpacketsizeandthenumberofprobingpackets(packettrainlength).A.ProbingPacketSizeTostudytheimpactoftheprobingpacketsizeonthemea-surementaccuracyoftheIGIandPTRalgorithms,weconductexperimentsontwoInternetpaths,usingprobingpacketsizesrangingfrom100to1400Byte.Werepeateachindividualmea-surement20times.Theentireexperimenttakesabout1h.OntheassumptionthatInternetpathpropertiesdonotchangemuchonthescaleofhours[25],wewouldexpectallmeasurementstohaveverysimilarresult.ThefirstInternetpathweuseisfromNWUtoCMU.Ithasapathcapacityof100Mb/s.ThemeasurementresultsareshowninFig.13(a)and(c)showshowtheavailablebandwidthmea-surementschangewiththeprobingpacketsize.Theavailable HUANDSTEENKISTE:EVALUATIONANDCHARACTERIZATIONOFAVAILABLEBANDWIDTHPROBINGTECHNIQUES889 (a)(b) Fig.13.IGIandPTRmeasurementswithdifferentprobingpacketsizesontwoInternetpaths.Graphs(a)and(b)showthefinalavailablebandwidthestiGraphs(c)and(d)showthegapconvergenceforindividualmeasurements.Theaxisistheinitialsourcegap,andtheaxisisthegapdifference,i.e.,thedestination(output)gapvalueminusthesource(input)gapvalue.ThepointsmarkedwithcirclesaretheturningpointswherethefinalestimatesarebandwidthmeasuredusingaTCPbulkdatatransfer(basedonthemethoddiscussedinSectionVI)is64Mb/s.Thepacketsizesthatresultintheclosestestimatesare500and700Byte.Forsmallerpacketsizes,bothmethodsunderestimatetheavailablebandwidthbyasignificantmargin.Forlargerprobingpacketsizes,thetwomethodsoverestimatetheavailablebandwidthbyamuchsmalleramount.Thereareatleasttworeasonswhysmallprobingpacketsizescanresultinhigherrorsintheavailablebandwidthestimation.First,asillustratedinFig.13(c),attheturningpointthegapvalueisproportionaltothepacketsize.Thismeansthatwithsmallpacketsizes,wewillhavesmallgaps,especiallyiftheavailablebandwidthishigh,asisthecasefortheNWUtoCMUpath.Theresultingprobingtrainismoresensitivetothebursti-nessofthecompetingtraffic.Thegraphfor100ByteprobingpacketsinFig.13(c)confirmsthis:thegapdifferencedoesnotconvergeasnicelyasitdoeswithlargerprobingpackets.Thesecondreasonisthatthesmallgapvaluesthatoccurwithsmallprobingpacketsarehardertomeasureaccurately,someasure-menterrorscanaffecttheresultsignificantly.Gapvaluesontheorderof10 arehardtogenerateandmeasureaccurately,es-peciallyforuser-levelapplications.Itislessclearwhywithlargerprobingpackets,theavailablebandwidthestimatesfurtherincreaseandinfactexceedthemea-suredbulkthroughput.Weconjecturethatthisisaresultoftheaggressivenessoftheprobingpackettrainflow.Probingflowswithlargerpacketsaremoreaggressivethanprobingflowswithsmallerpackets,sothey“observe”ahigheravailablebandwidth.ThepacketsizedistributiononInternethasclustersaround40,500,and1500Byte[26],soaflowwithonly1200or1500Bytepackets,forexample,ismoreaggressivethanaverage.ATCPbulkdatatransferislikelytousemostlymaximum-sizedpackets(1500Binthiscase),butitsdynamiccongestioncontrolbe-haviorreduceshowmuchbandwidthitcanuse.ThesecondexperimentisonthepathfromCORNELLtoCMU.TheresultsaresummarizedinFig.13(b)and(d).Thelinkcapacityofthebottlenecklinkisonly10Mb/s,asopposedto100Mb/sfortheNWUtoCMUpath.Asaresult,theavailablebandwidthissignificantlylower.TheresultsconfirmthemainresultsofthemeasurementsfortheNWUtoCMUpath.First,theavailablebandwidthestimatesincreasewiththepacketsize.Second,sincetheavailablebandwidthismuchlower,weareseeingfairlysmoothconvergenceofthegapdifference,evenforsmallprobingpacketsizes[Fig.13(d)].Finally,eventhoughweobserveniceconvergence,thebursti-nessofthecompetingtrafficdoesaffecttheprobeswithsmallpacketsmorethantheprobeswithlargerpackets.FortheIGIalgorithm,theresultswith100Byteprobingpacketare 890IEEEJOURNALONSELECTEDAREASINCOMMUNICATIONS,VOL.21,NO.6,AUGUST2003 Fig.14.Performancewithpackettrainsofdifferentlengths.suspiciousandhavealargevariance.BecausetheIGIalgorithmusesthechangesinindividualgapvaluesinsteadoftheaveragepackettrainrate(asusedbyPTR),itismoresensitivetosmallchangesingapvalues,forexampleasaresultofburstytrafficortrafficonnonbottlenecklinks.WediscussthispointinmoredetailinSectionVIII.Ourconclusionisthatingeneral,average-sizedprobingpacketsofabout500to700Bytearelikelytoyieldthemostrepresentativeavailablebandwidthestimate.Smallerpacketsizesmayunderestimatetheavailablerateandmaybemoresensitivetomeasurementerrors,whilelargerprobingpacketsizescanoverpredicttheavailablebandwidth.B.PacketTrainLengthandNumberofProbingPhasesThepackettrainlengthhasalargeimpactonthecostofthePTRandIGIalgorithms,sinceitaffectsboththenumberofpacketsthataresent(i.e.,theloadplacedonthenetwork)andtheprobingtime(i.e.,thelatencyassociatedwiththeprobingoperation).Anotherimportantparameter,thenumberofphasesneededtoconvergeonthebestinitialgapvalue(theturningpoint),istiedverycloselytothepackettrainlength.Intuitively,shorterpackettrainsprovidelessaccurateinformation,somorephasesmaybeneededtoconvergeontheturningpoint.Forthisreason,wewillstudythepackettrainlengthandthenumberofphasesintheIGI/PTRalgorithmtogether.InSectionIV,wementionedthattrainsof60packetsworkwell.Inthissection,weexperimentallyevaluatehowmuchwecanreducethisnumberwithoutasignificantlossinaccuracy.WeconductexperimentsoverthesametwoInternetpathsasintheprevioussection,i.e.,NWUtoCMUandCORNELLtoCMU.Foreachpath,weusepackettrainsofdifferentlengthstoestimatetheavailablebandwidth.Themeasurementstakeabouttwohours.SincetheavailablebandwidthovertheInternetisfairlystable[25],wedonotexpecttheavailablebandwidthtochangesignificantlyduringthe2-hperiod.Themeasurementswithdifferenttrainlengthsarealsointerleavedtofurtherreduceanypossiblebiastowardaspecifictrainlength.Fig.14showsthecumulativedistributionfunction(CDF)oftheestimatedavailablebandwidthusingIGI(top),andthenumberofprobingphasesneededtoconvergeontheturningpoint(bottom).ThedistributionsforthePTRmeasurementaresimilarandarenotincludedhere.Eachgraphhasfivecurves,correspondingtofivedifferentpackettrainlengths:8,16,24,32,and64.First,weobservethatshorterpackettrainsneedmorephasestoconverge,whichwehadalreadyconjecturedear-lier.Themeasurementsalsoshow,againnotsurprisingly,thatshorterpackettrainsresultinawiderrangeofavailableband-widthestimates,asshownbyaCDFthatismorespreadout.Thereasonisthatthecompetingtraffic(and,thus,theavail-ablebandwidth)isbursty,andsinceashorterpackettraincor-respondstoashortersamplinginterval,weareseeingawiderrangeofestimates.Note,however,thatasthepackettrainlengthincreases,theimpactofthepackettrainlengthonthedistribu-tionofthebandwidthestimatesbecomessmaller,i.e.,theesti-matesconvergeonaspecificvalue.Itisinterestingtocomparetheresultsforthetwopaths.FortheNWUtoCMUpath,changingthepackettrainlengthhasafairlysignificantimpactonthedistributionsforboththeavail-ablebandwidthandthephasecount.Inotherwords,increasingthepackettrainlengthhelpsinprovidingamorepredictableavailablebandwidthestimate.Usinglongertrainsisalso“re-warded”withareductioninthethenumberofprobingphases.Incontrast,fortheCORNELLtoCMUpaththeCDFfunctionsforboththeavailablebandwidthandphasecountarefairlysim-ilarfortrainlengthsof16packetsormore.ThereasonisthatthecompetingtrafficonthispathisnotasburstyasthatontheNWUtoCMUpath.Thedifferencebetweenthetwopathsraisesthequestionofwhatpackettrainlengthweshoulduseforavailablebandwidthestimation.Clearly,themostappropriatetrainlengthdependsonthepath.FortheNWUtoCMUpath,weprobablywouldwanttouseafairlylargevalue(32or64packets),whilefortheCORNELLtoCMUpath,atrainlengthof16packetsissufficient.Sincethedifferencebetweenthepathsappearstobecausedbytheburstinessofthetraffic,wedecidetousethechangesinthepacketgapstocharacterizetheburstinessofthe HUANDSTEENKISTE:EVALUATIONANDCHARACTERIZATIONOFAVAILABLEBANDWIDTHPROBINGTECHNIQUES891 Fig.15.Relativeburstinessmeasurementsbasedonthegapvaluesofaprobingtrain.Eachbarshowsthemedianvalueandthelinesoneachbarshowthe5and95%values.competingtraffic.Specifically,wedefinetherelativeburstiness where arethe gapmeasurementsofaprobingFig.15showstherelativeburstinessoftheIGImeasurementsattheturningpointforthetwopathsandforthedifferentpackettrainlengths.Werecordthedetailedgapvaluesattheturningpointfor65measurements(around20%ofthemeasurementscollected).TherelativeburstinessforthepathfromNWUtoCMUissignificantlyhigherthanthatforthepathfromCOR-NELLtoCMU.Interestingenough,theresultsforeight-packetprobingtrainsdonotfollowthistrend.Wesuspectthateightpacketsissimplynotlongenoughtogetareliablemeasurement(notethewidespread).Theseresultssuggestthatwecanreducethecostofprobingbydynamicallyadjustingthelengthofthepackettrain.Forex-ample,wecoulduseapackettrainof32packetsforthefirstfewphasesandusetheburstinessresultsofthosephasestoadjustthelengthoflaterpackettrains.Wedecidenottodothisbecause,astheresultsinTableIIshow,theIGI/PTRalgorithmisalreadyquitefast.Thedistributionoftheprobingphasecountsshowsthat80%ofthemeasurementsonlyneed4–6phasestocon-vergetotheturningpoint,sothecorrespondingprobingtimeisaround4–6RTTs.Dynamicallyadjustingthepackettrainlengthis,thus,notlikelytohavealargeimpactontheprobingtime.Ofcourse,wecouldmaketheburstinessinformationavailabletouserssotheycanknowhowvariabletheavailablebandwidthislikelytobeforshortdatatransfers.VIII.MULTIHOPTheIGIandPTRalgorithmsarebasedonthegapmodelpre-sentedinSectionIII.Itisderivedforasimplesingle-hopnet-work,ormoregenerally,foranetworkinwhichthebottlenecklinkisthetightlinkandtheeffectofallotherlinkscanbeig-nored.Inthissection,weusesimulationstostudymoregen-eralmultihopnetworks.Specifically,weaddresstwoquestions.First,howshouldweinterpretthemodelifthetightlinkisnot Fig.16.Simulationconfiguration.PsandPdareusedforprobing.C1s,C1d,C2s,C2d,C3s,andC3dareusedforthecompetingtrafficgeneration. Fig.17.Pretightlinkeffect.Here, isthebottlenecklinkcapacity,and isthetightlinkcapacity.thebottlenecklink,andwhataretheimplicationsfortheIGIandPTRmethod?Second,howdoesthecompetingtrafficonlinksotherthanthetightlinkaffecttheaccuracyofthealgorithms?A.TightLinkIsNottheBottleneckLinkWhenthetightlinkandthebottlenecklinkaredifferent,thegapmodelshowsthattheIGIalgorithmshouldusethe and valuesforthetightlinkwhenestimatingtheavailableband-width.Unfortunately,toolssuchasbprobeonlyestimatetheca-pacityofthebottlenecklink.Thiswillhaveanimpactontheaccuracyofthemethod.NotethatPTRdoesnotusethe and valuesexplicitly,soitwillnotbeaffectedbythistightlinkIntheremainderofthissection,wewillusens2[27]sim-ulationtoevaluatetheaccuracyofbothalgorithmsinthissce-nario.Whilesimulationhasthedrawbackthatitleavesoutmanyreal-worldeffects,ithastheadvantagethatwecanstudytopolo-giesthataredifficultorimpossibletobuild.WeusethesimulationtopologyshowninFig.16,using20,10,and20Mb/sforthelinkcapacitiesX,Y,andZ,respectively.BychangingthecompetingloadsC1,C2,andC3wecanchangethetightlinkofthepathandalsochangetheleveloftrafficonlinksotherthanthetightlink.Theprobingpacketsizeusedinthesimulationis700Byteandtheprobingpackettrainlengthis60.ThecompetingtrafficconsistsofCBRUDPtraffic.Notethatbypickinglinkcapacitiesthatarefairlyclose,theavail-ablebandwidthsondifferentlinksarelikelytobecloseaswell,whichisachallengingcase.Inthefirstsetofsimulations,wesetC2to3Mb/sandchangeC1from0to19Mb/s.WhenC1isintherange0–13Mb/s,thebottlenecklink R2,R3 isalsothetightlink,butwhenC1fallsin13–19Mb/s,thetightlinkis R1,R2 .Fig.17presentsthesimulationresults.Weseethatwhenthebottlenecklinkisequaltothetightlink( Mb/s),theIGImethodaccuratelypredictstheavailablebandwidth,asexpected.When R1,R2 isthetightlink,weshowtheIGIestimatesbasedonthe 892IEEEJOURNALONSELECTEDAREASINCOMMUNICATIONS,VOL.21,NO.6,AUGUST2003 Fig.18.Posttightlinkeffect.Here, isthebottlenecklinkcapacity,and isthetightlinkcapacity. and valuesforboththetight(“o”points)andbottlenecklinks(“x”points).Weseethattheresultsusingthetightlinkvaluesaremuchcloser.Theerroristheresultofinterferencefromcompetingtrafficonthe“nontight”link,aswediscussinmoredetailinthenextsubsection.Next,werunasimilarsetofsimulations,butwenowkeepC2fixedto3Mb/sandchangethecompetingtrafficC3from0to19Mb/s.Thetightlinkswitchesfrom R2,R3 to R3,R4 whenC3goesabove13Mb/s.Fig.18showsthattheresultsaresimilartothoseinFig.17:whenthetightlinkisnotthebottlenecklink( Mb/s),usingthe and valuesforthetightlinkgivesamoreaccuratepredictionfortheavailablebandwidthonthepath.However,theresultswhen Mb/sarelessclearthanforthepretightlinkcaseinFig.17,wewillexplainitinthenextsection.InFigs.17and18,wealsoplotthecorrespondingPTRvalues.ThePTRestimatesarealmostidenticaltotheIGIestimatesthatusethe and valuesforthetightlink.ThereasonisthatthePTRformuladoesnotexplicitlyuseanyinformationaboutthetightlinkcapacity.ThefactthattheIGIalgorithmusesthecapacityofthetightlinkexplicitlyisaproblembecauseweonlyhavetechniquesforidentifyingthelinkcapacityofthebottlenecklink,notthetightlink.Inpractice,thisisnotlikelytobeaproblem:weexpectthatonmanypaths,theaccesslinkfromtheclientnetworktotheISPwillbeboththebottleneckandthetightlink.OurInternetmeasurementsinSectionVIconfirmthis.B.InterferenceFromTrafficon“Nontight”LinksInamultihopnetwork,eachlinkwillpotentiallyaffectthegapvalueofapacketpairorpackettrain,sowehavetoef-fectivelyconcatenatemultipleinstancesofthesingle-hopgapmodel.Suchamultihopgapmodelishardtointerpret.How-ever,itisfairlyeasytoseethatitisthelinkwiththelowestunusedbandwidth(i.e.,thetightlink)thatwillhavethelargestimpactonthegapatthedestination.Theintuitionisasfollows.Onlinksthathavealotofunusedbandwidth,thepacketsoftheprobingflowarelikelytoencounteranemptyqueue,i.e.,theselinkswillhavealimitedimpactonthegapvalue.Ofcourse,theselinksmaystillhavesomeeffectonthegapvalues,asweanalyzeinthissectionusingthesimulationresultsfromthepre-vioussection.TheresultsinFig.17for Mb/sshowthatbothIGIandPTRareveryaccurate,evenwhenthereissignificantcompetingtrafficonalinkprecedingthetightlink.Interestingenough,thesecondsetofsimulationsshowadifferentresult. Fig.19.Combinedpretightandposttightlinkeffectswith20Mb/spretightandposttightlinkcapacities.TheresultsinFig.18for Mb/scorrespondtothecasethatthereissignificantcompetingtrafficonalinkfol-lowingthetightlink.WeobservethatwhilePTRisstillaccurate,theIGIaccuracysuffers.ThedifferentimpactonIGIofcompetingtrafficinlinksup-streamanddownstreamoftightlinkcanbeexplainedasfollows.Changesingapvaluesbeforethetightlinkwillbereshapedtherouterwhichthetightlinkconnectswith,andthecompetingtrafficonthetightlinkendsuphavingthedominatingimpact.Incontrast,anychangesingapvaluesthatarecausedbytrafficonlinksfollowingthetightlinkwilldirectlyaffecttheavailablebandwidthestimates,sotheyhavealargerimpact.SinceIGIisbasedonmorefine-graininformationthanPTR,itismoresen-sitivetothiseffect.InFig.19,weshowtheavailablebandwidth,asestimatedbyIGI,whenthereissignificantcompetingtrafficonboththelinksbeforeandafterthetightlink.Theactualavailableband-widthis7Mb/sforalldatapoints.Itisdeterminedbylink R2,R3 ,whichhas10Mb/scapacityand3Mb/scompetingtraffic(C2).Theresultsconfirmtheaboveobservation.Evensignifi-cantcompetingtrafficbeforethetightlinkhasalmostnoimpactontheaccuracy:thecurveisbasicallyflatalongthe Competingtrafficafterthetightlinkdoes,however,haveanef-fectand,notsurprisingly,itsimpactincreaseswiththelevelofcompetingtraffic.Notethattheabovesimulationsresultsaredesignedtohighlightaparticularlychallengingcase.Inpractice,itisnotcommontohavelinkswithcapacitiesand/oravailablebandwidthsthatarethissimilar.Insuchcases,theeffectofcompetingtrafficonotherlinksisveryminimal.Forexample,werunasetofsimulationssimilartothosedescribedabove,butwiththe R1,R2 and R3,R4 setto100Mb/sinsteadof20Mb/s.Thecapacityof R2,R3 anditcompetingtrafficthroughput(C2)keeptobe10and3Mb/s,respectively,i.e.,theavailablebandwidthisstill7Mb/s.TheresultsareshowninFig.20.WeseethattheIGImethodgivesaccurateresults—themeanvalueforthedatapointsinthisfigureis7.24Mb/s,andthestandarddeviationis0.10Mb/s.ThefactthatIGIandPTRtypicallyproduceverysimilarestimatesinourInternet HUANDSTEENKISTE:EVALUATIONANDCHARACTERIZATIONOFAVAILABLEBANDWIDTHPROBINGTECHNIQUES893 Fig.20.Combinedpretightandposttightlinkeffectswith100Mb/spretightandposttightlinkcapacities. Fig.21.Impactofinitialgaperror.experimentsshowsthattheresultsinFig.20aremuchmoretypicalthantheworstcaseresultsinFigs.17and18.C.TimingErrorsAnotherfactorthatcanreducetheaccuracyoftheIGIandPTRalgorithmsisthemeasurementerrorsinthegapvalues.Therearetwotypesofgapmeasurementerrors:theerrorsintheinitialgapvaluegeneratedbythesourcehostandthemeasure-menterrorsinthefinalgapvaluemeasuredonthedestinationToillustratetheeffectofsourcegapgenerationerror,weusethetopologyshowninFig.16,withX,Y,andZsetto20,10,and20Mb/s,respectively.TheflowC2istheonlycompetingflowandwechangeitsthroughputintherangeof0–9Mb/s.Foreachexperiment,theinitialgap( )isincrementedbyarandomvalue thatisuniformlydistributedin( , ),i.e., Werunsimulationsfor rangingfrom0–2ms,andfor value,wecollectresultswhenC2changesbetween0and9Mb/s.Fig.21showstheaverageabsoluteerrorintheIGIestimateasafunctionof .Weseethattheerrorissmall.Notetheturninggapvalueforthissimulationis0.3–1.7ms,sotheerrorsinflictedontheinitialgaparequitelarge.Webelievethatthereasonforthehigh-errortoleranceisthesameasthereasonforthelowsensitivityofIGItothepretightlinktraffic.Specifically,thetightlinkendsupreshapingthegapsaccordingtothecompetingtraffic,thus,ineffecthidingtheinitialgaperrors.Clearly,measurementerrorsonthedestinationsidewillhaveamoresignificantimpactsincetheywilldirectlychangethegapvaluesthatareusedintheIGIandPTRformulas.IX.CInthispaper,wepresentasimplegapmodelthatcaptureshowcompetingtrafficonanetworklinkaffectsthegapvalueofpacketpairsandpackettrains.Thegapmodelhelpsusidentifyunderwhatconditionspacketpairsprobingyieldsusefulinfor-mationabouttheavailablebandwidthalongapath.Italsoshowsthatthekeytogetusefulmeasurementsistocontroltheinitialgapofthepackettrain.Themostaccurateresultsareobtainedwhentheaverageoutputgapatthedestinationequalstheav-erageinitialgapatthesource.Wedesigntwotechniquesforestimatingavailablebandwidthbasedonthegapmodel.TheIGIalgorithmusestheinformationaboutchangesingapvaluesofapackettraintoestimatethecompetingbandwidthonthetightlinkofthepath.ThePTRmethodusestheaveragerateofthepackettrainasanestimateoftheavailablebandwidth.WecomparetheestimatesoftheIGIandPTRalgorithmswithPathloadestimatesandmeasuredTCPthroughputontheInternet.Theresultsshowthatallthreemethods(IGI,PTR,andPathload)haveasimilarmeasurementerror:inmostcasestheerrorislessthan30%.IGIandPTRtypicallyfinishinunder2swhilePathloadtakesalotlonger.Ananalysisofthealgorithmpropertiesprovidessuggestionsonhowtochoosetheprobingpacketsizeandtheprobingpackettrainlengthinordertoachievethebestmeasurementaccuracywiththeleastoverhead.Inthelastpartofthispaper,weusesimulationstostudythedynamicsofthemethodsinnetworkswithsignificanttrafficonmultiplelinksalongthepath.WeshowthattheIGImethodlosesaccuracyifthetightlinkisnotthebottlenecklink,orifthereissignificantcompetingtrafficonlinksfollowingthetightlink.Competingtrafficbeforethetightlinkhas,however,littleim-pactontheaccuracy.SincethePTRmethoddoesnotmakeuseofthedetailedchangesinthegapvaluesintheprobingpackettrain,itismuchlesssensitivetothepresenceoftrafficonlinksotherthanthetightlink.TheseresultssuggestthatthePTRmethodisthepreferredmethodforestimatingavailableCKNOWLEDGMENTTheauthorswouldliketothankD.Andersen,Y.-H.Chu,P.Dinda,andR.KarrerforallowingustousetheirresourcesinourInternetexperiments.TheywouldalsoliketothankM.Agrawal,R.Balan,Y.-H.Chu,J.Gao,A.-C.Huang,J.Lopez,andS.Raofortheirusefulcomments.[1]K.LaiandM.Baker,“Nettimer:Atoolformeasuringbottlenecklinkbandwidth,”inProc.USENIXSymp.InternetTechnologiesandSystemsMar.2001,pp.123–134.[2]C.Dovrolis,P.Ramanathan,andD.Moore,“Whatdopacketdispersiontechniquesmeasure?,”inProc.Conf.ComputerCommunication,Apr.2001,pp.905–914.[3]A.B.Downey.Clink:Atoolforestimatinginternetllinkcharacteristics.[Online].Available:http://rocky.wellesley.edu/downey/clink/ 894IEEEJOURNALONSELECTEDAREASINCOMMUNICATIONS,VOL.21,NO.6,AUGUST2003[4]V.Jacobson,“Pathchar–Atooltoinfercharacteristicsofinternetpaths,”presentedattheMathematicalSciencesResearchInstitute,Apr.1997,[Online].Available:ftp://ftp.ee.lbl.gov/pathchar/msri-talk.pdf.[5]R.L.CarterandM.E.Crovella,“Measuringbottlenecklinkspeedinpacket-switchednetworks,”BostonUniv.,Boston,MA,Comput.Sci.Dept.,Tech.Rep.,Mar.1996.[6]B.A.Mah.(2001)Pchar:Atoolformeasuringinternetpathcharac-teristics.[Online].Available:http://www.employees.org/bmah/Soft-ware/pchar/[7]S.Seshan,M.Stemm,andR.H.Katz,“SPAND:Sharedpassivenetworkperformancediscovery,”inProc.1stUsenixSymp.InternetTechnolo-giesSystems,Monterey,CA,Dec.1997,pp.135–146.[8]M.Stemm,S.Seshan,andR.H.Katz,“Anetworkmeasurementarchi-tectureforadaptiveapplications,”inProc.Conf.ComputerCommuni-,Mar.2000,pp.285–294.[9]M.JainandC.Dovrolis,“End-to-endavailablebandwidth:Measure-mentmethodology,dynamics,andrelationwithTCPthroughput,”inProc.ACMSIGCOMMSymp.CommunicationsArchitecturesProtocolsPittsburgh,PA,Aug.2002,pp.295–308. ,“Pathload:Ameasurementtoolforend-to-endavailableband-width,”inProc.PassiveActiveMeasurements,FortCollins,CO,Mar.[11]J.CurtisandT.McGregor,“Reviewofbandwidthestimationtech-niques,”inProc.NewZealandComputerScienceResearchStudents’Conf.,vol.8,NewZealand,Apr.2001.[12]V.Jacobson,“Congestionavoidanceandcontrol,”inProc.ACMSIG-COMMSymp.CommunicationsArchitecturesProtocols,Aug.1988,pp.[13]J.-C.Bolot,“End-to-endpacketdelayandlossbehaviorintheInternet,”Proc.ACMSIGCOMMSymp.CommunicationsArchitecturesProto-,SanFrancisco,CA,Sept.1993,pp.289–298.[14]S.Keshav.PacketPairFlowControl.[Online].Available:http://www.cs.cornell.edu/skeshav/doc/94/2–17.ps[15]R.L.CarterandM.E.Crovella,“Dynamicserverselectionusingbandwidthprobinginwide-areanetworks,”BostonUniv.,Boston,MA,Comput.Sci.Dept.,Tech.Rep.,Mar.1996.[16]V.Paxson,“MeasurementsandAnalysisofEnd-to-EndInternetDy-namics,”Ph.D.dissertation,Comput.Sci.Div.,U.C.Berkeley,Berkeley,CA,May1996.[17]M.MathisandJ.Mahdavi,“DiagnosingInternetcongestionwithatrans-portlayerperformancetool,”presentedattheINETConf.,Montreal,QC,Canada,June1996.[18]B.Melander,M.Bjorkman,andP.Gunningberg,“Anewend-to-endprobingandanalysismethodforestimatingbandwidthbottlenecks,”inProc.IEEEGLOBECOM–GlobalInternetSymp.,SanFrancisco,CA,Nov.2000,pp.415–420.[19]A.TirumalaandJ.Ferguson.Iperf.[Online].Available:http://dast.nlanr.[20]P.A.Dinda,T.Gross,R.Karrer,B.Lowekamp,N.Miller,P.Steenkiste,andD.Sutherland,“ThearchitectureoftheRemossystem,”inProc.10thIEEEInt.Symp.HighPerformanceDistributedComputing,Aug.2001,pp.252–265.[21]Libpcap[Online].Available:ftp://ftp.ee.lbl.gov/libpcap.tar.Z[22]J.Semke,J.Mahdavi,andM.Mathis,“AutomaticTCPbuffertuning,”inProc.ACMSIGCOMMSymp.CommunicationsArchitecturesProtocolsVancouver,BC,Canada,Sept.1998,pp.315–323.[23]J.Padhye,V.Firoiu,D.Towsley,andJ.Kurose,“ModelingTCPthroughput:Asimplemodelanditsempiricalvalidation,”inProc.ACMSIGCOMMSymp.CommunicationsArchitecturesProtocolsVancouver,BC,Canada,Sept.1998,pp.303–314.[24]R.Jain,TheArtofComputerSystemsPerformanceAnalysis.NewYork:Wiley,1991.[25]Y.Zhang,N.Duffield,V.Paxson,andS.Shenker,“OntheconstancyofInternetpathproperties,”inProc.ACMSIGCOMMInternetMeasure-mentWorkshop,SanFrancisco,CA,Nov.2001,pp.197–211.[26]K.Claffy,G.Miller,andK.Thompson,“Thenatureofthebeast:Recenttrafficmeasurementsfromaninternetbackbone,”presentedattheISOCINETConf.,July1998.[27]Ns2[Online].Available:http://www.isi.edu/nsnam/ns NingningHu(S’02)receivedtheB.S.degreeincomputersciencefromNanjingUniversity,Nanjing,China,in1998andisworkingtowardthePh.D.degreeintheComputerScienceDepartment,CarnegieMellonUniversity,Pittsburg,PA.Hisresearchinterestsareinnetworkmeasurement,networkingprotocols,andactivenetworkservices. PeterSteenkiste(M’00–SM’01)receivedtheB.S.degreeinelectricalengineeringfromtheUniversityofGhent,Belgium,in1982,andtheM.S.andPh.D.degreesinelectricalengineeringfromStanfordUni-versity,Stanford,CA,in1983and1987,respectively.HeisaProfessorintheDepartmentofComputerScienceandtheDepartmentElectricalandCom-puterEngineering,CarnegieMellonUniversity,Pittsburg,PA.Hisresearchinterestsareintheareasofnetworkinganddistributedsystem.