Accurately Measuring Denial of Service in Simulation and Testbed Experiments Jelena Mirkovic - PDF document

2experimentsinvolvinghumanusers(SectionVI).WesurveyrelatedworkinSectionVIIandconcludeinSectionVIII.Thispaper'scontributionsarethree-fold:(1)WeproposeanovelapproachtoDoSimpactmeasurementrelyingonapplication-specicQoSrequirements.Althoughourproposedmetricscombineseveralexistingapproaches,theirnoveltyliesin(i)thecarefulspecicationoftrafcmeasurementsthatreectservicedenialforthemostpopularapplications,and(ii)thedenitionofQoSthresholdsforeachmeasurementandeachapplicationclass,basedonextensivestudyoftheQoSliterature.(2)WeaggregatemultiplemeasurementsintointuitiveandinformativeDoSmetricsthatcanbedirectlyappliedtoexistingtestbedexperimentsandsimulations,andtoavarietyofDoSscenarios.(3)Wedemonstratethatourmetricsaccuratelycapturehumanperceptionofservicedenialbyconductingexperimentswithhumanusers.Admittedly,calculatingourmetricsismorecomplexthanlegacyones.Toeasethisprocess,wehavemadetheprogramusedforDoSmetricscalculationfromnetworktracesfreelyavailableathttp://www.isi.edu/mirkovic/dosmetric.II.EXISTINGMETRICSPriorDoSresearchhasfocusedonmeasuringdenialofser-vicethroughselectedlegitimatetrafcparameters:(a)packetloss,(b)trafcthroughputorgoodput,(c)request/responsedelay,(d)transactionduration,and(e)allocationofresources.Researchershaveusedbothsimplemetrics(singletrafcparameter)andcombinationsofthemtoreporttheimpactofanattackonthenetwork.Allexistingmetricsarenotquantitativebecausetheydonotspecifyrangesofloss,throughput,delay,durationorresourcesharesthatcorrespondtoservicedenial.Indeed,suchvaluescannotbespeciedingeneralbecausetheyhighlydependonthetypeofapplicationwhosetrafcco-existswiththeattack:10%lossofVoIPtrafcisdevastating,while10%lossofDNStrafcismerelyaglitch.Allexistingmetricsarefurthernotversatileandwepointoutbelowthecaseswheretheyfailtomeasureservicedenial.Theyareinaccuratesincetheyhavenotbeenproventocorrespondtoahumanuser'sperceptionofservicedenial.Lossisdenedasthenumberofpacketsorbyteslostduetotheinteractionofthelegitimatetrafcwiththeattack[1]orduetocollateraldamagefromadefense'soperation.Thelossmetricprimarilymeasuresthepresenceandextentofcongestioninthenetworkduetooodingattacks.Itcannotbeusedforattacksthatdonotcontinuallycreatecongestion,ordonotcongestnetworkresourcesatall.Examplesofsuchattacksarepulsingattacks[2],[3],TCPSYNoods[4],attacksthattargetapplicationresourcesandvulnerabilityattacksthatcrashapplicationsandhosts.Further,thelossmetrictypicallydoesnotdistinguishbetweenthetypesofpacketslost,whilesomepacketlosseshaveamoreprofoundimpactthanothers(forexample,alostSYNvsdatapacket)onservicequality.Throughputisdenedasthenumberofbytestransferredperunittimefromthesourcetothedestination.Goodputissimilar,butdoesnotcountretransmittedbytes[2],[5].BotharemeaningfulforTCP-basedtrafc,whichrespondstocongestionbyloweringitssendingrate.Indirectly,thesemetricscapturethepresenceandextentofcongestioninthenetworkandtheprolongeddurationoflegitimatetransactionsduetocongestion.Theycannotbeappliedtoapplicationsthataresensitivetojitterortolossofspecic(e.g.,control)packets,becauseahighthroughputlevelmaystillnotsatisfythequalityofservicerequiredbytheuser.Further,thesemetricsdonoteffectivelycaptureDoSimpactontrafcmixesconsistingofshortconnections,withafewpacketstobesenttotheserver.Suchconnectionsalreadyhavealowthroughputsoservicedenialmaybemasked.Request/responsedelayisdenedastheintervalbetweenwhenarequestisissuedandwhenacompleteresponseisreceivedfromthedestination[6].Itmeasuresservicedenialofinteractiveapplications(e.g,,telnet)well,butfailstomeasureitfornon-interactiveapplications(e.g.,email)whichhavemuchlargerthresholdsforacceptablerequest/responsedelay.Thismetricisalsoinapplicabletoone-waytrafc(e.g.,mediatrafc)whichdoesnotgenerateresponsesbutissensitivetoone-waydelay,lossandjitter.Transactiondurationisthetimeneededforanexchangeofameaningfulsetofmessagesbetweenasourceandadestination[7],[8],[9].Thismetricdependsheavilyonthevolumeofdatabeingtransferredandwhethertheapplicationisinteractiveandcongestion-sensitive.Itaccuratelymeasuresservicedenialforinteractiveapplications,suchasWebbrows-ing.Forone-waytrafc,suchasmediastreamingthatmaynotrespondtocongestionandrunsoverUDP,transactiondurationwillnotbeaffectedbytheattack.Durationofmanynon-interactivetransactionscanbeextendedwithoutcausingservicedenialbecausehumansexpectthatsuchtransactionsmayoccurwithsomedelay.Allocationofresourcesisthefractionofacriticalresource(usuallybandwidth)allocatedtolegitimatetrafcvs.attacktrafc[8],[10].Thismetricdoesnotprovideanyinsightintotheuser-perceivedservicequality.Itassumestheserviceisdeniedduetolackofresources,andappliesonlytooodingattacks.Further,itcannotcapturecollateraldamageofagivendefense.Forexample,adefensethatdrops90%oflegitimateand100%ofattacktrafc,wouldappearperfect,sinceitallocatesallremainingresourcestolegitimatetrafc.Weacknowledgethattheexistingmetricsconveysomenotionofdeniedservice,especiallywhenthedenialissevere.They,however,sufferfromtwomajordrawbacks:(1)Theymeasureasingletrafcparameterassumingthatitsdegra-dationalwayscorrespondstoservicedenial,whereastrafcparametersthatsignalservicedenialareactuallyapplication-specicandsomeattackscandenyservicewithoutaffectingthemonitoredparameter.(2)Theyfailtodenetheparam-eterrangerequiredforacceptableservicequality,whichisapplication-andtask-specic.Finally,theexistingmetricspredominantlycapturetheservicedenialatthenetworklayer,enroutetothevictimserver.Whilemanyattackstargetthisroute,someaffecttheserverhostortheapplicationdirectly,ortargetsupportingnetworkservices(suchasDNS),ortheroutefromtheservertolegitimateusers.Network-basedmetricsfailtocorrectlycapturetheimpactoftheseattacks. 4 Category One-waydelay Req/repdelay Loss Duration Jitter email(srv/srv) whole,RTT4h Usenet whole,RTT4h chat,typing RTT4s chat,typing somedatamustbesenttoserver chat,audio 150ms whole,RTT4s 3% 50ms chat,video 150ms whole,RTT4s 3% Web part,RTT4s 60s Web somedatamustbereceivedfromserver FTPData part,RTT10s 300% FTPControl part,RTT4s FTP somedatamustbeexchangedondatachannel FPSgames 150ms 3% RTSgames 500ms telnet part,RTT250ms telnet somedatamustbereceivedfromserver email(usr/srv) part,RTT4s 300% DNS whole4s ping whole4s media control media media audio,conv. 150ms whole,RTT4s 3% 50ms audio,messg. 2s whole,RTT4s 3% 50ms audio,stream 10s whole,RTT4s 1% 50ms videophone 150ms whole,RTT4s 3% video,stream 10s whole,RTT4s 1% TABLEIAPPLICATIONCATEGORIESANDTHEIRQOSREQUIREMENTS.B.MeasurementApproachDuringsimulation,collectionofnecessarytrafcmeasure-mentsusuallyimpliesslightsimulatormodication.Suchcollectionisachallengeintestbedexperimentation,andweexploredtwopossibleapproaches:(i)Instrumented-clients:instrumentingeachclientapplicationtocomputerequiredmeasurements,or(ii)Trace-based:usingreal,uninstrumentedapplicationsandtrafcgenerators,identifyingtransactionsincollectedpackettracesandcomputingtrafcmeasurements.Theinstrumentedclientapproachcanpreciselyidentifytrans-actions,butitlimitsthemetrics'usabilitytoopen-sourceclients.Wethusdecidedtousethetrace-basedapproach,sinceitiseasilyapplicabletomosttestscenariosandim-mediatelyusablebyotherresearchers.Inimplementingtrace-basedQoSevaluation,weencounteredseveralchallengesintransactionandrequest/responseidentication.Wesummarizeourhandlingofthesechallengeshere;moredetailsarein[20].TableIIshowshowweidentifytransactionsinthetracedata.Forinteractiveapplications,aninactivetime(userthinktime)followedbyanewuser'srequestdenotesanewtransaction.Atransactioniseitherapartialorentireow,whereowisdenedasalltrafcexchangedbetweentwoIPaddressesandportnumbers.Fortrafcrequiringmultipleows,suchasmediaorFTPtrafc,atransactionspansbothows.Weidentifyrequestsandresponsesusingthedataexchangebetweensendersandreceivers.LetAbeaclientthatinitiatessomeconversationwithserverB.ArequestisdenedasalldatapacketssentfromAtoB,beforeanydatapacketisreceivedfromB.AreplyisdenedasalldatapacketssentfromBtoA,beforeanynewrequestfromA.Fig.1illustratesrequestandreplyidentication,andmeasurementofpartialdelay,echodelayandwholedelay. Application Transaction email(srv/srv),Usenet TCPow chat,Web,telnet,email(usr/srv) TCPowandinactive�4s FTP TCPowandinactive�4s onbothcontrolanddatachannel games UDPowandinactive�4s DNS,ping Onerequest/responseexchange withuniquerequestID audioandvideo TCPow(controlchannel)and matchingUDPow(mediatrafc) TABLEIITRANSACTIONIDENTIFICATION. Fig.1.Illustrationofrequest/responseidentication.C.DoSMetricsWeaggregatethetransactionsuccess/failuremeasuresintoseveralintuitivecompositemetrics.Percentageoffailedtransactions(pft)perapplicationtype.ThismetricdirectlycapturestheimpactofaDoSattackonnetworkservicesbyquantifyingtheQoSexperiencedbyendusers.Foreachtransactionthatoverlapswiththeattack,weevaluatetransactionsuccessorfailureapplyingdenition3.Astraightforwardapproachtothepftcalculationisdividingthenumberoffailedtransactionsbythenumberofalltransactionsduringtheattack.Thisproducesbiasedresultsforclientsthatgeneratetransactionsserially.Ifaclientdoesnotgenerateeachrequestinadedicatedthread,timingofsubsequentrequestsdependsonthecompletionofpreviousrequests.Inthiscase,transactiondensityduringanattackwillbelowerthanwithoutanattack,sincetransactionsoverlappingtheattackwilllastlonger.Thisskewsthepftcalculationbecauseeachsuccessorfailurehasahigherinuenceonthepftvalueduringanattackthaninitsabsence.Inourexperiments,IRCandtelnetclientssufferedfromthisdeciency.Toremedythisproblem,wecalculatethepftvalueasthedifferencebetween1(100%)andtheratioofthenumberofsuccessfultransactionsdividedbythenumberofalltransactionsthatwouldhavebeeninitiatedbyagivenapplicationduringthesametimeiftheattackwerenotpresent.TheDoS-histmetricshowsthehistogramofpftmeasuresacrossapplications,andishelpfultounderstandeachapplica-tion'sresiliencetotheattack.TheDoS-levelmetricistheweightedaverageofpftmea-suresforallapplicationsofinterest:DoS-level=Pkpft(k)
wk,wherekspansallapplicationcategories,andwkisaweightassociatedwithacategoryk.WeintroducedthismetricsbecauseinsomeexperimentsitmaybeusefultoproduceasinglenumberthatdescribestheDoSimpact.But 5wecautionthatDoS-levelishighlydependentonthechosenapplicationweightsandthuscanbebiased.QoS-ratioistheratioofthedifferencebetweenatrans-action'strafcmeasurementanditscorrespondingthreshold,dividedbythisthreshold.TheQoSmetricforeachsuccessful transactionshowstheuser-perceivedservicequality,intherange(0;1],wherehighernumbersindicatebetterquality.Itisusefultoevaluateservicequalitydegradationduringattacks.WecomputeitbyaveragingQoS-ratiosforalltrafcmeasure-mentsofagiventransactionthathavedenedthresholds.Forfailed transactions,wecomputetherelatedQoS-degrademetric,toquantifyseverityofservicedenial.QoS-degradeistheabsolutevalueofQoS-ratioofthattransaction'smeasurementthatexceededitsQoSthresholdbythelargestmargin.Thismetricisintherange[0;+1).Intuitively,avalueNofQoS-degrademeansthattheserviceoffailedtransactionswasNtimesworsethanausercouldtolerate.Whilearguablyanydenialissignicantandthereisnoneedtoquantifyitsseverity,perceptionofDoSishighlysubjective.LowvaluesofQoS-degrade(e.g.,1)maysignifyservicequalitythatisacceptabletosomeusers.Thelifediagramshowsthebirthanddeathofeachtrans-actionintheexperimentwithhorizontalbars.Thex-axisistimeandthebarpositionshowsatransaction'sbirth(startofthebar)anddeath(itsend).Weshowfailedandsuccessfultransactionsonseparatediagrams,forclarity.Thismetriccanhelpquicklyshowwhichtransactionsfailedandindicateclustersthatmaypointtoacommoncause.Thefailureratioshowsthepercentageoflivetransactionsinthecurrent(1-second)intervalthatwillfailinthefuture.ThefailureratioisusefulforevaluationofDoSdefenses,tocapturethespeedofadefense'sresponse,andfortime-varyingattacks[2].Transactionsthatarebornduringtheattackareconsideredliveuntiltheycompletesuccessfullyorfail.Transactionsthatarebornbeforetheattackareconsideredliveaftertheattackstarts.Afailedtransactioncontributestothefailedtransactioncountinallintervalswhereitwaslive.IV.EVALUATIONINTESTBEDEXPERIMENTSWerstevaluateourmetricsinexperimentsontheDETERtestbed[21].ThetestbedislocatedattheUSCInformationSciencesInstituteandUCBerkeley,andallowssecurityre-searcherstoevaluateattacksanddefensesinacontrolledenvironment.A.TopologyFig.2showsourexperimentaltopology.Fourlegitimatenetworksandtwoattacknetworksareconnectedviafourcorerouters.Eachlegitimatenetworkhasfourservernodesandtwoclientnodes,andisconnectedtothecoreviaanaccessrouter.Linksbetweentheaccessrouterandthecorehave100Mbpsbandwidthand10–40msdelay,whileotherlinkshave1Gbpsbandwidthandnoaddeddelay.Thelocationofbottlenecksischosentomimichigh-bandwidthlocalnetworksthatconnectoveralimitedaccesslinktoanover-provisionedcore.Attacknetworkshosttwoattackerseach,andconnectdirectlytocorerouters. Fig.2.Experimentaltopology.B.BackgroundTrafcEachclientgeneratesamixtureofWeb,DNS,FTP,IRC,VoIP,pingandtelnettrafc.Weusedopen-sourceserversandclientswhenpossibletogeneraterealistictrafcattheapplication,transportandnetworklevel.Forexample,weusedanApacheserverandwgetclientforWebtrafc,bindserveranddigclientforDNStrafc,etc.Telnet,IRCandVoIPclientsandtheVoIPserverwerecustom-builtinPerl.Clientstalkwithserversintheirownandadjacentnetworks.Fig.2showsthetrafcpatterns.TrafcpatternsforIRCandVoIPdifferbecausethoseapplicationclientscouldnotsupportmultiplesimultaneousconnections.AllattackstargettheWebserverinnetwork4andcrossitsbottlenecklink,soonlythisnetwork'strafcshouldbeimpactedbytheattacks.Ourpreviouswork[20]usedasimilarexperimentalsetuptoillustrateourmetricsinrealistictrafcscenariosforvariousattacks.Here,weshowadifferentsetofexperimentswithonenovelattackscenario(SectionIV-D).Wemodiedthetopologyfrom[20]toensurethatbottlenecksoccuronlybeforetheattacktarget,tocreatemorerealisticattackcon-ditions.Weusedamorearticialtrafcmixthanin[20],withregularservicerequestarrivalsandidenticallesizesforeachapplication,toclearlyisolateandillustratefeaturesofourmetrics.Trafcparametersarechosentoproducethesametransactiondensityineachapplicationcategory(TableIII):roughly100transactionsforeachapplicationduring1,300seconds,whichistheattackduration.Alltransactionssucceedintheabsenceoftheattack.C.UDPBandwidthFloodOurrstexperimentisaUDPoodattack,frequentlyusedintheliteratureandfrequentlyobservedintheInternet.Thisattackcandenyserviceintwoways:(1)bygeneratingalargetrafcvolumethatexhaustsbandwidthonbottlenecklinks(morefrequentvariant),(2)bygeneratingahighpacketratethatexhauststheCPUatarouterleadingtothetarget.Wegeneratetherstattacktype:aUDPbandwidthood.Packet 6 Type Parameter(unit) Distribution telnet Requestinterarrivaltime 10s Responsesize 4KB Sessionduration 60s Timebetweensessions(s) 15s FTP Requestinterarrivaltime 13s Filesize 10KB Web Requestinterarrivaltime 13s Filesize 1KB DNS Requestinterarrivaltime 13s ping Requestinterarrivaltime 13s IRC Requestinterarrivaltime 5s Messagesize 10KB VoIP Packetinterarrivaltime 0.03s Talktime 8s Thinktime 5s TABLEIIILEGITIMATETRAFFICPARAMETERSANDTHEIRVALUES. Fig.3.UDPbandwidthood:DoS-histandDoS-levelmeasures.sizeshadrange[750B,1.25KB]andtotalpacketratewas200Kpps.Thisgeneratesavolumethatisroughly16timesthebottleneckbandwidth.Theexpectedeffectisthataccesslinkofnetwork4willbecomecongestedandtrafcbetweennetworks1and4,andnetworks3and4willbedeniedservice.Fig.3showstheDoS-histmeasuresforallsourceanddestinationnetworks,andtheDoS-levelmeasureassuming Fig.4.UDPbandwidthood:QoSmeasuresforsuccessfultransactions. Fig.5.UDPbandwidthood:QoS-degrademeasuresforfailedtransactions. Fig.6.UDPbandwidthood:Failureratiofortransactionsfromnetwork4tonetwork1.equalapplicationweights.Labelsatthetopofthegraphshowmeasuresthatbelongtothesamesourcenetwork,x-axislabelsdenotethedestinationnetwork,andthey-axisshowsthepftperapplication.Asexpected,onlytrafctoandfromnetwork4isaffected.Transactionsbetweennetworks1and4havesomewhathigherpftthantransactionsbetweennetworks3and4.Asimilartrendisalsonoticeableinotherexperiments,andoccursbecausetrafcbetweennetworks1and4sharesonemorerouterwiththeattack(routerB)thandoestrafcbetween3and4(crossesCandDbutnotB).DoS-levelisaround98%fortrafcbetween1and4,andaround91%fortrafcbetween3and4.Fig.4showstheQoSmeasure,averagedoversuccessfultransactions.Servicequalitydegradesamongtransactionsin-volvingserversorclientsinnetwork4.Othertransactionshaveconsistentlyhighservicequality.TheQoS-degrademeasureisshowninFig.5,averagedoverfailedtransactions.Whileasinglelargevaluecouldbiasthismetric,valuesinourexperimentswerefairlybalancedoverfailedtransactionsinthesameapplicationcategory.Transactionswithnetwork4experiencelargeservicedenial,receivingaservicewith10-300timesworsequalitythanexpected.Fig.6showsthefailureratiofortransactionsoriginating 7 Fig.7.UDPbandwidthood:Lifediagramofsuccessfultransactionsfromnetwork4tonetwork1. Fig.8.UDPbandwidthood:Lifediagramoffailedtransactionsfromnetwork4tonetwork1.fromnetwork4tonetwork1.Throughouttheattack,thefailureratiovaluestayscloseto1,illustratingthatnearlyallservicebetweenthesetwonetworksisdenied.Fig.7and8showthelifediagramsofsuccessfulandfailedtransactions.Thex-axisshowsthestartandendtimeofatransaction,thebarlengthrepresentstransactionduration,andthey-axisshowsthetransactionID.WeassignconsecutiveIDstotransactionsofthesametype.Allfailuresoccurduringtheattack,andalltransactionsfailregardlessoftheirapplicationtype.OneWebtransactionsucceedsduringtheattackbecauseitobtainsenoughbandwidthbychanceincompetitionwiththeattack.Notethedifferenceintransactiondensityduringtheattackbetweentelnetandotherapplications(Fig.8).TelnetandIRCclientsinourexperimentsgeneratetransactionsseriallyandthustheirtransactiondensityreduceswhenanattackprolongstransactions.Wenowcontrastourmetricswiththelegacymetrics:transactionduration,request/responsedelay,throughput,lossandresourceallocation.SincetheUDPbandwidthoodisthesimplestformofDoSattackthatdeniesservicethroughexcessivecongestion,weexpectthatmanyexistingmetricswilldowellinpredictingtransactionfailure.Aneffectivemetricwouldhaveaclearseparationofvaluesforsuccessful Fig.9.UDPbandwidthood:Transactioncdfwithrespecttoloss;transactionsoriginatedbynetwork4withnetwork1. Fig.10.UDPbandwidthood:Transactioncdfwithrespecttoaveragedatathroughput;transactionsoriginatedbynetwork4withnetwork1.andforfailedtransactions.Thecumulativedistributionfunction(cdf)ofmaximumlosswithina5-secintervalforalltransactionsoriginatedbynetwork4withnetwork1isshowninFig.9.Wealsoshowthecdfinthebaselinecase.Baselinetransactionsallhavezeroloss,andareclusteredattheoriginonthegraph.Successfultransactionsalsoallhavezeroloss,andquiteafewfailedtransactionshavehighloss(between0.5and1).However,manyfailedtransactionshavezeroloss,asshowninareaAinthegure,andfailbecausetheirotherQoSrequirementsarenotmet.ThisoverlapbetweenvaluesforsuccessfulandfailedtransactionsmakesthelossmetricinsufcientforDoSmeasurement.Fig.10showsthecdfofaveragedatathroughput(controlpacketsarenotcounted)foralltransactionsduringtheattackandforthebaselinecase.Theattackclearlylowersthetrans-actionthroughput—manyfailedtransactionshavethroughputclosetozero(andzero,notshownonthelog-scalegraph)andallsuccessfultransactionshaveahigherthroughput.However,thereisonetransactionthatfaileddespitehighthroughput,shownintheareaBinthegure.ThiswasaWebtransactionthatmanagedtoquicklydeliverarequesttotheserver;therequestwasacknowledgedbutthedatareplywaslost. 8 Fig.11.UDPbandwidthood:Transactioncdfwithrespecttorequest/replydelay;transactionsoriginatedbynetwork4withnetwork1. Fig.12.UDPbandwidthood:Transactioncdfwithrespecttoduration;transactionsoriginatedbynetwork4withnetwork1.Becausetransactionscanfailandstillhavehighthroughput,thethroughputmetricbyitselfcannotaccuratelymeasureDoS.Thecdfofrequest/replydelayforalltransactionsisshowninFig.11,duringtheattackandforthebaselinecase.Manyfailedtransactionshavehighdelaybutthereisasigni-cantoverlapindelayvaluesbetweenfailedandsuccessfultransactions,intheareaCinthegure.Thisoverlapmakesrequest/replydelayinsufcientforDoSmeasurement.Fig.12showsthecdfoftransactiondurationforalltrans-actionsduringtheattack,andforthebaseline.Theattackprolongsdurations,andsuccessfultransactionsnishsoonerthanfailedones.Thereisanarrowbutclearseparationofvalues,intheareaDinthegure.ThusthedurationmetriccouldpredictDoSinthisparticularexperimentbutwewillshowitfailsinexperimentswithotherattacktypes.Consideringresourceallocation,around97%ofbandwidthonnetwork4'saccesslinkwasconsumedbytheattack.ThisisclosetosomeDoS-histandDoS-levelvaluesfortransactionsbetweennetworks1and4,inFig.3,buthigherthantheDoSimpactontransactionsbetweennetworks3and4.Hence,resourceallocationmetricindicatesDoSimpactinthiscase,butisnotcompletelyaccurateinpredictingitsseverity.Theremainingexperimentsdiscussasubsetofthemetrics. Fig.13.UDPbandwidthood—low-rate:DoS-histandDoS-levelmeasures. Fig.14.UDPbandwidthood—low-rate:Transactioncdfwithrespecttoaveragethroughput;transactionsoriginatedbynetwork3withnetwork4.D.UDPBandwidthFlood—LowRateWenowillustratetheinadequacyofmetricsthatwereadequateforhigh-rateUDPbandwidthoods—duration,resourceallocation,andthroughput(ifweignoretheonefailedtransactionwithhighthroughput).WereducetherateoftheUDPoodattackto80%ofthebottlenecklinkbandwidth.Fig.13showstheDoS-histandDoS-levelmeasures.Traf-ctoandfromnetwork4suffersservicedenial,butthepercentageofimpairedtrafcvariesgreatlydependingonapplication.Webtransactionssuffertheleastservicedenial(8-20%),followedbytelnet,DNSandping.FTPislessimpactedwhentheserverisinnetwork4thanwhentheclientsarethere,becauseourFTPtransactionsaredownloads,somostdataowsfromservertoclient.ThisisalsowhyFTPsuffersmorethanWeb,telnet,DNSandpingwhennetwork4isthesourcenetwork.About50%ofIRCtransactionsfailand100%ofVoIPtransactionsfail.FortheQoS-degrademetric(notshowningraphduetospace),telnet,DNS,FTPandpingtrafchave10-100timesworseQoSthanrequired.Whentheserverisinnetwork4,Webtrafchas2-10timesdegradedserviceandVoIPhasonly0.15timesdegradation.Clearly,resourceallocationmetricscannotpredictsuchvariabilityinservicedenial:20%ofresourcesareallocatedtolegitimatetrafc. 9 Fig.15.UDPbandwidthood—low-rate:Transactioncdfwithrespecttoduration;transactionsoriginatedbynetwork3withnetwork4.Fig.14showsthecdfofaveragethroughputforalltransac-tionsduringtheattackinitiatedbynetwork3withnetwork4,andforthecorrespondingbaselinecase.Thereisasignicantoverlapofthroughputvaluesforsuccessfulandfailedtrans-actionsintheareaE,whichshowsthatthroughputbyitselfcannotaccuratelymeasureDoS.Fig.15showsthecdfofdurationforalltransactionsinitiatedbynetwork3withnetwork4duringtheattack,andforthecorrespondingbaselinecase.DurationsoffailedandsuccessfultransactionsoverlapinareaFinthegraph,showingthatdurationbyitselfcannotaccuratelymeasureDoS.Lossandrequest/replydelaydonotadequatelycaptureDoSimpactduetoalargeoverlapinvaluesforfailedandsuccessfultransactions.Weomitthesegraphsduetospace.E.TCPSYNFloodwithSyn-cookieDefenseAnotherpopularattackwithbothattackersandresearchersistheTCPSYNood[4].ItdeniesservicebysendingaTCPSYNoodthatconsumesOSmemoryatthetarget.ThisattackcanbelargelycounteredifthetargetdeploystheTCPsyn-cookiedefense[22],whichallocatesmemoryonlyafterthe3-wayhandshakeiscompleted.Sinceattackersdonotcompletethehandshake,theattackisthwarted.WegeneratedaTCPSYNoodtoport80ontheWebserverinnetwork4,sending500pps.Weturnedsyn-cookieson650secondsafterthestartofattack,attime715seconds.TheDoS-histandDoS-levelmeasuresareshowninFig.16.Asexpected,alltrafctonetwork4'sWebserversuffersservicedenial.Theseverityisaround50%,inlinewiththeexpectationthatalmostalltransactionsweredeniedservicebeforethesyn-cookiedefensewasturnedon,andnoneaf-terward.ThereisaslightDoSfortheVoIPtrafcfromnetwork1tonetwork4,when1outof100transactionsfailsbecauseofexcessiveloss.ThelossisduetoaggressiveTCPretransmissionsandisminor(3.3%)buthigherthanthe3%QoSthresholdforVoIP.Fig.17showstheWebtransactionfailureratiofromnetwork1tonetwork4.Duringtheattack,thevaluegoesto1,butrevertstozerowhensyn-cookiesaredeployed. Fig.16.TCPSYNoodwithsyn-cookies:DoS-histandDoS-levelmeasures. Fig.17.TCPSYNoodwithsyn-cookies:FailureratioforWebtrafcfromnetwork1tonetwork4.ThelifediagramsofsuccessfulandfailedtransactionsareshowninFig.18.OnlyWebtransactionsfailduringtheattack,andonlyduringaperiodwhensyn-cookiesareoff.Wesummarizelegacymetrics,forspacereasons.Dura-tionandlossmetricscaptureDoSimpactwellinthiscase,butthroughputandrequest/replydelayproduceoverlappingregionsforfailedandsuccessfulWebtransactionsandthuscannotmeasureDoSaccurately.Only18%ofbandwidthisconsumedbytheSYNood,yet100%ofwebtransactionsaredeniedservicewhensyncookiesareoff.ThebetterapproachtoresourceallocationmeasurementwouldbetomeasureoccupancyoftheTCPconnectiontableattheWebserver.WelackedtoolstoobtainthisinformationeasilyfromtheOS,butweinferfromtheWebtransactionsuccess/failuremetricsthatthetablewouldmostlybeoccupiedbyattackconnections.V.EVALUATIONINNS-2SIMULATIONSToextendtheapplicationofourproposedmetricstosimu-latedDDoSdefenseevaluation,wehaveportedthemetricstothepopularNS-2simulator[23].WeillustratetheDoSimpactmetricsinsmall-scaleexperimentsusingNS-2(version2.29),andcomparetheresultswithidenticalexperimentsontheDETERtestbed.Duringsimulations,wegenerateowsthat 10 Fig.18.TCPSYNoodwithsyn-cookies:Lifediagramofsuccessfulandfailedtransactionsfromnetwork1tonetwork4. (a)NS-2 (b)DETERFig.19.DoS-histandDoS-levelmeasuresinNS-2andDETERexperiments.eachrepresentatransactionandwecomputerequiredtrafcmeasurementsfromNS-2logs.Weuseasimplenetworktopologywithasinglelegitimateclient,anattacker,andaserver.Allnodesareconnectedtothesamerouter.Thelinkbetweentheserverandrouteris10Mbpswith10msdelay.Theothertwolinksare100Mbpsbandwidthwith10msdelay.Weuseaqueuesizeof100packets,withadrop-tailqueuingstrategy.Wegeneratethefollowinglegitimatetrafcbetweentheclientandtheserver:(1)WebandFTPtrafcwithlesize1000bytesand20srequestinterarrivalperiod.(2)Telnettrafcwith10ppsanda100-bytepacketsize.Duringthesimulation,westartanewtelnetconnectionevery60swithdurationof120s.(3)DNSandpingtrafcwith10srequestinterarrivalperiod.WeusethefollowingmodulesinNS-2togeneratethetrafc:Application/FTPforFTP,PagePool/WebTrafforWeb,Application/Telnetfortelnet,Agent/PingforICMP,andamodiedversionofAgent/Pingwithamaximumof3retransmissionswith5-stimeoutsforDNS.WegenerateaUDPoodthatoverwhelmsthebottlenecklinkwith10Mbps(moderateattack)or80Mbps(largeattack)rate.Fig.19showstheDoS-histmeasurefortheclient'strafctotheserverduringthetwoattacksfortheNS-2andDETERexperiments,andinno-attackcase.Thex-axisshowstheattackstrength,andthecolumnheightdenotestheresultof10testruns,witherrorbarsshown.SincethelegitimatetrafcpatternisxedfortheNS-2simulation,weachievevariabilitybyrandomlychoosingasmalldelay(10-100ms)toapplytotheattackstarttime.Thetrafcpatternintestbedexperimentsde-pendsonarandomseed.WealsoshowtheDoS-levelmeasureusingequalapplicationweights.Thetelnetapplicationisthemostaffectedbytheattackduetoitssmallecho-delaybound(250ms).DenialofserviceissimilarforDNSandping,eventhoughDNScanretransmitrequestsuptothreetimes,becausetheseretransmissionsoccuraftertheDNSrequest/responsedelaythresholdisexceeded(4s).Webtransactionssurvivetheattackbestbecauseofthegenerous(4s)delaythresholdandbecausethelostpacketsareretransmittedbyTCP.Athighattackrate(80Mbps),thepftofallapplicationsgoestoalmost100%.Comparingsimulationresultswithtestbedresults(Fig.19(a)vs19(b)),wendthattrendsinbothgraphsaresimilarbutmoretransactionsfailinsimulations.ThisisbecausethesoftwareroutersusedonthetestbedcanhandletheattacktrafcmuchbetterthanthesimpleoutputqueuingmodelusedinNS-2.Theresultsareconsistentwith[24],whichshowsmuchhigherthroughputandTCPcongestionwindowsizesintestbedexperimentscomparedtothesameexperimentsinNS-2.VI.EVALUATIONINHUMAN-USEREXPERIMENTSToevaluateourmetrics'abilitytocaptureahumanuser'sperceptionofservicedenial,wehaveconductedanexperimentwhereusersinteractwithaserverthatisoccasionallysubjectedtodenial-of-serviceattacks.Aftereachinteraction,auserrateshersatisfactionwithservicequalityandwecomparethisratingwithtwoofourdenial-of-servicemetrics:thetransactionsuccess/failuremetricandtheQoSmetric. 11 Fig.20.TopologyforHuman-userDoSexperiments.A.ServiceandContentWelimitedlegitimatetrafctoasingleapplication,Webbrowsing,tosimplifyuserinteractionwiththeserverandfacilitatewideparticipationinourexperiment.UsersinteractwiththeserverbybrowsingthroughasetofWebpages.TheyratetheirsatisfactionwiththeloadingspeedofeachpagebyllinginaWebformshowntotheleftofthepage,inaseparateframe.Wewantedtoprovideinterestingandcopyright-freecontenttoattractparticipantsandachievereasonablylonginterac-tionswiththeserver.Wedownloaded21selectpagesfromWikipedia[25],whichisahighlypopularonlineencyclopediathatallowscontentcopyingandmodicationunderthetermsoftheGNUFreeDocumentationLicense.These21pagesweregroupedintofourcontentcategories:Sports(geocaching,abseiling,aerobatics,fellrunning,Chileanrodeo,paintball),Music(blues,hiphop,rockandroll,heavymetal,therelation-shipbetweenmusicandmathematics),Film(StarWars,God-father,LordoftheRings,Casablanca,AnInconvenientTruth)andFamousPeople(WaltDisney,Shakespeare,ChristopherColumbus,BenjaminFranklin,Mozart).Wemodiedeachpagetotinto1-2screensoftext.B.ExperimentSetupBecausetheWebserverhadtobesubjectedtooccasionalDoSattacks,weneededacontrolled,isolatedenvironmentsuchastheDETERtestbed[21].However,ourdesiretoattractmanysurveyparticipantsdictatedtheneedforexperimentalmachinestobereachablebyusersfromoutsidethetestbed.DETERcurrentlyprohibitsanycommunicationbetweenexter-nalmachinesandexperimentalones,andthuscouldnothostourexperiment.Instead,weusedtheEmulabtestbed[26],whichissimilartotheDETERtestbedbutitallowsexternalWebrequeststoexperimentalnodes.AnaiveexperimentaltopologywoulduseoneWebserverintheEmulabtestbed,andtheusertrafcwouldreachtheserverdirectly.Suchatopologywasinadequateforourpurposes,forthefollowingreasons:(1)Usersmustreachtwotypesofpages(a)WikipediacontentonaserverthatmaybeaDoStarget,(b)welcomeandthank-youpages,andpageswithQoSratingformsthatmustalwaysbeloadedpromptlyregardlessofanattack.WeusedtwoWebservers—onetohostcontrolinformationforthesurvey(welcome,thank-youpagesandratingforms)andonetohostthecontentandbetheDoStarget.(2)ForDoSattacksthattargetbandwidth,usertrafcmustsharethebottlenecklinkwiththeattack.Thus,userWebrequestsmustbetunneledtothecontentWebserverinsteadofreachingitdirectly.Fig.20showsthetopologyusedintheexperiment.UsertrafcrstreachestheWebserverControl,whichhostscontrolinformation.Whenthesurveystarts,therightframeofthepagesdisplayedtotheuserpointstothehostNAT,whichactsasanetworkaddresstranslatorandtunnelstheuser'sWebrequeststotheWebserverContentoverthebottlenecklink,whichissharedwithattacktrafcfromhostsA1andA2.MachineR1isanaggregationrouterandmachineR2emulatesa10MbpslinkusingClick[27].Allphysicallinksare100Mbps.WeruntcpdumpforeachuseronthelinkleadingfromNATtoR1,anonymizingtheoutput.WeusethisoutputtocalculateourDoSmeasures.Therstpagedisplayedtoauseristheregistrationpage,withonlyonebuttonlabeled“Register.”AclickonthisbuttonassignsasequentialIDtotheuserandstartstcpdump;userratingswillbesavedunderthisIDandtcpdumpoutputwillbearthenamederivedfromtheID.Wenextgeneratearandomnumberintherange1–4.Number1triggersaUDPoodattackonthebottlenecklink,number2triggersthissameattackbutatasmallerrate,whichaimstodegradebutnottodenyservice.Number3triggersaSYNoodattackontheContentWebserverandnumber4doesnottriggeranyattack,i.e.,usersinthiscategoryformthecontrolgroup.Thenextpageisthewelcomepage,loadedfromControlserver,thatexplainsexperimentgoalsandsetup,andgivesinstructionstotheuser.Usersareaskedtoclickonatleast5pagesoftheirchoice,nottorepeatclicksandnottofollowexternallinksfromtheWikipediapages.Repeatedclickscanleadtoerroneousperceptionofservicequalitybecausetheydisplaythecontentfromthebrowser'scache,andexternal-pageclicksbypassthetestbed.Thewelcomepagealsodisplays21buttonsforthecontentpages,andabuttontoquitthesurvey.Clickingonacontentbuttongeneratesawebpagewithcontentontherightandtheratingformontheleft.IftheContentserverwereunderattack,thepageintherightframemaynotload,oritmaytakealongtimetoload.Usersratetheirsatisfactionwithservicequalityon1–4scale,where4means“Excellent”,3means“MostlyOK”,2means“Poor,butacceptable”and1means“Unacceptable.”Theratingformismultiple-choiceandallowsonlyasingleitemtobeselected.Theuserscanbrowsenaturally:theyareallowedtoratetheirsatisfactionatanytime;i.e.,theydidnothavetowaitforthepagetoloadcompletelyandtheydidnothavetoreadanycontent.Whentheuserclicksthe“Submit”buttonontheratingpage,thecontentlenameandtheratingaresavedinalogle.Aftertheratingissubmitted,thewelcomepageisdisplayedagain.Thesurveyendswhentheuserclicksthe“Quit”buttononthewelcomepage.Thethank-youpageisthendisplayedanduserratingsareshownsidebysidewithourDoSmetrics.WedescribetheprocessofmappingourmeasurestothesameratingscaleasusedbyhumanusersinSectionVI-C.Weexperimentedwiththefollowingattackdynamicsbefore 12settlingononeofthem:(1)Theattackstartsimmediatelyuponregistrationandlastsforalongtime.Thisgeneratespredictableresultsbecauseeitherallusertransactionsareaffectedbytheattackornoneare.Thisscenariowastoosimplistictovalidateourmetrics;wepreferredtohaveeachuserexperiencesomegoodandsomepoorservice.(2)TheattackstartswhenauserrequestsacontentpagedisplayingWikipediacontent.Thisscenariowouldbeidealbutithadtimingproblems.AWebrequestiscontainedinveryfewpacketssenttoserver—twotoopenaTCPconnectionandonetorequestaWebpage—andonlythispathisaffectedbytheattack.Requestpacketsaresentrapidlywhentheuserclicksonthecontentbutton.Iftheattackistriggeredsimultaneously,therewasaraceconditionbetweencreatingsufcientcongestionanduserpacketsreachingtheserver.If,ontheotherhand,wedelayedpageloaduntiltheattackhasstarted,thiswouldaffecttheuser'sperceptionofservicequalityandskewtheratingtowardlowervalues.(3)Theattackistriggeredupontheregistrationclick.Itrunsperiodically,thusagivenusermayexperiencesomehighqualityandsomelowqualitytransactions.Weoptedforthisscenariosinceitwasrichenoughtogenerateinterestingtestcasesforourmeasuresanddidnotsufferfromthetimingproblemspresentinscenario2.Eachattackstarts60secondsaftertheregistrationclick,lastsfor30secondsandrepeatsevery60secondsfortotalof10times.Wecarefullyselectedtheattackperiodanddurationtomaximizethechancethatattacktrafcoverlapswithuserrequests.Theattackisabortedwhentheuserquitsthesurvey.C.MappingourDoSMetricstoUser-CompatibleRatingsUserratingsofservicequalityareonthescale1–4,where2,3,4ratingsdenotesuccessfultransactionswithincreasingdegreesofusersatisfactionand1denotesfailedtransactions.TwoofourDoSmetricsarecomparablewithuserratings:thetransactionsuccess/failuremetricandtheQoSmetric.TheQoSmetricisonthescale(0;1]andiscalculatedonlyforsuccessfultransactions;wesetittozeroforfailedtransactions.HigherQoSvaluesdenotehigherservicequality.Wemappedtransactionsuccess/failureandtheQoSmeasureintothe1–4scaleasfollows(summarizedinTableIV).Ifatransactionfailed,ourratingofitsservicequalitywassetto1anditsQoSmeasurewassetto0.Ifitsucceeded,werunanoptimizationalgorithmtondthebestvaluesforthresholdsontheQoSmeasurethatdenotethelimitsbetweenratings2and3,andratings3and4.Forourexperimentalresults,thesethresholdswere0.87and0.88,respectively. QoS Rating 0 1 0but0.87 2 0.87but0.88 3 0.88 4 TABLEIVMAPPINGOFQoSMETRICTOUSERRATINGSCALE.Wealsohadtomaptransactionsintoclicks.Ourmeasuresarecalculatedpertransaction,whichincaseofWebservicemaydenotetheeventofinitiatingthecommunicationwiththeserver,partiallyorcompletelyloadingapage,orloadingeachembeddedobjectinapage.Thusoneuserclickusuallymapsintoseveraltransactions.WemapclicksintotransactionsbyrstidentifyingTCPconnectionsinthetcpdumpoutputassociatedwithoneWebpageload,thenrelatingourtransac-tionstotheseconnections(andthustopageloads),andnallypairingthepageloadswiththeuserclicksrecordedinourratinglogle,asexplainednext.IdenticationofTCPconnectionsassociatedwithonepageloadproceedsasfollows:(1)IdentifyTCPconnectionsinthecollectedtcpdumplebylookingfora3-wayhandshakeandallsubsequenttrafcbetweenthesameIPaddressesandportnumbersuntileitheraFINoraRESET.(2)IfaconnectioncontainsapacketwithanHTTPGETdirectiveinthecontenteld,parsethelenamefollowingthisdirective.Forlesendingin.htmlthisconnectiondenotesanewpageload.Forotherles,lookfortheReferereldinthepacketcontainingHTTPGET,andparsethenameofthereferringle,whichinourcasealwaysendsin.html.Thisconnectionisaddedtothepageloadofthereferringle.(3)IfaconnectiondoesnotcontainapacketwithanHTTPGETdirective,itisassociatedwitha“NOURL”pageload.Theseconnectionsusuallycontainapartialorfull3-wayhandshake,buttheservicedenialwassolargethattheconnectionneveradvancedtodataexchange.RelatingourtransactionstoTCPconnectionsinvolvedselectingtheTCPconnectionthathadthesameportnumbersasthegiventransactionandencompasseditsstartandendtimes.AfteralltransactionswerepairedwithTCPconnectionsandthuswithpageloads,wecalculatethesuccess/failureandtheQoSmeasureforeachpageload.Aload'ssuccess/failuremeasureisa“success”onlyifalltransactionsthataremappedtothisloadweresuccessful,otherwiseitisa“failure.”Aload'sQoSmeasureis0ifitssuccess/failuremeasureis“failure.”Otherwise,theQoSmeasureistheaverageofQoSmeasuresoftransactionsassociatedwiththispageload.PairingWebpageloadswithuserclicksfromtheratingloglewasperformedbypairingthelenamesfromtheloadswithURLsinthelogle.Ifwecannotndthenamefromthelogleamongourpageloads,wenextattempttopairthisclickwithour“NOURL”loadbasedontiming.Ifthisfails,theclickismarkedinvalid.Repeatedclicksarealsoconsideredinvalidbecausetheymaybeservedfromaclient'scache;anactioninvisibleinnetworktraces.D.ResultsWerecruitedexperimentparticipantsfromthefollowingpopulations:(1)graduatestudentsandfacultyattheUniversityofDelaware,(2)graduatestudentsatUCLA,(3)graduatestudentsatPurdueUniversity,(4)attendeesofSIGMETRICS2007,and(5)subscribersoftheTCCCmailinglist.Wekeptthesurveyopenforfourmonths(July-October2007)andhad101participantsand840clicks.32(3.8%)clickswereinvalid,leaving808validclicksfor100users.Assignmentofuserstoattackcategorieswasbalanced:23experiencedaUDPoodattack,28experiencedalow-rate 14DoSmetricforSkypeandotherVoIPtrafc.Inslow-motionbenchmarking[35],theauthorsusenetworktracescollectedattheclienttomeasureperformanceofthinclients.Theironlyperformancemeasureisthesumoftransactiondurationsinthebenchmark.VIII.CONCLUSIONSANDFUTUREWORKOnecannotunderstandacomplexphenomenonlikedenialofservicewithoutbeingabletomeasureitinanobjective,accurateway.Theworkdescribedhererepresentstherstattempttodeneaccurate,quantitativeandversatilemetricsformeasuringeffectivenessofdenialofserviceattacksanddefenses.Byfocusingontheissueofmeasuringhumanuserperceptionofapplication-levelservicequality,themetricscuttotheheartoftheproblemandavoidissuesofthespecicformoftheattackandlegitimatetrafcmix.Ourapproachisobjective,reproducible,andapplicabletoawidevarietyofattackanddefensemethodologies.Itsvaluehasbeendemonstratedinbothtestbedsandsimulationenvironments.Further,wehaveaddressedthemainconcernofmetricsthatfocusonanapplication-levelphenomenon–theaccuracyofthemetriccomparedtohumanperceptions–viatestswithhumansubjectsthatvalidatedourresults.Ourmetricsareusablebyotherresearchersintheirownwork.Theyoffertherstrealopportunitytocompareandcontrastdifferentdenialofserviceattacksanddefensesonanobjectivehead-to-headbasis.Weexpectthatthisworkwilladvancedenial-of-serviceresearchbyprovidingaclearmeasureofsuccessforanyproposeddefense,andhelpingresearchersgaininsightintostrengthsandweaknessesoftheirsolutions.WhileourDoSmetricsareanecessaryconditionforper-formancecomparisonofDoSdefenses,theyarenotsufcient.ArelatedproblemisdevisingstandardizedbenchmarksforDoSdefensetesting,soallproductsaretestedunderthesameconditions.Wehavedonesomepioneeringworkinthisarea[36]butoursisjustarst,smallstepandanengagementofawiderresearchcommunityisneededtocompletelyaddressthisproblem.REFERENCES[1]A.Yaar,A.Perrig,andD.Song.SIFF:AStatelessInternetFlowFiltertoMitigateDDoSFloodingAttacks.InProceedingsoftheIEEESecurityandPrivacySymposium,2004.[2]A.KuzmanovicandE.W.Knightly.Low-RateTCP-TargetedDenialofServiceAttacks(TheShrewvs.theMiceandElephants).InProc.ofACMSIGCOMM,August2003.[3]M.Guirguis,A.Bestavros,andI.Matta.ExploitingtheTransientsofAdaptationforRoQAttacksonInternetResources.InProceedingsofICNP,Oct2004.[4]CERTCC.CERTAdvisoryCA-1996-21TCPSYNFloodingandIPSpoongAttacks.http://www.cert.org/advisories/CA-1996-21.html,1996.[5]SrikanthKandula,DinaKatabi,MatthiasJacob,andArthurBerger.Botz-4-Sale:SurvivingOrganizedDDoSAttacksthatMimicFlashCrowds.InNSDI,2005.[6]HaniJamjoomandKangShin.PersistentDropping:AEfcientControlofTrafcAggregates.InACMSIGCOMMConference,2003.[7]X.Yang,D.Wetherall,andT.Anderson.ADoS-limitingNetworkArchitecture.InACMSIGCOMMConference,2005.[8]RatulMahajan,StevenM.Bellovin,SallyFloyd,JohnIoannidis,VernPaxson,andScottShenker.Controllinghighbandwidthaggregatesinthenetwork.InACMComputerCommunicationReview,July2001.[9]AngelosStavrou,AngelosD.Keromytis,JasonNieh,VishalMisra,andDanRubenstein.MOVE:AnEnd-to-EndSolutiontoNetworkDenialofService.InNDSS,2005.[10]G.Oikonomou,J.Mirkovic,P.Reiher,andM.Robinson.AFrameworkforCollaborativeDDoSDefense.InProceedingsofACSAC,December2006.[11]CooperativeAssociationforInternetDataAnalysis.CAIDAWebpage.http://www.caida.org.[12]WIDEProject.MAWIWorkingGroupTrafcArchive.http://tracer.csl.sony.co.jp/mawi/.[13]NortelNetworks.QoSPerformancerequirementsforUMTS.The3rdGenerationPartnershipProject(3GPP).http://www.3gpp.org/ftp/tsg sa/WG1 Serv/TSGS1 03-HCourt/Docs/Docs/s1-99362.pdf.[14]NinaBhatti,AnnaBouch,andAllanKuchinsky.QualityisintheEyeoftheBeholder:MeetingUsers'RequirementsforInternetQualityofService.TechnicalReportHPL-2000-4,HewlettPackard,2000.[15]L.YamamotoandJ.G.Beerends.Impactofnetworkperformanceparametersontheend-to-endperceivedspeechquality.InInProceedingsofEXPERTATMTrafcSymposium,September1997.[16]T.Beigbeder,R.Coughlan,C.Lusher,J.Plunkett,E.Agu,andM.Claypool.TheEffectsofLossandLatencyonUserPerformanceinUnrealTournament2003.InInProceedingsofACMNetGames2004.[17]NathanSheldon,EricGirard,SethBorg,MarkClaypool,andEmmanuelAgu.TheEffectofLatencyonUserPerformanceinWarcraftIII.InInProceedingsofACMNetGames2003.[18]B.N.ChunandD.E.Culler.User-centricPerformanceAnalysisofMarket-basedClusterBatchSchedulers.InInProceedingsofthe2ndIEEEInternationalSymposiumonClusterComputingandtheGrid,May2002.[19]J.Ash,M.Dolly,C.Dvorak,A.Morton,P.Taraporte,andY.E.Mghazli.Y.1541-QOSM–Y.1541QoSModelforNetworksUsingY.1541QoSClasses.NSISWorkingGroup,InternetDraft,Workinprogress,May2006.[20]J.Mirkovic,A.Hussain,B.Wilson,S.Fahmy,P.Reiher,R.Thomas,W.Yao,andS.Schwab.TowardsUser-CentricMetricsforDenial-Of-ServiceMeasurement.InInProceedingsoftheWorkshoponExperimentalComputerScience,June2007.[21]T.Benzel,R.Braden,D.Kim,C.Neuman,A.Joseph,K.Sklower,R.Ostrenga,andS.Schwab.ExperiencesWithDETER:ATestbedforSecurityResearch.In2ndIEEETridentComConference,March2006.[22]D.J.Bernstein.TCPsyncookies.http://cr.yp.to/syncookies.html.[23]TheNetworkSimulatorns2.NS-2Webpage.http://www.isi.edu/nsnam/ns/.[24]R.Chertov,S.Fahmy,andN.Shroff.EmulationversusSimulation:ACaseStudyofTCP-TargetedDenialofServiceAttacks.InProceedingsofthe2ndInternationalIEEECreateNetTridentComConference,February2006.[25]Wikipedia,theFreeEncyclopedia.http://www.wikipedia.com.[26]UniversityofUtah.Emulabtestbed.http://www.emulab.net.[27]E.Kohler,R.Morris,B.Chen,J.Jannotti,andM.F.Kaashoek.TheClickModularRouter.ACMTransactionsonComputerSystems,18(3):263–297,August2000.[28]TransactionProcessingPerformanceCouncil.TPCBenchmarks.http://www.tpc.org/information/benchmarks.asp.[29]StandardPerformanceEvaluationCorporation.SPECBenchmarksandPublishedResults.http://www.spec.org/benchmarks.html.[30]3GPP.The3rdGenerationPartnershipProject(3GPP).[31]M.W.Garrett.ServicearchitectureforATM:fromapplicationstoscheduling.IEEENetwork,10(3):6–14,May/June1996.[32]IRTFTMRGgroup.TheTransportModelingResearchGroup'sWebPage.http://www.icir.org/tmrg/.[33]KunchanLan,AleyaHussain,andDebojyotiDutta.TheEffectofMaliciousTrafcontheNetwork.InPassiveandActiveMeasurementWorkshop(PAM),April2003.[34]Kuan-TaChen,Chun-YingHuang,PollyHuang,andChin-LaungLei.QuantifyingSkypeUserSatisfaction.InProceedingsoftheACMSIGCOMM,September2006.[35]JasonNieh,S.JaeYang,andNaomiNovik.MeasuringThin-ClientPerformanceUsingSlow-MotionBenchmarking.ACMTransactionsonComputerSystems,21(1),February2003.[36]J.Mirkovic,S.Wei,A.Hussain,B.Wilson,R.Thomas,S.Schwab,S.Fahmy,R.Chertov,andP.Reiher.DDoSBenchmarksandExperimenter'sWorkbenchfortheDETERTestbed.InProceedingsofTridentcom,2007.