Catac ysm olicing Extreme Overloads in Internet Applications Bhuv an Urgaonkar Dept - PDF document

Cataclysm:PolicingExtremeOverloadsinInternetApplicationsBhuvanUrgaonkarDept.ofComputerScienceUniversityofMassachusettsAmehrst,MAbhuvan@cs.umass.eduPrashantShenoyDept.ofComputerScienceUniversityofMassachusettsAmherst,MAshenoy@cs.umass.eduABSTRACTInthispaperwepresenttheCataclysmserverplatformforhandlingex-tremeoverloadsinhostedInternetapplications.Theprimarycontributionofourworkistodevelopalowoverhead,highlyscalableadmissioncontroltechniqueforInternetapplications.Cataclysmprovidesseveraldesirablefeatures,suchasguaranteesonresponsetimebyconductingaccuratesize-basedadmissioncontrol,revenuemaximizationatmultipletime-scalesviapreferentialadmissionofimportantrequestsanddynamiccapacityprovi-sioning,andtheabilitytobeoperationalevenunderextremeoverloads.Cataclysmcantransparentlytrade-offtheaccuracyofitsdecisionmakingwiththeintensityoftheworkloadallowingittohandleincomingratesofseveraltensofthousandsofrequests/second.WeimplementaprototypeCataclysmhostingplatformonaLinuxclusteranddemonstratethebenetsofourintegratedapproachusingavarietyofworkloads.CategoriesandSubjectDescriptorsD.4.7[Software]:OperatingSystems—OrganizationandDesign;D.4.8[Software]:OperatingSystems—PerformanceGeneralTermsPerformance,Design,ExperimentationKeywordsInternetapplication,Overload,Sentry1.INTRODUCTIONDuringthepastdecade,therehasbeenadramaticincreaseinthepopularityofInternetapplicationssuchasonlinenews,onlineauc-tions,andelectroniccommerce.ItiswellknownthattheworkloadseenbyInternetapplicationsvariesovermultipletime-scalesandofteninanunpredictablefashionCertainworkloadvariationssuchastime-of-dayeffectsareeasytopredictandhandlebyappropriatecapacityprovisioning[10].Othervariationssuchasashcrowdsareoftenunpredictable.OnSeptember11th,2001,forinstance,theworkloadonapopularnewsWebsiteincreasedbyanorderofmagnitudeinthirtyminutes,withtheworkloaddoublingeverysevenminutesinthatperiod.Similarly,theloadone-commerceretailWebsitescanincreasedramaticallyduringthenaldaysofthepopularholidayseason.ThisresearchwassupportedinpartbyNSFgrantsCCR-9984030,CNS-0323597,andEIA-0080119.CopyrightisheldbytheInternationalWorldWideWebConferenceCom­mittee(IW3C2).Distributionofthesepapersislimitedtoclassroomuse,andpersonalusebyothers.WWW2005,May10­14,2005,Chiba,Japan.ACM1­59593­046­9/05/0005.Inthispaper,wefocusonhandlingextremeoverloadsseenbyInternetapplications.Informally,anextremeoverloadisascenariowheretheworkloadunexpectedlyincreasesbyuptoanorderofmagnitudeinafewtensofminutes.Ourgoalsare(i)todesignasystemthatremainsoperationaleveninthepresenceofanextremeoverloadandevenwhentheincomingrequestrateisseveraltimesgreaterthansystemcapacity,and(ii)tomaximizetherevenueduetotherequestsservicedbytheapplicationduringsuchanoverload.WeassumethatInternetapplicationsorservicesrunonahostingplatform—essentiallyaserverclusterthatrentsitsresourcestoap-plications.Applicationproviderspayforserverresources,andinturn,areprovidedperformanceguarantees,expressedintheformofaservicelevelagreement(SLA).Ahostingplatformcantakeoneormoreofthreeactionsduringanoverload:(i)addcapacitytotheapplicationbyallocatingidleorunder-usedservers,(ii)turnawayexcessrequestsandpreferentiallyserviceonly“important”requests,and(iii)degradetheperformanceofadmittedrequestsinordertoservicealargernumberofaggregaterequests.Wearguethatacomprehensiveapproachforhandlingextremeoverloadsshouldinvolveacombinationofalloftheabovetech-niques.Ahostingplatformshould,wheneverpossible,allocateadditionalcapacitytoanapplicationinordertohandleincreaseddemands.Theplatformshoulddegradeperformanceinordertotemporarilyincreaseeffectivecapacityduringoverloads.WhennocapacityadditionispossibleorwhentheSLAdoesnotpermitanyfurtherperformancedegradation,theplatformshouldturnawayex-cessrequests.Whiledoingso,theplatformshouldpreferentiallyadmitimportantrequestsandturnawaylessimportantrequeststomaximizeoverallrevenue.Forinstance,smallrequestsmaybepreferredoverlargerequests,ornancialtransactionsmaybepre-ferredovercasualbrowsingrequests.WepresentthedesignoftheCataclysmserverplatformtoachievethesegoals.CataclysmisspecicallydesignedtohandleextremeoverloadsinInternetapplicationsanddiffersfrompastworkintwosignicantrespects.First,sinceanextremeoverloadmayinvolverequestratesthatareanorderofmagnitudegreaterthanthecurrentlyallocatedca-pacity,theadmissioncontrollermustbeabletoquicklyexaminerequestsanddiscardalargefractionoftheserequests,whenneces-sary,withminimaloverheads.Thus,theefciencyoftheadmissioncontrollerisimportantduringheavyoverloads.Toaddressthisis-sue,weproposeverylowoverheadadmissioncontrolmechanismsthatcanscaletoveryhighrequestratesunderoverloads.Pastworkonadmissioncontrol[6,8,21,24]hasfocusedonthemechan-icsofpolicinganddidnotspecicallyconsiderthescalabilityofthesemechanisms.Inadditiontoimposingverylowoverheads,ourmechanismscanpreferentiallyadmitimportantrequestsduringanoverloadandtransparentlytrade-offtheaccuracyoftheirdecision Cataclysm servers for S1Cataclysm servers for S2Cataclysm Control Plane CataclysmNucleusrequests requestsApplication clients across the Internet CataclysmRequest PolicingLoad Balancing ProvisioningOS KernelNucleusFree Server PoolFigure1:TheCataclysmHostingPlatformArchitecture.makingwiththeintensityoftheworkload.Thetrade-offbetweenaccuracyandefciencyisanothercontributionofourworkanden-ablesourimplementationtoscaletoincomingratesofuptoafewtensofthousandsofrequests/s.Second,ourdynamicprovisioningmechanismemploysaG/G/1-basedqueuingmodelofareplicableapplicationinconjunctionwithonlinemeasurementstodynamicallyvarythenumberofserversal-locatedtoeachapplication.Anovelfeatureofourplatformisitsabilitytonotonlyvarythenumberofserversallocatedtoanappli-cationbutalsoothercomponentssuchastheadmissioncontrollerandtheloadbalancingswitches.Dynamicprovisioningofthelattercomponentshasnotbeenconsideredinpriorwork.WehaveimplementedaprototypeCataclysmhostingplatformonaclusteroftwentyLinuxservers.Wedemonstratetheeffective-nessofourintegratedoverloadcontrolapproachviaanexperimen-talevaluation.Ourresultsshowthat(i)preferentiallyadmittingrequestsbasedonimportanceandsizecanincreasetheutilityandeffectivecapacityofanapplication,(ii)ouradmissioncontrolishighlyscalableandremainsfunctionalevenforarrivalratesofafewthousandrequests/s,and(iii)oursolutionbasedonacombina-tionofadmissioncontrolanddynamicprovisioningiseffectiveinmeetingresponsetimetargetsandimprovingplatformrevenue.Therestofthispaperisorganizedasfollows.Section2providesanoverviewoftheproposedsystem.Sections3and4describethemechanismsthatconstituteouroverloadmanagementsolution.Section5describestheimplementationofourprototype.InSection6wepresenttheresultsofourexperimentalevaluation.Section7presentsrelatedworkandSection8concludesthispaper.2.SYSTEMOVERVIEWInthissection,wepresentthesystemmodelforourCataclysmhostingplatformandthemodelassumedforInternetapplicationsrunningontheplatform.2.1CataclysmHostingPlatformTheCataclysmhostingplatformconsistsofaclusterofcom-modityserversinterconnectedbyamodernLANtechnologysuchasgigabitEthernet.OneormorehighbandwidthlinksconnectthisclustertotheInternet.Eachnodeinthehostingplatformcantakeononeofthreeroles:cataclysmserver,cataclysmsentry,orcataclysmcontrolplane(seeFigure1).CataclysmServers:CataclysmserversarenodesthatrunInter-netapplications.Thehostingplatformmayhostmultipleapplica-tionsconcurrently.Eachapplicationisassumedtorunonasubsetofthenodes,andanodeisassumedtorunnomorethanoneappli-cationatanygiventime.Asubsetoftheserversmaybeunassignedandformthefreeserverpool.Thenumberofserversassignedtoanapplicationcanchangeovertimedependingonitsworkload.Eachserveralsorunsthecataclysmnucleus—asoftwarecomponentthatperformsonlinemeasurementsofapplication-specicresourceus-ages,whicharethenconveyedtotheothertwocomponentsthatwedescribenext.CataclysmSentry:Eachapplicationrunningontheplatformisassignedoneormoresentries.Asentryguardstheserversassignedtoanapplicationandisresponsiblefortwotasks.First,thesentrypolicesallrequeststoanapplication'sserverpool—incomingre-questsaresubjectedtoadmissioncontrolatthesentrytoensurethatthecontractedperformanceguaranteesaremet;excessrequestsareturnedawayduringoverloads.Second,eachsentryimplementsalayer-7switchthatperformsloadbalancingacrossserversallocatedtoanapplication.SincetherehasbeensubstantialresearchonloadbalancingtechniquesforclusteredInternetapplications[17],wedonotconsiderloadbalancingtechniquesinthiswork.CataclysmControlPlane:Thecontrolplaneisresponsiblefordynamicprovisioningofserversandsentriesinindividualapplica-tions.Ittrackstheresourceusagesonnodes,asreportedbycata-clysmnuclei,anddeterminestheresources(intermsofthenumberofserversandsentries)tobeallocatedtoeachapplication.Thecontrolplanerunsonadedicatedserveranditsscalabilityisnotofconcerninthedesignofourplatform.2.2ModelforInternetApplicationsTheInternetapplicationsconsideredinthisworkareassumedtobeinherentlyreplicable.Thatis,theapplicationisassumedtorunonaclusterofservers,anditisassumedthatrunningtheap-plicationonalargernumberofserversresultsinaneffectivein-creaseincapacity.Many,butbynomeansall,Internetapplica-tionsfallintothiscategory.VanillaclusteredWebserversareanexampleofareplicableapplication.Multi-tieredInternetapplica-tionsarepartiallyreplicable.Atypicalmulti-tieredapplicationhasthreecomponents:afront-endHTTPserver,amiddle-tierapplica-tionserver,andaback-enddatabaseserver.Thefront-endHTTPserveriseasilyreplicablebutisnotnecessarilythebottleneck.Themiddle-tier—afrequentbottleneck—canbeimplementedindif-ferentways.Onepopulartechniqueistouseserver-sidescript-ingsuchasApache'sphpfunctionality,ortousecgi-binscript-inglanguagessuchasperl.Ifthescriptsarewrittencarefullytohandleconcurrency,itispossibletoreplicatethemiddle-tieraswell.MorecomplexapplicationsuseJavaapplicationserverstoimplementthemiddle-tier.DynamicreplicationofJavaapplicationserversismorecomplexandtechniquesfordoingsoarebeyondthescopeofthispaper.Dynamicreplicationofback-enddatabasesisanopenresearchproblem.Consequently,mostdynamicreplica-tiontechniquesintheliterature,includingthiswork,assumethatthedatabaseissufcientlywellprovisionedanddoesnotbecomeabottleneckevenduringoverloads.GivenareplicableInternetapplication,weassumethattheap-plicationspeciesthedesiredperformanceguaranteesintheformofaservicelevelagreement(SLA).AnSLAprovidesadescriptionoftheQoSguaranteesthattheplatformwillprovidetotheapplica-tion.TheSLAweconsiderinourworkisdenedasfollows:AvgresptimeRofadmreq=8R1ifarrivalrate2[0;1)R2ifarrivalrate2[1;2):::Rkifarrivalrate2[k1;1)(1) ArrivalrateAvg.resp.timeforadmittedrequests10001sec1000-100002sec�100003secTable1:Asampleservice-levelagreement.TheSLAspeciestherevenuethatisgeneratedbyeachrequestthatmeetsitsresponsetimetarget.Table1illustratesanexampleSLA.EachInternetapplicationconsistsofL(L1)requestclasses:C1;:::;CL.Eachclasshasanassociatedrevenuethatanadmit-tedrequestyields—requestsofclassC1areassumedtoyieldthehighestrevenueandthoseofCLtheleast.Thenumberofre-questclassesLandthefunctionthatmapsrequeststoclassesisapplication-dependent.Toillustrate,anonlinebrokerageWebsitemaydenethreeclassesandmaymapnancialtransactionstoC1,othertypesofrequestssuchasbalanceinquiriestoC2,andca-sualbrowsingrequestsfromnon-customerstoC3.Anapplication'sSLAmayalsospecifylowerboundsontherequestarrivalratesthatitsclassesshouldalwaysbeabletosustain.3.CATACLYSMSENTRYDESIGNInthissection,wedescribethedesignofacataclysmsentry.Thesentryisresponsiblefortwotasks—requestpolicingandloadbal-ancing.Asindicatedearlier,theloadbalancingtechniqueusedinthesentryisnotafocusofthiswork,andweassumethesentryemploysalayer-7loadbalancingalgorithmsuchas[17].Therstkeyissuethatdrivesthedesignoftherequestpoliceristomaxi-mizetherevenueyieldedbytheadmittedrequestswhileprovidingthefollowingnotionofclass-baseddifferentiationtotheapplica-tion:eachclassshouldbeabletosustaintheminimumrequestratespeciedforitintheSLA.Givenourfocusonextremeoverloads,thedesignofthepolicerisalsoinuencedbythesecondkeyis-sueofscalability—ensuringverylowoverheadadmissioncontroltestsinordertoscaletoveryhighrequestarrivalratesseenduringoverloads.Thissectionelaboratesonthesetwoissues.3.1RequestPolicingBasicsThesentrymapseachincomingrequesttooneoftheclassesC1;:::;CL.Thepolicerneedstoguaranteetoeachclassanad-missionrateequaltotheminimumsustainableratedesiredbyit(recallourSLAfromSection2).Itdoessobyimplementingleakybuckets,oneforeachclass,thatadmitrequestsconrmingtotheserates.Requestsconrmingtotheseleakybucketsareforwardedtotheapplication.Leakybucketscanbeimplementedveryef-ciently,sodeterminingifanincomingrequestconrmstoaleakybucketisaninexpensiveoperation.Requestsinexcessoftheseratesundergofurtherprocessingasfollows.Eachclasshasaqueueassociatedwithit(seeFigure2);incomingrequestsareappendedtothecorrespondingclass-specicqueue.RequestswithineachclasscanbeprocessedeitherinFIFOorderorinorderoftheirservicetimes.Intheformercase,allrequestswithinaclassareassumedtobeequallyimportant,whereasinthelattercasesmallerrequestsaregivenpriorityoverlargerrequestswithineachclass.Admittedrequestsarehandedtotheloadbalancer,whichthenforwardsthemtooneofthecataclysmserversintheapplication'sserverpool.Thepolicerincorporatesthefollowingtwofeaturesinitspro-cessingoftherequeststhatareinexcessoftheguaranteedratestomaximizerevenue.(1)Thepolicerintroducesdifferentamountsofdelayinthepro-cessingofnewlyarrivedrequestsbelongingtodifferentclasses.Specically,requestsofclassCiareprocessedbythepoliceronceeveryditimeunits(d1=0d2:::dL);requestsar-rivingduringsuccessiveprocessinginstantswaitfortheirturnintheirclass-specicqueues.Thesedelayvalues,determinedperiod-ically,arechosentoreducethechanceofadmittinglessimportantrequestsintothesystemwhentheyarelikelytodenyservicetomoreimportantrequeststhatarriveshortlythereafter.InSection3.4weshowhowtopickthesedelayvaluessuchthattheproba-bilityofalessimportantrequestbeingadmittedintothesystemanddenyingservicetoamoreimportantrequestthatarriveslaterremainssufcientlysmall.(2)Thepolicerprocessesqueuedrequestsinthedecreasingor-derofimportance—requestsinC1aresubjectedtotheadmissioncontroltestrst,andthenthoseinC2andsoon.DoingsoensuresthatrequestsinclassCiaregivenhigherprioritythanthoseinclassCj,j�i.Theadmissioncontroltest—whichisdescribedindetailinthenextsection—admitsrequestssolongasthesystemhassuf-cientcapacitytomeetthecontractedSLA.Notethat,ifrequestsinacertainclassCifailtheadmissioncontroltest,allqueuedre-questsinlessimportantclassescanberejectedwithoutanyfurthertests.ethattheaboveadmissioncontrolstrategymeetsoneofourtwogoals—itpreferentiallyadmitsonlyimportantrequestsduringanoverloadandturnsawaylessimportantrequests.However,thestrategyneedstoinvoketheadmissioncontroltestoneachindi-vidualrequest,resultinginacomplexityofO(r),whereristhenumberofqueueduprequests.Further,whenrequestswithinaclassareexaminedinorderofservicetimesinsteadofFIFO,thecomplexityincreasestoO(r
log(r))duetotheneedtosortre-quests.Sincetheincomingrequestratecanbesubstantiallyhigherthancapacityduringanextremeoverload,runningtheadmissioncontroltestoneveryrequestorsortingrequestspriortoadmissioncontrolmaybesimplyinfeasible.Consequently,inwhatfollows,wepresenttwostrategiesforverylowoverheadadmissioncontrolthatscalewellduringoverloads.Wenotethatanewlyarrivingrequestimposestwotypesofcom-putationaloverheadsonthepolicer—(i)protocolprocessingand(ii)theadmissioncontroltestitself.Clearly,boththesecompo-nentsneedtoscaleforeffectivehandlingofoverloads.Whenproto-colprocessingstartsbecomingbottleneck,werespondbyincreas-ingthenumberofsentriesguardingtheoverloadedapplication—atechniquethatwedescribeindetailinSection4.2.Inthissectionwepresenttechniquestodealwiththescalabilityoftheadmissioncontroltest.3.2EfcientBatchProcessingOnepossibleapproachforreducingthepolicingoverheadistoprocessrequestsinbatches.Requestarrivalstendtobeveryburstyduringsevereoverloads,withalargenumberofrequestsarrivinginashortdurationoftime.Theserequestsarequeuedupintheap-propriateclass-specicqueuesatthesentry.Ourtechniqueexploitsthisfeaturebyconductingasingleadmissioncontroltestonanen-tirebatchofrequestswithinaclass,insteadofdoingsoforeachindividualrequest.Suchbatchprocessingcanamortizetheadmis-sioncontroloverheadoveralargernumberofrequests,especiallyduringoverloads.Toperformefcientbatch-basedadmissioncontrol,wedenebbucketswithineachrequestclass.Eachbuckethasarangeofrequestservicetimesassociatedwithit.Thesentryestimatestheservicetimeofarequestandthenhashesitintothebucketcor-respondingtothatservicetime.Toillustrate,arequestwithan estimatedservicetimeintherange(0;s1]ishashedtobucket1,thatwithservicetimeintherange(s1;s2]tobucket2,andsoon.Bydeninganappropriatehashingfunction,hashingarequesttoabucketcanbeimplementedefcientlyasaconstanttimeoperation.Bucket-basedhashingismotivatedbytworeasons.First,itgroupsrequestswithsimilarservicetimesandenablesthepolicertocon-ductasingleadmissioncontroltestbyassumingthatallrequestsinabucketimposesimilarservicedemands.Second,sincesuc-cessivebucketscontainrequestswithprogressivelylargerservicetimes,thetechniqueimplicitlygivesprioritytosmallerrequests.Moreover,nosortingofrequestsisnecessary—thehashingimplic-itly“sorts”requestswhenmappingthemintobuckets.









dddsilverbronzegoldclass goldclass silverClassifierLeaky bucketsClassspecific queuesAdmission controlFigure2:Workingofthecataclysmsentry.First,theclassarequestbelongstoisdetermined.Iftherequestconrmstotheleakybucketforitsclass,itisadmittedtotheapplicationwith-outanyfurtherprocessing.Otherwise,itisputintoitsclass-specicqueue.Theadmissioncontrolprocessestherequestsinvariousqueuesatfrequenciesgivenbytheclass-specicde-lays.Arequestisadmittedtotheapplicationifthereisenoughcapacity,elseitisdropped.Whentheadmissioncontrolisinvokedonarequestclass,itcon-siderseachnon-emptybucketinthatclassandconductsasingleadmissioncontroltestonallrequestsinthatbucket(i.e.,allre-questsinabucketaretreatedasabatch).Consequently,nomorethanbadmissioncontroltestsareneededwithineachclass,oneforeachbucket.SincethereareLrequestclasses,thisreducesthead-missioncontroloverheadtoO(b
L),whichissubstantiallysmallerthantheO(r)overheadforadmittingindividualrequests.Havingprovidedtheintuitionbehindbatch-basedadmissioncon-trol,wediscussthehashingprocessandtheadmissioncontroltestindetail.Inordertohasharequestintoabucket,thesentrymustrstestimatetheinherentservicetimeofthatrequest.Theinherentservicetimeofarequestisthetimeneededtoservicetherequestonalightlyloadedserver(i.e.,whentherequestdoesnotseeanyqueuingdelays).TheinherentservicetimeofarequestRisde-nedtobeSinherent=Rcpu+
Rdata(2)whereRcpuisthetotalCPUtimeneededtoserviceR,RdataistheIOtimeoftherequest(whichincludesthetimetofetchdatafromdisk,thetimetherequestisblockedonadatabasequery,thenetworktransfertime,etc.),andisanempiricallydeterminedconstant.Theinherentservicetimeisthenusedtohashtherequestintoanappropriatebucket—therequestmapstoabucketisuchthatsiSinherentsi+1.Thespecicadmissioncontroltestforeachbatchofrequestswithinabucketisasfollows.Letdenotethebatchsize(i.e.,thenumberofrequests)inabucket.LetQdenotetheestimatedqueu-ingdelayseenbyeachrequestinthebatch.Thequeuingdelayisthetimetherequesthastowaitatacataclysmserverbeforeitre-ceivesservice;thequeuingdelayisafunctionofthecurrentloadontheserveranditsestimationisdiscussedinSection3.5.Letdenotetheaveragenumberofrequests(connections)thatarecur-rentlybeingservicedbyaserverintheapplication'sserverpool.ThentherequestswithinabatchareadmittedifandonlyifthesumofthequeuingdelayseenbyarequestanditsactualservicetimedoesnotexceedthecontractedSLA.Thatis,Q++n
SRsla(3)whereSistheaverageinherentservicetimeofarequestinthebatch,nisthenumberofserversallocatedtotheapplication,andRslaisthedesiredresponsetime.Theterm+dne

Sisanestimateoftheactualservicetimeofthelastrequestinthebatch,andisdeterminedbyscalingtheinherentservicetimeSbytheserverload—whichisthenumberoftherequestscurrentlyinser-vice,i.e.,,plusthenumberofrequestsfromthebatchthatmightbeassignedtotheserveri.e,dne.1Ratherthanactuallycomputingthemeaninherentservicetimeoftherequestinabatch,itisap-proximatedasS=(si+si+1)=2,where(si;si+1]istheservicetimerangeassociatedwiththebucket.Asindicatedabove,theadmissioncontrolisinvokedforeachclassperiodically—onceeveryditimeunitsfornewlyarrivedre-questsofclassCi.Theinvocationismorefrequentforimpor-tantclassesandlessfrequentforlessimportantclasses,thatis,d1=0d2:::dL.Sincearequestmaywaitinabucketforuptoditimeunitsbeforeadmissioncontrolisinvokedforitsbatch,theabovetestismodiedasQ++n
SRsladi(4)Intheeventthisconditionissatised,allrequestsinthebatchareadmittedintothesystem.Otherwiserequestsinthebatcharedropped.Observethatintroducingthesedelaysintotheprocessingofcer-tainrequestsdoesnotcauseadegradationintheresponsetimeoftheadmittedrequestsbecausetheynowundergoamorestringentadmissioncontroltestasgivenby(4).However,thesedelayswouldhavetheeffectofreducingtheapplication'sthroughputwhenitisnotoverloaded.Therefore,thesedelaysshouldbechangeddynam-icallyasworkloadsofvariousclasseschange.Inparticular,theyshouldtendto0whentheapplicationhassufcientcapacitytohan-dlealltheincomingtrafc.WediscussinSection3.4howthesedelayvaluesaredynamicallyupdated.Techniquesforestimatingparameterssuchasthequeuingdelay,inherentservicetime,andthenumberofexistingconnectionsarediscussedinSection3.5.3.3ScalableThreshold­basedPolicingWenowpresentasecondapproachtofurtherreducethepolicingoverhead.Ourtechniquetradesefciencyofthepolicerforaccu-racyandreducestheoverheadtoafewarithmeticoperationsperrequest.Thekeyideabehindthistechniqueistoperiodicallypre-computethefractionofarrivingrequeststhatshouldbeadmittedin1Notethatwehavemadetheassumptionofperfectloadbalancingintheadmissioncontroltest(3).Oneapproachforcapturingloadimbalancescanbetoscaleandnbysuitablychosenskewfac-tors.Theseskewfactorscanbebasedonmeasurementsoftheloadimbalanceamongthereplicasoftheapplication. eachclassandthensimplyenforcetheselimitswithoutconductinganyadditionalper-requesttests.Again,incomingrequestsarerstclassiedandundergoaninexpensivetesttodetermineiftheycon-rmtotheleakybucketsfortheirclasses.Conrmingrequestsareadmittedtotheapplicationwithoutanyfurthertests.Otherrequestsundergoamorelightweightadmissioncontroltestthatwedescribenext.Ourtechniqueusesestimatesoffuturearrivalratesandservicedemandsineachclasstocomputeathreshold,whichisdenedtobeapair(classi,fractionpadmit).Thethresholdindicatesthatallrequestsinclassesmoreimportantthanishouldbeadmitted(padmit=1),requestsinclassishouldbeadmittedwithprobabil-itypadmit,andallrequestsinclasseslessimportantthanishouldbedropped(padmit=0).Wedeterminetheseparametersbasedonobservationsofarrivalratesandservicetimesineachclassesoverperiodsofmoderatelength(weuseperiodsoflength15sec).Denotingthearrivalratestoclasses1;:::;Lby1;:::;Landtheobservedaverageservicetimesbys1;:::;sL,thethreshold(i,padmit)iscomputedsuchthatj=iXj=1jsj1j=LXj=1minsj(5)andpadmit
isi+j=i1Xj=1jsj1j=LXj=1minsj(6)wheremindenotestheminimumguaranteedrateforclassj.Thus,admissioncontrolnowmerelyinvolvesapplyingthein-expensiveclassicationfunctiononanewrequesttodetermineitsclass,determiningifitconrmstotheleakybucketforthatclass(alsoalightweightoperation),andthenusingtheequallylightweightthresholdingfunction(ifitdoesnotconrmtotheleakybucket)todecideifitshouldbeadmitted.Observethatthisad-missioncontrolrequiresestimatesofper-classarrivalrates.Theseratesareclearlydifculttopredictduringunexpectedoverloads.However,itispossibletoreactfastbyupdatingourestimatesofthearrivalratesfrequently.Ourimplementationofthreshold-basedpolicingestimatesarrivalratesbycomputingexponentiallysmoothedaveragesofarrivalsover15secperiods.WewilldemonstratetheefcacyofthispolicerinanexperimentinSection6.3.Thethreshold-basedandthebatch-basedpolicingstrategiesneednotbemutuallyexclusive.Thesentrycanemploythemoreac-curatebatch-basedpolicingsolongastheincomingrequestratepermitsoneadmissioncontroltestperbatch.Iftheincomingrateincreasessignicantly,theprocessingdemandsofthebatch-basedpolicingmaysaturatethesentry.Insuchanevent,whentheloadatthesentryexceedsathreshold,thesentrycantradeaccuracyforefciencybydynamicallyswitchingtoathreshold-basedpolic-ingstrategy.Thisensuresgreaterscalabilityandrobustnessduringoverloads.Thesentryrevertstothebatch-basedadmissioncontrolwhentheloaddecreasesandstaysbelowthethresholdforasuf-cientlylongduration.Wewouldliketonotethatseveralexistingadmissioncontrolalgorithmssuchas[8,11,24](discussedinSec-tion7)arebasedondynamicallysetthresholdssuchasadmissionratesandcanbeimplementedasefcientlyasourthreshold-basedadmissioncontrol.Thenovelfeatureinourapproachistheexi-bilitytotrade-offtheaccuracyofadmissioncontrolforitscompu-tationaloverheaddependingontheloadonthesentry.3.4AnalysisofthePolicerInthissectionweshowhowthesentrycan,undercertainas-sumptions,computethedelayvaluesforvariousclassesbasedononlineobservations.Thegoalistopickdelayvaluessuchthattheprobabilityofanewlyarrivedrequestbeingdeniedserviceduetoanalreadyadmittedlessimportantrequestissmallerthanadesiredthreshold.Considerthefollowingsimpliedversionoftheadmissioncon-trolalgorithmpresentedinSection3.2:Assumethattheapplicationrunsononlyoneserver—itiseasytoextendtheanalysistothecaseofmultipleservers.Theadmissioncontrollerletsinanewrequestifandonlyifthetotalnumberofrequeststhathavebeenadmit-tedandarebeingprocessedbytheapplicationdoesnotexceedathresholdN.AssumetheapplicationconsistsofLrequestclassesC1;:::;CLindecreasingorderofimportance.Wemakethesim-plifyingassumptionofPoissonarrivalswithrates1;:::;L,andservicetimeswithknownCDFsFs1(:);:::;FsL(:)respectively.Asbefore,d1=0.ForsimplicityofexpositionweassumethatthedelayforclassC2isd,and8i�2;di+1=ki
di;(ki1).DenotebyAitheeventthatarequestofclassCihastobedroppedattheprocessinginstantm
di;(m�0)andthereisatleastonerequestofalessimportantclassCj;(j�i)stillinservice.Clearly,Pr(A1)=0andPr(AL)=0:Weareinterestedinensuring8i�1;Pr(Ai);01:(7)Consider1iL.ForAitooccur,allofthefollowingmusthold:(1)Xi:atleastonerequestofclassCiarrivesduringtheperiod[(m1)
di;m
di],(2)Yi:thenumberofrequestsinserviceattimem
diisN,(3)Zi:atleastoneoftherequestsbeingservicedbelongstooneoftheclassesCi+1;:::;CL.Wehave,Pr(Ai)=Pr(Xi^Yi^Zi)Duringoverloads,wecanassumethatthenumberofrequestsinservicewouldbeNwithahighprobabilitypdrop.Thepolicerwillrecordpdropovershortperiodsoftime.Also,XiandZiareindependent.ThisletsushavePr(Ai)Pr(Xi)
pdrop
Pr(Zi)(8)Pr(Xi)=1ei
di(9)DenotebyZji;(ijL)theeventthatatleastoneoftherequestsbeingservicedattimem
dibelongstotheclassj.Clearly,Pr(Zi)=j=LXj=i+1Pr(Zji)(10)LetusnowfocusonthetermPr(Zji).TheeventZjiisthedis-junctionofthefollowingevents,oneforeachl;(l�0):Plj:atleastonerequestofclassjarrivesduringtheperiod[m
di(l+1)
dj;m
dil
dj]andQl:atleastonerequestofclassjisadmittedattheprocessinginstantm
dil
djandRlj:theservicetimeofatleastoneadmittedrequestislongenoughsothatitisstillinserviceattimem
di.AsinEquation(9),Pr(Plj)=1ej
dj(11)ConsiderRlj.Duringanoverloadeachadmittedrequestcom-petesattheserverwith(N-1)otherrequestsduringmostofitslife-time.AfairapproximationthenistoassumethatarequesttakesNtimesitsservicetimetonish.Therefore,wehave,Pr(Rlj)=1Fsjl
djN(12) WeapproximateQlusingthefollowingreasoning.Duringover-loads,arequestofclassCjwillbeadmittedatprocessinginstanttonlyifthenumberofrequestsinserviceattimetislessthanN(theprobabilityofthisisapproximatedas(1pdrop))andnore-questofamoreimportantclassCharrivedduring[tdh;t].Thatis,Pr(Ql)(1pdrop)h=j1Yh=1ehdh(13)CombiningEquations(8)-(13),wegetthefollowingapproxima-tion.Pr(Ai),Pr(Ai)pdrop(1pdrop)(1eidi)
Pj=Lj=i+1(1ejdj)Qh=j1h=1ehdhP1=11FsjldjN)TheaboveapproximationofPr(Ai)providesaprocedureforiterativelycomputingthedivaluesusingnumericalmethods.Wepickdelayvaluesthatmakethetermontherighthandsidesmallerthanthedesiredboundforalli.Thisinturnguaranteesthattheinequalitiesin(7)aresatised.3.5OnlineParameterEstimationThebatch-basedandthreshold-basedpolicingalgorithmsrequireestimatesofanumberofsystemparameters.Theseparametersareestimatedusingonlinemeasurements.Thenucleirunningonthecataclysmserversandsentriescollectivelygatherandmaintainvariousstatisticsneededbythepolicer.Thefollowingstatisticsaremaintained:Arrivalratei:Sinceeachrequestismappedontoaclassatthesentry,itistrivialtousethisinformationtomeasuretheincomingarrivalratesineachclass.QueuingdelayQ:Thequeuingdelayincurredbyarequestismeasuredatthecataclysmserver.Thequeuingdelayisestimatedasthedifferencebetweenthetimetherequestar-rivesattheserverandthetimeitisacceptedbytheHTTPserverforservice(weassumethatthedelayincurredatthesentryisnegligible).Thenucleicanmeasurethesevaluesbyappropriatelyinstrumentingtheoperatingsystemkernel.Thenucleiperiodicallyreporttheobservedqueuingdelaystothesentry,whichthencomputesthemeandelaysacrossallserversintheapplication'spool.Numberofrequestsinservice:Thisparameterismeasuredatthecataclysmserver.Thenucleitrackthenumberofac-tiveconnectionsservicedbytheapplicationandperiodicallyreportthemeasuredvaluestothesentry.Thesentrythencomputesthemeanofthereportedvaluesacrossallserversfortheapplication.Requestservicetimes:Thisparameterisalsomeasuredattheserver.Theactualservicetimeofarequestismeasuredasthedifferencebetweenthearrivaltimeattheserverandthetimeatwhichthelastbyteoftheresponseissent.Themeasurementoftheinherentservicetimeismorecomplex.DoingsorequiresinstrumentationoftheOSkernelandsomeinstrumentationoftheapplicationitself.Thisinstrumenta-tionenablesthenucleustocomputetheCPUprocessingtimeforarequestaswellasthedurationforwhichtherequestedisblockedonI/O.Together,thesevaluesdeterminethein-herentservicetime(seeEquation2).Constant:TheconstantinEquation2ismeasuredusingofinemeasurementsonthecataclysmservers.WeexecuteseveralrequestswithdifferentCPUdemandsanddifferent-sizedresponsesunderlightloadconditionsandmeasuretheirexecutiontimes.WealsocomputetheCPUdemandsandtheI/Otimesasindicatedabove.Theconstantisthenesti-matedasthevaluethatminimizesthedifferencebetweentheactualexecutiontimeandtheinherentservicetimeinEq.2.Thesentryusespaststatisticstoestimatetheinherentservicetimeofanincomingrequestinordertomapitontoabucket.Todoso,thesentryusesahashtableformaintainingtheusagestatisticsfortherequestsithasadmittedsofar.Eachentryinthistablecon-sistsoftherequestedURL(whichisusedtocomputetheindexoftheentryinthetable)andavectoroftheresourceusagesforthisre-questasreportedbythevariousservers.RequestsforstaticcontentpossessthesameURLeverytimeandsoalwaysmaptothesameentryinthehashtable.TheURLforrequestsfordynamiccontent,ontheotherhand,maychange(e.g.,theargumentstoascriptmaybespeciedaspartoftheURL).Forsuchrequests,wegetridoftheargumentsandhashbasedonthenameofthescriptinvoked.Theresourceusagesforrequeststhatinvokethesescriptsmaychangedependingonthearguments.Wemaintainexponentiallydecayedaveragesoftheirusages.4.PROVISIONINGFORCATACLYSMSPolicingmechanismsmayturnawayasignicantfractionoftherequestsduringoverloads.Insuchascenario,anincreaseintheef-fectiveapplicationcapacityisnecessarytoreducetherequestdroprate.Thecataclysmcontrolplaneimplementsdynamicprovision-ingtovarythenumberofallocatedserversbasedonapplicationworkloads.Theapplication'sserverpoolisincreasedduringover-loadsbyallocatingserversfromthefreepoolorbyreassigningunder-usedserversfromotherapplications.Thecontrolplanecanalsodynamicallyprovisionsentryserverswhentheincomingre-questrateimposessignicantprocessingdemandsontheexistingsentries.Therestofthissectiondiscussestechniquesfordynami-callyprovisioningcataclysmserversandsentries.4.1Model­basedProvisioningforApplicationsWeusequeuingtheorytomodeltherevenueobtainedfromas-signingacertainnumberofserverstoareplicableapplicationunderagivenworkload.Ourmodeldoesnotmakeanyassumptionsaboutthenatureoftherequestarrivalprocessesortheservicetimes.OurabstractionofasinglereplicaofaserviceisaG/G/1queuingsys-temforwhichthefollowingboundisknown[13]:E[S]+2a+2b2
(E[R]E[S])1(14)HereE[R]istheaverageresponsetime,E[S]istheaverageservicetime,istherequestarrivalrate,=
E[S]istheutilization,and2aand2barethevarianceofinter-arrivaltimeandthevari-anceofservicetimerespectively.Itshouldbepointedoutthatanumberofsimilarmodelsforsimple,replicableapplicationshavebeenproposedinrecentwork([7,19,22])andanyofthesecouldpotentiallybeusedbyourprovisioningalgorithm.Thecompar-isonofourmodelwiththeseothermodelsisnotrelevanttoourcurrentdiscussionofoverloadmanagementandthereforebeyondthescopeofthiswork.Modelingofmorecomplex,multi-tieredInternetapplicationsispartofourongoingresearch.Inequality(14)isusedbytheprovisioningmechanismtodeter-minethenumberofserversneededbyanapplicationtosustainagivenrequestarrivalrate.Thedynamicprovisioningmechanism normallyoperatesinapredictivefashion.Itisinvokedperiodically(onceevery30minutesinourprototype)andusestheworkloadob-servationsinthepasttopredictfuturerequestarrivalrates.Itthendeterminesapartitionofserversamongtheapplicationsthatwouldmaximizetheexpectedrevenueduringthenextperiod.Sincethefocusofthispaperisonscalableadmissioncontrol,weomitthedetailsofourworkloadpredictionandserverallocationalgorithmshere[20].Sinceoverloadsareoftenunanticipated,asentryofanover-loadedapplicationcandynamicallyinvoketheprovisioningmech-anismwhenevertherequestdroprateexceedsacertainpre-denedvalue.Insuchascenario,theprovisioningmechanismoperatesinareactivemodetocountertheoverload.Themechanismmoni-torsdeviationsintheactualworkloadfromthepredictedworkloadandusesthesetodetectoverloadedapplications.Itallocatesaddi-tionalserverstotheoverloadedapplications—theseserversmaybefromthefreeserverpoolorunderutilizedserversfromotherappli-cations.Undesirableoscillationsinsuchallocationsarepreventedusingtwoconstraints:(i)alimitof
isimposedonthenumberofserversthatcanbeallocatedtoanapplicationinasinglestepinthereactivemodeand(ii)adelayoftimeunitsisimposedonthedurationbetweentwosuccessiveinvocationsoftheprovision-ingmechanisminthereactivemode(issetto5minutesinourprototype).RecallthatourSLApermitsdegradedresponsetimetargetsforhigherarrivalrates.TheprovisioningmechanismmaydegradetheresponsetimetotheextentpermittedbytheSLA,addmorecapacity,orabitofboth.Theoptimizationdrivesthesede-cisions,andtheresultingtargetresponsetimesareconveyedtotherequestpolicers.Thus,theseinteractionsenablecouplingofpolic-ing,provisioning,andadaptiveperformancedegradation.4.2SentryProvisioningIngeneral,allocationanddeallocationofsentriesissignicantlylessfrequentthanthatofcataclysmservers.Further,thenumberofsentriesneededbyanapplicationismuchsmallerthanthenumberofserversrunningit.Consequently,asimpleprovisioningschemesufcesfordynamicallyvaryingthenumberofsentriesassignedtoanapplication.OurschemeusestheCPUutilizationoftheexistingsentryserversasthebasisforallocatingadditionalsentries(ordeal-locatingactivesentries).Iftheutilizationofasentrystaysinexcessofapre-denedthresholdhighcpuforacertainperiodoftime,itrequeststhecontrolplaneforanadditionalsentryserver.Uponre-ceivingsuchrequestsfromoneormoresentriesofanapplication,thecontrolplaneassignseachanadditionalsentry.Similarly,iftheutilizationofasentrystaysbelowathresholdlowcpu,itisre-turnedtothefreepoolwhileensuringthattheapplicationhasatleastonesentryremaining.Wheneverthecontrolplaneassigns(orremoves)asentryservertoanapplication,itrepartitionstheappli-cation'sserverspoolequallyamongthevarioussentries.TheDNSentryfortheapplicationisalsoupdateduponeachallocationordeallocation;around-robinDNSschemeisusedtolooselyparti-tionincomingrequestsamongsentries.Sinceeachsentrymanagesamutuallyexclusivepoolofservers,itcanindependentlyperformadmissioncontrolandloadbalancingonarrivingrequests;theSLAiscollectivelymaintainedbyvirtueofmaintainingitateachsentry.5.IMPLEMENTATIONCONSIDERATIONSWeimplementedaprototypeCataclysmhostingplatformonaclusterof20Pentiummachinesconnectedviaa1GbpsethernetswitchandrunningLinux2.4.20.Eachmachineintheclusterrunsoneofthefollowingentities:(1)anapplicationreplica,(2)acata-clysmsentry,(3)thecataclysmprovisioning,(4)aworkloadgener-atorforanapplication.CataclysmSentry:WeusedKernelTCPVirtualServer(ktcpvs)version0.0.14[14]toimplementthepolicingmechanismsdescribedinSection3.ktcpvsisanopen-source,Layer-7loadbalancerim-plementedasaLinuxmodule.ItacceptsTCPconnectionsfromclients,opensseparateconnectionswithservers(oneforeachclient)andtransparentlyrelaysdatabetweenthese.WemodiedktcpvstoimplementallthesentrymechanismsdescribedinSections3and4.Thedetailsofourimplementationcanbefoundin[20].CataclysmProvisioning:Cataclysmprovisioningwasimple-mentedasauser-spacedaemonrunningonadedicatedmachine.Atstartup,itreadsinformationneededtocommunicatewiththesentriesandinformationabouttheserversintheclusterfromacongurationle.Thesentriesgatherandreportvariousstatisticsneededbytheprovisioningalgorithmperiodically.Theprovision-ingalgorithmcanbeinvokedinthereactiveorpredictivemodesasdiscussedinSection4.Afterdeterminingapartitioningofthecluster'sserversamongthehostedapplications,theprovisioningdaemonusesscriptstoremotelylogontothenodesrunningthesentriestoenforcethepartitioning.6.EXPERIMENTALEVALUATIONInthissectionwepresenttheexperimentalsetupfollowedbytheresultsofourexperimentalevaluation.6.1ExperimentalSetupThecataclysmsentrieswererunondual-processor1GHzma-chineswith1GBRAM.Thecataclysmcontrolplane(responsi-bleforprovisioning)wasrunonadual-processor450MHzma-chinewith1GBRAM.Themachinesusedascataclysmservershad2.8GHzprocessorsand512MBRAM.Finally,theworkloadgeneratorswererunonmachineswithprocessorspeedsvaryingfrom450MHzto1GHzandwithRAMsizesintherange128MB-512MB.AllmachinesranLinux2.4.20.InourexperimentsweconstructedreplicableapplicationsusingtheApache1.3.28WebserverwithPHPsupportenabled.ThelesetservicedbytheseWebserverscomprisedlesofsizevaryingfrom1kBto256kBtorepresenttherangefromsmalltextlestolargeimageles.Inaddition,theWebservershostPHPscriptswithdifferentcompu-tationaloverheads.Thedynamiccomponentofourworkloadcon-sistofrequestsforthesescripts.Inalltheexperiments,theSLApresentedinFigure1wasusedfortheapplications.Applicationrequestsaregeneratedusinghttperf[16],anopen-sourceWebworkloadgenerator.6.2RevenueMaximizationandClass­basedDifferentiationOurrstexperimentinvestigatestheefcacyofthemechanismsemployedbythecataclysmsentryforrevenuemaximizationandtoprovideclass-baseddifferentiationtorequestsduringoverloads.Thecataclysmprovisioningwaskeptturnedoffinthisexperiment.WeconstructedareplicatedWebserverconsistingofthreeApacheservers.Thisapplicationsupportedthreeclassesofrequests—Gold,SilverandBronzeindecreasingorderofrevenue.Theclassofare-questcouldbeuniquelydeterminedfromitsURL.Thedelayvaluesforthethreeclasseswerexedat0,50,and100msec,respectively.Theminimumsustainablerequestsratesdesiredbyallthreeclasseswerechosentobe0.TheworkloadconsistedofrequestsforasetofPHPscripts.ThecapacityofeachApacheserverforthisworkload(i.e.,therequestarrivalrateforwhichthe95thpercentileresponsetimeofthere-questswasbelowtheresponsetimetarget)wasdeterminedofineandwasfoundtobenearly60requests/sec.Figure3(a)showstheworkloadusedinthisexperiment.Nearlyalltherequestsar- 0501001502000100200300400500Arrival rate (req/sec)Time (sec)Arrival rateGOLDSILVERBRONZE(a)Arrivalrates0501001502000100200300400500Admission rate (req/sec)Time (sec)Admission rateGOLDSILVERBRONZE(b)Admissionrates00.20.40.60.810100200300400500Fraction admittedTime (sec)Fraction admittedGOLDSILVERBRONZE(c)Fractionadmitted 0 200 400 600 800 1000 1200010020030040050095th % resp. time (msec)Time (sec)95th percentile response timeGOLDSILVERBRONZE(d)95thresp.timeFigure3:Demonstrationoftheworkingoftheadmissioncon-trolduringanoverload.rivingtillt=130secwereadmittedbythesentry.Betweent=130secandt=195sec,theBronzerequestsweredroppedalmostex-clusively.Att=195secthearrivalrateofSilverrequestsshotupandreachednearly120requests/sec.TheadmissionrateofBronzerequestsdroppedtoalmostzerotoadmitasmanySilverrequestsaspossible.Att=210sec,thearrivalrateofGoldrequestsshotupto200requests/sec.ThesentrytotallysuppressedallarrivingBronzeandSilverrequestsnowandletinonlyGoldrequestsaslongastheincreasedarrivalrateofGoldrequestspersisted.Fig-ure3(c)isanalternaterepresentationofthesystembehaviorinthisexperimentanddepictsthevariationofthefractionofrequestsofthethreeclassesthatwereadmitted.Figure3(d)depictstheperfor-manceofadmittedrequests.Wendthatthesentryissuccessfulinmaintainingtheresponsetimebelow1000ms.6.3ScalableAdmissionControlWemeasuredtheCPUutilizationatthesentryserverfordiffer-entrequestarrivalratesforboththebatch-basedandthethreshold-basedadmissioncontrol.Figure4showsourobservationsofCPUutilizationwith95%condenceintervals.Sincewewereinter-estedonlyintheoverheadsoftheadmissioncontrolandnotinthedatacopyingoverheadsinherentinthedesignofthektcpvsswitch,weforcedthesentrytodropallrequestsafterconductingtheadmissioncontroltest.WeincreasedtherequestarrivalratestilltheCPUatthesentryserverbecamesaturated(nearly90%uti-lization).Weobservemorethanafour-foldimprovementinthesentry'sscalability—whereasthesentryCPUsaturatedat4000re-quests/secwiththebatch-basedadmissioncontrol,itwasabletohandlealmost19000requests/secwiththethreshold-basedadmis-sioncontrol.Asecondexperimentwasconductedtoinvestigatethedegrada-tioninthedecisionmakingduetothethreshold-basedadmissioncontroller.WerepeatedtheexperimentreportedinSection6.2(Fig-ure3)butforcedthesentrytoemploythethreshold-basedadmis-sioncontroller.Thethresholdsusedbytheadmissioncontrolwere02040608010005000100001500020000CPU usage (%)Arrival rate (req/sec)Scalability of the Admission ControlBatch-basedThreshold-basedFigure4:Scalabilityoftheadmissioncontrol.050100150200050100150200250300350400450500Admission rate (req/sec)Time (sec)Admission rateGOLDSILVERBRONZE05001000150020002500300035004000450005010015020025030035040045050095th % resp. time (msec)Time (sec)95th percentile response timeGOLDSILVERBRONZE(a)Admissionrates(b)95thperc.resp.timeFigure5:Performanceofthethreshold-basedadmissioncon-trol.Att=135sec,thethresholdwassettorejectallBronzerequests;att=180sec,itwasupdatedtorejectallBronzeandSilverrequests;att=210secitwasupdatedtoalsorejectGoldrequestswithaprobability0.5;nally,att=390sec,itwasagainsettorejectonlyBronzerequests.computedonceevery15sec.Figure5(a)showschangesinthead-missionratesforrequestsofthethreeclasses.Theimpactoftheinaccuraciesinherentinthethreshold-basedadmissioncontrollerresultedindegradedperformanceduringperiodswhenthethresh-oldchosenwasincorrect.Weobservetwosuchperiods(120-135secduringwhichallBronzerequestsweredroppedand190-210secduringwhichallBronzeandSilverrequestsweredroppedwhileGoldrequestswereadmittedwithprobabilityof0.5)duringwhichthe95thpercentileoftheresponsetimedeterioratedcomparedtothetargetof1000msec.Theresponsetimesduringtherestoftheexperimentwerekeptundercontrolduetothethresholdgettingupdatedtoastrictenoughvalue.6.4SentryProvisioningWeconductedanexperimenttodemonstratetheabilityofthesystemtodynamicallyprovisionadditionalsentriestoaheavilyoverloadedservice.Figure6showstheoutcomeofourexperi-ment.Theworkloadconsistedofrequestsforsmallstaticlessenttothesentrystartingat4000requests/secandincreasingby4000requests/seceveryminuteandisshowninFigure6(a).IftheCPUutilizationofthesentryserverremainedabove80%formorethan30sec,arequestwasissuedtothecontrolplaneforanadditionalsentry.Figure6(b)showsthevariationoftheCPUutilizationattherstsentry.Att=210sec,asecondsentrywasaddedtotheservice.SubsequentrequestsweredistributedequallybetweenthetwosentriescausingthearrivalrateandtheCPUutilizationattherstsentrytodrop.Athirdsentrywasaddedatt=420sec,when 0 5000 10000 15000 20000 25000 30000 35000 40000 45000 50000 0 100 200 300 400 500 600Arrival rate (req/sec)Time (sec)Arrival rates[S=1][S=2][S=3]Total arrival rateArrival rate to sentry 1 0 20 40 60 80 100 0 100 200 300 400 500 600CPU utilization (%)Time (sec)Sentry 1: CPU utilization[S=1][S=2][S=3][S=1][S=2][S=3]CPU utilization(a)Arrivalrates(b)CPUutilization:Sentry1.Figure6:Dynamicprovisioningofsentries.[S=n]meansthenumberofsentriesisnnow.thetotalarrivalratetotheservicehadreached32000requests/secoverwhelmingboththeexistingsentries.6.5CataclysmProvisioningWeconductedanexperimentwithtwoWebapplicationshostedonourCataclysmplatform.Thetotalnumberofcataclysmserversavailableinthisexperimentwas11.TheSLAsforboththeap-plicationswereidenticalandaredescribedinFigure1.Further,theSLAsimposedalowerboundof3onthenumberofserversthateachapplicationcouldbeassigned.Thedefaultprovisioningdurationusedbythecontrolplanewas30min.TheworkloadsforthetwoapplicationsconsistedofrequestsforanassortmentofPHPscriptsandlesinthesizerange1kB-128kB.Requestsweresentatasustainablebaseratetothetwoapplica-tionsthroughouttheexperiment.Overloadswerecreatedbysend-ingincreasednumberofrequestsforasmallsubsetofthescriptsandstaticles(tosimulateasubsetofthecontentbecomingpop-ular).Theexperimentbeganwiththetwoapplicationsrunningon3serverseach.Sentriesinvokedtheprovisioningalgorithmwhenmorethan50%oftherequestsweredroppedovera5mininterval.Figures7(a)and7(c)depictthearrivalratestothetwoapplica-tions.ThearrivalrateforApplication1wasmadetoincreaseinastep-likefashionstartingfrom100requests/sec,doublingroughlyonceevery5mintillitreachedapeakvalueof1600requests/sec.AtthispointApplication1washeavilyoverloadedwiththearrivalrateseveraltimeshigherthansystemcapacity(whichwasroughly60request/secperserverassignedtotheserviceasdeterminedbyofinemeasurements).Att=910secthesentry,havingobservedmorethan50%oftherequestbeingdropped,triggeredthepro-visioningalgorithmasdescribedinSection4.TheprovisioningalgorithmrespondedbypullingoneserverfromthefreepoolandaddingittoApplication1.Att=1210sec,anotherserverwasaddedtoApplication1fromthefreepool.ObserveinFigure7(a)theincreasesintheadmissionratescorrespondingtotheseadditionalserversbeingmadeavailabletoApplication1.Thenextinterest-ingeventwasthedefaultinvocationofprovisioningatt=1800sec.Theprovisioningalgorithmaddedallthe3serversremaininginthefreepooltotheheavilyoverloadedApplication1.Also,basedonrecentobservationofarrivalrates,itpredictedanarrivalrateintherange1000-10000requests/secanddegradedtheresponsetimetargetforApplication1to2000msecbasedonitsQoStable(seeFigure1).Inthelatterpartoftheexperiment,theoverloadofApplication1subsidedandApplication2gotoverloaded.ThefunctioningoftheprovisioningwasqualitativelysimilartowhenService1wasoverloaded.Figures7(b)and7(d)showthe95thpercentileresponsetimesforthetwoservicesduringtheexperi-ment.ThecontrolplanewasabletopredictchangestoarrivalratesanddegradetheresponsetimetargetaccordingtotheSLAresult- 0 500 1000 1500 2000 0 1000 2000 3000 4000 5000Rate (req/sec)Time (sec)Application 1: Arrival and admission rates[T,N=4][T,N=5][D,N=8][T,N=7][D,N=5]Arrival rateAdmission rate(a)Arrivalandadmissionrates05001000150020002500300001000200030004000500095th % resp. time (msec)Time (sec)Application 1: 95th resp time95th resp. timeresp. time target(b)95thresp.time 0 500 1000 1500 2000 0 1000 2000 3000 4000 5000Rate (req/sec)Time (sec)Application 2: Arrival and admission rates[T,N=4][D,N=6]Arrival rateAdmission rate(c)Arrivalandadmissionrates05001000150020002500300001000200030004000500095th % resp. time (msec)Time (sec)Application 2: 95th resp time95th resp. timeresp. time target(d)95thresp.timeFigure7:Dynamicprovisioningandadmissioncontrol:Per-formanceofApplications1and2.D:Defaultinvocationofpro-visioning,T:Provisioningtriggeredbyexcessivedrops,[N=n]:sizeoftheserversetisnnow.Onlyselectedprovisioningeventsareshown.inginanincreasednumberofrequestsbeingadmitted.Moreover,thesentrieswereabletokeeptheadmissionrateswellbelowsys-temcapacitytoachieveresponsetimeswithintheappropriatetargetwithonlysporadicviolations(whichwereonfewerthan4%oftheoccasions).RELATEDWORKPreviousworkonoverloadcontrolinInternetplatformsspansseveralareas.WebrieyreviewpriorworkthatismostrelevanttotheCataclysmplatformandreferthereaderto[20]foramoreextensivesurvey.AdmissionControlforInternetServices:Voigtetal.[23]presentkernel-basedadmissioncontrolmechanismstoprotectWebserversagainstoverloads—SYNpolicingcontrolstherateandburstatwhichnewconnectionsareaccepted,prioritizedlistenqueuere-ordersthelistenqueuebasedonpre-denedconnectionpriorities,HTTPheader-basedcontrolenablesratepolicingbasedonURLnames.WelshandCuller[24]proposeanoverloadmanagementsolutionforInternetservicesbuiltusingtheSEDAarchitecture.Asalientfeatureoftheirsolutionisfeedback-basedadmissioncon-trollersembeddedintoindividualstagesoftheservice.AmodelofaWebserverforthepurposeofperformancecontrolusingclassi-calfeedbackcontroltheorywasstudiedin[2];animplementationandevaluationusingtheApacheWebserverwasalsopresentedinthework.In[12],KanodiaandKnightlyutilizeamodelingtech-niquecalledserviceenvelopstodeviseanadmissioncontrolforWebservicesthatattemptstoprovidedifferentresponsetimetar-getsformultipleclassesofrequests.LiandJamin[15]presentameasurement-basedadmissioncontroltodistributebandwidthacrossclientsofunequalrequirement.In[8],theauthorspresentanadmissioncontrolsolutionformulti-tiere-commercesitesthat externallyobservesexecutioncostsofrequests,distinguishingdif-ferentrequeststypes.Kamraetal.presentacontroltheoreticap-proachforadmissioncontrolin[11].AbdelzaherandBhatti[1]proposetodealwithserveroverloadsbyadaptingdeliveredcon-tenttoloadconditions.ThisisadifferentkindofQoSdegradationthanwhatwehaveproposedinourwork,butitcanbeintegratedintoaCataclysmplatformbydeningappropriateSLAsbasedonit.Inthiswork,wefocusonthescalabilityofthepolicingmecha-nism—animportantissueduringoverloads—asopposedtopriorworkthatmostlyfocusedonthemechanicsofthepolicing.Further,unlikepriorwork,ourworkhasconsideredthetrade-offbetweenaccuracyandefciencyofpolicingtoscaletolargerequestrates,aswellastechniquestoprovisionsentries.DynamicProvisioninginClusters:ThenotionofanoverowserverpooltohandleunexpectedsurgesinworkloadwasproposedbyFoxetal.[9].Severaleffortshaveaddressedtheproblemofpro-visioningresourcesatthegranularityofindividualservers[3,19].Muse[4]presentsanarchitectureforresourcemanagementinahostingcenter.Museemploysaneconomicmodelfordynamicpro-visioningofresourcestomultipleapplications.Cluster-on-demandprovidesmechanismsforconstructinghierarchicalvirtualclustersinanon-demandfashion[5].Doyleetal.[7]presentamodel-basedutilityresourcemanagementfocusingoncoordinatedmanagementofmemoryandstorage.TheydevelopananalyticalmodelforaWebservicewithstaticcontent.Ranjanetal.[18]makeacaseformultipledatacentershostingreplicasofanapplication.Incaseofonedatacenterbecomingoverloaded,requestsmaybedivertedoveraWANtoothers.Ourfocusinthisworkwasonoverloadmanagementwithinasingledatacenter.Inthispaperweshowedtheutilityofcouplingpolicingandprovisioning,incontrasttopriorapproachesthatconsideredthesetechniquesinisolation.8.CONCLUSIONSInthispaperwepresentedCataclysm,acomprehensiveapproachforhandlingextremeoverloadsinahostingplatformrunningmul-tipleInternetservices.Theprimarycontributionofourworkwastodevelopalowoverhead,highlyscalableadmissioncontroltech-niqueforInternetapplications.Cataclysmprovidesseveraldesir-ablefeatures,suchasguaranteesonresponsetimebyconduct-ingaccuratesize-basedadmissioncontrol,revenuemaximizationatmultipletime-scalesviapreferentialadmissionofimportantre-questsanddynamiccapacityprovisioning,andtheabilitytobeop-erationalevenunderextremeoverloads.Thecataclysmsentrycantransparentlytrade-offtheaccuracyofitsdecisionmakingwiththeintensityoftheworkloadallowingittohandleincomingratesofupto19000requests/second.WeimplementedaprototypeCataclysmhostingplatformonaLinuxclusteranddemonstrateditsbenetsusingavarietyofworkloads.Aspartoffuturework,weplantoextendouroverloadman-agementtechniquestocomplex,multi-tieredInternetapplicationsservicingsession-orientedworkloads.9.REFERENCES[1]T.AbdelzaherandN.Bhatti.WebContentAdaptationtoImproveServerOverloadBehavior.InProceedingsoftheWorldWideWebConference(WWW8),Tornoto,1999.[2]T.Abdelzaher,K.G.Shin,andN.Bhatti.PerformanceGuaranteesforWebServerEnd-Systems:AControl-TheoreticalApproach.IEEETransactionsonParallelandDistributedSystems,13(1),Jan.2002.[3]K.Appleby,S.Fakhouri,L.Fong,M.K.G.Goldszmidt,S.Krishnakumar,D.Pazel,J.Pershing,andB.Rochwerger.Oceano-SLA-basedManagementofaComputingUtility.InProceedingsoftheIFIP/IEEESymposiumonIntegratedNetworkManagement,May2001.[4]J.Chase,D.Anderson,P.Thakar,A.Vahdat,andR.Doyle.ManagingEnergyandServerResourcesinHostingCenters.InProceedingsofthe18thSOSP,pages103–116,October2001.[5]J.Chase,L.Grit,D.Irwin,J.Moore,andS.Sprenkle.DynamicVirtualClustersinaGridSiteManager.InTwelfthInternationalSymposiumonHighPerformanceDistributedComputing(HPDC-12),June2003.[6]L.CherkasovaandP.Phaal.Sessionbasedadmissioncontrol:amechanismforimprovingperformanceofcommercialwebsites.InProceedingsofSeventhInternationalWorkshoponQualityofService,IEEE/IFIPevent,London,June1999.[7]R.Doyle,J.Chase,O.Asad,W.Jin,andA.Vahdat.Model-BasedResourceProvisioninginaWebServiceUtility.InProceedingsofthe4thUSITS,Mar.2003.[8]S.Elnikety,E.Nahum,J.Tracey,andW.Zwaenepoel.AMethodforTransparentAdmissionControlandRequestSchedulinginE-CommerceWebSites.InProceedingsofInternationalWWWConference,NewYork,USA,2004.[9]A.Fox,S.Gribble,Y.Chawathe,E.Brewer,andP.Gauthier.Cluster-basedScalableNetworkServices.InProceedingsofthe16thSOSP,December1997.[10]J.Hellerstein,F.Zhang,andP.Shahabuddin.AStatisticalApproachtoPredictiveDetection.ComputerNetworks,Jan.2000.[11]A.Kamra,V.Misra,andE.Nahum.Yaksha:AControllerforManagingthePerformanceof3-TieredWebsites.InProceedingsofthe12thIWQoS,2004.[12]V.KanodiaandE.Knightly.Multi-classlatency-boundedwebservers.InProceedingsofInternationalWorkshoponQualityofService(IWQoS'00),June2000.[13]L.Kleinrock.QueueingSystems,Volume2:ComputerApplications.JohnWileyandSons,Inc.,1976.[14]KernelTCPVirtualServer.http://www.linuxvirtualserver.org/software/.[15]S.LiandS.Jamin.Ameasurement-basedadmission-controlledWebserver.InProceedingsofINFOCOM2000,TelAviv,Israel,March2000.[16]D.MosbergerandT.Jin.httperf–AToolforMeasuringWebServerPerformance.InProceedingsoftheSIGMETRICSWorkshoponInternetServerPerformance,June1998.[17]V.Pai,M.Aron,G.Banga,M.Svendsen,P.Druschel,W.Zwaenepoel,andE.Nahum.Locality-AwareRequestDistributioninCluster-basedNetworkServers.InProceedingsofthe8thASPLOS,October1998.[18]S.Ranjan,R.Karrer,andE.Knightly.Widearearedirectionofdynamiccontentbyinternetdatacenters.InProceedingsofINFOCOM2004,HongKong,Mar.2004.[19]S.Ranjan,J.Rolia,H.Fu,andE.Knightly.Qos-drivenservermigrationforinternetdatacenters.InProceedingsoftheTenthInternationalWorkshoponQualityofService(IWQoS2002),May2002.[20]B.UrgaonkarandP.Shenoy.Cataclysm:HandlingExtremeOverloadsinInternetServices.Technicalreport,DepartmentofComputerScience,UniversityofMassachusetts,November2004.[21]A.VermaandS.Ghosal.OnAdmissionControlforProtMaximizationofNetworkedServiceProviders.InProceedingsofthe12thInternationalWorldWideWebConf.(WWW2003),Budapest,Hungary,May2003.[22]D.Villela,P.Pradhan,andD.Rubenstein.ProvisioningServersintheApplicationTierforE-commerceSystems.InProceedingsofthe12thIWQoS,June2004.[23]T.Voigt,R.Tewari,D.Freimuth,andA.Mehra.Kernelmechanismsforservicedifferrentiationinoverloadedwebservers.InProceedingsofUSENIXAnnualTechnicalConference,June2001.[24]M.WelshandD.Culler.AdaptiveOverloadControlforBusyInternetServers.InProceedingsofthe4thUSITS,March2003.