torontoedu Yashar Ganjali University of Toronto yganjalicstorontoedu ABSTRACT Limiting the overhead of frequent events on the control plane is essential for realizing a scalable SoftwareDe64257ned Network One way of limiting this overhead is to proce ID: 22989
Download Pdf The PPT/PDF document "Kandoo A Framework for Efcient and Scala..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Figure3:Kandoo'shighlevelarchitecture.DeploymentModel.ThedeploymentmodelofKandoocontrollersdependsonthecharacteristicsofanetwork.Forsoftwareswitches,localcontrollerscanbedirectlydeployedonthesameend-host.Similarly,ifwecanchangethesoftwareofaphysicalswitch,wecandeployKandoodirectlyontheswitch.Otherwise,wedeployKandoolocalcontrollersontheprocessingresourcesclosesttotheswitches.Insuchasetting,oneshouldprovisionthenumberoflocalcontrollersbasedontheworkloadandavailableprocessingresources.Notethatwecanuseahybridmodelinrealsettings.Forinstance,consideravirtualizeddeploymentenvironmentdepictedinFigure4,wherevirtualmachinesareconnectedtothenetworkusingsoftwareswitches.Inthisenvironment,wecanplacelocalcontrollersinend-hostsnexttosoftwareswitchesandinseparatenodesforotherswitches.Inourtoyexample(Figure2),wehavefourKandoocontrollers:threelocalcontrollerscontrollingtheswitchesandarootcontroller.Thelocalcontrollerscanbephysicallypositionedusinganydeploymentmodelexplainedabove.Notethat,inthisexample,wehavethemaximumnumberoflocalcontrollersrequired.ControlApplications.ControlapplicationsfunctionusingtheabstractionprovidedbythecontrollerandarenotawareofKandoointernals.TheyaregenerallyOpenFlowapplicationsandcanthereforesendOpenFlowmessagesandlistenonevents.Moreover,theycanemitKandooevents(i.e.,internalevents),whichcanbeconsumedbyotherapplications,andtheycanreplytotheapplicationthatemittedanevent.ControlapplicationsareloadedinlocalnamespacesandcancommunicateusingonlyKandooevents.ThisistoensurethatKandoodoesnotintroducefaultsbyooadingapplications.Inourexample,EelephantisaKandooeventthatcarriesmatchinginformationaboutthedetectedelephant ow(e.g.,itsOpenFlowmatchstructure)andisemittedbyAppdetect.Alocalcontrollercanrunanapplicationonlyiftheapplicationislocal.Inourexample,Apprerouteisnotlocal,i.e.,itmayinstall ow-entiresonanyswitchinthenetwork.Thus,therootcontrolleristheonlycontrollerabletorunAppreroute.Incontrast,Appdetectislocal;therefore,allcontrollerscanrunit.EventPropagation.Therootcontrollercansubscribe Figure4:Kandooinavirtualizedenvironment.Forsoftwareswitches,wecanleveragethesameend-hostforlocalcontrollers,and,forphysicalswitches,weuseseparateprocessingresources.tospeciceventsinthelocalcontrollersusingasimplemessagingchannelplusalteringcomponent.Oncethelocalcontrollerreceivesandlocallyprocessesanevent,itrelaystheeventtotherootcontrollerforfurtherprocessing.NotethatallcommunicationsbetweenKandoocontrollersareevent-basedandasynchronous.Inourexample,therootcontrollersubscribestoeventsoftypeEelephantinthelocalcontrollerssinceitisrunningAppreroutelisteningonEelephant.EelephantisredbyanAppdetectinstancedeployedononeofthelocalcontrollersandisrelayedtorootcontroller.NotethatiftherootcontrollerdoesnotsubscribetoEelephant,thelocalcontrollerswillnotrelayEelephantevents.Itisimportanttonotethatthedata owinKandooisnotalwaysbottom-up.Alocalapplicationcanexplicitlyrequestdatafromanapplicationdeployedontherootcontrollerbyemittinganevent,andapplicationsontherootcontrollerscansenddatabyreplyingtothatevent.Forinstance,wecanhaveatopologyservicerunningontherootcontrollerthatsendstopologyinformationtolocalapplicationsbyreplyingtoeventsofaspecictype.Reactivevs.Proactive.AlthoughKandooprovidesascalablemethodforeventhandling,westronglyrecommendpushingnetworkstateproactively.WeenvisionKandootobeusedasascalable,adaptivecontrolplane,wherethedefaultcongurationispushedproactivelyandisadaptivelyrenedafterwards.Inourtoyexample,defaultpathscanbepushedproactively,whileelephant owswillbereroutedadaptively.ImplementationDetails.WeimplementedKandooinamixtureofC,C++,andPython.Ourimplementationhasalowmemoryfootprintandsupportsdynamicallyloadableplug-ins,whichcanbeimplementedinC,Python,andJava.ItalsoprovidesanRPCAPIformoregeneralintegrationscenarios.OurimplementationofKandooisextremelymodular;anycomponentorback-endcanbeeasilyreplaced,whichsimpliesportingKandootophysicalswitches.Currently,KandoosupportsOpenFlow1.0(OpenFlow1.1and1.2supportisunderdevelopment).Fortheapplications,wecreateda\centralapplicationrepository"anddevelopedasimplepackagemanagement (a)Averagenumberofmessagesreceivedbythecontrollers (b)Averagenumberofbytesreceivedbythecon-trollersFigure6:ControlPlaneLoadfortheElephantFlowDetectionScenario.Theloadisbasedonthenumberofelephant owsinthenetwork.4.RELATEDWORKDatapathExtensions.TheproblemthatwetackleinthispaperisageneralizationofseveralpreviousattemptsatscalingSDNs.Aclassofsolutions,suchasDIFANE[21]andDevoFlow[5],addressthisproblembyextendingdataplanemechanismsofswitcheswiththeobjectiveofreducingtheloadtowardsthecontroller.DIFANEtriestopartlyooadforwardingdecisionsfromthecontrollertospecialswitches,calledauthorityswitches.Usingthisapproach,networkoperatorscanreducetheloadonthecontrollerandthelatenciesofruleinstallation.DevoFlow,ontheotherhand,introducesnewmechanismsinswitchestodispatchfarfewer\important"eventstothecontrolplane.Kandoohasthesamegoal,but,incontrasttoDIFANEandDevoFlow,itdoesnotextendswitches;instead,itmovescontrolplanefunctionsclosertoswitches.Kandoo'sapproachismoregeneralandworkswellindatacenters,butitmighthavealowerthroughputthanspecicextensionsimplementedinhardware.Interestingly,wecanuseKandootoprototypeandtestDIFANE,DevoFlow,orotherpotentialhardwareextensions.Forinstance,anauthorityswitchinDIFANEcanbeemulatedbyalocalKandoocontrollerthatmanagesasubsetofswitchesinthenetwork.Asanotherexample,DevoFlow'sextensionscanalsobeemulatedusingKandoocontrollersdirectlyinstalledonswitches.ThesecontrollersnotonlyreplacethefunctionalityofDIFANEorDevoFlow,butthey (a)Averagenumberofpacketsreceivedbythecontrollers (b)AveragenumberofbytesreceivedbythecontrollersFigure7:ControlPlaneLoadfortheElephantFlowDetectionScenario.Theloadisbasedonthenumberofnodesinthenetwork.alsoprovideaplatformtorunanylocalcontrolapplicationintheircontext.DistributedControllers.HyperFlow[18],Onix[8],SiBF[10],andDevolvedControllers[17]trytodistributethecontrolplanewhilemaintaininglogicallycentralized,eventuallyconsistentnetworkstate.Althoughtheseap-proacheshavetheirownmerits,theyimposelimitationsonapplicationstheycanrun.Thisisbecausetheyassumethatallapplicationsrequirethenetwork-widestate;hence,theycannotbeofmuchhelpwhenitcomestolocalcontrolapplications.Thatsaid,thedistributedcontrollerscanbeusedtorealizeascalablerootcontroller,thecontrollerthatrunsnon-localapplicationsinKandoo.Middleboxes.Middleboxarchitectures,suchasFlow-Stream[7],SideCar[15]andCoMb[13],providescalableprogrammabilityindataplanebyintercepting owsusingprocessingnodesinwhichnetworkapplicationsarede-ployed.Kandooisorthogonaltotheseapproachesinthesensethatitoperatesinthecontrolplane,butitprovidesasimilardistributionforcontrolapplications.InanetworkequippedwithFlowStream,SideCarorCoMb,Kandoocansharetheprocessingresourceswithmiddleboxes(giventhatcontrolanddataplaneapplicationsareisolated)inordertoincreaseresourceutilizationanddecreasethenumberofnodesusedbyKandoo.ActiveNetworks.ActiveNetworks(AN)andSDNsrepre-sentdi erentschoolsofthoughtonprogrammablenetworks.SDNsprovideprogrammablecontrolplanes,whereasANsallowprogrammabilityinnetworkingelementsatpacket transportgranularitybyrunningcodeencapsulatedinthepacket[20,12,19]orinstalledontheswitches[16,6].AnextremedeploymentofKandoocandeploylocalcontrollersonallswitchesinthenetwork.Insuchasetting,wecanemulatemostfunctionalityofANs.Thatsaid,Kandoodi ersfromactivenetworksintwoways.First,Kandoodoesnotprovidein-boundpacketprocessing;instead,itfollowsthefundamentallydi erentapproachproposedbySDNs.Second,Kandooisnotanall-or-nothingsolution(i.e.,thereisnoneedtohaveKandoosupportonswitches).UsingKandoo,networkoperatorscanstillgaineciencyandscalabilityusingcommoditymiddleboxes,eachcontrollingapartitionofswitches.5.CONCLUSIONKandooisahighlycongurableandscalablecontrolplane.Itusesasimpleyete ectiveapproachforcreatingadistributedcontrolplane:itprocessesfrequenteventsinhighlyreplicatedlocalcontrolapplicationsandrareeventsinacentrallocation.Asconrmedbyourexperiments,KandooscalesremarkablybetterthananormalOpenFlowimplementation,withoutmodifyingswitchesorusingsam-plingtechniques.Kandoocanco-existwithothercontrollersbyusingeitherFlowVisor[14]orcustomizedKandooadapters.Havingsaidthat,extrameasuresshouldbetakentoensureconsistency.ThemajorissueisthatKandoolocalcontrollersdonotpropagateanOpenFloweventunlesstherootcontrollersubscribestothatevent.Thus,withoutsubscribingtoallOpenFloweventsinalllocalcontrollers,wecannotguaranteethatexistingOpenFlowapplicationsworkasexpected.Movingforward,weareextendingKandootosupportnewcategoriesofcontrolapplicationsthatarenotnecessarilylocalbutthathavealimitedscope.Suchapplicationscanoperatebyhavingaccesstotheeventsgeneratedbyasubsetofswitches.Usingahierarchyofcontrollers(asopposedtothetwo-levelhierarchypresentedinthispaper),wecanextendKandootorunsuchapplications.Finally,wehavestartedportingKandootophysicalswitches.SwitchesequippedwithKandoocannativelyrunlocalcontrolapplications.6.REFERENCES[1]OpenFlowSwitchSpecication,Version1.2(WireProtocol0x03).http://tinyurl.com/84kelcj.[2]StationandMediaAccessControlConnectivityDiscovery,IEEEStandard802.1AB.http://tinyurl.com/6s739pe.[3]M.Alizadeh,A.Greenberg,D.A.Maltz,J.Padhye,P.Patel,B.Prabhakar,S.Sengupta,andM.Sridharan.Datacentertcp(dctcp).InProceedingsoftheACMSIGCOMM2010conference,pages63{74,2010.[4]A.R.Curtis,W.Kim,andP.Yalagandula.Mahout:Low-overheaddatacentertracmanagementusingend-host-basedelephantdetection.InProceedingsoftheIEEEINFOCOM2011conference,pages1629{1637,2011.[5]A.R.Curtis,J.C.Mogul,J.Tourrilhes,P.Yalagandula,P.Sharma,andS.Banerjee.DevoFlow:scaling owmanagementforhigh-performancenetworks.InProceedingsoftheACMSIGCOMM2011conference,pages254{265,2011.[6]D.DecasperandB.Plattner.DAN:distributedcodecachingforactivenetworks.InProceedingsoftheIEEEINFOCOM'98conference,volume2,pages609{616,1998.[7]A.Greenhalgh,F.Huici,M.Hoerdt,P.Papadimitriou,M.Handley,andL.Mathy.Flowprocessingandtheriseofcommoditynetworkhardware.SIGCOMMComput.Commun.Rev.,39(2):20{26,2009.[8]T.Koponen,M.Casado,N.Gude,J.Stribling,L.Poutievski,M.Zhu,R.Ramanathan,Y.Iwata,H.Inoue,T.Hama,andS.Shenker.Onix:adistributedcontrolplatformforlarge-scaleproductionnetworks.InProceedingsofthe9thUSENIXOSDIconference,pages1{6,2010.[9]B.Lantz,B.Heller,andN.McKeown.Anetworkinalaptop:rapidprototypingforsoftware-denednetworks.InProceedingsoftheACMSIGCOMMHotNetsworkshop,pages19:1{19:6,2010.[10]C.A.B.Macapuna,C.E.Rothenberg,andF.Magalh.In-packetbloomlterbaseddatacenternetworkingwithdistributedopen owcontrollers.InProceedingsofIEEEInternationalWorkshoponManagementofEmergingNetworksandServices,pages584{588,2010.[11]B.Pfa ,J.Pettit,T.Koponen,K.Amidon,M.Casado,andS.Shenker.Extendingnetworkingintothevirtualizationlayer.InProceedingsoftheACMSIGCOMMHotNetsworkshop,2009.[12]B.Schwartz,A.W.Jackson,W.T.Strayer,W.Zhou,R.D.Rockwell,andC.Partridge.Smartpackets:applyingactivenetworkstonetworkmanagement.ACMTransactionsonComputerSystems,18(1):67{88,Feb.2000.[13]V.Sekar,N.Egi,S.Ratnasamy,M.K.Reiter,andG.Shi.Designandimplementationofaconsolidatedmiddleboxarchitecture.InProceedingsofNSDI12,2012.[14]R.Sherwood,M.Chan,A.Covington,G.Gibb,M.Flajslik,N.Handigol,T.Huang,P.Kazemian,M.Kobayashi,J.Naous,S.Seetharaman,D.Underhill,T.Yabe,K.Yap,Y.Yiakoumis,H.Zeng,G.Appenzeller,R.Johari,N.McKeown,andG.Parulkar.CarvingresearchslicesoutofyourproductionnetworkswithOpenFlow.SIGCOMMCCR,40(1):129{130,Jan2010.[15]A.Shieh,S.Kandula,andE.G.Sirer.Sidecar:buildingprogrammabledatacenternetworkswithoutprogrammableswitches.InProceedingsoftheACMHotNetsworksop,pages21:1{21:6,2010.[16]J.M.Smith,D.J.Farber,C.A.Gunter,S.M.Nettles,D.C.Feldmeier,andW.D.Sincoskie.SwitchWare:acceleratingnetworkevolution(Whitepaper).Technicalreport,1996.[17]A.-W.Tam,K.Xi,andH.Chao.Useofdevolvedcontrollersindatacenternetworks.InProcessingsoftheIEEEComputerCommunicationsWorkshops,pages596{601,2011.[18]A.TootoonchianandY.Ganjali.Hyper ow:adistributedcontrolplaneforopen ow.InProceedingsofthe2010INMconference,pages3{3,2010.[19]D.J.Wetherall,J.V.Guttag,andD.L.Tennenhouse.ANTS:atoolkitforbuildinganddynamicallydeployingnetworkprotocols.InProceedingsoftheIEEEOpenArchitecturesandNetworkProgrammingconference,pages117{129,1998.[20]D.J.WetherallandD.L.Tennenhouse.TheACTIVEIPoption.InProceedingsofthe7thACMSIGOPSEuropeanworkshoponsystemssupportforworldwideapplications,pages33{40,1996.[21]M.Yu,J.Rexford,M.J.Freedman,andJ.Wang.Scalable ow-basednetworkingwithDIFANE.InProceedingsoftheACMSIGCOMM2010conference,pages351{362,2010. transportgranularitybyrunningcodeencapsulatedinthepacket[20,12,19]orinstalledontheswitches[16,6].AnextremedeploymentofKandoocandeploylocalcontrollersonallswitchesinthenetwork.Insuchasetting,wecanemulatemostfunctionalityofANs.Thatsaid,Kandoodi ersfromactivenetworksintwoways.First,Kandoodoesnotprovidein-boundpacketprocessing;instead,itfollowsthefundamentallydi erentapproachproposedbySDNs.Second,Kandooisnotanall-or-nothingsolution(i.e.,thereisnoneedtohaveKandoosupportonswitches).UsingKandoo,networkoperatorscanstillgaineciencyandscalabilityusingcommoditymiddleboxes,eachcontrollingapartitionofswitches.5.CONCLUSIONKandooisahighlycon\fgurableandscalablecontrolplane.Itusesasimpleyete ectiveapproachforcreatingadistributedcontrolplane:itprocessesfrequenteventsinhighlyreplicatedlocalcontrolapplicationsandrareeventsinacentrallocation.Ascon\frmedbyourexperiments,KandooscalesremarkablybetterthananormalOpenFlowimplementation,withoutmodifyingswitchesorusingsam-plingtechniques.Kandoocanco-existwithothercontrollersbyusingeitherFlowVisor[14]orcustomizedKandooadapters.Havingsaidthat,extrameasuresshouldbetakentoensureconsistency.ThemajorissueisthatKandoolocalcontrollersdonotpropagateanOpenFloweventunlesstherootcontrollersubscribestothatevent.Thus,withoutsubscribingtoallOpenFloweventsinalllocalcontrollers,wecannotguaranteethatexistingOpenFlowapplicationsworkasexpected.Movingforward,weareextendingKandootosupportnewcategoriesofcontrolapplicationsthatarenotnecessarilylocalbutthathavealimitedscope.Suchapplicationscanoperatebyhavingaccesstotheeventsgeneratedbyasubsetofswitches.Usingahierarchyofcontrollers(asopposedtothetwo-levelhierarchypresentedinthispaper),wecanextendKandootorunsuchapplications.Finally,wehavestartedportingKandootophysicalswitches.SwitchesequippedwithKandoocannativelyrunlocalcontrolapplications.6.REFERENCES[1]OpenFlowSwitchSpeci\fcation,Version1.2(WireProtocol0x03).http://tinyurl.com/84kelcj[2]StationandMediaAccessControlConnectivityDiscovery,IEEEStandard802.1AB.http://tinyurl.com/6s739pe[3]M.Alizadeh,A.Greenberg,D.A.Maltz,J.Padhye,P.Patel,B.Prabhakar,S.Sengupta,andM.Sridharan.Datacentertcp(dctcp).InProceedingsoftheACMSIGCOMM2010conference,pages63{74,2010.[4]A.R.Curtis,W.Kim,andP.Yalagandula.Mahout:Low-overheaddatacentertracmanagementusingend-host-basedelephantdetection.InProceedingsoftheIEEEINFOCOM2011conference,pages1629{1637,2011.[5]A.R.Curtis,J.C.Mogul,J.Tourrilhes,P.Yalagandula,P.Sharma,andS.Banerjee.DevoFlow:scaling\rowmanagementforhigh-performancenetworks.InProceedingsoftheACMSIGCOMM2011conference,pages254{265,2011.[6]D.DecasperandB.Plattner.DAN:distributedcodecachingforactivenetworks.InProceedingsoftheIEEEINFOCOM'98conference,volume2,pages609{616,1998.[7]A.Greenhalgh,F.Huici,M.Hoerdt,P.Papadimitriou,M.Handley,andL.Mathy.Flowprocessingandtheriseofcommoditynetworkhardware.SIGCOMMComput.Commun.Rev.,39(2):20{26,2009.[8]T.Koponen,M.Casado,N.Gude,J.Stribling,L.Poutievski,M.Zhu,R.Ramanathan,Y.Iwata,H.Inoue,T.Hama,andS.Shenker.Onix:adistributedcontrolplatformforlarge-scaleproductionnetworks.InProceedingsofthe9thUSENIXOSDIconference,pages1{6,2010.[9]B.Lantz,B.Heller,andN.McKeown.Anetworkinalaptop:rapidprototypingforsoftware-de\fnednetworks.InProceedingsoftheACMSIGCOMMHotNetsworkshoppages19:1{19:6,2010.[10]C.A.B.Macapuna,C.E.Rothenberg,andF.Magalh.In-packetbloom\flterbaseddatacenternetworkingwithdistributedopen\rowcontrollers.InProceedingsofIEEEInternationalWorkshoponManagementofEmergingNetworksandServices,pages584{588,2010.[11]B.Pfa ,J.Pettit,T.Koponen,K.Amidon,M.Casado,andS.Shenker.Extendingnetworkingintothevirtualizationlayer.InProceedingsoftheACMSIGCOMMHotNetsworkshop,2009.[12]B.Schwartz,A.W.Jackson,W.T.Strayer,W.Zhou,R.D.Rockwell,andC.Partridge.Smartpackets:applyingactivenetworkstonetworkmanagement.ACMTransactionsonComputerSystems,18(1):67{88,Feb.2000.[13]V.Sekar,N.Egi,S.Ratnasamy,M.K.Reiter,andG.Shi.Designandimplementationofaconsolidatedmiddleboxarchitecture.InProceedingsofNSDI12,2012.[14]R.Sherwood,M.Chan,A.Covington,G.Gibb,M.Flajslik,N.Handigol,T.Huang,P.Kazemian,M.Kobayashi,J.Naous,S.Seetharaman,D.Underhill,T.Yabe,K.Yap,Y.Yiakoumis,H.Zeng,G.Appenzeller,R.Johari,N.McKeown,andG.Parulkar.CarvingresearchslicesoutofyourproductionnetworkswithOpenFlow.SIGCOMMCCR,40(1):129{130,Jan2010.[15]A.Shieh,S.Kandula,andE.G.Sirer.Sidecar:buildingprogrammabledatacenternetworkswithoutprogrammableswitches.InProceedingsoftheACMHotNetsworksoppages21:1{21:6,2010.[16]J.M.Smith,D.J.Farber,C.A.Gunter,S.M.Nettles,D.C.Feldmeier,andW.D.Sincoskie.SwitchWare:acceleratingnetworkevolution(Whitepaper).Technicalreport,1996.[17]A.-W.Tam,K.Xi,andH.Chao.Useofdevolvedcontrollersindatacenternetworks.InProcessingsoftheIEEEComputerCommunicationsWorkshops,pages596{601,2011.[18]A.TootoonchianandY.Ganjali.Hyper\row:adistributedcontrolplaneforopen\row.InProceedingsofthe2010INMconference,pages3{3,2010.[19]D.J.Wetherall,J.V.Guttag,andD.L.Tennenhouse.ANTS:atoolkitforbuildinganddynamicallydeployingnetworkprotocols.InProceedingsoftheIEEEOpenArchitecturesandNetworkProgrammingconference,pages117{129,1998.[20]D.J.WetherallandD.L.Tennenhouse.TheACTIVEIPoption.InProceedingsofthe7thACMSIGOPSEuropeanworkshoponsystemssupportforworldwideapplicationspages33{40,1996.[21]M.Yu,J.Rexford,M.J.Freedman,andJ.Wang.Scalable\row-basednetworkingwithDIFANE.InProceedingsoftheACMSIGCOMM2010conference,pages351{362,2010. 24 (a)Averagenumberofmessagesreceivedbythecontrollers (b)Averagenumberofbytesreceivedbythecon-trollersFigure6:ControlPlaneLoadfortheElephantFlowDetectionScenario.Theloadisbasedonthenumberofelephant\rowsinthenetwork.4.RELATEDWORKDatapathExtensions.TheproblemthatwetackleinthispaperisageneralizationofseveralpreviousattemptsatscalingSDNs.Aclassofsolutions,suchasDIFANE[21]andDevoFlow[5],addressthisproblembyextendingdataplanemechanismsofswitcheswiththeobjectiveofreducingtheloadtowardsthecontroller.DIFANEtriestopartlyooadforwardingdecisionsfromthecontrollertospecialswitches,calledauthorityswitches.Usingthisapproach,networkoperatorscanreducetheloadonthecontrollerandthelatenciesofruleinstallation.DevoFlow,ontheotherhand,introducesnewmechanismsinswitchestodispatchfarfewer\important"eventstothecontrolplane.Kandoohasthesamegoal,but,incontrasttoDIFANEandDevoFlow,itdoesnotextendswitches;instead,itmovescontrolplanefunctionsclosertoswitches.Kandoo'sapproachismoregeneralandworkswellindatacenters,butitmighthavealowerthroughputthanspeci\fcextensionsimplementedinhardware.Interestingly,wecanuseKandootoprototypeandtestDIFANE,DevoFlow,orotherpotentialhardwareextensions.Forinstance,anauthorityswitchinDIFANEcanbeemulatedbyalocalKandoocontrollerthatmanagesasubsetofswitchesinthenetwork.Asanotherexample,DevoFlow'sextensionscanalsobeemulatedusingKandoocontrollersdirectlyinstalledonswitches.ThesecontrollersnotonlyreplacethefunctionalityofDIFANEorDevoFlow,butthey (a)Averagenumberofpacketsreceivedbythecontrollers (b)AveragenumberofbytesreceivedbythecontrollersFigure7:ControlPlaneLoadfortheElephantFlowDetectionScenario.Theloadisbasedonthenumberofnodesinthenetwork.alsoprovideaplatformtorunanylocalcontrolapplicationintheircontext.DistributedControllers.HyperFlow[18],Onix[8],SiBF[10],andDevolvedControllers[17]trytodistributethecontrolplanewhilemaintaininglogicallycentralized,eventuallyconsistentnetworkstate.Althoughtheseap-proacheshavetheirownmerits,theyimposelimitationsonapplicationstheycanrun.Thisisbecausetheyassumethatallapplicationsrequirethenetwork-widestate;hence,theycannotbeofmuchhelpwhenitcomestolocalcontrolapplications.Thatsaid,thedistributedcontrollerscanbeusedtorealizeascalablerootcontroller,thecontrollerthatrunsnon-localapplicationsinKandoo.Middleboxes.Middleboxarchitectures,suchasFlow-Stream[7],SideCar[15]andCoMb[13],providescalableprogrammabilityindataplanebyintercepting\rowsusingprocessingnodesinwhichnetworkapplicationsarede-ployed.Kandooisorthogonaltotheseapproachesinthesensethatitoperatesinthecontrolplane,butitprovidesasimilardistributionforcontrolapplications.InanetworkequippedwithFlowStream,SideCarorCoMb,Kandoocansharetheprocessingresourceswithmiddleboxes(giventhatcontrolanddataplaneapplicationsareisolated)inordertoincreaseresourceutilizationanddecreasethenumberofnodesusedbyKandoo.ActiveNetworks.ActiveNetworks(AN)andSDNsrepre-sentdi erentschoolsofthoughtonprogrammablenetworks.SDNsprovideprogrammablecontrolplanes,whereasANsallowprogrammabilityinnetworkingelementsatpacket 23 system,whichsupportsapplicationdependency.Whenbootingup,acontrollerdownloadstheinformationaboutallapplicationsfromtherepositoryandthenretrievestheapplicationsitcanrun.Single-nodePerformance.Single-nodeperformanceofKandooisofsigni\fcantimportancesincelocalcontrollersprocessfrequenteventsfromsingleswitches.Forthisreason,ourimplementationisecientandhaslowoverhead.AsingleKandoocontrollercanreachathroughputofmorethan1Mpkt-inpersecondfrom512switchesusingasinglethreadonaXeonE7-4807.3.EVALUATIONWehaveevaluatedKandoousingdi erentapplicationsinanemulatedenvironment.Forthesakeofspace,wepresentonlytheresultsobtainedfortheelephant\rowdetectionproblem.ThisevaluationdemonstratesthefeasibilityofKandootodistributethecontrolplaneatscaleandstronglysupportsourargument.Setup.Inourexperiments,werealizeatwo-layerhierarchyofKandoocontrollersasshowninFigure5.Ineachexperiment,weemulateanOpenFlownetworkusingaslightlymodi\fedversionofMininet[9]hostedonaphysicalserverequippedwith64GofRAMand4IntelXeon(R)E7-4807CPUs(eachwith6cores).WeuseOpenVSwitch1.4[11]asourkernel-levelsoftwareswitch. Figure5:ExperimentSetup.Weuseasimpletreetopology.EachswitchiscontrolledbyonelocalKandoocontroller.TherootKandoocontrollermanagesthelocalcontrollers.ElephantFlowDetection.WeimplementedElephantFlowDetectionapplications(i.e.rerouteandasdescribedinSection2.AsdepictedinFigure5,isdeployedonallKandoolocalcontrollers,whereasrerouteisdeployedonlyontherootcontroller.Ourimplementationofqueriesonlythetop-of-rack(ToR)switchestodetecttheelephant\rows.TodistinguishToRswitchesfromthecoreswitch,weimplementedasimplelinkdiscoverytechnique.\fresonequeryper\rowpersecondandreportsa\rowaselephantifithassentmorethan1MBofdata.Ourimplementationofrerouteinstallsnew\rowentriesonallswitchesforthedetectedelephant\row.LearningSwitch.Inadditiontothesetwoapplications,weuseasimplelearningswitchapplicationonallcontrollerstosetuppaths.ThisapplicationassociatesMACaddressestoportsandinstallsrespective\rowentriesontheswitch.Wenotethatthebandwidthconsumedforpathsetupisnegligiblecomparedtothebandwidthconsumptionforelephant\rowdetection.Thus,ourevaluationresultswouldstillapply,evenifweinstallallpathsproactively.Methodology.Inourexperiments,weaimtostudyhowthiscontrolplanescaleswithrespecttothenumberofelephant\rowsandnetworksizecomparedtoanormalOpen-Flownetwork(whereallthreeapplicationsareonasinglecontroller).Wemeasurethenumberofrequestsprocessedbyeachcontrollerandtheirbandwidthconsumption.WenotethatourgoalistodecreasetheloadontherootcontrollerinKandoo.Localcontrollershandleeventslocally,whichconsumefarlessbandwidthcomparedtoeventssenttotherootcontroller.Moreover,Kandoo'sdesignmakesiteasytoaddlocalcontrollerswhenneeded,e ectivelymakingtherootcontrollertheonlypotentialbottleneckintermsofscalability.Our\frstexperimentstudieshowtheseapplicationsscalewithrespecttothenumberofelephant\rowsinthenetwork.Inthisexperiment,weuseatreetopologyofdepth2andfanout6(i.e.,1coreswitch,6top-of-rackswitches,and36end-hosts).Eachend-hostinitiatesonehundredUDP\rowstoanyotherhostinthenetwork.ThissyntheticworkloadstressesKandoosincemost\rowsarenotlocal.Asreportedin[3],datacentertrachaslocality,andKandoowouldthereforeperformbetterinpractice.AsdepictedinFigure6,controlchannelconsumptionandtheloadonthecentralcontrollerareconsiderablylowerforKandoo,evenwhenall\rowsareelephant:\fvetimessmallerintermsofpackets(Figure6(a)),andanorderofmagnitudeintermsofbytes(Figure6(b)).Themainreasonisthat,unlikethenormalOpenFlow,thecentralcontrollerdoesnotneedtoquerytheToRswitches.Moreover,\fresoneeventforeachelephant\row,whichresultsinsigni\fcantlylessevents.TostudyKandoo'sscalabilitybasedonthenumberofnodesinthenetwork,we\fxratiooftheelephant\rowsat20%andexperimentwithnetworksofdi erentfanouts.AsillustratedinFigure7,theroleoflocalcontrollersismorepronouncedwhenwehavelargernetworks.Thesecontrollersscalelinearlywiththesizeofthenetworkande ectivelyshieldthecontrolchannelsfromfrequentevents.Consequently,Kandoo'srootcontrollerhandlessigni\fcantlylesseventscomparedtoanormalOpenFlownetwork.Basedonthesetwoexperiments,Kandooscalessignif-icantlybetterthananormalOpenFlownetwork.ItisimportanttonotethattheperformanceofKandoocanbeimprovedfurtherbysimpleoptimizationin.Forexample,theloadonthecentralcontrollercanbehalvedifqueriesonlythe\row-entriesinitiatedbythehostsdirectlyconnectedtotherespectiveswitch. Whenall\rowsareelephant,\fresaneventforeach\row.Thus,itre\rectsthemaximumnumberofeventsKandoowill\freforelephant\rowdetection. 22 Figure3:Kandoo'shighlevelarchitecture.DeploymentModel.ThedeploymentmodelofKandoocontrollersdependsonthecharacteristicsofanetwork.Forsoftwareswitches,localcontrollerscanbedirectlydeployedonthesameend-host.Similarly,ifwecanchangethesoftwareofaphysicalswitch,wecandeployKandoodirectlyontheswitch.Otherwise,wedeployKandoolocalcontrollersontheprocessingresourcesclosesttotheswitches.Insuchasetting,oneshouldprovisionthenumberoflocalcontrollersbasedontheworkloadandavailableprocessingresources.Notethatwecanuseahybridmodelinrealsettings.Forinstance,consideravirtualizeddeploymentenvironmentdepictedinFigure4,wherevirtualmachinesareconnectedtothenetworkusingsoftwareswitches.Inthisenvironment,wecanplacelocalcontrollersinend-hostsnexttosoftwareswitchesandinseparatenodesforotherswitches.Inourtoyexample(Figure2),wehavefourKandoocontrollers:threelocalcontrollerscontrollingtheswitchesandarootcontroller.Thelocalcontrollerscanbephysicallypositionedusinganydeploymentmodelexplainedabove.Notethat,inthisexample,wehavethemaximumnumberoflocalcontrollersrequired.ControlApplications.ControlapplicationsfunctionusingtheabstractionprovidedbythecontrollerandarenotawareofKandoointernals.TheyaregenerallyOpenFlowapplicationsandcanthereforesendOpenFlowmessagesandlistenonevents.Moreover,theycanemitKandooevents(i.e.,internalevents),whichcanbeconsumedbyotherapplications,andtheycanreplytotheapplicationthatemittedanevent.ControlapplicationsareloadedinlocalnamespacesandcancommunicateusingonlyKandooevents.ThisistoensurethatKandoodoesnotintroducefaultsbyooadingapplications.Inourexample,isaKandooeventthatcarriesmatchinginformationaboutthedetectedelephant\row(e.g.itsOpenFlowmatchstructure)andisemittedbyAlocalcontrollercanrunanapplicationonlyiftheapplicationislocal.Inourexample,rerouteisnotlocal,i.e.,itmayinstall\row-entiresonanyswitchinthenetwork.Thus,therootcontrolleristheonlycontrollerabletorunreroute.Incontrast,islocal;therefore,allcontrollerscanrunit.EventPropagation.Therootcontrollercansubscribe Figure4:Kandooinavirtualizedenvironment.Forsoftwareswitches,wecanleveragethesameend-hostforlocalcontrollers,and,forphysicalswitches,weuseseparateprocessingresources.tospeci\fceventsinthelocalcontrollersusingasimplemessagingchannelplusa\flteringcomponent.Oncethelocalcontrollerreceivesandlocallyprocessesanevent,itrelaystheeventtotherootcontrollerforfurtherprocessing.NotethatallcommunicationsbetweenKandoocontrollersareevent-basedandasynchronous.Inourexample,therootcontrollersubscribestoeventsoftypeinthelocalcontrollerssinceitisrunningreroutelisteningonis\fredbyaninstancedeployedononeofthelocalcontrollersandisrelayedtorootcontroller.Notethatiftherootcontrollerdoesnotsubscribeto,thelocalcontrollerswillnotrelayevents.Itisimportanttonotethatthedata\rowinKandooisnotalwaysbottom-up.Alocalapplicationcanexplicitlyrequestdatafromanapplicationdeployedontherootcontrollerbyemittinganevent,andapplicationsontherootcontrollerscansenddatabyreplyingtothatevent.Forinstance,wecanhaveatopologyservicerunningontherootcontrollerthatsendstopologyinformationtolocalapplicationsbyreplyingtoeventsofaspeci\fctype.Reactivevs.Proactive.AlthoughKandooprovidesascalablemethodforeventhandling,westronglyrecommendpushingnetworkstateproactively.WeenvisionKandootobeusedasascalable,adaptivecontrolplane,wherethedefaultcon\fgurationispushedproactivelyandisadaptivelyre\fnedafterwards.Inourtoyexample,defaultpathscanbepushedproactively,whileelephant\rowswillbereroutedadaptively.ImplementationDetails.WeimplementedKandooinamixtureofC,C++,andPython.Ourimplementationhasalowmemoryfootprintandsupportsdynamicallyloadableplug-ins,whichcanbeimplementedinC,Python,andJava.ItalsoprovidesanRPCAPIformoregeneralintegrationscenarios.OurimplementationofKandooisextremelymodular;anycomponentorback-endcanbeeasilyreplaced,whichsimpli\fesportingKandootophysicalswitches.Currently,KandoosupportsOpenFlow1.0(OpenFlow1.1and1.2supportisunderdevelopment).Fortheapplications,wecreateda\centralapplicationrepository"anddevelopedasimplepackagemanagement 21 Figure1:Kandoo'sTwoLevelsofControllers.Localcontrollershandlefrequentevents,whilealogicallycentralizedrootcontrollerhandlesrareevents.controlapplications.AsillustratedinFigure1,severallocalcontrollersaredeployedthroughoutthenetwork;eachofthesecontrollerscontrolsoneorahandfulofswitches.Therootcontroller,ontheotherhand,controlsalllocalcontrollers.Itiseasytorealizelocalcontrollerssincetheyaremerelyswitchproxiesfortherootcontroller,andtheydonotneedthenetwork-widestate.TheycanevenbeimplementeddirectlyinOpenFlowswitches.Interestingly,localcontrollerscanlinearlyscalewiththenumberofswitchesinanetwork.Thus,thecontrolplanescalesaslongasweprocessfrequenteventsinlocalapplicationsandshieldtherootcontrollerfromthesefrequentevents.Needlesstosay,Kandoocannothelpanycontrolapplicationsthatrequirenetwork-widestate(eventhoughitdoesnothurtthem,either).Webelievesuchapplicationsareintrinsicallyhardtoscale,andsolutionslikeOnix[8]andHyperFlow[18]providetherightframeworksforrunningsuchapplications.OurimplementationofKandooiscompletelycompliantwiththeOpenFlowspeci\fcations.DataandcontrolplanesaredecoupledinKandoo.Switchescanoperatewithouthavingalocalcontroller;controlapplicationsfunctionregardlessoftheirphysicallocation.ThemainadvantageofKandooisthatitgivesnetworkoperatorsthefreedomtocon\fgurethedeploymentmodelofcontrolplanefunctionalitiesbasedonthecharacteristicsofcontrolapplications.ThedesignandimplementationofKandooarepresentedinSections2.Ourexperimentscon\frmthatKandooscalesanorderofmagnitudebetterthananormalOpenFlownetworkandwouldleadtomorethan90%ofeventsbeingprocessedlocallyunderreasonableassumptions,asdescribedinSection3.ApplicationsofKandooarenotlimitedtotheevaluationscenariospresentedinthispaper.InSection4,webrie\rydiscussotherpotentialapplicationsofKandooandcompareittoexistingsolutions.WeconcludeourdiscussioninSection5.2.DESIGNANDIMPLEMENTATIONDesignobjectives.Kandooisdesignedwiththefollowinggoalsinmind.First,KandoomustbecompatiblewithOpenFlow:wedonotintroduceanynewdataplanefunctionalityinswitches,and,aslongastheysupportOpenFlow,Kandoosupportsthem,aswell.Second,Kandooautomaticallydistributescontrolapplicationswithoutanymanualintervention.Inotherwords,Kandoocontrol Figure2:ToyexampleforKandoo'sdesign:Inthisexample,twohostsareconnectedusingasimplelinetopology.EachswitchiscontrolledbyonelocalKandoocontroller.Therootcontrollercontrolsthelocalcontrollers.Inthisexample,wehavetwocontrolapplications:isalocalcontrolapplication,butrerouteisnon-local.applicationsarenotawareofhowtheyaredeployedinthenetwork,andapplicationdeveloperscanassumetheirapplicationswouldberunonacentralizedOpenFlowcontroller.TheonlyextrainformationKandooneedsisa\ragshowingwhetheracontrolapplicationislocalornot.Inwhatfollows,weexplainKandoo'sdesignusingatoyexample.WeshowhowKandoocanbeusedtorerouteelephant\rowsinasimplenetworkofthreeswitches(Figure2).Ourexamplehastwoapplications:(and(iirerouteconstantlyquerieseachswitchtodetectelephant\rows.Onceanelephant\rowisdetected,noti\fesreroute,whichinturnmayinstallorupdate\row-entriesonnetworkswitches.Itisextremelychallenging,ifnotimpossible,toimplementthisapplicationincurrentOpenFlownetworkswithoutmodifyingswitches[5].Ifswitchesarenotmodi\fed,a(logically)centralizedcontrolneedstofrequentlyqueryallswitches,whichwouldplaceaconsiderableloadoncontrolchannels.KandooController.AsshowninFigure3,Kandoohascontrollercomponentatitscore.ThiscomponenthasthesameroleasageneralOpenFlowcontroller,butithasKandoo-speci\fcextensionsforidentifyingapplicationrequirements,hidingthecomplexityoftheunderlyingdistributedapplicationmodel,andpropagatingeventsinthenetwork.AnetworkcontrolledbyKandoohasmultiplelocalcontrollersandalogicallycentralizedrootcontrollerThesecontrollerscollectivelyformKandoo'sdistributedcontrolplane.EachswitchiscontrolledbyonlyoneKandoocontroller,andeachKandoocontrollercancontrolmultipleswitches.Iftherootcontrollerneedstoinstall\row-entriesonswitchesofalocalcontroller,itdelegatestherequeststotherespectivelocalcontroller.Notethatforhighavailability,therootcontrollercanregisteritselfastheslavecontrollerforaspeci\fcswitch(thisbehaviorissupportedinOpenFlow1.2[1]). WenotethattherootcontrollerinKandoocanitselfbelogically/physicallydistributed.Infact,itisstraightforwardtoimplementKandoo'srootcontrollerusingOnix[8]orHyper\row[18]. 20 Kandoo:AFrameworkforEfcientandScalableOfoadingofControlApplicationsSoheilHassasYeganehUniversityofTorontosoheil@cs.toronto.eduYasharGanjaliUniversityofTorontoyganjali@cs.toronto.eduABSTRACTLimitingtheoverheadoffrequenteventsonthecontrolplaneisessentialforrealizingascalableSoftware-De\fnedNetwork.Onewayoflimitingthisoverheadistoprocessfrequenteventsinthedataplane.Thisrequiresmodifyingswitchesandcomesatthecostofvisibilityinthecontrolplane.Takinganalternativeroute,weproposeKandoo,aframeworkforpreservingscalabilitywithoutchangingswitches.Kandoohastwolayersofcontrollers:()thebot-tomlayerisagroupofcontrollerswithnointerconnection,andnoknowledgeofthenetwork-widestate,and(ii)thetoplayerisalogicallycentralizedcontrollerthatmaintainsthenetwork-widestate.Controllersatthebottomlayerrunonlylocalcontrolapplications(i.e.,applicationsthatcanfunctionusingthestateofasingleswitch)neardatapaths.Thesecontrollershandlemostofthefrequenteventsande ectivelyshieldthetoplayer.Kandoo'sdesignenablesnetworkoperatorstoreplicatelocalcontrollersondemandandrelievetheloadonthetoplayer,whichistheonlypotentialbottleneckintermsofscalability.OurevaluationsshowthatanetworkcontrolledbyKandoohasanorderofmagnitudelowercontrolchannelconsumptioncomparedtonormalOpenFlownetworks.CategoriesandSubjectDescriptorsC.2[Computer-communicationnetworks]:NetworkArchitectureandDesignKeywordsSoftware-De\fnedNetworks,DataCenterNetworks,Dis-tributedControlPlane1.INTRODUCTIONFrequentandresource-exhaustiveevents,suchas\rowarrivalsandnetwork-widestatisticscollectionevents,stressthecontrolplaneandconsequentlylimitthescalabilityofOpenFlownetworks[5,18,4].AlthoughonecanPermissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforprotorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitationontherstpage.Tocopyotherwise,torepublish,topostonserversortoredistributetolists,requirespriorspecicpermissionand/orafee.August13,2012,Helsinki,Finland.Copyright2012ACM978-1-4503-1477-0/12/08...$15.00.suppress\rowarrivalsbyproactivelypushingthenetworkstate,thisapproachfallsshortwhenitcomestoothertypesoffrequenteventssuchasnetwork-widestatisticscollection.Currentsolutionseitherviewthisasanintrinsiclimitationortrytoaddressitbymodifyingswitches.Forinstance,HyperFlow[18]canhandleafewthousandeventspersecond,andanythingbeyondthatisconsideredascalabilitylimitation.Incontrast,DIFANE[21]andDevoFlow[5]introducenewfunctionalitiesinswitchestosuppressfrequenteventsandtoreducetheloadonthecontrolplane.Tolimittheloadonthecontroller,frequenteventsshouldbehandledintheclosestvicinityofdatapaths,preferablywithoutmodifyingswitches.Addingnewprimitivestoswitchesisundesirable.ItbreaksthegeneralprinciplesofSoftware-De\fnedNetworks(SDNs),requireschangestothestandards,andnecessitatesthecostlyprocessofmodifyingswitches.Thus,theimportantquestionis:\Howcanwemovecontrolfunctionalitiestowarddatapaths,withoutintroducingnewdatapathmechanismsinswitches?Toanswerthisquestion,wefocuson:()environmentswhereprocessingpowerisreadilyavailableclosetoswitches(suchasdatacenternetworks)orcanbeeasilyadded(suchasenterprisenetworks),and(ii)applicationsthatarelocalinscope,i.e.,applicationsthatprocesseventsfromasingleswitchwithoutusingthenetwork-widestate.Weshowthatunderthesetwoconditionsonecanooadlocaleventprocessingtolocalresources,andthereforerealizeacontrolplanethathandlesfrequenteventsatscale.Wenotethatsomeapplications,suchasrouting,requirethenetwork-widestate,andcannotbeooadedtolocalprocessingresources.However,alargeclassofapplicationsareeitherlocal(e.g.,localpolicyenforcerandLinkLayerDiscoveryProtocol[2])orcanbedecomposedtomodulesthatarelocal(e.g.,elephant\rowdetectionmoduleinanelephant\rowreroutingapplication).Kandoo.Inthispaper,wepresentthedesignandimplementationofKandoo,anoveldistributedcontrolplanethatooadscontrolapplicationsoveravailableresourcesinthenetworkwithminimaldeveloperinterventionandwithoutviolatinganyrequirementsofcontrolapplications.Kandoo'scontrolplaneessentiallydistinguisheslocalcontrolapplicationsi.e.,applicationsthatprocesseventslocally)fromnon-localapplicationsi.e.,applicationsthatrequireaccesstothenetwork-widestate).Kandoocreatesatwo-levelhierarchyforcontrollers:(localcontrollersexecutelocalapplicationsascloseaspossibletoswitches,andii)alogicallycentralizedrootcontrollerrunsnon-local 19